-----============= acceptance-small: sanity-flr ============----- Mon Aug 31 14:36:57 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg627-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 14:37:15 (1788201435) === oleg627-client.virtnet: executing check_config_client /mnt/lustre oleg627-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg627-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9049a0148800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9049a0148800.idle_timeout=debug disable quota as required oleg627-server: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 14:37:52 (1788201472) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8217] == sanity-flr test 0a: lfs mirror create with -N option == 14:37:54 (1788201474) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:38:11 (1788201491) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:38:14 (1788201494) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:38:17 (1788201497) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:38:39 (1788201519) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:38:41 (1788201521) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 14:38:44 (1788201524) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9049a0148800.stats=clear osc.lustre-OST0001-osc-ffff9049a0148800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:38:52 (1788201532) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:39:06 (1788201546) PASS 0j (12s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:39:19 (1788201559) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (5s) == sanity-flr test 0m: mirror extend with -N= option ===== 14:39:24 (1788201564) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:39:32 (1788201572) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 14:39:41 (1788201581) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:39:48 (1788201588) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:39:56 (1788201596) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:40:04 (1788201604) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:40:15 (1788201615) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:40:22 (1788201622) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.22806 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746382 s, 14.0 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:40:31 (1788201631) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.250159 s, 37.7 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 14:40:40 (1788201640) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:40:49 (1788201649) Stopping /mnt/lustre-ost1 (opts:) on oleg627-server oleg627-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9049a0148800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9049a0148800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg627-server: oleg627-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg627-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg627-server oleg627-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9049a0148800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9049a0148800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg627-server: oleg627-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg627-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (69s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:41:58 (1788201718) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.546698 s, 15.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg627-server oleg627-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9049a0148800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9049a0148800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg627-server: oleg627-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg627-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:42:44 (1788201764) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg627-server oleg627-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9049a0148800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9049a0148800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg627-server: oleg627-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg627-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg627-server oleg627-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9049a0148800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9049a0148800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg627-server: oleg627-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg627-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (77s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:44:02 (1788201842) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.97852 s, 17.6 MB/s -rw-r--r-- 1 root root 52428800 Aug 31 14:44 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.048s user 0m0.001s sys 0m0.169s Stopping /mnt/lustre-ost1 (opts:) on oleg627-server oleg627-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9049a0148800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9049a0148800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.107s user 0m0.003s sys 0m0.185s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg627-server: oleg627-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg627-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg627-server oleg627-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9049a0148800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9049a0148800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.017s user 0m0.002s sys 0m0.165s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg627-server: oleg627-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg627-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg627-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9049a0148800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9049a0148800.ost_server_uuid in FULL state after 0 sec PASS 33b (93s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:45:34 (1788201934) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:45:38 (1788201938) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:45:42 (1788201942) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 14:45:46 (1788201946) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 14:45:55 (1788201955) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.566028 s, 22.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:46:09 (1788201969) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773538 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696853 s, 15.0 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:46:18 (1788201978) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:46:26 (1788201986) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00139522 s, 734 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00196904 s, 520 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00314919 s, 325 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000954955 s, 1.1 MB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 14:46:37 (1788201997) create test files with size 15743440 .. feb7f69c0ce0c781567c8f46f8d77078 - 4ca3ec6bfe3cfa8793bfc6ee8505ef98 - a90843e3584c48620496c3915cfb846f - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg627-client.virtnet /mnt/lustre (opts:) Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 14:47:00 (1788202020) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:47:13 (1788202033) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:47:19 (1788202039) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695803 s, 15.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0293582 s, 35.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 14:47:28 (1788202048) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0847246 s, 24.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.277283 s, 15.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.160768 s, 13.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 14:47:42 (1788202062) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 14:47:45 (1788202065) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:47:48 (1788202068) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617964 s, 17.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 14:47:56 (1788202076) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.261781 s, 12.0 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:48:07 (1788202087) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:48:18 (1788202098) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.310506 s, 33.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg627-client.virtnet /mnt/lustre (opts:) Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:48:32 (1788202112) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff904982fc0800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.450082 s, 23.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg627-client.virtnet /mnt/lustre (opts:) Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff904982be1000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:48:49 (1788202129) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:49:26 (1788202166) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (43s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:50:09 (1788202209) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 14:50:12 (1788202212) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.231026 s, 13.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0823895 s, 38.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x41b27a0f (nosync,stale) Mirror 2: 0x41b27a0f (nosync,stale) ** verify mirror 2 doesn't change original checksum: 1cb0d3b81704d9310b2fdf24ef2c587d - mirror 1 checksum: de424a713e02416503f584796b41bd67 - mirror 2 checksum: 1cb0d3b81704d9310b2fdf24ef2c587d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-31T14:50:16-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-31T14:50:16-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 1cb0d3b81704d9310b2fdf24ef2c587d - mirror 1 checksum: de424a713e02416503f584796b41bd67 - mirror 2 checksum: de424a713e02416503f584796b41bd67 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-31T14:50:16-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-31T14:50:16-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 14:50:21 (1788202221) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.028359 s, 37.0 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (6s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:50:27 (1788202227) Failing mds1 on oleg627-server Stopping /mnt/lustre-mds1 (opts:) on oleg627-server 14:50:39 (1788202239) shut down facet: mds1 facet_host: oleg627-server facet_failover_host: oleg627-server Failover mds1 to oleg627-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg627-server: oleg627-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-MDT0000 14:51:02 (1788202262) targets are mounted 14:51:02 (1788202262) facet_failover done oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172448 s, 60.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } PASS 50A (50s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:51:17 (1788202277) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0456574 s, 449 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0117574 s, 1.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:51:26 (1788202286) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0559657 s, 366 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00869458 s, 2.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:51:35 (1788202295) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00903858 s, 1.8 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:51:42 (1788202302) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0876378 s, 234 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:51:49 (1788202309) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00435203 s, 941 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0257791 s, 159 kB/s PASS 60a (6s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:51:55 (1788202315) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:52:16 (1788202336) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:52:44 (1788202364) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x48:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:53:08 (1788202388) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:53:31 (1788202411) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 14:53:37 (1788202417) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57542 mirror split pid 57543 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57542 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 57543 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 14:54:43 (1788202483) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 2485 skipping zero size read skipping zero size read fallocating to largest ever: 0x40000 000003 1788202485.332408 fallocate 0x030e7c thru 0x03ffff (0xf184 bytes) skipping zero size read skipping zero size read 000006 1788202485.342276 write 0x034f56 thru 0x03b49f (0x654a bytes) 000008 1788202485.348269 trunc from 0x03b4a0 to 0x03e98e (0x34ef bytes) 000009 1788202485.380710 mapread 0x00e4cf thru 0x017f59 (0x9a8b bytes) 000010 1788202485.419355 fallocate 0x02f00b thru 0x03a128 (0xb11e bytes) 000011 1788202485.428443 mapread 0x00c1d0 thru 0x00d510 (0x1341 bytes) 000013 1788202485.433231 trunc from 0x013aa2 to 0x01bda3 (0x8302 bytes) 000014 1788202485.441869 trunc from 0x0093de to 0x00cd3a (0x395d bytes) 000015 1788202485.447716 punch from 0x009eb0 to 0x017f26 (0xe077 bytes) 000016 1788202485.453589 punch from 0x029359 to 0x038b9f (0xf847 bytes) 000017 1788202485.472398 fallocate 0x03b924 thru 0x03ffff (0x46dc bytes) 000018 1788202485.488068 punch from 0x01cf7a to 0x01d54b (0x5d2 bytes) 000019 1788202485.501173 write 0x021c35 thru 0x0234f3 (0x18bf bytes) 000020 1788202485.526891 trunc from 0x00856b to 0x00bac6 (0x355c bytes) 000021 1788202485.537348 fallocate 0x00394e thru 0x00649c (0x2b4f bytes) 000023 1788202485.551514 fallocate 0x01ca8d thru 0x0263cd (0x9941 bytes) 000024 1788202485.565613 mapread 0x03009d thru 0x039f61 (0x9ec5 bytes) 000026 1788202485.605335 fallocate 0x0216e6 thru 0x02e94d (0xd268 bytes) 000027 1788202485.619250 fallocate 0x0215ad thru 0x02d355 (0xbda9 bytes) 000028 1788202485.643656 MIRROR_EXTEND to 2 mirrors 000029 1788202486.392883 mapwrite 0x00f3cc thru 0x01de98 (0xeacd bytes) 000030 1788202486.507313 read 0x01ca84 thru 0x01e8b4 (0x1e31 bytes) 000031 1788202486.508557 mapread 0x00d175 thru 0x01c86f (0xf6fb bytes) 000032 1788202486.519271 MIRROR_RESYNC 2 mirrors 000033 1788202487.169088 MIRROR_RESYNC 2 mirrors 000034 1788202487.565474 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000035 1788202487.851749 trunc from 0x00b497 to 0x01991a (0xe484 bytes) 000036 1788202487.878880 mapread 0x0252ae thru 0x03433d (0xf090 bytes) 000037 1788202487.942632 fallocate 0x0207ca thru 0x0278e1 (0x7118 bytes) 000038 1788202487.954561 trunc from 0x01ff32 to 0x02ac8f (0xad5e bytes) 000039 1788202487.960081 write 0x0287e7 thru 0x02e188 (0x59a2 bytes) 000040 1788202487.970412 trunc from 0x004c97 to 0x00b923 (0x6c8d bytes) 000041 1788202487.975225 mapwrite 0x03ba15 thru 0x03ffff (0x45eb bytes) 000042 1788202488.045186 read 0x02dc5e thru 0x03aa53 (0xcdf6 bytes) 000043 1788202488.047079 punch from 0x03a81d to 0x03ffff (0x57e3 bytes) 000044 1788202488.054037 write 0x0303b9 thru 0x039e47 (0x9a8f bytes) 000045 1788202488.056123 read 0x01f14c thru 0x0295fc (0xa4b1 bytes) 000046 1788202488.092372 mapwrite 0x0099bb thru 0x010f6f (0x75b5 bytes) 000047 1788202488.114511 trunc from 0x01c292 to 0x023cd1 (0x7a40 bytes) 000049 1788202488.149038 mapwrite 0x001f5b thru 0x0069c3 (0x4a69 bytes) 000051 1788202488.177308 trunc from 0x028d8f to 0x0371a7 (0xe419 bytes) 000053 1788202488.184302 fallocate 0x02f487 thru 0x030804 (0x137e bytes) 000054 1788202488.189390 read 0x0180d3 thru 0x01d896 (0x57c4 bytes) 000055 1788202488.198065 trunc from 0x0180bd to 0x01db8f (0x5ad3 bytes) 000056 1788202488.204973 punch from 0x030728 to 0x03c0e1 (0xb9ba bytes) 000057 1788202488.210703 trunc from 0x03b3aa to 0x03ffff (0x4c56 bytes) 000058 1788202488.216381 read 0x009662 thru 0x0120b5 (0x8a54 bytes) 000059 1788202488.217197 mapread 0x038821 thru 0x03ec9c (0x647c bytes) 000060 1788202488.228849 mapwrite 0x008512 thru 0x016c8c (0xe77b bytes) 000061 1788202488.253269 mapwrite 0x00c7fb thru 0x01183e (0x5044 bytes) 000062 1788202488.277468 punch from 0x03b868 to 0x03ffff (0x4798 bytes) 000063 1788202488.285695 punch from 0x007d05 to 0x009910 (0x1c0c bytes) 000064 1788202488.296296 trunc from 0x040000 to 0x01b31a (0x24ce5 bytes) 000067 1788202488.339457 mapwrite 0x020dd9 thru 0x021494 (0x6bc bytes) 000069 1788202488.430714 read 0x003645 thru 0x00ccbf (0x967b bytes) 000070 1788202488.459653 punch from 0x01b660 to 0x01bb09 (0x4aa bytes) 000071 1788202488.469757 punch from 0x00b1e1 to 0x0110b0 (0x5ed0 bytes) 000072 1788202488.480180 mapread 0x00f4e4 thru 0x017701 (0x821e bytes) 000073 1788202488.517707 trunc from 0x021495 to 0x01511c (0xc378 bytes) 000074 1788202488.551989 read 0x013cd7 thru 0x01511c (0x1446 bytes) 000075 1788202488.567063 MIRROR_EXTEND to 2 mirrors 000076 1788202489.440756 trunc from 0x00772b to 0x00a613 (0x2ee9 bytes) 000077 1788202489.511306 trunc from 0x01511d to 0x00c1d7 (0x8f45 bytes) 000078 1788202489.558388 read 0x004367 thru 0x00c1d7 (0x7e71 bytes) 000079 1788202489.568186 read 0x0082ba thru 0x00c1d7 (0x3f1e bytes) 000080 1788202489.572505 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xc1d8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc1d8): Mirror 2: 0x734177f2 (stale) CRC-32 checksum value for chunk [0, 0xc1d8): Mirror 1: 0x3dacfd3f Mirror 3: 0x3dacfd3f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc1d8: skipped 000081 1788202490.310453 read 0x002af9 thru 0x00c1d7 (0x96df bytes) 000082 1788202490.323136 write 0x01e101 thru 0x02db96 (0xfa96 bytes) 000083 1788202490.400920 mapwrite 0x02be77 thru 0x031ca8 (0x5e32 bytes) 000084 1788202490.514987 mapread 0x02e8f5 thru 0x030b4b (0x2257 bytes) 000085 1788202490.516079 mapwrite 0x01fc22 thru 0x020042 (0x421 bytes) 000086 1788202490.534981 read 0x0108b4 thru 0x017660 (0x6dad bytes) 000087 1788202490.547529 mapread 0x021864 thru 0x0229ef (0x118c bytes) 000088 1788202490.551458 trunc from 0x031ca9 to 0x01134f (0x20959 bytes) 000089 1788202490.583886 fallocate 0x02dcd4 thru 0x039763 (0xba90 bytes) 000090 1788202490.590408 mapread 0x010827 thru 0x01134f (0xb29 bytes) 000091 1788202490.591609 trunc from 0x0036d9 to 0x0083ea (0x4d12 bytes) 000092 1788202490.600414 fallocate 0x01d83e thru 0x01fcee (0x24b1 bytes) 000093 1788202490.615942 read 0x00c0a7 thru 0x01134f (0x52a9 bytes) 000095 1788202490.617374 read 0x0051c7 thru 0x01134f (0xc189 bytes) 000096 1788202490.633622 trunc from 0x011350 to 0x00b61a (0x5d35 bytes) 000097 1788202490.666085 mapwrite 0x0381f8 thru 0x03ffff (0x7e08 bytes) 000100 1788202490.750767 fallocate 0x013629 thru 0x01d64c (0xa024 bytes) 000101 1788202490.774757 trunc from 0x040000 to 0x00088f (0x3f770 bytes) 000102 1788202490.800120 punch from 0x0001ba to 0x00088f (0x6d6 bytes) 000103 1788202490.810491 trunc from 0x000890 to 0x02b4fc (0x2ac6d bytes) 000104 1788202490.839121 mapread 0x013966 thru 0x014c93 (0x132e bytes) 000105 1788202490.864920 write 0x018033 thru 0x01e26e (0x623c bytes) 000106 1788202490.884440 write 0x00507d thru 0x014ca5 (0xfc29 bytes) 000108 1788202490.895129 read 0x01f795 thru 0x02a026 (0xa892 bytes) 000109 1788202490.921650 fallocate 0x02c6d4 thru 0x0377d5 (0xb102 bytes) 000112 1788202490.940030 mapwrite 0x01a39d thru 0x021345 (0x6fa9 bytes) 000113 1788202490.970648 mapread 0x000895 thru 0x000d27 (0x493 bytes) 000114 1788202490.977729 mapread 0x00f5d6 thru 0x01244b (0x2e76 bytes) 000115 1788202490.991789 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b4fd): Mirror 3: 0xf39e9e56 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b4fd): Mirror 3: 0xf39e9e56 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b4fd: skipped 000116 1788202491.428086 mapwrite 0x00376d thru 0x00b1ec (0x7a80 bytes) 000117 1788202491.514277 read 0x009923 thru 0x00b28a (0x1968 bytes) 000118 1788202491.518102 mapread 0x009640 thru 0x017f52 (0xe913 bytes) 000119 1788202491.537688 trunc from 0x02b4fd to 0x03f221 (0x13d25 bytes) 000121 1788202491.566374 fallocate 0x0239df thru 0x0256b2 (0x1cd4 bytes) 000122 1788202491.572023 punch from 0x0171e7 to 0x01f94b (0x8765 bytes) 000123 1788202491.576182 write 0x035299 thru 0x036997 (0x16ff bytes) 000124 1788202491.603510 MIRROR_RESYNC 2 mirrors 000125 1788202492.457937 MIRROR_EXTEND to 3 mirrors 000126 1788202493.445446 trunc from 0x03f222 to 0x00d42c (0x31df5 bytes) 000127 1788202493.523547 read 0x009d60 thru 0x00d42c (0x36cd bytes) 000128 1788202493.540658 punch from 0x0087be to 0x00af82 (0x27c5 bytes) 000129 1788202493.583286 MIRROR_RESYNC 3 mirrors 000130 1788202494.291134 MIRROR_RESYNC 3 mirrors 000131 1788202494.626465 punch from 0x008619 to 0x009f1d (0x1905 bytes) 000132 1788202494.684486 write 0x026831 thru 0x031e31 (0xb601 bytes) 000133 1788202494.717640 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31e32): Mirror 4: 0xc529198a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31e32): Mirror 4: 0xc529198a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31e32: skipped 000134 1788202494.999829 punch from 0x02cb20 to 0x02d167 (0x648 bytes) 000135 1788202495.015214 fallocate 0x016290 thru 0x01cce6 (0x6a57 bytes) 000136 1788202495.026542 write 0x02fd26 thru 0x03e4b1 (0xe78c bytes) 000137 1788202495.046324 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e4b2): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e4b2): Mirror 4: 0x5527678f (stale) CRC-32 checksum value for chunk [0, 0x3e4b2): Mirror 1: 0xbf484fac Mirror 5: 0xbf484fac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e4b2: skipped 000138 1788202495.984364 punch from 0x009d12 to 0x018c20 (0xef0f bytes) 000140 1788202496.024479 fallocate 0x0127fc thru 0x01803f (0x5844 bytes) 000141 1788202496.034868 mapwrite 0x01f6b1 thru 0x02b1e5 (0xbb35 bytes) 000143 1788202496.107325 mapread 0x03578d thru 0x0391e8 (0x3a5c bytes) 000145 1788202496.128624 write 0x0145fa thru 0x01f744 (0xb14b bytes) 000146 1788202496.141448 trunc from 0x00e9da to 0x01980d (0xae34 bytes) 000147 1788202496.157033 trunc from 0x03e4b2 to 0x03dc06 (0x8ab bytes) 000148 1788202496.182415 mapwrite 0x001ab0 thru 0x002648 (0xb99 bytes) 000152 1788202496.210051 write 0x03cd33 thru 0x03ffff (0x32cd bytes) 000153 1788202496.212113 read 0x01c3a9 thru 0x01f67c (0x32d4 bytes) 000154 1788202496.213013 trunc from 0x040000 to 0x012556 (0x2daa9 bytes) 000155 1788202496.237464 mapread 0x002c7e thru 0x007973 (0x4cf6 bytes) 000156 1788202496.241284 fallocate 0x01bab4 thru 0x029763 (0xdcb0 bytes) 000157 1788202496.252692 mapwrite 0x0087a3 thru 0x014655 (0xbeb3 bytes) 000158 1788202496.337343 mapwrite 0x00c3ec thru 0x01496e (0x8583 bytes) 000159 1788202496.382214 trunc from 0x01496f to 0x020513 (0xbba5 bytes) 000160 1788202496.401020 mapread 0x00e101 thru 0x014699 (0x6599 bytes) 000161 1788202496.402663 trunc from 0x00036f to 0x007073 (0x6d05 bytes) 000162 1788202496.411265 fallocate 0x01de49 thru 0x02d954 (0xfb0c bytes) 000163 1788202496.421900 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d955): Mirror 4: 0x76df5a34 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d955): Mirror 4: 0x76df5a34 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d955: skipped 000164 1788202496.739798 mapwrite 0x035481 thru 0x03772d (0x22ad bytes) 000165 1788202496.823598 write 0x00b6f1 thru 0x00e445 (0x2d55 bytes) 000166 1788202496.841071 trunc from 0x000194 to 0x004a36 (0x48a3 bytes) 000167 1788202496.848638 mapwrite 0x00fa85 thru 0x01c7fd (0xcd79 bytes) 000168 1788202496.901121 mapwrite 0x000b30 thru 0x00859a (0x7a6b bytes) 000169 1788202496.935552 trunc from 0x03772e to 0x012fe9 (0x24744 bytes) 000170 1788202496.956448 trunc from 0x00a4f6 to 0x012fe9 (0x8af4 bytes) 000171 1788202496.968164 write 0x03c14e thru 0x03cc93 (0xb46 bytes) 000172 1788202496.969559 read 0x02ed81 thru 0x03576c (0x69ec bytes) 000173 1788202496.982123 trunc from 0x03cc94 to 0x000284 (0x3ca0f bytes) 000174 1788202497.019684 MIRROR_RESYNC 2 mirrors 000175 1788202497.636577 MIRROR_RESYNC 2 mirrors 000176 1788202498.030377 fallocate 0x0143a3 thru 0x0168f0 (0x254e bytes) 000177 1788202498.068293 write 0x01bcf9 thru 0x01cfa1 (0x12a9 bytes) 000178 1788202498.071073 fallocate 0x01730c thru 0x021323 (0xa018 bytes) 000179 1788202498.077075 trunc from 0x00a878 to 0x017df6 (0xd57f bytes) 000180 1788202498.083249 punch from 0x020275 to 0x021323 (0x10af bytes) 000181 1788202498.087924 mapread 0x016887 thru 0x021323 (0xaa9d bytes) 000183 1788202498.127870 write 0x038d26 thru 0x03ffff (0x72da bytes) 000184 1788202498.135357 MIRROR_RESYNC 2 mirrors 000185 1788202498.812973 trunc from 0x040000 to 0x02f115 (0x10eea bytes) 000186 1788202498.913213 mapread 0x01a492 thru 0x02677f (0xc2ee bytes) 000187 1788202498.951209 punch from 0x004ee3 to 0x005e0e (0xf2c bytes) 000188 1788202498.961566 mapread 0x014159 thru 0x021a7d (0xd925 bytes) 000190 1788202498.972159 trunc from 0x02f116 to 0x01af82 (0x14193 bytes) 000191 1788202499.011420 mapwrite 0x009fd2 thru 0x013b20 (0x9b4f bytes) 000192 1788202499.045993 fallocate 0x01902c thru 0x026582 (0xd557 bytes) 000193 1788202499.051462 punch from 0x021df7 to 0x026582 (0x478c bytes) 000194 1788202499.056447 trunc from 0x026583 to 0x00998b (0x1cbf7 bytes) 000195 1788202499.080717 fallocate 0x027bd4 thru 0x033288 (0xb6b5 bytes) 000196 1788202499.090211 fallocate 0x012244 thru 0x013b0a (0x18c7 bytes) 000197 1788202499.103562 fallocate 0x036204 thru 0x037ee3 (0x1ce0 bytes) 000198 1788202499.113783 fallocate 0x003e01 thru 0x007fd6 (0x41d6 bytes) 000199 1788202499.129993 mapwrite 0x005f3a thru 0x0073f1 (0x14b8 bytes) 000200 1788202499.153605 write 0x0312cf thru 0x03d86d (0xc59f bytes) 000201 1788202499.165504 trunc from 0x03d86e to 0x0279ea (0x15e83 bytes) 000204 1788202499.193663 write 0x032bf6 thru 0x038007 (0x5412 bytes) 000205 1788202499.196646 trunc from 0x038008 to 0x02695e (0x116a9 bytes) 000206 1788202499.233018 fallocate 0x004cb8 thru 0x00ca9c (0x7de5 bytes) 000207 1788202499.248283 write 0x00b05a thru 0x01819a (0xd141 bytes) 000208 1788202499.264693 trunc from 0x00e828 to 0x017de5 (0x95be bytes) 000210 1788202499.303617 trunc from 0x02695f to 0x027eee (0x1590 bytes) 000211 1788202499.332340 write 0x00c2a5 thru 0x014959 (0x86b5 bytes) 000212 1788202499.353947 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000215 1788202499.561435 mapread 0x018814 thru 0x01ab3e (0x232b bytes) 000216 1788202499.603785 punch from 0x014b30 to 0x01ce7a (0x834b bytes) 000217 1788202499.611069 punch from 0x005e58 to 0x00e6b8 (0x8861 bytes) 000218 1788202499.621744 fallocate 0x01610d thru 0x01c154 (0x6048 bytes) 000219 1788202499.632649 write 0x00afe5 thru 0x017aa7 (0xcac3 bytes) 000220 1788202499.658357 trunc from 0x027eef to 0x008357 (0x1fb97 bytes) 000221 1788202499.684469 trunc from 0x0082a1 to 0x008357 (0xb7 bytes) 000224 1788202499.697703 mapwrite 0x02a0e8 thru 0x035ff7 (0xbf10 bytes) 000225 1788202499.770762 read 0x015e58 thru 0x020d3e (0xaee7 bytes) 000226 1788202499.792471 read 0x00bb56 thru 0x014408 (0x88b3 bytes) 000227 1788202499.805663 write 0x031fc0 thru 0x03ffff (0xe040 bytes) 000232 1788202499.818710 trunc from 0x02298c to 0x022ba4 (0x219 bytes) 000233 1788202499.832120 punch from 0x03c884 to 0x03ffff (0x377c bytes) 000234 1788202499.862934 read 0x013291 thru 0x01e164 (0xaed4 bytes) 000235 1788202499.864920 trunc from 0x040000 to 0x03efa9 (0x1056 bytes) 000236 1788202499.895798 punch from 0x026c4a to 0x0323cd (0xb784 bytes) 000237 1788202499.903224 fallocate 0x03ba6c thru 0x03ffff (0x4594 bytes) 000238 1788202499.913510 write 0x01dfe5 thru 0x02d6a5 (0xf6c1 bytes) 000239 1788202499.922459 mapwrite 0x01b48e thru 0x02af3f (0xfab2 bytes) 000240 1788202499.962195 punch from 0x00aa4a to 0x012c90 (0x8247 bytes) 000242 1788202499.972550 mapread 0x00a2ca thru 0x00d2c1 (0x2ff8 bytes) 000243 1788202500.018186 mapread 0x01d7bd thru 0x01e809 (0x104d bytes) 000244 1788202500.022564 mapwrite 0x010faf thru 0x01b47e (0xa4d0 bytes) 000245 1788202500.064995 write 0x02f6fb thru 0x03859a (0x8ea0 bytes) 000246 1788202500.066513 trunc from 0x040000 to 0x01a391 (0x25c6e bytes) 000248 1788202500.095282 mapwrite 0x02d5ca thru 0x036379 (0x8db0 bytes) 000249 1788202500.258741 MIRROR_EXTEND to 2 mirrors 000250 1788202500.994114 MIRROR_EXTEND to 3 mirrors 000251 1788202501.800166 mapread 0x022cfd thru 0x027335 (0x4639 bytes) 000252 1788202501.823215 mapread 0x014cd8 thru 0x023db6 (0xf0df bytes) 000253 1788202501.834231 mapread 0x00b639 thru 0x00c43d (0xe05 bytes) 000254 1788202501.849706 fallocate 0x022667 thru 0x028088 (0x5a22 bytes) 000255 1788202501.904642 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3637a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3637a): Mirror 2: 0xe3bb3766 (stale) Mirror 3: 0xe3bb3766 (stale) CRC-32 checksum value for chunk [0, 0x3637a): Mirror 1: 0xe3bb3766 Mirror 4: 0xe3bb3766 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3637a: skipped 000256 1788202503.066115 fallocate 0x039da2 thru 0x03b9a6 (0x1c05 bytes) 000257 1788202503.121722 punch from 0x01db18 to 0x021e34 (0x431d bytes) 000258 1788202503.129911 trunc from 0x00a34a to 0x01361b (0x92d2 bytes) 000259 1788202503.139153 read 0x01b1b9 thru 0x028192 (0xcfda bytes) 000260 1788202503.150368 fallocate 0x02e6df thru 0x036b1e (0x8440 bytes) 000261 1788202503.159317 punch from 0x008a98 to 0x0154d5 (0xca3e bytes) 000262 1788202503.165192 mapread 0x02c651 thru 0x036b1e (0xa4ce bytes) 000263 1788202503.197432 read 0x027beb thru 0x02d619 (0x5a2f bytes) 000264 1788202503.198388 trunc from 0x014b17 to 0x017818 (0x2d02 bytes) 000265 1788202503.204127 fallocate 0x01c2dc thru 0x01d4a2 (0x11c7 bytes) 000266 1788202503.208959 read 0x020cba thru 0x023b54 (0x2e9b bytes) 000267 1788202503.209569 fallocate 0x032e0b thru 0x03af0b (0x8101 bytes) 000269 1788202503.216358 mapwrite 0x00af1f thru 0x019b9c (0xec7e bytes) 000270 1788202503.254192 mapread 0x004f68 thru 0x01196b (0xca04 bytes) 000271 1788202503.255659 mapread 0x0153ef thru 0x020474 (0xb086 bytes) 000273 1788202503.259445 mapwrite 0x02f884 thru 0x033964 (0x40e1 bytes) 000274 1788202503.282063 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36b1f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36b1f): Mirror 2: 0x4c9ab9ba (stale) Mirror 3: 0x4c9ab9ba (stale) Mirror 4: 0x4c9ab9ba (stale) CRC-32 checksum value for chunk [0, 0x36b1f): Mirror 1: 0x1d4e54c0 Mirror 5: 0x1d4e54c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36b1f: skipped 000275 1788202504.174240 fallocate 0x030fd2 thru 0x0390de (0x810d bytes) 000276 1788202504.235275 trunc from 0x0390df to 0x027fe8 (0x110f6 bytes) 000277 1788202504.280230 read 0x00f831 thru 0x01930d (0x9add bytes) 000278 1788202504.293151 fallocate 0x002d6b thru 0x00485f (0x1af5 bytes) 000279 1788202504.305165 MIRROR_RESYNC 5 mirrors 000280 1788202505.989353 read 0x015a49 thru 0x01b224 (0x57dc bytes) 000281 1788202505.998224 trunc from 0x027fe9 to 0x002028 (0x25fc0 bytes) 000282 1788202506.036536 read 0x000383 thru 0x002028 (0x1ca6 bytes) 000283 1788202506.048423 write 0x0004c0 thru 0x00977a (0x92bb bytes) 000284 1788202506.051339 punch from 0x009599 to 0x00977a (0x1e2 bytes) 000285 1788202506.088288 MIRROR_RESYNC 5 mirrors 000286 1788202507.055100 fallocate 0x0041f6 thru 0x011902 (0xd70d bytes) 000288 1788202507.093195 trunc from 0x00977b to 0x03b9d7 (0x3225d bytes) 000289 1788202507.135194 trunc from 0x03b9d8 to 0x039af2 (0x1ee5 bytes) 000290 1788202507.152755 mapwrite 0x0030be thru 0x0088ec (0x582f bytes) 000291 1788202507.195071 read 0x00e5e5 thru 0x014f51 (0x696d bytes) 000292 1788202507.202118 fallocate 0x00ed75 thru 0x018a20 (0x9cac bytes) 000293 1788202507.209945 write 0x00c971 thru 0x010aa6 (0x4136 bytes) 000294 1788202507.216335 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39af3): Mirror 2: 0xdfd392fb (stale) Mirror 3: 0xdfd392fb (stale) Mirror 4: 0xdfd392fb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39af3): Mirror 2: 0xdfd392fb (stale) Mirror 3: 0xdfd392fb (stale) Mirror 4: 0xdfd392fb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39af3: skipped 000295 1788202507.607407 write 0x02b1be thru 0x03382e (0x8671 bytes) 000296 1788202507.626253 trunc from 0x039af3 to 0x032934 (0x71be bytes) 000297 1788202507.642041 read 0x00d342 thru 0x017feb (0xacaa bytes) 000298 1788202507.652201 trunc from 0x032935 to 0x02c30a (0x662a bytes) 000299 1788202507.671780 mapread 0x024d03 thru 0x028240 (0x353e bytes) 000300 1788202507.689864 fallocate 0x0050d7 thru 0x00f21f (0xa149 bytes) 000301 1788202507.694787 punch from 0x004c8b to 0x00e9e6 (0x9d5c bytes) 000302 1788202507.700301 read 0x010452 thru 0x01aa7c (0xa62b bytes) 000303 1788202507.714389 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2c30b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c30b): Mirror 2: 0xc1ceb968 (stale) Mirror 3: 0xc1ceb968 (stale) Mirror 4: 0xc1ceb968 (stale) CRC-32 checksum value for chunk [0, 0x2c30b): Mirror 1: 0xea42a5c4 Mirror 5: 0xea42a5c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c30b: skipped 000304 1788202508.696206 mapwrite 0x01e3aa thru 0x023b1d (0x5774 bytes) 000306 1788202508.809964 fallocate 0x01cd13 thru 0x025aaa (0x8d98 bytes) 000307 1788202508.813764 trunc from 0x0246b1 to 0x027a48 (0x3398 bytes) 000308 1788202508.822433 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2c30b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c30b): Mirror 2: 0xc1ceb968 (stale) Mirror 3: 0xc1ceb968 (stale) Mirror 4: 0xc1ceb968 (stale) Mirror 5: 0xea42a5c4 (stale) CRC-32 checksum value for chunk [0, 0x2c30b): Mirror 1: 0x276ef6e0 Mirror 6: 0x276ef6e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c30b: skipped 000309 1788202509.705581 trunc from 0x01d89c to 0x0213a7 (0x3b0c bytes) 000310 1788202509.785288 write 0x03295a thru 0x03df81 (0xb628 bytes) 000311 1788202509.796654 trunc from 0x0119da to 0x01870a (0x6d31 bytes) 000312 1788202509.814949 mapwrite 0x025774 thru 0x033981 (0xe20e bytes) 000313 1788202509.887482 trunc from 0x03df82 to 0x037e61 (0x6120 bytes) 000314 1788202509.913384 fallocate 0x02d85c thru 0x035cc7 (0x846c bytes) 000315 1788202509.922766 write 0x0067a6 thru 0x010f7f (0xa7da bytes) 000316 1788202509.944637 MIRROR_RESYNC 6 mirrors 000317 1788202511.964090 write 0x01a250 thru 0x01ac3b (0x9ec bytes) 000318 1788202512.020716 mapwrite 0x01df04 thru 0x01e468 (0x565 bytes) 000319 1788202512.057327 trunc from 0x0375ac to 0x037764 (0x1b9 bytes) 000320 1788202512.067194 mapread 0x006b42 thru 0x012be5 (0xc0a4 bytes) 000321 1788202512.088101 read 0x028748 thru 0x0307f9 (0x80b2 bytes) 000322 1788202512.108703 punch from 0x003fcf to 0x012af1 (0xeb23 bytes) 000323 1788202512.118576 trunc from 0x0332b1 to 0x037e61 (0x4bb1 bytes) 000324 1788202512.127174 mapread 0x02af7a thru 0x034e9e (0x9f25 bytes) 000325 1788202512.142037 read 0x0240ca thru 0x033b6a (0xfaa1 bytes) 000326 1788202512.163659 trunc from 0x02aba7 to 0x02fd23 (0x517d bytes) 000328 1788202512.176734 read 0x01c595 thru 0x0238b4 (0x7320 bytes) 000329 1788202512.177970 trunc from 0x037e62 to 0x000fc0 (0x36ea1 bytes) 000330 1788202512.211702 trunc from 0x000fc1 to 0x01393a (0x1297a bytes) 000331 1788202512.240352 punch from 0x00f0c4 to 0x01393a (0x4877 bytes) 000332 1788202512.251295 mapwrite 0x00c583 thru 0x00fa15 (0x3493 bytes) 000333 1788202512.308181 write 0x018d48 thru 0x01c832 (0x3aeb bytes) 000336 1788202512.312256 read 0x004c2e thru 0x00995d (0x4d30 bytes) 000337 1788202512.313063 mapread 0x003e51 thru 0x00de51 (0xa001 bytes) 000338 1788202512.323632 MIRROR_RESYNC 6 mirrors 000339 1788202513.649057 read 0x00da52 thru 0x01792b (0x9eda bytes) 000340 1788202513.658971 trunc from 0x010997 to 0x014950 (0x3fba bytes) 000341 1788202513.696400 write 0x028162 thru 0x031f47 (0x9de6 bytes) 000342 1788202513.699891 trunc from 0x031f48 to 0x01e390 (0x13bb7 bytes) 000343 1788202513.718336 trunc from 0x0149cf to 0x01d11f (0x8751 bytes) 000344 1788202513.724347 fallocate 0x03dd62 thru 0x03ffff (0x229e bytes) 000345 1788202513.730528 mapwrite 0x0313ea thru 0x036938 (0x554f bytes) 000346 1788202513.772878 mapread 0x02eae6 thru 0x03ba1e (0xcf39 bytes) 000347 1788202513.774295 mapwrite 0x014c89 thru 0x0156ab (0xa23 bytes) 000348 1788202513.823437 read 0x030f48 thru 0x031e81 (0xf3a bytes) 000349 1788202513.824232 trunc from 0x040000 to 0x00c837 (0x337c8 bytes) 000352 1788202513.852663 read 0x009c08 thru 0x00c837 (0x2c30 bytes) 000353 1788202513.872381 write 0x007493 thru 0x00a4df (0x304d bytes) 000354 1788202513.873695 fallocate 0x02a2ec thru 0x039244 (0xef59 bytes) 000355 1788202513.882399 write 0x00cb6d thru 0x016f6d (0xa401 bytes) 000356 1788202513.886354 mapread 0x00417c thru 0x0095cb (0x5450 bytes) 000357 1788202513.888261 write 0x03191f thru 0x036a9e (0x5180 bytes) 000358 1788202513.893231 read 0x01b6c9 thru 0x01c547 (0xe7f bytes) 000359 1788202513.903676 write 0x00e3ff thru 0x014176 (0x5d78 bytes) 000360 1788202513.909046 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36a9f): Mirror 3: 0xde8e2004 (stale) Mirror 4: 0xde8e2004 (stale) Mirror 5: 0xde8e2004 (stale) Mirror 6: 0xde8e2004 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a9f): Mirror 3: 0xde8e2004 (stale) Mirror 4: 0xde8e2004 (stale) Mirror 5: 0xde8e2004 (stale) Mirror 6: 0xde8e2004 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a9f: skipped 000361 1788202514.439041 write 0x02bee7 thru 0x032d52 (0x6e6c bytes) 000362 1788202514.468958 punch from 0x02b1b4 to 0x034ec8 (0x9d15 bytes) 000363 1788202514.494966 mapread 0x02ac78 thru 0x03124f (0x65d8 bytes) 000364 1788202514.530407 punch from 0x008902 to 0x010fa1 (0x86a0 bytes) 000365 1788202514.536225 trunc from 0x036a9f to 0x035f19 (0xb85 bytes) 000366 1788202514.554462 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35f1a): Mirror 3: 0xabf1996c (stale) Mirror 4: 0xabf1996c (stale) Mirror 5: 0xabf1996c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35f1a): Mirror 3: 0xabf1996c (stale) Mirror 4: 0xabf1996c (stale) Mirror 5: 0xabf1996c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35f1a: skipped 000367 1788202514.912406 punch from 0x031a58 to 0x035c7f (0x4228 bytes) 000368 1788202514.917246 mapread 0x0238f9 thru 0x02a7c2 (0x6eca bytes) 000369 1788202514.933268 trunc from 0x00eaaf to 0x01e712 (0xfc64 bytes) 000370 1788202514.944400 mapwrite 0x02fff0 thru 0x03c679 (0xc68a bytes) 000371 1788202515.019628 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c67a): Mirror 4: 0x2fb403c (stale) Mirror 5: 0x2fb403c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c67a): Mirror 4: 0x2fb403c (stale) Mirror 5: 0x2fb403c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c67a: skipped 000372 1788202515.474330 write 0x03858b thru 0x03ffff (0x7a75 bytes) 000373 1788202515.497275 trunc from 0x040000 to 0x038678 (0x7987 bytes) 000374 1788202515.527897 trunc from 0x0112e4 to 0x01d1f6 (0xbf13 bytes) 000375 1788202515.547399 trunc from 0x038679 to 0x03a442 (0x1dca bytes) 000376 1788202515.590941 punch from 0x03835b to 0x03a442 (0x20e8 bytes) 000377 1788202515.603685 trunc from 0x035cbb to 0x03a442 (0x4788 bytes) 000378 1788202515.609142 fallocate 0x004118 thru 0x00f5fd (0xb4e6 bytes) 000379 1788202515.620158 mapread 0x0170d9 thru 0x02222e (0xb156 bytes) 000380 1788202515.678352 fallocate 0x02ae0f thru 0x033dbc (0x8fae bytes) 000381 1788202515.694598 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a443): Mirror 5: 0x8aedc97e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a443): Mirror 5: 0x8aedc97e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a443: skipped 000383 1788202516.072343 trunc from 0x03a443 to 0x03a8ee (0x4ac bytes) 000384 1788202516.092502 trunc from 0x030dfc to 0x03a8ee (0x9af3 bytes) 000385 1788202516.102175 punch from 0x028d66 to 0x037b7b (0xee16 bytes) 000386 1788202516.107172 trunc from 0x03a8ef to 0x003169 (0x37785 bytes) 000387 1788202516.126236 trunc from 0x00316a to 0x0121a1 (0xf038 bytes) 000388 1788202516.143136 write 0x023a4b thru 0x0284f3 (0x4aa9 bytes) 000389 1788202516.144679 write 0x03df42 thru 0x03ffff (0x20be bytes) 000390 1788202516.147269 trunc from 0x040000 to 0x01f8cf (0x20730 bytes) 000391 1788202516.165040 fallocate 0x031732 thru 0x03b213 (0x9ae2 bytes) 000392 1788202516.171134 punch from 0x02dd90 to 0x03242a (0x469b bytes) 000393 1788202516.175999 fallocate 0x034f05 thru 0x03ffff (0xb0fb bytes) 000394 1788202516.181102 mapread 0x016ba4 thru 0x024f20 (0xe37d bytes) 000396 1788202516.223909 write 0x03bb4f thru 0x03ffff (0x44b1 bytes) 000397 1788202516.232757 write 0x036a2f thru 0x03cf30 (0x6502 bytes) 000398 1788202516.241092 write 0x00164b thru 0x002c02 (0x15b8 bytes) 000399 1788202516.255000 mapwrite 0x00377a thru 0x004fb3 (0x183a bytes) 000400 1788202516.292533 mapwrite 0x009082 thru 0x00acea (0x1c69 bytes) 000401 1788202516.320357 MIRROR_RESYNC 2 mirrors 000402 1788202516.990026 mapwrite 0x02f9b0 thru 0x03f565 (0xfbb6 bytes) 000403 1788202517.102418 trunc from 0x03e636 to 0x03e643 (0xe bytes) 000404 1788202517.110960 mapread 0x003086 thru 0x012d78 (0xfcf3 bytes) 000405 1788202517.132132 mapwrite 0x01f01c thru 0x01fd4f (0xd34 bytes) 000407 1788202517.154525 trunc from 0x021f9e to 0x0262ee (0x4351 bytes) 000408 1788202517.167129 write 0x0009fc thru 0x003e1e (0x3423 bytes) 000409 1788202517.173364 read 0x00ce40 thru 0x01cd58 (0xff19 bytes) 000410 1788202517.174399 fallocate 0x00c6d0 thru 0x0147fb (0x812c bytes) 000411 1788202517.179535 read 0x037946 thru 0x03ffff (0x86ba bytes) 000412 1788202517.189369 mapread 0x037911 thru 0x03facc (0x81bc bytes) 000413 1788202517.190721 trunc from 0x01b544 to 0x0237a9 (0x8266 bytes) 000414 1788202517.195400 mapread 0x01418e thru 0x0223c3 (0xe236 bytes) 000415 1788202517.233342 mapread 0x02f462 thru 0x0335bc (0x415b bytes) 000416 1788202517.235218 punch from 0x038225 to 0x03ffff (0x7ddb bytes) 000417 1788202517.240608 read 0x032f4f thru 0x034bed (0x1c9f bytes) 000419 1788202517.242226 read 0x037513 thru 0x03ffff (0x8aed bytes) 000420 1788202517.259301 punch from 0x027341 to 0x0327e3 (0xb4a3 bytes) 000421 1788202517.267944 trunc from 0x03679d to 0x03e10c (0x7970 bytes) 000422 1788202517.289652 punch from 0x00ede7 to 0x014aef (0x5d09 bytes) 000423 1788202517.302937 fallocate 0x0313c0 thru 0x032b55 (0x1796 bytes) 000424 1788202517.310313 trunc from 0x02f12a to 0x03c69f (0xd576 bytes) 000425 1788202517.332400 trunc from 0x03abc0 to 0x03ffff (0x5440 bytes) 000426 1788202517.339344 mapread 0x025b89 thru 0x030aa6 (0xaf1e bytes) 000427 1788202517.398713 trunc from 0x026141 to 0x03372e (0xd5ee bytes) 000428 1788202517.408151 punch from 0x01ebbb to 0x02e353 (0xf799 bytes) 000429 1788202517.415183 write 0x03c9f6 thru 0x03ffff (0x360a bytes) 000430 1788202517.424120 mapwrite 0x01d306 thru 0x022916 (0x5611 bytes) 000431 1788202517.486195 trunc from 0x015b27 to 0x017daf (0x2289 bytes) 000432 1788202517.491786 read 0x028472 thru 0x0336fc (0xb28b bytes) 000433 1788202517.518131 read 0x00179a thru 0x0017c4 (0x2b bytes) 000434 1788202517.518910 punch from 0x020092 to 0x023134 (0x30a3 bytes) 000435 1788202517.527097 mapwrite 0x02ff0a thru 0x03b1ea (0xb2e1 bytes) 000436 1788202517.559043 mapread 0x00d177 thru 0x018948 (0xb7d2 bytes) 000437 1788202517.562193 trunc from 0x024d34 to 0x02f0cb (0xa398 bytes) 000438 1788202517.569737 mapwrite 0x018399 thru 0x020b8f (0x87f7 bytes) 000440 1788202517.612522 read 0x036b08 thru 0x036b8d (0x86 bytes) 000441 1788202517.614086 write 0x022b54 thru 0x02af9a (0x8447 bytes) 000442 1788202517.616753 write 0x034cbd thru 0x03915c (0x44a0 bytes) 000443 1788202517.619679 fallocate 0x023700 thru 0x03268a (0xef8b bytes) 000445 1788202517.631216 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x1cc92278 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x96999a9 Mirror 6: 0x96999a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000446 1788202519.037132 trunc from 0x040000 to 0x01688c (0x29773 bytes) 000448 1788202519.152013 fallocate 0x02ed66 thru 0x0393bc (0xa657 bytes) 000449 1788202519.180432 fallocate 0x008af7 thru 0x018586 (0xfa90 bytes) 000450 1788202519.190131 trunc from 0x004a47 to 0x00ba1d (0x6fd7 bytes) 000451 1788202519.216892 read 0x0001e3 thru 0x0031a0 (0x2fbe bytes) 000452 1788202519.238281 fallocate 0x0383df thru 0x03db63 (0x5785 bytes) 000453 1788202519.251102 mapread 0x03697a thru 0x0393bc (0x2a43 bytes) 000454 1788202519.301293 write 0x034a59 thru 0x03ffff (0xb5a7 bytes) 000455 1788202519.315168 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x1cc92278 (stale) Mirror 6: 0x96999a9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x17e94e01 Mirror 7: 0x17e94e01 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000456 1788202520.323695 trunc from 0x040000 to 0x031ab9 (0xe546 bytes) 000457 1788202520.394425 read 0x004e02 thru 0x00e584 (0x9783 bytes) 000459 1788202520.408371 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31aba): Mirror 6: 0xc6a4a38b (stale) Mirror 7: 0x75328265 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31aba): Mirror 6: 0xc6a4a38b (stale) Mirror 7: 0x75328265 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31aba: skipped 000460 1788202520.687867 write 0x02331b thru 0x027e3d (0x4b23 bytes) 000461 1788202520.704031 punch from 0x023be8 to 0x02e33d (0xa756 bytes) 000462 1788202520.740002 fallocate 0x03178c thru 0x03ffff (0xe874 bytes) 000463 1788202520.751985 read 0x02d384 thru 0x031ab9 (0x4736 bytes) 000464 1788202520.759762 write 0x034969 thru 0x03e132 (0x97ca bytes) 000465 1788202520.761481 trunc from 0x0394a4 to 0x03e132 (0x4c8f bytes) 000466 1788202520.791878 mapwrite 0x026a82 thru 0x030cdf (0xa25e bytes) 000467 1788202520.837493 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e133): Mirror 7: 0x59749a9e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e133): Mirror 7: 0x59749a9e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e133: skipped 000468 1788202521.147889 mapwrite 0x019a18 thru 0x02161f (0x7c08 bytes) 000469 1788202521.222939 fallocate 0x013166 thru 0x018758 (0x55f3 bytes) 000470 1788202521.243163 write 0x020d90 thru 0x029d66 (0x8fd7 bytes) 000471 1788202521.258802 mapwrite 0x009cde thru 0x00b656 (0x1979 bytes) 000472 1788202521.313138 mapread 0x02ca68 thru 0x02f285 (0x281e bytes) 000473 1788202521.335664 trunc from 0x03e133 to 0x035b84 (0x85ae bytes) 000474 1788202521.356552 punch from 0x02d7ff to 0x035b84 (0x8386 bytes) 000475 1788202521.365380 mapwrite 0x0352c9 thru 0x03bfce (0x6d06 bytes) 000476 1788202521.440896 trunc from 0x03bfcf to 0x036d3a (0x5294 bytes) 000477 1788202521.457904 read 0x030219 thru 0x036d3a (0x6b22 bytes) 000478 1788202521.458795 mapwrite 0x008b00 thru 0x017bf4 (0xf0f5 bytes) 000479 1788202521.487108 trunc from 0x036d3b to 0x02cb07 (0xa233 bytes) 000480 1788202521.516471 read 0x027777 thru 0x02b762 (0x3fec bytes) 000481 1788202521.527104 read 0x0142b9 thru 0x023393 (0xf0db bytes) 000482 1788202521.534251 trunc from 0x00c3ae to 0x00c42c (0x7f bytes) 000483 1788202521.548465 trunc from 0x0055d6 to 0x011f72 (0xc99d bytes) 000485 1788202521.566166 MIRROR_RESYNC 2 mirrors 000488 1788202522.705722 trunc from 0x01beb0 to 0x024dea (0x8f3b bytes) 000489 1788202522.768185 trunc from 0x02cb08 to 0x005829 (0x272de bytes) 000490 1788202522.810309 fallocate 0x02b3be thru 0x03701f (0xbc62 bytes) 000491 1788202522.837259 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x582a): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x582a): Mirror 7: 0x58da679f (stale) CRC-32 checksum value for chunk [0, 0x582a): Mirror 1: 0x58da679f Mirror 8: 0x58da679f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x582a: skipped 000493 1788202524.168979 trunc from 0x00582a to 0x004ac4 (0xd65 bytes) 000494 1788202524.235200 punch from 0x0029ff to 0x002fad (0x5af bytes) 000495 1788202524.242398 mapwrite 0x006da1 thru 0x0123e4 (0xb644 bytes) 000496 1788202524.327288 punch from 0x00c721 to 0x0123e4 (0x5cc4 bytes) 000497 1788202524.335327 punch from 0x00430c to 0x007737 (0x342c bytes) 000498 1788202524.349121 mapwrite 0x0381c4 thru 0x03ffff (0x7e3c bytes) 000499 1788202524.415020 mapwrite 0x0382c3 thru 0x03ffff (0x7d3d bytes) 000500 1788202524.460855 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xd4dc01a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xd4dc01a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000501 1788202524.789598 fallocate 0x02a9e7 thru 0x030f83 (0x659d bytes) 000502 1788202524.797163 write 0x00a2b4 thru 0x0181b8 (0xdf05 bytes) 000503 1788202524.815063 read 0x033dca thru 0x03ffff (0xc236 bytes) 000504 1788202524.827242 trunc from 0x024f05 to 0x031e1a (0xcf16 bytes) 000505 1788202524.833236 trunc from 0x040000 to 0x02cdf1 (0x1320e bytes) 000506 1788202524.849209 trunc from 0x02cdf2 to 0x02a8ac (0x2545 bytes) 000507 1788202524.868458 trunc from 0x01cfc0 to 0x026659 (0x969a bytes) 000508 1788202524.875318 trunc from 0x02a8ad to 0x0268e1 (0x3fcb bytes) 000509 1788202524.893944 punch from 0x0050cb to 0x00d9ae (0x88e4 bytes) 000511 1788202524.917136 read 0x01cf96 thru 0x02539c (0x8407 bytes) 000512 1788202524.937531 mapwrite 0x00a87d thru 0x00d0cc (0x2850 bytes) 000514 1788202524.964693 MIRROR_RESYNC 2 mirrors 000515 1788202526.182352 write 0x0099b1 thru 0x01182d (0x7e7d bytes) 000516 1788202526.256024 mapread 0x0127ec thru 0x01355c (0xd71 bytes) 000517 1788202526.270761 write 0x03826e thru 0x03ffff (0x7d92 bytes) 000518 1788202526.277054 read 0x03d37d thru 0x03dc03 (0x887 bytes) 000519 1788202526.278284 mapwrite 0x016ee0 thru 0x016fac (0xcd bytes) 000520 1788202526.359347 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x412f006 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x125ac48a Mirror 9: 0x125ac48a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000521 1788202527.273873 trunc from 0x040000 to 0x037b3f (0x84c0 bytes) 000522 1788202527.345747 read 0x0185b7 thru 0x023401 (0xae4b bytes) 000523 1788202527.354302 fallocate 0x03f0b3 thru 0x03ffff (0xf4d bytes) 000524 1788202527.362951 MIRROR_RESYNC 3 mirrors 000525 1788202528.362284 read 0x012305 thru 0x01a95d (0x8659 bytes) 000526 1788202528.395555 read 0x00806f thru 0x00e40e (0x63a0 bytes) 000527 1788202528.428998 trunc from 0x03c680 to 0x03ffff (0x3980 bytes) 000528 1788202528.484681 trunc from 0x027c60 to 0x0286a8 (0xa49 bytes) 000530 1788202528.499932 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xa7d93db8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xa7d93db8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000531 1788202528.914709 write 0x035f1a thru 0x03f6a9 (0x9790 bytes) 000532 1788202528.936727 read 0x01440e thru 0x01dcf3 (0x98e6 bytes) 000533 1788202528.946899 fallocate 0x00a0a4 thru 0x00b600 (0x155d bytes) 000534 1788202528.953194 trunc from 0x040000 to 0x00de36 (0x321c9 bytes) 000535 1788202528.980282 mapwrite 0x00d690 thru 0x01cd90 (0xf701 bytes) 000536 1788202529.071021 punch from 0x00e4e7 to 0x00f5fb (0x1115 bytes) 000538 1788202529.083841 write 0x00560d thru 0x0090c5 (0x3ab9 bytes) 000539 1788202529.098309 read 0x019efd thru 0x01cd90 (0x2e94 bytes) 000540 1788202529.099519 write 0x0015bb thru 0x008c0d (0x7653 bytes) 000541 1788202529.115950 punch from 0x016ec9 to 0x01cd90 (0x5ec8 bytes) 000542 1788202529.147667 MIRROR_RESYNC 2 mirrors 000545 1788202529.890378 read 0x008bac thru 0x00f234 (0x6689 bytes) 000546 1788202529.912644 mapread 0x019519 thru 0x01cd90 (0x3878 bytes) 000547 1788202529.955431 write 0x03827f thru 0x03f720 (0x74a2 bytes) 000549 1788202530.047680 write 0x018884 thru 0x01f42c (0x6ba9 bytes) 000550 1788202530.067853 trunc from 0x0104b8 to 0x0112d5 (0xe1e bytes) 000551 1788202530.074044 mapread 0x02b1f7 thru 0x02ec88 (0x3a92 bytes) 000553 1788202530.082788 fallocate 0x00c6ab thru 0x019d4e (0xd6a4 bytes) 000554 1788202530.088761 punch from 0x014aa1 to 0x01edcf (0xa32f bytes) 000555 1788202530.105791 write 0x024851 thru 0x028e87 (0x4637 bytes) 000556 1788202530.107009 write 0x033e1c thru 0x03ffff (0xc1e4 bytes) 000557 1788202530.110048 mapread 0x01ae80 thru 0x01d61d (0x279e bytes) 000558 1788202530.121246 mapwrite 0x006942 thru 0x00fb94 (0x9253 bytes) 000559 1788202530.157677 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000560 1788202530.424754 trunc from 0x040000 to 0x03305a (0xcfa5 bytes) 000562 1788202530.526368 read 0x021ab5 thru 0x02cd64 (0xb2b0 bytes) 000563 1788202530.552327 fallocate 0x03570b thru 0x03ffff (0xa8f5 bytes) 000565 1788202530.560753 mapread 0x01365b thru 0x01b45a (0x7e00 bytes) 000566 1788202530.583273 mapwrite 0x0020dc thru 0x010118 (0xe03d bytes) 000567 1788202530.630151 write 0x007520 thru 0x00cc0a (0x56eb bytes) 000569 1788202530.634476 mapwrite 0x00cc91 thru 0x00fe72 (0x31e2 bytes) 000570 1788202530.655499 mapread 0x002f44 thru 0x008662 (0x571f bytes) 000571 1788202530.659800 mapread 0x0310fd thru 0x03305a (0x1f5e bytes) 000572 1788202530.676641 trunc from 0x032108 to 0x03260a (0x503 bytes) 000574 1788202530.688236 read 0x007a7f thru 0x016cd9 (0xf25b bytes) 000576 1788202530.692354 trunc from 0x03305b to 0x00a87e (0x287dc bytes) 000577 1788202530.718917 read 0x006586 thru 0x00a87e (0x42f9 bytes) 000578 1788202530.720149 trunc from 0x009f89 to 0x00a87e (0x8f6 bytes) 000579 1788202530.731852 trunc from 0x005442 to 0x00a87e (0x543d bytes) 000580 1788202530.750792 trunc from 0x002668 to 0x00a87e (0x8217 bytes) 000581 1788202530.759905 fallocate 0x026b81 thru 0x028335 (0x17b5 bytes) 000582 1788202530.774519 write 0x00ae47 thru 0x00d16e (0x2328 bytes) 000583 1788202530.795441 fallocate 0x0259f8 thru 0x02639c (0x9a5 bytes) 000584 1788202530.807728 read 0x00bd27 thru 0x00d16e (0x1448 bytes) 000585 1788202530.814725 mapread 0x00cd97 thru 0x00d16e (0x3d8 bytes) 000586 1788202530.817079 punch from 0x00b50e to 0x00d16e (0x1c61 bytes) 000587 1788202530.847981 read 0x00048b thru 0x00c959 (0xc4cf bytes) 000588 1788202530.857894 trunc from 0x00d16f to 0x024e7b (0x17d0d bytes) 000590 1788202530.885667 write 0x01da1a thru 0x0287d6 (0xadbd bytes) 000591 1788202530.915200 punch from 0x008298 to 0x017948 (0xf6b1 bytes) 000592 1788202530.930990 mapread 0x021f68 thru 0x0287d6 (0x686f bytes) 000593 1788202530.936801 write 0x017704 thru 0x024062 (0xc95f bytes) 000594 1788202530.950748 punch from 0x02219a to 0x02430d (0x2174 bytes) 000595 1788202530.983935 trunc from 0x00fbfb to 0x01ac38 (0xb03e bytes) 000596 1788202530.992463 trunc from 0x00fee2 to 0x010711 (0x830 bytes) 000597 1788202530.999893 trunc from 0x0287d7 to 0x02bbd7 (0x3401 bytes) 000598 1788202531.023532 trunc from 0x00f64e to 0x014015 (0x49c8 bytes) 000599 1788202531.031760 trunc from 0x009e22 to 0x00eef5 (0x50d4 bytes) 000600 1788202531.039682 read 0x00728b thru 0x0116cb (0xa441 bytes) 000601 1788202531.057565 write 0x01d627 thru 0x029fce (0xc9a8 bytes) 000602 1788202531.061321 mapwrite 0x029b4e thru 0x032f8a (0x943d bytes) 000603 1788202531.117124 fallocate 0x012f42 thru 0x0133ed (0x4ac bytes) 000604 1788202531.124246 mapread 0x00f926 thru 0x015d60 (0x643b bytes) 000607 1788202531.130853 trunc from 0x032f8b to 0x004fb3 (0x2dfd7 bytes) 000609 1788202531.161055 read 0x001f19 thru 0x004fb3 (0x309b bytes) 000610 1788202531.162510 mapread 0x0022e0 thru 0x003e6c (0x1b8d bytes) 000612 1788202531.166495 write 0x0359e9 thru 0x03e67e (0x8c96 bytes) 000613 1788202531.169048 trunc from 0x0379d9 to 0x037b6d (0x195 bytes) 000614 1788202531.189150 fallocate 0x0266bc thru 0x029e78 (0x37bd bytes) 000615 1788202531.196767 read 0x03100f thru 0x03ddb5 (0xcda7 bytes) 000616 1788202531.211249 trunc from 0x012982 to 0x018059 (0x56d8 bytes) 000617 1788202531.217484 fallocate 0x0312e1 thru 0x032887 (0x15a7 bytes) 000618 1788202531.225532 write 0x0348b6 thru 0x03ffff (0xb74a bytes) 000619 1788202531.230019 write 0x0200db thru 0x02e1cc (0xe0f2 bytes) 000620 1788202531.238025 write 0x033970 thru 0x037554 (0x3be5 bytes) 000621 1788202531.239627 mapread 0x032c38 thru 0x03850e (0x58d7 bytes) 000622 1788202531.240705 trunc from 0x040000 to 0x0196e6 (0x26919 bytes) 000623 1788202531.262649 punch from 0x00920f to 0x018848 (0xf63a bytes) 000624 1788202531.271308 trunc from 0x01150e to 0x0196e6 (0x81d9 bytes) 000625 1788202531.277631 punch from 0x013b60 to 0x01793a (0x3ddb bytes) 000626 1788202531.284279 read 0x00ae6b thru 0x00e669 (0x37ff bytes) 000627 1788202531.300134 read 0x010d83 thru 0x0196e6 (0x8964 bytes) 000628 1788202531.301056 mapread 0x001bd5 thru 0x007952 (0x5d7e bytes) 000629 1788202531.303447 trunc from 0x0148cf to 0x01948d (0x4bbf bytes) 000631 1788202531.309505 trunc from 0x019251 to 0x0196e6 (0x496 bytes) 000632 1788202531.315694 punch from 0x001c81 to 0x005ecb (0x424b bytes) 000633 1788202531.323463 fallocate 0x0184f6 thru 0x025c92 (0xd79d bytes) 000634 1788202531.330742 read 0x010a4f thru 0x0180bb (0x766d bytes) 000635 1788202531.358009 punch from 0x01772a to 0x0196e6 (0x1fbd bytes) 000636 1788202531.364631 mapwrite 0x020ca5 thru 0x026dbf (0x611b bytes) 000638 1788202531.478940 mapwrite 0x02f240 thru 0x02f2b5 (0x76 bytes) 000639 1788202531.591797 mapread 0x01dbd8 thru 0x029fbf (0xc3e8 bytes) 000640 1788202531.598312 read 0x004c0c thru 0x00ff5f (0xb354 bytes) 000641 1788202531.602397 mapread 0x02e501 thru 0x02f2b5 (0xdb5 bytes) 000642 1788202531.603425 mapread 0x025978 thru 0x02f2b5 (0x993e bytes) 000644 1788202531.608288 trunc from 0x01c7b4 to 0x01f196 (0x29e3 bytes) 000645 1788202531.617439 read 0x017a9e thru 0x01ae2c (0x338f bytes) 000646 1788202531.633785 MIRROR_EXTEND to 2 mirrors 000647 1788202532.506279 read 0x027726 thru 0x02ccab (0x5586 bytes) 000648 1788202532.515407 mapread 0x0012ed thru 0x00b1b2 (0x9ec6 bytes) 000649 1788202532.534084 write 0x01cfa4 thru 0x01deed (0xf4a bytes) 000650 1788202532.573994 mapwrite 0x02c9d0 thru 0x02d7d5 (0xe06 bytes) 000651 1788202532.600760 fallocate 0x008fb5 thru 0x00ca20 (0x3a6c bytes) 000652 1788202532.605750 fallocate 0x00f4f4 thru 0x01c782 (0xd28f bytes) 000653 1788202532.610932 trunc from 0x02f2b6 to 0x00582c (0x29a89 bytes) 000654 1788202532.630880 fallocate 0x008af3 thru 0x00ef96 (0x64a4 bytes) 000655 1788202532.636974 trunc from 0x00582d to 0x017d49 (0x1251d bytes) 000656 1788202532.658503 read 0x009080 thru 0x009b47 (0xac8 bytes) 000657 1788202532.667589 mapwrite 0x006554 thru 0x00af80 (0x4a2d bytes) 000658 1788202532.702328 mapread 0x010bd0 thru 0x017d49 (0x717a bytes) 000659 1788202532.708154 trunc from 0x0105ef to 0x017d49 (0x775b bytes) 000660 1788202532.720213 read 0x013a65 thru 0x017d49 (0x42e5 bytes) 000661 1788202532.745463 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000662 1788202532.889009 fallocate 0x00164d thru 0x00e126 (0xcada bytes) 000664 1788202532.911406 fallocate 0x03dc95 thru 0x03ffff (0x236b bytes) 000665 1788202532.919984 write 0x03cceb thru 0x03ffff (0x3315 bytes) 000666 1788202532.933973 trunc from 0x019d20 to 0x01fb6c (0x5e4d bytes) 000667 1788202532.949569 trunc from 0x040000 to 0x00a19c (0x35e63 bytes) 000668 1788202532.987363 fallocate 0x03e4d1 thru 0x03ffff (0x1b2f bytes) 000669 1788202532.997620 punch from 0x0089d3 to 0x010301 (0x792f bytes) 000670 1788202533.015039 MIRROR_EXTEND to 2 mirrors 000671 1788202533.879237 read 0x00cdd1 thru 0x01c7b1 (0xf9e1 bytes) 000673 1788202533.892778 mapread 0x023f15 thru 0x02fd8c (0xbe78 bytes) 000674 1788202533.925241 mapread 0x03697b thru 0x03e1d4 (0x785a bytes) 000675 1788202533.935619 trunc from 0x03a2c9 to 0x03e240 (0x3f78 bytes) 000676 1788202533.975449 read 0x0380ed thru 0x03d045 (0x4f59 bytes) 000677 1788202533.994141 mapwrite 0x0378b0 thru 0x03ffff (0x8750 bytes) 000678 1788202534.040301 trunc from 0x0150fb to 0x01efee (0x9ef4 bytes) 000679 1788202534.047799 write 0x016dc4 thru 0x0261cb (0xf408 bytes) 000680 1788202534.062887 trunc from 0x00428f to 0x00f667 (0xb3d9 bytes) 000681 1788202534.085037 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6a65d0f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x73d03d1e Mirror 3: 0x73d03d1e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000682 1788202535.232215 mapwrite 0x006ca0 thru 0x011e47 (0xb1a8 bytes) 000683 1788202535.375056 mapwrite 0x003d66 thru 0x00e068 (0xa303 bytes) 000684 1788202535.420882 fallocate 0x007fca thru 0x016d0b (0xed42 bytes) 000687 1788202535.441673 mapread 0x00d33b thru 0x00fcf6 (0x29bc bytes) 000688 1788202535.448349 fallocate 0x03a7ce thru 0x03d17e (0x29b1 bytes) 000689 1788202535.466670 trunc from 0x040000 to 0x0083be (0x37c41 bytes) 000690 1788202535.508432 punch from 0x003a41 to 0x0083be (0x497e bytes) 000691 1788202535.536111 trunc from 0x0019f8 to 0x0083be (0x69c7 bytes) 000692 1788202535.553061 trunc from 0x0083bf to 0x02d49f (0x250e1 bytes) 000693 1788202535.596060 mapwrite 0x028622 thru 0x037e52 (0xf831 bytes) 000694 1788202535.710892 mapread 0x015565 thru 0x01ba37 (0x64d3 bytes) 000695 1788202535.734950 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x37e53): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37e53): Mirror 2: 0xf1bfe3b8 (stale) Mirror 3: 0x34f09594 (stale) CRC-32 checksum value for chunk [0, 0x37e53): Mirror 1: 0x834bb0a4 Mirror 4: 0x834bb0a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37e53: skipped 000696 1788202537.189444 mapread 0x00b141 thru 0x016bdb (0xba9b bytes) 000697 1788202537.273365 read 0x010812 thru 0x01fbc9 (0xf3b8 bytes) 000698 1788202537.296894 punch from 0x00b485 to 0x00ded2 (0x2a4e bytes) 000700 1788202537.373422 punch from 0x012e86 to 0x017cc8 (0x4e43 bytes) 000701 1788202537.381387 trunc from 0x037e53 to 0x016957 (0x214fb bytes) 000702 1788202537.420468 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x16958): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16958): Mirror 2: 0xc5d2e95a (stale) Mirror 3: 0x14866943 (stale) Mirror 4: 0x3f648291 (stale) CRC-32 checksum value for chunk [0, 0x16958): Mirror 1: 0x3f648291 Mirror 5: 0x3f648291 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16958: skipped 000703 1788202538.467666 fallocate 0x03c365 thru 0x03ffff (0x3c9b bytes) 000704 1788202538.525262 MIRROR_RESYNC 5 mirrors 000705 1788202539.773171 MIRROR_RESYNC 5 mirrors 000707 1788202540.232435 write 0x024f4b thru 0x02cdc8 (0x7e7e bytes) 000708 1788202540.284052 trunc from 0x0086b3 to 0x0168e0 (0xe22e bytes) 000709 1788202540.290019 write 0x02ed70 thru 0x036ec6 (0x8157 bytes) 000710 1788202540.292827 trunc from 0x036ec7 to 0x010e04 (0x260c2 bytes) 000711 1788202540.318518 read 0x00efd6 thru 0x010e04 (0x1e2f bytes) 000712 1788202540.325133 punch from 0x009d44 to 0x00a031 (0x2ee bytes) 000714 1788202540.338403 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10e05): Mirror 2: 0xc0ee717e (stale) Mirror 3: 0xc0ee717e (stale) Mirror 4: 0xc0ee717e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10e05): Mirror 2: 0xc0ee717e (stale) Mirror 3: 0xc0ee717e (stale) Mirror 4: 0xc0ee717e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10e05: skipped 000715 1788202540.726391 trunc from 0x010e05 to 0x02ae7e (0x1a07a bytes) 000716 1788202540.761988 trunc from 0x01200e to 0x01615a (0x414d bytes) 000717 1788202540.779780 read 0x012588 thru 0x0208bc (0xe335 bytes) 000718 1788202540.800697 mapread 0x00b7e6 thru 0x00cca2 (0x14bd bytes) 000719 1788202540.824763 mapwrite 0x024277 thru 0x02f499 (0xb223 bytes) 000720 1788202540.928141 trunc from 0x029b85 to 0x02f499 (0x5915 bytes) 000721 1788202540.937746 read 0x02f2a1 thru 0x02f499 (0x1f9 bytes) 000722 1788202540.956645 punch from 0x021198 to 0x02e2e0 (0xd149 bytes) 000723 1788202540.960995 trunc from 0x02216a to 0x02db90 (0xba27 bytes) 000724 1788202540.971379 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f49a): Mirror 3: 0xba2b65de (stale) Mirror 4: 0xba2b65de (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f49a): Mirror 3: 0xba2b65de (stale) Mirror 4: 0xba2b65de (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f49a: skipped 000725 1788202541.386733 punch from 0x007058 to 0x009dad (0x2d56 bytes) 000726 1788202541.397359 trunc from 0x02f49a to 0x00d585 (0x21f14 bytes) 000727 1788202541.428451 mapread 0x0045f6 thru 0x00b44b (0x6e56 bytes) 000728 1788202541.489281 punch from 0x007876 to 0x00d585 (0x5d10 bytes) 000729 1788202541.498394 trunc from 0x0024af to 0x00862c (0x617e bytes) 000730 1788202541.506891 write 0x0190d7 thru 0x019839 (0x763 bytes) 000731 1788202541.508223 trunc from 0x016f17 to 0x019839 (0x2923 bytes) 000732 1788202541.533469 punch from 0x005aa5 to 0x00c950 (0x6eac bytes) 000734 1788202541.548771 mapwrite 0x03a02a thru 0x03ffff (0x5fd6 bytes) 000735 1788202541.650406 MIRROR_RESYNC 3 mirrors 000736 1788202542.562952 mapread 0x00b2ba thru 0x01a3f8 (0xf13f bytes) 000738 1788202542.623248 trunc from 0x03e067 to 0x03e31f (0x2b9 bytes) 000739 1788202542.699725 trunc from 0x040000 to 0x03e1d3 (0x1e2c bytes) 000740 1788202542.734425 mapread 0x009680 thru 0x018b2b (0xf4ac bytes) 000741 1788202542.800206 MIRROR_RESYNC 3 mirrors 000742 1788202544.052445 trunc from 0x000fbb to 0x00dfcb (0xd011 bytes) 000743 1788202544.094572 fallocate 0x031f23 thru 0x0374f8 (0x55d6 bytes) 000745 1788202544.103674 read 0x02d46e thru 0x034326 (0x6eb9 bytes) 000746 1788202544.113668 trunc from 0x03e1d4 to 0x023d8b (0x1a448 bytes) 000747 1788202544.143989 trunc from 0x01671a to 0x01878c (0x2073 bytes) 000748 1788202544.154419 MIRROR_RESYNC 3 mirrors 000749 1788202545.024997 fallocate 0x02eddc thru 0x03416c (0x5391 bytes) 000750 1788202545.079991 mapread 0x02147c thru 0x023d8b (0x2910 bytes) 000751 1788202545.132374 mapread 0x00ef55 thru 0x01023d (0x12e9 bytes) 000752 1788202545.152532 mapread 0x016e30 thru 0x01b276 (0x4447 bytes) 000753 1788202545.176507 write 0x00942e thru 0x01397a (0xa54d bytes) 000754 1788202545.187722 trunc from 0x007c89 to 0x00bcf7 (0x406f bytes) 000755 1788202545.212562 mapread 0x022c17 thru 0x023d8b (0x1175 bytes) 000756 1788202545.218114 read 0x00a0f8 thru 0x0137e6 (0x96ef bytes) 000757 1788202545.236244 trunc from 0x02127d to 0x023d8b (0x2b0f bytes) 000758 1788202545.254329 mapread 0x01c565 thru 0x023d8b (0x7827 bytes) 000759 1788202545.267028 fallocate 0x012a52 thru 0x017cd3 (0x5282 bytes) 000760 1788202545.289044 punch from 0x01512d to 0x023d8b (0xec5f bytes) 000762 1788202545.306577 mapread 0x0062f3 thru 0x00769e (0x13ac bytes) 000763 1788202545.330373 write 0x022fc3 thru 0x02e1a8 (0xb1e6 bytes) 000764 1788202545.349939 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2e1a9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e1a9): Mirror 3: 0xe25681ec (stale) Mirror 4: 0xe25681ec (stale) CRC-32 checksum value for chunk [0, 0x2e1a9): Mirror 1: 0xf8a33870 Mirror 5: 0xf8a33870 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e1a9: skipped 000765 1788202546.308373 trunc from 0x004140 to 0x00a38d (0x624e bytes) 000766 1788202546.357714 fallocate 0x00503f thru 0x00a002 (0x4fc4 bytes) 000767 1788202546.368891 trunc from 0x02460e to 0x02e1a8 (0x9b9b bytes) 000768 1788202546.380625 fallocate 0x01d4ce thru 0x02cc3b (0xf76e bytes) 000769 1788202546.396420 trunc from 0x00b8b1 to 0x0196fe (0xde4e bytes) 000770 1788202546.406789 trunc from 0x02d4cf to 0x02e1a8 (0xcda bytes) 000771 1788202546.415284 punch from 0x019110 to 0x02851f (0xf410 bytes) 000772 1788202546.424161 fallocate 0x02a702 thru 0x03711c (0xca1b bytes) 000773 1788202546.430416 mapwrite 0x000e1c thru 0x00c863 (0xba48 bytes) 000774 1788202546.530191 read 0x01021c thru 0x013aa5 (0x388a bytes) 000775 1788202546.538194 punch from 0x00f6b8 to 0x017ce9 (0x8632 bytes) 000776 1788202546.544597 punch from 0x0143eb to 0x018a99 (0x46af bytes) 000777 1788202546.554083 write 0x00340c thru 0x006c81 (0x3876 bytes) 000778 1788202546.563301 mapread 0x02968c thru 0x02cf5b (0x38d0 bytes) 000779 1788202546.595549 write 0x029f59 thru 0x032bf4 (0x8c9c bytes) 000780 1788202546.600559 fallocate 0x022ed4 thru 0x024ee5 (0x2012 bytes) 000781 1788202546.614749 read 0x017183 thru 0x025785 (0xe603 bytes) 000782 1788202546.635630 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x32bf5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32bf5): Mirror 3: 0x1b376473 (stale) Mirror 4: 0x1b376473 (stale) Mirror 5: 0x6ba6a832 (stale) CRC-32 checksum value for chunk [0, 0x32bf5): Mirror 1: 0x16dc5cdd Mirror 6: 0x16dc5cdd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32bf5: skipped 000783 1788202547.696989 mapread 0x010791 thru 0x01fea1 (0xf711 bytes) 000785 1788202547.758160 punch from 0x02e562 to 0x032bf4 (0x4693 bytes) 000788 1788202547.833261 mapread 0x011346 thru 0x01407d (0x2d38 bytes) 000789 1788202547.859367 trunc from 0x008e5a to 0x0147cb (0xb972 bytes) 000790 1788202547.867028 write 0x004c19 thru 0x005fe8 (0x13d0 bytes) 000791 1788202547.896660 fallocate 0x032d59 thru 0x034b02 (0x1daa bytes) 000792 1788202547.902867 mapwrite 0x02310f thru 0x02b0d4 (0x7fc6 bytes) 000793 1788202547.936635 punch from 0x02e91c to 0x031a6e (0x3153 bytes) 000794 1788202547.942249 read 0x017329 thru 0x0248c8 (0xd5a0 bytes) 000795 1788202547.948076 read 0x029be7 thru 0x02f5c1 (0x59db bytes) 000796 1788202548.008629 mapread 0x01fe26 thru 0x0249b5 (0x4b90 bytes) 000797 1788202548.012091 read 0x031e29 thru 0x034b02 (0x2cda bytes) 000798 1788202548.013131 trunc from 0x019b8b to 0x0280d1 (0xe547 bytes) 000799 1788202548.020962 mapwrite 0x033179 thru 0x037388 (0x4210 bytes) 000801 1788202548.073872 write 0x01a9f3 thru 0x022c6f (0x827d bytes) 000802 1788202548.076103 mapread 0x0075fa thru 0x00d7df (0x61e6 bytes) 000803 1788202548.077782 mapread 0x0318f7 thru 0x035529 (0x3c33 bytes) 000804 1788202548.079757 fallocate 0x03f25f thru 0x03f55a (0x2fc bytes) 000806 1788202548.085769 write 0x016b7f thru 0x020a17 (0x9e99 bytes) 000807 1788202548.088189 punch from 0x012957 to 0x01dafc (0xb1a6 bytes) 000808 1788202548.140782 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3f55b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f55b): Mirror 3: 0x8d54f82c (stale) Mirror 4: 0x8d54f82c (stale) Mirror 5: 0x4dd1102a (stale) Mirror 6: 0xcb577e10 (stale) CRC-32 checksum value for chunk [0, 0x3f55b): Mirror 1: 0x2fe78f1d Mirror 7: 0x2fe78f1d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f55b: skipped 000809 1788202549.337261 punch from 0x008d0c to 0x0132b1 (0xa5a6 bytes) 000810 1788202549.370798 punch from 0x01b692 to 0x01bcf2 (0x661 bytes) 000811 1788202549.376266 read 0x03284f thru 0x03481c (0x1fce bytes) 000812 1788202549.382429 trunc from 0x00e93e to 0x018dc9 (0xa48c bytes) 000813 1788202549.387740 trunc from 0x03f55b to 0x03e7fc (0xd5e bytes) 000814 1788202549.449139 mapwrite 0x006fbf thru 0x00d516 (0x6558 bytes) 000815 1788202549.516977 mapread 0x00699b thru 0x0127c1 (0xbe27 bytes) 000817 1788202549.530239 punch from 0x0017eb to 0x0049bd (0x31d3 bytes) 000818 1788202549.535019 mapwrite 0x036a28 thru 0x03d16c (0x6745 bytes) 000819 1788202549.569923 trunc from 0x03e7fd to 0x036465 (0x8397 bytes) 000821 1788202549.590441 read 0x033edf thru 0x036465 (0x2587 bytes) 000822 1788202549.591724 trunc from 0x0155a5 to 0x0160e6 (0xb42 bytes) 000823 1788202549.604356 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36466): Mirror 3: 0x7b843991 (stale) Mirror 5: 0xb141b78 (stale) Mirror 6: 0xeadd8a91 (stale) Mirror 7: 0x28af8901 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36466): Mirror 3: 0x7b843991 (stale) Mirror 5: 0xb141b78 (stale) Mirror 6: 0xeadd8a91 (stale) Mirror 7: 0x28af8901 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36466: skipped 000824 1788202549.984749 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36466): Mirror 5: 0xb141b78 (stale) Mirror 6: 0xeadd8a91 (stale) Mirror 7: 0x28af8901 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36466): Mirror 5: 0xb141b78 (stale) Mirror 6: 0xeadd8a91 (stale) Mirror 7: 0x28af8901 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36466: skipped 000825 1788202550.292923 trunc from 0x02cfb4 to 0x036465 (0x94b2 bytes) 000827 1788202550.300897 mapread 0x028890 thru 0x031c57 (0x93c8 bytes) 000828 1788202550.324753 write 0x007250 thru 0x00b8b4 (0x4665 bytes) 000829 1788202550.339082 trunc from 0x036466 to 0x02f89a (0x6bcb bytes) 000830 1788202550.363180 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f89b): Mirror 5: 0x621af286 (stale) Mirror 7: 0xb4797db4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f89b): Mirror 5: 0x621af286 (stale) Mirror 7: 0xb4797db4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f89b: skipped 000831 1788202550.665041 mapread 0x02bf59 thru 0x02f89a (0x3942 bytes) 000833 1788202550.689977 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2f89b): Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f89b): Mirror 5: 0x621af286 (stale) Mirror 7: 0xb4797db4 (stale) CRC-32 checksum value for chunk [0, 0x2f89b): Mirror 1: 0x32f01895 Mirror 8: 0x32f01895 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f89b: skipped 000834 1788202551.746000 punch from 0x00d97a to 0x01d097 (0xf71e bytes) 000835 1788202551.794473 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f89b): Mirror 5: 0x621af286 (stale) Mirror 8: 0x32f01895 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f89b): Mirror 5: 0x621af286 (stale) Mirror 8: 0x32f01895 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f89b: skipped 000836 1788202552.148398 read 0x022a63 thru 0x027a1a (0x4fb8 bytes) 000837 1788202552.156668 trunc from 0x02f89b to 0x017713 (0x18187 bytes) 000838 1788202552.174255 trunc from 0x00f694 to 0x017713 (0x8080 bytes) 000839 1788202552.180757 mapwrite 0x03e11d thru 0x03ffff (0x1ee3 bytes) 000840 1788202552.239570 trunc from 0x02f67c to 0x03d7d5 (0xe15a bytes) 000841 1788202552.262258 mapread 0x012bc3 thru 0x02296b (0xfda9 bytes) 000842 1788202552.299180 trunc from 0x02e4b0 to 0x032c0d (0x475e bytes) 000843 1788202552.315251 read 0x0150f2 thru 0x01d384 (0x8293 bytes) 000844 1788202552.318024 trunc from 0x013251 to 0x0190b1 (0x5e61 bytes) 000845 1788202552.330192 trunc from 0x040000 to 0x00200e (0x3dff1 bytes) 000846 1788202552.359359 MIRROR_RESYNC 3 mirrors 000848 1788202553.024649 mapread 0x000431 thru 0x00200e (0x1bde bytes) 000849 1788202553.050152 punch from 0x0008cf to 0x00200e (0x1740 bytes) 000850 1788202553.097193 write 0x01702b thru 0x0223db (0xb3b1 bytes) 000851 1788202553.101400 fallocate 0x03fea3 thru 0x03ffff (0x15d bytes) 000852 1788202553.108105 trunc from 0x0223dc to 0x01611f (0xc2bc bytes) 000853 1788202553.144110 write 0x0143cf thru 0x01e220 (0x9e52 bytes) 000854 1788202553.149905 mapwrite 0x03991d thru 0x03ffff (0x66e3 bytes) 000856 1788202553.250228 mapread 0x011f33 thru 0x01509e (0x316c bytes) 000857 1788202553.267635 write 0x01641d thru 0x018c6e (0x2852 bytes) 000858 1788202553.268757 read 0x00f0b7 thru 0x01c56a (0xd4b4 bytes) 000859 1788202553.270435 write 0x01e433 thru 0x026d14 (0x88e2 bytes) 000860 1788202553.278939 fallocate 0x0044b6 thru 0x009a1c (0x5567 bytes) 000861 1788202553.285029 trunc from 0x008a02 to 0x016894 (0xde93 bytes) 000862 1788202553.323610 MIRROR_RESYNC 3 mirrors 000863 1788202554.017127 trunc from 0x00ba12 to 0x012039 (0x6628 bytes) 000864 1788202554.054744 write 0x021126 thru 0x02ec6e (0xdb49 bytes) 000865 1788202554.064636 trunc from 0x040000 to 0x00dac1 (0x3253e bytes) 000866 1788202554.107870 write 0x026046 thru 0x032da7 (0xcd62 bytes) 000867 1788202554.112143 trunc from 0x023564 to 0x02b5be (0x805b bytes) 000868 1788202554.128729 punch from 0x029300 to 0x02e154 (0x4e55 bytes) 000869 1788202554.132927 punch from 0x01de59 to 0x023b01 (0x5ca9 bytes) 000870 1788202554.139491 read 0x0106fd thru 0x01df3c (0xd840 bytes) 000871 1788202554.149841 mapread 0x02150a thru 0x02a805 (0x92fc bytes) 000872 1788202554.174692 fallocate 0x02bf94 thru 0x02c55f (0x5cc bytes) 000873 1788202554.184481 trunc from 0x032da8 to 0x015a2d (0x1d37a bytes) 000874 1788202554.218174 mapread 0x007b84 thru 0x015a2d (0xdeaa bytes) 000875 1788202554.238437 trunc from 0x015a2e to 0x01276b (0x32c2 bytes) 000876 1788202554.270333 punch from 0x00a9a0 to 0x00ec23 (0x4284 bytes) 000877 1788202554.284544 punch from 0x00ebcd to 0x01276b (0x3b9f bytes) 000878 1788202554.290712 fallocate 0x007265 thru 0x00fd24 (0x8ac0 bytes) 000879 1788202554.296778 write 0x0109d1 thru 0x01f400 (0xea30 bytes) 000880 1788202554.305145 trunc from 0x01f401 to 0x033b4f (0x1474f bytes) 000881 1788202554.331044 fallocate 0x0081ee thru 0x0150b2 (0xcec5 bytes) 000882 1788202554.339532 write 0x01424c thru 0x0231c7 (0xef7c bytes) 000883 1788202554.355192 trunc from 0x033b50 to 0x0251b9 (0xe996 bytes) 000885 1788202554.385440 write 0x00f223 thru 0x018888 (0x9666 bytes) 000886 1788202554.397219 mapread 0x014067 thru 0x01d22b (0x91c5 bytes) 000887 1788202554.398535 trunc from 0x0084c7 to 0x00bbe8 (0x3722 bytes) 000888 1788202554.409322 write 0x001cb5 thru 0x00be5b (0xa1a7 bytes) 000889 1788202554.415986 read 0x024ade thru 0x0251b9 (0x6dc bytes) 000890 1788202554.427057 read 0x00be20 thru 0x0121e2 (0x63c3 bytes) 000891 1788202554.428470 punch from 0x00537e to 0x00f300 (0x9f83 bytes) 000892 1788202554.452902 trunc from 0x00b991 to 0x010b14 (0x5184 bytes) 000893 1788202554.457998 write 0x00764e thru 0x012e2c (0xb7df bytes) 000894 1788202554.471500 read 0x002cdb thru 0x00e2df (0xb605 bytes) 000895 1788202554.476562 MIRROR_RESYNC 3 mirrors 000896 1788202555.141512 trunc from 0x0251ba to 0x0184a9 (0xcd10 bytes) 000898 1788202555.204947 mapwrite 0x02bb40 thru 0x02cf54 (0x1415 bytes) 000899 1788202555.260258 mapwrite 0x03f37e thru 0x03ffff (0xc82 bytes) 000900 1788202555.299732 punch from 0x02452e to 0x02c2e5 (0x7db8 bytes) 000901 1788202555.307292 write 0x039cf0 thru 0x03ffff (0x6310 bytes) 000902 1788202555.309692 trunc from 0x01dda3 to 0x02217e (0x43dc bytes) 000903 1788202555.319563 trunc from 0x00ca40 to 0x01529a (0x885b bytes) 000904 1788202555.328568 mapread 0x000b37 thru 0x00b71f (0xabe9 bytes) 000905 1788202555.339351 fallocate 0x016f64 thru 0x025adb (0xeb78 bytes) 000906 1788202555.345842 trunc from 0x040000 to 0x03121f (0xede0 bytes) 000907 1788202555.365618 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31220): Mirror 5: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31220): Mirror 5: 0xf76af94b (stale) Mirror 8: 0xf76af94b (stale) CRC-32 checksum value for chunk [0, 0x31220): Mirror 1: 0xeff8f236 Mirror 9: 0xeff8f236 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31220: skipped 000908 1788202556.299704 punch from 0x015c62 to 0x017a5f (0x1dfe bytes) 000909 1788202556.344246 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31220): Mirror 8: 0xf76af94b (stale) Mirror 9: 0xeff8f236 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31220): Mirror 8: 0xf76af94b (stale) Mirror 9: 0xeff8f236 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31220: skipped 000910 1788202556.609236 mapread 0x023125 thru 0x02ad44 (0x7c20 bytes) 000911 1788202556.622963 write 0x01acc7 thru 0x020e0b (0x6145 bytes) 000912 1788202556.634072 punch from 0x0191e3 to 0x01b797 (0x25b5 bytes) 000913 1788202556.647408 fallocate 0x0080fd thru 0x008fa6 (0xeaa bytes) 000914 1788202556.652005 punch from 0x017388 to 0x01bc5e (0x48d7 bytes) 000915 1788202556.657098 mapwrite 0x035c2b thru 0x03606a (0x440 bytes) 000916 1788202556.694169 read 0x0164a1 thru 0x02630c (0xfe6c bytes) 000917 1788202556.705158 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3606b): Mirror 9: 0x1125ec82 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3606b): Mirror 9: 0x1125ec82 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3606b: skipped 000918 1788202556.949030 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x53:0x0]: Invalid argument (22) 000919 1788202557.065695 trunc from 0x03606b to 0x01b97e (0x1a6ec bytes) 000920 1788202557.086704 write 0x002183 thru 0x00d664 (0xb4e2 bytes) 000921 1788202557.100458 trunc from 0x01609a to 0x01b97e (0x58e5 bytes) 000922 1788202557.108273 trunc from 0x01b97f to 0x00dd2f (0xdc4f bytes) 000923 1788202557.122309 trunc from 0x00dd30 to 0x01ee36 (0x11107 bytes) 000924 1788202557.136745 fallocate 0x03fa85 thru 0x03ffff (0x57b bytes) 000925 1788202557.144390 trunc from 0x003481 to 0x010315 (0xce95 bytes) 000927 1788202557.163621 write 0x02ac8a thru 0x038b47 (0xdebe bytes) 000928 1788202557.165711 read 0x002425 thru 0x00408f (0x1c6b bytes) 000930 1788202557.175784 write 0x03c78a thru 0x03ffff (0x3876 bytes) 000931 1788202557.178488 read 0x02586c thru 0x02ad3c (0x54d1 bytes) 000932 1788202557.187395 trunc from 0x024018 to 0x029c96 (0x5c7f bytes) 000933 1788202557.192376 read 0x00b119 thru 0x019d20 (0xec08 bytes) 000934 1788202557.193293 mapread 0x0266d8 thru 0x031c78 (0xb5a1 bytes) 000936 1788202557.202105 write 0x02d986 thru 0x0387e1 (0xae5c bytes) 000937 1788202557.203595 trunc from 0x040000 to 0x02eabe (0x11541 bytes) 000938 1788202557.222093 fallocate 0x03fb82 thru 0x03ffff (0x47e bytes) 000939 1788202557.228526 write 0x020d21 thru 0x029701 (0x89e1 bytes) 000940 1788202557.230542 read 0x03afaa thru 0x03ffff (0x5056 bytes) 000941 1788202557.242238 punch from 0x013f3a to 0x01fe2e (0xbef5 bytes) 000942 1788202557.249384 trunc from 0x040000 to 0x00e7f1 (0x3180e bytes) 000943 1788202557.265295 trunc from 0x0019d6 to 0x007370 (0x599b bytes) 000944 1788202557.275660 MIRROR_EXTEND to 2 mirrors 000945 1788202557.814014 mapwrite 0x0135bd thru 0x013d08 (0x74c bytes) 000946 1788202557.902797 trunc from 0x013d09 to 0x00b312 (0x89f6 bytes) 000947 1788202557.921390 trunc from 0x00b313 to 0x01a613 (0xf301 bytes) 000948 1788202557.941358 read 0x00b1c6 thru 0x0141d5 (0x9010 bytes) 000949 1788202557.954906 punch from 0x014860 to 0x01a613 (0x5db4 bytes) 000950 1788202557.963437 mapread 0x00725d thru 0x0167a1 (0xf545 bytes) 000952 1788202557.983915 mapwrite 0x02c1f3 thru 0x032f31 (0x6d3f bytes) 000954 1788202558.043059 punch from 0x00d67a to 0x00e639 (0xfc0 bytes) 000956 1788202558.049759 mapread 0x00f74f thru 0x01a755 (0xb007 bytes) 000957 1788202558.051522 read 0x01fb88 thru 0x02445f (0x48d8 bytes) 000958 1788202558.058916 write 0x0348e8 thru 0x03ffff (0xb718 bytes) 000959 1788202558.060422 write 0x00dfdd thru 0x012140 (0x4164 bytes) 000960 1788202558.063273 trunc from 0x040000 to 0x031869 (0xe796 bytes) 000961 1788202558.083782 punch from 0x01f1c8 to 0x029c7b (0xaab4 bytes) 000962 1788202558.092322 MIRROR_RESYNC 2 mirrors 000963 1788202558.677698 fallocate 0x0260a4 thru 0x02c042 (0x5f9f bytes) 000964 1788202558.737704 trunc from 0x02d7d2 to 0x031869 (0x4098 bytes) 000965 1788202558.743708 read 0x003290 thru 0x004abb (0x182c bytes) 000966 1788202558.755994 trunc from 0x03186a to 0x03a32f (0x8ac6 bytes) 000967 1788202558.799108 trunc from 0x033b41 to 0x03a32f (0x67ef bytes) 000968 1788202558.803091 trunc from 0x03a330 to 0x021983 (0x189ac bytes) 000969 1788202558.824591 mapwrite 0x006f1f thru 0x0163ac (0xf48e bytes) 000970 1788202558.889586 fallocate 0x004cfd thru 0x010bcf (0xbed3 bytes) 000971 1788202558.898043 MIRROR_RESYNC 2 mirrors 000972 1788202559.473976 write 0x017562 thru 0x01ed72 (0x7811 bytes) 000974 1788202559.521622 mapwrite 0x03fa1c thru 0x03ffff (0x5e4 bytes) 000975 1788202559.573908 mapwrite 0x03d255 thru 0x03ffff (0x2dab bytes) 000976 1788202559.622361 write 0x0270f6 thru 0x03379d (0xc6a8 bytes) 000977 1788202559.647803 punch from 0x009656 to 0x00993e (0x2e9 bytes) 000978 1788202559.652701 mapread 0x0263a1 thru 0x028102 (0x1d62 bytes) 000979 1788202559.668947 punch from 0x01a42c to 0x0237c0 (0x9395 bytes) 000980 1788202559.693469 MIRROR_RESYNC 2 mirrors 000981 1788202560.347518 trunc from 0x01c92e to 0x027951 (0xb024 bytes) 000982 1788202560.394280 mapwrite 0x02ff2f thru 0x03cf0c (0xcfde bytes) 000983 1788202560.461076 mapwrite 0x006489 thru 0x006bba (0x732 bytes) 000984 1788202560.496403 read 0x019892 thru 0x01db4d (0x42bc bytes) 000985 1788202560.506629 mapwrite 0x02e1c1 thru 0x0329cb (0x480b bytes) 000986 1788202560.528618 read 0x02ca28 thru 0x031cb0 (0x5289 bytes) 000987 1788202560.529595 trunc from 0x02091a to 0x02528a (0x4971 bytes) 000989 1788202560.543870 write 0x03e83e thru 0x03ffff (0x17c2 bytes) 000990 1788202560.545669 mapwrite 0x00fa11 thru 0x01e182 (0xe772 bytes) 000991 1788202560.577873 mapread 0x03bb06 thru 0x03ffff (0x44fa bytes) 000992 1788202560.583148 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x550e3b7c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3a94ec36 Mirror 3: 0x3a94ec36 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000993 1788202561.357923 punch from 0x01c097 to 0x021d47 (0x5cb1 bytes) 000995 1788202561.413104 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x550e3b7c (stale) Mirror 3: 0x3a94ec36 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5922e8d2 Mirror 4: 0x5922e8d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000996 1788202562.176490 mapread 0x01442d thru 0x01db02 (0x96d6 bytes) 000997 1788202562.217250 trunc from 0x00941b to 0x017684 (0xe26a bytes) 000998 1788202562.263331 read 0x0099d7 thru 0x015352 (0xb97c bytes) 000999 1788202562.279482 write 0x019111 thru 0x025540 (0xc430 bytes) 001000 1788202562.296092 punch from 0x025224 to 0x030a8a (0xb867 bytes) All operations completed A-OK! PASS 70a (83s) == sanity-flr test 71: check mirror extend parent fid ==== 14:56:06 (1788202566) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1b0:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (8s) == sanity-flr test 72: check append on FLR file ========== 14:56:14 (1788202574) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.21978 s, 9.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16951 s, 6.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 14:56:21 (1788202581) Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre2 Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3523104 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7044480 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 6252928 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 7110336 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7423392 .. Extending file size to 7314464 .. Extending file size to 9062560 .. Extending file size to 9101696 .. Extending file size to 7989152 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9161760 .. Extending file size to 8536160 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61446 61448 61449 61451 61453 done 192.168.206.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg627-client.virtnet /mnt/lustre2 (opts:) 192.168.206.127@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg627-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:57:41 (1788202661) Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre2 Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 926432 .. Extending file size to 922656 .. failed Extending file size to 4253952 .. Extending file size to 3665856 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6516864 .. Extending file size to 8733120 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8862848 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4903712 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8926592 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 5819616 .. Extending file size to 7559488 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8970848 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7738176 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8855232 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8958528 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 63103 63104 63106 63108 63110 192.168.206.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg627-client.virtnet /mnt/lustre2 (opts:) 192.168.206.127@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg627-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:58:58 (1788202738) Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1268896 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2100 1269204 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25348 3581672 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17716 3589304 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43064 7170976 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2408 1268896 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2100 1269204 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25348 3581672 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17716 3589304 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43064 7170976 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.101026 s, 20.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.181108 s, 11.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x54402cd1 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg627-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:59:16 (1788202756) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104936 s, 20.0 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 14:59:22 (1788202762) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x98:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14b:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x148:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:59:28 (1788202768) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.137835 s, 15.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x247:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x249:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:59:36 (1788202776) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0874282 s, 24.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x149:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x149:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14a:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:59:43 (1788202783) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141477 s, 14.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:00:16 (1788202816) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg627-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553941 s, 18.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x248:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x248:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x249:0x0] } PASS 204d (47s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:01:03 (1788202863) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg627-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0731919 s, 28.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (52s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:01:56 (1788202916) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg627-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg627-server: Pool lustre.test_204f created Adding targets to pool oleg627-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg627-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141894 s, 14.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x24d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg627-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg627-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg627-server: Pool lustre.test_204f destroyed PASS 204f (75s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:03:11 (1788202991) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:03:18 (1788202998) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:03:25 (1788203005) oleg627-server: Pool lustre.test_206 created Adding targets to pool oleg627-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg627-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg627-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg627-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg627-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:03:51 (1788203031) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125738 s, 8.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411139 s, 25.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x8d371759 Mirror 3: 0x8d371759 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:04:02 (1788203042) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:04:05 (1788203045) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:04:09 (1788203049) Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0559978 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2050000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x10f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x10f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x10f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1a40000 Read this (4 bytes): BBBB 192.168.206.127@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg627-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:04:44 (1788203084) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0224198 s, 365 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:04:51 (1788203091) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500963 s, 20.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb7 lmm_fid: [0x200000405:0xb7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb7 lmm_fid: [0x200000405:0xb7:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x159:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611319 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-31 15:04:52.000000000 -0400 Modify: 2026-08-31 15:04:52.000000000 -0400 Change: 2026-08-31 15:04:53.000000000 -0400 Birth: 2026-08-31 15:04:53.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:04:58 (1788203098) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239169 s, 43.8 MB/s oleg627-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg627-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg627-server: O/280000401/d26/15a: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:05:19 (1788203119) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.199576 s, 205 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0182384 s, 225 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 31 15:05 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbc:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 31 15:05 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:05:26 (1788203126) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:05:33 (1788203133) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1722 sec =========== 15:05:40 (1788203140) === sanity-flr: start cleanup 15:05:41 (1788203141) === === sanity-flr: finish cleanup 15:05:49 (1788203149) ===