-----============= acceptance-small: sanity-flr ============----- Mon Jul 6 10:56:27 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg222-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 10:56:39 (1783349799) === oleg222-client.virtnet: executing check_config_client /mnt/lustre oleg222-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg222-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9a07c8c4d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a07c8c4d800.idle_timeout=debug disable quota as required oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 10:57:03 (1783349823) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8210] == sanity-flr test 0a: lfs mirror create with -N option == 10:57:05 (1783349825) PASS 0a (10s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:57:15 (1783349835) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:57:17 (1783349837) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:57:19 (1783349839) PASS 0d (14s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:57:33 (1783349853) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:57:36 (1783349856) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 10:57:38 (1783349858) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9a07c8c4d800.stats=clear osc.lustre-OST0001-osc-ffff9a07c8c4d800.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:57:42 (1783349862) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (7s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:57:49 (1783349869) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:57:54 (1783349874) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:57:56 (1783349876) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (5s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:58:01 (1783349881) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (6s) == sanity-flr test 2: create components from existing files ========================================================== 10:58:07 (1783349887) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:58:11 (1783349891) PASS 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:58:16 (1783349896) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:58:22 (1783349902) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (11s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:58:38 (1783349918) PASS 7 (11s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:58:49 (1783349929) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 2.08089 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161903 s, 6.5 MB/s PASS 21 (17s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:59:06 (1783349946) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 1.38126 s, 14.4 MB/s PASS 22 (17s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:59:24 (1783349964) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (15s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:59:39 (1783349979) Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a07c8c4d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a07c8c4d800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a07c8c4d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a07c8c4d800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg222-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (71s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 11:00:50 (1783350050) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.522978 s, 18.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a07c8c4d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a07c8c4d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 11:01:34 (1783350094) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a07c8c4d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a07c8c4d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a07c8c4d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a07c8c4d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg222-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (76s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 11:02:51 (1783350171) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.29125 s, 15.9 MB/s -rw-r--r-- 1 root root 52428800 Jul 6 11:02 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.322s user 0m0.000s sys 0m0.162s Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a07c8c4d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a07c8c4d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.232s user 0m0.000s sys 0m0.171s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a07c8c4d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a07c8c4d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.124s user 0m0.002s sys 0m0.173s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg222-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg222-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a07c8c4d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a07c8c4d800.ost_server_uuid in FULL state after 0 sec PASS 33b (92s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 11:04:23 (1783350263) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 11:04:26 (1783350266) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 11:04:30 (1783350270) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 11:04:34 (1783350274) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 11:04:41 (1783350281) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.677642 s, 18.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 11:04:55 (1783350295) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0263337 s, 39.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036972 s, 28.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 11:05:02 (1783350302) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 11:05:11 (1783350311) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00141094 s, 726 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00212311 s, 482 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00109906 s, 932 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00127659 s, 802 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 11:05:22 (1783350322) create test files with size 15732637 .. 7858cb9478157cf7cfad88e9508d1a64 - f18030713d1c5959ca6bdbba8505faf1 - dc457a659077c8cb9acb4045dd38f069 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 11:05:47 (1783350347) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 11:06:03 (1783350363) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 11:06:10 (1783350370) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693072 s, 15.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215921 s, 48.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc0:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 11:06:20 (1783350380) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118915 s, 17.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.404265 s, 10.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.227486 s, 9.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 11:06:38 (1783350398) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 11:06:42 (1783350402) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 11:06:45 (1783350405) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0476757 s, 22.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 11:06:54 (1783350414) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.325347 s, 9.7 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 11:07:05 (1783350425) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xca:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xca:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xca:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 11:07:15 (1783350435) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.389105 s, 26.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 11:07:27 (1783350447) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9a07c602b000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.341395 s, 30.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9a07ca017800.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 11:07:46 (1783350466) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (41s) == sanity-flr test 46: Verify setstripe --copy option ==== 11:08:28 (1783350508) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 11:09:10 (1783350550) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 11:09:14 (1783350554) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.300559 s, 10.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.154796 s, 20.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xc5378a9e (nosync,stale) Mirror 2: 0xc5378a9e (nosync,stale) ** verify mirror 2 doesn't change original checksum: 87dbe1eb5a44e7cf2308e43889c1fef2 - mirror 1 checksum: cd0d326fb2ff70ece989a2637d551771 - mirror 2 checksum: 87dbe1eb5a44e7cf2308e43889c1fef2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-06T11:09:17-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-06T11:09:17-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 87dbe1eb5a44e7cf2308e43889c1fef2 - mirror 1 checksum: cd0d326fb2ff70ece989a2637d551771 - mirror 2 checksum: cd0d326fb2ff70ece989a2637d551771 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-06T11:09:17-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-06T11:09:17-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 11:09:26 (1783350566) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496523 s, 21.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (10s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 11:09:36 (1783350576) Failing mds1 on oleg222-server Stopping /mnt/lustre-mds1 (opts:) on oleg222-server 11:09:50 (1783350590) shut down facet: mds1 facet_host: oleg222-server facet_failover_host: oleg222-server Failover mds1 to oleg222-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 11:10:21 (1783350621) targets are mounted 11:10:21 (1783350621) facet_failover done oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234192 s, 44.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:10:40 (1783350640) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.059253 s, 346 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0228502 s, 896 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:10:51 (1783350651) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0590341 s, 347 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0107 s, 1.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:11:03 (1783350663) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0213369 s, 768 kB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:11:12 (1783350672) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0846919 s, 242 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:11:21 (1783350681) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0127336 s, 322 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0890378 s, 46.0 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:11:29 (1783350689) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (22s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:11:51 (1783350711) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:12:21 (1783350741) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:12:45 (1783350765) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:13:11 (1783350791) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 11:13:19 (1783350799) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57365 mirror split pid 57367 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57365 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57367 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 11:14:27 (1783350867) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 868 truncating to largest ever: 0x15321 000001 1783350869.327960 trunc from 00000000 to 0x015320 (0x15321 bytes) 000002 1783350869.360512 mapread 0x00ebba thru 0x0139fa (0x4e41 bytes) truncating to largest ever: 0x1ce38 000004 1783350869.430022 trunc from 0x015321 to 0x01ce37 (0x7b17 bytes) 000005 1783350869.480735 read 0x0070b4 thru 0x012f64 (0xbeb1 bytes) 000006 1783350869.486149 write 0x00db47 thru 0x0118c9 (0x3d83 bytes) 000007 1783350869.487593 trunc from 0x007d75 to 0x012286 (0xa512 bytes) 000008 1783350869.523578 trunc from 0x007c29 to 0x01238d (0xa765 bytes) 000010 1783350869.534388 fallocate 0x000b2f thru 0x00ea84 (0xdf56 bytes) 000011 1783350869.541713 punch from 0x00257e to 0x00a912 (0x8395 bytes) 000012 1783350869.552197 read 0x0135bb thru 0x01ce37 (0x987d bytes) 000014 1783350869.567589 mapread 0x000726 thru 0x002ce9 (0x25c4 bytes) 000015 1783350869.573805 trunc from 0x0180a1 to 0x01ce37 (0x4d97 bytes) 000016 1783350869.592591 MIRROR_EXTEND to 2 mirrors 000017 1783350870.303800 mapwrite 0x015106 thru 0x01fb63 (0xaa5e bytes) 000018 1783350870.434121 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1fb64): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fb64): Mirror 2: 0x7dfe3850 (stale) CRC-32 checksum value for chunk [0, 0x1fb64): Mirror 1: 0x7efef9ff Mirror 3: 0x7efef9ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fb64: skipped 000019 1783350871.219560 mapwrite 0x0379d3 thru 0x03ce27 (0x5455 bytes) 000020 1783350871.336205 write 0x004e99 thru 0x011bae (0xcd16 bytes) 000021 1783350871.348545 read 0x014ea0 thru 0x01b36f (0x64d0 bytes) fallocating to largest ever: 0x40000 000022 1783350871.358691 fallocate 0x031f31 thru 0x03ffff (0xe0cf bytes) 000023 1783350871.364241 mapread 0x024b26 thru 0x027e63 (0x333e bytes) 000024 1783350871.375098 trunc from 0x033ea6 to 0x039aa5 (0x5c00 bytes) 000025 1783350871.381866 read 0x00c4a1 thru 0x00d09c (0xbfc bytes) 000026 1783350871.382440 fallocate 0x01eb20 thru 0x01ec17 (0xf8 bytes) 000027 1783350871.386749 write 0x006960 thru 0x00b05e (0x46ff bytes) 000028 1783350871.387673 mapwrite 0x01543f thru 0x01bc5f (0x6821 bytes) 000029 1783350871.408165 mapwrite 0x01ca32 thru 0x01d2de (0x8ad bytes) 000030 1783350871.433897 write 0x018959 thru 0x01c014 (0x36bc bytes) 000031 1783350871.435515 read 0x0327e9 thru 0x03ba76 (0x928e bytes) 000032 1783350871.436764 mapwrite 0x003836 thru 0x00d529 (0x9cf4 bytes) 000033 1783350871.460102 trunc from 0x0226f3 to 0x0277b9 (0x50c7 bytes) 000034 1783350871.465329 trunc from 0x01dd2b to 0x0224f5 (0x47cb bytes) 000035 1783350871.472078 trunc from 0x024f0d to 0x027156 (0x224a bytes) 000036 1783350871.476453 mapwrite 0x007026 thru 0x00d413 (0x63ee bytes) 000037 1783350871.497799 MIRROR_RESYNC 3 mirrors 000038 1783350872.620055 MIRROR_RESYNC 3 mirrors 000039 1783350873.012563 punch from 0x018578 to 0x026437 (0xdec0 bytes) 000040 1783350873.063386 mapwrite 0x03a10a thru 0x03ffff (0x5ef6 bytes) 000041 1783350873.128294 fallocate 0x01eb06 thru 0x0296db (0xabd6 bytes) 000042 1783350873.135893 fallocate 0x0341b0 thru 0x03ffff (0xbe50 bytes) 000043 1783350873.141205 punch from 0x0294f8 to 0x029bc6 (0x6cf bytes) 000044 1783350873.146653 trunc from 0x03711c to 0x03c4be (0x53a3 bytes) 000045 1783350873.151920 write 0x003f16 thru 0x008db3 (0x4e9e bytes) 000047 1783350873.166497 mapwrite 0x00b24d thru 0x011455 (0x6209 bytes) 000048 1783350873.198587 write 0x004350 thru 0x009920 (0x55d1 bytes) 000049 1783350873.201197 trunc from 0x040000 to 0x03a73b (0x58c4 bytes) 000050 1783350873.226020 mapwrite 0x03b993 thru 0x03f2f2 (0x3960 bytes) 000051 1783350873.302016 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f2f3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2f3): Mirror 2: 0xc6906dd4 (stale) Mirror 3: 0xc6906dd4 (stale) CRC-32 checksum value for chunk [0, 0x3f2f3): Mirror 1: 0x8a1a7ac2 Mirror 4: 0x8a1a7ac2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2f3: skipped 000053 1783350874.308091 mapread 0x00d722 thru 0x01348f (0x5d6e bytes) 000054 1783350874.345805 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f2f3): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f2f3): Mirror 2: 0xc6906dd4 (stale) Mirror 3: 0xc6906dd4 (stale) CRC-32 checksum value for chunk [0, 0x3f2f3): Mirror 1: 0x8a1a7ac2 Mirror 4: 0x8a1a7ac2 Mirror 5: 0x8a1a7ac2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f2f3: skipped 000055 1783350875.365363 mapread 0x027634 thru 0x036c89 (0xf656 bytes) 000056 1783350875.399934 punch from 0x02fc32 to 0x03b818 (0xbbe7 bytes) 000057 1783350875.450124 read 0x0397ef thru 0x03f2f2 (0x5b04 bytes) 000059 1783350875.475250 MIRROR_RESYNC 5 mirrors 000060 1783350876.999408 MIRROR_RESYNC 5 mirrors 000062 1783350877.564019 mapread 0x026b6e thru 0x02d8ce (0x6d61 bytes) 000063 1783350877.589977 write 0x00918d thru 0x015391 (0xc205 bytes) 000064 1783350877.625971 mapread 0x01cb2e thru 0x01f2f3 (0x27c6 bytes) 000065 1783350877.636897 trunc from 0x01d965 to 0x020bc6 (0x3262 bytes) 000066 1783350877.646339 MIRROR_RESYNC 5 mirrors 000067 1783350878.598660 MIRROR_RESYNC 5 mirrors 000068 1783350879.096242 mapwrite 0x00a661 thru 0x01796c (0xd30c bytes) 000069 1783350879.335605 trunc from 0x03f2f3 to 0x032836 (0xcabc bytes) 000070 1783350879.364227 mapread 0x0001e7 thru 0x0092c8 (0x90e2 bytes) 000071 1783350879.388957 mapwrite 0x00672d thru 0x006ba9 (0x47d bytes) 000072 1783350879.419445 read 0x008de5 thru 0x010575 (0x7791 bytes) 000073 1783350879.420143 mapread 0x009fbd thru 0x00e9c2 (0x4a06 bytes) 000074 1783350879.421152 fallocate 0x03d3e5 thru 0x03ffff (0x2c1b bytes) 000075 1783350879.430788 punch from 0x0040ca to 0x00dd22 (0x9c59 bytes) 000076 1783350879.436979 mapwrite 0x02c025 thru 0x0339e3 (0x79bf bytes) 000077 1783350879.486786 mapwrite 0x0322e8 thru 0x035208 (0x2f21 bytes) 000078 1783350879.516453 write 0x038f3b thru 0x03bb70 (0x2c36 bytes) 000079 1783350879.517921 punch from 0x005302 to 0x01437d (0xf07c bytes) 000080 1783350879.525008 trunc from 0x040000 to 0x029176 (0x16e89 bytes) 000081 1783350879.546646 fallocate 0x007943 thru 0x0116ea (0x9da8 bytes) 000082 1783350879.561082 MIRROR_RESYNC 5 mirrors 000083 1783350880.793711 mapwrite 0x023ee2 thru 0x02617a (0x2299 bytes) 000084 1783350880.928962 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x29177): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29177): Mirror 2: 0x9746ce6b (stale) Mirror 3: 0x9746ce6b (stale) Mirror 4: 0x9746ce6b (stale) Mirror 5: 0x9746ce6b (stale) CRC-32 checksum value for chunk [0, 0x29177): Mirror 1: 0xd6d6520f Mirror 6: 0xd6d6520f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29177: skipped 000085 1783350881.858779 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x29177): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29177): Mirror 2: 0x9746ce6b (stale) Mirror 3: 0x9746ce6b (stale) Mirror 4: 0x9746ce6b (stale) Mirror 5: 0x9746ce6b (stale) CRC-32 checksum value for chunk [0, 0x29177): Mirror 1: 0xd6d6520f Mirror 6: 0xd6d6520f Mirror 7: 0xd6d6520f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29177: skipped 000086 1783350883.059661 fallocate 0x016caa thru 0x019586 (0x28dd bytes) 000087 1783350883.098736 trunc from 0x029177 to 0x032163 (0x8fed bytes) 000088 1783350883.134517 mapread 0x0278e6 thru 0x029781 (0x1e9c bytes) 000089 1783350883.158114 fallocate 0x013978 thru 0x01b0c4 (0x774d bytes) 000090 1783350883.163885 punch from 0x0193df to 0x02197d (0x859f bytes) 000091 1783350883.168724 fallocate 0x03a9bc thru 0x03ffff (0x5644 bytes) 000094 1783350883.180012 MIRROR_RESYNC 7 mirrors 000095 1783350885.411293 trunc from 0x030b6f to 0x036426 (0x58b8 bytes) 000096 1783350885.447796 trunc from 0x040000 to 0x02d582 (0x12a7d bytes) 000097 1783350885.467143 mapwrite 0x016261 thru 0x0190e3 (0x2e83 bytes) 000098 1783350885.512044 trunc from 0x01226c to 0x014df2 (0x2b87 bytes) 000099 1783350885.516758 trunc from 0x00973c to 0x0158aa (0xc16f bytes) 000100 1783350885.522276 mapread 0x015564 thru 0x022ae6 (0xd583 bytes) 000101 1783350885.538613 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d583): Mirror 2: 0x4be1bd6d (stale) Mirror 3: 0x4be1bd6d (stale) Mirror 5: 0x4be1bd6d (stale) Mirror 6: 0x4be1bd6d (stale) Mirror 7: 0x4be1bd6d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d583): Mirror 2: 0x4be1bd6d (stale) Mirror 3: 0x4be1bd6d (stale) Mirror 5: 0x4be1bd6d (stale) Mirror 6: 0x4be1bd6d (stale) Mirror 7: 0x4be1bd6d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d583: skipped 000102 1783350885.964286 trunc from 0x02d583 to 0x00810f (0x25473 bytes) 000103 1783350885.994733 write 0x0004a3 thru 0x001757 (0x12b5 bytes) 000104 1783350886.010603 read 0x006864 thru 0x00810f (0x18ac bytes) 000105 1783350886.018948 write 0x0277c0 thru 0x02d437 (0x5c78 bytes) 000106 1783350886.021430 trunc from 0x02d438 to 0x0244c5 (0x8f72 bytes) 000107 1783350886.039773 read 0x000216 thru 0x00d348 (0xd133 bytes) 000108 1783350886.050294 trunc from 0x01b126 to 0x01d6ed (0x25c8 bytes) 000109 1783350886.055453 fallocate 0x01947d thru 0x01e544 (0x50c8 bytes) 000112 1783350886.062609 punch from 0x011717 to 0x014d75 (0x365f bytes) 000113 1783350886.067719 mapread 0x01ef84 thru 0x0244c5 (0x5542 bytes) 000114 1783350886.070577 trunc from 0x01a883 to 0x02192a (0x70a8 bytes) 000118 1783350886.076840 punch from 0x02435c to 0x0244c5 (0x16a bytes) 000119 1783350886.085708 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x244c6): Mirror 3: 0xc9465eee (stale) Mirror 5: 0xc9465eee (stale) Mirror 6: 0xc9465eee (stale) Mirror 7: 0xc9465eee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x244c6): Mirror 3: 0xc9465eee (stale) Mirror 5: 0xc9465eee (stale) Mirror 6: 0xc9465eee (stale) Mirror 7: 0xc9465eee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x244c6: skipped 000121 1783350886.441693 MIRROR_RESYNC 5 mirrors 000122 1783350887.256954 read 0x0047d1 thru 0x006686 (0x1eb6 bytes) 000123 1783350887.279963 write 0x03e29c thru 0x03ffff (0x1d64 bytes) 000124 1783350887.318168 fallocate 0x027213 thru 0x02e248 (0x7036 bytes) 000125 1783350887.325301 trunc from 0x040000 to 0x015b1d (0x2a4e2 bytes) 000127 1783350887.368305 trunc from 0x015b1e to 0x01b660 (0x5b43 bytes) 000129 1783350887.391624 read 0x00f258 thru 0x010aed (0x1896 bytes) 000130 1783350887.413261 write 0x012049 thru 0x01c94a (0xa902 bytes) 000131 1783350887.414603 trunc from 0x00302d to 0x00ab6a (0x7b3e bytes) 000132 1783350887.429308 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c94b): Mirror 3: 0xa3dd3f59 (stale) Mirror 5: 0xa3dd3f59 (stale) Mirror 6: 0xa3dd3f59 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c94b): Mirror 3: 0xa3dd3f59 (stale) Mirror 5: 0xa3dd3f59 (stale) Mirror 6: 0xa3dd3f59 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c94b: skipped 000133 1783350887.808901 trunc from 0x01c94b to 0x01f8e7 (0x2f9d bytes) 000134 1783350887.847651 mapread 0x0067e1 thru 0x010273 (0x9a93 bytes) 000135 1783350887.865833 trunc from 0x0177e7 to 0x01d3e7 (0x5c01 bytes) 000136 1783350887.871005 read 0x004886 thru 0x0061c1 (0x193c bytes) 000137 1783350887.871959 fallocate 0x03344c thru 0x038b81 (0x5736 bytes) 000138 1783350887.879379 write 0x00e780 thru 0x019809 (0xb08a bytes) 000139 1783350887.891648 mapwrite 0x0010ea thru 0x00232f (0x1246 bytes) 000140 1783350887.918800 read 0x009911 thru 0x010ec0 (0x75b0 bytes) 000142 1783350887.920653 write 0x03255f thru 0x037c30 (0x56d2 bytes) 000143 1783350887.923926 write 0x0391ac thru 0x03d07a (0x3ecf bytes) 000144 1783350887.924967 punch from 0x034076 to 0x038ea7 (0x4e32 bytes) 000145 1783350887.946128 write 0x002897 thru 0x00f0d4 (0xc83e bytes) 000146 1783350887.956804 punch from 0x0093dc to 0x00a055 (0xc7a bytes) 000147 1783350887.980675 trunc from 0x02f5d4 to 0x0348c2 (0x52ef bytes) 000148 1783350887.995247 fallocate 0x02096f thru 0x02e3ad (0xda3f bytes) 000149 1783350888.006900 write 0x018c5e thru 0x01fbfd (0x6fa0 bytes) 000150 1783350888.041423 mapread 0x023dec thru 0x02ae8e (0x70a3 bytes) 000152 1783350888.066568 fallocate 0x01f9d0 thru 0x02b6fc (0xbd2d bytes) 000153 1783350888.072399 trunc from 0x03d07b to 0x00a229 (0x32e51 bytes) 000154 1783350888.100168 punch from 0x0068d7 to 0x00a229 (0x3953 bytes) 000155 1783350888.120714 read 0x001019 thru 0x00596d (0x4955 bytes) 000157 1783350888.122041 trunc from 0x008ae2 to 0x00a229 (0x1748 bytes) 000159 1783350888.133605 trunc from 0x001581 to 0x00a229 (0x8ca9 bytes) 000160 1783350888.142704 mapread 0x009bf0 thru 0x00a229 (0x63a bytes) 000162 1783350888.167607 fallocate 0x02bebf thru 0x03a516 (0xe658 bytes) 000163 1783350888.180200 write 0x038c08 thru 0x03e869 (0x5c62 bytes) 000164 1783350888.186049 trunc from 0x00d67d to 0x01659a (0x8f1e bytes) 000165 1783350888.196619 mapread 0x00948c thru 0x01593d (0xc4b2 bytes) 000166 1783350888.238194 trunc from 0x03e86a to 0x000016 (0x3e853 bytes) 000168 1783350888.255162 read 0x000005 thru 0x000016 (0x12 bytes) 000169 1783350888.255773 trunc from 0x000017 to 0x03e7b6 (0x3e7a0 bytes) 000170 1783350888.277109 mapread 0x0156ce thru 0x01f788 (0xa0bb bytes) 000171 1783350888.300168 mapwrite 0x018931 thru 0x0259f5 (0xd0c5 bytes) 000172 1783350888.353790 fallocate 0x000497 thru 0x009e40 (0x99aa bytes) 000173 1783350888.377542 MIRROR_RESYNC 4 mirrors 000174 1783350889.269546 mapread 0x034b6e thru 0x03e145 (0x95d8 bytes) 000175 1783350889.295665 mapwrite 0x00579c thru 0x0074b0 (0x1d15 bytes) 000176 1783350889.389691 trunc from 0x03e7b7 to 0x039564 (0x5252 bytes) 000177 1783350889.434994 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39565): Mirror 5: 0x409d732 (stale) Mirror 6: 0x409d732 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39565): Mirror 5: 0x409d732 (stale) Mirror 6: 0x409d732 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39565: skipped 000178 1783350889.783132 mapwrite 0x0315e1 thru 0x033de5 (0x2805 bytes) 000180 1783350889.831206 punch from 0x02bed6 to 0x033d36 (0x7e61 bytes) 000181 1783350889.842372 MIRROR_RESYNC 3 mirrors 000182 1783350890.985973 mapread 0x01a9b2 thru 0x020d1f (0x636e bytes) 000183 1783350891.013256 read 0x0173c5 thru 0x01be46 (0x4a82 bytes) 000184 1783350891.014339 mapwrite 0x038809 thru 0x03ffff (0x77f7 bytes) 000185 1783350891.108877 trunc from 0x039cb3 to 0x03ffff (0x634d bytes) 000186 1783350891.114889 read 0x02082a thru 0x02a884 (0xa05b bytes) 000187 1783350891.125135 read 0x024820 thru 0x0326d4 (0xdeb5 bytes) 000188 1783350891.125875 trunc from 0x040000 to 0x021bd3 (0x1e42c bytes) 000189 1783350891.144192 read 0x00fe5d thru 0x01bc08 (0xbdac bytes) 000190 1783350891.145020 write 0x01c0e0 thru 0x02261d (0x653e bytes) 000191 1783350891.146326 write 0x03ffaf thru 0x03ffff (0x51 bytes) 000192 1783350891.148778 mapread 0x032f4c thru 0x03ecb9 (0xbd6e bytes) 000193 1783350891.157346 trunc from 0x025d01 to 0x02646e (0x76e bytes) 000194 1783350891.160724 read 0x036dc5 thru 0x03eda7 (0x7fe3 bytes) 000195 1783350891.161232 read 0x00dce3 thru 0x013e56 (0x6174 bytes) 000196 1783350891.161779 mapwrite 0x0265e1 thru 0x02d944 (0x7364 bytes) 000197 1783350891.189545 trunc from 0x040000 to 0x0229fa (0x1d605 bytes) 000198 1783350891.217713 MIRROR_RESYNC 3 mirrors 000199 1783350891.939641 punch from 0x01bc6f to 0x0229fa (0x6d8c bytes) 000201 1783350891.971528 mapwrite 0x01e845 thru 0x023cfc (0x54b8 bytes) 000202 1783350892.102524 mapwrite 0x00c495 thru 0x014ffc (0x8b68 bytes) 000203 1783350892.182980 punch from 0x011183 to 0x011d71 (0xbef bytes) 000204 1783350892.196907 trunc from 0x023cfd to 0x02d822 (0x9b26 bytes) 000205 1783350892.228635 read 0x01fbec thru 0x0254ad (0x58c2 bytes) 000206 1783350892.250951 read 0x026c97 thru 0x0294e1 (0x284b bytes) 000207 1783350892.283280 punch from 0x0275f7 to 0x02d822 (0x622c bytes) 000208 1783350892.293295 punch from 0x00419d to 0x0055f9 (0x145d bytes) 000209 1783350892.305525 write 0x01c83c thru 0x0288f9 (0xc0be bytes) 000210 1783350892.315096 trunc from 0x014488 to 0x01dcb0 (0x9829 bytes) 000211 1783350892.341278 trunc from 0x02d823 to 0x01e7ac (0xf076 bytes) 000212 1783350892.366932 punch from 0x012c00 to 0x01e7ac (0xbbad bytes) 000213 1783350892.373714 mapread 0x00e48d thru 0x01035f (0x1ed3 bytes) 000214 1783350892.374761 read 0x00c67a thru 0x016be7 (0xa56e bytes) 000215 1783350892.396734 mapread 0x015c16 thru 0x01e7ac (0x8b97 bytes) 000216 1783350892.398096 trunc from 0x013c4d to 0x01e7ac (0xab60 bytes) 000217 1783350892.404804 read 0x000216 thru 0x00c724 (0xc50f bytes) 000218 1783350892.412946 MIRROR_RESYNC 3 mirrors 000219 1783350893.258318 write 0x03a9f3 thru 0x03ffff (0x560d bytes) 000220 1783350893.293193 write 0x028e38 thru 0x0386f2 (0xf8bb bytes) 000221 1783350893.304008 punch from 0x03aea3 to 0x03fa72 (0x4bd0 bytes) 000222 1783350893.324764 trunc from 0x040000 to 0x012549 (0x2dab6 bytes) 000223 1783350893.372883 punch from 0x006a14 to 0x00a32c (0x3919 bytes) 000224 1783350893.379505 read 0x00f78c thru 0x012549 (0x2dbe bytes) 000225 1783350893.387079 trunc from 0x00766f to 0x00f047 (0x79d9 bytes) 000226 1783350893.394637 mapwrite 0x0224f5 thru 0x031719 (0xf225 bytes) 000227 1783350893.466698 MIRROR_RESYNC 3 mirrors 000228 1783350894.632370 mapwrite 0x02907b thru 0x02f502 (0x6488 bytes) 000229 1783350894.744344 write 0x00241e thru 0x003593 (0x1176 bytes) 000230 1783350894.754664 mapread 0x010b85 thru 0x018c20 (0x809c bytes) 000231 1783350894.812981 MIRROR_RESYNC 3 mirrors 000232 1783350895.811568 write 0x01d7bf thru 0x02af93 (0xd7d5 bytes) 000233 1783350895.854034 write 0x0143ce thru 0x023f78 (0xfbab bytes) 000234 1783350895.861648 write 0x038faf thru 0x03ffff (0x7051 bytes) 000235 1783350895.865390 trunc from 0x00fa15 to 0x013431 (0x3a1d bytes) 000236 1783350895.871941 mapread 0x01069d thru 0x019427 (0x8d8b bytes) 000237 1783350895.892180 trunc from 0x040000 to 0x0282f2 (0x17d0d bytes) 000238 1783350895.919035 punch from 0x01df1d to 0x0282f2 (0xa3d6 bytes) 000239 1783350895.949367 write 0x028586 thru 0x03705f (0xeada bytes) 000240 1783350895.960803 punch from 0x00ece4 to 0x017696 (0x89b3 bytes) 000241 1783350895.977242 punch from 0x02431a to 0x026364 (0x204b bytes) 000242 1783350895.986041 trunc from 0x037060 to 0x0187f8 (0x1e867 bytes) 000243 1783350896.015359 mapwrite 0x006d36 thru 0x00bb80 (0x4e4b bytes) 000244 1783350896.058572 punch from 0x006963 to 0x0099b0 (0x304e bytes) 000245 1783350896.064637 mapread 0x00bf72 thru 0x0138a1 (0x7930 bytes) 000246 1783350896.083690 punch from 0x007bd5 to 0x00be1b (0x4247 bytes) 000247 1783350896.093280 fallocate 0x00562c thru 0x010e86 (0xb85b bytes) 000248 1783350896.104558 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x187f9): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x187f9): Mirror 5: 0xb768afb7 (stale) Mirror 6: 0xb768afb7 (stale) CRC-32 checksum value for chunk [0, 0x187f9): Mirror 1: 0x7ebc23dd Mirror 7: 0x7ebc23dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x187f9: skipped 000249 1783350896.974981 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x187f9): Mirror 6: 0xb768afb7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x187f9): Mirror 6: 0xb768afb7 (stale) CRC-32 checksum value for chunk [0, 0x187f9): Mirror 1: 0x7ebc23dd Mirror 7: 0x7ebc23dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x187f9: skipped 000250 1783350897.561409 fallocate 0x011c7e thru 0x01b5a8 (0x992b bytes) 000251 1783350897.652770 read 0x00804a thru 0x017710 (0xf6c7 bytes) 000252 1783350897.668705 trunc from 0x003bc1 to 0x013174 (0xf5b4 bytes) 000253 1783350897.678892 fallocate 0x037132 thru 0x03ffff (0x8ece bytes) 000254 1783350897.690440 trunc from 0x0097a8 to 0x016d58 (0xd5b1 bytes) 000256 1783350897.697383 mapread 0x007c50 thru 0x00901e (0x13cf bytes) 000258 1783350897.721586 trunc from 0x01b5a9 to 0x0068c6 (0x14ce2 bytes) 000259 1783350897.775043 read 0x000ea4 thru 0x0046d1 (0x382e bytes) 000260 1783350897.782934 write 0x01a15c thru 0x0254cb (0xb370 bytes) 000261 1783350897.785512 fallocate 0x0194fb thru 0x029444 (0xff4a bytes) 000262 1783350897.790743 mapwrite 0x02b8a7 thru 0x0376f5 (0xbe4f bytes) 000263 1783350897.871314 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x376f6): Mirror 7: 0xcd1d01b7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x376f6): Mirror 7: 0xcd1d01b7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x376f6: skipped 000264 1783350898.151545 trunc from 0x0075d3 to 0x017058 (0xfa86 bytes) 000265 1783350898.156365 read 0x0036c5 thru 0x0113e3 (0xdd1f bytes) 000266 1783350898.177226 punch from 0x0023e5 to 0x01062b (0xe247 bytes) 000267 1783350898.191687 read 0x02f53d thru 0x031c03 (0x26c7 bytes) 000269 1783350898.217747 trunc from 0x021be3 to 0x027ee0 (0x62fe bytes) 000270 1783350898.229075 trunc from 0x00efe0 to 0x0126c2 (0x36e3 bytes) 000271 1783350898.236922 trunc from 0x0376f6 to 0x03c090 (0x499b bytes) 000272 1783350898.275174 punch from 0x0216a5 to 0x02b156 (0x9ab2 bytes) 000273 1783350898.282037 read 0x00000e thru 0x0006a3 (0x696 bytes) 000274 1783350898.283196 fallocate 0x00c993 thru 0x01a093 (0xd701 bytes) 000275 1783350898.288167 trunc from 0x0016df to 0x00799b (0x62bd bytes) 000276 1783350898.295741 mapread 0x02dccb thru 0x0378fd (0x9c33 bytes) 000277 1783350898.303239 MIRROR_RESYNC 2 mirrors 000278 1783350899.059154 write 0x002277 thru 0x00f3f7 (0xd181 bytes) 000279 1783350899.106751 mapread 0x013628 thru 0x01bd09 (0x86e2 bytes) 000281 1783350899.127777 mapwrite 0x039b32 thru 0x03ffff (0x64ce bytes) 000283 1783350899.202980 trunc from 0x025b6a to 0x0307b6 (0xac4d bytes) 000284 1783350899.209466 fallocate 0x03c4c9 thru 0x03cab0 (0x5e8 bytes) 000285 1783350899.213676 trunc from 0x033b73 to 0x039d76 (0x6204 bytes) 000286 1783350899.219434 mapwrite 0x00113f thru 0x010d73 (0xfc35 bytes) 000287 1783350899.255246 mapread 0x0304ff thru 0x033f18 (0x3a1a bytes) 000288 1783350899.268582 fallocate 0x00d5e6 thru 0x017727 (0xa142 bytes) 000289 1783350899.274508 write 0x000185 thru 0x00c7c8 (0xc644 bytes) 000290 1783350899.286733 trunc from 0x040000 to 0x02869a (0x17965 bytes) 000292 1783350899.308158 write 0x0139fa thru 0x01dc1d (0xa224 bytes) 000293 1783350899.315053 mapread 0x004828 thru 0x00a485 (0x5c5e bytes) 000294 1783350899.323617 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2869b): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2869b): Mirror 7: 0xd70e504d (stale) CRC-32 checksum value for chunk [0, 0x2869b): Mirror 1: 0x26dcef75 Mirror 8: 0x26dcef75 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2869b: skipped 000295 1783350900.060999 mapread 0x00bd6e thru 0x01434e (0x85e1 bytes) 000296 1783350900.080692 trunc from 0x019110 to 0x027a90 (0xe981 bytes) 000297 1783350900.124380 write 0x029892 thru 0x02c2ae (0x2a1d bytes) 000298 1783350900.140491 mapread 0x025db6 thru 0x02c2ae (0x64f9 bytes) 000299 1783350900.167620 trunc from 0x00808a to 0x00b724 (0x369b bytes) 000300 1783350900.173532 trunc from 0x02c2af to 0x03d3eb (0x1113d bytes) 000301 1783350900.237292 punch from 0x01a866 to 0x01b71d (0xeb8 bytes) 000302 1783350900.248862 write 0x021924 thru 0x029437 (0x7b14 bytes) 000304 1783350900.253384 trunc from 0x0196a3 to 0x021f4a (0x88a8 bytes) 000305 1783350900.282295 punch from 0x01b684 to 0x020ebf (0x583c bytes) 000306 1783350900.288196 trunc from 0x03d3ec to 0x0236b3 (0x19d38 bytes) 000307 1783350900.309575 trunc from 0x0236b4 to 0x01ecab (0x4a08 bytes) 000308 1783350900.339789 trunc from 0x00a9be to 0x0152b5 (0xa8f8 bytes) 000309 1783350900.349550 fallocate 0x0297cd thru 0x02c0d9 (0x290d bytes) 000310 1783350900.356379 read 0x00426e thru 0x00850e (0x42a1 bytes) 000311 1783350900.378066 read 0x000434 thru 0x008048 (0x7c15 bytes) 000312 1783350900.398233 fallocate 0x0355f3 thru 0x03ffff (0xaa0d bytes) 000313 1783350900.407091 fallocate 0x015645 thru 0x024871 (0xf22d bytes) 000314 1783350900.416046 trunc from 0x02c0da to 0x0005f1 (0x2bae8 bytes) 000315 1783350900.443526 punch from 0x00020d to 0x0005f1 (0x3e5 bytes) 000316 1783350900.450086 punch from 0x00011d to 0x0005f1 (0x4d5 bytes) 000317 1783350900.459676 trunc from 0x0005f2 to 0x01199b (0x113aa bytes) 000318 1783350900.476584 punch from 0x001bd1 to 0x008404 (0x6834 bytes) 000319 1783350900.485004 write 0x03f3c0 thru 0x03ffff (0xc40 bytes) 000320 1783350900.494369 MIRROR_RESYNC 3 mirrors 000321 1783350901.180633 read 0x01d2dc thru 0x020872 (0x3597 bytes) 000322 1783350901.190109 trunc from 0x040000 to 0x022993 (0x1d66c bytes) 000324 1783350901.316142 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22994): Mirror 8: 0x37df1e6c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22994): Mirror 8: 0x37df1e6c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22994: skipped 000325 1783350901.598706 mapwrite 0x008aa6 thru 0x00cfac (0x4507 bytes) 000326 1783350901.630216 mapwrite 0x00bcad thru 0x00f974 (0x3cc8 bytes) 000327 1783350901.659998 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) 000328 1783350901.848740 trunc from 0x022994 to 0x0045e4 (0x1e3af bytes) 000329 1783350901.871740 trunc from 0x00080c to 0x0045e4 (0x3dd9 bytes) 000330 1783350901.878049 write 0x03357a thru 0x037210 (0x3c97 bytes) 000331 1783350901.879739 fallocate 0x024cf2 thru 0x02e72c (0x9a3b bytes) 000332 1783350901.891895 trunc from 0x00a438 to 0x011af3 (0x76bc bytes) 000333 1783350901.896481 trunc from 0x037211 to 0x02f5b8 (0x7c58 bytes) 000334 1783350901.917888 fallocate 0x03616b thru 0x03ffff (0x9e95 bytes) 000335 1783350901.923182 mapread 0x02a853 thru 0x02fa5b (0x5209 bytes) 000336 1783350901.972254 punch from 0x01f4d2 to 0x029f93 (0xaac2 bytes) 000337 1783350901.981125 punch from 0x01c550 to 0x0210ec (0x4b9d bytes) 000338 1783350901.996794 mapread 0x02a846 thru 0x02d277 (0x2a32 bytes) 000340 1783350902.003709 fallocate 0x02bd6e thru 0x03047e (0x4711 bytes) 000341 1783350902.010789 trunc from 0x03f0fe to 0x03ffff (0xf02 bytes) 000343 1783350902.023698 MIRROR_EXTEND to 2 mirrors 000344 1783350902.764076 trunc from 0x0290e7 to 0x0316f1 (0x860b bytes) 000345 1783350902.817749 mapwrite 0x023338 thru 0x032d75 (0xfa3e bytes) 000346 1783350902.868723 trunc from 0x040000 to 0x0227bb (0x1d844 bytes) 000347 1783350902.895128 read 0x00e4bb thru 0x011931 (0x3477 bytes) 000348 1783350902.903119 fallocate 0x01c372 thru 0x027966 (0xb5f5 bytes) 000349 1783350902.920909 read 0x02562f thru 0x027966 (0x2338 bytes) 000350 1783350902.930090 read 0x003f60 thru 0x013ae5 (0xfb86 bytes) 000351 1783350902.934624 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x27967): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27967): Mirror 2: 0x2fb831c4 (stale) CRC-32 checksum value for chunk [0, 0x27967): Mirror 1: 0x2fb831c4 Mirror 3: 0x2fb831c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27967: skipped 000352 1783350903.747385 punch from 0x025a38 to 0x027966 (0x1f2f bytes) 000353 1783350903.798984 read 0x000f77 thru 0x00b877 (0xa901 bytes) 000354 1783350903.820110 mapwrite 0x02d0c4 thru 0x03b02a (0xdf67 bytes) 000355 1783350903.899283 fallocate 0x00a80e thru 0x018978 (0xe16b bytes) 000356 1783350903.904779 mapread 0x003ba5 thru 0x00a4b6 (0x6912 bytes) 000357 1783350903.907182 mapread 0x030f0a thru 0x038390 (0x7487 bytes) 000358 1783350903.908929 fallocate 0x00ddf9 thru 0x00e7b5 (0x9bd bytes) 000359 1783350903.917458 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b02b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b02b): Mirror 2: 0x2afad0c9 (stale) Mirror 3: 0x2afad0c9 (stale) CRC-32 checksum value for chunk [0, 0x3b02b): Mirror 1: 0x4b6e3b2e Mirror 4: 0x4b6e3b2e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b02b: skipped 000360 1783350904.743468 fallocate 0x0084e8 thru 0x00b96e (0x3487 bytes) 000361 1783350904.811141 punch from 0x014790 to 0x015413 (0xc84 bytes) 000362 1783350904.820322 mapread 0x025a8e thru 0x0262fd (0x870 bytes) 000363 1783350904.834529 trunc from 0x029416 to 0x036c6c (0xd857 bytes) 000364 1783350904.849000 mapwrite 0x014cdb thru 0x020ff8 (0xc31e bytes) 000366 1783350904.964438 punch from 0x0322c2 to 0x03b02a (0x8d69 bytes) 000367 1783350904.972312 punch from 0x02e8d8 to 0x034b2e (0x6257 bytes) 000368 1783350904.982737 mapread 0x00e8c6 thru 0x018f69 (0xa6a4 bytes) 000369 1783350904.988800 trunc from 0x03b02b to 0x032ab5 (0x8575 bytes) 000370 1783350905.026149 trunc from 0x01dbef to 0x021cca (0x40dc bytes) 000371 1783350905.032311 fallocate 0x004b31 thru 0x014078 (0xf548 bytes) 000372 1783350905.047204 punch from 0x00be98 to 0x00e53c (0x26a5 bytes) 000373 1783350905.059119 mapwrite 0x0304dc thru 0x038986 (0x84ab bytes) 000376 1783350905.162756 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38987): Mirror 3: 0x823f94d0 (stale) Mirror 4: 0xe6133565 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38987): Mirror 3: 0x823f94d0 (stale) Mirror 4: 0xe6133565 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38987: skipped 000377 1783350905.635083 mapread 0x00ad6a thru 0x01305a (0x82f1 bytes) 000378 1783350905.661681 trunc from 0x0237f5 to 0x032aed (0xf2f9 bytes) 000380 1783350905.669129 write 0x00dc3e thru 0x016a83 (0x8e46 bytes) 000381 1783350905.685238 trunc from 0x038987 to 0x005a13 (0x32f73 bytes) 000382 1783350905.710614 trunc from 0x005a14 to 0x013df4 (0xe3e1 bytes) 000384 1783350905.736598 read 0x00c845 thru 0x012699 (0x5e55 bytes) 000385 1783350905.746572 fallocate 0x0025a7 thru 0x00aeba (0x8914 bytes) 000386 1783350905.753516 punch from 0x0137b2 to 0x013df4 (0x643 bytes) 000387 1783350905.761553 punch from 0x00f8af to 0x013492 (0x3be4 bytes) 000388 1783350905.767295 trunc from 0x013df5 to 0x024e13 (0x1101f bytes) 000389 1783350905.785708 fallocate 0x01cdd4 thru 0x0267b3 (0x99e0 bytes) 000390 1783350905.794454 mapread 0x014b3f thru 0x018de3 (0x42a5 bytes) 000391 1783350905.806711 mapwrite 0x01fe2a thru 0x02dc5b (0xde32 bytes) 000392 1783350905.890512 mapwrite 0x02836b thru 0x02ff14 (0x7baa bytes) 000393 1783350905.945245 fallocate 0x0312cd thru 0x033f89 (0x2cbd bytes) 000394 1783350905.955225 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33f8a): Mirror 4: 0x53922d7b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33f8a): Mirror 4: 0x53922d7b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33f8a: skipped 000395 1783350906.213147 mapwrite 0x01ac13 thru 0x02031c (0x570a bytes) 000396 1783350906.276648 trunc from 0x033f8a to 0x021cbe (0x122cb bytes) 000397 1783350906.326985 trunc from 0x006a11 to 0x010c36 (0xa226 bytes) 000398 1783350906.332744 punch from 0x00e10a to 0x0128aa (0x47a1 bytes) 000399 1783350906.338264 mapwrite 0x00c429 thru 0x0170dd (0xacb5 bytes) 000401 1783350906.377039 write 0x0337f1 thru 0x03fd91 (0xc5a1 bytes) 000402 1783350906.383674 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fd92): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd92): Mirror 4: 0xd9f62abb (stale) CRC-32 checksum value for chunk [0, 0x3fd92): Mirror 1: 0x2c0f6e73 Mirror 5: 0x2c0f6e73 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd92: skipped 000403 1783350907.320961 write 0x010f01 thru 0x01a638 (0x9738 bytes) 000404 1783350907.385175 mapwrite 0x0280a1 thru 0x031c9a (0x9bfa bytes) 000405 1783350907.429571 write 0x00850e thru 0x01740a (0xeefd bytes) 000406 1783350907.444064 MIRROR_RESYNC 3 mirrors 000407 1783350908.493589 read 0x018dfa thru 0x01cefb (0x4102 bytes) 000408 1783350908.521805 read 0x00c24f thru 0x0107bc (0x456e bytes) 000409 1783350908.548417 trunc from 0x03fd92 to 0x006957 (0x3943a bytes) 000410 1783350908.644254 trunc from 0x002898 to 0x0063ea (0x3b53 bytes) 000411 1783350908.653189 trunc from 0x006958 to 0x006a7c (0x125 bytes) 000412 1783350908.674855 write 0x0234a7 thru 0x028fd1 (0x5b2b bytes) 000413 1783350908.677521 punch from 0x01f08d to 0x028fd1 (0x9f45 bytes) 000414 1783350908.704147 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x28fd2): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28fd2): Mirror 4: 0x494813a1 (stale) Mirror 5: 0x494813a1 (stale) CRC-32 checksum value for chunk [0, 0x28fd2): Mirror 1: 0xc30dda21 Mirror 6: 0xc30dda21 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28fd2: skipped 000415 1783350909.552751 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28fd2): Mirror 4: 0x494813a1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28fd2): Mirror 4: 0x494813a1 (stale) CRC-32 checksum value for chunk [0, 0x28fd2): Mirror 1: 0xc30dda21 Mirror 6: 0xc30dda21 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28fd2: skipped 000416 1783350910.111662 read 0x009476 thru 0x0134e8 (0xa073 bytes) 000417 1783350910.142592 mapwrite 0x01c134 thru 0x02462e (0x84fb bytes) 000418 1783350910.249862 read 0x014e92 thru 0x016762 (0x18d1 bytes) 000419 1783350910.268094 fallocate 0x0167c8 thru 0x018a50 (0x2289 bytes) 000421 1783350910.276469 write 0x03e9e5 thru 0x03ffff (0x161b bytes) 000422 1783350910.282729 write 0x028c95 thru 0x02913d (0x4a9 bytes) 000423 1783350910.291333 punch from 0x02bc51 to 0x02d9c0 (0x1d70 bytes) 000424 1783350910.296585 trunc from 0x040000 to 0x030d83 (0xf27c bytes) 000425 1783350910.316736 trunc from 0x030d84 to 0x037884 (0x6b01 bytes) 000426 1783350910.343064 trunc from 0x031fd5 to 0x037884 (0x58b0 bytes) 000427 1783350910.351135 read 0x0368bd thru 0x037884 (0xfc8 bytes) 000428 1783350910.359405 read 0x02b44d thru 0x030416 (0x4fca bytes) 000429 1783350910.364628 MIRROR_RESYNC 3 mirrors 000430 1783350911.363387 MIRROR_EXTEND to 4 mirrors 000431 1783350912.180547 mapread 0x00db0e thru 0x00f335 (0x1828 bytes) 000432 1783350912.204517 mapread 0x0118d0 thru 0x01681c (0x4f4d bytes) 000433 1783350912.205625 trunc from 0x037885 to 0x037e12 (0x58e bytes) 000434 1783350912.269537 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37e13): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37e13): Mirror 4: 0x723947c8 (stale) Mirror 6: 0x723947c8 (stale) Mirror 7: 0x723947c8 (stale) CRC-32 checksum value for chunk [0, 0x37e13): Mirror 1: 0x723947c8 Mirror 8: 0x723947c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37e13: skipped 000435 1783350913.374515 read 0x018daf thru 0x025e1e (0xd070 bytes) 000436 1783350913.392476 mapread 0x00bf82 thru 0x0129cf (0x6a4e bytes) 000437 1783350913.425450 read 0x022863 thru 0x02db0d (0xb2ab bytes) 000438 1783350913.443469 fallocate 0x03db68 thru 0x03ffff (0x2498 bytes) 000439 1783350913.485444 trunc from 0x0333dd to 0x03dcbc (0xa8e0 bytes) 000440 1783350913.497385 write 0x03a462 thru 0x03ffff (0x5b9e bytes) 000441 1783350913.541007 write 0x00345a thru 0x006f24 (0x3acb bytes) 000442 1783350913.554601 trunc from 0x040000 to 0x00dc20 (0x323df bytes) 000443 1783350913.601202 mapwrite 0x003288 thru 0x005c9b (0x2a14 bytes) 000444 1783350913.618747 trunc from 0x007120 to 0x00ad4d (0x3c2e bytes) 000445 1783350913.624584 write 0x03a463 thru 0x03d7fe (0x339c bytes) 000446 1783350913.638409 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d7ff): Mirror 6: 0x673c6244 (stale) Mirror 7: 0x673c6244 (stale) Mirror 8: 0x673c6244 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d7ff): Mirror 6: 0x673c6244 (stale) Mirror 7: 0x673c6244 (stale) Mirror 8: 0x673c6244 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d7ff: skipped 000447 1783350914.100719 trunc from 0x00f15b to 0x011a25 (0x28cb bytes) 000448 1783350914.108579 trunc from 0x03d7ff to 0x00491d (0x38ee1 bytes) 000449 1783350914.129530 write 0x01225b thru 0x0176a5 (0x544b bytes) 000450 1783350914.131379 read 0x01006b thru 0x0176a5 (0x763b bytes) 000451 1783350914.137960 punch from 0x002707 to 0x00b4b9 (0x8db3 bytes) 000453 1783350914.144714 trunc from 0x0176a6 to 0x039e7d (0x227d8 bytes) 000454 1783350914.163859 read 0x0023a1 thru 0x006b3f (0x479f bytes) 000455 1783350914.177584 fallocate 0x01e05f thru 0x022ce6 (0x4c88 bytes) 000456 1783350914.184562 read 0x01c1fd thru 0x028c9d (0xcaa1 bytes) 000457 1783350914.189732 fallocate 0x007574 thru 0x00b50e (0x3f9b bytes) 000458 1783350914.197184 trunc from 0x024f06 to 0x0305f4 (0xb6ef bytes) 000459 1783350914.203603 trunc from 0x039e7e to 0x00a31a (0x2fb63 bytes) 000460 1783350914.225418 trunc from 0x006395 to 0x00a31a (0x3f86 bytes) 000461 1783350914.234567 trunc from 0x00a31b to 0x012d43 (0x8a29 bytes) 000462 1783350914.264738 mapwrite 0x03a2f3 thru 0x03aa51 (0x75f bytes) 000463 1783350914.316056 read 0x0044c9 thru 0x012e2f (0xe967 bytes) 000464 1783350914.320389 write 0x004c88 thru 0x00d26f (0x85e8 bytes) 000465 1783350914.322132 trunc from 0x03aa52 to 0x001f03 (0x38b4e bytes) 000467 1783350914.355108 fallocate 0x02b868 thru 0x034ecb (0x9664 bytes) 000468 1783350914.365224 write 0x02364a thru 0x0301d3 (0xcb8a bytes) 000469 1783350914.367138 write 0x021830 thru 0x02a394 (0x8b65 bytes) 000470 1783350914.389509 trunc from 0x0301d4 to 0x019836 (0x1699d bytes) 000472 1783350914.432342 write 0x00a593 thru 0x015c74 (0xb6e2 bytes) 000473 1783350914.444053 mapwrite 0x016354 thru 0x024b57 (0xe804 bytes) 000474 1783350914.631107 mapread 0x01633e thru 0x024b57 (0xe81a bytes) 000475 1783350914.635515 mapwrite 0x02b806 thru 0x02b808 (0x3 bytes) 000476 1783350914.699613 mapread 0x024273 thru 0x02b808 (0x7596 bytes) 000477 1783350914.779687 mapwrite 0x00a9ee thru 0x00c9b4 (0x1fc7 bytes) 000478 1783350914.795196 fallocate 0x00376d thru 0x006462 (0x2cf6 bytes) 000479 1783350914.801467 write 0x00e443 thru 0x018972 (0xa530 bytes) 000480 1783350914.804722 mapread 0x019169 thru 0x0283d4 (0xf26c bytes) 000481 1783350914.807168 punch from 0x028387 to 0x02b808 (0x3482 bytes) 000482 1783350914.814322 trunc from 0x026e35 to 0x02b808 (0x49d4 bytes) 000483 1783350914.820958 read 0x00bd1c thru 0x019a4e (0xdd33 bytes) 000484 1783350914.821625 mapwrite 0x023c02 thru 0x023eb4 (0x2b3 bytes) 000485 1783350914.853172 mapwrite 0x03acc1 thru 0x03ce08 (0x2148 bytes) 000486 1783350914.902858 punch from 0x003559 to 0x005133 (0x1bdb bytes) 000487 1783350914.909606 fallocate 0x01bc32 thru 0x01d574 (0x1943 bytes) 000488 1783350914.921422 punch from 0x00d021 to 0x0178ae (0xa88e bytes) 000489 1783350914.950483 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ce09): Mirror 7: 0xbf29e32b (stale) Mirror 8: 0xbf29e32b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce09): Mirror 7: 0xbf29e32b (stale) Mirror 8: 0xbf29e32b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce09: skipped 000490 1783350915.320121 trunc from 0x021415 to 0x028335 (0x6f21 bytes) 000491 1783350915.327697 mapwrite 0x011792 thru 0x01fa80 (0xe2ef bytes) 000492 1783350915.386025 punch from 0x0369d7 to 0x03ce08 (0x6432 bytes) 000493 1783350915.395253 trunc from 0x03ce09 to 0x012692 (0x2a776 bytes) 000494 1783350915.436443 trunc from 0x004fd6 to 0x0116ff (0xc72a bytes) 000495 1783350915.447284 read 0x001ff0 thru 0x00b9b4 (0x99c5 bytes) 000496 1783350915.467233 mapwrite 0x00d708 thru 0x00db4f (0x448 bytes) 000497 1783350915.496332 trunc from 0x012693 to 0x012f6f (0x8dd bytes) 000498 1783350915.535399 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12f70): Mirror 8: 0x9d48a0be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12f70): Mirror 8: 0x9d48a0be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12f70: skipped 000499 1783350915.789750 read 0x006ce6 thru 0x012f6f (0xc28a bytes) 000501 1783350915.799478 mapwrite 0x0384cd thru 0x03f940 (0x7474 bytes) 000503 1783350915.881539 punch from 0x003cec to 0x007515 (0x382a bytes) 000504 1783350915.887127 read 0x00a7e5 thru 0x00fc0a (0x5426 bytes) 000505 1783350915.891582 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) 000506 1783350916.048570 read 0x018417 thru 0x0187f3 (0x3dd bytes) 000508 1783350916.063359 trunc from 0x015870 to 0x01c6ea (0x6e7b bytes) 000509 1783350916.071899 punch from 0x008d16 to 0x017e63 (0xf14e bytes) 000510 1783350916.076457 read 0x002cea thru 0x00e5b9 (0xb8d0 bytes) 000511 1783350916.085731 mapread 0x008684 thru 0x00f3c5 (0x6d42 bytes) 000512 1783350916.086747 write 0x00609c thru 0x014ac9 (0xea2e bytes) 000513 1783350916.097094 trunc from 0x03f941 to 0x00b2e5 (0x3465b bytes) 000514 1783350916.126567 trunc from 0x00b2e6 to 0x02f8df (0x245fa bytes) 000515 1783350916.153279 write 0x039763 thru 0x03ffff (0x689d bytes) 000516 1783350916.155107 write 0x03c6c1 thru 0x03ffff (0x393f bytes) 000517 1783350916.156210 read 0x016a1c thru 0x01bd73 (0x5358 bytes) 000518 1783350916.165619 fallocate 0x01f713 thru 0x02dafc (0xe3ea bytes) 000519 1783350916.175693 read 0x01040a thru 0x01a4fb (0xa0f2 bytes) 000520 1783350916.183856 punch from 0x031bad to 0x03b502 (0x9956 bytes) 000521 1783350916.204437 punch from 0x013b73 to 0x01ed28 (0xb1b6 bytes) 000522 1783350916.209521 trunc from 0x040000 to 0x0266c2 (0x1993d bytes) 000523 1783350916.230424 MIRROR_EXTEND to 2 mirrors 000524 1783350916.886625 write 0x026659 thru 0x02a2ba (0x3c62 bytes) 000525 1783350916.944543 MIRROR_RESYNC 2 mirrors 000526 1783350917.542713 punch from 0x011668 to 0x01e9fc (0xd395 bytes) 000527 1783350917.577846 trunc from 0x02a2bb to 0x024d5c (0x555e bytes) 000528 1783350917.599965 mapread 0x003e13 thru 0x012b3d (0xed2b bytes) 000529 1783350917.646804 trunc from 0x001756 to 0x0116f0 (0xff9b bytes) 000530 1783350917.658413 read 0x0128c6 thru 0x014f48 (0x2683 bytes) 000531 1783350917.659328 punch from 0x01acd0 to 0x024d5c (0xa08d bytes) 000532 1783350917.673501 write 0x011624 thru 0x01a097 (0x8a74 bytes) 000533 1783350917.692832 mapwrite 0x02292d thru 0x023234 (0x908 bytes) 000534 1783350917.747665 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x24d5d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24d5d): Mirror 2: 0xd53d68c6 (stale) CRC-32 checksum value for chunk [0, 0x24d5d): Mirror 1: 0xa11cbaf7 Mirror 3: 0xa11cbaf7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24d5d: skipped 000535 1783350918.487389 trunc from 0x024d5d to 0x00001f (0x24d3d bytes) 000536 1783350918.533695 trunc from 0x000020 to 0x00bbee (0xbbcf bytes) 000537 1783350918.547714 trunc from 0x00854d to 0x00bbee (0x36a2 bytes) 000538 1783350918.551051 trunc from 0x000aa6 to 0x00b418 (0xa973 bytes) 000539 1783350918.560376 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbbef): Mirror 3: 0xa26a4efa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbbef): Mirror 3: 0xa26a4efa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbbef: skipped 000540 1783350918.877612 fallocate 0x02da4a thru 0x03af93 (0xd54a bytes) 000541 1783350918.891312 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3af94): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3af94): Mirror 3: 0xdae63106 (stale) CRC-32 checksum value for chunk [0, 0x3af94): Mirror 1: 0x612f1778 Mirror 4: 0x612f1778 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3af94: skipped 000542 1783350919.725068 mapread 0x00f08c thru 0x0171f0 (0x8165 bytes) 000543 1783350919.743708 mapwrite 0x03ef4f thru 0x03ffff (0x10b1 bytes) 000544 1783350919.830596 trunc from 0x040000 to 0x026275 (0x19d8a bytes) 000545 1783350919.888000 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26276): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26276): Mirror 3: 0xc3c36957 (stale) Mirror 4: 0xd06ed8d5 (stale) CRC-32 checksum value for chunk [0, 0x26276): Mirror 1: 0xd06ed8d5 Mirror 5: 0xd06ed8d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26276: skipped 000546 1783350920.791407 mapread 0x01ebb9 thru 0x026275 (0x76bd bytes) 000547 1783350920.853322 mapwrite 0x0105ec thru 0x01aa96 (0xa4ab bytes) 000548 1783350921.044591 trunc from 0x008d51 to 0x016b97 (0xde47 bytes) 000549 1783350921.057056 mapwrite 0x02c1ab thru 0x0327f6 (0x664c bytes) 000550 1783350921.202656 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x327f7): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x327f7): Mirror 3: 0xd995d44b (stale) Mirror 4: 0xda1864df (stale) Mirror 5: 0xda1864df (stale) CRC-32 checksum value for chunk [0, 0x327f7): Mirror 1: 0xa9cc4655 Mirror 6: 0xa9cc4655 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x327f7: skipped 000551 1783350921.967578 write 0x03ca2d thru 0x03ffff (0x35d3 bytes) 000552 1783350922.004697 trunc from 0x040000 to 0x0390eb (0x6f14 bytes) 000553 1783350922.061133 mapread 0x034542 thru 0x0390eb (0x4baa bytes) 000554 1783350922.093476 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x390ec): Mirror 4: 0xc73caa79 (stale) Mirror 5: 0xc73caa79 (stale) Mirror 6: 0xb2490bc2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x390ec): Mirror 4: 0xc73caa79 (stale) Mirror 5: 0xc73caa79 (stale) Mirror 6: 0xb2490bc2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x390ec: skipped 000555 1783350922.472504 mapread 0x01f4ea thru 0x02ad89 (0xb8a0 bytes) 000556 1783350922.514013 trunc from 0x00518e to 0x012ac1 (0xd934 bytes) 000557 1783350922.518985 trunc from 0x0390ec to 0x004432 (0x34cb9 bytes) 000558 1783350922.550916 MIRROR_RESYNC 4 mirrors 000559 1783350923.298270 mapwrite 0x02b78a thru 0x03a5b1 (0xee28 bytes) 000560 1783350923.452335 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a5b2): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a5b2): Mirror 4: 0x4fc61554 (stale) Mirror 5: 0x4fc61554 (stale) Mirror 6: 0x4fc61554 (stale) CRC-32 checksum value for chunk [0, 0x3a5b2): Mirror 1: 0x760613a2 Mirror 7: 0x760613a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a5b2: skipped 000561 1783350924.188263 trunc from 0x0038eb to 0x005eaa (0x25c0 bytes) 000562 1783350924.231860 read 0x0178d7 thru 0x019bad (0x22d7 bytes) 000563 1783350924.239394 punch from 0x0069b9 to 0x0125a7 (0xbbef bytes) 000564 1783350924.245645 write 0x00a7a3 thru 0x016366 (0xbbc4 bytes) 000565 1783350924.283079 mapread 0x020d28 thru 0x02198e (0xc67 bytes) 000566 1783350924.295371 punch from 0x01c0f8 to 0x02a112 (0xe01b bytes) 000567 1783350924.303494 write 0x009de8 thru 0x017f8f (0xe1a8 bytes) 000568 1783350924.317987 read 0x0329c5 thru 0x03a5b1 (0x7bed bytes) 000569 1783350924.330440 punch from 0x006421 to 0x009518 (0x30f8 bytes) 000570 1783350924.346422 read 0x02bc23 thru 0x036fec (0xb3ca bytes) 000571 1783350924.353556 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a5b2): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a5b2): Mirror 4: 0x4fc61554 (stale) Mirror 5: 0x4fc61554 (stale) Mirror 6: 0x4fc61554 (stale) Mirror 7: 0x760613a2 (stale) CRC-32 checksum value for chunk [0, 0x3a5b2): Mirror 1: 0xed5fe7b Mirror 8: 0xed5fe7b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a5b2: skipped 000572 1783350925.131625 trunc from 0x03a5b2 to 0x03f5be (0x500d bytes) 000573 1783350925.231595 fallocate 0x0241f7 thru 0x02867c (0x4486 bytes) 000574 1783350925.237991 trunc from 0x03f5bf to 0x004f6d (0x3a651 bytes) 000575 1783350925.268758 trunc from 0x0039e5 to 0x004f6d (0x1589 bytes) 000577 1783350925.277567 trunc from 0x004f6e to 0x036d8c (0x31e1f bytes) 000578 1783350925.324570 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36d8d): Mirror 4: 0x9538499f (stale) Mirror 5: 0x9538499f (stale) Mirror 6: 0x9538499f (stale) Mirror 8: 0x8f608e1c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d8d): Mirror 4: 0x9538499f (stale) Mirror 5: 0x9538499f (stale) Mirror 6: 0x9538499f (stale) Mirror 8: 0x8f608e1c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d8d: skipped 000579 1783350925.754727 fallocate 0x025520 thru 0x02d605 (0x80e6 bytes) 000580 1783350925.765538 MIRROR_RESYNC 5 mirrors 000581 1783350926.587549 mapread 0x00e368 thru 0x01bfe5 (0xdc7e bytes) 000582 1783350926.618964 trunc from 0x036d8d to 0x009152 (0x2dc3a bytes) 000583 1783350926.690929 mapread 0x004ba0 thru 0x009152 (0x45b3 bytes) 000584 1783350926.708003 write 0x02b62a thru 0x02bcd4 (0x6ab bytes) 000585 1783350926.710023 punch from 0x013325 to 0x019bf5 (0x68d1 bytes) 000586 1783350926.717994 punch from 0x02b77c to 0x02bcd4 (0x559 bytes) 000587 1783350926.734976 mapread 0x025edd thru 0x0291b7 (0x32db bytes) 000588 1783350926.744027 fallocate 0x00d9a2 thru 0x016815 (0x8e74 bytes) 000589 1783350926.754257 trunc from 0x02bcd5 to 0x011829 (0x1a4ab bytes) 000590 1783350926.807034 write 0x01eab5 thru 0x020bd4 (0x2120 bytes) 000591 1783350926.814432 MIRROR_RESYNC 5 mirrors 000592 1783350927.979359 trunc from 0x01d77b to 0x020bd4 (0x345a bytes) 000593 1783350928.024599 trunc from 0x002b6b to 0x00b8ae (0x8d44 bytes) 000594 1783350928.032706 mapwrite 0x0056a1 thru 0x00a41a (0x4d7a bytes) 000595 1783350928.076562 mapread 0x01225a thru 0x0132d8 (0x107f bytes) 000596 1783350928.088180 mapwrite 0x007241 thru 0x016a8a (0xf84a bytes) 000597 1783350928.122585 fallocate 0x016a84 thru 0x0206bc (0x9c39 bytes) 000598 1783350928.129334 trunc from 0x020bd5 to 0x0222fc (0x1728 bytes) 000599 1783350928.161184 read 0x009377 thru 0x0120ce (0x8d58 bytes) 000600 1783350928.162882 write 0x00d901 thru 0x016c80 (0x9380 bytes) 000601 1783350928.165509 trunc from 0x0222fd to 0x0396e6 (0x173ea bytes) 000602 1783350928.188479 trunc from 0x0396e7 to 0x032448 (0x729e bytes) 000603 1783350928.200310 mapwrite 0x01af99 thru 0x02620e (0xb276 bytes) 000604 1783350928.251905 mapwrite 0x037bd2 thru 0x03ffff (0x842e bytes) 000605 1783350928.317698 trunc from 0x040000 to 0x00a515 (0x35aea bytes) 000606 1783350928.350002 mapread 0x00737d thru 0x00a515 (0x3199 bytes) 000607 1783350928.351100 read 0x000263 thru 0x00a515 (0xa2b3 bytes) 000608 1783350928.362686 write 0x005e6f thru 0x0080f0 (0x2282 bytes) 000609 1783350928.365762 punch from 0x009a0f to 0x00a515 (0xb07 bytes) 000610 1783350928.372902 fallocate 0x00a608 thru 0x00b4b9 (0xeb2 bytes) 000611 1783350928.380398 mapread 0x004c05 thru 0x009fae (0x53aa bytes) 000612 1783350928.388871 write 0x03b51f thru 0x03ffff (0x4ae1 bytes) 000613 1783350928.395256 write 0x0329bb thru 0x037d24 (0x536a bytes) 000614 1783350928.411773 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x14759f49 (stale) Mirror 5: 0x14759f49 (stale) Mirror 6: 0x14759f49 (stale) Mirror 8: 0x14759f49 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x91da4aef Mirror 9: 0x91da4aef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000615 1783350929.323693 trunc from 0x040000 to 0x020f99 (0x1f066 bytes) 000616 1783350929.414207 trunc from 0x020f9a to 0x02d599 (0xc600 bytes) 000617 1783350929.430203 trunc from 0x0291c2 to 0x02d599 (0x43d8 bytes) 000618 1783350929.433693 fallocate 0x0042d8 thru 0x00e241 (0x9f6a bytes) 000619 1783350929.437139 punch from 0x02708a to 0x0289a7 (0x191e bytes) 000620 1783350929.442384 trunc from 0x021315 to 0x026f5e (0x5c4a bytes) 000621 1783350929.448688 write 0x0109c8 thru 0x016dc6 (0x63ff bytes) 000622 1783350929.462693 mapread 0x0265d2 thru 0x02d599 (0x6fc8 bytes) 000623 1783350929.496126 punch from 0x013e94 to 0x01b854 (0x79c1 bytes) 000624 1783350929.517865 MIRROR_RESYNC 6 mirrors 000625 1783350931.136140 read 0x005508 thru 0x013954 (0xe44d bytes) 000626 1783350931.150212 trunc from 0x02d59a to 0x037aae (0xa515 bytes) 000627 1783350931.181302 mapwrite 0x004f1c thru 0x00f734 (0xa819 bytes) 000628 1783350931.221879 mapread 0x0151d7 thru 0x01fe2e (0xac58 bytes) 000630 1783350931.243195 read 0x029ee8 thru 0x037aae (0xdbc7 bytes) 000631 1783350931.252287 mapwrite 0x00573b thru 0x00aab9 (0x537f bytes) 000632 1783350931.268321 mapread 0x00880c thru 0x016c1f (0xe414 bytes) 000633 1783350931.269665 mapwrite 0x021344 thru 0x029b03 (0x87c0 bytes) 000634 1783350931.295017 mapwrite 0x03dee7 thru 0x03ffff (0x2119 bytes) 000635 1783350931.332586 write 0x030f2c thru 0x039426 (0x84fb bytes) 000636 1783350931.333906 write 0x02678e thru 0x028af7 (0x236a bytes) 000637 1783350931.334796 mapread 0x0362f8 thru 0x03edd3 (0x8adc bytes) 000638 1783350931.336538 trunc from 0x040000 to 0x02b9b6 (0x14649 bytes) 000639 1783350931.355149 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2b9b7): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b9b7): Mirror 4: 0x3ed987bf (stale) Mirror 5: 0x3ed987bf (stale) Mirror 6: 0x3ed987bf (stale) Mirror 8: 0x3ed987bf (stale) Mirror 9: 0x3ed987bf (stale) CRC-32 checksum value for chunk [0, 0x2b9b7): Mirror 1: 0x9362156e Mirror 10: 0x9362156e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b9b7: skipped 000640 1783350932.076301 fallocate 0x015469 thru 0x01c9cf (0x7567 bytes) 000641 1783350932.134511 fallocate 0x0063b0 thru 0x011b10 (0xb761 bytes) 000643 1783350932.140636 read 0x017d9b thru 0x02125f (0x94c5 bytes) 000644 1783350932.149901 trunc from 0x02b9b7 to 0x015433 (0x16583 bytes) 000645 1783350932.195405 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x15434): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15434): Mirror 4: 0x43c1397c (stale) Mirror 5: 0x43c1397c (stale) Mirror 6: 0x43c1397c (stale) Mirror 8: 0x43c1397c (stale) Mirror 9: 0x43c1397c (stale) Mirror 10: 0xb44314e1 (stale) CRC-32 checksum value for chunk [0, 0x15434): Mirror 1: 0xb44314e1 Mirror 11: 0xb44314e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15434: skipped 000646 1783350932.939426 read 0x00d9ec thru 0x0125c6 (0x4bdb bytes) 000647 1783350932.957878 trunc from 0x015434 to 0x00cd4c (0x86e7 bytes) 000648 1783350933.046401 punch from 0x00994c to 0x00cd4c (0x3401 bytes) 000650 1783350933.053470 trunc from 0x00cd4d to 0x00ddaf (0x1063 bytes) 000651 1783350933.071250 read 0x005b59 thru 0x006fe6 (0x148e bytes) 000652 1783350933.081324 mapread 0x001102 thru 0x00ddaf (0xccae bytes) 000653 1783350933.082725 trunc from 0x00896a to 0x00d066 (0x46fd bytes) 000655 1783350933.089573 mapwrite 0x02f137 thru 0x0387bf (0x9689 bytes) 000656 1783350933.143374 write 0x01b7f7 thru 0x01dcad (0x24b7 bytes) 000657 1783350933.145414 read 0x01c2f1 thru 0x022720 (0x6430 bytes) 000658 1783350933.146731 mapread 0x034522 thru 0x036a83 (0x2562 bytes) 000659 1783350933.148451 mapread 0x02fe26 thru 0x030ada (0xcb5 bytes) 000660 1783350933.157292 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x387c0): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x387c0): Mirror 4: 0x766ad242 (stale) Mirror 5: 0x766ad242 (stale) Mirror 6: 0x766ad242 (stale) Mirror 8: 0x766ad242 (stale) Mirror 9: 0x766ad242 (stale) Mirror 10: 0x835d4df (stale) Mirror 11: 0x6143241a (stale) CRC-32 checksum value for chunk [0, 0x387c0): Mirror 1: 0x984dd1b9 Mirror 12: 0x984dd1b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x387c0: skipped 000661 1783350934.303544 trunc from 0x02b67a to 0x0303f9 (0x4d80 bytes) 000662 1783350934.351725 trunc from 0x0387c0 to 0x0000a3 (0x3871c bytes) 000663 1783350934.404633 write 0x035ed1 thru 0x03ffff (0xa12f bytes) 000664 1783350934.409241 write 0x02baf5 thru 0x0370f4 (0xb600 bytes) 000665 1783350934.416799 read 0x03ed73 thru 0x03f158 (0x3e6 bytes) 000666 1783350934.421865 MIRROR_RESYNC 9 mirrors 000667 1783350935.847219 trunc from 0x040000 to 0x01965e (0x269a1 bytes) 000668 1783350935.917226 trunc from 0x017694 to 0x01965e (0x1fcb bytes) 000669 1783350935.927592 trunc from 0x01965f to 0x0158af (0x3daf bytes) 000671 1783350935.956870 read 0x0111b5 thru 0x0158af (0x46fb bytes) 000672 1783350935.971198 mapread 0x00e32f thru 0x0158af (0x7581 bytes) 000673 1783350935.992634 trunc from 0x004f87 to 0x00a48e (0x5508 bytes) 000674 1783350936.001315 trunc from 0x00b07a to 0x0110a4 (0x602b bytes) 000675 1783350936.005215 fallocate 0x012b90 thru 0x01bb77 (0x8fe8 bytes) 000676 1783350936.009057 write 0x035b98 thru 0x03f63e (0x9aa7 bytes) 000677 1783350936.010492 read 0x03d2bc thru 0x03f63e (0x2383 bytes) 000678 1783350936.011499 trunc from 0x003f13 to 0x006575 (0x2663 bytes) 000679 1783350936.016246 read 0x0213ab thru 0x029014 (0x7c6a bytes) 000680 1783350936.023600 write 0x02d4f4 thru 0x02e66a (0x1177 bytes) 000681 1783350936.024450 punch from 0x00643b to 0x013c52 (0xd818 bytes) 000682 1783350936.028919 write 0x00f6b4 thru 0x01a3f2 (0xad3f bytes) 000683 1783350936.033831 punch from 0x021908 to 0x02c89a (0xaf93 bytes) 000684 1783350936.038394 trunc from 0x03076b to 0x033511 (0x2da7 bytes) 000685 1783350936.042701 fallocate 0x01c292 thru 0x024e13 (0x8b82 bytes) 000686 1783350936.047368 trunc from 0x025784 to 0x0332fa (0xdb77 bytes) 000687 1783350936.062232 mapread 0x03a6b5 thru 0x03f63e (0x4f8a bytes) 000688 1783350936.063146 trunc from 0x035bc3 to 0x03636d (0x7ab bytes) 000689 1783350936.067284 mapread 0x01cf33 thru 0x029d51 (0xce1f bytes) 000690 1783350936.115465 write 0x02d869 thru 0x02f94a (0x20e2 bytes) 000691 1783350936.129913 mapwrite 0x01bff1 thru 0x024d84 (0x8d94 bytes) 000693 1783350936.152605 write 0x039dce thru 0x03ffff (0x6232 bytes) 000694 1783350936.154399 read 0x028632 thru 0x02d3bc (0x4d8b bytes) 000695 1783350936.168645 trunc from 0x0105c8 to 0x010a38 (0x471 bytes) 000696 1783350936.174174 read 0x02c18a thru 0x03c14d (0xffc4 bytes) 000697 1783350936.175271 trunc from 0x040000 to 0x00329c (0x3cd63 bytes) 000698 1783350936.197668 trunc from 0x003141 to 0x00329c (0x15c bytes) 000699 1783350936.203121 mapwrite 0x0119a9 thru 0x01822f (0x6887 bytes) 000700 1783350936.266386 trunc from 0x018230 to 0x03040b (0x181dc bytes) 000702 1783350936.292045 trunc from 0x02e11c to 0x03040b (0x22f0 bytes) 000703 1783350936.298216 mapwrite 0x035f25 thru 0x03ffff (0xa0db bytes) 000704 1783350936.352384 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x205446d7 (stale) Mirror 5: 0x205446d7 (stale) Mirror 6: 0x205446d7 (stale) Mirror 8: 0x205446d7 (stale) Mirror 9: 0x205446d7 (stale) Mirror 10: 0x205446d7 (stale) Mirror 12: 0x205446d7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x205446d7 (stale) Mirror 5: 0x205446d7 (stale) Mirror 6: 0x205446d7 (stale) Mirror 8: 0x205446d7 (stale) Mirror 9: 0x205446d7 (stale) Mirror 10: 0x205446d7 (stale) Mirror 12: 0x205446d7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000705 1783350936.910107 punch from 0x030864 to 0x03fe34 (0xf5d1 bytes) 000706 1783350936.921650 mapwrite 0x03a629 thru 0x03c518 (0x1ef0 bytes) 000707 1783350936.963287 read 0x0147fa thru 0x015735 (0xf3c bytes) 000708 1783350936.976224 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x205446d7 (stale) Mirror 5: 0x205446d7 (stale) Mirror 6: 0x205446d7 (stale) Mirror 8: 0x205446d7 (stale) Mirror 9: 0x205446d7 (stale) Mirror 10: 0x205446d7 (stale) Mirror 12: 0x205446d7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd468aad4 Mirror 13: 0xd468aad4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000709 1783350938.377527 trunc from 0x03a721 to 0x03de2d (0x370d bytes) 000710 1783350938.478175 punch from 0x039f65 to 0x03c479 (0x2515 bytes) 000711 1783350938.484343 trunc from 0x040000 to 0x0034e8 (0x3cb17 bytes) 000712 1783350938.517316 trunc from 0x0034e9 to 0x032652 (0x2f16a bytes) 000713 1783350938.536083 trunc from 0x032653 to 0x03aa2f (0x83dd bytes) 000714 1783350938.556461 punch from 0x0190ca to 0x02075a (0x7691 bytes) 000715 1783350938.563616 fallocate 0x004f50 thru 0x00a05a (0x510b bytes) 000716 1783350938.569321 fallocate 0x038829 thru 0x039b93 (0x136b bytes) 000717 1783350938.575082 mapread 0x01deba thru 0x025297 (0x73de bytes) 000718 1783350938.609124 trunc from 0x03aa30 to 0x012010 (0x28a1f bytes) 000719 1783350938.627522 punch from 0x000720 to 0x002ee2 (0x27c3 bytes) 000720 1783350938.647045 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12011): Mirror 4: 0xb43e1645 (stale) Mirror 5: 0xb43e1645 (stale) Mirror 6: 0xb43e1645 (stale) Mirror 8: 0xb43e1645 (stale) Mirror 10: 0xb43e1645 (stale) Mirror 12: 0xb43e1645 (stale) Mirror 13: 0xb573006c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12011): Mirror 4: 0xb43e1645 (stale) Mirror 5: 0xb43e1645 (stale) Mirror 6: 0xb43e1645 (stale) Mirror 8: 0xb43e1645 (stale) Mirror 10: 0xb43e1645 (stale) Mirror 12: 0xb43e1645 (stale) Mirror 13: 0xb573006c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12011: skipped 000721 1783350939.082526 MIRROR_RESYNC 8 mirrors 000722 1783350940.405087 punch from 0x00a570 to 0x00de9c (0x392d bytes) 000724 1783350940.471566 mapwrite 0x025699 thru 0x02f2e8 (0x9c50 bytes) 000725 1783350940.578934 punch from 0x001aa4 to 0x00e48a (0xc9e7 bytes) 000726 1783350940.586857 mapwrite 0x020460 thru 0x0215b0 (0x1151 bytes) 000727 1783350940.610132 trunc from 0x02f2e9 to 0x01e52a (0x10dbe bytes) 000729 1783350940.633172 trunc from 0x002506 to 0x0042c9 (0x1dc4 bytes) 000730 1783350940.642255 mapwrite 0x00620e thru 0x009e0f (0x3c02 bytes) 000731 1783350940.676323 mapwrite 0x0105c2 thru 0x019940 (0x937f bytes) 000732 1783350940.704639 mapread 0x00ee41 thru 0x01a569 (0xb729 bytes) 000733 1783350940.718299 trunc from 0x01e52b to 0x016d23 (0x7807 bytes) 000735 1783350940.735504 trunc from 0x0001ce to 0x000486 (0x2b9 bytes) 000736 1783350940.740627 punch from 0x001d4f to 0x00c1f0 (0xa4a2 bytes) 000737 1783350940.748518 mapread 0x014f97 thru 0x016d23 (0x1d8d bytes) 000738 1783350940.751178 fallocate 0x01216a thru 0x021e24 (0xfcbb bytes) 000739 1783350940.760019 fallocate 0x02dca9 thru 0x02eb2f (0xe87 bytes) 000741 1783350940.766237 read 0x01fb5c thru 0x028462 (0x8907 bytes) 000742 1783350940.782796 punch from 0x024a36 to 0x02eb2f (0xa0fa bytes) 000743 1783350940.808065 mapwrite 0x01e9f6 thru 0x02715e (0x8769 bytes) 000744 1783350940.850546 trunc from 0x02eb30 to 0x03cdbc (0xe28d bytes) 000745 1783350940.879154 write 0x022483 thru 0x02ea92 (0xc610 bytes) 000746 1783350940.883622 trunc from 0x03cdbd to 0x02aa0e (0x123ae bytes) 000747 1783350940.908703 trunc from 0x02aa0f to 0x02a1a3 (0x86b bytes) 000749 1783350940.947796 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a1a4): Mirror 4: 0xc9d968b2 (stale) Mirror 6: 0xc9d968b2 (stale) Mirror 8: 0xc9d968b2 (stale) Mirror 10: 0xc9d968b2 (stale) Mirror 12: 0xc9d968b2 (stale) Mirror 13: 0xc9d968b2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a1a4): Mirror 4: 0xc9d968b2 (stale) Mirror 6: 0xc9d968b2 (stale) Mirror 8: 0xc9d968b2 (stale) Mirror 10: 0xc9d968b2 (stale) Mirror 12: 0xc9d968b2 (stale) Mirror 13: 0xc9d968b2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a1a4: skipped 000751 1783350941.430393 read 0x023271 thru 0x029a05 (0x6795 bytes) 000752 1783350941.440763 trunc from 0x02686d to 0x02a1a3 (0x3937 bytes) 000754 1783350941.444888 trunc from 0x02991b to 0x02a1a3 (0x889 bytes) 000757 1783350941.449419 mapwrite 0x03b232 thru 0x03ffff (0x4dce bytes) 000758 1783350941.548061 read 0x013f3e thru 0x014146 (0x209 bytes) 000759 1783350941.572742 read 0x004210 thru 0x006a0d (0x27fe bytes) 000760 1783350941.573702 read 0x031ad8 thru 0x0353b8 (0x38e1 bytes) 000761 1783350941.575314 fallocate 0x0109e5 thru 0x016cae (0x62ca bytes) 000762 1783350941.582353 mapread 0x035dee thru 0x03accb (0x4ede bytes) 000764 1783350941.586486 mapread 0x031f3b thru 0x039ec1 (0x7f87 bytes) 000765 1783350941.587714 mapread 0x0303b1 thru 0x038cb8 (0x8908 bytes) 000766 1783350941.590476 mapwrite 0x0265ff thru 0x02bed3 (0x58d5 bytes) 000767 1783350941.627475 mapwrite 0x027fd5 thru 0x02f1fd (0x7229 bytes) 000768 1783350941.658373 mapread 0x020702 thru 0x02f83c (0xf13b bytes) 000769 1783350941.659887 mapread 0x0252de thru 0x02be8d (0x6bb0 bytes) 000771 1783350941.662116 mapread 0x02e629 thru 0x037ddc (0x97b4 bytes) 000772 1783350941.663428 mapread 0x03b6d2 thru 0x03b949 (0x278 bytes) 000773 1783350941.664601 mapwrite 0x03c7f5 thru 0x03ffff (0x380b bytes) 000774 1783350941.687455 trunc from 0x03efe6 to 0x03ffff (0x101a bytes) 000775 1783350941.694466 punch from 0x013ef6 to 0x01b02b (0x7136 bytes) 000776 1783350941.703735 mapread 0x00d2fe thru 0x010f92 (0x3c95 bytes) 000777 1783350941.704718 write 0x00c638 thru 0x011151 (0x4b1a bytes) 000778 1783350941.705859 mapread 0x021098 thru 0x025817 (0x4780 bytes) 000779 1783350941.713498 mapwrite 0x02a7ce thru 0x032e11 (0x8644 bytes) 000780 1783350941.744654 fallocate 0x02bcfb thru 0x0363f1 (0xa6f7 bytes) 000781 1783350941.752572 punch from 0x004b59 to 0x00dd13 (0x91bb bytes) 000782 1783350941.757693 trunc from 0x040000 to 0x0286bd (0x17942 bytes) 000783 1783350941.787365 mapread 0x004d3e thru 0x00eab5 (0x9d78 bytes) 000784 1783350941.799181 trunc from 0x006cf8 to 0x00d44f (0x6758 bytes) 000785 1783350941.806135 write 0x00f8c2 thru 0x0114a6 (0x1be5 bytes) 000786 1783350941.808215 write 0x018834 thru 0x025abc (0xd289 bytes) 000788 1783350941.811748 trunc from 0x02316a to 0x0286bd (0x5554 bytes) 000789 1783350941.832967 trunc from 0x0286be to 0x03c16b (0x13aae bytes) 000790 1783350941.852921 read 0x02ff0e thru 0x0363de (0x64d1 bytes) 000791 1783350941.877246 fallocate 0x01f42f thru 0x029ea1 (0xaa73 bytes) 000792 1783350941.884230 fallocate 0x02be1d thru 0x02ca6c (0xc50 bytes) 000793 1783350941.894192 write 0x010303 thru 0x01518e (0x4e8c bytes) 000794 1783350941.898402 punch from 0x01cace to 0x021ecf (0x5402 bytes) 000795 1783350941.906491 trunc from 0x03c16c to 0x03446b (0x7d00 bytes) 000796 1783350941.930376 mapread 0x026d80 thru 0x02f544 (0x87c5 bytes) 000798 1783350941.932152 punch from 0x02b435 to 0x030d76 (0x5942 bytes) 000800 1783350941.947104 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3446c): Mirror 4: 0xc858b9c8 (stale) Mirror 6: 0xc858b9c8 (stale) Mirror 8: 0xc858b9c8 (stale) Mirror 10: 0xc858b9c8 (stale) Mirror 12: 0xc858b9c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3446c): Mirror 4: 0xc858b9c8 (stale) Mirror 6: 0xc858b9c8 (stale) Mirror 8: 0xc858b9c8 (stale) Mirror 10: 0xc858b9c8 (stale) Mirror 12: 0xc858b9c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3446c: skipped 000801 1783350942.376159 write 0x0356df thru 0x03ffff (0xa921 bytes) 000803 1783350942.381089 trunc from 0x040000 to 0x033b4d (0xc4b2 bytes) 000804 1783350942.401024 write 0x0245e4 thru 0x02f1fc (0xac19 bytes) 000805 1783350942.411544 fallocate 0x00416f thru 0x00b02c (0x6ebe bytes) 000806 1783350942.416056 trunc from 0x02b2bf to 0x031acb (0x680d bytes) 000807 1783350942.435451 mapwrite 0x02c0c1 thru 0x030ba3 (0x4ae3 bytes) 000808 1783350942.481358 read 0x00957f thru 0x01463e (0xb0c0 bytes) 000809 1783350942.492281 mapread 0x00fe17 thru 0x014977 (0x4b61 bytes) 000810 1783350942.493657 fallocate 0x00dd3c thru 0x019d8a (0xc04f bytes) 000811 1783350942.504938 MIRROR_RESYNC 6 mirrors 000812 1783350944.334963 trunc from 0x033b4e to 0x03c80a (0x8cbd bytes) 000813 1783350944.432648 MIRROR_RESYNC 6 mirrors 000814 1783350946.088258 fallocate 0x02b835 thru 0x02e46f (0x2c3b bytes) 000815 1783350946.134448 trunc from 0x03c80b to 0x02ccc9 (0xfb41 bytes) 000816 1783350946.155164 fallocate 0x02330f thru 0x02b797 (0x8489 bytes) 000817 1783350946.173771 punch from 0x004d55 to 0x004d91 (0x3d bytes) 000818 1783350946.179640 mapread 0x01845a thru 0x023b51 (0xb6f8 bytes) 000819 1783350946.225007 trunc from 0x0298d8 to 0x02ccc9 (0x33f2 bytes) 000820 1783350946.229708 mapread 0x008a28 thru 0x00f90a (0x6ee3 bytes) 000822 1783350946.241471 trunc from 0x02ccca to 0x01fd16 (0xcfb3 bytes) 000823 1783350946.254649 trunc from 0x01fd17 to 0x006bc8 (0x1914e bytes) 000824 1783350946.271462 read 0x00649e thru 0x006bc8 (0x72b bytes) 000825 1783350946.272179 trunc from 0x001b8b to 0x004023 (0x2499 bytes) 000826 1783350946.275614 write 0x000a3d thru 0x007c2d (0x71f1 bytes) 000827 1783350946.279034 read 0x000c53 thru 0x0022d1 (0x167f bytes) 000828 1783350946.282134 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x7c2e): Mirror 6: 0xa5537e9f (stale) Mirror 8: 0xa5537e9f (stale) Mirror 10: 0xa5537e9f (stale) Mirror 12: 0xa5537e9f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7c2e): Mirror 6: 0xa5537e9f (stale) Mirror 8: 0xa5537e9f (stale) Mirror 10: 0xa5537e9f (stale) Mirror 12: 0xa5537e9f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7c2e: skipped 000829 1783350946.551593 mapread 0x000e8e thru 0x005002 (0x4175 bytes) 000830 1783350946.577670 trunc from 0x007c2e to 0x03249a (0x2a86d bytes) 000831 1783350946.602093 mapread 0x00d74b thru 0x01a418 (0xccce bytes) 000832 1783350946.619189 mapwrite 0x0287ed thru 0x02a0ea (0x18fe bytes) 000833 1783350946.660706 read 0x02672c thru 0x03249a (0xbd6f bytes) 000834 1783350946.669683 trunc from 0x0260bd to 0x02cc86 (0x6bca bytes) 000835 1783350946.680864 mapread 0x0277e1 thru 0x03249a (0xacba bytes) 000837 1783350946.696103 punch from 0x005ef0 to 0x00e595 (0x86a6 bytes) 000838 1783350946.702029 write 0x024b2d thru 0x034721 (0xfbf5 bytes) 000839 1783350946.704445 mapread 0x01f191 thru 0x02054e (0x13be bytes) 000840 1783350946.719328 mapwrite 0x0371ee thru 0x03f6c9 (0x84dc bytes) 000841 1783350946.770112 mapwrite 0x03471b thru 0x03ffff (0xb8e5 bytes) 000842 1783350946.804273 trunc from 0x040000 to 0x01c700 (0x238ff bytes) 000843 1783350946.827548 fallocate 0x008c69 thru 0x00c946 (0x3cde bytes) 000844 1783350946.833319 trunc from 0x015362 to 0x01c700 (0x739f bytes) 000845 1783350946.841107 read 0x007c1b thru 0x010d99 (0x917f bytes) 000846 1783350946.850369 punch from 0x014646 to 0x017833 (0x31ee bytes) 000848 1783350946.858076 punch from 0x000f61 to 0x00e6bf (0xd75f bytes) 000849 1783350946.864228 read 0x0050d1 thru 0x007d78 (0x2ca8 bytes) 000850 1783350946.877216 fallocate 0x026d3a thru 0x02c58f (0x5856 bytes) 000851 1783350946.881098 read 0x01b5e5 thru 0x01c700 (0x111c bytes) 000852 1783350946.881833 trunc from 0x00b876 to 0x016eeb (0xb676 bytes) 000853 1783350946.885458 write 0x019415 thru 0x0278c2 (0xe4ae bytes) 000854 1783350946.886764 mapread 0x000250 thru 0x00604b (0x5dfc bytes) 000855 1783350946.887552 mapwrite 0x02050d thru 0x02ee72 (0xe966 bytes) 000856 1783350946.947708 write 0x030701 thru 0x031f65 (0x1865 bytes) 000857 1783350946.948599 fallocate 0x03cafc thru 0x03ffff (0x3504 bytes) 000858 1783350946.954152 mapread 0x01948d thru 0x01b85a (0x23ce bytes) 000859 1783350946.956116 trunc from 0x030c5d to 0x031f65 (0x1309 bytes) 000860 1783350946.972779 punch from 0x002772 to 0x007f86 (0x5815 bytes) 000861 1783350946.978888 fallocate 0x03f927 thru 0x03ffff (0x6d9 bytes) 000862 1783350946.989878 MIRROR_RESYNC 5 mirrors 000863 1783350948.346691 read 0x0330ed thru 0x03ffff (0xcf13 bytes) 000864 1783350948.360304 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) 000865 1783350948.702202 mapwrite 0x01d765 thru 0x023d44 (0x65e0 bytes) 000866 1783350948.805434 trunc from 0x02db81 to 0x03bd48 (0xe1c8 bytes) 000867 1783350948.811265 trunc from 0x03adf8 to 0x03ffff (0x5208 bytes) 000868 1783350948.816969 punch from 0x028125 to 0x035c8a (0xdb66 bytes) 000869 1783350948.821487 trunc from 0x040000 to 0x01e787 (0x21878 bytes) 000870 1783350948.869764 trunc from 0x006e31 to 0x013163 (0xc333 bytes) 000871 1783350948.896991 MIRROR_RESYNC 4 mirrors 000872 1783350950.094912 write 0x03646c thru 0x03a54b (0x40e0 bytes) 000873 1783350950.124075 trunc from 0x025c7a to 0x02d5e5 (0x796c bytes) 000874 1783350950.132020 read 0x014486 thru 0x0173da (0x2f55 bytes) 000875 1783350950.144734 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a54c): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a54c): Mirror 6: 0xae5d0e85 (stale) Mirror 8: 0xae5d0e85 (stale) Mirror 12: 0xae5d0e85 (stale) CRC-32 checksum value for chunk [0, 0x3a54c): Mirror 1: 0x36eb6c09 Mirror 13: 0x36eb6c09 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a54c: skipped 000876 1783350951.056634 mapwrite 0x037832 thru 0x037f49 (0x718 bytes) 000877 1783350951.187197 trunc from 0x039321 to 0x03a54b (0x122b bytes) 000879 1783350951.196793 mapread 0x010778 thru 0x01ae2d (0xa6b6 bytes) 000880 1783350951.216242 trunc from 0x001d88 to 0x00ddd8 (0xc051 bytes) 000881 1783350951.224621 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a54c): Mirror 6: 0xae5d0e85 (stale) Mirror 8: 0xae5d0e85 (stale) Mirror 12: 0xae5d0e85 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a54c): Mirror 6: 0xae5d0e85 (stale) Mirror 8: 0xae5d0e85 (stale) Mirror 12: 0xae5d0e85 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a54c: skipped 000882 1783350951.615680 trunc from 0x001874 to 0x007911 (0x609e bytes) 000883 1783350951.619484 write 0x01dcb6 thru 0x027bdd (0x9f28 bytes) 000884 1783350951.628544 write 0x01ebdd thru 0x0256ac (0x6ad0 bytes) 000885 1783350951.629478 trunc from 0x03a54c to 0x03ed4f (0x4804 bytes) 000886 1783350951.665568 trunc from 0x03ed50 to 0x033d74 (0xafdb bytes) 000887 1783350951.682025 mapwrite 0x00bc6c thru 0x00c818 (0xbad bytes) 000888 1783350951.706886 write 0x028e77 thru 0x02bfc4 (0x314e bytes) 000889 1783350951.726729 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x33d75): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33d75): Mirror 6: 0x9cb21ec0 (stale) Mirror 8: 0x9cb21ec0 (stale) Mirror 12: 0x9cb21ec0 (stale) CRC-32 checksum value for chunk [0, 0x33d75): Mirror 1: 0x6efd8aaa Mirror 13: 0x6efd8aaa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33d75: skipped 000890 1783350952.759395 mapread 0x031186 thru 0x033d74 (0x2bef bytes) 000891 1783350952.813317 MIRROR_RESYNC 5 mirrors 000892 1783350953.787499 punch from 0x021153 to 0x028a23 (0x78d1 bytes) 000893 1783350953.820038 trunc from 0x033d75 to 0x02e604 (0x5770 bytes) 000894 1783350953.834992 mapwrite 0x02bd88 thru 0x033ae3 (0x7d5c bytes) 000895 1783350953.894113 fallocate 0x01a463 thru 0x0266c9 (0xc267 bytes) 000896 1783350953.899217 trunc from 0x033ae4 to 0x019044 (0x1aa9f bytes) 000897 1783350953.914745 trunc from 0x019045 to 0x01abd7 (0x1b93 bytes) 000898 1783350953.930978 write 0x001318 thru 0x00d073 (0xbd5c bytes) 000900 1783350953.942871 fallocate 0x00bb16 thru 0x018f71 (0xd45c bytes) 000901 1783350953.948156 write 0x01352f thru 0x015b50 (0x2622 bytes) 000902 1783350953.958316 trunc from 0x01167c to 0x0129fc (0x1381 bytes) 000903 1783350953.963350 trunc from 0x00a114 to 0x015028 (0xaf15 bytes) 000904 1783350953.979665 mapread 0x001b65 thru 0x0035e8 (0x1a84 bytes) 000905 1783350953.981492 punch from 0x003b1c to 0x009114 (0x55f9 bytes) 000906 1783350953.992748 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1abd8): Mirror 6: 0x6de85e94 (stale) Mirror 8: 0x6de85e94 (stale) Mirror 13: 0x6de85e94 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1abd8): Mirror 6: 0x6de85e94 (stale) Mirror 8: 0x6de85e94 (stale) Mirror 13: 0x6de85e94 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1abd8: skipped 000908 1783350954.312797 read 0x008677 thru 0x013c95 (0xb61f bytes) 000910 1783350954.323929 trunc from 0x0136d9 to 0x015ac3 (0x23eb bytes) 000911 1783350954.329045 read 0x016c40 thru 0x01abd7 (0x3f98 bytes) 000912 1783350954.342977 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1abd8): Mirror 8: 0x6de85e94 (stale) Mirror 13: 0x6de85e94 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1abd8): Mirror 8: 0x6de85e94 (stale) Mirror 13: 0x6de85e94 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1abd8: skipped 000913 1783350954.626083 trunc from 0x01abd8 to 0x01a2bd (0x91a bytes) 000914 1783350954.645473 trunc from 0x01a2be to 0x00ca77 (0xd846 bytes) 000915 1783350954.659497 punch from 0x001bc2 to 0x00a012 (0x8451 bytes) 000916 1783350954.664089 trunc from 0x00ca78 to 0x0124c0 (0x5a49 bytes) 000917 1783350954.682138 trunc from 0x0124c1 to 0x0114d8 (0xfe8 bytes) 000918 1783350954.696272 mapwrite 0x00c496 thru 0x0163be (0x9f29 bytes) 000919 1783350954.749416 read 0x012840 thru 0x0163be (0x3b7f bytes) 000920 1783350954.751294 mapread 0x01109d thru 0x0163be (0x5322 bytes) 000921 1783350954.753577 mapwrite 0x032ecf thru 0x036377 (0x34a9 bytes) 000922 1783350954.794130 punch from 0x01062b to 0x01d6fb (0xd0d1 bytes) 000923 1783350954.799640 write 0x0092da thru 0x01773c (0xe463 bytes) 000925 1783350954.807151 trunc from 0x036378 to 0x017da7 (0x1e5d0 bytes) 000926 1783350954.821757 write 0x022ffd thru 0x02dded (0xadf1 bytes) 000927 1783350954.827800 trunc from 0x02ddee to 0x031eef (0x4102 bytes) 000928 1783350954.846065 trunc from 0x00c085 to 0x00caf8 (0xa74 bytes) 000929 1783350954.860208 mapread 0x01ef39 thru 0x024d17 (0x5ddf bytes) 000930 1783350954.872125 write 0x014928 thru 0x01eb4a (0xa223 bytes) 000932 1783350954.875067 trunc from 0x031ef0 to 0x03a931 (0x8a42 bytes) 000934 1783350954.894934 trunc from 0x03a932 to 0x029646 (0x112eb bytes) 000935 1783350954.923023 punch from 0x02108e to 0x022d84 (0x1cf7 bytes) 000936 1783350954.930179 fallocate 0x02bd56 thru 0x02f7a3 (0x3a4e bytes) 000937 1783350954.937029 mapwrite 0x01f875 thru 0x026c68 (0x73f4 bytes) 000938 1783350954.962105 fallocate 0x024177 thru 0x032ad4 (0xe95e bytes) 000939 1783350954.972475 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x29647): Mirror 8: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29647): Mirror 8: 0xe7773784 (stale) Mirror 13: 0xe7773784 (stale) CRC-32 checksum value for chunk [0, 0x29647): Mirror 1: 0x36662251 Mirror 14: 0x36662251 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29647: skipped 000940 1783350955.867564 MIRROR_RESYNC 4 mirrors 000941 1783350956.480261 MIRROR_EXTEND to 5 mirrors 000942 1783350957.117705 trunc from 0x029647 to 0x03c25c (0x12c16 bytes) 000943 1783350957.202414 fallocate 0x03e560 thru 0x03ffff (0x1aa0 bytes) 000944 1783350957.212373 read 0x0254fa thru 0x02c504 (0x700b bytes) 000946 1783350957.223384 punch from 0x03cada to 0x03ffff (0x3526 bytes) 000947 1783350957.229355 read 0x013e27 thru 0x01be30 (0x800a bytes) 000948 1783350957.240290 trunc from 0x016982 to 0x017bf4 (0x1273 bytes) 000949 1783350957.248720 read 0x010292 thru 0x017627 (0x7396 bytes) 000950 1783350957.254455 mapread 0x02829b thru 0x028ff9 (0xd5f bytes) 000951 1783350957.255854 mapwrite 0x03733d thru 0x03c0f8 (0x4dbc bytes) 000952 1783350957.281441 write 0x00550b thru 0x0106e5 (0xb1db bytes) 000953 1783350957.286596 read 0x0230aa thru 0x0295f0 (0x6547 bytes) 000954 1783350957.296422 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1a4:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x78c4d11e (stale) Mirror 13: 0x78c4d11e (stale) Mirror 15: 0x78c4d11e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x78c4d11e (stale) Mirror 13: 0x78c4d11e (stale) Mirror 15: 0x78c4d11e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000955 1783350957.713376 mapwrite 0x01e06a thru 0x025a8e (0x7a25 bytes) 000956 1783350957.767635 fallocate 0x00437a thru 0x005d2c (0x19b3 bytes) 000957 1783350957.777029 mapread 0x0202cb thru 0x02ec50 (0xe986 bytes) 000958 1783350957.786410 mapwrite 0x028f97 thru 0x02a0b7 (0x1121 bytes) 000959 1783350957.802649 trunc from 0x034cb3 to 0x03ffff (0xb34d bytes) 000960 1783350957.810066 punch from 0x02fc42 to 0x03aa9d (0xae5c bytes) 000961 1783350957.815988 punch from 0x02dea6 to 0x03531b (0x7476 bytes) 000962 1783350957.820699 mapwrite 0x00ee1b thru 0x01e438 (0xf61e bytes) 000963 1783350957.862233 trunc from 0x040000 to 0x00a83f (0x357c0 bytes) 000965 1783350957.885759 trunc from 0x00a840 to 0x0388b1 (0x2e072 bytes) 000966 1783350957.904083 mapread 0x00b08e thru 0x0121b1 (0x7124 bytes) 000967 1783350957.921199 trunc from 0x024554 to 0x0312f3 (0xcda0 bytes) 000968 1783350957.927014 read 0x002621 thru 0x00c9e4 (0xa3c4 bytes) 000970 1783350957.934214 mapread 0x011bf9 thru 0x0156ff (0x3b07 bytes) 000971 1783350957.945083 fallocate 0x01813a thru 0x024cda (0xcba1 bytes) 000972 1783350957.952109 punch from 0x036ae1 to 0x0388b1 (0x1dd1 bytes) 000973 1783350957.959020 punch from 0x00d8c0 to 0x019f8d (0xc6ce bytes) 000974 1783350957.965245 fallocate 0x02bb92 thru 0x033419 (0x7888 bytes) 000975 1783350957.973059 read 0x01b64b thru 0x022f36 (0x78ec bytes) 000976 1783350957.999954 read 0x00be1f thru 0x0170a5 (0xb287 bytes) 000977 1783350958.001241 trunc from 0x0388b2 to 0x020945 (0x17f6c bytes) 000978 1783350958.022195 trunc from 0x011b5c to 0x018db5 (0x725a bytes) 000979 1783350958.030560 punch from 0x0119d8 to 0x012d46 (0x136f bytes) 000980 1783350958.039441 fallocate 0x01e0bd thru 0x02626f (0x81b3 bytes) 000981 1783350958.059559 MIRROR_RESYNC 4 mirrors 000982 1783350959.244384 mapwrite 0x005773 thru 0x0077fb (0x2089 bytes) 000983 1783350959.340065 read 0x00321e thru 0x00f33d (0xc120 bytes) 000984 1783350959.341674 mapread 0x0075db thru 0x017052 (0xfa78 bytes) 000985 1783350959.356158 mapwrite 0x020e2e thru 0x030dbf (0xff92 bytes) 000986 1783350959.464050 trunc from 0x030dc0 to 0x03a5a2 (0x97e3 bytes) 000987 1783350959.487415 mapread 0x01fa99 thru 0x0235d9 (0x3b41 bytes) 000988 1783350959.488703 trunc from 0x03a5a3 to 0x0021bf (0x383e3 bytes) 000989 1783350959.532032 punch from 0x002050 to 0x0021bf (0x170 bytes) 000991 1783350959.542903 read 0x001bfc thru 0x0021bf (0x5c4 bytes) 000992 1783350959.571963 trunc from 0x0021c0 to 0x01268b (0x104cc bytes) 000993 1783350959.600743 trunc from 0x01268c to 0x02444a (0x11dbf bytes) 000994 1783350959.632285 fallocate 0x0313bf thru 0x03dac0 (0xc702 bytes) 000995 1783350959.648171 MIRROR_RESYNC 4 mirrors 000996 1783350960.555293 fallocate 0x035aff thru 0x03ffff (0xa501 bytes) 000997 1783350960.596042 trunc from 0x01864c to 0x023dbb (0xb770 bytes) 000998 1783350960.607103 trunc from 0x02444b to 0x00c061 (0x183e9 bytes) 001000 1783350960.630223 read 0x006359 thru 0x00c061 (0x5d09 bytes) All operations completed A-OK! PASS 70a (98s) == sanity-flr test 71: check mirror extend parent fid ==== 11:16:05 (1783350965) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x56:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 11:16:14 (1783350974) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.176307 s, 11.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127751 s, 8.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 11:16:21 (1783350981) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 8599040 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3886304 .. done Extending file size to 5292288 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8364704 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8767680 .. Extending file size to 9142144 .. Extending file size to 9222560 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8548448 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8576608 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8807264 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61213 61214 61215 61217 61218 done 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) 192.168.202.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (75s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:17:36 (1783351056) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8016608 .. Extending file size to 6778656 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8261152 .. failed Extending file size to 7681728 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8288224 .. Extending file size to 8556096 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8678144 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8856480 .. Extending file size to 9081568 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62820 62822 62823 62825 62826 done 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) 192.168.202.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:18:53 (1783351133) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2308 1268996 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2224 1269080 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24508 3582512 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18628 3588392 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43136 7170904 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2308 1268996 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2224 1269080 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24508 3582512 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18628 3588392 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43136 7170904 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0631091 s, 33.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.211663 s, 9.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xca24c879 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:19:09 (1783351149) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.110957 s, 18.9 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 11:19:15 (1783351155) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x65:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } PASS 203 (5s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:19:20 (1783351160) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0677168 s, 31.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x250:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x24f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:19:27 (1783351167) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0974887 s, 21.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:19:33 (1783351173) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.136994 s, 15.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:20:03 (1783351203) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg222-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772012 s, 13.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x251:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x251:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x250:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x251:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x251:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x250:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x252:0x0] } PASS 204d (49s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:20:52 (1783351252) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg222-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.100772 s, 20.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (44s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:21:36 (1783351296) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg222-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg222-server: Pool lustre.test_204f created Adding targets to pool oleg222-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg222-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0804788 s, 26.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg222-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg222-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg222-server: Pool lustre.test_204f destroyed PASS 204f (63s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:22:39 (1783351359) PASS 205a (5s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:22:44 (1783351364) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:22:49 (1783351369) oleg222-server: Pool lustre.test_206 created Adding targets to pool oleg222-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg222-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg222-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg222-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg222-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:23:12 (1783351392) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596229 s, 17.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353151 s, 29.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xeeb99ed5 Mirror 3: 0xeeb99ed5 PASS 207 (7s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:23:19 (1783351399) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:23:21 (1783351401) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:23:23 (1783351403) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0355304 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x20d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2640000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2640000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2640000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1280000 Read this (4 bytes): BBBB 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) PASS 209a (28s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:23:51 (1783351431) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00590871 s, 1.4 MB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:23:57 (1783351437) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285669 s, 36.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x86 lmm_fid: [0x200000405:0x86:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x86 lmm_fid: [0x200000405:0x86:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x140:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611270 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-06 11:23:58.000000000 -0400 Modify: 2026-07-06 11:23:58.000000000 -0400 Change: 2026-07-06 11:24:00.000000000 -0400 Birth: 2026-07-06 11:23:58.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:24:04 (1783351444) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316597 s, 33.1 MB/s oleg222-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg222-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg222-server: O/280000401/d29/13d: File not found by ext2_lookup PASS 210b (16s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:24:20 (1783351460) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.158629 s, 258 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0182881 s, 224 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 6 11:24 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x8b:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 6 11:24 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:24:27 (1783351467) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:24:35 (1783351475) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1693 sec =========== 11:24:42 (1783351482) === sanity-flr: start cleanup 11:24:43 (1783351483) === === sanity-flr: finish cleanup 11:24:50 (1783351490) ===