-----============= acceptance-small: sanity-flr ============----- Sat Sep 5 16:39:58 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg616-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 16:40:18 (1788640818) === oleg616-client.virtnet: executing check_config_client /mnt/lustre oleg616-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg616-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff91b7c88ab800.idle_timeout=debug osc.lustre-OST0001-osc-ffff91b7c88ab800.idle_timeout=debug disable quota as required oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 16:41:02 (1788640862) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8224] == sanity-flr test 0a: lfs mirror create with -N option == 16:41:05 (1788640865) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:41:23 (1788640883) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:41:26 (1788640886) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:41:29 (1788640889) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:41:54 (1788640914) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:41:57 (1788640917) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 16:42:01 (1788640921) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff91b7c88ab800.stats=clear osc.lustre-OST0001-osc-ffff91b7c88ab800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 16:42:10 (1788640930) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 16:42:20 (1788640940) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:42:28 (1788640948) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 16:42:32 (1788640952) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:42:39 (1788640959) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 16:42:48 (1788640968) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:42:57 (1788640977) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:43:05 (1788640985) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:43:12 (1788640992) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:43:23 (1788641003) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:43:30 (1788641010) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 1.79409 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998881 s, 10.5 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:43:42 (1788641022) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 0.627333 s, 23.4 MB/s PASS 22 (12s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 16:43:54 (1788641034) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:44:06 (1788641046) Stopping /mnt/lustre-ost1 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91b7c88ab800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91b7c88ab800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg616-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91b7c88ab800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91b7c88ab800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg616-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (74s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:45:20 (1788641120) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.405244 s, 18.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91b7c88ab800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91b7c88ab800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg616-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:46:03 (1788641163) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91b7c88ab800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91b7c88ab800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg616-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91b7c88ab800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91b7c88ab800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg616-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (80s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:47:23 (1788641243) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.60283 s, 14.6 MB/s -rw-r--r-- 1 root root 52428800 Sep 5 16:47 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.018s user 0m0.001s sys 0m0.175s Stopping /mnt/lustre-ost1 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91b7c88ab800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91b7c88ab800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.159s user 0m0.001s sys 0m0.186s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg616-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg616-server oleg616-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91b7c88ab800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91b7c88ab800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.978s user 0m0.000s sys 0m0.184s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg616-server: oleg616-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg616-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg616-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91b7c88ab800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91b7c88ab800.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:49:01 (1788641341) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:49:06 (1788641346) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:49:09 (1788641349) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 16:49:12 (1788641352) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 16:49:19 (1788641359) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.513677 s, 24.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:49:30 (1788641370) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360962 s, 29.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284442 s, 36.9 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:49:36 (1788641376) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:49:43 (1788641383) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000758131 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00082525 s, 1.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0016872 s, 607 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00381569 s, 268 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 16:49:52 (1788641392) create test files with size 15745250 .. 69c804ba960b36d49eb7e6b1f46f97bf - 401704bc68e022b07fec2b3ca6215829 - f0083422d33a5aa5e2ae7b2135036cd5 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg616-client.virtnet /mnt/lustre (opts:) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre PASS 37 (17s) == sanity-flr test 38: resync ============================ 16:50:09 (1788641409) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:50:20 (1788641420) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:50:28 (1788641428) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533424 s, 19.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xae:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244615 s, 42.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 16:50:36 (1788641436) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.107903 s, 19.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.2281 s, 18.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.176235 s, 11.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (12s) == sanity-flr test 42: lfs mirror verify ================= 16:50:48 (1788641448) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 16:50:52 (1788641452) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:50:56 (1788641456) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337609 s, 31.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 16:51:05 (1788641465) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.246554 s, 12.8 MB/s PASS 44a (8s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:51:13 (1788641473) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:51:23 (1788641483) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.283903 s, 36.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg616-client.virtnet /mnt/lustre (opts:) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:51:32 (1788641492) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff91b7c5e0b800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.26741 s, 39.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg616-client.virtnet /mnt/lustre (opts:) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff91b7c98a6800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:51:45 (1788641505) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:52:22 (1788641542) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:53:00 (1788641580) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 16:53:03 (1788641583) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.228345 s, 13.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.128841 s, 24.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x9b45da75 (nosync,stale) Mirror 2: 0x9b45da75 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 782e2050218d5ee54852623581d264a7 - mirror 1 checksum: ade1c10e083f0a5de7882f1471f6abf7 - mirror 2 checksum: 782e2050218d5ee54852623581d264a7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T16:53:07-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T16:53:07-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 782e2050218d5ee54852623581d264a7 - mirror 1 checksum: ade1c10e083f0a5de7882f1471f6abf7 - mirror 2 checksum: ade1c10e083f0a5de7882f1471f6abf7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T16:53:07-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T16:53:07-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 16:53:12 (1788641592) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355128 s, 29.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:53:21 (1788641601) Failing mds1 on oleg616-server Stopping /mnt/lustre-mds1 (opts:) on oleg616-server 16:53:36 (1788641616) shut down facet: mds1 facet_host: oleg616-server facet_failover_host: oleg616-server Failover mds1 to oleg616-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg616-server: oleg616-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-MDT0000 16:54:07 (1788641647) targets are mounted 16:54:07 (1788641647) facet_failover done oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0515091 s, 20.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:54:25 (1788641665) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0535907 s, 382 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0292535 s, 700 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:54:36 (1788641676) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0313066 s, 654 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0107523 s, 1.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:54:46 (1788641686) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00813946 s, 2.0 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:54:54 (1788641694) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0361284 s, 567 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:55:02 (1788641702) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00879338 s, 466 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0265285 s, 154 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:55:11 (1788641711) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:55:34 (1788641734) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:56:03 (1788641763) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:56:26 (1788641786) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:56:51 (1788641811) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 16:56:58 (1788641818) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57569 mirror split pid 57570 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57569 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 57570 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 16:58:07 (1788641887) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 1889 skipping zero length punch hole skipping zero length zero range truncating to largest ever: 0x1eca7 000003 1788641889.823525 trunc from 00000000 to 0x01eca6 (0x1eca7 bytes) 000006 1788641889.839486 trunc from 0x014460 to 0x016dc7 (0x2968 bytes) 000007 1788641889.844156 punch from 0x011de8 to 0x0170a1 (0x52ba bytes) fallocating to largest ever: 0x1f051 000008 1788641889.850400 fallocate 0x01b8f6 thru 0x01f050 (0x375b bytes) 000009 1788641889.855848 read 0x01abdf thru 0x01f050 (0x4472 bytes) 000012 1788641889.868472 fallocate 0x00093b thru 0x00a979 (0xa03f bytes) 000013 1788641889.879094 trunc from 0x01f051 to 0x017d09 (0x7347 bytes) 000014 1788641889.898105 trunc from 0x00f5f2 to 0x0167a6 (0x71b5 bytes) 000015 1788641889.903975 punch from 0x015073 to 0x0172bc (0x224a bytes) 000017 1788641889.910096 write 0x018039 thru 0x0244aa (0xc472 bytes) 000018 1788641889.912855 trunc from 0x001ee2 to 0x00f495 (0xd5b4 bytes) 000020 1788641889.917472 fallocate 0x0011ba thru 0x005407 (0x424e bytes) 000021 1788641889.922811 mapwrite 0x0318f0 thru 0x03c9ef (0xb100 bytes) 000022 1788641889.994761 mapread 0x02641e thru 0x033653 (0xd236 bytes) 000023 1788641890.004714 read 0x02fde9 thru 0x036ab9 (0x6cd1 bytes) 000024 1788641890.008971 MIRROR_EXTEND to 2 mirrors 000025 1788641890.737859 punch from 0x01ce42 to 0x027a02 (0xabc1 bytes) 000026 1788641890.784032 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) 000027 1788641890.984175 write 0x002a7b thru 0x00bc24 (0x91aa bytes) 000028 1788641891.005587 mapwrite 0x0340c6 thru 0x03da65 (0x99a0 bytes) zero_range to largest ever: 0x3bfbd 000029 1788641891.060885 trunc from 0x037850 to 0x03bfbc (0x476d bytes) 000030 1788641891.069000 write 0x03e0f8 thru 0x03ffff (0x1f08 bytes) 000031 1788641891.070299 write 0x02b895 thru 0x03b478 (0xfbe4 bytes) 000032 1788641891.079855 read 0x011d96 thru 0x01c811 (0xaa7c bytes) 000034 1788641891.088795 write 0x017270 thru 0x0247f4 (0xd585 bytes) 000035 1788641891.094103 mapwrite 0x00a586 thru 0x00f1fd (0x4c78 bytes) 000036 1788641891.116048 write 0x027be6 thru 0x02bf8b (0x43a6 bytes) 000037 1788641891.120829 mapwrite 0x004a71 thru 0x006121 (0x16b1 bytes) 000038 1788641891.136039 trunc from 0x040000 to 0x02b03b (0x14fc4 bytes) 000039 1788641891.152832 write 0x00893f thru 0x01109b (0x875d bytes) 000040 1788641891.154123 write 0x00ba45 thru 0x0122ae (0x686a bytes) 000044 1788641891.166470 read 0x01252c thru 0x0161b5 (0x3c8a bytes) 000045 1788641891.167171 trunc from 0x00e2b3 to 0x012a1e (0x476c bytes) 000046 1788641891.184915 read 0x021af5 thru 0x02b03b (0x9547 bytes) 000048 1788641891.196668 fallocate 0x020e4e thru 0x026eb9 (0x606c bytes) 000049 1788641891.200811 write 0x022654 thru 0x022a3e (0x3eb bytes) 000050 1788641891.201582 mapread 0x01f49f thru 0x028409 (0x8f6b bytes) 000051 1788641891.202431 trunc from 0x000333 to 0x003c06 (0x38d4 bytes) 000052 1788641891.211034 trunc from 0x01d383 to 0x02aa51 (0xd6cf bytes) 000053 1788641891.219067 mapwrite 0x005b80 thru 0x00c5da (0x6a5b bytes) 000054 1788641891.231901 trunc from 0x01c068 to 0x0218d4 (0x586d bytes) 000055 1788641891.236274 read 0x01b2f7 thru 0x0259f8 (0xa702 bytes) 000057 1788641891.247834 MIRROR_EXTEND to 2 mirrors 000058 1788641891.872984 trunc from 0x02b03c to 0x01c801 (0xe83a bytes) 000059 1788641892.002418 mapread 0x0175d6 thru 0x01c801 (0x522c bytes) 000060 1788641892.034441 MIRROR_RESYNC 2 mirrors 000062 1788641892.890053 mapwrite 0x004af3 thru 0x011c57 (0xd165 bytes) 000063 1788641893.034643 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1c802): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c802): Mirror 2: 0x38baba33 (stale) CRC-32 checksum value for chunk [0, 0x1c802): Mirror 1: 0x1f0a26df Mirror 3: 0x1f0a26df lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c802: skipped 000064 1788641894.015781 fallocate 0x0254bb thru 0x02a8b9 (0x53ff bytes) 000065 1788641894.044772 read 0x00260e thru 0x00695d (0x4350 bytes) 000066 1788641894.053848 punch from 0x023c86 to 0x0247d7 (0xb52 bytes) 000067 1788641894.058028 trunc from 0x028cd2 to 0x02a8b9 (0x1be8 bytes) 000068 1788641894.062818 mapread 0x01ef68 thru 0x01f244 (0x2dd bytes) 000069 1788641894.071236 mapwrite 0x02a39c thru 0x02c27d (0x1ee2 bytes) 000070 1788641894.119896 fallocate 0x02a09c thru 0x0345eb (0xa550 bytes) 000071 1788641894.124931 mapread 0x029f5b thru 0x02c27d (0x2323 bytes) 000072 1788641894.126118 punch from 0x002150 to 0x00eb81 (0xca32 bytes) 000073 1788641894.132221 trunc from 0x01e339 to 0x0278be (0x9586 bytes) 000074 1788641894.137067 mapread 0x01cb36 thru 0x022ab0 (0x5f7b bytes) 000075 1788641894.149583 fallocate 0x02d16b thru 0x03b0e6 (0xdf7c bytes) 000076 1788641894.154557 write 0x02848f thru 0x02e092 (0x5c04 bytes) 000077 1788641894.156909 punch from 0x00b345 to 0x00b609 (0x2c5 bytes) 000078 1788641894.161728 write 0x01a1c1 thru 0x029650 (0xf490 bytes) 000079 1788641894.164099 trunc from 0x02e093 to 0x03b520 (0xd48e bytes) 000080 1788641894.195187 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b521): Mirror 3: 0x9a420fb8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b521): Mirror 3: 0x9a420fb8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b521: skipped 000081 1788641894.534054 write 0x03c88b thru 0x03ffff (0x3775 bytes) 000082 1788641894.544993 punch from 0x013d97 to 0x01b069 (0x72d3 bytes) 000083 1788641894.559869 mapread 0x01ccb2 thru 0x0231e4 (0x6533 bytes) 000084 1788641894.598625 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x549fea43 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9a12bf4d Mirror 4: 0x9a12bf4d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000085 1788641895.534460 mapread 0x032279 thru 0x03ffff (0xdd87 bytes) 000086 1788641895.652951 trunc from 0x00c62b to 0x00ea16 (0x23ec bytes) 000088 1788641895.759736 mapread 0x018369 thru 0x023c0a (0xb8a2 bytes) 000089 1788641895.824374 trunc from 0x040000 to 0x018fd1 (0x2702e bytes) 000090 1788641895.910781 write 0x019fc3 thru 0x028855 (0xe893 bytes) 000091 1788641895.915392 write 0x035d31 thru 0x03acd1 (0x4fa1 bytes) 000093 1788641895.917351 mapwrite 0x026f45 thru 0x02e2c2 (0x737e bytes) 000094 1788641895.964855 punch from 0x01d524 to 0x02276b (0x5248 bytes) 000095 1788641895.972578 trunc from 0x03acd2 to 0x02e02b (0xcca6 bytes) 000096 1788641895.998534 MIRROR_RESYNC 3 mirrors 000097 1788641897.103193 trunc from 0x02e02c to 0x014db1 (0x1927a bytes) 000098 1788641897.142624 write 0x0320d6 thru 0x03810f (0x603a bytes) 000100 1788641897.145079 punch from 0x029119 to 0x02b398 (0x2280 bytes) 000101 1788641897.149647 trunc from 0x038110 to 0x01d7c3 (0x1a94c bytes) 000102 1788641897.169031 write 0x0064c6 thru 0x00f592 (0x90cd bytes) 000103 1788641897.179027 read 0x01972e thru 0x01d763 (0x4036 bytes) 000104 1788641897.187276 mapwrite 0x004509 thru 0x0113fb (0xcef3 bytes) 000105 1788641897.223403 trunc from 0x01d7c4 to 0x03862f (0x1ae6c bytes) 000106 1788641897.237487 mapread 0x0205c1 thru 0x030094 (0xfad4 bytes) 000107 1788641897.259017 mapwrite 0x03a824 thru 0x03ffff (0x57dc bytes) 000108 1788641897.300260 fallocate 0x00296e thru 0x01166b (0xecfe bytes) zero_range to largest ever: 0x40000 000110 1788641897.308180 trunc from 0x0334e8 to 0x03ffff (0xcb18 bytes) 000111 1788641897.312816 trunc from 0x02c65a to 0x0348b5 (0x825c bytes) 000112 1788641897.318223 punch from 0x03f6fa to 0x03ffff (0x906 bytes) 000113 1788641897.322771 punch from 0x020426 to 0x0260e8 (0x5cc3 bytes) 000114 1788641897.328298 read 0x037dca thru 0x03e4a5 (0x66dc bytes) 000115 1788641897.339292 mapwrite 0x02a45c thru 0x02dd0d (0x38b2 bytes) 000116 1788641897.358598 trunc from 0x040000 to 0x01a1eb (0x25e14 bytes) 000117 1788641897.378637 fallocate 0x01c595 thru 0x01eac9 (0x2535 bytes) 000118 1788641897.384903 trunc from 0x01a1ec to 0x008839 (0x119b2 bytes) 000119 1788641897.406087 trunc from 0x0049a1 to 0x007034 (0x2694 bytes) 000120 1788641897.412561 fallocate 0x01af80 thru 0x01c09f (0x1120 bytes) 000121 1788641897.417818 trunc from 0x01c0a0 to 0x010fec (0xb0b3 bytes) 000123 1788641897.435274 punch from 0x007af8 to 0x010fec (0x94f5 bytes) 000124 1788641897.441126 trunc from 0x00b3ad to 0x010fec (0x5c40 bytes) 000125 1788641897.444851 punch from 0x0089eb to 0x00b205 (0x281b bytes) 000126 1788641897.448963 fallocate 0x00c92f thru 0x0104a2 (0x3b74 bytes) 000127 1788641897.453003 write 0x028e46 thru 0x0380d1 (0xf28c bytes) 000128 1788641897.464847 mapwrite 0x028938 thru 0x0312bd (0x8986 bytes) 000129 1788641897.485408 punch from 0x028c78 to 0x02d2e5 (0x466e bytes) 000130 1788641897.490858 write 0x01cfcf thru 0x027dde (0xae10 bytes) 000131 1788641897.500661 trunc from 0x000913 to 0x0104e5 (0xfbd3 bytes) 000132 1788641897.506001 trunc from 0x0380d2 to 0x002a73 (0x3565e bytes) 000134 1788641897.526184 mapread 0x002196 thru 0x002a73 (0x8de bytes) 000135 1788641897.532649 trunc from 0x001a46 to 0x002a73 (0x102e bytes) 000137 1788641897.539860 read 0x0027ff thru 0x002a73 (0x275 bytes) 000139 1788641897.546158 fallocate 0x017eab thru 0x026e66 (0xefbc bytes) 000141 1788641897.551778 mapread 0x000a9d thru 0x002a73 (0x1fd7 bytes) 000142 1788641897.552650 write 0x008c6a thru 0x00b6ff (0x2a96 bytes) 000143 1788641897.555025 trunc from 0x00b700 to 0x00c2cb (0xbcc bytes) 000144 1788641897.570081 punch from 0x005b92 to 0x00c2cb (0x673a bytes) 000145 1788641897.585066 mapread 0x00a9a9 thru 0x00c2cb (0x1923 bytes) 000146 1788641897.593891 trunc from 0x00737a to 0x00c2cb (0x4f52 bytes) 000147 1788641897.601155 trunc from 0x006027 to 0x008352 (0x232c bytes) 000149 1788641897.612115 mapread 0x0057e9 thru 0x00c2cb (0x6ae3 bytes) 000150 1788641897.658264 read 0x00c180 thru 0x00c2cb (0x14c bytes) 000151 1788641897.659631 trunc from 0x005380 to 0x00c2cb (0x6f4c bytes) 000152 1788641897.665042 read 0x006b69 thru 0x00c2cb (0x5763 bytes) 000154 1788641897.675243 write 0x02cdd5 thru 0x0385d0 (0xb7fc bytes) 000155 1788641897.677386 mapread 0x00f028 thru 0x01714f (0x8128 bytes) 000156 1788641897.686793 trunc from 0x023db8 to 0x024fe4 (0x122d bytes) 000157 1788641897.692093 read 0x00f595 thru 0x016388 (0x6df4 bytes) 000158 1788641897.693020 mapread 0x006ca6 thru 0x00d4d0 (0x682b bytes) 000159 1788641897.695230 fallocate 0x00f877 thru 0x013822 (0x3fac bytes) 000160 1788641897.700076 punch from 0x028607 to 0x03566c (0xd066 bytes) 000161 1788641897.713848 trunc from 0x0385d1 to 0x023c4a (0x14986 bytes) 000162 1788641897.731026 mapread 0x0201a1 thru 0x023c4a (0x3aaa bytes) 000163 1788641897.739154 punch from 0x0217b5 to 0x023c4a (0x2496 bytes) 000164 1788641897.746871 write 0x028cf5 thru 0x038042 (0xf34e bytes) 000165 1788641897.751518 punch from 0x017938 to 0x01c4d0 (0x4b99 bytes) 000166 1788641897.755990 read 0x031c8d thru 0x031f40 (0x2b4 bytes) 000167 1788641897.756916 trunc from 0x016951 to 0x0263f6 (0xfaa6 bytes) 000168 1788641897.766625 trunc from 0x038043 to 0x03a8a8 (0x2866 bytes) 000169 1788641897.787488 write 0x022028 thru 0x025009 (0x2fe2 bytes) 000170 1788641897.798971 read 0x02b934 thru 0x038377 (0xca44 bytes) 000171 1788641897.799940 punch from 0x027c73 to 0x02adda (0x3168 bytes) 000172 1788641897.824016 mapwrite 0x027883 thru 0x031ccb (0xa449 bytes) 000173 1788641897.869291 write 0x01f3b4 thru 0x02a9a5 (0xb5f2 bytes) 000174 1788641897.879241 mapread 0x02f3f7 thru 0x03a8a8 (0xb4b2 bytes) 000175 1788641897.883625 fallocate 0x017146 thru 0x021e7a (0xad35 bytes) 000176 1788641897.889608 write 0x010a87 thru 0x01ae2a (0xa3a4 bytes) 000177 1788641897.915562 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a8a9): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a8a9): Mirror 3: 0x194afbfb (stale) Mirror 4: 0x194afbfb (stale) CRC-32 checksum value for chunk [0, 0x3a8a9): Mirror 1: 0x7d77f9a2 Mirror 5: 0x7d77f9a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a8a9: skipped 000178 1788641898.962809 punch from 0x0361d8 to 0x03a8a8 (0x46d1 bytes) 000179 1788641899.009689 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a8a9): Mirror 4: 0x194afbfb (stale) Mirror 5: 0x7d77f9a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a8a9): Mirror 4: 0x194afbfb (stale) Mirror 5: 0x7d77f9a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a8a9: skipped 000180 1788641899.319592 punch from 0x037e1a to 0x03a8a8 (0x2a8f bytes) 000181 1788641899.325075 trunc from 0x007d0d to 0x00ec88 (0x6f7c bytes) 000182 1788641899.334185 punch from 0x01f97d to 0x02c7e6 (0xce6a bytes) 000183 1788641899.355770 mapwrite 0x022496 thru 0x02fb1c (0xd687 bytes) 000185 1788641899.492752 fallocate 0x002cff thru 0x00ea7d (0xbd7f bytes) 000186 1788641899.508129 mapread 0x00c9c9 thru 0x0169b7 (0x9fef bytes) 000187 1788641899.548546 fallocate 0x02f609 thru 0x031ee4 (0x28dc bytes) 000188 1788641899.553140 fallocate 0x00e1b2 thru 0x01066a (0x24b9 bytes) 000189 1788641899.557850 mapwrite 0x02c9a1 thru 0x036e5d (0xa4bd bytes) 000190 1788641899.611915 trunc from 0x01ecd1 to 0x022ac8 (0x3df8 bytes) 000191 1788641899.621019 write 0x038896 thru 0x03c6c5 (0x3e30 bytes) 000192 1788641899.652608 trunc from 0x03c6c6 to 0x0106b7 (0x2c00e bytes) 000193 1788641899.697932 read 0x00eec2 thru 0x0106b7 (0x17f6 bytes) 000195 1788641899.703152 mapwrite 0x030989 thru 0x033283 (0x28fb bytes) 000196 1788641899.741913 punch from 0x01bf35 to 0x01bf53 (0x1f bytes) 000197 1788641899.746929 write 0x02035e thru 0x02f3f0 (0xf093 bytes) 000198 1788641899.753706 trunc from 0x027271 to 0x02af4e (0x3cde bytes) 000199 1788641899.767985 fallocate 0x02ab82 thru 0x0375fa (0xca79 bytes) 000200 1788641899.773684 punch from 0x0254cc to 0x0263a1 (0xed6 bytes) 000201 1788641899.778140 fallocate 0x01457e thru 0x02304a (0xeacd bytes) 000202 1788641899.783160 mapread 0x017ff1 thru 0x019a4c (0x1a5c bytes) 000203 1788641899.792009 mapwrite 0x00848b thru 0x00f74c (0x72c2 bytes) 000204 1788641899.807540 read 0x0284da thru 0x03072e (0x8255 bytes) 000205 1788641899.817409 fallocate 0x03796d thru 0x037eab (0x53f bytes) 000206 1788641899.825413 trunc from 0x028711 to 0x031157 (0x8a47 bytes) 000207 1788641899.837222 trunc from 0x0375fb to 0x02f749 (0x7eb1 bytes) 000208 1788641899.879135 read 0x01124c thru 0x01aaf6 (0x98ab bytes) 000209 1788641899.879831 mapwrite 0x00d857 thru 0x0139b1 (0x615b bytes) 000210 1788641899.912187 trunc from 0x012987 to 0x01fd6c (0xd3e6 bytes) 000211 1788641899.917428 read 0x008d42 thru 0x016fed (0xe2ac bytes) 000212 1788641899.946642 fallocate 0x02a25a thru 0x03141e (0x71c5 bytes) 000213 1788641899.959866 punch from 0x02b228 to 0x03141e (0x61f7 bytes) 000214 1788641899.967966 mapwrite 0x0352bb thru 0x03e956 (0x969c bytes) 000215 1788641900.032941 fallocate 0x008e96 thru 0x00afc7 (0x2132 bytes) 000216 1788641900.046221 read 0x02709a thru 0x027497 (0x3fe bytes) 000217 1788641900.046867 mapwrite 0x0012b4 thru 0x00825f (0x6fac bytes) 000218 1788641900.088433 read 0x0099f9 thru 0x00eec3 (0x54cb bytes) 000219 1788641900.089338 punch from 0x0233be to 0x02781b (0x445e bytes) 000221 1788641900.105015 mapread 0x003414 thru 0x011d1b (0xe908 bytes) 000222 1788641900.106272 punch from 0x03337f to 0x03b0a1 (0x7d23 bytes) 000223 1788641900.113008 trunc from 0x03e957 to 0x02c95d (0x11ff9 bytes) 000224 1788641900.132111 trunc from 0x01dab8 to 0x028cb5 (0xb1fe bytes) 000225 1788641900.140965 punch from 0x02c64e to 0x02c95d (0x310 bytes) 000226 1788641900.154624 mapwrite 0x001311 thru 0x00387e (0x256e bytes) 000228 1788641900.177910 mapread 0x01e4cd thru 0x0257a8 (0x72dc bytes) 000229 1788641900.187416 mapread 0x01758b thru 0x025b23 (0xe599 bytes) 000230 1788641900.188363 mapwrite 0x01e19e thru 0x026a7a (0x88dd bytes) 000231 1788641900.212840 read 0x02bb63 thru 0x02c95d (0xdfb bytes) 000235 1788641900.217718 mapwrite 0x01f0dd thru 0x020851 (0x1775 bytes) 000236 1788641900.235867 punch from 0x02c313 to 0x02c95d (0x64b bytes) 000237 1788641900.241814 write 0x015cf0 thru 0x01c63e (0x694f bytes) 000238 1788641900.248039 trunc from 0x0134b1 to 0x015788 (0x22d8 bytes) 000240 1788641900.269850 punch from 0x00c44a to 0x010299 (0x3e50 bytes) 000241 1788641900.274527 trunc from 0x01b21a to 0x01ea52 (0x3839 bytes) 000242 1788641900.278671 mapwrite 0x008d09 thru 0x012968 (0x9c60 bytes) 000243 1788641900.313582 mapwrite 0x0304cc thru 0x03b814 (0xb349 bytes) 000244 1788641900.354662 mapread 0x02c94c thru 0x0363cb (0x9a80 bytes) 000246 1788641900.356734 trunc from 0x03b815 to 0x021acc (0x19d48 bytes) 000247 1788641900.378867 fallocate 0x0398d4 thru 0x03ddf0 (0x451d bytes) 000248 1788641900.383547 trunc from 0x00c90a to 0x017288 (0xa97f bytes) 000249 1788641900.388926 write 0x021568 thru 0x02e9c3 (0xd45c bytes) 000250 1788641900.391931 write 0x023666 thru 0x031891 (0xe22c bytes) 000251 1788641900.393967 write 0x03f103 thru 0x03ffff (0xefd bytes) 000252 1788641900.396023 fallocate 0x022edd thru 0x026a24 (0x3b48 bytes) 000253 1788641900.400282 trunc from 0x00444a to 0x00831e (0x3ed5 bytes) 000254 1788641900.404819 punch from 0x034c05 to 0x03ffff (0xb3fb bytes) 000255 1788641900.421702 MIRROR_RESYNC 3 mirrors 000256 1788641902.065892 trunc from 0x040000 to 0x031a12 (0xe5ed bytes) 000257 1788641902.126780 write 0x013957 thru 0x01c7aa (0x8e54 bytes) 000258 1788641902.139651 punch from 0x010b42 to 0x0149d5 (0x3e94 bytes) 000259 1788641902.157518 mapread 0x010c5c thru 0x0158f7 (0x4c9c bytes) 000260 1788641902.175006 trunc from 0x031a13 to 0x02d519 (0x44f9 bytes) 000261 1788641902.192989 mapwrite 0x024cef thru 0x028d94 (0x40a6 bytes) 000262 1788641902.223134 punch from 0x007d9c to 0x012553 (0xa7b8 bytes) 000263 1788641902.244591 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d51a): Mirror 5: 0x108a87c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d51a): Mirror 5: 0x108a87c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d51a: skipped 000265 1788641902.711532 write 0x018a40 thru 0x025663 (0xcc24 bytes) 000266 1788641902.735933 mapwrite 0x00094c thru 0x00b598 (0xac4d bytes) 000268 1788641902.800214 read 0x0054eb thru 0x0096dd (0x41f3 bytes) 000269 1788641902.800842 trunc from 0x019ae4 to 0x01ad9a (0x12b7 bytes) 000270 1788641902.811031 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) 000271 1788641902.975947 write 0x03ea94 thru 0x03f4d4 (0xa41 bytes) 000272 1788641902.984902 trunc from 0x03f4d5 to 0x03a049 (0x548b bytes) 000273 1788641903.021477 write 0x01a989 thru 0x01b0e0 (0x758 bytes) 000274 1788641903.044226 trunc from 0x03a04a to 0x00ecf8 (0x2b351 bytes) 000275 1788641903.060769 fallocate 0x022ef2 thru 0x0243d0 (0x14df bytes) 000276 1788641903.072463 mapread 0x019f90 thru 0x01b02a (0x109b bytes) 000277 1788641903.132730 MIRROR_EXTEND to 2 mirrors 000278 1788641903.716894 mapwrite 0x02ab55 thru 0x031f46 (0x73f2 bytes) 000279 1788641903.817723 fallocate 0x027af4 thru 0x03284b (0xad58 bytes) 000280 1788641903.823466 mapwrite 0x00fef7 thru 0x014960 (0x4a6a bytes) 000281 1788641903.850194 trunc from 0x02b0e8 to 0x02f99c (0x48b5 bytes) 000282 1788641903.859028 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31f47): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31f47): Mirror 2: 0x9d95c90e (stale) CRC-32 checksum value for chunk [0, 0x31f47): Mirror 1: 0xda102035 Mirror 3: 0xda102035 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31f47: skipped 000283 1788641904.685663 trunc from 0x004c87 to 0x0129fd (0xdd77 bytes) 000284 1788641904.734861 mapread 0x01c09d thru 0x023445 (0x73a9 bytes) 000285 1788641904.766221 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31f47): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31f47): Mirror 2: 0x9d95c90e (stale) Mirror 3: 0xda102035 (stale) CRC-32 checksum value for chunk [0, 0x31f47): Mirror 1: 0x9584e643 Mirror 4: 0x9584e643 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31f47: skipped 000286 1788641905.749614 fallocate 0x03cdb9 thru 0x03d7e8 (0xa30 bytes) 000287 1788641905.783845 punch from 0x00f6c0 to 0x01dfaa (0xe8eb bytes) 000288 1788641905.791794 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d7e9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d7e9): Mirror 2: 0x44de2e5c (stale) Mirror 3: 0x888f9282 (stale) Mirror 4: 0x20877a6d (stale) CRC-32 checksum value for chunk [0, 0x3d7e9): Mirror 1: 0xae5cd3ab Mirror 5: 0xae5cd3ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d7e9: skipped 000289 1788641906.729732 read 0x037a8c thru 0x03d7e8 (0x5d5d bytes) 000290 1788641906.746484 mapread 0x00f3bd thru 0x015a28 (0x666c bytes) 000291 1788641906.783719 mapread 0x024bb5 thru 0x02ce13 (0x825f bytes) 000292 1788641906.808177 mapread 0x037bc5 thru 0x03d7e8 (0x5c24 bytes) 000293 1788641906.821044 trunc from 0x003c11 to 0x008693 (0x4a83 bytes) 000294 1788641906.861476 punch from 0x02dde7 to 0x0387f2 (0xaa0c bytes) 000295 1788641906.870686 trunc from 0x016260 to 0x017c56 (0x19f7 bytes) 000296 1788641906.875670 trunc from 0x01b36f to 0x028d26 (0xd9b8 bytes) 000297 1788641906.880221 read 0x00d951 thru 0x01cbcd (0xf27d bytes) 000298 1788641906.890786 trunc from 0x00dc8f to 0x018222 (0xa594 bytes) 000300 1788641906.896022 write 0x03fae7 thru 0x03ffff (0x519 bytes) 000301 1788641906.901660 read 0x00adbe thru 0x016aee (0xbd31 bytes) 000302 1788641906.909988 trunc from 0x0378df to 0x03a8c4 (0x2fe6 bytes) 000303 1788641906.914056 trunc from 0x040000 to 0x03f1f6 (0xe09 bytes) 000304 1788641906.929720 trunc from 0x02c260 to 0x036c12 (0xa9b3 bytes) 000305 1788641906.935460 punch from 0x02e338 to 0x03bb64 (0xd82d bytes) 000306 1788641906.940011 trunc from 0x03f1f7 to 0x034df0 (0xa406 bytes) 000307 1788641906.954526 fallocate 0x03a005 thru 0x03f050 (0x504c bytes) 000308 1788641906.958835 trunc from 0x034df1 to 0x0282bb (0xcb35 bytes) 000309 1788641906.972737 write 0x033917 thru 0x03ffff (0xc6e9 bytes) 000311 1788641906.976797 punch from 0x01f5f0 to 0x027bd6 (0x85e7 bytes) 000312 1788641906.981719 mapwrite 0x015eb9 thru 0x016f83 (0x10cb bytes) 000313 1788641906.996668 trunc from 0x005a2b to 0x00fee8 (0xa4be bytes) 000316 1788641907.002820 read 0x01ed3f thru 0x02539d (0x665f bytes) 000317 1788641907.010903 fallocate 0x030893 thru 0x03d677 (0xcde5 bytes) 000319 1788641907.015919 trunc from 0x035c48 to 0x03ffff (0xa3b8 bytes) 000320 1788641907.020348 read 0x03f6fb thru 0x03ffff (0x905 bytes) 000321 1788641907.028821 fallocate 0x02af9f thru 0x02e822 (0x3884 bytes) 000322 1788641907.033973 read 0x00d341 thru 0x018987 (0xb647 bytes) 000323 1788641907.034677 fallocate 0x03e193 thru 0x03ffff (0x1e6d bytes) 000324 1788641907.043140 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdf5a80ac (stale) Mirror 3: 0x92161e1c (stale) Mirror 4: 0x2b6c1ef9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdf5a80ac (stale) Mirror 3: 0x92161e1c (stale) Mirror 4: 0x2b6c1ef9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000325 1788641907.316880 punch from 0x023b1f to 0x02dba3 (0xa085 bytes) 000326 1788641907.324777 punch from 0x00de06 to 0x00e7a5 (0x9a0 bytes) 000327 1788641907.328473 mapread 0x01905f thru 0x022789 (0x972b bytes) 000328 1788641907.342866 mapwrite 0x03949b thru 0x03ffff (0x6b65 bytes) 000329 1788641907.382646 punch from 0x029237 to 0x02e027 (0x4df1 bytes) 000330 1788641907.388689 read 0x026ad4 thru 0x03242d (0xb95a bytes) 000331 1788641907.403027 read 0x03d700 thru 0x03ffff (0x2900 bytes) 000332 1788641907.403716 punch from 0x0019da to 0x005da5 (0x43cc bytes) 000333 1788641907.409242 trunc from 0x024db5 to 0x0344d1 (0xf71d bytes) 000334 1788641907.417945 mapwrite 0x0176d9 thru 0x02312e (0xba56 bytes) 000335 1788641907.445906 read 0x003d8e thru 0x00756a (0x37dd bytes) 000337 1788641907.455381 fallocate 0x00a744 thru 0x016d8c (0xc649 bytes) 000338 1788641907.459837 punch from 0x017e56 to 0x01c6dc (0x4887 bytes) 000339 1788641907.472190 mapwrite 0x0125a6 thru 0x019fc7 (0x7a22 bytes) 000340 1788641907.511790 write 0x03fce4 thru 0x03ffff (0x31c bytes) 000341 1788641907.517607 trunc from 0x01eb9d to 0x029ed0 (0xb334 bytes) 000343 1788641907.522921 mapwrite 0x016b0c thru 0x024976 (0xde6b bytes) 000344 1788641907.600164 trunc from 0x040000 to 0x021687 (0x1e978 bytes) 000345 1788641907.616374 mapread 0x014e84 thru 0x01ff3b (0xb0b8 bytes) 000346 1788641907.617801 punch from 0x0200cf to 0x021687 (0x15b9 bytes) 000347 1788641907.622833 punch from 0x0161cf to 0x01ddb1 (0x7be3 bytes) 000348 1788641907.627865 trunc from 0x019b1b to 0x01ce66 (0x334c bytes) 000350 1788641907.637220 read 0x0110aa thru 0x01eeab (0xde02 bytes) 000351 1788641907.646504 write 0x0121ac thru 0x0125a4 (0x3f9 bytes) 000352 1788641907.647604 write 0x014a42 thru 0x018d71 (0x4330 bytes) 000353 1788641907.650347 trunc from 0x01507e to 0x020eeb (0xbe6e bytes) 000354 1788641907.665651 read 0x017700 thru 0x021687 (0x9f88 bytes) 000355 1788641907.675941 fallocate 0x03baa8 thru 0x03ffff (0x4558 bytes) 000356 1788641907.681874 fallocate 0x013178 thru 0x022703 (0xf58c bytes) 000357 1788641907.687880 mapread 0x0112ba thru 0x018f36 (0x7c7d bytes) 000358 1788641907.695350 MIRROR_RESYNC 4 mirrors 000359 1788641908.857442 mapread 0x006a4b thru 0x00d931 (0x6ee7 bytes) 000360 1788641908.885590 MIRROR_RESYNC 4 mirrors 000361 1788641909.285641 mapwrite 0x026d97 thru 0x02e5e1 (0x784b bytes) 000362 1788641909.394650 trunc from 0x040000 to 0x031cff (0xe300 bytes) 000363 1788641909.426116 punch from 0x024df6 to 0x02ed9b (0x9fa6 bytes) 000364 1788641909.431719 trunc from 0x007149 to 0x0130c7 (0xbf7f bytes) 000365 1788641909.436067 mapread 0x011f37 thru 0x017665 (0x572f bytes) 000367 1788641909.446595 mapread 0x0077db thru 0x00ae04 (0x362a bytes) 000368 1788641909.454817 mapwrite 0x02602f thru 0x035ea8 (0xfe7a bytes) 000369 1788641909.499611 trunc from 0x035ea9 to 0x027e13 (0xe095 bytes) 000370 1788641909.513914 mapread 0x0055e9 thru 0x00c348 (0x6d60 bytes) 000373 1788641909.520823 MIRROR_RESYNC 4 mirrors 000374 1788641910.499096 mapwrite 0x03767c thru 0x03ffff (0x8984 bytes) 000375 1788641910.572149 write 0x013a32 thru 0x018778 (0x4d47 bytes) 000376 1788641910.581705 trunc from 0x040000 to 0x00b6a3 (0x3495c bytes) 000377 1788641910.598666 mapread 0x0005bc thru 0x0090f6 (0x8b3b bytes) 000378 1788641910.619154 fallocate 0x03fe80 thru 0x03ffff (0x180 bytes) 000380 1788641910.630201 write 0x002aca thru 0x0072fe (0x4835 bytes) 000381 1788641910.633343 fallocate 0x000cbe thru 0x000fca (0x30d bytes) 000382 1788641910.645994 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xb6a4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb6a4): Mirror 2: 0xdf3c694e (stale) Mirror 3: 0xdf3c694e (stale) Mirror 4: 0xdf3c694e (stale) CRC-32 checksum value for chunk [0, 0xb6a4): Mirror 1: 0x2fff9219 Mirror 5: 0x2fff9219 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb6a4: skipped 000383 1788641911.456045 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xb6a4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb6a4): Mirror 2: 0xdf3c694e (stale) Mirror 3: 0xdf3c694e (stale) Mirror 4: 0xdf3c694e (stale) CRC-32 checksum value for chunk [0, 0xb6a4): Mirror 1: 0x2fff9219 Mirror 5: 0x2fff9219 Mirror 6: 0x2fff9219 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb6a4: skipped 000384 1788641912.400481 fallocate 0x03241f thru 0x03dfe4 (0xbbc6 bytes) 000385 1788641912.458459 fallocate 0x027779 thru 0x033998 (0xc220 bytes) 000386 1788641912.463073 read 0x020170 thru 0x028771 (0x8602 bytes) 000387 1788641912.474982 trunc from 0x00502e to 0x00ec31 (0x9c04 bytes) 000388 1788641912.480208 read 0x015d3d thru 0x016a7d (0xd41 bytes) 000389 1788641912.490394 read 0x025ec9 thru 0x02f464 (0x959c bytes) 000390 1788641912.491016 trunc from 0x018749 to 0x019ae3 (0x139b bytes) 000391 1788641912.496543 mapwrite 0x011d8c thru 0x01ff42 (0xe1b7 bytes) 000392 1788641912.529104 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dfe5): Mirror 2: 0x7dfd5433 (stale) Mirror 3: 0x7dfd5433 (stale) Mirror 4: 0x7dfd5433 (stale) Mirror 5: 0x9387885a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dfe5): Mirror 2: 0x7dfd5433 (stale) Mirror 3: 0x7dfd5433 (stale) Mirror 4: 0x7dfd5433 (stale) Mirror 5: 0x9387885a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dfe5: skipped 000393 1788641912.869083 fallocate 0x01b1ad thru 0x01edb8 (0x3c0c bytes) 000394 1788641912.876188 read 0x031d7d thru 0x03852c (0x67b0 bytes) 000395 1788641912.894487 trunc from 0x034dc2 to 0x03c6ca (0x7909 bytes) 000396 1788641912.900666 write 0x01c169 thru 0x02a6d1 (0xe569 bytes) 000398 1788641912.929643 write 0x00c6ba thru 0x016855 (0xa19c bytes) 000399 1788641912.940567 trunc from 0x0093ef to 0x0176c2 (0xe2d4 bytes) 000400 1788641912.960507 trunc from 0x02e38a to 0x0333ce (0x5045 bytes) 000401 1788641912.968501 punch from 0x03b336 to 0x03dfe4 (0x2caf bytes) 000402 1788641912.973798 mapread 0x00f661 thru 0x011948 (0x22e8 bytes) 000403 1788641912.994237 fallocate 0x026c24 thru 0x02eaf1 (0x7ece bytes) 000404 1788641913.000115 trunc from 0x03c716 to 0x03dfe4 (0x18cf bytes) 000405 1788641913.004100 read 0x00fb78 thru 0x01f709 (0xfb92 bytes) 000406 1788641913.015651 mapread 0x0061b1 thru 0x01227b (0xc0cb bytes) 000407 1788641913.016846 mapwrite 0x02f906 thru 0x0335e4 (0x3cdf bytes) 000408 1788641913.040559 MIRROR_RESYNC 5 mirrors 000409 1788641914.738061 MIRROR_EXTEND to 6 mirrors 000411 1788641915.614622 write 0x0031ce thru 0x00ec33 (0xba66 bytes) 000412 1788641915.723680 trunc from 0x03dfe5 to 0x003953 (0x3a691 bytes) 000413 1788641915.774687 mapwrite 0x00d354 thru 0x0103c7 (0x3074 bytes) 000414 1788641915.859037 punch from 0x00fdc8 to 0x0103c7 (0x600 bytes) 000415 1788641915.866293 read 0x0059c4 thru 0x0103c7 (0xaa04 bytes) 000416 1788641915.872267 mapwrite 0x007035 thru 0x00e768 (0x7734 bytes) 000417 1788641915.905909 mapread 0x00a4d2 thru 0x00ca0a (0x2539 bytes) 000418 1788641915.906832 mapwrite 0x039449 thru 0x03dbc1 (0x4779 bytes) 000419 1788641915.950169 mapwrite 0x02cb4b thru 0x030440 (0x38f6 bytes) 000420 1788641915.977426 fallocate 0x02499e thru 0x0297ed (0x4e50 bytes) 000421 1788641915.989645 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3dbc2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dbc2): Mirror 2: 0xe019ed62 (stale) Mirror 3: 0xe019ed62 (stale) Mirror 4: 0xe019ed62 (stale) Mirror 5: 0xe019ed62 (stale) Mirror 6: 0xe019ed62 (stale) CRC-32 checksum value for chunk [0, 0x3dbc2): Mirror 1: 0xa1ddf099 Mirror 7: 0xa1ddf099 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dbc2: skipped 000422 1788641917.106163 write 0x006572 thru 0x00856d (0x1ffc bytes) 000423 1788641917.169885 fallocate 0x009d80 thru 0x017a0c (0xdc8d bytes) 000424 1788641917.176822 mapwrite 0x01d05f thru 0x029fc5 (0xcf67 bytes) 000425 1788641917.233647 trunc from 0x00b758 to 0x010eb8 (0x5761 bytes) 000426 1788641917.239859 trunc from 0x03dbc2 to 0x02efef (0xebd2 bytes) 000427 1788641917.262061 mapwrite 0x00fac9 thru 0x01dc11 (0xe149 bytes) 000429 1788641917.307953 fallocate 0x0131ca thru 0x02157b (0xe3b2 bytes) 000430 1788641917.318965 mapwrite 0x02bbec thru 0x03a9d7 (0xedec bytes) 000431 1788641917.394752 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3a9d8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a9d8): Mirror 2: 0xf2936f8f (stale) Mirror 3: 0xf2936f8f (stale) Mirror 4: 0xf2936f8f (stale) Mirror 5: 0xf2936f8f (stale) Mirror 6: 0xf2936f8f (stale) Mirror 7: 0xc680d2c6 (stale) CRC-32 checksum value for chunk [0, 0x3a9d8): Mirror 1: 0x58ee7604 Mirror 8: 0x58ee7604 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a9d8: skipped 000432 1788641918.362969 trunc from 0x01272b to 0x01289a (0x170 bytes) 000433 1788641918.452921 mapread 0x0337bb thru 0x034e94 (0x16da bytes) 000435 1788641918.471452 mapwrite 0x0180fc thru 0x01f91a (0x781f bytes) 000436 1788641918.506143 trunc from 0x03a9d8 to 0x00af0c (0x2facb bytes) 000437 1788641918.565711 trunc from 0x00af0d to 0x03958d (0x2e681 bytes) 000438 1788641918.586208 trunc from 0x03958e to 0x00d7f3 (0x2bd9a bytes) 000439 1788641918.608531 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xd7f4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd7f4): Mirror 2: 0xff614906 (stale) Mirror 3: 0xff614906 (stale) Mirror 4: 0xff614906 (stale) Mirror 5: 0xff614906 (stale) Mirror 6: 0xff614906 (stale) Mirror 7: 0xbda95f49 (stale) Mirror 8: 0x4c984be4 (stale) CRC-32 checksum value for chunk [0, 0xd7f4): Mirror 1: 0x27824128 Mirror 9: 0x27824128 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd7f4: skipped 000440 1788641919.522784 punch from 0x00b7cf to 0x00d7f3 (0x2025 bytes) 000441 1788641919.593823 trunc from 0x004832 to 0x00b0b5 (0x6884 bytes) 000442 1788641919.603426 fallocate 0x02650e thru 0x03042f (0x9f22 bytes) 000443 1788641919.610285 punch from 0x007cd6 to 0x00d7f3 (0x5b1e bytes) 000444 1788641919.621657 trunc from 0x00d7f4 to 0x03baf8 (0x2e305 bytes) 000445 1788641919.673812 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3baf9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3baf9): Mirror 2: 0x9f75a9b9 (stale) Mirror 3: 0x9f75a9b9 (stale) Mirror 4: 0x9f75a9b9 (stale) Mirror 5: 0x9f75a9b9 (stale) Mirror 6: 0x9f75a9b9 (stale) Mirror 7: 0x8671bb1f (stale) Mirror 8: 0x72460a69 (stale) Mirror 9: 0xacf4327c (stale) CRC-32 checksum value for chunk [0, 0x3baf9): Mirror 1: 0x62cb0e11 Mirror 10: 0x62cb0e11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3baf9: skipped 000446 1788641921.137932 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3baf9): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3baf9): Mirror 2: 0x9f75a9b9 (stale) Mirror 3: 0x9f75a9b9 (stale) Mirror 4: 0x9f75a9b9 (stale) Mirror 5: 0x9f75a9b9 (stale) Mirror 6: 0x9f75a9b9 (stale) Mirror 7: 0x8671bb1f (stale) Mirror 8: 0x72460a69 (stale) Mirror 9: 0xacf4327c (stale) CRC-32 checksum value for chunk [0, 0x3baf9): Mirror 1: 0x62cb0e11 Mirror 10: 0x62cb0e11 Mirror 11: 0x62cb0e11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3baf9: skipped 000447 1788641922.723763 trunc from 0x0045c8 to 0x006f6c (0x29a5 bytes) 000448 1788641922.785880 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3baf9): Mirror 2: 0x9f75a9b9 (stale) Mirror 3: 0x9f75a9b9 (stale) Mirror 4: 0x9f75a9b9 (stale) Mirror 5: 0x9f75a9b9 (stale) Mirror 6: 0x9f75a9b9 (stale) Mirror 7: 0x8671bb1f (stale) Mirror 8: 0x72460a69 (stale) Mirror 10: 0x62cb0e11 (stale) Mirror 11: 0x62cb0e11 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3baf9): Mirror 2: 0x9f75a9b9 (stale) Mirror 3: 0x9f75a9b9 (stale) Mirror 4: 0x9f75a9b9 (stale) Mirror 5: 0x9f75a9b9 (stale) Mirror 6: 0x9f75a9b9 (stale) Mirror 7: 0x8671bb1f (stale) Mirror 8: 0x72460a69 (stale) Mirror 10: 0x62cb0e11 (stale) Mirror 11: 0x62cb0e11 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3baf9: skipped 000450 1788641923.551951 trunc from 0x03baf9 to 0x01fa00 (0x1c0f8 bytes) 000451 1788641923.576132 read 0x0178fc thru 0x01fa00 (0x8105 bytes) 000452 1788641923.590660 write 0x0255d9 thru 0x02a30b (0x4d33 bytes) 000453 1788641923.595467 MIRROR_RESYNC 10 mirrors 000454 1788641925.581133 fallocate 0x026911 thru 0x035eda (0xf5ca bytes) 000455 1788641925.615512 punch from 0x01209b to 0x017016 (0x4f7c bytes) 000456 1788641925.621482 punch from 0x00b49f to 0x016e8f (0xb9f1 bytes) 000457 1788641925.633609 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2a30c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a30c): Mirror 2: 0x9cb5fd23 (stale) Mirror 3: 0x9cb5fd23 (stale) Mirror 4: 0x9cb5fd23 (stale) Mirror 5: 0x9cb5fd23 (stale) Mirror 6: 0x9cb5fd23 (stale) Mirror 7: 0x9cb5fd23 (stale) Mirror 8: 0x9cb5fd23 (stale) Mirror 10: 0x9cb5fd23 (stale) Mirror 11: 0x9cb5fd23 (stale) CRC-32 checksum value for chunk [0, 0x2a30c): Mirror 1: 0x9cb5fd23 Mirror 12: 0x9cb5fd23 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a30c: skipped 000458 1788641926.610382 trunc from 0x02a30c to 0x02501a (0x52f1 bytes) 000459 1788641926.672901 punch from 0x00e33b to 0x019e02 (0xbac8 bytes) 000460 1788641926.683131 mapread 0x00d1fc thru 0x01b2a3 (0xe0a8 bytes) 000461 1788641926.731804 write 0x03cc93 thru 0x03ffff (0x336d bytes) 000462 1788641926.735522 fallocate 0x00a5cc thru 0x00ce9d (0x28d2 bytes) 000463 1788641926.740570 trunc from 0x040000 to 0x028d10 (0x172ef bytes) 000465 1788641926.758903 trunc from 0x01a829 to 0x027cc3 (0xd49b bytes) 000466 1788641926.764341 fallocate 0x038ae1 thru 0x03ffff (0x751f bytes) 000467 1788641926.772062 read 0x033400 thru 0x03bbd3 (0x87d4 bytes) 000468 1788641926.781593 read 0x024ab3 thru 0x024f40 (0x48e bytes) 000469 1788641926.792285 read 0x0254e3 thru 0x028d43 (0x3861 bytes) 000470 1788641926.793586 mapwrite 0x03493a thru 0x03a960 (0x6027 bytes) 000472 1788641926.815536 fallocate 0x010c5d thru 0x020172 (0xf516 bytes) 000473 1788641926.820454 fallocate 0x014f2a thru 0x015750 (0x827 bytes) 000474 1788641926.829433 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb56a6aa5 (stale) Mirror 3: 0xb56a6aa5 (stale) Mirror 4: 0xb56a6aa5 (stale) Mirror 6: 0xb56a6aa5 (stale) Mirror 7: 0xb56a6aa5 (stale) Mirror 8: 0xb56a6aa5 (stale) Mirror 10: 0xb56a6aa5 (stale) Mirror 11: 0xb56a6aa5 (stale) Mirror 12: 0xb56a6aa5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb56a6aa5 (stale) Mirror 3: 0xb56a6aa5 (stale) Mirror 4: 0xb56a6aa5 (stale) Mirror 6: 0xb56a6aa5 (stale) Mirror 7: 0xb56a6aa5 (stale) Mirror 8: 0xb56a6aa5 (stale) Mirror 10: 0xb56a6aa5 (stale) Mirror 11: 0xb56a6aa5 (stale) Mirror 12: 0xb56a6aa5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000475 1788641927.535951 mapwrite 0x004cc6 thru 0x00ec1b (0x9f56 bytes) 000476 1788641927.585897 read 0x02d8df thru 0x0375a7 (0x9cc9 bytes) 000477 1788641927.594814 mapread 0x039990 thru 0x03a477 (0xae8 bytes) 000478 1788641927.604741 mapread 0x032761 thru 0x035b24 (0x33c4 bytes) 000479 1788641927.605691 read 0x03ccad thru 0x03fd6d (0x30c1 bytes) 000480 1788641927.610817 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb56a6aa5 (stale) Mirror 3: 0xb56a6aa5 (stale) Mirror 4: 0xb56a6aa5 (stale) Mirror 6: 0xb56a6aa5 (stale) Mirror 7: 0xb56a6aa5 (stale) Mirror 8: 0xb56a6aa5 (stale) Mirror 11: 0xb56a6aa5 (stale) Mirror 12: 0xb56a6aa5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb56a6aa5 (stale) Mirror 3: 0xb56a6aa5 (stale) Mirror 4: 0xb56a6aa5 (stale) Mirror 6: 0xb56a6aa5 (stale) Mirror 7: 0xb56a6aa5 (stale) Mirror 8: 0xb56a6aa5 (stale) Mirror 11: 0xb56a6aa5 (stale) Mirror 12: 0xb56a6aa5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000481 1788641928.089901 trunc from 0x040000 to 0x012b47 (0x2d4b8 bytes) 000482 1788641928.122131 read 0x009c3e thru 0x010591 (0x6954 bytes) 000483 1788641928.135233 fallocate 0x000a21 thru 0x009881 (0x8e61 bytes) 000484 1788641928.141734 mapwrite 0x016bd1 thru 0x01edb7 (0x81e7 bytes) 000486 1788641928.209944 trunc from 0x006565 to 0x00da90 (0x752c bytes) 000487 1788641928.218209 mapread 0x01d45f thru 0x01edb7 (0x1959 bytes) 000488 1788641928.229213 MIRROR_RESYNC 9 mirrors 000489 1788641930.422834 mapwrite 0x027019 thru 0x0303fe (0x93e6 bytes) 000490 1788641930.533850 fallocate 0x00b637 thru 0x01603a (0xaa04 bytes) 000491 1788641930.539992 mapread 0x02e56d thru 0x0303fe (0x1e92 bytes) 000492 1788641930.545902 trunc from 0x0303ff to 0x03fff4 (0xfbf6 bytes) 000493 1788641930.569013 punch from 0x0391b3 to 0x03dcbd (0x4b0b bytes) 000494 1788641930.575050 trunc from 0x03fff5 to 0x0094c7 (0x36b2d bytes) 000495 1788641930.592936 mapread 0x007173 thru 0x0094c7 (0x2355 bytes) 000496 1788641930.602405 mapread 0x001f81 thru 0x007410 (0x5490 bytes) 000497 1788641930.610669 MIRROR_RESYNC 9 mirrors 000498 1788641932.330977 punch from 0x003fef to 0x0094c7 (0x54d9 bytes) 000499 1788641932.367576 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x94c8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x94c8): Mirror 2: 0xd6d4627a (stale) Mirror 3: 0xd6d4627a (stale) Mirror 4: 0xd6d4627a (stale) Mirror 6: 0xd6d4627a (stale) Mirror 7: 0xd6d4627a (stale) Mirror 8: 0xd6d4627a (stale) Mirror 11: 0xd6d4627a (stale) Mirror 12: 0xd6d4627a (stale) CRC-32 checksum value for chunk [0, 0x94c8): Mirror 1: 0x1d130773 Mirror 13: 0x1d130773 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x94c8: skipped 000500 1788641933.799608 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x94c8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x94c8): Mirror 2: 0xd6d4627a (stale) Mirror 3: 0xd6d4627a (stale) Mirror 4: 0xd6d4627a (stale) Mirror 6: 0xd6d4627a (stale) Mirror 7: 0xd6d4627a (stale) Mirror 8: 0xd6d4627a (stale) Mirror 11: 0xd6d4627a (stale) Mirror 12: 0xd6d4627a (stale) CRC-32 checksum value for chunk [0, 0x94c8): Mirror 1: 0x1d130773 Mirror 13: 0x1d130773 Mirror 14: 0x1d130773 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x94c8: skipped 000501 1788641934.830969 fallocate 0x00bbfa thru 0x01112f (0x5536 bytes) 000502 1788641934.907938 mapread 0x004253 thru 0x008a46 (0x47f4 bytes) 000503 1788641934.934518 write 0x00ae07 thru 0x00f31f (0x4519 bytes) 000504 1788641934.957834 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11130): Mirror 3: 0xbf609fa9 (stale) Mirror 4: 0xbf609fa9 (stale) Mirror 6: 0xbf609fa9 (stale) Mirror 7: 0xbf609fa9 (stale) Mirror 8: 0xbf609fa9 (stale) Mirror 11: 0xbf609fa9 (stale) Mirror 12: 0xbf609fa9 (stale) Mirror 13: 0xa43148b6 (stale) Mirror 14: 0xa43148b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11130): Mirror 3: 0xbf609fa9 (stale) Mirror 4: 0xbf609fa9 (stale) Mirror 6: 0xbf609fa9 (stale) Mirror 7: 0xbf609fa9 (stale) Mirror 8: 0xbf609fa9 (stale) Mirror 11: 0xbf609fa9 (stale) Mirror 12: 0xbf609fa9 (stale) Mirror 13: 0xa43148b6 (stale) Mirror 14: 0xa43148b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11130: skipped 000505 1788641935.439889 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x11130): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11130): Mirror 3: 0xbf609fa9 (stale) Mirror 4: 0xbf609fa9 (stale) Mirror 6: 0xbf609fa9 (stale) Mirror 7: 0xbf609fa9 (stale) Mirror 8: 0xbf609fa9 (stale) Mirror 11: 0xbf609fa9 (stale) Mirror 12: 0xbf609fa9 (stale) Mirror 13: 0xa43148b6 (stale) Mirror 14: 0xa43148b6 (stale) CRC-32 checksum value for chunk [0, 0x11130): Mirror 1: 0x1252420d Mirror 15: 0x1252420d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11130: skipped 000506 1788641936.574022 trunc from 0x011130 to 0x01f51d (0xe3ee bytes) 000510 1788641936.678591 read 0x017743 thru 0x018e1e (0x16dc bytes) 000511 1788641936.688190 mapwrite 0x01de49 thru 0x0211fa (0x33b2 bytes) 000513 1788641936.764360 trunc from 0x0003e0 to 0x00bbb9 (0xb7da bytes) 000514 1788641936.781511 MIRROR_RESYNC 11 mirrors 000515 1788641939.750045 write 0x02ce7d thru 0x036f4b (0xa0cf bytes) 000516 1788641939.812882 punch from 0x012f3f to 0x0193a9 (0x646b bytes) 000517 1788641939.825665 mapwrite 0x002988 thru 0x010376 (0xd9ef bytes) 000518 1788641939.985939 write 0x00cea6 thru 0x017410 (0xa56b bytes) 000519 1788641939.994476 fallocate 0x006ebf thru 0x015892 (0xe9d4 bytes) 000520 1788641940.008218 write 0x00d0f7 thru 0x0178a7 (0xa7b1 bytes) 000521 1788641940.010088 read 0x03483f thru 0x034925 (0xe7 bytes) 000523 1788641940.011444 read 0x036299 thru 0x036f4b (0xcb3 bytes) 000524 1788641940.012695 trunc from 0x030d5c to 0x033fd8 (0x327d bytes) 000525 1788641940.025906 fallocate 0x007463 thru 0x016e0f (0xf9ad bytes) 000527 1788641940.060064 trunc from 0x036f4c to 0x01fd55 (0x171f6 bytes) 000528 1788641940.105929 fallocate 0x028089 thru 0x02c3f7 (0x436f bytes) 000529 1788641940.112954 mapwrite 0x027e44 thru 0x02b06b (0x3228 bytes) 000531 1788641940.206729 punch from 0x0248f7 to 0x02c3f7 (0x7b01 bytes) 000532 1788641940.217555 mapwrite 0x016e8a thru 0x023236 (0xc3ad bytes) 000533 1788641940.285917 punch from 0x01eff7 to 0x029ff5 (0xafff bytes) 000535 1788641940.299702 trunc from 0x02c3f8 to 0x01e168 (0xe28f bytes) 000536 1788641940.319614 mapread 0x00d24f thru 0x0160e1 (0x8e93 bytes) 000537 1788641940.331454 MIRROR_RESYNC 11 mirrors 000538 1788641942.894878 trunc from 0x01e169 to 0x032367 (0x141ff bytes) 000539 1788641942.983852 mapwrite 0x00f479 thru 0x01772a (0x82b2 bytes) 000540 1788641943.049142 mapwrite 0x00cf6f thru 0x0134eb (0x657d bytes) 000541 1788641943.093958 mapread 0x00383c thru 0x012fe0 (0xf7a5 bytes) 000542 1788641943.107694 fallocate 0x00aef4 thru 0x00d689 (0x2796 bytes) 000543 1788641943.121350 punch from 0x017eb0 to 0x01c641 (0x4792 bytes) 000545 1788641943.128588 fallocate 0x01a086 thru 0x02787e (0xd7f9 bytes) 000546 1788641943.136483 read 0x0010fa thru 0x007a4c (0x6953 bytes) 000547 1788641943.142158 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x32368): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32368): Mirror 3: 0x474f8701 (stale) Mirror 4: 0x474f8701 (stale) Mirror 6: 0x474f8701 (stale) Mirror 7: 0x474f8701 (stale) Mirror 8: 0x474f8701 (stale) Mirror 11: 0x474f8701 (stale) Mirror 12: 0x474f8701 (stale) Mirror 13: 0x474f8701 (stale) Mirror 14: 0x474f8701 (stale) Mirror 15: 0x474f8701 (stale) CRC-32 checksum value for chunk [0, 0x32368): Mirror 1: 0x1224ab20 Mirror 16: 0x1224ab20 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32368: skipped 000548 1788641944.859134 fallocate 0x0056f5 thru 0x00d7cd (0x80d9 bytes) 000549 1788641944.905573 punch from 0x001369 to 0x00305d (0x1cf5 bytes) 000550 1788641944.913698 trunc from 0x026cf0 to 0x02e856 (0x7b67 bytes) 000551 1788641944.920937 read 0x00347b thru 0x0058c1 (0x2447 bytes) 000552 1788641944.927806 punch from 0x029f72 to 0x03033d (0x63cc bytes) 000553 1788641944.932629 mapwrite 0x00acef thru 0x01a909 (0xfc1b bytes) 000554 1788641944.997661 punch from 0x00c8e9 to 0x00fd6a (0x3482 bytes) 000555 1788641945.028159 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x32368): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32368): Mirror 3: 0x474f8701 (stale) Mirror 4: 0x474f8701 (stale) Mirror 6: 0x474f8701 (stale) Mirror 7: 0x474f8701 (stale) Mirror 8: 0x474f8701 (stale) Mirror 11: 0x474f8701 (stale) Mirror 12: 0x474f8701 (stale) Mirror 13: 0x474f8701 (stale) Mirror 14: 0x474f8701 (stale) Mirror 15: 0x474f8701 (stale) Mirror 16: 0x1224ab20 (stale) CRC-32 checksum value for chunk [0, 0x32368): Mirror 1: 0x91150c7 Mirror 17: 0x91150c7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32368: skipped 000556 1788641946.375028 fallocate 0x00a359 thru 0x017590 (0xd238 bytes) 000557 1788641946.424857 trunc from 0x002335 to 0x00bd08 (0x99d4 bytes) 000558 1788641946.430573 write 0x02eaa4 thru 0x038dcb (0xa328 bytes) 000559 1788641946.442674 trunc from 0x02c309 to 0x035143 (0x8e3b bytes) 000560 1788641946.487861 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x38dcc): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38dcc): Mirror 3: 0xd609fe80 (stale) Mirror 4: 0xd609fe80 (stale) Mirror 6: 0xd609fe80 (stale) Mirror 7: 0xd609fe80 (stale) Mirror 8: 0xd609fe80 (stale) Mirror 11: 0xd609fe80 (stale) Mirror 12: 0xd609fe80 (stale) Mirror 13: 0xd609fe80 (stale) Mirror 14: 0xd609fe80 (stale) Mirror 15: 0xd609fe80 (stale) Mirror 16: 0x19dac90f (stale) Mirror 17: 0xa550e59e (stale) CRC-32 checksum value for chunk [0, 0x38dcc): Mirror 1: 0x157b318 Mirror 18: 0x157b318 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38dcc: skipped 000561 1788641948.711931 mapread 0x00ea6e thru 0x011942 (0x2ed5 bytes) 000562 1788641948.785461 mapread 0x0095a8 thru 0x00ecc9 (0x5722 bytes) 000563 1788641948.810556 trunc from 0x02b28f to 0x0365f1 (0xb363 bytes) 000565 1788641948.890354 mapread 0x00cc13 thru 0x00dd8e (0x117c bytes) 000566 1788641948.926456 write 0x02f62c thru 0x033eb6 (0x488b bytes) 000568 1788641948.959693 punch from 0x01bc36 to 0x023a3d (0x7e08 bytes) 000569 1788641948.969515 read 0x020e62 thru 0x023676 (0x2815 bytes) 000570 1788641948.974485 read 0x0385eb thru 0x038dcb (0x7e1 bytes) 000571 1788641949.022772 mapwrite 0x00086e thru 0x008dd7 (0x856a bytes) 000572 1788641949.074990 write 0x02c506 thru 0x0355ef (0x90ea bytes) 000573 1788641949.076185 punch from 0x01afd4 to 0x021433 (0x6460 bytes) 000574 1788641949.086993 mapwrite 0x0329c8 thru 0x0378dc (0x4f15 bytes) 000575 1788641949.151052 read 0x02115c thru 0x024e69 (0x3d0e bytes) 000576 1788641949.168118 trunc from 0x038dcc to 0x0139c9 (0x25402 bytes) 000577 1788641949.248715 write 0x03b4dc thru 0x03ffff (0x4b24 bytes) 000578 1788641949.280691 MIRROR_RESYNC 14 mirrors 000579 1788641954.300616 trunc from 0x040000 to 0x009363 (0x36c9c bytes) 000580 1788641954.390573 fallocate 0x01266b thru 0x0142c3 (0x1c59 bytes) 000581 1788641954.396033 trunc from 0x0142c4 to 0x00cd8f (0x7534 bytes) 000582 1788641954.427823 read 0x002dc1 thru 0x00b1dc (0x841c bytes) 000583 1788641954.442118 punch from 0x00c67e to 0x00cd8f (0x712 bytes) 000584 1788641954.462035 read 0x0037e6 thru 0x00b05f (0x787a bytes) 000585 1788641954.465900 trunc from 0x00cd90 to 0x005fa7 (0x6de8 bytes) 000586 1788641954.492702 read 0x003e79 thru 0x005fa7 (0x212f bytes) 000587 1788641954.496966 trunc from 0x005ecf to 0x005fa7 (0xd9 bytes) 000588 1788641954.506414 read 0x004037 thru 0x005fa7 (0x1f71 bytes) 000589 1788641954.522377 read 0x0041b0 thru 0x005fa7 (0x1df8 bytes) 000590 1788641954.528113 MIRROR_RESYNC 14 mirrors 000591 1788641955.780681 write 0x0005f4 thru 0x00dc3e (0xd64b bytes) 000592 1788641955.815381 punch from 0x0003ac to 0x005117 (0x4d6c bytes) 000593 1788641955.828988 mapwrite 0x039af9 thru 0x03fb17 (0x601f bytes) 000594 1788641955.885418 punch from 0x039639 to 0x03fb17 (0x64df bytes) 000595 1788641955.890777 mapwrite 0x03eb61 thru 0x03ffff (0x149f bytes) 000596 1788641955.928928 MIRROR_SPLIT mirror 3 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xaed840cd (stale) Mirror 4: 0xaed840cd (stale) Mirror 7: 0xaed840cd (stale) Mirror 8: 0xaed840cd (stale) Mirror 11: 0xaed840cd (stale) Mirror 12: 0xaed840cd (stale) Mirror 13: 0xaed840cd (stale) Mirror 14: 0xaed840cd (stale) Mirror 15: 0xaed840cd (stale) Mirror 16: 0xaed840cd (stale) Mirror 17: 0xaed840cd (stale) Mirror 18: 0xaed840cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xaed840cd (stale) Mirror 4: 0xaed840cd (stale) Mirror 7: 0xaed840cd (stale) Mirror 8: 0xaed840cd (stale) Mirror 11: 0xaed840cd (stale) Mirror 12: 0xaed840cd (stale) Mirror 13: 0xaed840cd (stale) Mirror 14: 0xaed840cd (stale) Mirror 15: 0xaed840cd (stale) Mirror 16: 0xaed840cd (stale) Mirror 17: 0xaed840cd (stale) Mirror 18: 0xaed840cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000597 1788641956.421835 mapread 0x01ba0e thru 0x026087 (0xa67a bytes) 000598 1788641956.453432 read 0x00d438 thru 0x00e151 (0xd1a bytes) 000599 1788641956.461127 trunc from 0x040000 to 0x037550 (0x8aaf bytes) 000600 1788641956.479822 trunc from 0x01d962 to 0x01f8e3 (0x1f82 bytes) 000601 1788641956.488714 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x37551): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37551): Mirror 3: 0x29f6bc41 (stale) Mirror 4: 0x29f6bc41 (stale) Mirror 7: 0x29f6bc41 (stale) Mirror 8: 0x29f6bc41 (stale) Mirror 11: 0x29f6bc41 (stale) Mirror 12: 0x29f6bc41 (stale) Mirror 13: 0x29f6bc41 (stale) Mirror 14: 0x29f6bc41 (stale) Mirror 15: 0x29f6bc41 (stale) Mirror 16: 0x29f6bc41 (stale) Mirror 17: 0x29f6bc41 (stale) Mirror 18: 0x29f6bc41 (stale) CRC-32 checksum value for chunk [0, 0x37551): Mirror 1: 0x1ea4ba8e Mirror 19: 0x1ea4ba8e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37551: skipped 000603 1788641957.381920 read 0x001839 thru 0x00c336 (0xaafe bytes) 000604 1788641957.397644 mapread 0x031227 thru 0x037550 (0x632a bytes) 000606 1788641957.465959 MIRROR_RESYNC 14 mirrors 000607 1788641960.568749 read 0x02d60b thru 0x037550 (0x9f46 bytes) 000608 1788641960.578821 read 0x01ff4b thru 0x02cae1 (0xcb97 bytes) 000609 1788641960.590815 MIRROR_EXTEND to 15 mirrors 000610 1788641961.698565 trunc from 0x037551 to 0x02f67e (0x7ed2 bytes) 000612 1788641961.782724 mapread 0x00eb1d thru 0x01d299 (0xe77d bytes) 000613 1788641961.807814 trunc from 0x01f3f3 to 0x02ab80 (0xb78e bytes) 000614 1788641961.812289 mapwrite 0x0114b6 thru 0x0178ac (0x63f7 bytes) 000615 1788641961.837461 fallocate 0x03dbcc thru 0x03ffff (0x2434 bytes) 000616 1788641961.841496 trunc from 0x027d05 to 0x034e83 (0xd17f bytes) 000617 1788641961.856275 fallocate 0x0176c5 thru 0x026799 (0xf0d5 bytes) 000618 1788641961.866229 mapread 0x0061f5 thru 0x00fe34 (0x9c40 bytes) 000619 1788641961.875814 punch from 0x02b4b3 to 0x02b4ef (0x3d bytes) 000620 1788641961.880300 trunc from 0x0389eb to 0x03dfd1 (0x55e7 bytes) 000621 1788641961.885214 mapwrite 0x004433 thru 0x011218 (0xcde6 bytes) 000622 1788641961.908492 write 0x033b6d thru 0x03ffff (0xc493 bytes) 000623 1788641961.915990 fallocate 0x00d21f thru 0x00e9f0 (0x17d2 bytes) 000624 1788641961.923399 punch from 0x00f109 to 0x011a2a (0x2922 bytes) 000625 1788641961.933242 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6abea55 (stale) Mirror 4: 0x6abea55 (stale) Mirror 7: 0x6abea55 (stale) Mirror 8: 0x6abea55 (stale) Mirror 11: 0x6abea55 (stale) Mirror 12: 0x6abea55 (stale) Mirror 13: 0x6abea55 (stale) Mirror 14: 0x6abea55 (stale) Mirror 15: 0x6abea55 (stale) Mirror 16: 0x6abea55 (stale) Mirror 17: 0x6abea55 (stale) Mirror 18: 0x6abea55 (stale) Mirror 19: 0x6abea55 (stale) Mirror 20: 0x6abea55 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2d4047c Mirror 21: 0x2d4047c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000627 1788641963.514219 mapwrite 0x00a82e thru 0x0163ae (0xbb81 bytes) 000628 1788641963.683786 trunc from 0x012df9 to 0x021f3d (0xf145 bytes) 000630 1788641963.693985 write 0x0313f1 thru 0x036215 (0x4e25 bytes) 000631 1788641963.727740 punch from 0x000878 to 0x010187 (0xf910 bytes) 000632 1788641963.738775 trunc from 0x040000 to 0x0275c1 (0x18a3e bytes) 000633 1788641963.785329 mapread 0x01bb31 thru 0x020d12 (0x51e2 bytes) 000634 1788641963.797605 mapwrite 0x0220d7 thru 0x031f28 (0xfe52 bytes) 000635 1788641963.903869 trunc from 0x031f29 to 0x004bd8 (0x2d350 bytes) 000636 1788641963.927045 MIRROR_RESYNC 16 mirrors 000637 1788641965.940886 mapwrite 0x0353b2 thru 0x03ffff (0xac4e bytes) 000639 1788641966.115410 write 0x03ff24 thru 0x03ffff (0xdc bytes) 000640 1788641966.116766 fallocate 0x01a4e7 thru 0x01d50d (0x3027 bytes) 000641 1788641966.121390 read 0x0089b6 thru 0x01882e (0xfe79 bytes) 000642 1788641966.132415 mapread 0x00596d thru 0x007e66 (0x24fa bytes) 000643 1788641966.140046 mapwrite 0x03d570 thru 0x03ffff (0x2a90 bytes) 000644 1788641966.151439 mapwrite 0x026e8b thru 0x02ea36 (0x7bac bytes) 000645 1788641966.174921 punch from 0x038d45 to 0x03ffff (0x72bb bytes) 000647 1788641966.181542 write 0x02bff4 thru 0x036e2d (0xae3a bytes) 000648 1788641966.183439 mapread 0x02c5ea thru 0x039f66 (0xd97d bytes) 000649 1788641966.191604 trunc from 0x040000 to 0x03322c (0xcdd3 bytes) 000650 1788641966.213471 punch from 0x01d384 to 0x0265c2 (0x923f bytes) 000651 1788641966.219021 trunc from 0x0274ef to 0x03322c (0xbd3e bytes) 000652 1788641966.235969 MIRROR_RESYNC 16 mirrors 000653 1788641969.456819 mapwrite 0x00a9b2 thru 0x010c63 (0x62b2 bytes) 000654 1788641969.594726 write 0x029e23 thru 0x037a51 (0xdc2f bytes) 000655 1788641969.606791 trunc from 0x037a52 to 0x0335c5 (0x448c bytes) 000656 1788641969.639242 mapwrite 0x023d8f thru 0x033962 (0xfbd4 bytes) 000657 1788641969.740794 write 0x03490a thru 0x0380de (0x37d5 bytes) 000658 1788641969.745986 trunc from 0x02dc32 to 0x032bc8 (0x4f97 bytes) 000659 1788641969.757956 read 0x035603 thru 0x0380de (0x2adc bytes) 000660 1788641969.758744 punch from 0x012c2f to 0x01ebd8 (0xbfaa bytes) 000661 1788641969.773868 read 0x01f8cb thru 0x021f55 (0x268b bytes) 000662 1788641969.774764 trunc from 0x0380df to 0x01a2d1 (0x1de0d bytes) 000663 1788641969.805580 mapwrite 0x00e056 thru 0x00e595 (0x540 bytes) 000664 1788641969.820527 fallocate 0x01f6a9 thru 0x02ba7b (0xc3d3 bytes) 000665 1788641969.825281 trunc from 0x01a2d2 to 0x02607c (0xbdab bytes) 000666 1788641969.842739 punch from 0x00f272 to 0x0195f3 (0xa382 bytes) 000667 1788641969.847708 mapread 0x013084 thru 0x01c4b2 (0x942f bytes) 000668 1788641969.857778 trunc from 0x007001 to 0x00a20e (0x320e bytes) 000670 1788641969.863284 mapread 0x0039f4 thru 0x00c0a0 (0x86ad bytes) 000671 1788641969.883336 fallocate 0x0210f7 thru 0x02221c (0x1126 bytes) 000672 1788641969.887660 punch from 0x021d74 to 0x02607c (0x4309 bytes) 000673 1788641969.891789 trunc from 0x016bc2 to 0x01e1fa (0x7639 bytes) 000674 1788641969.896773 trunc from 0x01a63b to 0x01af60 (0x926 bytes) 000675 1788641969.901483 punch from 0x022e80 to 0x02607c (0x31fd bytes) 000676 1788641969.907901 fallocate 0x02e757 thru 0x02e8d9 (0x183 bytes) 000677 1788641969.915560 MIRROR_RESYNC 16 mirrors 000678 1788641973.145070 trunc from 0x02e8da to 0x004d9a (0x29b3f bytes) 000679 1788641973.183128 mapwrite 0x0247ab thru 0x02a734 (0x5f8a bytes) 000680 1788641973.229531 write 0x00d221 thru 0x01416f (0x6f4f bytes) 000681 1788641973.238597 trunc from 0x021a8d to 0x02597e (0x3ef2 bytes) 000682 1788641973.243523 mapread 0x01e15d thru 0x0241e0 (0x6084 bytes) 000683 1788641973.255229 fallocate 0x0187a6 thru 0x01f335 (0x6b90 bytes) 000684 1788641973.259693 write 0x01b653 thru 0x0288f2 (0xd2a0 bytes) 000685 1788641973.262843 fallocate 0x000d05 thru 0x002f4a (0x2246 bytes) 000686 1788641973.268642 write 0x03ef82 thru 0x03ffff (0x107e bytes) 000687 1788641973.270602 punch from 0x000ef4 to 0x008263 (0x7370 bytes) 000688 1788641973.275362 write 0x013d45 thru 0x01fbe9 (0xbea5 bytes) 000690 1788641973.277872 trunc from 0x02de79 to 0x03cf50 (0xf0d8 bytes) 000691 1788641973.282002 trunc from 0x040000 to 0x03ea23 (0x15dc bytes) 000692 1788641973.297182 mapwrite 0x02eb58 thru 0x03e953 (0xfdfc bytes) 000693 1788641973.347390 trunc from 0x03ea24 to 0x03e82d (0x1f6 bytes) 000695 1788641973.366259 trunc from 0x00807e to 0x00ab70 (0x2af3 bytes) 000697 1788641973.371788 trunc from 0x023f00 to 0x02d17f (0x9280 bytes) 000698 1788641973.378359 mapwrite 0x012aa6 thru 0x01ca4d (0x9fa8 bytes) 000699 1788641973.408774 punch from 0x028eca to 0x02b739 (0x2870 bytes) 000700 1788641973.413929 mapwrite 0x00b67f thru 0x017939 (0xc2bb bytes) 000701 1788641973.455402 punch from 0x017d66 to 0x026124 (0xe3bf bytes) 000702 1788641973.460417 punch from 0x007fdd to 0x01619b (0xe1bf bytes) 000703 1788641973.465041 punch from 0x00d8f6 to 0x01438b (0x6a96 bytes) 000704 1788641973.469888 fallocate 0x0156de thru 0x018fba (0x38dd bytes) 000705 1788641973.474620 mapwrite 0x013036 thru 0x01cb3a (0x9b05 bytes) 000709 1788641973.512169 trunc from 0x03e82e to 0x0116b1 (0x2d17c bytes) 000711 1788641973.536539 read 0x00c008 thru 0x00d771 (0x176a bytes) 000712 1788641973.537487 mapwrite 0x03bbc7 thru 0x03ffff (0x4439 bytes) 000713 1788641973.575774 trunc from 0x03fdcc to 0x03ffff (0x234 bytes) 000714 1788641973.580776 trunc from 0x0194c7 to 0x028952 (0xf48c bytes) 000716 1788641973.586034 trunc from 0x040000 to 0x02c338 (0x13cc7 bytes) 000717 1788641973.601814 fallocate 0x01fc05 thru 0x0258ba (0x5cb6 bytes) 000718 1788641973.606698 trunc from 0x02c339 to 0x03a73f (0xe407 bytes) 000719 1788641973.619188 mapread 0x013bd2 thru 0x01fd94 (0xc1c3 bytes) 000720 1788641973.633699 fallocate 0x019dce thru 0x01c038 (0x226b bytes) 000721 1788641973.638291 trunc from 0x03a740 to 0x01fdbc (0x1a983 bytes) 000722 1788641973.657160 MIRROR_SPLIT mirror 2 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1fdbd): Mirror 3: 0xf867aae6 (stale) Mirror 7: 0xf867aae6 (stale) Mirror 8: 0xf867aae6 (stale) Mirror 11: 0xf867aae6 (stale) Mirror 12: 0xf867aae6 (stale) Mirror 13: 0xf867aae6 (stale) Mirror 14: 0xf867aae6 (stale) Mirror 15: 0xf867aae6 (stale) Mirror 16: 0xf867aae6 (stale) Mirror 17: 0xf867aae6 (stale) Mirror 18: 0xf867aae6 (stale) Mirror 19: 0xf867aae6 (stale) Mirror 20: 0xf867aae6 (stale) Mirror 21: 0xf867aae6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fdbd): Mirror 3: 0xf867aae6 (stale) Mirror 7: 0xf867aae6 (stale) Mirror 8: 0xf867aae6 (stale) Mirror 11: 0xf867aae6 (stale) Mirror 12: 0xf867aae6 (stale) Mirror 13: 0xf867aae6 (stale) Mirror 14: 0xf867aae6 (stale) Mirror 15: 0xf867aae6 (stale) Mirror 16: 0xf867aae6 (stale) Mirror 17: 0xf867aae6 (stale) Mirror 18: 0xf867aae6 (stale) Mirror 19: 0xf867aae6 (stale) Mirror 20: 0xf867aae6 (stale) Mirror 21: 0xf867aae6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fdbd: skipped 000723 1788641974.516028 fallocate 0x01ece7 thru 0x0264ae (0x77c8 bytes) 000724 1788641974.528806 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x264af): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x264af): Mirror 3: 0x548c8f6d (stale) Mirror 7: 0x548c8f6d (stale) Mirror 8: 0x548c8f6d (stale) Mirror 11: 0x548c8f6d (stale) Mirror 12: 0x548c8f6d (stale) Mirror 13: 0x548c8f6d (stale) Mirror 14: 0x548c8f6d (stale) Mirror 15: 0x548c8f6d (stale) Mirror 16: 0x548c8f6d (stale) Mirror 17: 0x548c8f6d (stale) Mirror 18: 0x548c8f6d (stale) Mirror 19: 0x548c8f6d (stale) Mirror 20: 0x548c8f6d (stale) Mirror 21: 0x548c8f6d (stale) CRC-32 checksum value for chunk [0, 0x264af): Mirror 1: 0x8b856706 Mirror 22: 0x8b856706 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x264af: skipped 000725 1788641975.877259 punch from 0x00d5e6 to 0x01a168 (0xcb83 bytes) 000727 1788641975.997185 MIRROR_RESYNC 16 mirrors 000728 1788641977.931438 trunc from 0x023a16 to 0x0264ae (0x2a99 bytes) 000729 1788641977.970451 mapwrite 0x01f7d9 thru 0x0273aa (0x7bd2 bytes) 000731 1788641978.023417 write 0x021236 thru 0x02443a (0x3205 bytes) 000733 1788641978.026584 trunc from 0x006f53 to 0x009300 (0x23ae bytes) 000734 1788641978.030517 read 0x023113 thru 0x0273aa (0x4298 bytes) 000736 1788641978.032254 mapread 0x015711 thru 0x017df5 (0x26e5 bytes) 000737 1788641978.041278 read 0x009ff1 thru 0x00da54 (0x3a64 bytes) 000738 1788641978.050426 write 0x0286ef thru 0x02a151 (0x1a63 bytes) 000739 1788641978.052478 mapread 0x009ec5 thru 0x01649c (0xc5d8 bytes) 000740 1788641978.053754 fallocate 0x0290b7 thru 0x032a2d (0x9977 bytes) 000741 1788641978.058856 fallocate 0x00a88a thru 0x0100f9 (0x5870 bytes) 000742 1788641978.064757 mapread 0x021c67 thru 0x028d55 (0x70ef bytes) 000743 1788641978.066543 write 0x03db24 thru 0x03ffff (0x24dc bytes) 000744 1788641978.067680 read 0x03ea9b thru 0x03ffff (0x1565 bytes) 000745 1788641978.069281 fallocate 0x019877 thru 0x0224dd (0x8c67 bytes) 000747 1788641978.075529 write 0x01be4b thru 0x023af9 (0x7caf bytes) 000748 1788641978.077528 trunc from 0x040000 to 0x005ef6 (0x3a109 bytes) 000749 1788641978.098249 read 0x0052b6 thru 0x005ef6 (0xc41 bytes) 000750 1788641978.098947 trunc from 0x005ef7 to 0x01a75c (0x14866 bytes) 000751 1788641978.116063 trunc from 0x01a75d to 0x03cd50 (0x225f4 bytes) 000752 1788641978.131190 write 0x008878 thru 0x014e8c (0xc615 bytes) 000753 1788641978.141835 write 0x00a6ec thru 0x00aad0 (0x3e5 bytes) 000754 1788641978.142545 trunc from 0x03cd51 to 0x004a01 (0x3834f bytes) 000755 1788641978.155885 write 0x00941e thru 0x00b569 (0x214c bytes) 000756 1788641978.156898 mapwrite 0x0158b3 thru 0x01c7c9 (0x6f17 bytes) 000757 1788641978.241427 mapread 0x01792e thru 0x0181e4 (0x8b7 bytes) 000758 1788641978.243553 fallocate 0x038ee8 thru 0x03ffff (0x7118 bytes) 000759 1788641978.253293 MIRROR_SPLIT mirror 12 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c7ca): Mirror 3: 0x64fea41d (stale) Mirror 7: 0x64fea41d (stale) Mirror 8: 0x64fea41d (stale) Mirror 11: 0x64fea41d (stale) Mirror 12: 0x64fea41d (stale) Mirror 13: 0x64fea41d (stale) Mirror 14: 0x64fea41d (stale) Mirror 15: 0x64fea41d (stale) Mirror 16: 0x64fea41d (stale) Mirror 17: 0x64fea41d (stale) Mirror 18: 0x64fea41d (stale) Mirror 20: 0x64fea41d (stale) Mirror 21: 0x64fea41d (stale) Mirror 22: 0x64fea41d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c7ca): Mirror 3: 0x64fea41d (stale) Mirror 7: 0x64fea41d (stale) Mirror 8: 0x64fea41d (stale) Mirror 11: 0x64fea41d (stale) Mirror 12: 0x64fea41d (stale) Mirror 13: 0x64fea41d (stale) Mirror 14: 0x64fea41d (stale) Mirror 15: 0x64fea41d (stale) Mirror 16: 0x64fea41d (stale) Mirror 17: 0x64fea41d (stale) Mirror 18: 0x64fea41d (stale) Mirror 20: 0x64fea41d (stale) Mirror 21: 0x64fea41d (stale) Mirror 22: 0x64fea41d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c7ca: skipped 000761 1788641978.804908 trunc from 0x01c7ca to 0x02a070 (0xd8a7 bytes) 000762 1788641978.829056 trunc from 0x01f39e to 0x0293ae (0xa011 bytes) 000763 1788641978.833191 mapread 0x010453 thru 0x010489 (0x37 bytes) 000764 1788641978.842195 read 0x0072c1 thru 0x013a12 (0xc752 bytes) 000765 1788641978.853570 MIRROR_RESYNC 15 mirrors 000766 1788641981.880434 write 0x008ac7 thru 0x016820 (0xdd5a bytes) 000767 1788641981.918197 read 0x006a2c thru 0x010fb0 (0xa585 bytes) 000768 1788641981.926991 read 0x0092a1 thru 0x018775 (0xf4d5 bytes) 000769 1788641981.947557 write 0x00a573 thru 0x013589 (0x9017 bytes) 000770 1788641981.949336 punch from 0x002061 to 0x00d96d (0xb90d bytes) 000771 1788641981.962109 trunc from 0x006b2a to 0x015329 (0xe800 bytes) 000772 1788641981.966778 punch from 0x026c9d to 0x02a070 (0x33d4 bytes) 000773 1788641981.971267 mapwrite 0x00c6bd thru 0x012f34 (0x6878 bytes) 000774 1788641982.014909 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x2a071): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a071): Mirror 3: 0x99a400e0 (stale) Mirror 7: 0x99a400e0 (stale) Mirror 8: 0x99a400e0 (stale) Mirror 11: 0x99a400e0 (stale) Mirror 12: 0x99a400e0 (stale) Mirror 13: 0x99a400e0 (stale) Mirror 14: 0x99a400e0 (stale) Mirror 15: 0x99a400e0 (stale) Mirror 16: 0x99a400e0 (stale) Mirror 17: 0x99a400e0 (stale) Mirror 18: 0x99a400e0 (stale) Mirror 20: 0x99a400e0 (stale) Mirror 21: 0x99a400e0 (stale) Mirror 22: 0x99a400e0 (stale) CRC-32 checksum value for chunk [0, 0x2a071): Mirror 1: 0x8772f780 Mirror 23: 0x8772f780 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a071: skipped 000775 1788641982.855124 MIRROR_RESYNC 16 mirrors 000776 1788641985.648540 write 0x014f08 thru 0x018843 (0x393c bytes) 000777 1788641985.698048 MIRROR_RESYNC 16 mirrors 000778 1788641988.417279 read 0x018f80 thru 0x020724 (0x77a5 bytes) 000779 1788641988.431049 MIRROR_SPLIT mirror 15 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) 000780 1788641989.159866 write 0x015f8e thru 0x0160cc (0x13f bytes) 000781 1788641989.246653 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a071): Mirror 7: 0x81efe8f8 (stale) Mirror 8: 0x81efe8f8 (stale) Mirror 11: 0x81efe8f8 (stale) Mirror 12: 0x81efe8f8 (stale) Mirror 13: 0x81efe8f8 (stale) Mirror 14: 0x81efe8f8 (stale) Mirror 15: 0x81efe8f8 (stale) Mirror 16: 0x81efe8f8 (stale) Mirror 17: 0x81efe8f8 (stale) Mirror 18: 0x81efe8f8 (stale) Mirror 20: 0x81efe8f8 (stale) Mirror 21: 0x81efe8f8 (stale) Mirror 22: 0x81efe8f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a071): Mirror 7: 0x81efe8f8 (stale) Mirror 8: 0x81efe8f8 (stale) Mirror 11: 0x81efe8f8 (stale) Mirror 12: 0x81efe8f8 (stale) Mirror 13: 0x81efe8f8 (stale) Mirror 14: 0x81efe8f8 (stale) Mirror 15: 0x81efe8f8 (stale) Mirror 16: 0x81efe8f8 (stale) Mirror 17: 0x81efe8f8 (stale) Mirror 18: 0x81efe8f8 (stale) Mirror 20: 0x81efe8f8 (stale) Mirror 21: 0x81efe8f8 (stale) Mirror 22: 0x81efe8f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a071: skipped 000782 1788641990.090556 write 0x01fd52 thru 0x025ab3 (0x5d62 bytes) 000783 1788641990.102927 punch from 0x027a0d to 0x02a070 (0x2664 bytes) 000784 1788641990.112673 fallocate 0x029c40 thru 0x035994 (0xbd55 bytes) 000785 1788641990.117219 trunc from 0x0145eb to 0x0167ad (0x21c3 bytes) 000786 1788641990.123245 punch from 0x028990 to 0x02a070 (0x16e1 bytes) 000787 1788641990.127545 fallocate 0x028352 thru 0x02972d (0x13dc bytes) 000788 1788641990.139455 punch from 0x019d2d to 0x0246a1 (0xa975 bytes) 000789 1788641990.160823 trunc from 0x02a071 to 0x021ded (0x8283 bytes) 000791 1788641990.184532 mapread 0x018691 thru 0x021ded (0x975d bytes) 000794 1788641990.203738 punch from 0x01bc7e to 0x01cb05 (0xe88 bytes) 000795 1788641990.208775 write 0x00ec11 thru 0x016ebb (0x82ab bytes) 000796 1788641990.214562 write 0x037e37 thru 0x03ffff (0x81c9 bytes) 000797 1788641990.215800 punch from 0x033ab7 to 0x03a5dc (0x6b26 bytes) 000800 1788641990.241098 trunc from 0x03b31c to 0x03ffff (0x4ce4 bytes) 000801 1788641990.249417 read 0x00228f thru 0x01105a (0xedcc bytes) 000802 1788641990.278632 write 0x00e415 thru 0x0191ec (0xadd8 bytes) 000803 1788641990.282684 trunc from 0x040000 to 0x005ac0 (0x3a53f bytes) 000804 1788641990.310043 mapwrite 0x01805a thru 0x0210d5 (0x907c bytes) 000805 1788641990.359740 MIRROR_RESYNC 14 mirrors 000806 1788641992.510223 trunc from 0x0210d6 to 0x0277d1 (0x66fc bytes) 000807 1788641992.546111 trunc from 0x0277d2 to 0x02bbe3 (0x4412 bytes) 000808 1788641992.565521 read 0x025672 thru 0x02bbe3 (0x6572 bytes) 000810 1788641992.576046 read 0x000059 thru 0x00ba0c (0xb9b4 bytes) 000811 1788641992.587160 mapread 0x00de92 thru 0x01c89d (0xea0c bytes) 000813 1788641992.589301 mapread 0x02a5e9 thru 0x02bbe3 (0x15fb bytes) 000814 1788641992.590283 mapwrite 0x022198 thru 0x0302be (0xe127 bytes) 000815 1788641992.630282 fallocate 0x02d5ca thru 0x030e1c (0x3853 bytes) 000816 1788641992.634715 read 0x01048b thru 0x01ef82 (0xeaf8 bytes) 000817 1788641992.635767 fallocate 0x00d5b3 thru 0x0128eb (0x5339 bytes) 000818 1788641992.639636 read 0x00161f thru 0x00df2a (0xc90c bytes) 000819 1788641992.640629 fallocate 0x01a2d9 thru 0x01c543 (0x226b bytes) 000820 1788641992.645112 fallocate 0x03c68a thru 0x03ffff (0x3976 bytes) 000821 1788641992.649698 trunc from 0x013d17 to 0x01f197 (0xb481 bytes) 000822 1788641992.654637 trunc from 0x030aec to 0x030e1c (0x331 bytes) 000823 1788641992.659512 punch from 0x00ba8c to 0x00e4a3 (0x2a18 bytes) 000824 1788641992.663410 punch from 0x0301fa to 0x030e1c (0xc23 bytes) 000825 1788641992.668134 fallocate 0x0046b1 thru 0x00c8e0 (0x8230 bytes) 000826 1788641992.672711 mapwrite 0x00e3f7 thru 0x016daa (0x89b4 bytes) 000827 1788641992.698563 read 0x01b637 thru 0x02199e (0x6368 bytes) 000828 1788641992.699277 read 0x000d7d thru 0x00ab14 (0x9d98 bytes) 000829 1788641992.700476 write 0x014a31 thru 0x01c63c (0x7c0c bytes) 000830 1788641992.702270 trunc from 0x030e1d to 0x0013a9 (0x2fa73 bytes) 000831 1788641992.717883 punch from 0x000ed8 to 0x0013a9 (0x4d2 bytes) 000832 1788641992.726369 trunc from 0x000aa6 to 0x0013a9 (0x904 bytes) 000833 1788641992.730028 mapwrite 0x02d987 thru 0x03b6ab (0xdd25 bytes) 000834 1788641992.773784 trunc from 0x025577 to 0x03201e (0xcaa8 bytes) 000835 1788641992.778505 trunc from 0x011cb8 to 0x012dcc (0x1115 bytes) 000836 1788641992.782832 trunc from 0x0267be to 0x032785 (0xbfc8 bytes) 000837 1788641992.790263 fallocate 0x024c48 thru 0x029dba (0x5173 bytes) 000838 1788641992.794775 mapwrite 0x00f7d1 thru 0x010f15 (0x1745 bytes) 000839 1788641992.806733 fallocate 0x01793f thru 0x01c05d (0x471f bytes) 000840 1788641992.816225 write 0x02e925 thru 0x035aa5 (0x7181 bytes) 000841 1788641992.822515 write 0x007d2b thru 0x016ca5 (0xef7b bytes) 000842 1788641992.825267 trunc from 0x029439 to 0x037df8 (0xe9c0 bytes) 000843 1788641992.843782 punch from 0x03818a to 0x03b6ab (0x3522 bytes) 000844 1788641992.850560 write 0x01d5de thru 0x025465 (0x7e88 bytes) 000846 1788641992.861482 trunc from 0x01024d to 0x01d8d7 (0xd68b bytes) 000847 1788641992.874047 mapread 0x01796b thru 0x017ebb (0x551 bytes) 000848 1788641992.879474 mapread 0x026348 thru 0x02b2aa (0x4f63 bytes) 000849 1788641992.910661 trunc from 0x03b6ac to 0x038dcb (0x28e0 bytes) 000850 1788641992.926220 mapwrite 0x0381f4 thru 0x03ffff (0x7e0c bytes) 000851 1788641992.991047 mapwrite 0x0145a1 thru 0x023315 (0xed75 bytes) 000852 1788641993.106066 fallocate 0x02689e thru 0x03154f (0xacb2 bytes) 000853 1788641993.111813 trunc from 0x00e3f2 to 0x010793 (0x23a2 bytes) 000854 1788641993.131286 fallocate 0x01ae9a thru 0x02322d (0x8394 bytes) 000855 1788641993.135467 mapwrite 0x032685 thru 0x038e04 (0x6780 bytes) 000856 1788641993.238157 fallocate 0x02ba3a thru 0x03a512 (0xead9 bytes) 000857 1788641993.244860 trunc from 0x036990 to 0x03daab (0x711c bytes) 000858 1788641993.259144 MIRROR_SPLIT mirror 7 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x9b296db3 (stale) Mirror 8: 0x9b296db3 (stale) Mirror 11: 0x9b296db3 (stale) Mirror 12: 0x9b296db3 (stale) Mirror 13: 0x9b296db3 (stale) Mirror 14: 0x9b296db3 (stale) Mirror 16: 0x9b296db3 (stale) Mirror 17: 0x9b296db3 (stale) Mirror 18: 0x9b296db3 (stale) Mirror 20: 0x9b296db3 (stale) Mirror 21: 0x9b296db3 (stale) Mirror 22: 0x9b296db3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x9b296db3 (stale) Mirror 8: 0x9b296db3 (stale) Mirror 11: 0x9b296db3 (stale) Mirror 12: 0x9b296db3 (stale) Mirror 13: 0x9b296db3 (stale) Mirror 14: 0x9b296db3 (stale) Mirror 16: 0x9b296db3 (stale) Mirror 17: 0x9b296db3 (stale) Mirror 18: 0x9b296db3 (stale) Mirror 20: 0x9b296db3 (stale) Mirror 21: 0x9b296db3 (stale) Mirror 22: 0x9b296db3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000859 1788641994.071714 trunc from 0x040000 to 0x02915d (0x16ea2 bytes) 000860 1788641994.096752 read 0x00a68a thru 0x010b43 (0x64ba bytes) 000861 1788641994.108664 trunc from 0x02915e to 0x000a1f (0x2873e bytes) 000862 1788641994.124215 trunc from 0x000a20 to 0x00221d (0x17fe bytes) 000863 1788641994.145475 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x221e): Mirror 7: 0x55b08982 (stale) Mirror 8: 0x55b08982 (stale) Mirror 11: 0x55b08982 (stale) Mirror 12: 0x55b08982 (stale) Mirror 13: 0x55b08982 (stale) Mirror 14: 0x55b08982 (stale) Mirror 16: 0x55b08982 (stale) Mirror 17: 0x55b08982 (stale) Mirror 18: 0x55b08982 (stale) Mirror 20: 0x55b08982 (stale) Mirror 21: 0x55b08982 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x221e): Mirror 7: 0x55b08982 (stale) Mirror 8: 0x55b08982 (stale) Mirror 11: 0x55b08982 (stale) Mirror 12: 0x55b08982 (stale) Mirror 13: 0x55b08982 (stale) Mirror 14: 0x55b08982 (stale) Mirror 16: 0x55b08982 (stale) Mirror 17: 0x55b08982 (stale) Mirror 18: 0x55b08982 (stale) Mirror 20: 0x55b08982 (stale) Mirror 21: 0x55b08982 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x221e: skipped 000864 1788641994.593875 read 0x001ac3 thru 0x00221d (0x75b bytes) 000865 1788641994.602622 mapwrite 0x000f75 thru 0x00cbff (0xbc8b bytes) 000866 1788641994.668417 punch from 0x0032e5 to 0x00cbff (0x991b bytes) 000867 1788641994.674134 write 0x03a10f thru 0x03a92c (0x81e bytes) 000868 1788641994.676103 mapread 0x005d26 thru 0x007019 (0x12f4 bytes) 000869 1788641994.687228 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a92d): Mirror 7: 0x7b79448d (stale) Mirror 8: 0x7b79448d (stale) Mirror 11: 0x7b79448d (stale) Mirror 12: 0x7b79448d (stale) Mirror 13: 0x7b79448d (stale) Mirror 14: 0x7b79448d (stale) Mirror 16: 0x7b79448d (stale) Mirror 17: 0x7b79448d (stale) Mirror 18: 0x7b79448d (stale) Mirror 20: 0x7b79448d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a92d): Mirror 7: 0x7b79448d (stale) Mirror 8: 0x7b79448d (stale) Mirror 11: 0x7b79448d (stale) Mirror 12: 0x7b79448d (stale) Mirror 13: 0x7b79448d (stale) Mirror 14: 0x7b79448d (stale) Mirror 16: 0x7b79448d (stale) Mirror 17: 0x7b79448d (stale) Mirror 18: 0x7b79448d (stale) Mirror 20: 0x7b79448d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a92d: skipped 000870 1788641995.260841 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3a92d): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a92d): Mirror 7: 0x7b79448d (stale) Mirror 8: 0x7b79448d (stale) Mirror 11: 0x7b79448d (stale) Mirror 12: 0x7b79448d (stale) Mirror 13: 0x7b79448d (stale) Mirror 14: 0x7b79448d (stale) Mirror 16: 0x7b79448d (stale) Mirror 17: 0x7b79448d (stale) Mirror 18: 0x7b79448d (stale) Mirror 20: 0x7b79448d (stale) CRC-32 checksum value for chunk [0, 0x3a92d): Mirror 1: 0x902756cb Mirror 21: 0x902756cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a92d: skipped 000871 1788641996.235160 MIRROR_RESYNC 12 mirrors 000872 1788641998.377990 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) 000874 1788641998.857273 fallocate 0x00d67e thru 0x011110 (0x3a93 bytes) 000875 1788641998.897906 write 0x03bf0c thru 0x03e283 (0x2378 bytes) 000876 1788641998.902797 punch from 0x02683f to 0x02ccce (0x6490 bytes) 000877 1788641998.907049 trunc from 0x03e284 to 0x03bbb2 (0x26d1 bytes) 000878 1788641998.932453 fallocate 0x018816 thru 0x01e509 (0x5cf4 bytes) 000879 1788641998.939629 mapwrite 0x02c3f7 thru 0x03a696 (0xe2a0 bytes) 000880 1788641998.984697 punch from 0x01d306 to 0x02438a (0x7085 bytes) 000881 1788641999.001242 MIRROR_RESYNC 11 mirrors 000882 1788642001.278083 fallocate 0x022855 thru 0x02497a (0x2126 bytes) 000883 1788642001.311739 mapread 0x02f827 thru 0x03862b (0x8e05 bytes) 000884 1788642001.327515 read 0x0031a2 thru 0x00663f (0x349e bytes) 000885 1788642001.336220 write 0x03d57f thru 0x03ffff (0x2a81 bytes) 000886 1788642001.337962 mapread 0x026b83 thru 0x035a84 (0xef02 bytes) 000887 1788642001.351975 fallocate 0x037342 thru 0x03ffff (0x8cbe bytes) 000888 1788642001.363006 mapwrite 0x023b1b thru 0x025c85 (0x216b bytes) 000889 1788642001.397155 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x619c48a3 (stale) Mirror 8: 0x619c48a3 (stale) Mirror 11: 0x619c48a3 (stale) Mirror 12: 0x619c48a3 (stale) Mirror 13: 0x619c48a3 (stale) Mirror 14: 0x619c48a3 (stale) Mirror 17: 0x619c48a3 (stale) Mirror 18: 0x619c48a3 (stale) Mirror 20: 0x619c48a3 (stale) Mirror 21: 0x619c48a3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfb430cf3 Mirror 22: 0xfb430cf3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000890 1788642002.408486 fallocate 0x025360 thru 0x0325f2 (0xd293 bytes) 000891 1788642002.449885 MIRROR_RESYNC 12 mirrors 000892 1788642006.082573 mapwrite 0x0057fb thru 0x0105c0 (0xadc6 bytes) 000893 1788642006.190083 mapread 0x026a08 thru 0x0354a2 (0xea9b bytes) 000894 1788642006.234551 write 0x0180f9 thru 0x020e07 (0x8d0f bytes) 000895 1788642006.250462 trunc from 0x031d2b to 0x03cf40 (0xb216 bytes) 000896 1788642006.266761 mapwrite 0x01fab4 thru 0x0270e6 (0x7633 bytes) 000897 1788642006.303781 read 0x030599 thru 0x0350ea (0x4b52 bytes) 000898 1788642006.314215 trunc from 0x040000 to 0x014bf0 (0x2b40f bytes) 000899 1788642006.344958 MIRROR_RESYNC 12 mirrors 000900 1788642008.901884 mapwrite 0x022ea7 thru 0x0284e6 (0x5640 bytes) 000901 1788642009.077474 trunc from 0x0284e7 to 0x01b0be (0xd428 bytes) 000902 1788642009.106132 write 0x019255 thru 0x021df5 (0x8ba1 bytes) 000903 1788642009.122551 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x21df6): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21df6): Mirror 7: 0x172d64c3 (stale) Mirror 8: 0x172d64c3 (stale) Mirror 11: 0x172d64c3 (stale) Mirror 12: 0x172d64c3 (stale) Mirror 13: 0x172d64c3 (stale) Mirror 14: 0x172d64c3 (stale) Mirror 17: 0x172d64c3 (stale) Mirror 18: 0x172d64c3 (stale) Mirror 20: 0x172d64c3 (stale) Mirror 21: 0x172d64c3 (stale) Mirror 22: 0x172d64c3 (stale) CRC-32 checksum value for chunk [0, 0x21df6): Mirror 1: 0x435a6d8a Mirror 23: 0x435a6d8a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21df6: skipped 000904 1788642010.736594 write 0x025584 thru 0x029bd2 (0x464f bytes) 000905 1788642010.818873 mapread 0x01f985 thru 0x029bd2 (0xa24e bytes) 000907 1788642010.846520 punch from 0x008585 to 0x017031 (0xeaad bytes) 000908 1788642010.854035 write 0x022db1 thru 0x02c1e1 (0x9431 bytes) 000909 1788642010.855602 trunc from 0x014528 to 0x023bd9 (0xf6b2 bytes) 000910 1788642010.883470 punch from 0x00b4fa to 0x00b979 (0x480 bytes) 000911 1788642010.892457 mapwrite 0x0017a4 thru 0x00b3d2 (0x9c2f bytes) 000914 1788642010.941152 mapwrite 0x00c06a thru 0x015369 (0x9300 bytes) 000915 1788642010.978118 read 0x0064fb thru 0x012405 (0xbf0b bytes) 000917 1788642010.980640 punch from 0x019dcf to 0x024966 (0xab98 bytes) 000918 1788642010.986382 trunc from 0x02c1e2 to 0x03f349 (0x13168 bytes) 000919 1788642011.014919 mapread 0x0079c7 thru 0x00ccf3 (0x532d bytes) 000920 1788642011.018783 read 0x038d34 thru 0x03f349 (0x6616 bytes) 000921 1788642011.036802 MIRROR_RESYNC 13 mirrors 000922 1788642013.689502 trunc from 0x03f34a to 0x023312 (0x1c037 bytes) 000923 1788642013.741103 punch from 0x0033d6 to 0x0128b3 (0xf4de bytes) 000924 1788642013.750372 mapread 0x005b3b thru 0x008f1e (0x33e4 bytes) 000925 1788642013.788098 read 0x006397 thru 0x009a7d (0x36e7 bytes) 000927 1788642013.789857 punch from 0x00b35f to 0x01679a (0xb43c bytes) 000928 1788642013.797509 mapwrite 0x020dc8 thru 0x0244dc (0x3715 bytes) 000929 1788642013.912809 trunc from 0x0244dd to 0x029c74 (0x5798 bytes) 000930 1788642013.952287 mapwrite 0x02e4d3 thru 0x031964 (0x3492 bytes) 000931 1788642014.120610 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x31965): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31965): Mirror 7: 0x139ec134 (stale) Mirror 8: 0x139ec134 (stale) Mirror 11: 0x139ec134 (stale) Mirror 12: 0x139ec134 (stale) Mirror 13: 0x139ec134 (stale) Mirror 14: 0x139ec134 (stale) Mirror 17: 0x139ec134 (stale) Mirror 18: 0x139ec134 (stale) Mirror 20: 0x139ec134 (stale) Mirror 21: 0x139ec134 (stale) Mirror 22: 0x139ec134 (stale) Mirror 23: 0x139ec134 (stale) CRC-32 checksum value for chunk [0, 0x31965): Mirror 1: 0xce63dccf Mirror 24: 0xce63dccf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31965: skipped 000933 1788642016.472097 mapread 0x0266a4 thru 0x02aa7d (0x43da bytes) 000934 1788642016.587008 trunc from 0x031965 to 0x016e9e (0x1aac6 bytes) 000935 1788642016.780001 write 0x01c515 thru 0x02310a (0x6bf6 bytes) 000936 1788642016.791599 mapwrite 0x01aa52 thru 0x020bf1 (0x61a0 bytes) 000937 1788642016.857267 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2310b): Mirror 7: 0xcb62118 (stale) Mirror 8: 0xcb62118 (stale) Mirror 11: 0xcb62118 (stale) Mirror 13: 0xcb62118 (stale) Mirror 14: 0xcb62118 (stale) Mirror 17: 0xcb62118 (stale) Mirror 18: 0xcb62118 (stale) Mirror 20: 0xcb62118 (stale) Mirror 21: 0xcb62118 (stale) Mirror 22: 0xcb62118 (stale) Mirror 23: 0xcb62118 (stale) Mirror 24: 0xb7b3d615 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2310b): Mirror 7: 0xcb62118 (stale) Mirror 8: 0xcb62118 (stale) Mirror 11: 0xcb62118 (stale) Mirror 13: 0xcb62118 (stale) Mirror 14: 0xcb62118 (stale) Mirror 17: 0xcb62118 (stale) Mirror 18: 0xcb62118 (stale) Mirror 20: 0xcb62118 (stale) Mirror 21: 0xcb62118 (stale) Mirror 22: 0xcb62118 (stale) Mirror 23: 0xcb62118 (stale) Mirror 24: 0xb7b3d615 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2310b: skipped 000938 1788642018.149024 mapwrite 0x0220de thru 0x02b4d7 (0x93fa bytes) 000939 1788642018.257240 mapread 0x0109c3 thru 0x019ac4 (0x9102 bytes) 000940 1788642018.293875 fallocate 0x00f0f8 thru 0x0183c3 (0x92cc bytes) 000941 1788642018.301785 mapwrite 0x009cb9 thru 0x01302f (0x9377 bytes) 000942 1788642018.355694 write 0x034da8 thru 0x039aa1 (0x4cfa bytes) 000944 1788642018.359158 mapwrite 0x00e00d thru 0x014d20 (0x6d14 bytes) 000945 1788642018.384171 read 0x030e50 thru 0x033c6a (0x2e1b bytes) 000946 1788642018.393734 punch from 0x02b586 to 0x02e3e5 (0x2e60 bytes) 000947 1788642018.410830 trunc from 0x039aa2 to 0x0082e6 (0x317bb bytes) 000948 1788642018.441780 mapread 0x0069d1 thru 0x0082e6 (0x1916 bytes) 000949 1788642018.476625 mapwrite 0x01e91e thru 0x02282c (0x3f0f bytes) 000951 1788642018.567461 trunc from 0x02282d to 0x03e20c (0x1b9e0 bytes) 000952 1788642018.588657 punch from 0x0288be to 0x031a00 (0x9143 bytes) 000953 1788642018.600133 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3e20d): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e20d): Mirror 7: 0x4bd89479 (stale) Mirror 8: 0x4bd89479 (stale) Mirror 11: 0x4bd89479 (stale) Mirror 13: 0x4bd89479 (stale) Mirror 14: 0x4bd89479 (stale) Mirror 17: 0x4bd89479 (stale) Mirror 18: 0x4bd89479 (stale) Mirror 20: 0x4bd89479 (stale) Mirror 21: 0x4bd89479 (stale) Mirror 22: 0x4bd89479 (stale) Mirror 23: 0x4bd89479 (stale) Mirror 24: 0x8259dd9d (stale) CRC-32 checksum value for chunk [0, 0x3e20d): Mirror 1: 0x297c1eac Mirror 25: 0x297c1eac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e20d: skipped 000954 1788642020.583170 fallocate 0x022c28 thru 0x02721b (0x45f4 bytes) 000955 1788642020.691729 fallocate 0x02f8d1 thru 0x035be4 (0x6314 bytes) 000956 1788642020.702274 write 0x02e137 thru 0x03449b (0x6365 bytes) 000958 1788642020.733789 punch from 0x039f80 to 0x03e20c (0x428d bytes) 000959 1788642020.739258 trunc from 0x03e20d to 0x0171d7 (0x27035 bytes) 000960 1788642020.776539 fallocate 0x03b790 thru 0x03d7ca (0x203b bytes) 000961 1788642020.786806 trunc from 0x03d7cb to 0x012f0c (0x2a8be bytes) 000962 1788642020.824520 trunc from 0x012f0d to 0x013254 (0x348 bytes) 000963 1788642020.875708 MIRROR_RESYNC 14 mirrors 000964 1788642023.824049 mapwrite 0x017600 thru 0x0258f1 (0xe2f2 bytes) 000965 1788642024.011955 read 0x009027 thru 0x00fe55 (0x6e2f bytes) 000966 1788642024.028799 mapread 0x009a28 thru 0x00a77a (0xd53 bytes) 000967 1788642024.029668 write 0x002a1f thru 0x00d21c (0xa7fe bytes) 000968 1788642024.040095 read 0x00908f thru 0x00eae5 (0x5a57 bytes) 000969 1788642024.042566 punch from 0x01158f to 0x013ca8 (0x271a bytes) 000970 1788642024.064759 read 0x0093ac thru 0x0155b7 (0xc20c bytes) 000971 1788642024.084450 MIRROR_RESYNC 14 mirrors 000972 1788642026.452792 trunc from 0x0258f2 to 0x016b03 (0xedee bytes) 000973 1788642026.509233 mapwrite 0x028469 thru 0x02ae91 (0x2a29 bytes) 000976 1788642026.570993 punch from 0x012df6 to 0x01d9e4 (0xabef bytes) 000977 1788642026.580046 MIRROR_RESYNC 14 mirrors 000978 1788642028.792501 fallocate 0x0332fb thru 0x03ffff (0xcd05 bytes) 000979 1788642028.861965 mapread 0x032bd6 thru 0x03ffff (0xd42a bytes) 000980 1788642028.900003 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x53bd2cc7 (stale) Mirror 8: 0x53bd2cc7 (stale) Mirror 11: 0x53bd2cc7 (stale) Mirror 13: 0x53bd2cc7 (stale) Mirror 14: 0x53bd2cc7 (stale) Mirror 17: 0x53bd2cc7 (stale) Mirror 18: 0x53bd2cc7 (stale) Mirror 20: 0x53bd2cc7 (stale) Mirror 21: 0x53bd2cc7 (stale) Mirror 22: 0x53bd2cc7 (stale) Mirror 23: 0x53bd2cc7 (stale) Mirror 24: 0x53bd2cc7 (stale) Mirror 25: 0x53bd2cc7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x53bd2cc7 Mirror 26: 0x53bd2cc7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000981 1788642030.328239 punch from 0x023b23 to 0x0267c7 (0x2ca5 bytes) 000982 1788642030.383531 fallocate 0x0075d1 thru 0x015a55 (0xe485 bytes) 000983 1788642030.394377 trunc from 0x040000 to 0x000488 (0x3fb77 bytes) 000984 1788642030.424097 mapread 0x000179 thru 0x000488 (0x310 bytes) 000985 1788642030.430385 punch from 0x00044f to 0x000488 (0x3a bytes) 000986 1788642030.435469 write 0x014ea5 thru 0x01713b (0x2297 bytes) 000987 1788642030.437359 mapwrite 0x00c4e5 thru 0x01498f (0x84ab bytes) 000988 1788642030.468285 mapwrite 0x00cb7e thru 0x00e2ee (0x1771 bytes) 000989 1788642030.480063 mapwrite 0x0035f8 thru 0x00d766 (0xa16f bytes) 000990 1788642030.515943 mapwrite 0x01c5de thru 0x01d129 (0xb4c bytes) 000991 1788642030.597381 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x1d12a): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d12a): Mirror 7: 0x11ae8aa5 (stale) Mirror 8: 0x11ae8aa5 (stale) Mirror 11: 0x11ae8aa5 (stale) Mirror 13: 0x11ae8aa5 (stale) Mirror 14: 0x11ae8aa5 (stale) Mirror 17: 0x11ae8aa5 (stale) Mirror 18: 0x11ae8aa5 (stale) Mirror 20: 0x11ae8aa5 (stale) Mirror 21: 0x11ae8aa5 (stale) Mirror 22: 0x11ae8aa5 (stale) Mirror 23: 0x11ae8aa5 (stale) Mirror 24: 0x11ae8aa5 (stale) Mirror 25: 0x11ae8aa5 (stale) Mirror 26: 0x11ae8aa5 (stale) CRC-32 checksum value for chunk [0, 0x1d12a): Mirror 1: 0x14c834ae Mirror 27: 0x14c834ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d12a: skipped 000992 1788642031.708985 mapread 0x00d4ab thru 0x00ef76 (0x1acc bytes) 000994 1788642031.743052 mapwrite 0x02dc1d thru 0x02e119 (0x4fd bytes) 000995 1788642031.840096 fallocate 0x030e97 thru 0x03e528 (0xd692 bytes) 000996 1788642031.849882 MIRROR_SPLIT mirror 8 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x194:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e529): Mirror 7: 0xb0637675 (stale) Mirror 8: 0xb0637675 (stale) Mirror 11: 0xb0637675 (stale) Mirror 13: 0xb0637675 (stale) Mirror 14: 0xb0637675 (stale) Mirror 17: 0xb0637675 (stale) Mirror 18: 0xb0637675 (stale) Mirror 21: 0xb0637675 (stale) Mirror 22: 0xb0637675 (stale) Mirror 23: 0xb0637675 (stale) Mirror 24: 0xb0637675 (stale) Mirror 25: 0xb0637675 (stale) Mirror 26: 0xb0637675 (stale) Mirror 27: 0x4f7d1773 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e529): Mirror 7: 0xb0637675 (stale) Mirror 8: 0xb0637675 (stale) Mirror 11: 0xb0637675 (stale) Mirror 13: 0xb0637675 (stale) Mirror 14: 0xb0637675 (stale) Mirror 17: 0xb0637675 (stale) Mirror 18: 0xb0637675 (stale) Mirror 21: 0xb0637675 (stale) Mirror 22: 0xb0637675 (stale) Mirror 23: 0xb0637675 (stale) Mirror 24: 0xb0637675 (stale) Mirror 25: 0xb0637675 (stale) Mirror 26: 0xb0637675 (stale) Mirror 27: 0x4f7d1773 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e529: skipped 000998 1788642032.541695 mapread 0x0348dd thru 0x03e528 (0x9c4c bytes) 001000 1788642032.600584 write 0x039e9c thru 0x03ffff (0x6164 bytes) All operations completed A-OK! PASS 70a (151s) == sanity-flr test 71: check mirror extend parent fid ==== 17:00:38 (1788642038) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1d2:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 17:00:48 (1788642048) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.272561 s, 7.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105954 s, 9.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 17:00:57 (1788642057) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre2 Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4828768 .. Extending file size to 7109888 .. Extending file size to 8066592 .. Extending file size to 6749280 .. Extending file size to 6926528 .. Extending file size to 7508192 .. Extending file size to 8337664 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8231296 .. Extending file size to 8612800 .. Extending file size to 8773728 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8759872 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 61403 61404 61405 61407 61408 done 192.168.206.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg616-client.virtnet /mnt/lustre2 (opts:) 192.168.206.116@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg616-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:02:20 (1788642140) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre2 Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4797216 .. Extending file size to 1896224 .. Extending file size to 7558880 .. Extending file size to 8143360 .. Extending file size to 8551392 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8784128 .. Extending file size to 8972800 .. Extending file size to 8532672 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8822432 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 63021 63022 63025 63026 63029 failed 192.168.206.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg616-client.virtnet /mnt/lustre2 (opts:) 192.168.206.116@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg616-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 17:03:36 (1788642216) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2388 1268916 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25016 3582004 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17552 3589468 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42568 7171472 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2240 1269064 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2388 1268916 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25016 3582004 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17552 3589468 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42568 7171472 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170117 s, 12.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.173099 s, 12.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x6ba79e18 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg616-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 17:03:55 (1788642235) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.166714 s, 12.6 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 17:04:03 (1788642243) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x63:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 17:04:11 (1788642251) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0672751 s, 31.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x239:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (6s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 17:04:18 (1788642258) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.136896 s, 15.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x110:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x111:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 17:04:25 (1788642265) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134218 s, 15.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:04:59 (1788642299) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg616-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134289 s, 7.8 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23c:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:05:52 (1788642352) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg616-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0691903 s, 30.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (52s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 17:06:43 (1788642403) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg616-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg616-server: Pool lustre.test_204f created Adding targets to pool oleg616-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg616-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.149273 s, 14.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x23e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg616-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg616-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg616-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 17:07:57 (1788642477) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 17:08:05 (1788642485) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 17:08:13 (1788642493) oleg616-server: Pool lustre.test_206 created Adding targets to pool oleg616-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg616-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg616-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg616-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg616-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 17:08:39 (1788642519) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0636353 s, 16.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324248 s, 32.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1899aca5 Mirror 3: 0x1899aca5 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 17:08:49 (1788642529) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 17:08:52 (1788642532) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 17:08:56 (1788642536) Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0482796 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xd30000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2560000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2560000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2560000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1c30000 Read this (4 bytes): BBBB 192.168.206.116@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg616-client.virtnet /mnt/lustre2 (opts:) PASS 209a (33s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 17:09:29 (1788642569) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0125739 s, 652 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 17:09:36 (1788642576) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056537 s, 18.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x80 lmm_fid: [0x200000405:0x80:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x11d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611264 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 17:09:38.000000000 -0400 Modify: 2026-09-05 17:09:38.000000000 -0400 Change: 2026-09-05 17:09:40.000000000 -0400 Birth: 2026-09-05 17:09:39.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 17:09:44 (1788642584) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531612 s, 19.7 MB/s oleg616-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg616-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg616-server: O/280000401/d26/13a: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 17:10:08 (1788642608) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.245287 s, 167 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.026711 s, 153 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 5 17:10 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x85:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 5 17:10 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 17:10:15 (1788642615) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 17:10:22 (1788642622) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1830 sec =========== 17:10:30 (1788642630) === sanity-flr: start cleanup 17:10:31 (1788642631) === === sanity-flr: finish cleanup 17:10:39 (1788642639) ===