-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 10:44:23 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg223-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 10:44:46 (1787841886) === oleg223-client.virtnet: executing check_config_client /mnt/lustre oleg223-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg223-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8c3e8851d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8c3e8851d000.idle_timeout=debug disable quota as required oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 10:45:40 (1787841940) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8232] == sanity-flr test 0a: lfs mirror create with -N option == 10:45:43 (1787841943) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:46:04 (1787841964) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:46:09 (1787841969) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:46:13 (1787841973) PASS 0d (30s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:46:43 (1787842003) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:46:47 (1787842007) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 10:46:51 (1787842011) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8c3e8851d000.stats=clear osc.lustre-OST0001-osc-ffff8c3e8851d000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:47:00 (1787842020) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:47:10 (1787842030) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:47:20 (1787842040) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:47:24 (1787842044) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:47:34 (1787842054) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 10:47:45 (1787842065) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:47:53 (1787842073) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:48:02 (1787842082) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:48:11 (1787842091) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:48:23 (1787842103) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:48:30 (1787842110) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.592302 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102442 s, 10.2 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:48:39 (1787842119) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.619367 s, 30.5 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:48:50 (1787842130) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:49:01 (1787842141) Stopping /mnt/lustre-ost1 (opts:) on oleg223-server oleg223-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c3e8851d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c3e8851d000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg223-server: oleg223-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg223-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg223-server oleg223-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c3e8851d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c3e8851d000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg223-server: oleg223-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg223-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (74s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:50:15 (1787842215) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.244638 s, 17.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg223-server oleg223-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c3e8851d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c3e8851d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg223-server: oleg223-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg223-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:51:01 (1787842261) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg223-server oleg223-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c3e8851d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c3e8851d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg223-server: oleg223-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg223-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg223-server oleg223-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c3e8851d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c3e8851d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg223-server: oleg223-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg223-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (84s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:52:25 (1787842345) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.38919 s, 15.5 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 10:52 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.296s user 0m0.000s sys 0m0.200s Stopping /mnt/lustre-ost1 (opts:) on oleg223-server oleg223-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8c3e8851d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8c3e8851d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.806s user 0m0.002s sys 0m0.155s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg223-server: oleg223-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg223-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg223-server oleg223-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8c3e8851d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c3e8851d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.147s user 0m0.000s sys 0m0.196s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg223-server: oleg223-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg223-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg223-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8c3e8851d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8c3e8851d000.ost_server_uuid in FULL state after 0 sec PASS 33b (96s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:54:01 (1787842441) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:54:04 (1787842444) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:54:08 (1787842448) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 10:54:11 (1787842451) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 10:54:20 (1787842460) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.609647 s, 20.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:54:33 (1787842473) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336651 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462711 s, 22.7 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:54:41 (1787842481) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:54:50 (1787842490) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0011547 s, 887 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00165497 s, 619 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00320473 s, 320 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.003904 s, 262 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 10:55:01 (1787842501) create test files with size 15757440 .. 997f199efd5890a565d22ba64e26197f - d26dc87bd9295dd14873809ee1516a3f - e51eba56ab257cb93d857261db058dda - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg223-client.virtnet /mnt/lustre (opts:) Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 10:55:27 (1787842527) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (21s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:55:48 (1787842548) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:55:56 (1787842556) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126786 s, 8.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244983 s, 42.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 10:56:07 (1787842567) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117231 s, 17.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.258667 s, 16.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.198904 s, 10.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 10:56:26 (1787842586) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 10:56:30 (1787842590) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:56:34 (1787842594) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564917 s, 18.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 43c: read prefer uncompressed mirror == 10:56:43 (1787842603) llite.lustre-ffff8c3e887b0800.enable_compression=1 striped dir -i1 -c2 -H crush /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (8s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 10:56:51 (1787842611) llite.lustre-ffff8c3e887b0800.enable_compression=1 striped dir -i1 -c2 -H crush /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735402 s, 14.3 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337475 s, 31.1 MB/s **verify components PASS 43d (10s) == sanity-flr test 44a: lfs mirror split check =========== 10:57:01 (1787842621) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.200992 s, 15.7 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:57:11 (1787842631) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xce:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:57:22 (1787842642) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.246968 s, 42.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x9:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg223-client.virtnet /mnt/lustre (opts:) Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:57:32 (1787842652) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8c3e8879c800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.318936 s, 32.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg223-client.virtnet /mnt/lustre (opts:) Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8c3e89f25000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:57:48 (1787842668) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:58:26 (1787842706) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (42s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:59:08 (1787842748) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 10:59:12 (1787842752) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.367256 s, 8.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.155881 s, 20.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xcd1efa61 (nosync,stale) Mirror 2: 0xcd1efa61 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 3dc8e131ac457d318940c420e2b4d050 - mirror 1 checksum: 50ba96ebbc5822cdd9da4d7991a6c03d - mirror 2 checksum: 3dc8e131ac457d318940c420e2b4d050 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T10:59:15-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T10:59:15-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 3dc8e131ac457d318940c420e2b4d050 - mirror 1 checksum: 50ba96ebbc5822cdd9da4d7991a6c03d - mirror 2 checksum: 50ba96ebbc5822cdd9da4d7991a6c03d - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T10:59:15-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T10:59:15-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 10:59:23 (1787842763) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677997 s, 15.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:59:31 (1787842771) Failing mds1 on oleg223-server Stopping /mnt/lustre-mds1 (opts:) on oleg223-server 10:59:43 (1787842783) shut down facet: mds1 facet_host: oleg223-server facet_failover_host: oleg223-server Failover mds1 to oleg223-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg223-server: oleg223-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg223-client: oleg223-server: ssh exited with exit code 1 Started lustre-MDT0000 11:00:17 (1787842817) targets are mounted 11:00:17 (1787842817) facet_failover done oleg223-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299094 s, 35.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 50A (66s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 11:00:37 (1787842837) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0453277 s, 452 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.019554 s, 1.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (12s) == sanity-flr test 50b: mirror rsync handles sparseness == 11:00:49 (1787842849) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0969205 s, 211 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0108856 s, 1.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 11:01:00 (1787842860) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00736083 s, 2.2 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 11:01:10 (1787842870) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0789335 s, 259 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 11:01:19 (1787842879) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0065303 s, 627 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0479306 s, 85.5 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 11:01:28 (1787842888) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 11:01:53 (1787842913) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 11:02:23 (1787842943) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 11:02:49 (1787842969) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 11:03:14 (1787842994) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 11:03:23 (1787843003) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 58864 mirror split pid 58865 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58864 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58865 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 11:04:32 (1787843072) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 3075 skipping zero length punch hole skipping zero length punch hole skipping zero size read skipping zero length zero range 000005 1787843076.273085 write 0x03bc16 thru 0x03ffff (0x43ea bytes) 000006 1787843076.290161 mapwrite 0x01926c thru 0x0256d1 (0xc466 bytes) 000007 1787843076.427919 mapread 0x02e5d0 thru 0x02e9c1 (0x3f2 bytes) zero_range to largest ever: 0x1e919 000009 1787843076.450538 trunc from 0x00f019 to 0x01e918 (0xf900 bytes) 000010 1787843076.458542 punch from 0x02c82a to 0x0344bf (0x7c96 bytes) 000011 1787843076.478060 MIRROR_EXTEND to 2 mirrors 000012 1787843077.973713 read 0x03a184 thru 0x03ffff (0x5e7c bytes) 000013 1787843077.986088 mapread 0x03435f thru 0x03efc6 (0xac68 bytes) 000014 1787843078.014459 write 0x03085c thru 0x03ed54 (0xe4f9 bytes) 000015 1787843078.129156 mapread 0x0157ed thru 0x018eda (0x36ee bytes) 000016 1787843078.145532 read 0x00e593 thru 0x013c8e (0x56fc bytes) 000018 1787843078.152076 mapread 0x020e5f thru 0x02ba00 (0xaba2 bytes) 000019 1787843078.198126 fallocate 0x009548 thru 0x00df5e (0x4a17 bytes) fallocating to largest ever: 0x2e1c2 000020 1787843078.217695 fallocate 0x01ed62 thru 0x02e1c1 (0xf460 bytes) 000021 1787843078.231146 trunc from 0x0274dd to 0x03092a (0x944e bytes) 000022 1787843078.279428 write 0x0059cf thru 0x01387e (0xdeb0 bytes) 000023 1787843078.294604 punch from 0x03bb9a to 0x03d1a6 (0x160d bytes) 000024 1787843078.306942 trunc from 0x040000 to 0x01f303 (0x20cfc bytes) 000025 1787843078.356821 write 0x0121ac thru 0x01302f (0xe84 bytes) 000026 1787843078.359705 mapread 0x00472a thru 0x00f59e (0xae75 bytes) 000027 1787843078.361379 trunc from 0x0197de to 0x01f303 (0x5b26 bytes) 000028 1787843078.369135 read 0x007c4f thru 0x013872 (0xbc24 bytes) 000029 1787843078.369820 mapwrite 0x010b93 thru 0x01670a (0x5b78 bytes) 000030 1787843078.414849 mapread 0x00651c thru 0x013b6e (0xd653 bytes) 000031 1787843078.418052 read 0x01574c thru 0x01f303 (0x9bb8 bytes) 000032 1787843078.431571 mapwrite 0x038bc4 thru 0x03ffff (0x743c bytes) 000033 1787843078.506571 mapread 0x035b90 thru 0x038fb9 (0x342a bytes) 000035 1787843078.510175 write 0x030355 thru 0x0371a5 (0x6e51 bytes) 000036 1787843078.516143 mapread 0x0336e1 thru 0x03ffff (0xc91f bytes) 000037 1787843078.517505 punch from 0x013736 to 0x013ae3 (0x3ae bytes) 000039 1787843078.524290 mapwrite 0x025791 thru 0x0309f7 (0xb267 bytes) 000040 1787843078.553861 read 0x0108fa thru 0x011287 (0x98e bytes) 000043 1787843078.556031 punch from 0x022d45 to 0x02ec92 (0xbf4e bytes) 000044 1787843078.563117 trunc from 0x00e95b to 0x01ddff (0xf4a5 bytes) 000046 1787843078.569066 mapread 0x03db1e thru 0x03ffff (0x24e2 bytes) 000047 1787843078.570305 trunc from 0x02d359 to 0x0306aa (0x3352 bytes) 000048 1787843078.574145 mapwrite 0x0198a7 thru 0x025bf0 (0xc34a bytes) 000049 1787843078.633813 fallocate 0x01a0a8 thru 0x01da0c (0x3965 bytes) 000050 1787843078.642458 trunc from 0x040000 to 0x015b67 (0x2a498 bytes) 000051 1787843078.662365 write 0x01a75d thru 0x01c872 (0x2116 bytes) 000052 1787843078.664886 trunc from 0x010785 to 0x0180ff (0x797b bytes) 000053 1787843078.671892 mapwrite 0x00f3d7 thru 0x0168a1 (0x74cb bytes) 000054 1787843078.736417 write 0x01d2c0 thru 0x028b84 (0xb8c5 bytes) 000055 1787843078.738028 mapwrite 0x014d02 thru 0x016838 (0x1b37 bytes) 000056 1787843078.765933 read 0x0039b0 thru 0x009025 (0x5676 bytes) 000057 1787843078.772181 MIRROR_RESYNC 2 mirrors 000058 1787843079.682349 MIRROR_RESYNC 2 mirrors fallocating to largest ever: 0x40000 000059 1787843080.225054 fallocate 0x039ab5 thru 0x03ffff (0x654b bytes) 000060 1787843080.322595 punch from 0x020c95 to 0x02243d (0x17a9 bytes) 000062 1787843080.331136 mapwrite 0x015a1e thru 0x015f79 (0x55c bytes) 000063 1787843080.355339 punch from 0x009d04 to 0x010370 (0x666d bytes) 000064 1787843080.361821 write 0x00a823 thru 0x00f5db (0x4db9 bytes) 000065 1787843080.383340 mapwrite 0x031f4d thru 0x03be83 (0x9f37 bytes) 000066 1787843080.510559 read 0x0006ad thru 0x003180 (0x2ad4 bytes) 000067 1787843080.522858 trunc from 0x03be84 to 0x00af08 (0x30f7b bytes) 000069 1787843080.567358 trunc from 0x004593 to 0x007daf (0x381d bytes) 000070 1787843080.578066 fallocate 0x0052b5 thru 0x009cc7 (0x4a13 bytes) 000071 1787843080.598147 trunc from 0x007376 to 0x00af08 (0x3b93 bytes) 000072 1787843080.617165 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) 000073 1787843080.938853 mapread 0x001e42 thru 0x005af1 (0x3cb0 bytes) 000074 1787843080.989749 mapwrite 0x03743f thru 0x03ae13 (0x39d5 bytes) 000075 1787843081.131085 mapwrite 0x024a76 thru 0x02df2a (0x94b5 bytes) 000076 1787843081.216556 trunc from 0x03ae14 to 0x0365fa (0x4819 bytes) 000077 1787843081.233873 read 0x02d4fd thru 0x02d527 (0x2b bytes) 000078 1787843081.234502 read 0x011941 thru 0x01ebef (0xd2af bytes) 000079 1787843081.248782 mapwrite 0x032a99 thru 0x0348bf (0x1e27 bytes) 000080 1787843081.280756 trunc from 0x01dc21 to 0x02b6b8 (0xda98 bytes) 000081 1787843081.305580 mapread 0x002333 thru 0x00ab0b (0x87d9 bytes) 000083 1787843081.308545 mapwrite 0x03ab1a thru 0x03ffff (0x54e6 bytes) 000084 1787843081.401895 trunc from 0x03473b to 0x03ffff (0xb8c5 bytes) 000085 1787843081.409184 mapwrite 0x03a34a thru 0x03ffff (0x5cb6 bytes) 000086 1787843081.518661 read 0x03f3d0 thru 0x03ffff (0xc30 bytes) 000087 1787843081.520645 write 0x03a326 thru 0x03ffff (0x5cda bytes) 000088 1787843081.522773 mapread 0x02a8b5 thru 0x02be8f (0x15db bytes) 000089 1787843081.544294 fallocate 0x03605d thru 0x03ffff (0x9fa3 bytes) 000090 1787843081.551535 trunc from 0x040000 to 0x01df95 (0x2206a bytes) 000091 1787843081.598797 write 0x00db1d thru 0x0148a9 (0x6d8d bytes) 000093 1787843081.604062 punch from 0x017de4 to 0x019e2d (0x204a bytes) 000096 1787843081.616572 punch from 0x005270 to 0x0134f8 (0xe289 bytes) 000097 1787843081.645878 read 0x00e5d0 thru 0x00f2f4 (0xd25 bytes) 000098 1787843081.689593 trunc from 0x01df96 to 0x0387dc (0x1a847 bytes) 000099 1787843081.713597 trunc from 0x0387dd to 0x03d650 (0x4e74 bytes) 000103 1787843081.760484 write 0x00a944 thru 0x014cb7 (0xa374 bytes) 000104 1787843081.764527 read 0x005870 thru 0x0058d1 (0x62 bytes) 000105 1787843081.766781 trunc from 0x03d651 to 0x0017fe (0x3be52 bytes) 000106 1787843081.789782 read 0x0004cf thru 0x0017fe (0x1330 bytes) 000107 1787843081.790438 punch from 0x000581 to 0x0017fe (0x127e bytes) 000108 1787843081.800461 mapwrite 0x015fed thru 0x02232c (0xc340 bytes) 000109 1787843081.937241 punch from 0x00a5fa to 0x013afd (0x9504 bytes) 000110 1787843081.943516 trunc from 0x0152a3 to 0x02232c (0xd08a bytes) 000111 1787843081.959355 punch from 0x01d1fc to 0x02232c (0x5131 bytes) 000112 1787843081.977248 write 0x018faf thru 0x023dc4 (0xae16 bytes) 000113 1787843081.990047 read 0x012e89 thru 0x017673 (0x47eb bytes) 000115 1787843082.038582 trunc from 0x001dc9 to 0x00c0d5 (0xa30d bytes) 000116 1787843082.067500 trunc from 0x019900 to 0x01ce9a (0x359b bytes) 000117 1787843082.099851 trunc from 0x023dc5 to 0x0119dd (0x123e7 bytes) 000118 1787843082.120043 trunc from 0x01114b to 0x0119dd (0x893 bytes) 000119 1787843082.141591 mapread 0x0007e7 thru 0x0090ef (0x8909 bytes) 000120 1787843082.217578 write 0x0059e1 thru 0x00e999 (0x8fb9 bytes) 000121 1787843082.227960 MIRROR_EXTEND to 2 mirrors 000122 1787843083.053035 mapread 0x00975c thru 0x0119dd (0x8282 bytes) 000123 1787843083.078818 punch from 0x00f0b7 to 0x0119dd (0x2927 bytes) 000124 1787843083.172719 write 0x03800c thru 0x03b38c (0x3381 bytes) 000125 1787843083.194586 read 0x02ce6a thru 0x03a788 (0xd91f bytes) 000126 1787843083.212416 trunc from 0x03b38d to 0x011831 (0x29b5b bytes) 000127 1787843083.283266 MIRROR_RESYNC 2 mirrors 000128 1787843083.892791 trunc from 0x011832 to 0x0347c7 (0x22f96 bytes) 000129 1787843083.977330 fallocate 0x00bb2f thru 0x010ccb (0x519d bytes) 000130 1787843083.991439 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) 000131 1787843084.154428 write 0x0036c9 thru 0x00e23d (0xab75 bytes) 000132 1787843084.204941 mapwrite 0x005942 thru 0x01568b (0xfd4a bytes) 000133 1787843084.268799 mapread 0x01183f thru 0x0186f7 (0x6eb9 bytes) 000134 1787843084.269840 read 0x023aaa thru 0x02b144 (0x769b bytes) 000135 1787843084.287098 read 0x02d316 thru 0x0347c7 (0x74b2 bytes) 000137 1787843084.298998 mapwrite 0x03f0db thru 0x03ffff (0xf25 bytes) 000138 1787843084.370849 mapwrite 0x00c629 thru 0x01695e (0xa336 bytes) 000139 1787843084.409404 mapread 0x017ee8 thru 0x027cc0 (0xfdd9 bytes) 000140 1787843084.412820 write 0x02c02c thru 0x034b2f (0x8b04 bytes) 000141 1787843084.415313 read 0x01a5d4 thru 0x02966e (0xf09b bytes) 000142 1787843084.425181 MIRROR_EXTEND to 2 mirrors 000143 1787843085.122266 trunc from 0x0284b1 to 0x0359e6 (0xd536 bytes) 000144 1787843085.146600 trunc from 0x040000 to 0x035751 (0xa8ae bytes) 000145 1787843085.177815 mapread 0x001f15 thru 0x002541 (0x62d bytes) 000146 1787843085.184548 read 0x00984a thru 0x00f99a (0x6151 bytes) 000147 1787843085.194905 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) 000150 1787843085.375305 write 0x02c9af thru 0x03973c (0xcd8e bytes) 000151 1787843085.388474 read 0x0040dd thru 0x0068ce (0x27f2 bytes) 000152 1787843085.393714 mapread 0x01f658 thru 0x01f799 (0x142 bytes) 000153 1787843085.399072 fallocate 0x025bd9 thru 0x028687 (0x2aaf bytes) 000154 1787843085.403935 write 0x00f7f5 thru 0x01a787 (0xaf93 bytes) 000157 1787843085.415815 mapread 0x00fe1a thru 0x01ab9a (0xad81 bytes) 000158 1787843085.418850 read 0x01fd22 thru 0x026de0 (0x70bf bytes) 000159 1787843085.436308 punch from 0x0319b5 to 0x036f5d (0x55a9 bytes) 000160 1787843085.471297 read 0x0381f7 thru 0x03973c (0x1546 bytes) 000161 1787843085.472071 trunc from 0x0098fe to 0x00ab0a (0x120d bytes) 000163 1787843085.495650 mapwrite 0x022b6b thru 0x02330b (0x7a1 bytes) 000164 1787843085.518292 fallocate 0x002206 thru 0x00972d (0x7528 bytes) 000165 1787843085.523869 mapread 0x01000b thru 0x010f09 (0xeff bytes) 000166 1787843085.524590 mapread 0x01bf0d thru 0x0209e7 (0x4adb bytes) 000167 1787843085.527685 fallocate 0x036de3 thru 0x03ffff (0x921d bytes) 000170 1787843085.535817 punch from 0x004f0e to 0x00c64a (0x773d bytes) 000171 1787843085.547595 trunc from 0x02a4a7 to 0x02a7ec (0x346 bytes) 000172 1787843085.555840 trunc from 0x02421f to 0x02e6b1 (0xa493 bytes) 000173 1787843085.568593 punch from 0x020bd9 to 0x0218c8 (0xcf0 bytes) 000174 1787843085.574445 read 0x0167ae thru 0x025e54 (0xf6a7 bytes) 000177 1787843085.612163 mapwrite 0x027ef4 thru 0x02fdc8 (0x7ed5 bytes) 000178 1787843085.664779 trunc from 0x040000 to 0x03dd32 (0x22cd bytes) 000179 1787843085.719923 punch from 0x0254c0 to 0x027364 (0x1ea5 bytes) 000180 1787843085.741131 mapread 0x005c6b thru 0x012e2f (0xd1c5 bytes) 000181 1787843085.742201 write 0x02a64c thru 0x02e8aa (0x425f bytes) 000182 1787843085.743250 trunc from 0x03dd33 to 0x0310cf (0xcc63 bytes) 000183 1787843085.793080 trunc from 0x0310d0 to 0x00c1a5 (0x24f2a bytes) 000184 1787843085.859559 read 0x00a418 thru 0x00c1a5 (0x1d8e bytes) 000185 1787843085.861159 fallocate 0x02168e thru 0x02464a (0x2fbd bytes) 000186 1787843085.887983 read 0x01a0fa thru 0x01b4e3 (0x13ea bytes) 000187 1787843085.918992 mapread 0x013299 thru 0x0172c9 (0x4031 bytes) 000188 1787843085.925938 read 0x003a30 thru 0x00ceb3 (0x9484 bytes) 000189 1787843085.927905 read 0x01a8da thru 0x02464a (0x9d71 bytes) 000190 1787843085.934716 fallocate 0x0306d9 thru 0x03cc65 (0xc58d bytes) 000191 1787843085.950968 fallocate 0x014e1e thru 0x017ecc (0x30af bytes) 000193 1787843085.982341 trunc from 0x03cc66 to 0x006f72 (0x35cf3 bytes) 000194 1787843086.030864 trunc from 0x006f73 to 0x01b2ca (0x14358 bytes) 000195 1787843086.067651 trunc from 0x01b2cb to 0x0249ab (0x96e1 bytes) 000196 1787843086.099151 mapwrite 0x00a1f2 thru 0x00e847 (0x4656 bytes) 000197 1787843086.238401 read 0x01b727 thru 0x01f787 (0x4061 bytes) 000199 1787843086.249317 fallocate 0x014728 thru 0x01a19e (0x5a77 bytes) 000202 1787843086.271793 write 0x0363cc thru 0x03ffff (0x9c34 bytes) 000203 1787843086.279131 read 0x00e0a8 thru 0x01d894 (0xf7ed bytes) 000204 1787843086.280959 trunc from 0x000080 to 0x000e31 (0xdb2 bytes) 000205 1787843086.297495 mapread 0x02b3b2 thru 0x02f4f4 (0x4143 bytes) 000207 1787843086.335449 write 0x023f16 thru 0x033520 (0xf60b bytes) 000208 1787843086.338472 trunc from 0x040000 to 0x0240b7 (0x1bf48 bytes) 000209 1787843086.370263 fallocate 0x0106ab thru 0x01486f (0x41c5 bytes) 000210 1787843086.398009 punch from 0x017887 to 0x022d77 (0xb4f1 bytes) 000212 1787843086.420712 MIRROR_EXTEND to 2 mirrors 000213 1787843087.565406 trunc from 0x0240b8 to 0x01e8eb (0x57cc bytes) 000214 1787843087.656334 trunc from 0x009687 to 0x00d953 (0x42cd bytes) 000215 1787843087.665959 fallocate 0x036842 thru 0x03e898 (0x8057 bytes) 000216 1787843087.673244 punch from 0x019216 to 0x023574 (0xa35f bytes) 000217 1787843087.681116 mapread 0x00a9d7 thru 0x00e43a (0x3a64 bytes) 000218 1787843087.715147 mapread 0x02b806 thru 0x02f100 (0x38fb bytes) 000220 1787843087.739175 fallocate 0x0277d4 thru 0x02d733 (0x5f60 bytes) 000221 1787843087.750691 write 0x035e28 thru 0x039340 (0x3519 bytes) 000223 1787843087.764342 read 0x03be12 thru 0x03e898 (0x2a87 bytes) 000224 1787843087.771296 fallocate 0x013558 thru 0x01e051 (0xaafa bytes) 000225 1787843087.776177 trunc from 0x00fd94 to 0x01b2ee (0xb55b bytes) 000226 1787843087.783251 trunc from 0x00c901 to 0x00ee3c (0x253c bytes) 000227 1787843087.793597 fallocate 0x020fdf thru 0x024dc4 (0x3de6 bytes) 000228 1787843087.800003 fallocate 0x034232 thru 0x03e85e (0xa62d bytes) 000229 1787843087.808028 mapwrite 0x017ffd thru 0x0274c9 (0xf4cd bytes) 000230 1787843087.882894 mapwrite 0x0250e2 thru 0x02f6d5 (0xa5f4 bytes) 000231 1787843087.936327 mapwrite 0x03908e thru 0x039d78 (0xceb bytes) 000233 1787843087.962987 mapread 0x032e97 thru 0x03e898 (0xba02 bytes) 000234 1787843087.978289 mapwrite 0x01a22c thru 0x021494 (0x7269 bytes) 000235 1787843088.048158 write 0x015727 thru 0x024133 (0xea0d bytes) 000236 1787843088.058714 trunc from 0x03e899 to 0x0208ca (0x1dfce bytes) 000237 1787843088.113728 mapread 0x0194b1 thru 0x0208ca (0x741a bytes) 000238 1787843088.115046 read 0x00b4c0 thru 0x01b388 (0xfec9 bytes) 000239 1787843088.140798 write 0x039f2c thru 0x03ffff (0x60d4 bytes) 000240 1787843088.146775 fallocate 0x009c1d thru 0x01512e (0xb512 bytes) 000241 1787843088.158481 fallocate 0x02c153 thru 0x02cbc0 (0xa6e bytes) 000242 1787843088.167236 punch from 0x0072cc to 0x00e376 (0x70ab bytes) 000243 1787843088.173144 fallocate 0x0146f5 thru 0x01fdca (0xb6d6 bytes) 000244 1787843088.186298 trunc from 0x017bd8 to 0x020eaa (0x92d3 bytes) 000245 1787843088.240890 MIRROR_RESYNC 2 mirrors 000246 1787843090.055181 mapread 0x00bbb6 thru 0x014ed2 (0x931d bytes) 000248 1787843090.144829 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) 000249 1787843090.442246 punch from 0x02a809 to 0x02dde1 (0x35d9 bytes) 000250 1787843090.514842 mapwrite 0x02811f thru 0x032a86 (0xa968 bytes) 000251 1787843090.573271 write 0x03c959 thru 0x03fb86 (0x322e bytes) 000252 1787843090.594704 punch from 0x036128 to 0x03f047 (0x8f20 bytes) 000254 1787843090.644946 mapread 0x028429 thru 0x02c54d (0x4125 bytes) 000256 1787843090.659633 mapread 0x024a41 thru 0x0302db (0xb89b bytes) 000257 1787843090.664463 fallocate 0x0141ad thru 0x01bf89 (0x7ddd bytes) 000258 1787843090.675992 trunc from 0x040000 to 0x033716 (0xc8e9 bytes) 000259 1787843090.711334 punch from 0x02f098 to 0x033716 (0x467f bytes) 000261 1787843090.732781 write 0x02b79e thru 0x0351b9 (0x9a1c bytes) 000262 1787843090.736438 trunc from 0x0351ba to 0x0350af (0x10a bytes) 000263 1787843090.760793 mapread 0x00f990 thru 0x01a664 (0xacd5 bytes) 000264 1787843090.782948 trunc from 0x000430 to 0x0024c0 (0x2091 bytes) 000266 1787843090.789382 punch from 0x00aa0a to 0x00bd58 (0x134f bytes) 000267 1787843090.792871 trunc from 0x0184f0 to 0x024a63 (0xc574 bytes) 000268 1787843090.796855 fallocate 0x02bcd5 thru 0x039a3c (0xdd68 bytes) 000269 1787843090.799890 write 0x03bcf8 thru 0x03ffff (0x4308 bytes) 000270 1787843090.800753 punch from 0x03ddbd to 0x03ffff (0x2243 bytes) 000271 1787843090.811972 mapwrite 0x02d9dc thru 0x02f025 (0x164a bytes) 000272 1787843090.826241 fallocate 0x002cc0 thru 0x0089ad (0x5cee bytes) 000274 1787843090.835814 trunc from 0x01428c to 0x01fa83 (0xb7f8 bytes) 000275 1787843090.844032 mapread 0x0318e5 thru 0x03610e (0x482a bytes) 000276 1787843090.854988 read 0x00dcbb thru 0x019354 (0xb69a bytes) 000277 1787843090.868219 punch from 0x01dfb7 to 0x0288d8 (0xa922 bytes) 000278 1787843090.877727 MIRROR_EXTEND to 2 mirrors 000279 1787843091.559209 fallocate 0x03b77e thru 0x03ffff (0x4882 bytes) 000280 1787843091.612560 trunc from 0x040000 to 0x0100f2 (0x2ff0d bytes) 000281 1787843091.640187 read 0x00f941 thru 0x0100f2 (0x7b2 bytes) 000282 1787843091.644353 punch from 0x0044b4 to 0x0100c8 (0xbc15 bytes) 000283 1787843091.647946 mapwrite 0x0214f9 thru 0x026627 (0x512f bytes) 000284 1787843091.707885 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x26628): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26628): Mirror 2: 0xb0280266 (stale) CRC-32 checksum value for chunk [0, 0x26628): Mirror 1: 0x58298e94 Mirror 3: 0x58298e94 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26628: skipped 000286 1787843092.454064 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26628): Mirror 2: 0xb0280266 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26628): Mirror 2: 0xb0280266 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26628: skipped 000287 1787843092.694948 read 0x019507 thru 0x025191 (0xbc8b bytes) 000288 1787843092.704091 read 0x00b28a thru 0x00d332 (0x20a9 bytes) 000289 1787843092.721270 fallocate 0x02afb6 thru 0x0313d2 (0x641d bytes) 000290 1787843092.740240 mapread 0x013910 thru 0x01d2e2 (0x99d3 bytes) 000291 1787843092.776861 write 0x032c8c thru 0x03bc5e (0x8fd3 bytes) 000292 1787843092.781125 read 0x037760 thru 0x03bc5e (0x44ff bytes) 000294 1787843092.781923 write 0x015858 thru 0x01c7b9 (0x6f62 bytes) 000295 1787843092.782767 write 0x012c7e thru 0x013fa3 (0x1326 bytes) 000296 1787843092.789871 write 0x035a63 thru 0x03a483 (0x4a21 bytes) 000297 1787843092.793859 MIRROR_RESYNC 2 mirrors 000298 1787843093.431596 trunc from 0x03bc5f to 0x021970 (0x1a2ee bytes) 000299 1787843093.489454 punch from 0x01a256 to 0x021970 (0x771b bytes) 000300 1787843093.497167 trunc from 0x01ef67 to 0x021970 (0x2a0a bytes) 000301 1787843093.501775 punch from 0x013cab to 0x018aab (0x4e01 bytes) 000302 1787843093.508179 mapwrite 0x02596d thru 0x032e17 (0xd4ab bytes) 000303 1787843093.594269 punch from 0x01a511 to 0x0236c4 (0x91b4 bytes) 000304 1787843093.604598 mapwrite 0x037709 thru 0x03b37d (0x3c75 bytes) 000305 1787843093.659355 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b37e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b37e): Mirror 2: 0x98c8514c (stale) CRC-32 checksum value for chunk [0, 0x3b37e): Mirror 1: 0x9ea21b3e Mirror 3: 0x9ea21b3e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b37e: skipped 000306 1787843094.815672 punch from 0x0301f8 to 0x03888b (0x8694 bytes) 000307 1787843094.923943 fallocate 0x03968a thru 0x03ffff (0x6976 bytes) 000308 1787843094.936466 mapwrite 0x000951 thru 0x00619f (0x584f bytes) 000309 1787843094.991426 mapwrite 0x026198 thru 0x02c6f5 (0x655e bytes) 000310 1787843095.031824 mapread 0x015af2 thru 0x018c57 (0x3166 bytes) 000311 1787843095.049231 trunc from 0x040000 to 0x03baed (0x4512 bytes) 000312 1787843095.098262 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3baee): Mirror 3: 0xc979f17d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3baee): Mirror 3: 0xc979f17d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3baee: skipped 000313 1787843095.468114 punch from 0x02b20d to 0x0388ec (0xd6e0 bytes) 000314 1787843095.479448 mapwrite 0x02f9fe thru 0x03f162 (0xf765 bytes) 000315 1787843095.572718 mapread 0x00b82a thru 0x0121be (0x6995 bytes) 000316 1787843095.587190 trunc from 0x03f163 to 0x021b68 (0x1d5fa bytes) 000317 1787843095.605399 fallocate 0x018682 thru 0x01ce19 (0x4798 bytes) 000318 1787843095.611309 trunc from 0x00ca47 to 0x012ebb (0x6475 bytes) 000319 1787843095.617350 punch from 0x00f060 to 0x00fdbb (0xd5c bytes) 000320 1787843095.623354 trunc from 0x021b69 to 0x02309a (0x1532 bytes) 000321 1787843095.647309 MIRROR_RESYNC 2 mirrors 000322 1787843096.378154 trunc from 0x00f223 to 0x01c55b (0xd339 bytes) 000323 1787843096.421511 mapread 0x00133b thru 0x00acd2 (0x9998 bytes) 000324 1787843096.442876 write 0x00094b thru 0x00c250 (0xb906 bytes) 000325 1787843096.453293 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2309b): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2309b): Mirror 3: 0xcc1b9a32 (stale) CRC-32 checksum value for chunk [0, 0x2309b): Mirror 1: 0x28005859 Mirror 4: 0x28005859 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2309b: skipped 000326 1787843097.338342 read 0x0159f3 thru 0x018512 (0x2b20 bytes) 000327 1787843097.346553 trunc from 0x0212af to 0x02309a (0x1dec bytes) 000328 1787843097.407428 punch from 0x0134c0 to 0x02309a (0xfbdb bytes) 000329 1787843097.414591 mapwrite 0x01cd6a thru 0x023534 (0x67cb bytes) 000330 1787843097.510052 trunc from 0x023535 to 0x0107d1 (0x12d63 bytes) 000331 1787843097.528729 read 0x005864 thru 0x0107d1 (0xaf6e bytes) 000332 1787843097.542293 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x107d2): Mirror 3: 0xb22fd84f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x107d2): Mirror 3: 0xb22fd84f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x107d2: skipped 000333 1787843097.800485 fallocate 0x00d78b thru 0x01b687 (0xdefd bytes) 000334 1787843097.807704 trunc from 0x010a64 to 0x01526e (0x480b bytes) 000335 1787843097.818468 MIRROR_RESYNC 2 mirrors 000337 1787843098.750344 read 0x0051a5 thru 0x010370 (0xb1cc bytes) 000338 1787843098.770357 trunc from 0x01b688 to 0x03e2cd (0x22c46 bytes) 000339 1787843098.865162 trunc from 0x03e2ce to 0x001f02 (0x3c3cb bytes) 000340 1787843098.896072 read 0x0004b2 thru 0x001f02 (0x1a51 bytes) 000341 1787843098.920985 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1f03): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f03): Mirror 3: 0xb53a4719 (stale) CRC-32 checksum value for chunk [0, 0x1f03): Mirror 1: 0xb53a4719 Mirror 4: 0xb53a4719 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f03: skipped 000342 1787843099.573727 trunc from 0x001f03 to 0x027639 (0x25737 bytes) 000343 1787843099.612808 MIRROR_RESYNC 3 mirrors 000344 1787843100.270437 read 0x00df40 thru 0x012740 (0x4801 bytes) 000345 1787843100.281872 mapwrite 0x03d847 thru 0x03ffff (0x27b9 bytes) 000346 1787843100.381549 trunc from 0x00ab0a to 0x010fa3 (0x649a bytes) 000347 1787843100.387557 trunc from 0x00500c to 0x013bac (0xeba1 bytes) 000348 1787843100.393360 read 0x01deb4 thru 0x027a98 (0x9be5 bytes) 000349 1787843100.407327 trunc from 0x02e772 to 0x03dc73 (0xf502 bytes) 000350 1787843100.414403 mapwrite 0x019d6c thru 0x01ad31 (0xfc6 bytes) 000351 1787843100.427083 trunc from 0x040000 to 0x02a04d (0x15fb2 bytes) 000352 1787843100.466501 MIRROR_RESYNC 3 mirrors 000353 1787843101.528239 mapwrite 0x00bfde thru 0x019262 (0xd285 bytes) 000354 1787843101.679394 read 0x00d371 thru 0x014073 (0x6d03 bytes) 000355 1787843101.681298 punch from 0x012007 to 0x01d399 (0xb393 bytes) 000356 1787843101.694173 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2a04e): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a04e): Mirror 3: 0x63301796 (stale) Mirror 4: 0x63301796 (stale) CRC-32 checksum value for chunk [0, 0x2a04e): Mirror 1: 0x3d91cf82 Mirror 5: 0x3d91cf82 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a04e: skipped 000357 1787843103.011549 mapwrite 0x03cf34 thru 0x03ffff (0x30cc bytes) 000358 1787843103.145577 fallocate 0x025fd0 thru 0x029b6a (0x3b9b bytes) 000359 1787843103.154602 write 0x038883 thru 0x03c286 (0x3a04 bytes) 000360 1787843103.160306 MIRROR_RESYNC 4 mirrors 000361 1787843104.199250 fallocate 0x00183e thru 0x00d9d7 (0xc19a bytes) 000362 1787843104.238408 read 0x01f698 thru 0x026a32 (0x739b bytes) 000363 1787843104.245259 trunc from 0x031dff to 0x03b690 (0x9892 bytes) 000364 1787843104.249204 mapread 0x035765 thru 0x03ffff (0xa89b bytes) 000365 1787843104.269740 trunc from 0x040000 to 0x003171 (0x3ce8e bytes) 000366 1787843104.309250 fallocate 0x000b6d thru 0x00fcfa (0xf18e bytes) 000367 1787843104.317152 trunc from 0x00d49a to 0x00e5b8 (0x111f bytes) 000368 1787843104.322343 mapwrite 0x03aa85 thru 0x03ffff (0x557b bytes) 000369 1787843104.356438 mapread 0x019391 thru 0x01b32f (0x1f9f bytes) 000370 1787843104.367447 trunc from 0x040000 to 0x02d46c (0x12b93 bytes) 000371 1787843104.380422 punch from 0x016452 to 0x01b324 (0x4ed3 bytes) 000372 1787843104.384647 trunc from 0x02d46d to 0x01f3b4 (0xe0b8 bytes) 000373 1787843104.401435 punch from 0x00cc1d to 0x00dd7c (0x1160 bytes) 000374 1787843104.406106 fallocate 0x01002d thru 0x016228 (0x61fc bytes) 000376 1787843104.411361 trunc from 0x01f3b5 to 0x036fb9 (0x17c05 bytes) 000377 1787843104.431942 write 0x027534 thru 0x02bf2a (0x49f7 bytes) 000378 1787843104.441884 mapwrite 0x02a8dd thru 0x033d12 (0x9436 bytes) 000379 1787843104.470180 trunc from 0x004515 to 0x00ee0d (0xa8f9 bytes) 000380 1787843104.475783 write 0x0127a5 thru 0x01d073 (0xa8cf bytes) 000381 1787843104.477490 trunc from 0x02a31a to 0x0324ca (0x81b1 bytes) 000382 1787843104.483125 write 0x034c02 thru 0x039bcb (0x4fca bytes) 000383 1787843104.484463 mapwrite 0x02b0db thru 0x02ec99 (0x3bbf bytes) 000384 1787843104.512124 read 0x003bff thru 0x0132b4 (0xf6b6 bytes) 000385 1787843104.524812 punch from 0x0001f1 to 0x0049b5 (0x47c5 bytes) 000386 1787843104.531901 mapwrite 0x02f842 thru 0x037549 (0x7d08 bytes) 000387 1787843104.553159 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39bcc): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39bcc): Mirror 3: 0x90069e51 (stale) Mirror 4: 0x90069e51 (stale) Mirror 5: 0x90069e51 (stale) CRC-32 checksum value for chunk [0, 0x39bcc): Mirror 1: 0xc6c6caf1 Mirror 6: 0xc6c6caf1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39bcc: skipped 000388 1787843105.455434 write 0x014d5f thru 0x01e179 (0x941b bytes) 000389 1787843105.544333 punch from 0x036777 to 0x039bcb (0x3455 bytes) 000390 1787843105.552395 write 0x028e58 thru 0x038d95 (0xff3e bytes) 000391 1787843105.573576 trunc from 0x039bcc to 0x022c97 (0x16f34 bytes) 000392 1787843105.699672 write 0x0169bd thru 0x01c2cc (0x5910 bytes) 000393 1787843105.702356 mapwrite 0x03ca7c thru 0x03f9df (0x2f64 bytes) 000394 1787843105.790071 mapwrite 0x0394cc thru 0x03e0fa (0x4c2f bytes) 000395 1787843105.878706 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f9e0): Mirror 3: 0x3a4f854a (stale) Mirror 4: 0x3a4f854a (stale) Mirror 6: 0x732ac48c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9e0): Mirror 3: 0x3a4f854a (stale) Mirror 4: 0x3a4f854a (stale) Mirror 6: 0x732ac48c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9e0: skipped 000396 1787843106.278145 trunc from 0x02cd95 to 0x02e462 (0x16ce bytes) 000397 1787843106.283797 fallocate 0x01b947 thru 0x024010 (0x86ca bytes) 000398 1787843106.288982 punch from 0x026989 to 0x02781a (0xe92 bytes) 000399 1787843106.294552 write 0x006d55 thru 0x00d240 (0x64ec bytes) 000400 1787843106.310257 punch from 0x00975d to 0x012889 (0x912d bytes) 000401 1787843106.324377 mapwrite 0x01144a thru 0x01557c (0x4133 bytes) 000402 1787843106.359783 mapread 0x02657f thru 0x034503 (0xdf85 bytes) 000403 1787843106.385238 trunc from 0x03f9e0 to 0x02f772 (0x1026d bytes) 000404 1787843106.405917 trunc from 0x02f773 to 0x0119a2 (0x1ddd0 bytes) 000405 1787843106.422727 trunc from 0x010edc to 0x0119a2 (0xac7 bytes) 000406 1787843106.429802 trunc from 0x002da2 to 0x007d3b (0x4f9a bytes) 000408 1787843106.436648 punch from 0x008d12 to 0x0119a2 (0x8c91 bytes) 000409 1787843106.443243 read 0x000b66 thru 0x010420 (0xf8bb bytes) 000410 1787843106.456352 trunc from 0x011568 to 0x0119a2 (0x43b bytes) 000411 1787843106.461470 trunc from 0x0119a3 to 0x0225cc (0x10c2a bytes) 000412 1787843106.488606 fallocate 0x01359f thru 0x01b1f0 (0x7c52 bytes) 000413 1787843106.494557 write 0x02aca6 thru 0x036a61 (0xbdbc bytes) 000414 1787843106.504094 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36a62): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36a62): Mirror 3: 0x4aae9608 (stale) Mirror 4: 0x4aae9608 (stale) Mirror 6: 0x5398079f (stale) CRC-32 checksum value for chunk [0, 0x36a62): Mirror 1: 0x6f4055c Mirror 7: 0x6f4055c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36a62: skipped 000415 1787843107.510410 read 0x02e1a5 thru 0x036a61 (0x88bd bytes) 000416 1787843107.522314 mapread 0x00828b thru 0x00e905 (0x667b bytes) 000417 1787843107.549459 trunc from 0x00e164 to 0x0186cb (0xa568 bytes) 000418 1787843107.599741 fallocate 0x0331fa thru 0x03eb3d (0xb944 bytes) 000419 1787843107.609957 punch from 0x018bcd to 0x025abf (0xcef3 bytes) 000420 1787843107.618309 mapread 0x01fa50 thru 0x029a36 (0x9fe7 bytes) 000421 1787843107.646084 trunc from 0x02a85a to 0x02ea0a (0x41b1 bytes) 000422 1787843107.670303 write 0x006724 thru 0x00ae69 (0x4746 bytes) 000423 1787843107.725308 mapread 0x0184df thru 0x01b9b6 (0x34d8 bytes) 000424 1787843107.727448 punch from 0x023236 to 0x02fb7a (0xc945 bytes) 000425 1787843107.761208 MIRROR_RESYNC 5 mirrors 000426 1787843109.082164 trunc from 0x03eb3e to 0x01bf2c (0x22c11 bytes) 000427 1787843109.146901 punch from 0x001464 to 0x00d0a9 (0xbc46 bytes) 000428 1787843109.152472 punch from 0x011728 to 0x01bf2c (0xa805 bytes) 000429 1787843109.163746 mapwrite 0x0234a6 thru 0x02fe67 (0xc9c2 bytes) 000430 1787843109.254723 mapread 0x023a97 thru 0x02c63f (0x8ba9 bytes) 000431 1787843109.257339 fallocate 0x03bd83 thru 0x03ffff (0x427d bytes) 000432 1787843109.265952 trunc from 0x002b23 to 0x0052b6 (0x2794 bytes) 000433 1787843109.272683 read 0x02fbdb thru 0x03cbe7 (0xd00d bytes) 000434 1787843109.285812 punch from 0x02fc8b to 0x03c8a5 (0xcc1b bytes) 000435 1787843109.292766 punch from 0x02efdb to 0x037900 (0x8926 bytes) 000436 1787843109.299852 read 0x02909a thru 0x03824c (0xf1b3 bytes) 000437 1787843109.312830 trunc from 0x02ce59 to 0x0379c3 (0xab6b bytes) 000438 1787843109.319134 punch from 0x010097 to 0x01bb5b (0xbac5 bytes) 000439 1787843109.326162 trunc from 0x040000 to 0x0202ab (0x1fd54 bytes) 000440 1787843109.346310 trunc from 0x0202ac to 0x02b8e0 (0xb635 bytes) 000441 1787843109.365121 write 0x00a0cc thru 0x01164a (0x757f bytes) 000442 1787843109.375754 read 0x003cd8 thru 0x00eca1 (0xafca bytes) 000443 1787843109.377245 mapwrite 0x02564c thru 0x029df7 (0x47ac bytes) 000444 1787843109.405515 mapread 0x00d1cb thru 0x0181ac (0xafe2 bytes) 000445 1787843109.413494 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b8e1): Mirror 3: 0x452d1832 (stale) Mirror 6: 0x452d1832 (stale) Mirror 7: 0x452d1832 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b8e1): Mirror 3: 0x452d1832 (stale) Mirror 6: 0x452d1832 (stale) Mirror 7: 0x452d1832 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b8e1: skipped 000446 1787843109.737395 trunc from 0x0179fe to 0x01ac6c (0x326f bytes) 000447 1787843109.742924 trunc from 0x02b8e1 to 0x00cba3 (0x1ed3d bytes) 000448 1787843109.770847 read 0x00c9ad thru 0x00cba3 (0x1f7 bytes) 000449 1787843109.775168 punch from 0x00449c to 0x00cba3 (0x8708 bytes) 000450 1787843109.783324 punch from 0x005d69 to 0x00cba3 (0x6e3b bytes) 000451 1787843109.788153 trunc from 0x00cba4 to 0x03a6e7 (0x2db44 bytes) 000452 1787843109.805836 mapwrite 0x035041 thru 0x03ffff (0xafbf bytes) 000453 1787843109.876189 write 0x02691c thru 0x0276dd (0xdc2 bytes) 000454 1787843109.884848 fallocate 0x00e17b thru 0x0121b8 (0x403e bytes) 000455 1787843109.889489 read 0x01052f thru 0x019ebf (0x9991 bytes) 000456 1787843109.900029 mapwrite 0x01ff37 thru 0x02e93a (0xea04 bytes) 000457 1787843109.923503 trunc from 0x040000 to 0x023fad (0x1c052 bytes) 000458 1787843109.970099 mapread 0x00cbb8 thru 0x00f1bb (0x2604 bytes) 000459 1787843109.975243 write 0x00a5f4 thru 0x014de5 (0xa7f2 bytes) 000460 1787843109.977497 trunc from 0x010849 to 0x013c5f (0x3417 bytes) 000461 1787843109.994979 trunc from 0x023fae to 0x018e9c (0xb111 bytes) 000462 1787843110.021432 trunc from 0x007341 to 0x01710a (0xfdca bytes) 000463 1787843110.028748 fallocate 0x002e0b thru 0x00c73c (0x9932 bytes) 000464 1787843110.034336 mapwrite 0x000731 thru 0x0012cc (0xb9c bytes) 000465 1787843110.081481 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x18e9d): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18e9d): Mirror 3: 0xc42168c4 (stale) Mirror 6: 0xc42168c4 (stale) Mirror 7: 0xc42168c4 (stale) CRC-32 checksum value for chunk [0, 0x18e9d): Mirror 1: 0x54802bf9 Mirror 8: 0x54802bf9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18e9d: skipped 000466 1787843111.269578 read 0x0058da thru 0x007f99 (0x26c0 bytes) 000468 1787843111.279713 trunc from 0x018e9d to 0x0243bc (0xb520 bytes) 000469 1787843111.365552 fallocate 0x00089f thru 0x00517a (0x48dc bytes) 000470 1787843111.369369 mapwrite 0x024b0c thru 0x02737f (0x2874 bytes) 000471 1787843111.447037 mapread 0x026bd0 thru 0x02737f (0x7b0 bytes) 000472 1787843111.447966 mapread 0x0216a5 thru 0x02737f (0x5cdb bytes) 000473 1787843111.448820 mapread 0x00a388 thru 0x016b8d (0xc806 bytes) 000474 1787843111.481458 fallocate 0x024b7c thru 0x032460 (0xd8e5 bytes) 000475 1787843111.493977 write 0x01b2d0 thru 0x023ebe (0x8bef bytes) 000476 1787843111.501130 fallocate 0x031173 thru 0x0370e7 (0x5f75 bytes) 000477 1787843111.510803 fallocate 0x00cc65 thru 0x0100c6 (0x3462 bytes) 000478 1787843111.526570 fallocate 0x03c438 thru 0x03cc0e (0x7d7 bytes) 000479 1787843111.534339 punch from 0x024df4 to 0x02ff90 (0xb19d bytes) 000480 1787843111.542492 write 0x013f23 thru 0x0231ab (0xf289 bytes) 000481 1787843111.545117 mapwrite 0x014184 thru 0x014916 (0x793 bytes) 000482 1787843111.585950 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x370e8): Mirror 3: 0xd6447c9b (stale) Mirror 6: 0xd6447c9b (stale) Mirror 8: 0xb873a56d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x370e8): Mirror 3: 0xd6447c9b (stale) Mirror 6: 0xd6447c9b (stale) Mirror 8: 0xb873a56d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x370e8: skipped 000483 1787843112.049362 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x370e8): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x370e8): Mirror 3: 0xd6447c9b (stale) Mirror 6: 0xd6447c9b (stale) Mirror 8: 0xb873a56d (stale) CRC-32 checksum value for chunk [0, 0x370e8): Mirror 1: 0x62c58589 Mirror 9: 0x62c58589 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x370e8: skipped 000484 1787843112.957575 punch from 0x02ece5 to 0x03152f (0x284b bytes) 000485 1787843113.008487 fallocate 0x03a356 thru 0x03f25d (0x4f08 bytes) 000486 1787843113.019337 mapread 0x018c4b thru 0x02571c (0xcad2 bytes) 000487 1787843113.051159 trunc from 0x006493 to 0x0125da (0xc148 bytes) 000488 1787843113.056423 mapwrite 0x03c12e thru 0x03cf3f (0xe12 bytes) 000489 1787843113.081428 mapread 0x017b62 thru 0x01e809 (0x6ca8 bytes) 000490 1787843113.082677 mapwrite 0x0228a1 thru 0x023fba (0x171a bytes) 000491 1787843113.095101 read 0x033258 thru 0x03c935 (0x96de bytes) 000492 1787843113.104085 trunc from 0x03f25e to 0x009ee4 (0x35379 bytes) 000493 1787843113.131543 punch from 0x001a48 to 0x0036c8 (0x1c81 bytes) 000494 1787843113.139280 mapwrite 0x0116b1 thru 0x01ad91 (0x96e1 bytes) 000495 1787843113.209540 mapwrite 0x0035ea thru 0x00d655 (0xa06c bytes) 000496 1787843113.238205 read 0x00edc8 thru 0x0127e1 (0x3a1a bytes) 000497 1787843113.239218 write 0x03bc00 thru 0x03bd66 (0x167 bytes) 000499 1787843113.241214 punch from 0x03b642 to 0x03bd66 (0x725 bytes) 000500 1787843113.251452 fallocate 0x036c61 thru 0x03dfa5 (0x7345 bytes) 000501 1787843113.257315 read 0x015b14 thru 0x0227c4 (0xccb1 bytes) 000502 1787843113.267312 fallocate 0x037e7e thru 0x03ffff (0x8182 bytes) 000503 1787843113.276943 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3bd67): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd67): Mirror 3: 0x81dd659b (stale) Mirror 6: 0x81dd659b (stale) Mirror 8: 0x45ba3580 (stale) Mirror 9: 0xb46c341a (stale) CRC-32 checksum value for chunk [0, 0x3bd67): Mirror 1: 0x1ea940ad Mirror 10: 0x1ea940ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd67: skipped 000504 1787843114.540290 write 0x0219d3 thru 0x02ce3f (0xb46d bytes) 000505 1787843114.614289 trunc from 0x03bd67 to 0x00b43b (0x3092b bytes) 000507 1787843114.665138 trunc from 0x00b43c to 0x01b233 (0xfdf8 bytes) 000508 1787843114.690102 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1b234): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b234): Mirror 3: 0xc20782ca (stale) Mirror 6: 0xc20782ca (stale) Mirror 8: 0x3ff38d00 (stale) Mirror 9: 0x407f46e8 (stale) Mirror 10: 0x511e39a9 (stale) CRC-32 checksum value for chunk [0, 0x1b234): Mirror 1: 0x5556e4a9 Mirror 11: 0x5556e4a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b234: skipped 000509 1787843115.809056 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b234): Mirror 3: 0xc20782ca (stale) Mirror 6: 0xc20782ca (stale) Mirror 8: 0x3ff38d00 (stale) Mirror 9: 0x407f46e8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b234): Mirror 3: 0xc20782ca (stale) Mirror 6: 0xc20782ca (stale) Mirror 8: 0x3ff38d00 (stale) Mirror 9: 0x407f46e8 (stale) CRC-32 checksum value for chunk [0, 0x1b234): Mirror 1: 0x5556e4a9 Mirror 11: 0x5556e4a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b234: skipped 000510 1787843116.402174 fallocate 0x023f96 thru 0x02c0a3 (0x810e bytes) 000511 1787843116.473842 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1b234): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b234): Mirror 3: 0xc20782ca (stale) Mirror 6: 0xc20782ca (stale) Mirror 8: 0x3ff38d00 (stale) Mirror 9: 0x407f46e8 (stale) Mirror 11: 0x5556e4a9 (stale) CRC-32 checksum value for chunk [0, 0x1b234): Mirror 1: 0x5556e4a9 Mirror 12: 0x5556e4a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b234: skipped 000513 1787843117.735507 punch from 0x007e8c to 0x010020 (0x8195 bytes) 000514 1787843117.788476 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1b234): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b234): Mirror 3: 0xc20782ca (stale) Mirror 6: 0xc20782ca (stale) Mirror 8: 0x3ff38d00 (stale) Mirror 9: 0x407f46e8 (stale) Mirror 11: 0x5556e4a9 (stale) Mirror 12: 0x5556e4a9 (stale) CRC-32 checksum value for chunk [0, 0x1b234): Mirror 1: 0x505cf729 Mirror 13: 0x505cf729 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b234: skipped 000515 1787843118.793326 trunc from 0x00efdb to 0x01abe1 (0xbc07 bytes) 000518 1787843118.873481 read 0x0125a2 thru 0x01909c (0x6afb bytes) 000519 1787843118.884465 write 0x033140 thru 0x039d48 (0x6c09 bytes) 000520 1787843118.898911 MIRROR_RESYNC 8 mirrors 000521 1787843121.128723 read 0x0384a5 thru 0x039d48 (0x18a4 bytes) 000522 1787843121.134170 mapread 0x01bcc3 thru 0x0229df (0x6d1d bytes) 000523 1787843121.158383 punch from 0x00fd7b to 0x018959 (0x8bdf bytes) 000524 1787843121.206542 trunc from 0x039d49 to 0x00f63f (0x2a709 bytes) 000526 1787843121.230046 punch from 0x00bd8b to 0x00f63f (0x38b5 bytes) 000527 1787843121.234415 fallocate 0x015804 thru 0x02366b (0xde68 bytes) 000528 1787843121.239705 mapwrite 0x027bb6 thru 0x02fba7 (0x7ff2 bytes) 000529 1787843121.283500 mapread 0x013055 thru 0x017657 (0x4603 bytes) 000530 1787843121.294249 punch from 0x023f24 to 0x0296a6 (0x5783 bytes) 000531 1787843121.298303 mapwrite 0x00d86c thru 0x015edc (0x8671 bytes) 000532 1787843121.321389 write 0x010e4f thru 0x014644 (0x37f6 bytes) 000533 1787843121.324794 mapwrite 0x027da0 thru 0x030b7a (0x8ddb bytes) 000534 1787843121.419463 fallocate 0x028053 thru 0x0284aa (0x458 bytes) 000536 1787843121.426786 mapwrite 0x026d9f thru 0x0332ff (0xc561 bytes) 000537 1787843121.482587 trunc from 0x0150c5 to 0x01b490 (0x63cc bytes) 000538 1787843121.489534 punch from 0x0097dc to 0x01258d (0x8db2 bytes) 000539 1787843121.496077 fallocate 0x0095c4 thru 0x011777 (0x81b4 bytes) 000540 1787843121.510704 mapwrite 0x010717 thru 0x01f26e (0xeb58 bytes) 000541 1787843121.560217 read 0x00b9c2 thru 0x018892 (0xced1 bytes) 000542 1787843121.561462 trunc from 0x021ccf to 0x02ea87 (0xcdb9 bytes) 000543 1787843121.566325 mapread 0x013c40 thru 0x014235 (0x5f6 bytes) 000544 1787843121.567284 read 0x019223 thru 0x01e5b4 (0x5392 bytes) 000545 1787843121.567900 punch from 0x029d46 to 0x0332ff (0x95ba bytes) 000546 1787843121.573336 read 0x015ac5 thru 0x024d15 (0xf251 bytes) 000548 1787843121.594035 punch from 0x00ee00 to 0x018877 (0x9a78 bytes) 000549 1787843121.600599 write 0x00146e thru 0x00d78d (0xc320 bytes) 000550 1787843121.604021 trunc from 0x033300 to 0x02e0d2 (0x522d bytes) 000551 1787843121.620586 read 0x00c6ab thru 0x00e534 (0x1e8a bytes) 000552 1787843121.633037 MIRROR_RESYNC 8 mirrors 000553 1787843123.392260 trunc from 0x027171 to 0x02c684 (0x5514 bytes) 000554 1787843123.439722 trunc from 0x010b52 to 0x013749 (0x2bf8 bytes) 000555 1787843123.449891 mapread 0x01d84d thru 0x02022f (0x29e3 bytes) 000556 1787843123.469192 trunc from 0x02e0d3 to 0x0377bd (0x96eb bytes) 000557 1787843123.487538 mapread 0x02bd6e thru 0x0351cc (0x945f bytes) 000558 1787843123.508040 trunc from 0x0377be to 0x01db34 (0x19c89 bytes) 000559 1787843123.523578 punch from 0x01c5e0 to 0x01db34 (0x1555 bytes) 000560 1787843123.529687 fallocate 0x028719 thru 0x02cab3 (0x439b bytes) 000561 1787843123.535063 trunc from 0x0152bd to 0x024f98 (0xfcdc bytes) 000562 1787843123.540459 mapread 0x008000 thru 0x011ce9 (0x9cea bytes) 000563 1787843123.568197 trunc from 0x02ab59 to 0x02cab3 (0x1f5b bytes) 000565 1787843123.573971 trunc from 0x02cab4 to 0x038729 (0xbc76 bytes) 000566 1787843123.591013 trunc from 0x03872a to 0x01b52d (0x1d1fc bytes) 000567 1787843123.617003 punch from 0x004efe to 0x0080c8 (0x31cb bytes) 000568 1787843123.623481 trunc from 0x00eead to 0x01a6ca (0xb81e bytes) 000570 1787843123.634149 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1b52e): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b52e): Mirror 3: 0x30df37a4 (stale) Mirror 6: 0x30df37a4 (stale) Mirror 8: 0x30df37a4 (stale) Mirror 9: 0x30df37a4 (stale) Mirror 11: 0x30df37a4 (stale) Mirror 12: 0x30df37a4 (stale) Mirror 13: 0x30df37a4 (stale) CRC-32 checksum value for chunk [0, 0x1b52e): Mirror 1: 0xca43c664 Mirror 14: 0xca43c664 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b52e: skipped 000571 1787843124.717877 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1b52e): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b52e): Mirror 3: 0x30df37a4 (stale) Mirror 6: 0x30df37a4 (stale) Mirror 8: 0x30df37a4 (stale) Mirror 9: 0x30df37a4 (stale) Mirror 11: 0x30df37a4 (stale) Mirror 12: 0x30df37a4 (stale) Mirror 13: 0x30df37a4 (stale) CRC-32 checksum value for chunk [0, 0x1b52e): Mirror 1: 0xca43c664 Mirror 14: 0xca43c664 Mirror 15: 0xca43c664 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b52e: skipped 000573 1787843125.669809 fallocate 0x02f3ce thru 0x034a3c (0x566f bytes) 000574 1787843125.715039 mapwrite 0x01ef0b thru 0x0223d6 (0x34cc bytes) 000575 1787843125.819405 write 0x037eed thru 0x03ffff (0x8113 bytes) 000576 1787843125.841355 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1b7b2206 (stale) Mirror 6: 0x1b7b2206 (stale) Mirror 8: 0x1b7b2206 (stale) Mirror 9: 0x1b7b2206 (stale) Mirror 11: 0x1b7b2206 (stale) Mirror 12: 0x1b7b2206 (stale) Mirror 14: 0x4ee162d2 (stale) Mirror 15: 0x4ee162d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1b7b2206 (stale) Mirror 6: 0x1b7b2206 (stale) Mirror 8: 0x1b7b2206 (stale) Mirror 9: 0x1b7b2206 (stale) Mirror 11: 0x1b7b2206 (stale) Mirror 12: 0x1b7b2206 (stale) Mirror 14: 0x4ee162d2 (stale) Mirror 15: 0x4ee162d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000577 1787843126.464394 trunc from 0x025ca4 to 0x033f18 (0xe275 bytes) 000579 1787843126.477463 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1b7b2206 (stale) Mirror 8: 0x1b7b2206 (stale) Mirror 9: 0x1b7b2206 (stale) Mirror 11: 0x1b7b2206 (stale) Mirror 12: 0x1b7b2206 (stale) Mirror 14: 0x4ee162d2 (stale) Mirror 15: 0x4ee162d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1b7b2206 (stale) Mirror 8: 0x1b7b2206 (stale) Mirror 9: 0x1b7b2206 (stale) Mirror 11: 0x1b7b2206 (stale) Mirror 12: 0x1b7b2206 (stale) Mirror 14: 0x4ee162d2 (stale) Mirror 15: 0x4ee162d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000580 1787843126.970339 mapread 0x030905 thru 0x0351b3 (0x48af bytes) 000581 1787843127.000979 fallocate 0x02554e thru 0x02cc70 (0x7723 bytes) 000582 1787843127.006596 trunc from 0x040000 to 0x01b0bd (0x24f42 bytes) 000583 1787843127.026268 mapread 0x01394b thru 0x01b0bd (0x7773 bytes) 000584 1787843127.041113 MIRROR_RESYNC 8 mirrors 000585 1787843128.850760 mapread 0x005131 thru 0x0099ea (0x48ba bytes) 000586 1787843128.870586 punch from 0x019368 to 0x01b0bd (0x1d56 bytes) 000587 1787843128.908766 punch from 0x014fa3 to 0x01b0bd (0x611b bytes) 000588 1787843128.913380 mapread 0x010648 thru 0x01655c (0x5f15 bytes) 000589 1787843128.942404 fallocate 0x0193c8 thru 0x028662 (0xf29b bytes) 000590 1787843128.953319 mapwrite 0x021eab thru 0x02bba0 (0x9cf6 bytes) 000591 1787843129.068565 mapread 0x025f8a thru 0x02bba0 (0x5c17 bytes) 000592 1787843129.069593 trunc from 0x02bba1 to 0x02e159 (0x25b9 bytes) 000593 1787843129.086021 mapwrite 0x02718b thru 0x02bd4d (0x4bc3 bytes) 000595 1787843129.102091 mapwrite 0x01b92a thru 0x02a321 (0xe9f8 bytes) 000596 1787843129.124405 punch from 0x02b834 to 0x02e159 (0x2926 bytes) 000597 1787843129.129952 punch from 0x00fe23 to 0x01e0e6 (0xe2c4 bytes) 000598 1787843129.135511 write 0x012ddc thru 0x01df51 (0xb176 bytes) 000600 1787843129.150592 trunc from 0x0028f7 to 0x012060 (0xf76a bytes) 000601 1787843129.164267 write 0x03a549 thru 0x03a8bf (0x377 bytes) 000603 1787843129.165917 trunc from 0x0145dc to 0x01bad8 (0x74fd bytes) 000604 1787843129.171913 fallocate 0x00db60 thru 0x012d1b (0x51bc bytes) 000605 1787843129.176973 mapwrite 0x02f015 thru 0x032d8a (0x3d76 bytes) 000606 1787843129.202224 read 0x01f190 thru 0x02b36f (0xc1e0 bytes) 000607 1787843129.203101 fallocate 0x00d018 thru 0x01cfae (0xff97 bytes) 000608 1787843129.208074 write 0x03509d thru 0x03cb3d (0x7aa1 bytes) 000609 1787843129.210883 fallocate 0x02f54e thru 0x038c4e (0x9701 bytes) 000610 1787843129.215719 mapwrite 0x01142d thru 0x0169dd (0x55b1 bytes) 000611 1787843129.259197 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3cb3e): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb3e): Mirror 6: 0x2d907395 (stale) Mirror 8: 0x2d907395 (stale) Mirror 9: 0x2d907395 (stale) Mirror 11: 0x2d907395 (stale) Mirror 12: 0x2d907395 (stale) Mirror 14: 0x2d907395 (stale) Mirror 15: 0x2d907395 (stale) CRC-32 checksum value for chunk [0, 0x3cb3e): Mirror 1: 0x62e53c05 Mirror 16: 0x62e53c05 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb3e: skipped 000612 1787843130.722513 fallocate 0x004702 thru 0x00a926 (0x6225 bytes) 000613 1787843130.797511 MIRROR_RESYNC 9 mirrors 000614 1787843134.413132 write 0x02e940 thru 0x03c3bc (0xda7d bytes) 000615 1787843134.535265 MIRROR_RESYNC 9 mirrors 000616 1787843137.443890 fallocate 0x022805 thru 0x02b45d (0x8c59 bytes) 000617 1787843137.479895 trunc from 0x01df97 to 0x02175f (0x37c9 bytes) 000618 1787843137.491054 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3cb3e): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb3e): Mirror 6: 0xc4aa77e2 (stale) Mirror 8: 0xc4aa77e2 (stale) Mirror 9: 0xc4aa77e2 (stale) Mirror 11: 0xc4aa77e2 (stale) Mirror 12: 0xc4aa77e2 (stale) Mirror 14: 0xc4aa77e2 (stale) Mirror 15: 0xc4aa77e2 (stale) Mirror 16: 0xc4aa77e2 (stale) CRC-32 checksum value for chunk [0, 0x3cb3e): Mirror 1: 0xaf671f36 Mirror 17: 0xaf671f36 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb3e: skipped 000620 1787843138.895228 write 0x00ccb0 thru 0x00f2fe (0x264f bytes) 000621 1787843138.965178 read 0x01bddb thru 0x024db4 (0x8fda bytes) 000622 1787843139.002762 fallocate 0x015cb2 thru 0x018551 (0x28a0 bytes) 000623 1787843139.011381 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3cb3e): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb3e): Mirror 6: 0xc4aa77e2 (stale) Mirror 8: 0xc4aa77e2 (stale) Mirror 9: 0xc4aa77e2 (stale) Mirror 11: 0xc4aa77e2 (stale) Mirror 12: 0xc4aa77e2 (stale) Mirror 14: 0xc4aa77e2 (stale) Mirror 15: 0xc4aa77e2 (stale) Mirror 16: 0xc4aa77e2 (stale) Mirror 17: 0xaf671f36 (stale) CRC-32 checksum value for chunk [0, 0x3cb3e): Mirror 1: 0x75f5d5fa Mirror 18: 0x75f5d5fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb3e: skipped 000624 1787843140.909149 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cb3e): Mirror 8: 0xc4aa77e2 (stale) Mirror 9: 0xc4aa77e2 (stale) Mirror 11: 0xc4aa77e2 (stale) Mirror 12: 0xc4aa77e2 (stale) Mirror 14: 0xc4aa77e2 (stale) Mirror 15: 0xc4aa77e2 (stale) Mirror 16: 0xc4aa77e2 (stale) Mirror 17: 0xaf671f36 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb3e): Mirror 8: 0xc4aa77e2 (stale) Mirror 9: 0xc4aa77e2 (stale) Mirror 11: 0xc4aa77e2 (stale) Mirror 12: 0xc4aa77e2 (stale) Mirror 14: 0xc4aa77e2 (stale) Mirror 15: 0xc4aa77e2 (stale) Mirror 16: 0xc4aa77e2 (stale) Mirror 17: 0xaf671f36 (stale) CRC-32 checksum value for chunk [0, 0x3cb3e): Mirror 1: 0x75f5d5fa Mirror 18: 0x75f5d5fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb3e: skipped 000625 1787843141.625240 trunc from 0x011877 to 0x020180 (0xe90a bytes) 000626 1787843141.692295 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cb3e): Mirror 8: 0xc4aa77e2 (stale) Mirror 9: 0xc4aa77e2 (stale) Mirror 12: 0xc4aa77e2 (stale) Mirror 14: 0xc4aa77e2 (stale) Mirror 15: 0xc4aa77e2 (stale) Mirror 16: 0xc4aa77e2 (stale) Mirror 17: 0xaf671f36 (stale) Mirror 18: 0x75f5d5fa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb3e): Mirror 8: 0xc4aa77e2 (stale) Mirror 9: 0xc4aa77e2 (stale) Mirror 12: 0xc4aa77e2 (stale) Mirror 14: 0xc4aa77e2 (stale) Mirror 15: 0xc4aa77e2 (stale) Mirror 16: 0xc4aa77e2 (stale) Mirror 17: 0xaf671f36 (stale) Mirror 18: 0x75f5d5fa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb3e: skipped 000629 1787843142.316893 mapwrite 0x024da3 thru 0x02e9d5 (0x9c33 bytes) 000630 1787843142.391934 trunc from 0x03cb3e to 0x00ccda (0x2fe63 bytes) 000631 1787843142.425334 write 0x01a5cf thru 0x01edbc (0x47ee bytes) 000632 1787843142.426857 read 0x008702 thru 0x014dbf (0xc6be bytes) 000633 1787843142.437133 trunc from 0x011409 to 0x0181d2 (0x6dca bytes) 000634 1787843142.447941 MIRROR_RESYNC 9 mirrors 000635 1787843144.681889 trunc from 0x005627 to 0x00e5d3 (0x8fad bytes) 000636 1787843144.765222 mapwrite 0x03bd83 thru 0x03ffff (0x427d bytes) 000637 1787843144.820167 trunc from 0x040000 to 0x02d3ce (0x12c31 bytes) 000638 1787843144.840219 write 0x013682 thru 0x01b74d (0x80cc bytes) 000640 1787843144.856210 trunc from 0x02d3cf to 0x03e19d (0x10dcf bytes) 000641 1787843144.876444 mapread 0x00dc6a thru 0x015418 (0x77af bytes) 000642 1787843144.885893 read 0x033903 thru 0x0361a2 (0x28a0 bytes) 000643 1787843144.893246 fallocate 0x0306de thru 0x03ed52 (0xe675 bytes) 000644 1787843144.898517 mapread 0x02f673 thru 0x037da5 (0x8733 bytes) 000645 1787843144.907648 mapread 0x02b50b thru 0x03ad0f (0xf805 bytes) 000646 1787843144.917523 trunc from 0x03e19e to 0x028200 (0x15f9d bytes) 000647 1787843144.939237 mapread 0x025320 thru 0x028200 (0x2ee1 bytes) 000648 1787843144.951575 punch from 0x008f4b to 0x00f702 (0x67b8 bytes) 000650 1787843144.961886 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x28201): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28201): Mirror 8: 0xc57bc3c3 (stale) Mirror 9: 0xc57bc3c3 (stale) Mirror 12: 0xc57bc3c3 (stale) Mirror 14: 0xc57bc3c3 (stale) Mirror 15: 0xc57bc3c3 (stale) Mirror 16: 0xc57bc3c3 (stale) Mirror 17: 0xc57bc3c3 (stale) Mirror 18: 0xc57bc3c3 (stale) CRC-32 checksum value for chunk [0, 0x28201): Mirror 1: 0x31d8b867 Mirror 19: 0x31d8b867 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28201: skipped 000651 1787843146.032511 read 0x01a13b thru 0x028200 (0xe0c6 bytes) 000652 1787843146.043166 mapread 0x017686 thru 0x01f291 (0x7c0c bytes) 000653 1787843146.061879 mapwrite 0x015ce5 thru 0x01d9d0 (0x7cec bytes) 000654 1787843146.145511 write 0x016c65 thru 0x024e81 (0xe21d bytes) 000655 1787843146.158450 read 0x009601 thru 0x01905d (0xfa5d bytes) 000656 1787843146.169552 mapwrite 0x03ad23 thru 0x03ffff (0x52dd bytes) 000657 1787843146.224136 trunc from 0x020f46 to 0x02b089 (0xa144 bytes) 000658 1787843146.229677 mapwrite 0x02eb3f thru 0x03af38 (0xc3fa bytes) 000659 1787843146.249328 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xa45b3ab1 (stale) Mirror 9: 0xa45b3ab1 (stale) Mirror 12: 0xa45b3ab1 (stale) Mirror 14: 0xa45b3ab1 (stale) Mirror 15: 0xa45b3ab1 (stale) Mirror 16: 0xa45b3ab1 (stale) Mirror 17: 0xa45b3ab1 (stale) Mirror 18: 0xa45b3ab1 (stale) Mirror 19: 0x35bc2998 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd06c669c Mirror 20: 0xd06c669c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000660 1787843147.309115 write 0x021d4c thru 0x02e529 (0xc7de bytes) 000661 1787843147.369024 mapwrite 0x00bf5d thru 0x00c0f6 (0x19a bytes) 000662 1787843147.395099 mapwrite 0x021db2 thru 0x02d459 (0xb6a8 bytes) 000664 1787843147.416124 write 0x011b45 thru 0x01d502 (0xb9be bytes) 000665 1787843147.427240 mapwrite 0x00cd7b thru 0x017523 (0xa7a9 bytes) 000666 1787843147.458479 read 0x0360cb thru 0x03ffff (0x9f35 bytes) 000667 1787843147.467119 write 0x02a143 thru 0x039e25 (0xfce3 bytes) 000668 1787843147.470860 read 0x03409e thru 0x03e860 (0xa7c3 bytes) 000669 1787843147.472984 fallocate 0x0304af thru 0x03e571 (0xe0c3 bytes) 000670 1787843147.479730 trunc from 0x040000 to 0x026b50 (0x194af bytes) 000671 1787843147.534846 write 0x03d195 thru 0x03ffff (0x2e6b bytes) 000672 1787843147.536075 trunc from 0x00c396 to 0x0187ed (0xc458 bytes) 000673 1787843147.544452 mapread 0x01ee89 thru 0x02d395 (0xe50d bytes) 000674 1787843147.562546 mapread 0x03f549 thru 0x03ffff (0xab7 bytes) 000675 1787843147.569870 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xa45b3ab1 (stale) Mirror 9: 0xa45b3ab1 (stale) Mirror 12: 0xa45b3ab1 (stale) Mirror 14: 0xa45b3ab1 (stale) Mirror 15: 0xa45b3ab1 (stale) Mirror 16: 0xa45b3ab1 (stale) Mirror 17: 0xa45b3ab1 (stale) Mirror 18: 0xa45b3ab1 (stale) Mirror 19: 0x35bc2998 (stale) Mirror 20: 0xd06c669c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x34640ac3 Mirror 21: 0x34640ac3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000676 1787843149.063201 read 0x03330e thru 0x0351df (0x1ed2 bytes) 000677 1787843149.071350 write 0x03ede6 thru 0x03ffff (0x121a bytes) 000678 1787843149.120253 mapread 0x021700 thru 0x02afbe (0x98bf bytes) 000679 1787843149.143110 read 0x018dd9 thru 0x01c64a (0x3872 bytes) 000680 1787843149.155016 read 0x038028 thru 0x03ae23 (0x2dfc bytes) 000682 1787843149.156162 trunc from 0x040000 to 0x0177f2 (0x2880d bytes) 000683 1787843149.194773 fallocate 0x0245ce thru 0x0336e7 (0xf11a bytes) 000685 1787843149.214479 trunc from 0x0177f3 to 0x02ba7d (0x1428b bytes) 000686 1787843149.260184 mapread 0x01a8da thru 0x02388f (0x8fb6 bytes) 000687 1787843149.278124 mapwrite 0x03f4bb thru 0x03ffff (0xb45 bytes) 000688 1787843149.366148 write 0x03bf57 thru 0x03ffff (0x40a9 bytes) 000689 1787843149.370814 trunc from 0x040000 to 0x038d24 (0x72db bytes) 000690 1787843149.417927 trunc from 0x038d25 to 0x034de5 (0x3f3f bytes) 000691 1787843149.490885 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x34de6): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34de6): Mirror 8: 0xae49e45a (stale) Mirror 9: 0xae49e45a (stale) Mirror 12: 0xae49e45a (stale) Mirror 14: 0xae49e45a (stale) Mirror 15: 0xae49e45a (stale) Mirror 16: 0xae49e45a (stale) Mirror 17: 0xae49e45a (stale) Mirror 18: 0xae49e45a (stale) Mirror 19: 0x717b83dd (stale) Mirror 20: 0x95c8f4a1 (stale) Mirror 21: 0x6c2bf905 (stale) CRC-32 checksum value for chunk [0, 0x34de6): Mirror 1: 0x32b42ec4 Mirror 22: 0x32b42ec4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34de6: skipped 000692 1787843150.714183 punch from 0x0334d3 to 0x034de5 (0x1913 bytes) 000694 1787843150.765438 trunc from 0x00b019 to 0x00d729 (0x2711 bytes) 000695 1787843150.772109 punch from 0x013e79 to 0x017d7c (0x3f04 bytes) 000696 1787843150.782919 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x34de6): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34de6): Mirror 8: 0xae49e45a (stale) Mirror 9: 0xae49e45a (stale) Mirror 12: 0xae49e45a (stale) Mirror 14: 0xae49e45a (stale) Mirror 15: 0xae49e45a (stale) Mirror 16: 0xae49e45a (stale) Mirror 17: 0xae49e45a (stale) Mirror 18: 0xae49e45a (stale) Mirror 19: 0x717b83dd (stale) Mirror 20: 0x95c8f4a1 (stale) Mirror 21: 0x6c2bf905 (stale) Mirror 22: 0x32b42ec4 (stale) CRC-32 checksum value for chunk [0, 0x34de6): Mirror 1: 0xfee528a8 Mirror 23: 0xfee528a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34de6: skipped 000697 1787843151.900323 fallocate 0x000347 thru 0x008255 (0x7f0f bytes) 000698 1787843151.955362 MIRROR_RESYNC 14 mirrors 000700 1787843155.117301 punch from 0x018f27 to 0x020cb4 (0x7d8e bytes) 000701 1787843155.215336 write 0x03d030 thru 0x03ffff (0x2fd0 bytes) 000702 1787843155.216705 write 0x008528 thru 0x00dd5d (0x5836 bytes) 000703 1787843155.251139 trunc from 0x040000 to 0x038958 (0x76a7 bytes) 000705 1787843155.357815 MIRROR_SPLIT mirror 2 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38959): Mirror 8: 0x4412840e (stale) Mirror 12: 0x4412840e (stale) Mirror 14: 0x4412840e (stale) Mirror 15: 0x4412840e (stale) Mirror 16: 0x4412840e (stale) Mirror 17: 0x4412840e (stale) Mirror 18: 0x4412840e (stale) Mirror 19: 0x4412840e (stale) Mirror 20: 0x4412840e (stale) Mirror 21: 0x4412840e (stale) Mirror 22: 0x4412840e (stale) Mirror 23: 0x4412840e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38959): Mirror 8: 0x4412840e (stale) Mirror 12: 0x4412840e (stale) Mirror 14: 0x4412840e (stale) Mirror 15: 0x4412840e (stale) Mirror 16: 0x4412840e (stale) Mirror 17: 0x4412840e (stale) Mirror 18: 0x4412840e (stale) Mirror 19: 0x4412840e (stale) Mirror 20: 0x4412840e (stale) Mirror 21: 0x4412840e (stale) Mirror 22: 0x4412840e (stale) Mirror 23: 0x4412840e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38959: skipped 000706 1787843156.587016 fallocate 0x005585 thru 0x00d2f2 (0x7d6e bytes) 000708 1787843156.598165 MIRROR_SPLIT mirror 3 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38959): Mirror 8: 0x4412840e (stale) Mirror 12: 0x4412840e (stale) Mirror 15: 0x4412840e (stale) Mirror 16: 0x4412840e (stale) Mirror 17: 0x4412840e (stale) Mirror 18: 0x4412840e (stale) Mirror 19: 0x4412840e (stale) Mirror 20: 0x4412840e (stale) Mirror 21: 0x4412840e (stale) Mirror 22: 0x4412840e (stale) Mirror 23: 0x4412840e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38959): Mirror 8: 0x4412840e (stale) Mirror 12: 0x4412840e (stale) Mirror 15: 0x4412840e (stale) Mirror 16: 0x4412840e (stale) Mirror 17: 0x4412840e (stale) Mirror 18: 0x4412840e (stale) Mirror 19: 0x4412840e (stale) Mirror 20: 0x4412840e (stale) Mirror 21: 0x4412840e (stale) Mirror 22: 0x4412840e (stale) Mirror 23: 0x4412840e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38959: skipped 000709 1787843157.541324 fallocate 0x02de69 thru 0x035e98 (0x8030 bytes) 000711 1787843157.554460 trunc from 0x038959 to 0x001cd9 (0x36c7f bytes) 000712 1787843157.591457 trunc from 0x001cda to 0x032ca5 (0x30fcc bytes) 000713 1787843157.611406 trunc from 0x032ca6 to 0x01c2c1 (0x169e4 bytes) 000714 1787843157.643838 MIRROR_RESYNC 12 mirrors 000715 1787843159.520548 read 0x0078e6 thru 0x00d93b (0x6056 bytes) 000716 1787843159.531519 trunc from 0x00c742 to 0x00cba0 (0x45f bytes) 000717 1787843159.591524 fallocate 0x02a491 thru 0x02cb70 (0x26e0 bytes) 000718 1787843159.607251 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c2c2): Mirror 8: 0xe8b03072 (stale) Mirror 12: 0xe8b03072 (stale) Mirror 15: 0xe8b03072 (stale) Mirror 17: 0xe8b03072 (stale) Mirror 18: 0xe8b03072 (stale) Mirror 19: 0xe8b03072 (stale) Mirror 20: 0xe8b03072 (stale) Mirror 21: 0xe8b03072 (stale) Mirror 22: 0xe8b03072 (stale) Mirror 23: 0xe8b03072 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c2c2): Mirror 8: 0xe8b03072 (stale) Mirror 12: 0xe8b03072 (stale) Mirror 15: 0xe8b03072 (stale) Mirror 17: 0xe8b03072 (stale) Mirror 18: 0xe8b03072 (stale) Mirror 19: 0xe8b03072 (stale) Mirror 20: 0xe8b03072 (stale) Mirror 21: 0xe8b03072 (stale) Mirror 22: 0xe8b03072 (stale) Mirror 23: 0xe8b03072 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c2c2: skipped 000719 1787843160.203723 trunc from 0x01c2c2 to 0x0258fd (0x963c bytes) 000720 1787843160.223741 mapread 0x01c706 thru 0x0258fd (0x91f8 bytes) 000721 1787843160.244297 trunc from 0x0258fe to 0x00c791 (0x1916c bytes) 000722 1787843160.266439 mapread 0x008509 thru 0x00c791 (0x4289 bytes) 000723 1787843160.276435 fallocate 0x025d15 thru 0x02f1bf (0x94ab bytes) 000724 1787843160.281824 write 0x0003d0 thru 0x0103a7 (0xffd8 bytes) 000725 1787843160.288331 trunc from 0x0103a8 to 0x03a762 (0x2a3bb bytes) 000726 1787843160.305270 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a763): Mirror 8: 0x51852514 (stale) Mirror 12: 0x51852514 (stale) Mirror 15: 0x51852514 (stale) Mirror 17: 0x51852514 (stale) Mirror 18: 0x51852514 (stale) Mirror 19: 0x51852514 (stale) Mirror 21: 0x51852514 (stale) Mirror 22: 0x51852514 (stale) Mirror 23: 0x51852514 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a763): Mirror 8: 0x51852514 (stale) Mirror 12: 0x51852514 (stale) Mirror 15: 0x51852514 (stale) Mirror 17: 0x51852514 (stale) Mirror 18: 0x51852514 (stale) Mirror 19: 0x51852514 (stale) Mirror 21: 0x51852514 (stale) Mirror 22: 0x51852514 (stale) Mirror 23: 0x51852514 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a763: skipped 000727 1787843161.234761 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3a763): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a763): Mirror 8: 0x51852514 (stale) Mirror 12: 0x51852514 (stale) Mirror 15: 0x51852514 (stale) Mirror 17: 0x51852514 (stale) Mirror 18: 0x51852514 (stale) Mirror 19: 0x51852514 (stale) Mirror 21: 0x51852514 (stale) Mirror 22: 0x51852514 (stale) Mirror 23: 0x51852514 (stale) CRC-32 checksum value for chunk [0, 0x3a763): Mirror 1: 0xe8a7fd75 Mirror 24: 0xe8a7fd75 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a763: skipped 000729 1787843163.099064 trunc from 0x016831 to 0x02634e (0xfb1e bytes) 000730 1787843163.161948 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a763): Mirror 8: 0x51852514 (stale) Mirror 12: 0x51852514 (stale) Mirror 15: 0x51852514 (stale) Mirror 17: 0x51852514 (stale) Mirror 18: 0x51852514 (stale) Mirror 19: 0x51852514 (stale) Mirror 21: 0x51852514 (stale) Mirror 22: 0x51852514 (stale) Mirror 24: 0xe8a7fd75 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a763): Mirror 8: 0x51852514 (stale) Mirror 12: 0x51852514 (stale) Mirror 15: 0x51852514 (stale) Mirror 17: 0x51852514 (stale) Mirror 18: 0x51852514 (stale) Mirror 19: 0x51852514 (stale) Mirror 21: 0x51852514 (stale) Mirror 22: 0x51852514 (stale) Mirror 24: 0xe8a7fd75 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a763: skipped 000731 1787843163.706603 read 0x005bf5 thru 0x00b555 (0x5961 bytes) 000732 1787843163.715586 fallocate 0x03bd0f thru 0x03ffff (0x42f1 bytes) 000733 1787843163.722471 mapwrite 0x00d110 thru 0x011a95 (0x4986 bytes) 000734 1787843163.765294 fallocate 0x03b9ba thru 0x03ffff (0x4646 bytes) 000735 1787843163.771216 mapread 0x019355 thru 0x025802 (0xc4ae bytes) 000736 1787843163.804899 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x313e1633 (stale) Mirror 12: 0x313e1633 (stale) Mirror 15: 0x313e1633 (stale) Mirror 17: 0x313e1633 (stale) Mirror 19: 0x313e1633 (stale) Mirror 21: 0x313e1633 (stale) Mirror 22: 0x313e1633 (stale) Mirror 24: 0x387433d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x313e1633 (stale) Mirror 12: 0x313e1633 (stale) Mirror 15: 0x313e1633 (stale) Mirror 17: 0x313e1633 (stale) Mirror 19: 0x313e1633 (stale) Mirror 21: 0x313e1633 (stale) Mirror 22: 0x313e1633 (stale) Mirror 24: 0x387433d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000737 1787843164.438098 read 0x02f4d5 thru 0x037c4f (0x877b bytes) 000738 1787843164.460204 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x313e1633 (stale) Mirror 12: 0x313e1633 (stale) Mirror 15: 0x313e1633 (stale) Mirror 17: 0x313e1633 (stale) Mirror 19: 0x313e1633 (stale) Mirror 21: 0x313e1633 (stale) Mirror 22: 0x313e1633 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x313e1633 (stale) Mirror 12: 0x313e1633 (stale) Mirror 15: 0x313e1633 (stale) Mirror 17: 0x313e1633 (stale) Mirror 19: 0x313e1633 (stale) Mirror 21: 0x313e1633 (stale) Mirror 22: 0x313e1633 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000739 1787843165.260657 mapwrite 0x0285ee thru 0x0357b9 (0xd1cc bytes) 000740 1787843165.397716 punch from 0x01011d to 0x011686 (0x156a bytes) 000741 1787843165.417044 trunc from 0x040000 to 0x014b66 (0x2b499 bytes) 000742 1787843165.507098 fallocate 0x00eabd thru 0x013f3f (0x5483 bytes) 000743 1787843165.525831 write 0x035c48 thru 0x03ffff (0xa3b8 bytes) 000744 1787843165.544196 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x313e1633 (stale) Mirror 12: 0x313e1633 (stale) Mirror 15: 0x313e1633 (stale) Mirror 17: 0x313e1633 (stale) Mirror 19: 0x313e1633 (stale) Mirror 21: 0x313e1633 (stale) Mirror 22: 0x313e1633 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x73bf67b4 Mirror 23: 0x73bf67b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000746 1787843166.728593 trunc from 0x039649 to 0x03c083 (0x2a3b bytes) 000747 1787843166.784801 mapread 0x024879 thru 0x031a29 (0xd1b1 bytes) 000749 1787843166.820451 mapread 0x02698a thru 0x02d677 (0x6cee bytes) 000750 1787843166.821938 punch from 0x02eae1 to 0x034ffc (0x651c bytes) 000751 1787843166.826955 punch from 0x01f116 to 0x021dd0 (0x2cbb bytes) 000752 1787843166.834637 mapwrite 0x03f2ef thru 0x03ffff (0xd11 bytes) 000754 1787843166.858391 mapread 0x021efb thru 0x02cdd8 (0xaede bytes) 000755 1787843166.874812 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x313e1633 (stale) Mirror 12: 0x313e1633 (stale) Mirror 15: 0x313e1633 (stale) Mirror 17: 0x313e1633 (stale) Mirror 19: 0x313e1633 (stale) Mirror 21: 0x313e1633 (stale) Mirror 22: 0x313e1633 (stale) Mirror 23: 0x73bf67b4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x13532c0f Mirror 24: 0x13532c0f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000756 1787843168.220461 trunc from 0x025780 to 0x034520 (0xeda1 bytes) 000757 1787843168.272465 mapread 0x03f81d thru 0x03ffff (0x7e3 bytes) 000758 1787843168.314043 punch from 0x029960 to 0x02df24 (0x45c5 bytes) 000760 1787843168.349992 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x313e1633 (stale) Mirror 12: 0x313e1633 (stale) Mirror 15: 0x313e1633 (stale) Mirror 17: 0x313e1633 (stale) Mirror 21: 0x313e1633 (stale) Mirror 22: 0x313e1633 (stale) Mirror 23: 0x73bf67b4 (stale) Mirror 24: 0x13532c0f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x313e1633 (stale) Mirror 12: 0x313e1633 (stale) Mirror 15: 0x313e1633 (stale) Mirror 17: 0x313e1633 (stale) Mirror 21: 0x313e1633 (stale) Mirror 22: 0x313e1633 (stale) Mirror 23: 0x73bf67b4 (stale) Mirror 24: 0x13532c0f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000761 1787843168.936891 trunc from 0x040000 to 0x001ddd (0x3e222 bytes) 000762 1787843168.961993 mapread 0x00133a thru 0x001ddd (0xaa4 bytes) 000763 1787843168.969157 fallocate 0x0363bd thru 0x03ffff (0x9c43 bytes) 000764 1787843168.974760 trunc from 0x000f91 to 0x001ddd (0xe4d bytes) 000765 1787843168.981486 trunc from 0x001dde to 0x031b27 (0x2fd4a bytes) 000766 1787843169.001766 mapread 0x02ea93 thru 0x031b27 (0x3095 bytes) 000767 1787843169.020572 mapwrite 0x00beeb thru 0x01bc69 (0xfd7f bytes) 000768 1787843169.061982 mapread 0x0266ff thru 0x031b27 (0xb429 bytes) 000769 1787843169.072997 mapwrite 0x01cb3d thru 0x0234f1 (0x69b5 bytes) 000770 1787843169.096324 punch from 0x02ad59 to 0x02f5f7 (0x489f bytes) 000771 1787843169.101534 mapread 0x01acfc thru 0x025b5a (0xae5f bytes) 000772 1787843169.103953 read 0x000480 thru 0x0073ce (0x6f4f bytes) 000773 1787843169.111474 trunc from 0x01a4d8 to 0x02a1b0 (0xfcd9 bytes) 000774 1787843169.133480 fallocate 0x03d636 thru 0x03f7f5 (0x21c0 bytes) 000775 1787843169.146153 read 0x01b38a thru 0x024a5e (0x96d5 bytes) 000776 1787843169.167760 mapread 0x02a98c thru 0x02c2b8 (0x192d bytes) 000777 1787843169.169219 write 0x00d4c1 thru 0x019b18 (0xc658 bytes) 000779 1787843169.172520 write 0x00186e thru 0x0109f4 (0xf187 bytes) 000780 1787843169.174734 write 0x001669 thru 0x00e44a (0xcde2 bytes) 000781 1787843169.177345 read 0x02c5c2 thru 0x032cd8 (0x6717 bytes) 000782 1787843169.178553 read 0x0256a0 thru 0x0321ea (0xcb4b bytes) 000783 1787843169.181019 mapwrite 0x01097a thru 0x01f10d (0xe794 bytes) 000784 1787843169.205664 mapwrite 0x03d2ea thru 0x03df0b (0xc22 bytes) 000785 1787843169.219505 write 0x027c49 thru 0x02a145 (0x24fd bytes) 000786 1787843169.220725 trunc from 0x03f7f6 to 0x02e869 (0x10f8c bytes) 000787 1787843169.250480 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e86a): Mirror 8: 0xfde516d6 (stale) Mirror 15: 0xfde516d6 (stale) Mirror 17: 0xfde516d6 (stale) Mirror 21: 0xfde516d6 (stale) Mirror 22: 0xfde516d6 (stale) Mirror 23: 0xaf383eaa (stale) Mirror 24: 0xaf383eaa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e86a): Mirror 8: 0xfde516d6 (stale) Mirror 15: 0xfde516d6 (stale) Mirror 17: 0xfde516d6 (stale) Mirror 21: 0xfde516d6 (stale) Mirror 22: 0xfde516d6 (stale) Mirror 23: 0xaf383eaa (stale) Mirror 24: 0xaf383eaa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e86a: skipped 000788 1787843169.788816 mapwrite 0x0113fb thru 0x017438 (0x603e bytes) 000789 1787843169.865746 read 0x01e45c thru 0x01f84b (0x13f0 bytes) 000790 1787843169.886307 fallocate 0x001d4f thru 0x006bb3 (0x4e65 bytes) 000791 1787843169.914889 mapread 0x01772e thru 0x025d3f (0xe612 bytes) 000792 1787843169.982735 write 0x00d734 thru 0x019327 (0xbbf4 bytes) 000793 1787843170.001501 trunc from 0x02e86a to 0x02a442 (0x4427 bytes) 000794 1787843170.035237 mapwrite 0x0277fd thru 0x02c533 (0x4d37 bytes) 000795 1787843170.119289 trunc from 0x02bc8f to 0x02c533 (0x8a5 bytes) 000796 1787843170.124410 mapwrite 0x00a12c thru 0x010403 (0x62d8 bytes) 000798 1787843170.152253 read 0x0145f7 thru 0x015d66 (0x1770 bytes) 000800 1787843170.154469 mapwrite 0x02be78 thru 0x02ffa3 (0x412c bytes) 000801 1787843170.205982 trunc from 0x0033f9 to 0x00accd (0x78d5 bytes) 000802 1787843170.215000 punch from 0x00048a to 0x005025 (0x4b9c bytes) 000803 1787843170.223280 read 0x0038d3 thru 0x00d30e (0x9a3c bytes) 000804 1787843170.235456 punch from 0x02eb8e to 0x02ffa3 (0x1416 bytes) 000805 1787843170.241684 punch from 0x0028f2 to 0x005403 (0x2b12 bytes) 000806 1787843170.248688 punch from 0x00a606 to 0x012dae (0x87a9 bytes) 000807 1787843170.257816 punch from 0x003f47 to 0x00b568 (0x7622 bytes) 000808 1787843170.264511 fallocate 0x00b812 thru 0x01b221 (0xfa10 bytes) 000809 1787843170.272759 trunc from 0x02ffa4 to 0x0142ec (0x1bcb7 bytes) 000810 1787843170.300675 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x142ed): Mirror 8: 0xf503643a (stale) Mirror 15: 0xf503643a (stale) Mirror 21: 0xf503643a (stale) Mirror 22: 0xf503643a (stale) Mirror 23: 0x3979a0d4 (stale) Mirror 24: 0x3979a0d4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x142ed): Mirror 8: 0xf503643a (stale) Mirror 15: 0xf503643a (stale) Mirror 21: 0xf503643a (stale) Mirror 22: 0xf503643a (stale) Mirror 23: 0x3979a0d4 (stale) Mirror 24: 0x3979a0d4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x142ed: skipped 000811 1787843170.816285 fallocate 0x02f558 thru 0x03bcad (0xc756 bytes) 000812 1787843170.823523 mapread 0x00b857 thru 0x0142ec (0x8a96 bytes) 000813 1787843170.849745 fallocate 0x009d11 thru 0x016910 (0xcc00 bytes) 000814 1787843170.856794 trunc from 0x010cb2 to 0x012999 (0x1ce8 bytes) 000815 1787843170.869159 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x16911): Mirror 8: (stale - read error) Mirror 15: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16911): Mirror 8: 0x3bc55776 (stale) Mirror 15: 0x3bc55776 (stale) Mirror 21: 0x3bc55776 (stale) Mirror 22: 0x3bc55776 (stale) Mirror 23: 0x78d37e18 (stale) Mirror 24: 0x78d37e18 (stale) CRC-32 checksum value for chunk [0, 0x16911): Mirror 1: 0xb884f5a1 Mirror 25: 0xb884f5a1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16911: skipped 000816 1787843172.337895 punch from 0x01477e to 0x016910 (0x2193 bytes) 000817 1787843172.450724 mapwrite 0x031779 thru 0x039c93 (0x851b bytes) 000819 1787843172.617848 trunc from 0x039c94 to 0x004430 (0x35863 bytes) 000820 1787843172.674749 write 0x03d89e thru 0x03e463 (0xbc6 bytes) 000822 1787843172.679481 read 0x021f60 thru 0x022bd9 (0xc7a bytes) 000823 1787843172.691272 mapwrite 0x02a463 thru 0x02c2f8 (0x1e96 bytes) 000824 1787843172.740244 fallocate 0x038840 thru 0x03ffff (0x77c0 bytes) 000825 1787843172.760338 mapwrite 0x011c65 thru 0x01fb85 (0xdf21 bytes) 000826 1787843172.837932 mapwrite 0x0300ef thru 0x03e4ab (0xe3bd bytes) 000827 1787843172.939018 fallocate 0x00d751 thru 0x014027 (0x68d7 bytes) 000828 1787843172.952980 mapread 0x0391e3 thru 0x03ffff (0x6e1d bytes) 000829 1787843172.963162 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x313e1633 (stale) Mirror 15: 0x313e1633 (stale) Mirror 22: 0x313e1633 (stale) Mirror 23: 0x73bf67b4 (stale) Mirror 24: 0x13532c0f (stale) Mirror 25: 0x36d22bb5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x313e1633 (stale) Mirror 15: 0x313e1633 (stale) Mirror 22: 0x313e1633 (stale) Mirror 23: 0x73bf67b4 (stale) Mirror 24: 0x13532c0f (stale) Mirror 25: 0x36d22bb5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000830 1787843173.781913 punch from 0x01cf43 to 0x024569 (0x7627 bytes) 000831 1787843173.791358 trunc from 0x040000 to 0x0028fa (0x3d705 bytes) 000832 1787843173.844193 trunc from 0x001f98 to 0x0028fa (0x963 bytes) 000833 1787843173.860146 fallocate 0x0258ae thru 0x028d09 (0x345c bytes) 000834 1787843173.873846 trunc from 0x001892 to 0x0028fa (0x1069 bytes) 000835 1787843173.883865 mapwrite 0x015170 thru 0x01abf5 (0x5a86 bytes) 000836 1787843174.021809 punch from 0x000c52 to 0x00d995 (0xcd44 bytes) 000838 1787843174.035089 punch from 0x015c83 to 0x01abf5 (0x4f73 bytes) 000839 1787843174.044418 fallocate 0x03831e thru 0x03ffff (0x7ce2 bytes) 000840 1787843174.051876 fallocate 0x002d35 thru 0x009a88 (0x6d54 bytes) 000841 1787843174.065544 trunc from 0x040000 to 0x0185f8 (0x27a07 bytes) 000842 1787843174.114096 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x185f9): Mirror 8: 0x2e0d052f (stale) Mirror 15: 0x2e0d052f (stale) Mirror 22: 0x2e0d052f (stale) Mirror 23: 0xa68a5b11 (stale) Mirror 24: 0xa68a5b11 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x185f9): Mirror 8: 0x2e0d052f (stale) Mirror 15: 0x2e0d052f (stale) Mirror 22: 0x2e0d052f (stale) Mirror 23: 0xa68a5b11 (stale) Mirror 24: 0xa68a5b11 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x185f9: skipped 000843 1787843174.734401 fallocate 0x03a8dd thru 0x03fe16 (0x553a bytes) 000844 1787843174.750918 fallocate 0x005f46 thru 0x010f86 (0xb041 bytes) 000845 1787843174.758701 trunc from 0x00dc02 to 0x017c31 (0xa030 bytes) 000846 1787843174.772985 punch from 0x010e16 to 0x0120c7 (0x12b2 bytes) 000847 1787843174.782445 mapread 0x00855a thru 0x0144c1 (0xbf68 bytes) 000848 1787843174.835004 trunc from 0x03fe17 to 0x02c576 (0x138a0 bytes) 000849 1787843174.874951 read 0x00712c thru 0x0110e3 (0x9fb8 bytes) 000850 1787843174.883068 MIRROR_RESYNC 6 mirrors 000851 1787843176.445761 mapread 0x01ceb6 thru 0x02a5da (0xd725 bytes) 000852 1787843176.499699 mapread 0x019eb9 thru 0x021f7a (0x80c2 bytes) 000853 1787843176.509694 MIRROR_EXTEND to 7 mirrors 000854 1787843177.378915 punch from 0x002307 to 0x011370 (0xf06a bytes) 000855 1787843177.432746 write 0x03dc31 thru 0x03ffff (0x23cf bytes) 000856 1787843177.438906 trunc from 0x040000 to 0x038912 (0x76ed bytes) 000857 1787843177.458503 punch from 0x010458 to 0x014696 (0x423f bytes) 000858 1787843177.463797 trunc from 0x038913 to 0x029bc0 (0xed52 bytes) 000859 1787843177.485468 read 0x016bde thru 0x01d853 (0x6c76 bytes) 000860 1787843177.495881 write 0x002fe1 thru 0x005230 (0x2250 bytes) 000861 1787843177.510187 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x29bc1): Mirror 8: (stale - read error) Mirror 15: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29bc1): Mirror 8: 0x39e804f1 (stale) Mirror 15: 0x39e804f1 (stale) Mirror 22: 0x39e804f1 (stale) Mirror 23: 0x39e804f1 (stale) Mirror 24: 0x39e804f1 (stale) Mirror 25: 0x39e804f1 (stale) CRC-32 checksum value for chunk [0, 0x29bc1): Mirror 1: 0xbdbdb5a Mirror 26: 0xbdbdb5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29bc1: skipped 000862 1787843178.905315 mapwrite 0x020c6f thru 0x024f8d (0x431f bytes) 000863 1787843178.995115 read 0x01657d thru 0x01bc3c (0x56c0 bytes) 000864 1787843179.002126 trunc from 0x029bc1 to 0x011c63 (0x17f5d bytes) 000865 1787843179.027353 mapread 0x00c818 thru 0x00d23d (0xa26 bytes) 000866 1787843179.039552 mapwrite 0x022c3b thru 0x02b6bb (0x8a81 bytes) 000867 1787843179.110151 mapwrite 0x00ca62 thru 0x01321b (0x67ba bytes) 000868 1787843179.137090 trunc from 0x02b6bc to 0x00acfe (0x209bd bytes) 000869 1787843179.151827 mapread 0x004db5 thru 0x00acfe (0x5f4a bytes) 000870 1787843179.160913 mapwrite 0x014603 thru 0x024201 (0xfbff bytes) 000872 1787843179.215601 punch from 0x0200c1 to 0x024201 (0x4141 bytes) 000873 1787843179.221826 punch from 0x018c26 to 0x01b857 (0x2c32 bytes) 000874 1787843179.227870 mapread 0x00aa3f thru 0x0103b5 (0x5977 bytes) 000876 1787843179.237122 trunc from 0x01c353 to 0x022380 (0x602e bytes) 000877 1787843179.242595 punch from 0x01ceef to 0x01e56b (0x167d bytes) 000878 1787843179.247403 read 0x01c3a3 thru 0x02013e (0x3d9c bytes) 000879 1787843179.263185 punch from 0x00a692 to 0x0162c9 (0xbc38 bytes) 000880 1787843179.269257 trunc from 0x024202 to 0x032ee3 (0xece2 bytes) 000881 1787843179.290842 write 0x03ec32 thru 0x03ffff (0x13ce bytes) 000882 1787843179.292963 fallocate 0x019d3a thru 0x019ff8 (0x2bf bytes) 000883 1787843179.304339 mapwrite 0x00a1af thru 0x00ae03 (0xc55 bytes) 000884 1787843179.339598 write 0x01c273 thru 0x01edc8 (0x2b56 bytes) 000885 1787843179.341294 mapread 0x023350 thru 0x03191c (0xe5cd bytes) 000887 1787843179.379495 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x312373c (stale) Mirror 15: 0x312373c (stale) Mirror 22: 0x312373c (stale) Mirror 23: 0x312373c (stale) Mirror 25: 0x312373c (stale) Mirror 26: 0x178b745b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x312373c (stale) Mirror 15: 0x312373c (stale) Mirror 22: 0x312373c (stale) Mirror 23: 0x312373c (stale) Mirror 25: 0x312373c (stale) Mirror 26: 0x178b745b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000888 1787843180.096014 MIRROR_RESYNC 7 mirrors 000889 1787843184.117429 punch from 0x036421 to 0x039d34 (0x3914 bytes) 000891 1787843184.169494 fallocate 0x00b9c1 thru 0x011949 (0x5f89 bytes) 000892 1787843184.176582 punch from 0x01cd3a to 0x01d560 (0x827 bytes) 000893 1787843184.183155 fallocate 0x024045 thru 0x02ae96 (0x6e52 bytes) 000894 1787843184.190170 trunc from 0x040000 to 0x03e66d (0x1992 bytes) 000896 1787843184.213803 punch from 0x0230d4 to 0x02602f (0x2f5c bytes) 000898 1787843184.223197 fallocate 0x032c1b thru 0x035126 (0x250c bytes) 000899 1787843184.228589 trunc from 0x0389fa to 0x03b694 (0x2c9b bytes) 000900 1787843184.234226 trunc from 0x03e66e to 0x002170 (0x3c4fd bytes) 000901 1787843184.255988 trunc from 0x002171 to 0x02a7e6 (0x28676 bytes) 000902 1787843184.278578 read 0x017ef9 thru 0x019410 (0x1518 bytes) 000903 1787843184.288425 mapread 0x00c1ea thru 0x00c7cb (0x5e2 bytes) 000904 1787843184.295506 fallocate 0x03959c thru 0x03ffff (0x6a64 bytes) 000906 1787843184.311842 mapread 0x00927a thru 0x010822 (0x75a9 bytes) 000907 1787843184.330839 trunc from 0x00854b to 0x0091b8 (0xc6e bytes) 000908 1787843184.336042 fallocate 0x00eb76 thru 0x01b2e3 (0xc76e bytes) 000909 1787843184.341655 trunc from 0x02a7e7 to 0x01461a (0x161cc bytes) 000910 1787843184.361577 mapread 0x00a644 thru 0x01461a (0x9fd7 bytes) 000911 1787843184.370585 write 0x03aa0f thru 0x03ffff (0x55f1 bytes) 000912 1787843184.373104 trunc from 0x040000 to 0x00ba42 (0x345bd bytes) 000913 1787843184.395050 MIRROR_RESYNC 7 mirrors 000914 1787843186.140564 write 0x018298 thru 0x021d19 (0x9a82 bytes) 000915 1787843186.189460 punch from 0x004f7e to 0x009f3e (0x4fc1 bytes) 000916 1787843186.192907 read 0x0005c8 thru 0x003ff1 (0x3a2a bytes) 000917 1787843186.198498 read 0x0108c2 thru 0x014456 (0x3b95 bytes) 000918 1787843186.212057 read 0x005eb6 thru 0x006b14 (0xc5f bytes) 000919 1787843186.213971 trunc from 0x00d498 to 0x01bccb (0xe834 bytes) 000920 1787843186.240184 read 0x0196d3 thru 0x01a211 (0xb3f bytes) 000921 1787843186.261109 punch from 0x003e77 to 0x009323 (0x54ad bytes) 000922 1787843186.273683 trunc from 0x010100 to 0x01f1d4 (0xf0d5 bytes) 000924 1787843186.290767 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x21d1a): Mirror 8: (stale - read error) Mirror 15: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21d1a): Mirror 8: 0x60983400 (stale) Mirror 15: 0x60983400 (stale) Mirror 22: 0x60983400 (stale) Mirror 23: 0x60983400 (stale) Mirror 25: 0x60983400 (stale) Mirror 26: 0x60983400 (stale) CRC-32 checksum value for chunk [0, 0x21d1a): Mirror 1: 0xdb18bf2d Mirror 27: 0xdb18bf2d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21d1a: skipped 000925 1787843187.362648 fallocate 0x017cce thru 0x018d0b (0x103e bytes) 000926 1787843187.418724 write 0x014aa1 thru 0x02020e (0xb76e bytes) 000927 1787843187.448460 read 0x00a60b thru 0x0183dc (0xddd2 bytes) 000930 1787843187.463499 read 0x0061ea thru 0x0109f0 (0xa807 bytes) 000931 1787843187.475829 punch from 0x01f9c1 to 0x021d19 (0x2359 bytes) 000933 1787843187.509532 write 0x013612 thru 0x0146e6 (0x10d5 bytes) 000934 1787843187.510686 read 0x001b6d thru 0x010d9f (0xf233 bytes) 000935 1787843187.511362 punch from 0x016751 to 0x021d19 (0xb5c9 bytes) 000936 1787843187.522297 read 0x01eee3 thru 0x021d19 (0x2e37 bytes) 000937 1787843187.556192 fallocate 0x0289ed thru 0x02cd6b (0x437f bytes) 000939 1787843187.577355 fallocate 0x028fb2 thru 0x02de9d (0x4eec bytes) 000940 1787843187.585440 trunc from 0x02de9e to 0x02ecac (0xe0f bytes) 000941 1787843187.671714 punch from 0x006193 to 0x013a11 (0xd87f bytes) 000942 1787843187.691391 mapwrite 0x01ecf4 thru 0x0214fb (0x2808 bytes) 000943 1787843187.705834 punch from 0x00ff9f to 0x01f790 (0xf7f2 bytes) 000944 1787843187.711167 trunc from 0x02ecad to 0x005f2b (0x28d81 bytes) 000945 1787843187.739021 read 0x005bec thru 0x005f2b (0x340 bytes) 000946 1787843187.740871 fallocate 0x004658 thru 0x0066ed (0x2096 bytes) 000947 1787843187.747327 write 0x022aed thru 0x02bca5 (0x91b9 bytes) 000948 1787843187.750603 mapwrite 0x036261 thru 0x03d10b (0x6eab bytes) 000949 1787843187.803467 trunc from 0x0249b2 to 0x028d78 (0x43c7 bytes) 000950 1787843187.810129 trunc from 0x03d10c to 0x0109dd (0x2c72e bytes) 000951 1787843187.833361 trunc from 0x00a9a1 to 0x00adfb (0x45b bytes) 000952 1787843187.838652 mapread 0x006c5f thru 0x00ab75 (0x3f17 bytes) 000953 1787843187.854576 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x109de): Mirror 8: 0xeeef0b20 (stale) Mirror 15: 0xeeef0b20 (stale) Mirror 22: 0xeeef0b20 (stale) Mirror 23: 0xeeef0b20 (stale) Mirror 25: 0xeeef0b20 (stale) Mirror 26: 0xeeef0b20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x109de): Mirror 8: 0xeeef0b20 (stale) Mirror 15: 0xeeef0b20 (stale) Mirror 22: 0xeeef0b20 (stale) Mirror 23: 0xeeef0b20 (stale) Mirror 25: 0xeeef0b20 (stale) Mirror 26: 0xeeef0b20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x109de: skipped 000954 1787843188.412598 read 0x0054eb thru 0x005d44 (0x85a bytes) 000955 1787843188.419078 write 0x03502b thru 0x03d23b (0x8211 bytes) 000956 1787843188.429360 mapwrite 0x00b159 thru 0x01772f (0xc5d7 bytes) 000957 1787843188.481031 mapread 0x00e79a thru 0x0144e7 (0x5d4e bytes) 000958 1787843188.482148 punch from 0x01114a to 0x020727 (0xf5de bytes) 000959 1787843188.488182 mapread 0x002425 thru 0x0080d9 (0x5cb5 bytes) 000961 1787843188.497005 trunc from 0x01fc55 to 0x02c22d (0xc5d9 bytes) 000962 1787843188.509875 MIRROR_RESYNC 7 mirrors 000963 1787843190.983798 mapwrite 0x029f97 thru 0x037599 (0xd603 bytes) 000964 1787843191.150474 read 0x02f19d thru 0x03b15d (0xbfc1 bytes) 000965 1787843191.151537 fallocate 0x0350e9 thru 0x037ad0 (0x29e8 bytes) 000966 1787843191.159360 write 0x00a76c thru 0x013a85 (0x931a bytes) 000968 1787843191.182314 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d23c): Mirror 8: 0x36255020 (stale) Mirror 15: 0x36255020 (stale) Mirror 22: 0x36255020 (stale) Mirror 23: 0x36255020 (stale) Mirror 25: 0x36255020 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d23c): Mirror 8: 0x36255020 (stale) Mirror 15: 0x36255020 (stale) Mirror 22: 0x36255020 (stale) Mirror 23: 0x36255020 (stale) Mirror 25: 0x36255020 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d23c: skipped 000969 1787843191.651817 mapwrite 0x013757 thru 0x016304 (0x2bae bytes) 000970 1787843191.685595 mapread 0x002ad0 thru 0x0099ad (0x6ede bytes) 000971 1787843191.699089 punch from 0x001642 to 0x005bb2 (0x4571 bytes) 000972 1787843191.704716 mapwrite 0x03487a thru 0x03ffff (0xb786 bytes) 000973 1787843191.764696 read 0x006096 thru 0x015fb5 (0xff20 bytes) 000974 1787843191.766513 trunc from 0x0173a7 to 0x021938 (0xa592 bytes) 000975 1787843191.776697 read 0x0365f5 thru 0x037e30 (0x183c bytes) 000976 1787843191.777296 mapread 0x0065c0 thru 0x01527a (0xecbb bytes) 000977 1787843191.778351 trunc from 0x0376d2 to 0x03ffff (0x892e bytes) 000978 1787843191.793466 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 15: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x6d1ee377 (stale) Mirror 15: 0x6d1ee377 (stale) Mirror 22: 0x6d1ee377 (stale) Mirror 23: 0x6d1ee377 (stale) Mirror 25: 0x6d1ee377 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf0f9c11d Mirror 26: 0xf0f9c11d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000979 1787843193.141076 trunc from 0x040000 to 0x025979 (0x1a686 bytes) 000980 1787843193.228745 MIRROR_RESYNC 7 mirrors 000981 1787843195.553684 fallocate 0x0097b9 thru 0x018872 (0xf0ba bytes) 000984 1787843195.604014 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2597a): Mirror 8: (stale - read error) Mirror 15: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2597a): Mirror 8: 0xa41ea95d (stale) Mirror 15: 0xa41ea95d (stale) Mirror 22: 0xa41ea95d (stale) Mirror 23: 0xa41ea95d (stale) Mirror 25: 0xa41ea95d (stale) Mirror 26: 0xa41ea95d (stale) CRC-32 checksum value for chunk [0, 0x2597a): Mirror 1: 0xa41ea95d Mirror 27: 0xa41ea95d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2597a: skipped 000985 1787843196.810855 mapwrite 0x0210f1 thru 0x021596 (0x4a6 bytes) 000986 1787843196.924373 read 0x005599 thru 0x00960f (0x4077 bytes) 000987 1787843196.944474 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2597a): Mirror 8: 0xa41ea95d (stale) Mirror 15: 0xa41ea95d (stale) Mirror 23: 0xa41ea95d (stale) Mirror 25: 0xa41ea95d (stale) Mirror 26: 0xa41ea95d (stale) Mirror 27: 0xa41ea95d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2597a): Mirror 8: 0xa41ea95d (stale) Mirror 15: 0xa41ea95d (stale) Mirror 23: 0xa41ea95d (stale) Mirror 25: 0xa41ea95d (stale) Mirror 26: 0xa41ea95d (stale) Mirror 27: 0xa41ea95d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2597a: skipped 000988 1787843197.477767 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ad:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2597a): Mirror 8: 0xa41ea95d (stale) Mirror 23: 0xa41ea95d (stale) Mirror 25: 0xa41ea95d (stale) Mirror 26: 0xa41ea95d (stale) Mirror 27: 0xa41ea95d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2597a): Mirror 8: 0xa41ea95d (stale) Mirror 23: 0xa41ea95d (stale) Mirror 25: 0xa41ea95d (stale) Mirror 26: 0xa41ea95d (stale) Mirror 27: 0xa41ea95d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2597a: skipped 000990 1787843198.107271 trunc from 0x02597a to 0x0006f0 (0x25289 bytes) 000991 1787843198.144696 mapread 0x000041 thru 0x0006f0 (0x6b0 bytes) 000992 1787843198.160194 trunc from 0x0006f1 to 0x00fa9e (0xf3ae bytes) 000993 1787843198.177489 mapwrite 0x018aee thru 0x01d0d8 (0x45eb bytes) 000994 1787843198.235488 trunc from 0x01d0d9 to 0x032df4 (0x15d1c bytes) 000995 1787843198.259190 read 0x000375 thru 0x009553 (0x91df bytes) 000996 1787843198.270296 write 0x02aad7 thru 0x03698f (0xbeb9 bytes) 000999 1787843198.277920 trunc from 0x036990 to 0x039a59 (0x30ca bytes) 001000 1787843198.294652 trunc from 0x039a5a to 0x032071 (0x79e8 bytes) All operations completed A-OK! PASS 70a (132s) == sanity-flr test 71: check mirror extend parent fid ==== 11:06:44 (1787843204) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x1d:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 11:06:56 (1787843216) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195777 s, 10.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fa:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990068 s, 10.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fa:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fe:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fa:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fe:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2fd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2fb:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 11:07:06 (1787843226) Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre2 Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2180352 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9363200 .. done Extending file size to 4610624 .. Extending file size to 7030336 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9088704 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 62654 62655 62656 62659 62664 192.168.202.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg223-client.virtnet /mnt/lustre2 (opts:) 192.168.202.123@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg223-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:08:31 (1787843311) Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre2 Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1493344 .. Extending file size to 7461472 .. done Extending file size to 2926784 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 6151840 .. Extending file size to 6389792 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8869120 .. failed Extending file size to 7143200 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7670304 .. Extending file size to 8628544 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8705952 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 64358 64359 64361 64364 64365 done 192.168.202.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg223-client.virtnet /mnt/lustre2 (opts:) 192.168.202.123@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg223-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (86s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:09:57 (1787843397) Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2712 1268592 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1864 1269440 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 23776 3583244 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18460 3588560 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42236 7171804 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2712 1268592 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1864 1269440 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 23776 3583244 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18460 3588560 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42236 7171804 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0831812 s, 25.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.259702 s, 8.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x860b574d (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.202.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg223-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:10:19 (1787843419) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.158528 s, 13.2 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 11:10:27 (1787843427) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x202:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x31a:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x31a:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x31b:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x316:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:10:34 (1787843434) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0811479 s, 25.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x31c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x317:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x318:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:10:44 (1787843444) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.135272 s, 15.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x31e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x319:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x31e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x319:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31a:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:10:53 (1787843453) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.126319 s, 16.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x31f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x31f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31c:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:11:27 (1787843487) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg223-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0648657 s, 16.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x30:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x30:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2f:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:12:21 (1787843541) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg223-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134663 s, 15.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x320:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x31e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x320:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (49s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:13:10 (1787843590) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg223-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg223-server: Pool lustre.test_204f created Adding targets to pool oleg223-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg223-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138337 s, 15.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x31:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x31:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg223-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg223-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg223-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:14:25 (1787843665) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:14:33 (1787843673) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:14:42 (1787843682) oleg223-server: Pool lustre.test_206 created Adding targets to pool oleg223-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg223-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg223-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg223-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg223-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:15:08 (1787843708) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450539 s, 23.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565689 s, 18.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xb6e02777 Mirror 3: 0xb6e02777 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:15:19 (1787843719) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:15:24 (1787843724) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:15:28 (1787843728) Starting client: oleg223-client.virtnet: -o user_xattr,flock 192.168.202.123@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.056997 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xa50000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1cc0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1cc0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1cc0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x12c0000 Read this (4 bytes): BBBB 192.168.202.123@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg223-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:16:10 (1787843770) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0312754 s, 262 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:16:18 (1787843778) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597783 s, 17.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x223 lmm_fid: [0x200000405:0x223:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x223 lmm_fid: [0x200000405:0x223:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x329:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611683 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 11:16:20.000000000 -0400 Modify: 2026-08-27 11:16:20.000000000 -0400 Change: 2026-08-27 11:16:21.000000000 -0400 Birth: 2026-08-27 11:16:19.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:16:26 (1787843786) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833292 s, 12.6 MB/s oleg223-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg223-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg223-server: O/280000401/d10/32a: File not found by ext2_lookup PASS 210b (24s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:16:51 (1787843811) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.272579 s, 150 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0281163 s, 146 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 11:16 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x228:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 11:16 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:17:00 (1787843820) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:17:09 (1787843829) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1975 sec =========== 11:17:20 (1787843840) === sanity-flr: start cleanup 11:17:22 (1787843842) === === sanity-flr: finish cleanup 11:17:31 (1787843851) ===