-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 14:35:29 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg150-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 14:35:49 (1787855749) === oleg150-client.virtnet: executing check_config_client /mnt/lustre oleg150-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg150-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ac045ff0000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ac045ff0000.idle_timeout=debug disable quota as required oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 14:36:35 (1787855795) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8218] == sanity-flr test 0a: lfs mirror create with -N option == 14:36:38 (1787855798) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 14:36:59 (1787855819) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 14:37:02 (1787855822) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 14:37:05 (1787855825) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 14:37:34 (1787855854) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 14:37:39 (1787855859) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (5s) == sanity-flr test 0g: lfs mirror create flags support === 14:37:44 (1787855864) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9ac045ff0000.stats=clear osc.lustre-OST0001-osc-ffff9ac045ff0000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 14:37:54 (1787855874) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 14:38:06 (1787855886) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 14:38:15 (1787855895) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 14:38:19 (1787855899) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 14:38:28 (1787855908) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 14:38:38 (1787855918) PASS 2 (10s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 14:38:48 (1787855928) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 14:38:56 (1787855936) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 14:39:04 (1787855944) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 14:39:16 (1787855956) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 14:39:23 (1787855963) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 1.26135 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961881 s, 10.9 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 14:39:34 (1787855974) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.649275 s, 30.7 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 14:39:43 (1787855983) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 14:39:55 (1787855995) Stopping /mnt/lustre-ost1 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ac045ff0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ac045ff0000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg150-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ac045ff0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ac045ff0000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg150-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (78s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 14:41:13 (1787856073) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.416109 s, 22.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ac045ff0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ac045ff0000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg150-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 14:41:59 (1787856119) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ac045ff0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ac045ff0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg150-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ac045ff0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ac045ff0000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg150-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (93s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 14:43:33 (1787856213) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.24618 s, 16.2 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 14:43 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.029s user 0m0.001s sys 0m0.184s Stopping /mnt/lustre-ost1 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9ac045ff0000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9ac045ff0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.048s user 0m0.001s sys 0m0.201s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg150-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg150-server oleg150-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9ac045ff0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ac045ff0000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.326s user 0m0.000s sys 0m0.181s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg150-server: oleg150-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg150-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg150-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ac045ff0000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9ac045ff0000.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 14:45:11 (1787856311) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 14:45:14 (1787856314) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 14:45:17 (1787856317) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 14:45:21 (1787856321) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 14:45:29 (1787856329) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.608166 s, 20.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 14:45:43 (1787856343) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0543387 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0288299 s, 36.4 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 14:45:52 (1787856352) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 14:46:01 (1787856361) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0017154 s, 597 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00112531 s, 910 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0013514 s, 758 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000906173 s, 1.1 MB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 14:46:11 (1787856371) create test files with size 15745930 .. b406cd6be545b4bde97b32123b7b3ac7 - f72160bd8924d10ba5458a732b2b4a5d - 12d0cfec36c5081b8664833a003e69a2 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg150-client.virtnet /mnt/lustre (opts:) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 14:46:38 (1787856398) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 14:46:53 (1787856413) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 14:47:00 (1787856420) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12948 s, 8.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545714 s, 19.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 14:47:11 (1787856431) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0579337 s, 36.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.204428 s, 20.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117763 s, 17.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 14:47:26 (1787856446) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 14:47:30 (1787856450) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 14:47:34 (1787856454) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449293 s, 23.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 43c: read prefer uncompressed mirror == 14:47:42 (1787856462) llite.lustre-ffff9ac061a7d800.enable_compression=1 striped dir -i1 -c2 -H all_char /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (8s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 14:47:50 (1787856470) llite.lustre-ffff9ac061a7d800.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330547 s, 31.7 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328584 s, 31.9 MB/s **verify components PASS 43d (9s) == sanity-flr test 44a: lfs mirror split check =========== 14:48:00 (1787856480) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.357292 s, 8.8 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 14:48:12 (1787856492) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcc:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcc:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 14:48:24 (1787856504) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.658079 s, 15.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x9:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg150-client.virtnet /mnt/lustre (opts:) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (16s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 14:48:41 (1787856521) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9ac04640b800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.302973 s, 34.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg150-client.virtnet /mnt/lustre (opts:) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9ac072888000.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 14:48:58 (1787856538) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 14:49:38 (1787856578) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (43s) == sanity-flr test 47: Verify mirror obj alloc =========== 14:50:22 (1787856622) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 14:50:26 (1787856626) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.275968 s, 11.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.26964 s, 11.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x150414a9 (nosync,stale) Mirror 2: 0x150414a9 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 850a5cd49c5fb5338880701276545ace - mirror 1 checksum: 68cfdfac267f2a507e656d58ee95eee4 - mirror 2 checksum: 850a5cd49c5fb5338880701276545ace - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T14:50:29-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T14:50:29-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 850a5cd49c5fb5338880701276545ace - mirror 1 checksum: 68cfdfac267f2a507e656d58ee95eee4 - mirror 2 checksum: 68cfdfac267f2a507e656d58ee95eee4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T14:50:29-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T14:50:29-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd7:0x0] } PASS 48 (13s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 14:50:39 (1787856639) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0231972 s, 45.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 14:50:47 (1787856647) Failing mds1 on oleg150-server Stopping /mnt/lustre-mds1 (opts:) on oleg150-server 14:50:59 (1787856659) shut down facet: mds1 facet_host: oleg150-server facet_failover_host: oleg150-server Failover mds1 to oleg150-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Started lustre-MDT0000 14:51:35 (1787856695) targets are mounted 14:51:35 (1787856695) facet_failover done oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331303 s, 31.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd9:0x0] } PASS 50A (68s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 14:51:55 (1787856715) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0314665 s, 651 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.017993 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 14:52:06 (1787856726) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0735889 s, 278 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0105781 s, 1.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 14:52:16 (1787856736) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0165743 s, 989 kB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 14:52:24 (1787856744) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.07616 s, 269 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 14:52:32 (1787856752) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00564916 s, 725 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0374613 s, 109 kB/s PASS 60a (11s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 14:52:44 (1787856764) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 14:53:08 (1787856788) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 14:53:38 (1787856818) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 14:54:03 (1787856843) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 14:54:28 (1787856868) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 14:54:36 (1787856876) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 58852 mirror split pid 58853 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58852 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58853 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 14:55:43 (1787856943) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 6946 fallocating to largest ever: 0x3f4ce 000002 1787856946.640512 fallocate 0x03e61f thru 0x03f4cd (0xeaf bytes) 000003 1787856946.649985 mapwrite 0x0064b1 thru 0x014545 (0xe095 bytes) 000004 1787856946.781495 punch from 0x03706a to 0x03f4cd (0x8464 bytes) 000005 1787856946.800773 read 0x03afd2 thru 0x03f4cd (0x44fc bytes) 000006 1787856946.825069 mapwrite 0x03626a thru 0x039dea (0x3b81 bytes) 000007 1787856946.870278 write 0x032481 thru 0x036f6a (0x4aea bytes) 000010 1787856946.884256 fallocate 0x0393d7 thru 0x03a921 (0x154b bytes) 000012 1787856946.903535 read 0x012e78 thru 0x01ac28 (0x7db1 bytes) 000013 1787856946.922636 write 0x017424 thru 0x024ce9 (0xd8c6 bytes) 000014 1787856946.924172 read 0x023937 thru 0x026189 (0x2853 bytes) 000016 1787856946.928123 mapread 0x02dd28 thru 0x02e0e0 (0x3b9 bytes) 000017 1787856946.935815 punch from 0x006980 to 0x01188c (0xaf0d bytes) fallocating to largest ever: 0x40000 000018 1787856946.959245 fallocate 0x03d268 thru 0x03ffff (0x2d98 bytes) 000020 1787856946.972330 trunc from 0x003df8 to 0x0109fb (0xcc04 bytes) 000021 1787856946.997768 trunc from 0x031650 to 0x03a7e0 (0x9191 bytes) 000023 1787856947.026182 write 0x014356 thru 0x022f6b (0xec16 bytes) 000024 1787856947.028224 mapread 0x015d5c thru 0x016f22 (0x11c7 bytes) 000025 1787856947.032290 trunc from 0x01917b to 0x0233e0 (0xa266 bytes) 000026 1787856947.068318 punch from 0x03e4fd to 0x03f4cd (0xfd1 bytes) 000027 1787856947.088625 trunc from 0x03f4ce to 0x011b06 (0x2d9c7 bytes) 000028 1787856947.118845 mapread 0x00484b thru 0x008917 (0x40cd bytes) 000029 1787856947.151312 mapwrite 0x0100b1 thru 0x01d4e0 (0xd430 bytes) 000031 1787856947.245414 mapwrite 0x030801 thru 0x034a73 (0x4273 bytes) 000034 1787856947.361053 MIRROR_EXTEND to 2 mirrors 000035 1787856948.314622 fallocate 0x00c89c thru 0x019793 (0xcef8 bytes) 000036 1787856948.369638 write 0x016542 thru 0x017baa (0x1669 bytes) 000037 1787856948.391486 read 0x001f63 thru 0x0103b3 (0xe451 bytes) 000038 1787856948.409957 mapwrite 0x03adb8 thru 0x03ffff (0x5248 bytes) 000040 1787856948.529262 write 0x012d2d thru 0x012ff8 (0x2cc bytes) 000041 1787856948.537615 trunc from 0x01399e to 0x01f219 (0xb87c bytes) 000042 1787856948.546954 write 0x022f23 thru 0x029294 (0x6372 bytes) 000043 1787856948.558949 punch from 0x01fa6a to 0x020829 (0xdc0 bytes) 000044 1787856948.569461 trunc from 0x03343c to 0x037465 (0x402a bytes) 000045 1787856948.574949 trunc from 0x025a48 to 0x0317c9 (0xbd82 bytes) 000047 1787856948.597853 mapread 0x03d770 thru 0x03ffff (0x2890 bytes) 000049 1787856948.602155 mapread 0x037602 thru 0x03ffff (0x89fe bytes) 000050 1787856948.617960 mapwrite 0x0144b0 thru 0x017bd4 (0x3725 bytes) 000051 1787856948.649495 trunc from 0x040000 to 0x00b362 (0x34c9d bytes) 000052 1787856948.666328 fallocate 0x02de22 thru 0x03dc0f (0xfdee bytes) 000053 1787856948.673846 write 0x031230 thru 0x03fb7b (0xe94c bytes) 000055 1787856948.677942 read 0x01ce57 thru 0x02171a (0x48c4 bytes) 000056 1787856948.688169 read 0x035e46 thru 0x03fb7b (0x9d36 bytes) 000057 1787856948.689285 write 0x03b945 thru 0x03e3a2 (0x2a5e bytes) 000058 1787856948.691310 write 0x03c1ce thru 0x03ffff (0x3e32 bytes) 000059 1787856948.699347 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7127281c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2f7337b0 Mirror 3: 0x2f7337b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000060 1787856949.646345 mapwrite 0x000e5b thru 0x00ace6 (0x9e8c bytes) 000061 1787856949.763264 read 0x02a183 thru 0x0311f0 (0x706e bytes) 000062 1787856949.776718 mapwrite 0x020e40 thru 0x02ff1a (0xf0db bytes) 000063 1787856949.827776 MIRROR_RESYNC 3 mirrors 000065 1787856950.969924 mapwrite 0x03df1a thru 0x03ffff (0x20e6 bytes) 000066 1787856951.112682 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4c06c545 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4c06c545 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000067 1787856951.436246 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4c06c545 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x49b52f1c Mirror 3: 0x49b52f1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000068 1787856952.822235 MIRROR_RESYNC 3 mirrors 000069 1787856953.570123 mapwrite 0x00e8b3 thru 0x012e97 (0x45e5 bytes) 000070 1787856953.715071 mapwrite 0x038692 thru 0x03ffff (0x796e bytes) 000071 1787856953.783170 write 0x026089 thru 0x02a3e9 (0x4361 bytes) 000072 1787856953.793964 punch from 0x01d540 to 0x022711 (0x51d2 bytes) 000073 1787856953.805580 write 0x0001d4 thru 0x003fec (0x3e19 bytes) 000074 1787856953.820416 fallocate 0x00ffa6 thru 0x0188ba (0x8915 bytes) 000075 1787856953.840863 write 0x008159 thru 0x0169ef (0xe897 bytes) 000076 1787856953.843045 mapread 0x006c0d thru 0x00af8d (0x4381 bytes) 000077 1787856953.857900 trunc from 0x00c014 to 0x0112de (0x52cb bytes) 000078 1787856953.921599 punch from 0x00c525 to 0x01236f (0x5e4b bytes) 000079 1787856953.929062 punch from 0x008052 to 0x0129ea (0xa999 bytes) 000080 1787856953.951307 fallocate 0x018914 thru 0x023611 (0xacfe bytes) 000082 1787856953.967970 mapread 0x024fa8 thru 0x0276b4 (0x270d bytes) 000083 1787856954.002531 write 0x00cd2f thru 0x00ee3e (0x2110 bytes) 000084 1787856954.037284 punch from 0x02f207 to 0x03752a (0x8324 bytes) 000085 1787856954.065277 MIRROR_RESYNC 3 mirrors 000086 1787856955.267028 MIRROR_RESYNC 3 mirrors 000087 1787856955.772055 trunc from 0x040000 to 0x0025c7 (0x3da38 bytes) 000088 1787856955.874820 read 0x000b0d thru 0x0025c7 (0x1abb bytes) 000089 1787856955.887900 punch from 0x00120a to 0x0025c7 (0x13be bytes) 000091 1787856955.895145 write 0x014e6c thru 0x015e33 (0xfc8 bytes) 000092 1787856955.904940 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15e34): Mirror 3: 0xcbb9251b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15e34): Mirror 3: 0xcbb9251b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15e34: skipped 000093 1787856956.330627 punch from 0x0019ef to 0x0112ba (0xf8cc bytes) 000094 1787856956.336622 trunc from 0x015e34 to 0x023f9f (0xe16c bytes) 000095 1787856956.364333 fallocate 0x036c44 thru 0x03ffff (0x93bc bytes) 000096 1787856956.388473 trunc from 0x023fa0 to 0x011825 (0x1277a bytes) 000097 1787856956.424811 punch from 0x00cdc2 to 0x00d386 (0x5c5 bytes) 000098 1787856956.430106 read 0x011164 thru 0x011825 (0x6c2 bytes) 000099 1787856956.455479 trunc from 0x00e671 to 0x011825 (0x31b5 bytes) 000100 1787856956.481775 trunc from 0x002c61 to 0x0059be (0x2d5e bytes) 000101 1787856956.489807 mapwrite 0x0397a9 thru 0x03ffff (0x6857 bytes) 000102 1787856956.600975 trunc from 0x03d70f to 0x03ffff (0x28f1 bytes) 000103 1787856956.615928 write 0x010d9d thru 0x012338 (0x159c bytes) 000104 1787856956.634683 mapwrite 0x003229 thru 0x009224 (0x5ffc bytes) 000106 1787856956.731110 read 0x01a698 thru 0x01f745 (0x50ae bytes) 000107 1787856956.754926 fallocate 0x008900 thru 0x00b660 (0x2d61 bytes) 000108 1787856956.764563 read 0x0296fc thru 0x02c2db (0x2be0 bytes) 000109 1787856956.765560 fallocate 0x0014af thru 0x00e3be (0xcf10 bytes) 000110 1787856956.775564 mapread 0x02bbc4 thru 0x031fd0 (0x640d bytes) 000111 1787856956.776490 mapwrite 0x009267 thru 0x015760 (0xc4fa bytes) 000112 1787856956.808039 read 0x035830 thru 0x03ffff (0xa7d0 bytes) 000113 1787856956.808709 mapread 0x0039e7 thru 0x012d1c (0xf336 bytes) 000114 1787856956.809672 write 0x00c78f thru 0x0134b8 (0x6d2a bytes) 000115 1787856956.815502 write 0x0171a0 thru 0x02103d (0x9e9e bytes) 000117 1787856956.817467 read 0x0170d4 thru 0x0255a3 (0xe4d0 bytes) 000118 1787856956.819033 write 0x01af14 thru 0x02539e (0xa48b bytes) 000119 1787856956.821273 trunc from 0x040000 to 0x037cff (0x8300 bytes) 000120 1787856956.851237 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37d00): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37d00): Mirror 3: 0x5bbce1c9 (stale) CRC-32 checksum value for chunk [0, 0x37d00): Mirror 1: 0x434b8d50 Mirror 4: 0x434b8d50 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37d00: skipped 000121 1787856958.124051 trunc from 0x00df7a to 0x011594 (0x361b bytes) 000123 1787856958.185804 mapwrite 0x00b3c8 thru 0x01382c (0x8465 bytes) 000124 1787856958.231920 mapread 0x02e64e thru 0x037cff (0x96b2 bytes) 000125 1787856958.251485 mapread 0x023321 thru 0x02d3c6 (0xa0a6 bytes) 000126 1787856958.263474 trunc from 0x00ce2f to 0x00e9bd (0x1b8f bytes) 000127 1787856958.269162 write 0x022234 thru 0x0292b7 (0x7084 bytes) 000128 1787856958.275353 punch from 0x01938b to 0x027086 (0xdcfc bytes) 000129 1787856958.300456 read 0x01fad6 thru 0x02ecfd (0xf228 bytes) 000130 1787856958.308547 mapwrite 0x00c8da thru 0x012778 (0x5e9f bytes) 000132 1787856958.330461 fallocate 0x015e21 thru 0x024299 (0xe479 bytes) 000133 1787856958.335270 fallocate 0x015c78 thru 0x016318 (0x6a1 bytes) 000135 1787856958.340475 mapread 0x0065f5 thru 0x009e78 (0x3884 bytes) 000136 1787856958.341784 trunc from 0x0241b5 to 0x02ab06 (0x6952 bytes) 000137 1787856958.346714 trunc from 0x00ef06 to 0x015f30 (0x702b bytes) 000141 1787856958.354660 punch from 0x02c2ac to 0x037b1a (0xb86f bytes) 000142 1787856958.359843 read 0x004163 thru 0x010186 (0xc024 bytes) 000143 1787856958.371100 mapwrite 0x007625 thru 0x00d54d (0x5f29 bytes) 000145 1787856958.388170 fallocate 0x02d1dc thru 0x02d630 (0x455 bytes) 000146 1787856958.393704 mapread 0x0039c3 thru 0x0135b6 (0xfbf4 bytes) 000147 1787856958.394753 trunc from 0x037d00 to 0x02b834 (0xc4cb bytes) 000148 1787856958.425413 write 0x01a19f thru 0x024e8c (0xacee bytes) 000149 1787856958.428440 trunc from 0x008ef1 to 0x00bc93 (0x2da3 bytes) 000150 1787856958.434227 read 0x0293c3 thru 0x02b834 (0x2472 bytes) 000151 1787856958.435222 mapwrite 0x0359c9 thru 0x03ffff (0xa637 bytes) 000152 1787856958.494296 punch from 0x0109d0 to 0x01f385 (0xe9b6 bytes) 000153 1787856958.509755 read 0x033946 thru 0x03efdf (0xb69a bytes) 000155 1787856958.512990 trunc from 0x00cad2 to 0x00e715 (0x1c44 bytes) 000156 1787856958.518865 fallocate 0x023134 thru 0x026019 (0x2ee6 bytes) 000157 1787856958.527341 mapread 0x022a9c thru 0x026451 (0x39b6 bytes) 000158 1787856958.534555 punch from 0x00f576 to 0x013c09 (0x4694 bytes) 000159 1787856958.557191 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x209443f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x209443f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000160 1787856958.860967 write 0x0282b9 thru 0x0338ad (0xb5f5 bytes) 000161 1787856958.877643 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x209443f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1ab1a10f Mirror 4: 0x1ab1a10f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000163 1787856960.075784 read 0x022f21 thru 0x02c302 (0x93e2 bytes) 000164 1787856960.089270 mapwrite 0x023844 thru 0x025931 (0x20ee bytes) 000165 1787856960.166953 fallocate 0x00f4e3 thru 0x01d744 (0xe262 bytes) 000167 1787856960.175299 read 0x00df86 thru 0x015815 (0x7890 bytes) 000168 1787856960.186363 mapread 0x03b799 thru 0x03ffff (0x4867 bytes) 000169 1787856960.187482 mapwrite 0x0232d4 thru 0x0301f3 (0xcf20 bytes) 000171 1787856960.206887 trunc from 0x03d47f to 0x03ffff (0x2b81 bytes) 000172 1787856960.212145 read 0x021fcb thru 0x02a0e8 (0x811e bytes) 000174 1787856960.213483 mapwrite 0x028d28 thru 0x0358c4 (0xcb9d bytes) 000175 1787856960.233831 mapread 0x020ab8 thru 0x0276d3 (0x6c1c bytes) 000176 1787856960.234751 trunc from 0x0311e7 to 0x03df8a (0xcda4 bytes) 000177 1787856960.240132 mapwrite 0x037faa thru 0x03ffff (0x8056 bytes) 000178 1787856960.269784 trunc from 0x040000 to 0x0368b1 (0x974e bytes) 000179 1787856960.301289 fallocate 0x02e450 thru 0x0327f0 (0x43a1 bytes) 000180 1787856960.306799 write 0x02f872 thru 0x03cee8 (0xd677 bytes) 000181 1787856960.308935 fallocate 0x0390aa thru 0x03ffff (0x6f56 bytes) 000183 1787856960.315588 fallocate 0x01bd2b thru 0x01be91 (0x167 bytes) 000184 1787856960.320116 mapwrite 0x03d62c thru 0x03ffff (0x29d4 bytes) 000185 1787856960.363092 trunc from 0x0346c8 to 0x0371b7 (0x2af0 bytes) 000186 1787856960.368677 fallocate 0x023d66 thru 0x02d4c4 (0x975f bytes) 000187 1787856960.374549 fallocate 0x0360de thru 0x037c1a (0x1b3d bytes) 000188 1787856960.381925 punch from 0x03727b to 0x03c5ca (0x5350 bytes) 000189 1787856960.390965 fallocate 0x012f00 thru 0x01ef70 (0xc071 bytes) 000191 1787856960.399346 mapwrite 0x0275be thru 0x035336 (0xdd79 bytes) 000192 1787856960.445324 punch from 0x02df56 to 0x02f830 (0x18db bytes) 000193 1787856960.450246 mapwrite 0x0002ed thru 0x0058fe (0x5612 bytes) 000194 1787856960.469140 trunc from 0x028b85 to 0x02e36a (0x57e6 bytes) 000195 1787856960.474321 punch from 0x00a07a to 0x010485 (0x640c bytes) 000196 1787856960.479576 punch from 0x039a40 to 0x03ffff (0x65c0 bytes) 000197 1787856960.484880 trunc from 0x010864 to 0x01b3d4 (0xab71 bytes) 000199 1787856960.495525 MIRROR_RESYNC 3 mirrors 000201 1787856961.443967 trunc from 0x001620 to 0x007c15 (0x65f6 bytes) 000202 1787856961.529124 trunc from 0x00d6fe to 0x017ea8 (0xa7ab bytes) 000204 1787856961.550082 trunc from 0x040000 to 0x02225b (0x1dda4 bytes) 000205 1787856961.589098 mapread 0x01db57 thru 0x02225b (0x4705 bytes) 000206 1787856961.664946 write 0x017a01 thru 0x01df1a (0x651a bytes) 000207 1787856961.676122 mapwrite 0x02bd8b thru 0x03b527 (0xf79d bytes) 000208 1787856961.826726 MIRROR_RESYNC 3 mirrors 000209 1787856963.056088 fallocate 0x010cde thru 0x01b0c8 (0xa3eb bytes) 000210 1787856963.098483 mapwrite 0x008d39 thru 0x00ae8c (0x2154 bytes) 000211 1787856963.131433 trunc from 0x032ace to 0x037bb2 (0x50e5 bytes) 000212 1787856963.141317 write 0x003366 thru 0x011dc1 (0xea5c bytes) 000213 1787856963.143735 trunc from 0x03b528 to 0x02ce63 (0xe6c4 bytes) 000214 1787856963.166478 write 0x0241ed thru 0x026b73 (0x2987 bytes) 000215 1787856963.178928 trunc from 0x011475 to 0x020059 (0xebe5 bytes) 000216 1787856963.200195 punch from 0x005e1e to 0x011555 (0xb738 bytes) 000217 1787856963.206311 mapread 0x01de0e thru 0x02c304 (0xe4f7 bytes) 000218 1787856963.217207 punch from 0x004d5e to 0x012f7e (0xe221 bytes) 000219 1787856963.221880 write 0x000514 thru 0x0016c0 (0x11ad bytes) 000220 1787856963.230592 read 0x026e0b thru 0x02ce63 (0x6059 bytes) 000221 1787856963.231353 read 0x0030b9 thru 0x00d4ad (0xa3f5 bytes) 000222 1787856963.241877 punch from 0x011698 to 0x01d742 (0xc0ab bytes) 000223 1787856963.246558 mapread 0x022976 thru 0x0240dc (0x1767 bytes) 000224 1787856963.247526 mapwrite 0x0000a7 thru 0x00af13 (0xae6d bytes) 000225 1787856963.268665 mapwrite 0x02a916 thru 0x038002 (0xd6ed bytes) 000226 1787856963.313157 mapread 0x00fb5f thru 0x0183d6 (0x8878 bytes) 000227 1787856963.382948 write 0x0128b4 thru 0x0196e6 (0x6e33 bytes) 000229 1787856963.407983 write 0x02786a thru 0x02be38 (0x45cf bytes) 000230 1787856963.412588 mapwrite 0x027d2c thru 0x03053e (0x8813 bytes) 000231 1787856963.479513 write 0x02f873 thru 0x037287 (0x7a15 bytes) 000232 1787856963.480586 mapwrite 0x0254fd thru 0x0290b1 (0x3bb5 bytes) 000233 1787856963.506024 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38003): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38003): Mirror 3: 0x660f1ad1 (stale) Mirror 4: 0x660f1ad1 (stale) CRC-32 checksum value for chunk [0, 0x38003): Mirror 1: 0x8f7e92cc Mirror 5: 0x8f7e92cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38003: skipped 000234 1787856964.627553 write 0x029222 thru 0x02e840 (0x561f bytes) 000235 1787856964.758841 MIRROR_RESYNC 4 mirrors 000236 1787856966.002589 trunc from 0x0298d6 to 0x02ae62 (0x158d bytes) 000237 1787856966.088319 mapwrite 0x03dcef thru 0x03ffff (0x2311 bytes) 000238 1787856966.173194 trunc from 0x040000 to 0x012d76 (0x2d289 bytes) 000239 1787856966.201100 trunc from 0x00db0d to 0x012d76 (0x526a bytes) 000240 1787856966.211534 punch from 0x006910 to 0x00d2a1 (0x6992 bytes) 000241 1787856966.217409 punch from 0x001d96 to 0x007ec4 (0x612f bytes) 000242 1787856966.223330 punch from 0x00724d to 0x012d76 (0xbb2a bytes) 000243 1787856966.228529 read 0x00da08 thru 0x012d76 (0x536f bytes) 000244 1787856966.240990 read 0x009315 thru 0x012d76 (0x9a62 bytes) 000245 1787856966.251294 trunc from 0x012d77 to 0x0008c3 (0x124b3 bytes) 000247 1787856966.273457 trunc from 0x0008c4 to 0x00f4f7 (0xec34 bytes) 000248 1787856966.289726 mapwrite 0x01224b thru 0x02007f (0xde35 bytes) 000249 1787856966.340061 fallocate 0x037927 thru 0x03ffff (0x86d9 bytes) 000250 1787856966.347033 fallocate 0x01128b thru 0x020f60 (0xfcd6 bytes) 000251 1787856966.353626 mapread 0x01899d thru 0x028237 (0xf89b bytes) 000252 1787856966.378110 mapwrite 0x0358df thru 0x03ffff (0xa721 bytes) 000253 1787856966.396874 write 0x010d94 thru 0x016c71 (0x5ede bytes) 000254 1787856966.403478 read 0x03a50d thru 0x03ce5a (0x294e bytes) 000255 1787856966.404185 trunc from 0x040000 to 0x00a667 (0x35998 bytes) 000256 1787856966.437780 MIRROR_RESYNC 4 mirrors 000257 1787856967.202850 write 0x008678 thru 0x011a56 (0x93df bytes) 000258 1787856967.252244 read 0x00285a thru 0x0084c1 (0x5c68 bytes) 000259 1787856967.260434 write 0x01bd18 thru 0x01c5af (0x898 bytes) 000260 1787856967.262247 punch from 0x0089d4 to 0x0169d8 (0xe005 bytes) 000261 1787856967.284512 MIRROR_RESYNC 4 mirrors 000262 1787856968.539438 trunc from 0x00fdf9 to 0x016661 (0x6869 bytes) 000263 1787856968.588344 trunc from 0x01c5b0 to 0x020bdb (0x462c bytes) 000264 1787856968.613108 punch from 0x005e1c to 0x0111b6 (0xb39b bytes) 000265 1787856968.618610 fallocate 0x009f1d thru 0x015d08 (0xbdec bytes) 000266 1787856968.623760 read 0x017b4f thru 0x020bdb (0x908d bytes) 000267 1787856968.633080 trunc from 0x00dba1 to 0x018719 (0xab79 bytes) 000268 1787856968.638067 fallocate 0x015a90 thru 0x0253e5 (0xf956 bytes) 000269 1787856968.646609 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20bdc): Mirror 4: 0xdfc5198 (stale) Mirror 5: 0xdfc5198 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20bdc): Mirror 4: 0xdfc5198 (stale) Mirror 5: 0xdfc5198 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20bdc: skipped 000271 1787856968.971220 mapwrite 0x000af8 thru 0x00f1a7 (0xe6b0 bytes) 000272 1787856969.030453 punch from 0x008914 to 0x00af47 (0x2634 bytes) 000273 1787856969.043620 mapread 0x0205b3 thru 0x020bdb (0x629 bytes) 000274 1787856969.069872 write 0x03239d thru 0x039e74 (0x7ad8 bytes) 000275 1787856969.081614 MIRROR_RESYNC 3 mirrors 000276 1787856970.041078 mapwrite 0x027193 thru 0x02fb17 (0x8985 bytes) 000277 1787856970.155132 read 0x016473 thru 0x02086f (0xa3fd bytes) 000278 1787856970.170248 punch from 0x01f3b2 to 0x020ca4 (0x18f3 bytes) 000279 1787856970.180194 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39e75): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e75): Mirror 4: 0x8c6aeb98 (stale) Mirror 5: 0x8c6aeb98 (stale) CRC-32 checksum value for chunk [0, 0x39e75): Mirror 1: 0x9aadd8b8 Mirror 6: 0x9aadd8b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e75: skipped 000280 1787856971.183343 trunc from 0x0118c7 to 0x01f992 (0xe0cc bytes) 000281 1787856971.250409 read 0x03435c thru 0x039e74 (0x5b19 bytes) 000282 1787856971.272602 fallocate 0x00cca6 thru 0x0165e5 (0x9940 bytes) 000283 1787856971.283579 punch from 0x0174fe to 0x01adb7 (0x38ba bytes) 000284 1787856971.290936 fallocate 0x0277d1 thru 0x02c50f (0x4d3f bytes) 000286 1787856971.307437 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39e75): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e75): Mirror 4: 0x8c6aeb98 (stale) Mirror 5: 0x8c6aeb98 (stale) Mirror 6: 0x9aadd8b8 (stale) CRC-32 checksum value for chunk [0, 0x39e75): Mirror 1: 0x1ec5decd Mirror 7: 0x1ec5decd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e75: skipped 000287 1787856972.706245 mapwrite 0x02a4a6 thru 0x034fd3 (0xab2e bytes) 000288 1787856972.911312 trunc from 0x024f42 to 0x032339 (0xd3f8 bytes) 000289 1787856972.921240 mapwrite 0x035d87 thru 0x03fe41 (0xa0bb bytes) 000290 1787856973.000756 punch from 0x016243 to 0x01aad6 (0x4894 bytes) 000291 1787856973.008470 trunc from 0x03ed4c to 0x03fe41 (0x10f6 bytes) 000292 1787856973.021418 MIRROR_RESYNC 5 mirrors 000294 1787856974.563718 trunc from 0x03fe42 to 0x027eda (0x17f67 bytes) 000295 1787856974.607240 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x27edb): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27edb): Mirror 4: 0xc89218d8 (stale) Mirror 5: 0xc89218d8 (stale) Mirror 6: 0xc89218d8 (stale) Mirror 7: 0xc89218d8 (stale) CRC-32 checksum value for chunk [0, 0x27edb): Mirror 1: 0xc89218d8 Mirror 8: 0xc89218d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27edb: skipped 000296 1787856975.615871 trunc from 0x021f9a to 0x025b36 (0x3b9d bytes) 000297 1787856975.657685 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x27edb): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27edb): Mirror 4: 0xc89218d8 (stale) Mirror 5: 0xc89218d8 (stale) Mirror 6: 0xc89218d8 (stale) Mirror 7: 0xc89218d8 (stale) Mirror 8: 0xc89218d8 (stale) CRC-32 checksum value for chunk [0, 0x27edb): Mirror 1: 0xc89218d8 Mirror 9: 0xc89218d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27edb: skipped 000298 1787856976.555994 write 0x0331f3 thru 0x03a9ac (0x77ba bytes) 000299 1787856976.605023 read 0x0001bb thru 0x00a694 (0xa4da bytes) 000300 1787856976.621131 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a9ad): Mirror 4: 0x77775ebf (stale) Mirror 5: 0x77775ebf (stale) Mirror 6: 0x77775ebf (stale) Mirror 8: 0x47009a29 (stale) Mirror 9: 0x47009a29 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a9ad): Mirror 4: 0x77775ebf (stale) Mirror 5: 0x77775ebf (stale) Mirror 6: 0x77775ebf (stale) Mirror 8: 0x47009a29 (stale) Mirror 9: 0x47009a29 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a9ad: skipped 000301 1787856977.012562 read 0x00859d thru 0x008e58 (0x8bc bytes) 000302 1787856977.019502 mapwrite 0x01cfde thru 0x01d076 (0x99 bytes) 000304 1787856977.045790 trunc from 0x02ef7b to 0x0355ef (0x6675 bytes) 000305 1787856977.054892 mapread 0x039bac thru 0x03a9ac (0xe01 bytes) 000306 1787856977.064203 write 0x03cdf1 thru 0x03ffff (0x320f bytes) 000307 1787856977.071907 mapwrite 0x0214bd thru 0x0303a6 (0xeeea bytes) 000308 1787856977.107829 read 0x010ddb thru 0x016d54 (0x5f7a bytes) 000309 1787856977.120855 trunc from 0x035a41 to 0x0396e7 (0x3ca7 bytes) 000310 1787856977.126468 write 0x01e480 thru 0x02877e (0xa2ff bytes) 000311 1787856977.128208 punch from 0x0222bc to 0x024257 (0x1f9c bytes) 000312 1787856977.147298 mapread 0x002dad thru 0x005645 (0x2899 bytes) 000313 1787856977.148880 read 0x0156bb thru 0x017d60 (0x26a6 bytes) 000314 1787856977.151861 mapread 0x003586 thru 0x00b0b9 (0x7b34 bytes) 000315 1787856977.152840 mapwrite 0x02630f thru 0x034284 (0xdf76 bytes) 000316 1787856977.188078 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x3113614d (stale) Mirror 5: 0x3113614d (stale) Mirror 8: 0xcbe61edd (stale) Mirror 9: 0xcbe61edd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x3113614d (stale) Mirror 5: 0x3113614d (stale) Mirror 8: 0xcbe61edd (stale) Mirror 9: 0xcbe61edd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000318 1787856977.666563 write 0x03910d thru 0x03ffff (0x6ef3 bytes) 000319 1787856977.677733 punch from 0x012872 to 0x0211ef (0xe97e bytes) 000320 1787856977.692269 write 0x014971 thru 0x01ee17 (0xa4a7 bytes) 000321 1787856977.711963 write 0x029432 thru 0x0342d0 (0xae9f bytes) 000322 1787856977.728770 mapread 0x031b9f thru 0x032a7b (0xedd bytes) 000324 1787856977.734033 trunc from 0x011ac9 to 0x0181dd (0x6715 bytes) 000325 1787856977.762862 trunc from 0x040000 to 0x01d9ab (0x22654 bytes) 000326 1787856977.782836 read 0x0040a2 thru 0x00a91f (0x687e bytes) 000327 1787856977.791869 mapread 0x014630 thru 0x01c40c (0x7ddd bytes) 000328 1787856977.815812 write 0x035bf9 thru 0x03b7d0 (0x5bd8 bytes) 000329 1787856977.817082 mapread 0x012fc2 thru 0x0135bd (0x5fc bytes) 000330 1787856977.817924 mapread 0x00914a thru 0x00fb19 (0x69d0 bytes) 000331 1787856977.825803 mapread 0x00f01a thru 0x013855 (0x483c bytes) 000332 1787856977.827103 mapwrite 0x01124e thru 0x018170 (0x6f23 bytes) 000333 1787856977.847340 trunc from 0x02c739 to 0x0345b0 (0x7e78 bytes) 000334 1787856977.851993 trunc from 0x03b7d1 to 0x01bc8c (0x1fb44 bytes) 000335 1787856977.872941 mapread 0x011f38 thru 0x01bc8c (0x9d55 bytes) 000336 1787856977.874347 mapwrite 0x0196b6 thru 0x0260a1 (0xc9ec bytes) 000337 1787856977.930923 fallocate 0x021818 thru 0x02ab84 (0x936d bytes) 000338 1787856977.937352 mapread 0x0146d5 thru 0x01b21c (0x6b48 bytes) 000339 1787856977.938963 read 0x0208af thru 0x02380f (0x2f61 bytes) 000341 1787856977.940575 fallocate 0x0322c3 thru 0x03e8db (0xc619 bytes) 000342 1787856977.945288 write 0x0010bb thru 0x002f00 (0x1e46 bytes) 000343 1787856977.953801 mapwrite 0x02bf40 thru 0x03010d (0x41ce bytes) 000344 1787856977.996855 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3010e): Mirror 4: 0xde1f5740 (stale) Mirror 5: 0xde1f5740 (stale) Mirror 8: 0xde1f5740 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3010e): Mirror 4: 0xde1f5740 (stale) Mirror 5: 0xde1f5740 (stale) Mirror 8: 0xde1f5740 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3010e: skipped 000345 1787856978.317191 trunc from 0x03010e to 0x010717 (0x1f9f6 bytes) 000346 1787856978.334956 read 0x00256c thru 0x005440 (0x2ed5 bytes) 000347 1787856978.342039 trunc from 0x010718 to 0x00ffb9 (0x75e bytes) 000349 1787856978.361639 MIRROR_RESYNC 4 mirrors 000350 1787856979.056057 trunc from 0x00ffba to 0x022b59 (0x12ba0 bytes) 000351 1787856979.176040 mapread 0x011a41 thru 0x016d26 (0x52e6 bytes) 000352 1787856979.237902 write 0x00cd44 thru 0x013bf9 (0x6eb6 bytes) 000353 1787856979.239229 mapread 0x0066ca thru 0x014b6b (0xe4a2 bytes) 000355 1787856979.256262 read 0x0071c6 thru 0x007358 (0x193 bytes) 000356 1787856979.262831 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x22b5a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22b5a): Mirror 4: 0xb8963fe2 (stale) Mirror 5: 0xb8963fe2 (stale) Mirror 8: 0xb8963fe2 (stale) CRC-32 checksum value for chunk [0, 0x22b5a): Mirror 1: 0xe283a3a9 Mirror 9: 0xe283a3a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22b5a: skipped 000357 1787856980.361770 read 0x00e84e thru 0x019b41 (0xb2f4 bytes) 000358 1787856980.376998 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x22b5a): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22b5a): Mirror 4: 0xb8963fe2 (stale) Mirror 5: 0xb8963fe2 (stale) Mirror 8: 0xb8963fe2 (stale) CRC-32 checksum value for chunk [0, 0x22b5a): Mirror 1: 0xe283a3a9 Mirror 9: 0xe283a3a9 Mirror 10: 0xe283a3a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22b5a: skipped 000359 1787856981.497285 mapwrite 0x0226de thru 0x022759 (0x7c bytes) 000360 1787856981.629979 read 0x00c24f thru 0x018962 (0xc714 bytes) 000361 1787856981.654300 punch from 0x0165c7 to 0x01ef1e (0x8958 bytes) 000362 1787856981.668840 mapread 0x003b9f thru 0x00b8cd (0x7d2f bytes) 000363 1787856981.680300 fallocate 0x01d14f thru 0x021cd8 (0x4b8a bytes) 000364 1787856981.697244 mapread 0x01754c thru 0x020203 (0x8cb8 bytes) 000365 1787856981.723318 mapread 0x013a98 thru 0x021b45 (0xe0ae bytes) 000366 1787856981.728143 punch from 0x01caf5 to 0x01d817 (0xd23 bytes) 000367 1787856981.733098 mapwrite 0x02a9bb thru 0x02d0dd (0x2723 bytes) 000369 1787856981.795152 trunc from 0x0148ef to 0x020a80 (0xc192 bytes) 000370 1787856981.800222 read 0x02739a thru 0x02d0dd (0x5d44 bytes) 000371 1787856981.801089 punch from 0x0242dd to 0x027a3e (0x3762 bytes) 000374 1787856981.809869 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2d0de): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d0de): Mirror 4: 0xba7c14d0 (stale) Mirror 5: 0xba7c14d0 (stale) Mirror 8: 0xba7c14d0 (stale) Mirror 9: 0x41da76eb (stale) Mirror 10: 0x41da76eb (stale) CRC-32 checksum value for chunk [0, 0x2d0de): Mirror 1: 0x12c6d66f Mirror 11: 0x12c6d66f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d0de: skipped 000375 1787856982.700006 mapread 0x006077 thru 0x0124d4 (0xc45e bytes) 000376 1787856982.736485 MIRROR_RESYNC 7 mirrors 000379 1787856984.189571 mapread 0x003167 thru 0x00accb (0x7b65 bytes) 000380 1787856984.209143 punch from 0x026209 to 0x02d0dd (0x6ed5 bytes) 000381 1787856984.250361 fallocate 0x03d426 thru 0x03e56d (0x1148 bytes) 000382 1787856984.255201 punch from 0x020c55 to 0x0238b5 (0x2c61 bytes) 000383 1787856984.262297 trunc from 0x02d0de to 0x020bba (0xc523 bytes) 000384 1787856984.285337 fallocate 0x01e4af thru 0x020458 (0x1faa bytes) 000385 1787856984.291140 punch from 0x00ef88 to 0x00f67a (0x6f3 bytes) 000388 1787856984.297157 mapread 0x00a4c2 thru 0x015e50 (0xb98f bytes) 000389 1787856984.313973 trunc from 0x020bbb to 0x038049 (0x1748f bytes) 000390 1787856984.334956 MIRROR_RESYNC 7 mirrors 000391 1787856985.633219 mapread 0x022103 thru 0x0280d3 (0x5fd1 bytes) 000392 1787856985.671180 write 0x027073 thru 0x031a0d (0xa99b bytes) 000393 1787856985.726011 trunc from 0x03804a to 0x039587 (0x153e bytes) 000394 1787856985.750971 fallocate 0x02122b thru 0x0261c7 (0x4f9d bytes) 000395 1787856985.756908 mapread 0x030985 thru 0x039587 (0x8c03 bytes) 000396 1787856985.771918 trunc from 0x0354d6 to 0x039587 (0x40b2 bytes) 000397 1787856985.782475 mapread 0x0277c1 thru 0x0319eb (0xa22b bytes) 000398 1787856985.790765 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x39588): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39588): Mirror 4: 0xa7d61b67 (stale) Mirror 5: 0xa7d61b67 (stale) Mirror 8: 0xa7d61b67 (stale) Mirror 9: 0xa7d61b67 (stale) Mirror 10: 0xa7d61b67 (stale) Mirror 11: 0xa7d61b67 (stale) CRC-32 checksum value for chunk [0, 0x39588): Mirror 1: 0x933da018 Mirror 12: 0x933da018 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39588: skipped 000399 1787856987.038851 write 0x0228b3 thru 0x027338 (0x4a86 bytes) 000400 1787856987.101105 MIRROR_RESYNC 8 mirrors 000402 1787856989.130711 mapwrite 0x013a1e thru 0x01f1ec (0xb7cf bytes) 000403 1787856989.286826 fallocate 0x02a3c1 thru 0x02fb56 (0x5796 bytes) 000404 1787856989.300188 fallocate 0x020599 thru 0x02d2af (0xcd17 bytes) 000405 1787856989.310532 punch from 0x02dd7d to 0x039587 (0xb80b bytes) 000406 1787856989.321445 mapwrite 0x0028fe thru 0x00f5cf (0xccd2 bytes) 000407 1787856989.368928 mapwrite 0x01af5c thru 0x01ef39 (0x3fde bytes) 000409 1787856989.396563 trunc from 0x00c912 to 0x014e3a (0x8529 bytes) 000410 1787856989.415146 MIRROR_RESYNC 8 mirrors 000411 1787856991.629466 trunc from 0x00ba84 to 0x014f10 (0x948d bytes) 000412 1787856991.717098 trunc from 0x039588 to 0x02d296 (0xc2f1 bytes) 000413 1787856991.750167 read 0x0019e5 thru 0x009165 (0x7781 bytes) 000414 1787856991.761676 trunc from 0x013c1a to 0x015f56 (0x233d bytes) 000415 1787856991.775970 trunc from 0x02d297 to 0x02cacd (0x7c9 bytes) 000416 1787856991.800984 trunc from 0x02cace to 0x00315e (0x2996f bytes) 000417 1787856991.850686 mapwrite 0x00ed5b thru 0x0126e6 (0x398c bytes) 000418 1787856991.906102 mapread 0x0029b1 thru 0x008aa2 (0x60f2 bytes) 000419 1787856991.915787 trunc from 0x000b68 to 0x00884e (0x7ce7 bytes) 000420 1787856991.921588 fallocate 0x0096e3 thru 0x012991 (0x92af bytes) 000422 1787856991.927055 read 0x00b4da thru 0x012991 (0x74b8 bytes) 000423 1787856991.927991 mapread 0x005895 thru 0x010d3d (0xb4a9 bytes) 000425 1787856991.943844 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12992): Mirror 5: 0xe352c634 (stale) Mirror 8: 0xe352c634 (stale) Mirror 9: 0xe352c634 (stale) Mirror 10: 0xe352c634 (stale) Mirror 11: 0xe352c634 (stale) Mirror 12: 0xe352c634 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12992): Mirror 5: 0xe352c634 (stale) Mirror 8: 0xe352c634 (stale) Mirror 9: 0xe352c634 (stale) Mirror 10: 0xe352c634 (stale) Mirror 11: 0xe352c634 (stale) Mirror 12: 0xe352c634 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12992: skipped 000426 1787856992.524446 mapwrite 0x00f74d thru 0x01d646 (0xdefa bytes) 000427 1787856992.625832 mapwrite 0x03ddf4 thru 0x03ffff (0x220c bytes) 000428 1787856992.676677 trunc from 0x040000 to 0x000abb (0x3f544 bytes) 000429 1787856992.696552 mapread 0x000399 thru 0x000abb (0x723 bytes) 000430 1787856992.704754 trunc from 0x000abc to 0x006242 (0x5787 bytes) 000431 1787856992.718685 write 0x02820c thru 0x037f76 (0xfd6b bytes) 000433 1787856992.722677 mapwrite 0x01d83d thru 0x022008 (0x47cc bytes) 000435 1787856992.753435 mapwrite 0x034b10 thru 0x039437 (0x4928 bytes) 000436 1787856992.810044 read 0x01165d thru 0x013af7 (0x249b bytes) 000437 1787856992.818584 trunc from 0x039438 to 0x02401c (0x1541b bytes) 000439 1787856992.845191 trunc from 0x0028ef to 0x0123ed (0xfaff bytes) 000440 1787856992.851516 write 0x016deb thru 0x0214c1 (0xa6d7 bytes) 000441 1787856992.854827 mapread 0x01694f thru 0x01a4a6 (0x3b58 bytes) 000442 1787856992.856925 read 0x009e7c thru 0x011906 (0x7a8b bytes) 000443 1787856992.867837 fallocate 0x00a1cc thru 0x017f74 (0xdda9 bytes) 000444 1787856992.873905 mapread 0x013ba7 thru 0x018a14 (0x4e6e bytes) 000447 1787856992.876036 fallocate 0x01b899 thru 0x026e3d (0xb5a5 bytes) 000448 1787856992.882774 trunc from 0x026e3e to 0x023f97 (0x2ea6 bytes) 000449 1787856992.903237 mapread 0x016ad3 thru 0x023812 (0xcd40 bytes) 000450 1787856992.905845 mapwrite 0x0354ae thru 0x03ffff (0xab52 bytes) 000451 1787856992.959119 trunc from 0x02c6e3 to 0x037587 (0xaea5 bytes) 000453 1787856992.969050 trunc from 0x01ef96 to 0x028521 (0x958c bytes) 000454 1787856992.975689 fallocate 0x002f04 thru 0x0030a0 (0x19d bytes) 000455 1787856992.982334 trunc from 0x040000 to 0x02b72c (0x148d3 bytes) 000456 1787856993.007809 trunc from 0x00dc43 to 0x015240 (0x75fe bytes) 000457 1787856993.014017 write 0x0143f3 thru 0x01f406 (0xb014 bytes) 000458 1787856993.027001 fallocate 0x003db4 thru 0x00b5d5 (0x7822 bytes) 000459 1787856993.032496 fallocate 0x016c0d thru 0x01d4f0 (0x68e4 bytes) 000460 1787856993.040746 mapread 0x023040 thru 0x027de3 (0x4da4 bytes) 000461 1787856993.077728 write 0x00e165 thru 0x01a329 (0xc1c5 bytes) 000462 1787856993.091573 fallocate 0x036113 thru 0x03e609 (0x84f7 bytes) 000463 1787856993.103720 read 0x02130f thru 0x030380 (0xf072 bytes) 000465 1787856993.135062 mapread 0x03e14b thru 0x03e609 (0x4bf bytes) 000466 1787856993.137104 write 0x012234 thru 0x01c9c5 (0xa792 bytes) 000467 1787856993.138583 punch from 0x008ea5 to 0x00c28d (0x33e9 bytes) 000468 1787856993.153066 punch from 0x01b39e to 0x01cea6 (0x1b09 bytes) 000469 1787856993.172883 mapwrite 0x03908c thru 0x03ffff (0x6f74 bytes) 000470 1787856993.227773 write 0x004c6f thru 0x009f7d (0x530f bytes) 000471 1787856993.235689 mapwrite 0x024928 thru 0x028a26 (0x40ff bytes) 000473 1787856993.257479 punch from 0x01f0f8 to 0x028d9f (0x9ca8 bytes) 000474 1787856993.268533 mapread 0x02b5d4 thru 0x02f930 (0x435d bytes) 000475 1787856993.270580 read 0x02e91e thru 0x03b2eb (0xc9ce bytes) 000477 1787856993.273104 trunc from 0x040000 to 0x035b10 (0xa4ef bytes) 000479 1787856993.299589 read 0x028d7e thru 0x02913e (0x3c1 bytes) 000480 1787856993.323878 mapwrite 0x00e873 thru 0x013ac1 (0x524f bytes) 000481 1787856993.345479 mapwrite 0x032326 thru 0x03ffff (0xdcda bytes) 000482 1787856993.433589 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xeb688e66 (stale) Mirror 8: 0xeb688e66 (stale) Mirror 9: 0xeb688e66 (stale) Mirror 10: 0xeb688e66 (stale) Mirror 11: 0xeb688e66 (stale) Mirror 12: 0xeb688e66 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x75b0e270 Mirror 13: 0x75b0e270 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000483 1787856994.578756 read 0x01b1f1 thru 0x02b129 (0xff39 bytes) 000485 1787856994.602648 mapwrite 0x03bd71 thru 0x03ffff (0x428f bytes) 000486 1787856994.752661 mapwrite 0x03b037 thru 0x03ffff (0x4fc9 bytes) 000487 1787856994.794012 punch from 0x00d0b8 to 0x012fe6 (0x5f2f bytes) 000488 1787856994.810929 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xeb688e66 (stale) Mirror 8: 0xeb688e66 (stale) Mirror 9: 0xeb688e66 (stale) Mirror 10: 0xeb688e66 (stale) Mirror 11: 0xeb688e66 (stale) Mirror 12: 0xeb688e66 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xeb688e66 (stale) Mirror 8: 0xeb688e66 (stale) Mirror 9: 0xeb688e66 (stale) Mirror 10: 0xeb688e66 (stale) Mirror 11: 0xeb688e66 (stale) Mirror 12: 0xeb688e66 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000489 1787856995.525486 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xeb688e66 (stale) Mirror 8: 0xeb688e66 (stale) Mirror 10: 0xeb688e66 (stale) Mirror 11: 0xeb688e66 (stale) Mirror 12: 0xeb688e66 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xeb688e66 (stale) Mirror 8: 0xeb688e66 (stale) Mirror 10: 0xeb688e66 (stale) Mirror 11: 0xeb688e66 (stale) Mirror 12: 0xeb688e66 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000490 1787856996.162025 mapread 0x020e52 thru 0x02bb80 (0xad2f bytes) 000491 1787856996.205682 trunc from 0x0323bb to 0x03418c (0x1dd2 bytes) 000494 1787856996.216712 trunc from 0x040000 to 0x010c05 (0x2f3fa bytes) 000495 1787856996.297012 write 0x007e19 thru 0x00b505 (0x36ed bytes) 000496 1787856996.312883 mapwrite 0x025744 thru 0x033bd4 (0xe491 bytes) 000497 1787856996.388713 mapread 0x018d5a thru 0x01c213 (0x34ba bytes) 000498 1787856996.408680 trunc from 0x017453 to 0x020e2c (0x99da bytes) 000499 1787856996.418135 punch from 0x0159d8 to 0x01d054 (0x767d bytes) 000500 1787856996.427797 fallocate 0x01a78a thru 0x01f480 (0x4cf7 bytes) 000502 1787856996.433596 punch from 0x005a7c to 0x00806d (0x25f2 bytes) 000503 1787856996.440866 fallocate 0x01a3b7 thru 0x029c82 (0xf8cc bytes) 000504 1787856996.446963 trunc from 0x033bd5 to 0x01955c (0x1a678 bytes) 000505 1787856996.467731 trunc from 0x010bfd to 0x016ce9 (0x60ed bytes) 000506 1787856996.473189 mapread 0x00d547 thru 0x014ff0 (0x7aaa bytes) 000507 1787856996.483697 punch from 0x01129d to 0x01955c (0x82c0 bytes) 000508 1787856996.491588 fallocate 0x02ff4a thru 0x036b4b (0x6c02 bytes) 000509 1787856996.502513 trunc from 0x02fe86 to 0x036b4b (0x6cc6 bytes) 000510 1787856996.507933 write 0x014ddf thru 0x0210a3 (0xc2c5 bytes) 000511 1787856996.528508 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36b4c): Mirror 8: 0xe81c09f2 (stale) Mirror 10: 0xe81c09f2 (stale) Mirror 11: 0xe81c09f2 (stale) Mirror 12: 0xe81c09f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36b4c): Mirror 8: 0xe81c09f2 (stale) Mirror 10: 0xe81c09f2 (stale) Mirror 11: 0xe81c09f2 (stale) Mirror 12: 0xe81c09f2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36b4c: skipped 000512 1787856996.964310 read 0x01b713 thru 0x02858a (0xce78 bytes) 000513 1787856996.976598 trunc from 0x00d9f1 to 0x01cd15 (0xf325 bytes) 000514 1787856996.982713 mapwrite 0x014668 thru 0x01ad6f (0x6708 bytes) 000515 1787856997.028893 trunc from 0x00ddcc to 0x01108e (0x32c3 bytes) 000516 1787856997.035714 mapwrite 0x01ee13 thru 0x02324f (0x443d bytes) 000517 1787856997.056620 write 0x0161ed thru 0x016294 (0xa8 bytes) 000518 1787856997.063734 mapread 0x033ade thru 0x0358f2 (0x1e15 bytes) 000519 1787856997.081965 MIRROR_RESYNC 5 mirrors 000520 1787856998.718952 write 0x03d8cc thru 0x03ffff (0x2734 bytes) 000521 1787856998.794004 write 0x031c8b thru 0x036633 (0x49a9 bytes) 000522 1787856998.808886 trunc from 0x01f3b8 to 0x0288a5 (0x94ee bytes) 000523 1787856998.814619 mapread 0x00ff7b thru 0x0147fd (0x4883 bytes) 000524 1787856998.851388 read 0x02bb02 thru 0x037cdc (0xc1db bytes) 000525 1787856998.863282 read 0x025194 thru 0x02b44f (0x62bc bytes) 000526 1787856998.885226 write 0x037c56 thru 0x03e3df (0x678a bytes) 000527 1787856998.886588 trunc from 0x040000 to 0x02cee5 (0x1311a bytes) 000528 1787856998.922848 mapwrite 0x018162 thru 0x01feca (0x7d69 bytes) 000529 1787856998.950977 read 0x0167c0 thru 0x0236c8 (0xcf09 bytes) 000530 1787856998.959639 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2cee6): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cee6): Mirror 8: 0xd7b1ff2e (stale) Mirror 10: 0xd7b1ff2e (stale) Mirror 11: 0xd7b1ff2e (stale) Mirror 12: 0xd7b1ff2e (stale) CRC-32 checksum value for chunk [0, 0x2cee6): Mirror 1: 0x6fc0b53a Mirror 13: 0x6fc0b53a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cee6: skipped 000532 1787857000.292742 write 0x01cc04 thru 0x021bde (0x4fdb bytes) 000533 1787857000.382043 mapwrite 0x002461 thru 0x0111ed (0xed8d bytes) 000534 1787857000.471599 fallocate 0x007f50 thru 0x01657e (0xe62f bytes) 000535 1787857000.479552 write 0x03db3d thru 0x03ffff (0x24c3 bytes) 000536 1787857000.481045 read 0x01d694 thru 0x029836 (0xc1a3 bytes) 000537 1787857000.519431 trunc from 0x00e8cf to 0x01bab0 (0xd1e2 bytes) 000538 1787857000.526146 punch from 0x01bb14 to 0x0279ed (0xbeda bytes) 000539 1787857000.531621 mapread 0x01453b thru 0x022b06 (0xe5cc bytes) 000540 1787857000.551758 mapread 0x02f117 thru 0x03b1cb (0xc0b5 bytes) 000541 1787857000.566253 trunc from 0x0071c8 to 0x016c2e (0xfa67 bytes) 000542 1787857000.570645 trunc from 0x040000 to 0x00f9ee (0x30611 bytes) 000543 1787857000.590975 fallocate 0x02b45f thru 0x036aa2 (0xb644 bytes) 000544 1787857000.617560 mapwrite 0x00f8c5 thru 0x0179a4 (0x80e0 bytes) 000545 1787857000.709333 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x179a5): Mirror 8: 0x98cd4847 (stale) Mirror 10: 0x98cd4847 (stale) Mirror 12: 0x98cd4847 (stale) Mirror 13: 0x98cd4847 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x179a5): Mirror 8: 0x98cd4847 (stale) Mirror 10: 0x98cd4847 (stale) Mirror 12: 0x98cd4847 (stale) Mirror 13: 0x98cd4847 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x179a5: skipped 000546 1787857001.240696 trunc from 0x007849 to 0x015ce4 (0xe49c bytes) 000547 1787857001.253050 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x179a5): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x179a5): Mirror 8: 0x98cd4847 (stale) Mirror 10: 0x98cd4847 (stale) Mirror 12: 0x98cd4847 (stale) Mirror 13: 0x98cd4847 (stale) CRC-32 checksum value for chunk [0, 0x179a5): Mirror 1: 0xce5b22c3 Mirror 14: 0xce5b22c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x179a5: skipped 000548 1787857002.406802 mapwrite 0x00a79e thru 0x00bfd4 (0x1837 bytes) 000549 1787857002.543267 read 0x0032fb thru 0x00b4ed (0x81f3 bytes) 000550 1787857002.558551 trunc from 0x0179a5 to 0x022357 (0xa9b3 bytes) 000551 1787857002.597133 MIRROR_RESYNC 6 mirrors 000552 1787857004.618357 punch from 0x010d46 to 0x019daa (0x9065 bytes) 000553 1787857004.654183 fallocate 0x0189dd thru 0x01ad64 (0x2388 bytes) 000554 1787857004.660086 mapwrite 0x00e866 thru 0x019f1d (0xb6b8 bytes) 000556 1787857004.715978 write 0x00adf2 thru 0x0111b6 (0x63c5 bytes) 000558 1787857004.721851 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22358): Mirror 10: 0x62b470eb (stale) Mirror 12: 0x62b470eb (stale) Mirror 13: 0x62b470eb (stale) Mirror 14: 0x62b470eb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22358): Mirror 10: 0x62b470eb (stale) Mirror 12: 0x62b470eb (stale) Mirror 13: 0x62b470eb (stale) Mirror 14: 0x62b470eb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22358: skipped 000559 1787857005.063251 mapwrite 0x0278eb thru 0x033bf5 (0xc30b bytes) 000560 1787857005.141544 fallocate 0x036ad2 thru 0x03afd9 (0x4508 bytes) 000561 1787857005.152083 mapread 0x031ec7 thru 0x033bf5 (0x1d2f bytes) 000562 1787857005.153812 write 0x03c590 thru 0x03ffff (0x3a70 bytes) 000563 1787857005.156832 punch from 0x009c00 to 0x0122dc (0x86dd bytes) 000564 1787857005.163484 punch from 0x00db91 to 0x014007 (0x6477 bytes) 000565 1787857005.170183 mapwrite 0x0183a2 thru 0x01ed1c (0x697b bytes) 000566 1787857005.213802 trunc from 0x00102a to 0x00b6fa (0xa6d1 bytes) 000567 1787857005.225488 mapread 0x032b22 thru 0x03f9b1 (0xce90 bytes) 000568 1787857005.236802 fallocate 0x03f6b4 thru 0x03ffff (0x94c bytes) 000569 1787857005.249445 write 0x00b032 thru 0x014c4a (0x9c19 bytes) 000571 1787857005.265448 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xe73713c6 (stale) Mirror 12: 0xe73713c6 (stale) Mirror 13: 0xe73713c6 (stale) Mirror 14: 0xe73713c6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7f4dde Mirror 15: 0x7f4dde lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000573 1787857006.158047 write 0x01f40b thru 0x025a77 (0x666d bytes) 000574 1787857006.215177 trunc from 0x040000 to 0x00c559 (0x33aa6 bytes) 000575 1787857006.271100 punch from 0x00679f to 0x00c559 (0x5dbb bytes) 000576 1787857006.277450 trunc from 0x0082f3 to 0x00c559 (0x4267 bytes) 000577 1787857006.282805 punch from 0x007ea2 to 0x00c559 (0x46b8 bytes) 000578 1787857006.287748 punch from 0x007e94 to 0x00c559 (0x46c6 bytes) 000579 1787857006.293166 write 0x02e550 thru 0x02e92c (0x3dd bytes) 000580 1787857006.294663 read 0x021946 thru 0x0228c2 (0xf7d bytes) 000581 1787857006.308541 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2e92d): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e92d): Mirror 10: 0xd4d7a3fd (stale) Mirror 12: 0xd4d7a3fd (stale) Mirror 13: 0xd4d7a3fd (stale) Mirror 14: 0xd4d7a3fd (stale) Mirror 15: 0x1179baf7 (stale) CRC-32 checksum value for chunk [0, 0x2e92d): Mirror 1: 0x93005a66 Mirror 16: 0x93005a66 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e92d: skipped 000582 1787857007.452038 read 0x0032a2 thru 0x012d17 (0xfa76 bytes) 000583 1787857007.464425 fallocate 0x034ca4 thru 0x03f130 (0xa48d bytes) 000584 1787857007.537333 trunc from 0x02e92d to 0x03df05 (0xf5d9 bytes) 000585 1787857007.612526 mapwrite 0x00463d thru 0x005b66 (0x152a bytes) 000587 1787857007.638508 mapwrite 0x036693 thru 0x03ec15 (0x8583 bytes) 000588 1787857007.694302 trunc from 0x010b51 to 0x01aadc (0x9f8c bytes) 000589 1787857007.700779 punch from 0x00f631 to 0x017b26 (0x84f6 bytes) 000590 1787857007.709785 trunc from 0x001bc7 to 0x00b6a8 (0x9ae2 bytes) 000591 1787857007.719456 mapwrite 0x026c43 thru 0x02851f (0x18dd bytes) 000592 1787857007.749274 trunc from 0x0064d3 to 0x0102d6 (0x9e04 bytes) 000594 1787857007.755687 write 0x0135da thru 0x01437f (0xda6 bytes) 000595 1787857007.765628 punch from 0x0013b8 to 0x0072bb (0x5f04 bytes) 000596 1787857007.770737 punch from 0x01d94e to 0x02473b (0x6dee bytes) 000598 1787857007.776913 mapread 0x00c704 thru 0x017553 (0xae50 bytes) 000599 1787857007.806911 read 0x026446 thru 0x030c63 (0xa81e bytes) 000600 1787857007.808117 fallocate 0x02fc72 thru 0x032abe (0x2e4d bytes) 000601 1787857007.815839 mapread 0x01ea21 thru 0x024e3d (0x641d bytes) 000602 1787857007.830490 mapread 0x035669 thru 0x0379cf (0x2367 bytes) 000603 1787857007.831924 read 0x024516 thru 0x02f0fb (0xabe6 bytes) 000604 1787857007.833275 mapread 0x00ef9a thru 0x01051f (0x1586 bytes) 000605 1787857007.837473 trunc from 0x03ec16 to 0x00cb0d (0x32108 bytes) 000606 1787857007.870621 write 0x003f4c thru 0x00a79f (0x6854 bytes) 000607 1787857007.874563 trunc from 0x000bac to 0x001167 (0x5bc bytes) 000608 1787857007.881678 mapread 0x007dac thru 0x0089d5 (0xc2a bytes) 000609 1787857007.883341 write 0x005626 thru 0x00a751 (0x512c bytes) 000610 1787857007.889023 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xcb0e): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcb0e): Mirror 10: 0x96bea6e4 (stale) Mirror 12: 0x96bea6e4 (stale) Mirror 13: 0x96bea6e4 (stale) Mirror 14: 0x96bea6e4 (stale) Mirror 15: 0xe3a67942 (stale) Mirror 16: 0xa719b5d0 (stale) CRC-32 checksum value for chunk [0, 0xcb0e): Mirror 1: 0xee67a78c Mirror 17: 0xee67a78c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcb0e: skipped 000611 1787857009.181696 punch from 0x004de5 to 0x00cb0d (0x7d29 bytes) 000612 1787857009.269098 write 0x000432 thru 0x00d052 (0xcc21 bytes) 000614 1787857009.298075 write 0x00d7d0 thru 0x019e78 (0xc6a9 bytes) 000615 1787857009.302672 mapwrite 0x02b295 thru 0x02f169 (0x3ed5 bytes) 000616 1787857009.402724 read 0x01c022 thru 0x027d2f (0xbd0e bytes) 000617 1787857009.441261 MIRROR_RESYNC 8 mirrors 000618 1787857010.995311 write 0x017e55 thru 0x01a6bf (0x286b bytes) 000619 1787857011.053646 fallocate 0x00f6db thru 0x01af0c (0xb832 bytes) 000620 1787857011.061104 read 0x0020c0 thru 0x00480e (0x274f bytes) 000621 1787857011.067305 write 0x00d195 thru 0x011a9f (0x490b bytes) 000622 1787857011.076680 trunc from 0x02158a to 0x02bcb8 (0xa72f bytes) 000623 1787857011.085007 mapwrite 0x01d977 thru 0x01e911 (0xf9b bytes) 000624 1787857011.115411 trunc from 0x02f16a to 0x015df5 (0x19374 bytes) 000626 1787857011.139032 trunc from 0x00f08f to 0x015df5 (0x6d67 bytes) 000627 1787857011.148455 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x15df6): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15df6): Mirror 10: 0x7b5f75fe (stale) Mirror 12: 0x7b5f75fe (stale) Mirror 13: 0x7b5f75fe (stale) Mirror 14: 0x7b5f75fe (stale) Mirror 15: 0x7b5f75fe (stale) Mirror 16: 0x7b5f75fe (stale) Mirror 17: 0x7b5f75fe (stale) CRC-32 checksum value for chunk [0, 0x15df6): Mirror 1: 0x408cbcf7 Mirror 18: 0x408cbcf7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15df6: skipped 000628 1787857012.235032 mapread 0x00afff thru 0x00bc3a (0xc3c bytes) 000629 1787857012.251485 write 0x02279d thru 0x02f110 (0xc974 bytes) 000630 1787857012.299413 read 0x002548 thru 0x009fb7 (0x7a70 bytes) 000631 1787857012.309990 punch from 0x0040df to 0x008366 (0x4288 bytes) 000632 1787857012.315073 mapwrite 0x03e474 thru 0x03ffff (0x1b8c bytes) 000633 1787857012.396979 read 0x031840 thru 0x03d2f5 (0xbab6 bytes) 000634 1787857012.411685 write 0x00df54 thru 0x01abe9 (0xcc96 bytes) 000635 1787857012.413876 read 0x01fcfa thru 0x02a0d3 (0xa3da bytes) 000636 1787857012.415356 fallocate 0x00c09d thru 0x01bfa5 (0xff09 bytes) 000637 1787857012.424338 fallocate 0x03dd6c thru 0x03ffff (0x2294 bytes) 000639 1787857012.431358 mapwrite 0x031bcd thru 0x038c5a (0x708e bytes) 000640 1787857012.455045 punch from 0x000f89 to 0x001ebd (0xf35 bytes) 000641 1787857012.465875 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xfc74d931 (stale) Mirror 12: 0xfc74d931 (stale) Mirror 13: 0xfc74d931 (stale) Mirror 14: 0xfc74d931 (stale) Mirror 15: 0xfc74d931 (stale) Mirror 16: 0xfc74d931 (stale) Mirror 17: 0xfc74d931 (stale) Mirror 18: 0x18948d7d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb7ecadc9 Mirror 19: 0xb7ecadc9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000643 1787857013.850456 fallocate 0x01bfca thru 0x0299b6 (0xd9ed bytes) 000644 1787857013.906644 fallocate 0x03bf89 thru 0x03ffff (0x4077 bytes) 000645 1787857013.914300 mapwrite 0x01e721 thru 0x026dc7 (0x86a7 bytes) 000646 1787857013.987298 punch from 0x032bde to 0x037419 (0x483c bytes) 000647 1787857013.997829 read 0x001eaa thru 0x002d19 (0xe70 bytes) 000648 1787857014.005441 trunc from 0x040000 to 0x02e0ff (0x11f00 bytes) 000649 1787857014.065462 trunc from 0x00049d to 0x005868 (0x53cc bytes) 000651 1787857014.076714 fallocate 0x0063ae thru 0x0088f5 (0x2548 bytes) 000652 1787857014.083014 trunc from 0x02e100 to 0x01f751 (0xe9ae bytes) 000653 1787857014.104525 punch from 0x0145be to 0x015634 (0x1077 bytes) 000654 1787857014.109192 trunc from 0x01f752 to 0x017cac (0x7aa5 bytes) 000655 1787857014.123812 mapread 0x005bf7 thru 0x00fb07 (0x9f11 bytes) 000656 1787857014.132915 write 0x020b83 thru 0x024f32 (0x43b0 bytes) 000657 1787857014.134195 fallocate 0x01797a thru 0x01d5fc (0x5c83 bytes) 000659 1787857014.139205 mapwrite 0x030f85 thru 0x0342fa (0x3376 bytes) 000660 1787857014.203894 trunc from 0x0342fb to 0x00a69b (0x29c5f bytes) 000661 1787857014.240354 fallocate 0x01088c thru 0x02000b (0xf780 bytes) 000662 1787857014.249323 read 0x006558 thru 0x00a69b (0x4144 bytes) 000663 1787857014.252565 punch from 0x004bd5 to 0x00a69b (0x5ac7 bytes) 000664 1787857014.263060 trunc from 0x00a69c to 0x00589e (0x4dfd bytes) 000665 1787857014.293756 fallocate 0x0299ef thru 0x032cf7 (0x9309 bytes) 000666 1787857014.300198 write 0x01d49f thru 0x0259d6 (0x8538 bytes) 000667 1787857014.322344 fallocate 0x039172 thru 0x03ffff (0x6e8e bytes) 000668 1787857014.335150 mapread 0x012d6c thru 0x021528 (0xe7bd bytes) 000669 1787857014.336187 punch from 0x0184f9 to 0x01ab7b (0x2683 bytes) 000670 1787857014.345121 punch from 0x01a739 to 0x01c3df (0x1ca7 bytes) 000672 1787857014.361192 mapread 0x01f6bb thru 0x02f429 (0xfd6f bytes) 000673 1787857014.366670 read 0x0396db thru 0x03ffff (0x6925 bytes) 000674 1787857014.383815 fallocate 0x00a87e thru 0x00f23f (0x49c2 bytes) 000676 1787857014.404744 trunc from 0x01eacd to 0x02e3d8 (0xf90c bytes) 000677 1787857014.422544 read 0x028b94 thru 0x03154e (0x89bb bytes) 000678 1787857014.433736 mapread 0x037d1d thru 0x03ffff (0x82e3 bytes) 000679 1787857014.434752 mapwrite 0x000ab0 thru 0x00598b (0x4edc bytes) 000680 1787857014.466201 punch from 0x01029a to 0x01f51a (0xf281 bytes) 000681 1787857014.478465 mapread 0x000e1a thru 0x0033fb (0x25e2 bytes) 000682 1787857014.479519 mapread 0x03f032 thru 0x03ffff (0xfce bytes) 000683 1787857014.481224 read 0x008dc4 thru 0x00b65f (0x289c bytes) 000684 1787857014.482995 trunc from 0x017232 to 0x025715 (0xe4e4 bytes) 000685 1787857014.492727 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xfc74d931 (stale) Mirror 12: 0xfc74d931 (stale) Mirror 13: 0xfc74d931 (stale) Mirror 14: 0xfc74d931 (stale) Mirror 15: 0xfc74d931 (stale) Mirror 16: 0xfc74d931 (stale) Mirror 17: 0xfc74d931 (stale) Mirror 18: 0x18948d7d (stale) Mirror 19: 0xb7ecadc9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd38c71f3 Mirror 20: 0xd38c71f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000686 1787857015.298011 punch from 0x01cf71 to 0x0207a6 (0x3836 bytes) 000687 1787857015.330918 MIRROR_RESYNC 11 mirrors 000688 1787857016.947859 fallocate 0x00da0b thru 0x00f534 (0x1b2a bytes) 000689 1787857017.011756 read 0x0214cb thru 0x0246db (0x3211 bytes) 000690 1787857017.026694 punch from 0x0068f9 to 0x010089 (0x9791 bytes) 000692 1787857017.035200 read 0x0003cb thru 0x00b432 (0xb068 bytes) 000693 1787857017.069146 mapread 0x02af20 thru 0x02b0e6 (0x1c7 bytes) 000694 1787857017.070062 fallocate 0x03569d thru 0x03ffff (0xa963 bytes) 000695 1787857017.076222 read 0x0314c5 thru 0x03f73f (0xe27b bytes) 000696 1787857017.077179 mapwrite 0x008ca5 thru 0x017227 (0xe583 bytes) 000698 1787857017.102362 punch from 0x007d41 to 0x009c16 (0x1ed6 bytes) 000699 1787857017.108975 read 0x01334c thru 0x01f96c (0xc621 bytes) 000700 1787857017.114335 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xd38c71f3 (stale) Mirror 13: 0xd38c71f3 (stale) Mirror 14: 0xd38c71f3 (stale) Mirror 15: 0xd38c71f3 (stale) Mirror 16: 0xd38c71f3 (stale) Mirror 17: 0xd38c71f3 (stale) Mirror 18: 0xd38c71f3 (stale) Mirror 19: 0xd38c71f3 (stale) Mirror 20: 0xd38c71f3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xd38c71f3 (stale) Mirror 13: 0xd38c71f3 (stale) Mirror 14: 0xd38c71f3 (stale) Mirror 15: 0xd38c71f3 (stale) Mirror 16: 0xd38c71f3 (stale) Mirror 17: 0xd38c71f3 (stale) Mirror 18: 0xd38c71f3 (stale) Mirror 19: 0xd38c71f3 (stale) Mirror 20: 0xd38c71f3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000701 1787857017.902524 punch from 0x036b6f to 0x03b796 (0x4c28 bytes) 000702 1787857017.912169 trunc from 0x040000 to 0x02f02c (0x10fd3 bytes) 000703 1787857017.969859 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2f02d): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f02d): Mirror 12: 0xecb4fa66 (stale) Mirror 13: 0xecb4fa66 (stale) Mirror 14: 0xecb4fa66 (stale) Mirror 15: 0xecb4fa66 (stale) Mirror 16: 0xecb4fa66 (stale) Mirror 17: 0xecb4fa66 (stale) Mirror 18: 0xecb4fa66 (stale) Mirror 19: 0xecb4fa66 (stale) Mirror 20: 0xecb4fa66 (stale) CRC-32 checksum value for chunk [0, 0x2f02d): Mirror 1: 0x6a1332f7 Mirror 21: 0x6a1332f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f02d: skipped 000704 1787857019.146013 MIRROR_RESYNC 11 mirrors 000705 1787857021.696245 write 0x013d86 thru 0x01f689 (0xb904 bytes) 000706 1787857021.788191 read 0x007b2d thru 0x0092fd (0x17d1 bytes) 000707 1787857021.818055 mapread 0x013681 thru 0x014517 (0xe97 bytes) 000708 1787857021.822984 mapwrite 0x029e41 thru 0x0375d8 (0xd798 bytes) 000709 1787857021.989921 fallocate 0x00bfd6 thru 0x010fb4 (0x4fdf bytes) 000710 1787857022.000064 mapwrite 0x0265c7 thru 0x031253 (0xac8d bytes) 000712 1787857022.062113 fallocate 0x01dead thru 0x02878c (0xa8e0 bytes) 000713 1787857022.068138 read 0x006276 thru 0x0065cc (0x357 bytes) 000714 1787857022.085668 fallocate 0x01d5e4 thru 0x02c9b0 (0xf3cd bytes) 000715 1787857022.104221 write 0x02a32e thru 0x02edae (0x4a81 bytes) 000716 1787857022.105722 write 0x01c1a7 thru 0x01e989 (0x27e3 bytes) 000717 1787857022.113583 write 0x000019 thru 0x00ffa2 (0xff8a bytes) 000718 1787857022.115317 trunc from 0x0375d9 to 0x020623 (0x16fb5 bytes) 000719 1787857022.146907 read 0x02041e thru 0x020623 (0x206 bytes) 000720 1787857022.147968 mapwrite 0x003837 thru 0x012236 (0xea00 bytes) 000721 1787857022.212253 mapwrite 0x014144 thru 0x01fb4e (0xba0b bytes) 000722 1787857022.238851 trunc from 0x00256e to 0x01223b (0xfcce bytes) 000723 1787857022.248020 trunc from 0x020624 to 0x00ea48 (0x11bdb bytes) 000724 1787857022.285582 read 0x008a4b thru 0x00e05c (0x5612 bytes) 000725 1787857022.308173 trunc from 0x00ea49 to 0x01107b (0x2633 bytes) 000726 1787857022.332573 write 0x02c026 thru 0x03ba6a (0xfa45 bytes) 000727 1787857022.336203 read 0x02a79a thru 0x02dbcc (0x3433 bytes) 000728 1787857022.350227 mapwrite 0x010707 thru 0x013385 (0x2c7f bytes) 000729 1787857022.375618 trunc from 0x03ba6b to 0x01fb38 (0x1bf32 bytes) 000731 1787857022.394595 trunc from 0x0111fc to 0x01d7fc (0xc601 bytes) 000733 1787857022.402844 trunc from 0x01fb39 to 0x038a50 (0x18f18 bytes) 000734 1787857022.427478 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38a51): Mirror 12: 0x1ea1ed2e (stale) Mirror 13: 0x1ea1ed2e (stale) Mirror 14: 0x1ea1ed2e (stale) Mirror 15: 0x1ea1ed2e (stale) Mirror 16: 0x1ea1ed2e (stale) Mirror 17: 0x1ea1ed2e (stale) Mirror 18: 0x1ea1ed2e (stale) Mirror 19: 0x1ea1ed2e (stale) Mirror 20: 0x1ea1ed2e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a51): Mirror 12: 0x1ea1ed2e (stale) Mirror 13: 0x1ea1ed2e (stale) Mirror 14: 0x1ea1ed2e (stale) Mirror 15: 0x1ea1ed2e (stale) Mirror 16: 0x1ea1ed2e (stale) Mirror 17: 0x1ea1ed2e (stale) Mirror 18: 0x1ea1ed2e (stale) Mirror 19: 0x1ea1ed2e (stale) Mirror 20: 0x1ea1ed2e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a51: skipped 000735 1787857023.082309 mapwrite 0x01d4b9 thru 0x02717d (0x9cc5 bytes) 000736 1787857023.146641 mapwrite 0x006e77 thru 0x00cc4f (0x5dd9 bytes) 000737 1787857023.177791 write 0x02f2e1 thru 0x03d91c (0xe63c bytes) 000738 1787857023.193439 MIRROR_RESYNC 10 mirrors 000739 1787857026.494560 mapread 0x010a49 thru 0x01ab88 (0xa140 bytes) 000740 1787857026.514091 trunc from 0x0197a0 to 0x020682 (0x6ee3 bytes) 000742 1787857026.601177 mapread 0x01314c thru 0x0213d8 (0xe28d bytes) 000743 1787857026.628716 write 0x01e66f thru 0x024987 (0x6319 bytes) 000744 1787857026.632229 write 0x0053a7 thru 0x00fdde (0xaa38 bytes) 000745 1787857026.639831 mapread 0x035ad8 thru 0x03d91c (0x7e45 bytes) 000746 1787857026.657467 trunc from 0x03d91d to 0x03f00b (0x16ef bytes) 000747 1787857026.687115 fallocate 0x0072ae thru 0x012e86 (0xbbd9 bytes) 000748 1787857026.692762 write 0x03cdad thru 0x03ffff (0x3253 bytes) 000749 1787857026.694695 trunc from 0x040000 to 0x0207ad (0x1f852 bytes) 000750 1787857026.710771 punch from 0x01999e to 0x0207ad (0x6e10 bytes) 000751 1787857026.733746 punch from 0x008e7c to 0x010093 (0x7218 bytes) 000752 1787857026.748452 MIRROR_RESYNC 10 mirrors 000753 1787857029.581052 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000754 1787857030.034097 read 0x003cca thru 0x00e7ab (0xaae2 bytes) 000755 1787857030.050526 read 0x0146e8 thru 0x017d7b (0x3694 bytes) 000756 1787857030.061670 write 0x00756a thru 0x00f786 (0x821d bytes) 000757 1787857030.122440 mapread 0x0154ec thru 0x018375 (0x2e8a bytes) 000758 1787857030.144275 fallocate 0x0202a1 thru 0x0290ac (0x8e0c bytes) 000759 1787857030.149292 mapread 0x004046 thru 0x009230 (0x51eb bytes) 000760 1787857030.169963 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x207ae): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x207ae): Mirror 13: 0x406498c0 (stale) Mirror 14: 0x406498c0 (stale) Mirror 15: 0x406498c0 (stale) Mirror 16: 0x406498c0 (stale) Mirror 17: 0x406498c0 (stale) Mirror 18: 0x406498c0 (stale) Mirror 19: 0x406498c0 (stale) Mirror 20: 0x406498c0 (stale) CRC-32 checksum value for chunk [0, 0x207ae): Mirror 1: 0xd715ac79 Mirror 21: 0xd715ac79 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x207ae: skipped 000761 1787857031.293251 fallocate 0x01d298 thru 0x027a9b (0xa804 bytes) 000762 1787857031.345072 MIRROR_RESYNC 10 mirrors 000763 1787857033.228195 MIRROR_EXTEND to 11 mirrors 000764 1787857034.080798 MIRROR_RESYNC 11 mirrors 000765 1787857034.633027 mapread 0x018086 thru 0x019552 (0x14cd bytes) 000766 1787857034.641795 punch from 0x014650 to 0x0207ad (0xc15e bytes) 000767 1787857034.677232 mapwrite 0x02b253 thru 0x02f33b (0x40e9 bytes) 000768 1787857034.732312 punch from 0x00832e to 0x00be53 (0x3b26 bytes) 000769 1787857034.742680 fallocate 0x03d2cf thru 0x03ffff (0x2d31 bytes) 000770 1787857034.747039 write 0x0115ad thru 0x02049d (0xeef1 bytes) 000771 1787857034.757924 mapwrite 0x01e95a thru 0x02e8a7 (0xff4e bytes) 000772 1787857034.785760 write 0x020f37 thru 0x02eef9 (0xdfc3 bytes) 000773 1787857034.786923 trunc from 0x021cd0 to 0x02fa59 (0xdd8a bytes) 000774 1787857034.799586 trunc from 0x040000 to 0x018484 (0x27b7b bytes) 000775 1787857034.827931 MIRROR_RESYNC 11 mirrors 000776 1787857036.731566 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000777 1787857037.272221 write 0x03d662 thru 0x03ffff (0x299e bytes) 000778 1787857037.313659 write 0x0334c2 thru 0x03e4d9 (0xb018 bytes) 000779 1787857037.323594 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x7cdc5248 (stale) Mirror 14: 0x7cdc5248 (stale) Mirror 16: 0x7cdc5248 (stale) Mirror 17: 0x7cdc5248 (stale) Mirror 18: 0x7cdc5248 (stale) Mirror 19: 0x7cdc5248 (stale) Mirror 20: 0x7cdc5248 (stale) Mirror 21: 0x7cdc5248 (stale) Mirror 22: 0x7cdc5248 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x780d8a56 Mirror 23: 0x780d8a56 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000780 1787857038.589499 trunc from 0x037638 to 0x03e4a7 (0x6e70 bytes) 000781 1787857038.650313 write 0x030597 thru 0x03c0e2 (0xbb4c bytes) 000782 1787857038.671611 trunc from 0x040000 to 0x02410e (0x1bef1 bytes) 000783 1787857038.703079 mapread 0x002c50 thru 0x009572 (0x6923 bytes) 000784 1787857038.725322 read 0x0030c0 thru 0x010a39 (0xd97a bytes) 000785 1787857038.726748 write 0x038476 thru 0x03ffff (0x7b8a bytes) 000786 1787857038.730053 trunc from 0x040000 to 0x00bf1f (0x340e0 bytes) 000787 1787857038.752769 trunc from 0x005d68 to 0x006ef7 (0x1190 bytes) 000788 1787857038.760609 trunc from 0x00bf20 to 0x014f7b (0x905c bytes) 000789 1787857038.782665 MIRROR_RESYNC 11 mirrors 000790 1787857041.509140 trunc from 0x006f09 to 0x014f7b (0xe073 bytes) 000791 1787857041.569977 punch from 0x00f41d to 0x014f7b (0x5b5f bytes) 000792 1787857041.582141 write 0x023b0c thru 0x026408 (0x28fd bytes) 000793 1787857041.598951 MIRROR_RESYNC 11 mirrors 000795 1787857044.221799 MIRROR_EXTEND to 12 mirrors 000796 1787857045.073783 mapread 0x00e002 thru 0x015547 (0x7546 bytes) 000797 1787857045.095982 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) 000799 1787857045.715503 read 0x01b87d thru 0x0244b4 (0x8c38 bytes) 000800 1787857045.728970 mapread 0x0004aa thru 0x00212a (0x1c81 bytes) 000801 1787857045.748494 punch from 0x0165ef to 0x02262a (0xc03c bytes) 000802 1787857045.820936 MIRROR_RESYNC 11 mirrors 000803 1787857048.744956 trunc from 0x0018b0 to 0x00e703 (0xce54 bytes) 000804 1787857048.786609 mapwrite 0x020b04 thru 0x024580 (0x3a7d bytes) 000805 1787857048.834874 trunc from 0x01788b to 0x01aa16 (0x318c bytes) 000806 1787857048.840768 trunc from 0x026409 to 0x0397a5 (0x1339d bytes) 000807 1787857048.863703 fallocate 0x00e447 thru 0x010ac4 (0x267e bytes) 000808 1787857048.876825 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x397a6): Mirror 16: 0x166e0980 (stale) Mirror 17: 0x166e0980 (stale) Mirror 18: 0x166e0980 (stale) Mirror 19: 0x166e0980 (stale) Mirror 20: 0x166e0980 (stale) Mirror 21: 0x166e0980 (stale) Mirror 22: 0x166e0980 (stale) Mirror 23: 0x166e0980 (stale) Mirror 24: 0x166e0980 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x397a6): Mirror 16: 0x166e0980 (stale) Mirror 17: 0x166e0980 (stale) Mirror 18: 0x166e0980 (stale) Mirror 19: 0x166e0980 (stale) Mirror 20: 0x166e0980 (stale) Mirror 21: 0x166e0980 (stale) Mirror 22: 0x166e0980 (stale) Mirror 23: 0x166e0980 (stale) Mirror 24: 0x166e0980 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x397a6: skipped 000809 1787857049.566898 punch from 0x0051b7 to 0x0095a3 (0x43ed bytes) 000810 1787857049.586693 trunc from 0x0397a6 to 0x0226ee (0x170b7 bytes) 000811 1787857049.609269 trunc from 0x01c378 to 0x021c9b (0x5924 bytes) 000812 1787857049.617246 mapwrite 0x0393df thru 0x03de1e (0x4a40 bytes) 000813 1787857049.715123 mapread 0x0189b0 thru 0x01c694 (0x3ce5 bytes) 000814 1787857049.745695 mapwrite 0x007462 thru 0x01428c (0xce2b bytes) 000815 1787857049.809641 trunc from 0x02653a to 0x036482 (0xff49 bytes) 000816 1787857049.820123 mapread 0x0018b6 thru 0x00e241 (0xc98c bytes) 000818 1787857049.821995 fallocate 0x03f805 thru 0x03ffff (0x7fb bytes) 000819 1787857049.837518 trunc from 0x03de1f to 0x00b0fb (0x32d23 bytes) 000820 1787857049.867529 write 0x026a10 thru 0x034af9 (0xe0ea bytes) 000821 1787857049.872364 write 0x039143 thru 0x03fce5 (0x6ba3 bytes) 000822 1787857049.879840 mapwrite 0x01851e thru 0x02609a (0xdb7d bytes) 000823 1787857049.992352 write 0x003386 thru 0x00e98a (0xb605 bytes) 000824 1787857050.005450 trunc from 0x039fb9 to 0x03b8fb (0x1943 bytes) 000825 1787857050.021634 mapread 0x02f9e4 thru 0x035930 (0x5f4d bytes) 000826 1787857050.047446 trunc from 0x005936 to 0x00eeb6 (0x9581 bytes) 000827 1787857050.086091 write 0x00d8cc thru 0x013aae (0x61e3 bytes) 000828 1787857050.107651 mapread 0x03a34e thru 0x03fce5 (0x5998 bytes) 000829 1787857050.110252 trunc from 0x03fce6 to 0x000154 (0x3fb91 bytes) 000831 1787857050.138355 mapread 0x000109 thru 0x000154 (0x4c bytes) 000832 1787857050.149631 write 0x01b360 thru 0x01d5db (0x227c bytes) 000833 1787857050.152948 mapread 0x00cbb4 thru 0x00f811 (0x2c5e bytes) 000834 1787857050.184939 trunc from 0x01d5dc to 0x017ade (0x5afd bytes) 000835 1787857050.210727 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17adf): Mirror 16: 0x9d1ff5af (stale) Mirror 17: 0x9d1ff5af (stale) Mirror 18: 0x9d1ff5af (stale) Mirror 19: 0x9d1ff5af (stale) Mirror 20: 0x9d1ff5af (stale) Mirror 22: 0x9d1ff5af (stale) Mirror 23: 0x9d1ff5af (stale) Mirror 24: 0x9d1ff5af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17adf): Mirror 16: 0x9d1ff5af (stale) Mirror 17: 0x9d1ff5af (stale) Mirror 18: 0x9d1ff5af (stale) Mirror 19: 0x9d1ff5af (stale) Mirror 20: 0x9d1ff5af (stale) Mirror 22: 0x9d1ff5af (stale) Mirror 23: 0x9d1ff5af (stale) Mirror 24: 0x9d1ff5af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17adf: skipped 000836 1787857050.988593 trunc from 0x017adf to 0x01ebb3 (0x70d5 bytes) 000837 1787857051.033450 fallocate 0x02a069 thru 0x037ed5 (0xde6d bytes) 000838 1787857051.043245 write 0x01d3ae thru 0x02aa20 (0xd673 bytes) 000839 1787857051.052785 trunc from 0x02aa21 to 0x0182e5 (0x1273b bytes) 000840 1787857051.097217 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x182e6): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x182e6): Mirror 16: 0x4057883d (stale) Mirror 17: 0x4057883d (stale) Mirror 18: 0x4057883d (stale) Mirror 19: 0x4057883d (stale) Mirror 20: 0x4057883d (stale) Mirror 22: 0x4057883d (stale) Mirror 23: 0x4057883d (stale) Mirror 24: 0x4057883d (stale) CRC-32 checksum value for chunk [0, 0x182e6): Mirror 1: 0x3e1afa64 Mirror 25: 0x3e1afa64 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x182e6: skipped 000841 1787857052.445710 mapwrite 0x0120c6 thru 0x019bf7 (0x7b32 bytes) 000842 1787857052.535172 fallocate 0x004355 thru 0x01175c (0xd408 bytes) 000843 1787857052.545860 write 0x014d3c thru 0x019f24 (0x51e9 bytes) 000844 1787857052.547889 trunc from 0x00a6d6 to 0x017dda (0xd705 bytes) 000845 1787857052.556235 read 0x00520c thru 0x011bba (0xc9af bytes) 000846 1787857052.563585 mapread 0x0155f4 thru 0x019f24 (0x4931 bytes) 000847 1787857052.572129 punch from 0x010b8a to 0x016107 (0x557e bytes) 000848 1787857052.591722 MIRROR_RESYNC 10 mirrors 000849 1787857054.586586 fallocate 0x0330d2 thru 0x03ffff (0xcf2e bytes) 000850 1787857054.628041 MIRROR_RESYNC 10 mirrors 000851 1787857057.566054 read 0x005432 thru 0x00d0a1 (0x7c70 bytes) 000852 1787857057.576424 MIRROR_EXTEND to 11 mirrors 000853 1787857058.687192 mapwrite 0x00186b thru 0x004978 (0x310e bytes) 000854 1787857058.807529 trunc from 0x040000 to 0x026970 (0x1968f bytes) 000855 1787857058.844977 trunc from 0x026971 to 0x03583a (0xeeca bytes) 000856 1787857058.887193 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3583b): Mirror 16: 0x5ac72175 (stale) Mirror 17: 0x5ac72175 (stale) Mirror 18: 0x5ac72175 (stale) Mirror 19: 0x5ac72175 (stale) Mirror 22: 0x5ac72175 (stale) Mirror 23: 0x5ac72175 (stale) Mirror 24: 0x5ac72175 (stale) Mirror 25: 0x5ac72175 (stale) Mirror 26: 0x5ac72175 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3583b): Mirror 16: 0x5ac72175 (stale) Mirror 17: 0x5ac72175 (stale) Mirror 18: 0x5ac72175 (stale) Mirror 19: 0x5ac72175 (stale) Mirror 22: 0x5ac72175 (stale) Mirror 23: 0x5ac72175 (stale) Mirror 24: 0x5ac72175 (stale) Mirror 25: 0x5ac72175 (stale) Mirror 26: 0x5ac72175 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3583b: skipped 000857 1787857059.482457 mapread 0x001049 thru 0x004524 (0x34dc bytes) 000859 1787857059.513827 trunc from 0x01f388 to 0x02311f (0x3d98 bytes) 000860 1787857059.520764 trunc from 0x03583b to 0x016259 (0x1f5e1 bytes) 000861 1787857059.566825 fallocate 0x012660 thru 0x01f816 (0xd1b7 bytes) 000862 1787857059.575832 mapread 0x012a0a thru 0x01f816 (0xce0d bytes) 000863 1787857059.595902 punch from 0x014780 to 0x01f816 (0xb097 bytes) 000864 1787857059.607952 write 0x00c52f thru 0x012efd (0x69cf bytes) 000865 1787857059.613876 mapread 0x00f44e thru 0x01a00d (0xabc0 bytes) 000866 1787857059.624276 trunc from 0x01f817 to 0x005e22 (0x199f4 bytes) 000867 1787857059.640166 trunc from 0x0030c3 to 0x005e22 (0x2d60 bytes) 000868 1787857059.645707 punch from 0x00279a to 0x005e22 (0x3689 bytes) 000869 1787857059.651625 trunc from 0x005e23 to 0x0165f6 (0x107d4 bytes) 000870 1787857059.671629 trunc from 0x0165f7 to 0x0173ff (0xe09 bytes) 000871 1787857059.687020 trunc from 0x017400 to 0x019344 (0x1f45 bytes) 000872 1787857059.706966 punch from 0x0180cb to 0x019344 (0x127a bytes) 000874 1787857059.718568 trunc from 0x004170 to 0x00ff69 (0xbdfa bytes) 000875 1787857059.727325 write 0x009a3b thru 0x019445 (0xfa0b bytes) 000876 1787857059.746181 mapwrite 0x02a512 thru 0x03131b (0x6e0a bytes) 000877 1787857059.807780 fallocate 0x030c42 thru 0x0345b4 (0x3973 bytes) 000878 1787857059.818672 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3131c): Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3131c): Mirror 16: 0x87be2064 (stale) Mirror 17: 0x87be2064 (stale) Mirror 18: 0x87be2064 (stale) Mirror 19: 0x87be2064 (stale) Mirror 22: 0x87be2064 (stale) Mirror 23: 0x87be2064 (stale) Mirror 24: 0x87be2064 (stale) Mirror 25: 0x87be2064 (stale) Mirror 26: 0x87be2064 (stale) CRC-32 checksum value for chunk [0, 0x3131c): Mirror 1: 0x6ac5d673 Mirror 27: 0x6ac5d673 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3131c: skipped 000879 1787857061.301644 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3131c): Mirror 16: 0x87be2064 (stale) Mirror 17: 0x87be2064 (stale) Mirror 18: 0x87be2064 (stale) Mirror 19: 0x87be2064 (stale) Mirror 22: 0x87be2064 (stale) Mirror 23: 0x87be2064 (stale) Mirror 24: 0x87be2064 (stale) Mirror 25: 0x87be2064 (stale) Mirror 26: 0x87be2064 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3131c): Mirror 16: 0x87be2064 (stale) Mirror 17: 0x87be2064 (stale) Mirror 18: 0x87be2064 (stale) Mirror 19: 0x87be2064 (stale) Mirror 22: 0x87be2064 (stale) Mirror 23: 0x87be2064 (stale) Mirror 24: 0x87be2064 (stale) Mirror 25: 0x87be2064 (stale) Mirror 26: 0x87be2064 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3131c: skipped 000880 1787857062.078665 mapwrite 0x028614 thru 0x0318c7 (0x92b4 bytes) 000881 1787857062.145626 trunc from 0x0117da to 0x012d49 (0x1570 bytes) 000882 1787857062.159307 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x318c8): Mirror 16: 0x59f23b97 (stale) Mirror 18: 0x59f23b97 (stale) Mirror 19: 0x59f23b97 (stale) Mirror 22: 0x59f23b97 (stale) Mirror 23: 0x59f23b97 (stale) Mirror 24: 0x59f23b97 (stale) Mirror 25: 0x59f23b97 (stale) Mirror 26: 0x59f23b97 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x318c8): Mirror 16: 0x59f23b97 (stale) Mirror 18: 0x59f23b97 (stale) Mirror 19: 0x59f23b97 (stale) Mirror 22: 0x59f23b97 (stale) Mirror 23: 0x59f23b97 (stale) Mirror 24: 0x59f23b97 (stale) Mirror 25: 0x59f23b97 (stale) Mirror 26: 0x59f23b97 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x318c8: skipped 000883 1787857062.692256 punch from 0x010ed8 to 0x011218 (0x341 bytes) 000884 1787857062.697864 mapread 0x01f4eb thru 0x025fba (0x6ad0 bytes) 000885 1787857062.724828 mapread 0x01e5a0 thru 0x027534 (0x8f95 bytes) 000887 1787857062.726398 read 0x01a3f7 thru 0x0268cb (0xc4d5 bytes) 000888 1787857062.727667 write 0x00daaa thru 0x0109a6 (0x2efd bytes) 000889 1787857062.740945 mapwrite 0x02541b thru 0x02ab7a (0x5760 bytes) 000890 1787857062.793964 trunc from 0x0318c8 to 0x036b43 (0x527c bytes) 000891 1787857062.822353 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x36b44): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36b44): Mirror 16: 0x66e76121 (stale) Mirror 18: 0x66e76121 (stale) Mirror 19: 0x66e76121 (stale) Mirror 22: 0x66e76121 (stale) Mirror 23: 0x66e76121 (stale) Mirror 24: 0x66e76121 (stale) Mirror 25: 0x66e76121 (stale) Mirror 26: 0x66e76121 (stale) CRC-32 checksum value for chunk [0, 0x36b44): Mirror 1: 0x7d5bb6ee Mirror 27: 0x7d5bb6ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36b44: skipped 000892 1787857064.577996 punch from 0x01c668 to 0x024440 (0x7dd9 bytes) 000894 1787857064.642020 write 0x00f1a9 thru 0x015ebb (0x6d13 bytes) 000895 1787857064.674360 trunc from 0x036b44 to 0x008470 (0x2e6d3 bytes) 000896 1787857064.693217 trunc from 0x0013b1 to 0x008470 (0x70c0 bytes) 000897 1787857064.700452 trunc from 0x008471 to 0x000f04 (0x756c bytes) 000898 1787857064.716143 mapread 0x0004a3 thru 0x000f04 (0xa62 bytes) 000900 1787857064.733941 write 0x01c4cf thru 0x01d931 (0x1463 bytes) 000901 1787857064.735048 punch from 0x018cab to 0x01d931 (0x4c87 bytes) 000902 1787857064.750638 mapwrite 0x0385c5 thru 0x03c7ec (0x4228 bytes) 000903 1787857064.830664 fallocate 0x02204d thru 0x02efdb (0xcf8f bytes) 000904 1787857064.842231 trunc from 0x028b07 to 0x0336e4 (0xabde bytes) 000905 1787857064.849595 trunc from 0x0290b5 to 0x036cb7 (0xdc03 bytes) 000906 1787857064.862260 mapread 0x02eb81 thru 0x02fdc5 (0x1245 bytes) 000907 1787857064.879531 mapwrite 0x027aae thru 0x02ccc2 (0x5215 bytes) 000908 1787857064.907518 mapread 0x005912 thru 0x014fed (0xf6dc bytes) 000909 1787857064.935153 fallocate 0x014ab5 thru 0x020e86 (0xc3d2 bytes) 000910 1787857064.942766 trunc from 0x03c7ed to 0x017215 (0x255d7 bytes) 000911 1787857064.966404 trunc from 0x0068e9 to 0x0114df (0xabf7 bytes) 000912 1787857064.982952 fallocate 0x03991d thru 0x03ffff (0x66e3 bytes) 000914 1787857064.995157 punch from 0x007a72 to 0x0147e0 (0xcd6f bytes) 000915 1787857065.010743 trunc from 0x040000 to 0x00757a (0x38a85 bytes) 000916 1787857065.049805 read 0x0058ca thru 0x00757a (0x1cb1 bytes) 000917 1787857065.067014 trunc from 0x00757b to 0x0196c1 (0x12147 bytes) 000918 1787857065.113488 fallocate 0x02f57d thru 0x03e28e (0xed12 bytes) 000919 1787857065.122828 trunc from 0x03e28f to 0x01e07a (0x20214 bytes) 000920 1787857065.146802 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x1e07b): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e07b): Mirror 16: 0xe63e2d1 (stale) Mirror 18: 0xe63e2d1 (stale) Mirror 19: 0xe63e2d1 (stale) Mirror 22: 0xe63e2d1 (stale) Mirror 23: 0xe63e2d1 (stale) Mirror 24: 0xe63e2d1 (stale) Mirror 25: 0xe63e2d1 (stale) Mirror 26: 0xe63e2d1 (stale) Mirror 27: 0x69cce0d (stale) CRC-32 checksum value for chunk [0, 0x1e07b): Mirror 1: 0x77d39423 Mirror 28: 0x77d39423 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e07b: skipped 000921 1787857066.158470 trunc from 0x00e665 to 0x00efc3 (0x95f bytes) 000922 1787857066.194826 trunc from 0x01e07b to 0x00c8e7 (0x11793 bytes) 000923 1787857066.247248 read 0x0075a2 thru 0x00b64f (0x40ae bytes) 000924 1787857066.257225 trunc from 0x00c8e8 to 0x00afa3 (0x1944 bytes) 000925 1787857066.270580 mapwrite 0x0361a1 thru 0x03ffff (0x9e5f bytes) 000926 1787857066.390599 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xb2066633 (stale) Mirror 18: 0xb2066633 (stale) Mirror 19: 0xb2066633 (stale) Mirror 22: 0xb2066633 (stale) Mirror 23: 0xb2066633 (stale) Mirror 24: 0xb2066633 (stale) Mirror 25: 0xb2066633 (stale) Mirror 26: 0xb2066633 (stale) Mirror 27: 0x89d8709f (stale) Mirror 28: 0x906b7569 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xde9fe8b3 Mirror 29: 0xde9fe8b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000927 1787857068.587136 punch from 0x030f2c to 0x03652e (0x5603 bytes) 000928 1787857068.693194 mapwrite 0x033cec thru 0x03d790 (0x9aa5 bytes) 000929 1787857068.786062 trunc from 0x040000 to 0x028bfe (0x17401 bytes) 000930 1787857068.851095 trunc from 0x018ccd to 0x01cbb4 (0x3ee8 bytes) 000931 1787857068.883016 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x28bff): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28bff): Mirror 16: 0x80eab446 (stale) Mirror 18: 0x80eab446 (stale) Mirror 19: 0x80eab446 (stale) Mirror 22: 0x80eab446 (stale) Mirror 23: 0x80eab446 (stale) Mirror 24: 0x80eab446 (stale) Mirror 25: 0x80eab446 (stale) Mirror 26: 0x80eab446 (stale) Mirror 27: 0x57998aba (stale) Mirror 28: 0x8c15cc (stale) Mirror 29: 0x8c15cc (stale) CRC-32 checksum value for chunk [0, 0x28bff): Mirror 1: 0x8c15cc Mirror 30: 0x8c15cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28bff: skipped 000932 1787857070.074761 trunc from 0x028bff to 0x01bb9b (0xd063 bytes) 000933 1787857070.126036 punch from 0x0028c5 to 0x0090a6 (0x67e2 bytes) 000934 1787857070.131693 read 0x00ac64 thru 0x019095 (0xe432 bytes) 000935 1787857070.160772 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x1bb9c): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bb9c): Mirror 16: 0xdf83de3c (stale) Mirror 18: 0xdf83de3c (stale) Mirror 19: 0xdf83de3c (stale) Mirror 22: 0xdf83de3c (stale) Mirror 23: 0xdf83de3c (stale) Mirror 24: 0xdf83de3c (stale) Mirror 25: 0xdf83de3c (stale) Mirror 26: 0xdf83de3c (stale) Mirror 27: 0x6443895c (stale) Mirror 28: 0xd135df85 (stale) Mirror 29: 0xd135df85 (stale) Mirror 30: 0xd135df85 (stale) CRC-32 checksum value for chunk [0, 0x1bb9c): Mirror 1: 0xd135df85 Mirror 31: 0xd135df85 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bb9c: skipped 000936 1787857071.597755 read 0x0165f6 thru 0x01bb9b (0x55a6 bytes) 000937 1787857071.607689 trunc from 0x00da7b to 0x010f09 (0x348f bytes) 000938 1787857071.644026 fallocate 0x00e9fe thru 0x01d35e (0xe961 bytes) 000939 1787857071.651050 read 0x00c83c thru 0x00dd1a (0x14df bytes) 000940 1787857071.662748 MIRROR_SPLIT mirror 9 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1bb9c): Mirror 16: 0xdf83de3c (stale) Mirror 18: 0xdf83de3c (stale) Mirror 19: 0xdf83de3c (stale) Mirror 22: 0xdf83de3c (stale) Mirror 23: 0xdf83de3c (stale) Mirror 24: 0xdf83de3c (stale) Mirror 25: 0xdf83de3c (stale) Mirror 26: 0xdf83de3c (stale) Mirror 28: 0xd135df85 (stale) Mirror 29: 0xd135df85 (stale) Mirror 30: 0xd135df85 (stale) Mirror 31: 0xd135df85 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bb9c): Mirror 16: 0xdf83de3c (stale) Mirror 18: 0xdf83de3c (stale) Mirror 19: 0xdf83de3c (stale) Mirror 22: 0xdf83de3c (stale) Mirror 23: 0xdf83de3c (stale) Mirror 24: 0xdf83de3c (stale) Mirror 25: 0xdf83de3c (stale) Mirror 26: 0xdf83de3c (stale) Mirror 28: 0xd135df85 (stale) Mirror 29: 0xd135df85 (stale) Mirror 30: 0xd135df85 (stale) Mirror 31: 0xd135df85 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bb9c: skipped 000941 1787857072.206582 trunc from 0x017fb3 to 0x01bb9b (0x3be9 bytes) 000942 1787857072.210789 trunc from 0x004918 to 0x008935 (0x401e bytes) 000943 1787857072.216896 MIRROR_RESYNC 13 mirrors 000944 1787857074.640526 write 0x0158f0 thru 0x0207f5 (0xaf06 bytes) 000945 1787857074.713022 write 0x0174f0 thru 0x02676f (0xf280 bytes) 000946 1787857074.722227 mapread 0x00dba9 thru 0x0110d0 (0x3528 bytes) 000947 1787857074.765765 fallocate 0x01bf23 thru 0x01f638 (0x3716 bytes) 000948 1787857074.785476 mapwrite 0x03dd94 thru 0x03ffff (0x226c bytes) 000949 1787857074.889432 MIRROR_RESYNC 13 mirrors 000950 1787857077.935276 punch from 0x00c920 to 0x016262 (0x9943 bytes) 000951 1787857077.968514 read 0x011b84 thru 0x0212db (0xf758 bytes) 000952 1787857077.980443 mapread 0x03d9e7 thru 0x03ffff (0x2619 bytes) 000954 1787857077.997299 mapread 0x03fe5d thru 0x03ffff (0x1a3 bytes) 000956 1787857077.999072 fallocate 0x03d970 thru 0x03ffff (0x2690 bytes) 000957 1787857078.003339 punch from 0x00e7d3 to 0x015a97 (0x72c5 bytes) 000958 1787857078.007067 trunc from 0x016d9e to 0x023e6a (0xd0cd bytes) 000959 1787857078.011537 trunc from 0x040000 to 0x0242f2 (0x1bd0d bytes) 000960 1787857078.028339 mapwrite 0x003756 thru 0x00f63c (0xbee7 bytes) 000961 1787857078.054062 mapread 0x01a14f thru 0x0242f2 (0xa1a4 bytes) 000962 1787857078.066304 trunc from 0x002d92 to 0x00c444 (0x96b3 bytes) 000963 1787857078.071006 mapread 0x0027ee thru 0x009150 (0x6963 bytes) 000965 1787857078.089332 trunc from 0x0242f3 to 0x0397f8 (0x15506 bytes) 000966 1787857078.113925 trunc from 0x0397f9 to 0x028587 (0x11271 bytes) 000967 1787857078.130906 write 0x03aba7 thru 0x03ffff (0x5459 bytes) 000968 1787857078.132055 mapwrite 0x017f4a thru 0x024c49 (0xcd00 bytes) 000970 1787857078.150176 read 0x03c997 thru 0x03ffff (0x3669 bytes) 000971 1787857078.154248 mapread 0x02938b thru 0x0376f9 (0xe36f bytes) 000972 1787857078.174240 trunc from 0x040000 to 0x013500 (0x2caff bytes) 000973 1787857078.239416 trunc from 0x013501 to 0x02959a (0x1609a bytes) 000974 1787857078.279102 mapread 0x0109bf thru 0x01497b (0x3fbd bytes) 000976 1787857078.317572 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x2959b): Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2959b): Mirror 16: 0x3bb18bee (stale) Mirror 18: 0x3bb18bee (stale) Mirror 19: 0x3bb18bee (stale) Mirror 22: 0x3bb18bee (stale) Mirror 23: 0x3bb18bee (stale) Mirror 24: 0x3bb18bee (stale) Mirror 25: 0x3bb18bee (stale) Mirror 26: 0x3bb18bee (stale) Mirror 28: 0x3bb18bee (stale) Mirror 29: 0x3bb18bee (stale) Mirror 30: 0x3bb18bee (stale) Mirror 31: 0x3bb18bee (stale) CRC-32 checksum value for chunk [0, 0x2959b): Mirror 1: 0xe175b464 Mirror 32: 0xe175b464 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2959b: skipped 000978 1787857079.591139 mapread 0x01f66b thru 0x02959a (0x9f30 bytes) 000979 1787857079.639297 mapwrite 0x016fbc thru 0x023949 (0xc98e bytes) 000980 1787857079.789590 read 0x00b5be thru 0x017ce6 (0xc729 bytes) 000981 1787857079.798512 trunc from 0x00fe7a to 0x019917 (0x9a9e bytes) 000982 1787857079.804659 mapread 0x01ce34 thru 0x0228bd (0x5a8a bytes) 000983 1787857079.806166 trunc from 0x02959b to 0x01133c (0x1825e bytes) 000984 1787857079.885883 fallocate 0x015b2d thru 0x01ff72 (0xa446 bytes) 000985 1787857079.894225 trunc from 0x01133d to 0x0079ad (0x998f bytes) 000986 1787857079.908361 mapwrite 0x02feee thru 0x037cf8 (0x7e0b bytes) 000987 1787857079.946907 trunc from 0x037cf9 to 0x032ad4 (0x5224 bytes) 000988 1787857079.958513 trunc from 0x02b41c to 0x032ad4 (0x76b9 bytes) 000989 1787857079.962641 mapread 0x030424 thru 0x031769 (0x1346 bytes) 000990 1787857079.969320 write 0x02f44d thru 0x033c06 (0x47ba bytes) 000991 1787857079.970475 fallocate 0x012bd3 thru 0x013655 (0xa83 bytes) 000992 1787857079.973851 trunc from 0x025a0d to 0x026ed8 (0x14cc bytes) 000993 1787857079.978897 punch from 0x00233a to 0x00bdb0 (0x9a77 bytes) 000995 1787857079.983138 punch from 0x01bfd6 to 0x023295 (0x72c0 bytes) 000997 1787857079.988486 trunc from 0x00ff7f to 0x01d325 (0xd3a7 bytes) 000998 1787857079.995321 MIRROR_SPLIT mirror 10 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x55:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33c07): Mirror 16: 0xd30bf1aa (stale) Mirror 18: 0xd30bf1aa (stale) Mirror 19: 0xd30bf1aa (stale) Mirror 22: 0xd30bf1aa (stale) Mirror 23: 0xd30bf1aa (stale) Mirror 24: 0xd30bf1aa (stale) Mirror 25: 0xd30bf1aa (stale) Mirror 26: 0xd30bf1aa (stale) Mirror 28: 0xd30bf1aa (stale) Mirror 30: 0xd30bf1aa (stale) Mirror 31: 0xd30bf1aa (stale) Mirror 32: 0xa805d0e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33c07): Mirror 16: 0xd30bf1aa (stale) Mirror 18: 0xd30bf1aa (stale) Mirror 19: 0xd30bf1aa (stale) Mirror 22: 0xd30bf1aa (stale) Mirror 23: 0xd30bf1aa (stale) Mirror 24: 0xd30bf1aa (stale) Mirror 25: 0xd30bf1aa (stale) Mirror 26: 0xd30bf1aa (stale) Mirror 28: 0xd30bf1aa (stale) Mirror 30: 0xd30bf1aa (stale) Mirror 31: 0xd30bf1aa (stale) Mirror 32: 0xa805d0e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33c07: skipped 000999 1787857080.669200 trunc from 0x033c07 to 0x008755 (0x2b4b1 bytes) 001000 1787857080.694347 mapread 0x002f81 thru 0x008755 (0x57d5 bytes) All operations completed A-OK! PASS 70a (143s) == sanity-flr test 71: check mirror extend parent fid ==== 14:58:06 (1787857086) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x96:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 14:58:18 (1787857098) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.220081 s, 9.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800429 s, 13.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13f:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13f:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x140:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 14:58:27 (1787857107) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre2 Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3985120 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8887744 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 4310592 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7846240 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7881664 .. failed Extending file size to 8185568 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8226080 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8336512 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8585664 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62587 62589 62590 62593 62594 done 192.168.201.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre2 (opts:) 192.168.201.150@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:59:48 (1787857188) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre2 Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3793536 .. Extending file size to 8625472 .. failed Extending file size to 7903136 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8255456 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8274592 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8919360 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8419264 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8797664 .. Waiting 64186 64187 64189 64190 64193 done 192.168.201.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre2 (opts:) 192.168.201.150@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:01:09 (1787857269) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2600 1268704 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2028 1269276 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25180 3581840 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17520 3589500 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42700 7171340 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2600 1268704 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2028 1269276 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25180 3581840 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17520 3589500 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42700 7171340 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124416 s, 16.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.319733 s, 6.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x2cfda59d (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.201.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:01:32 (1787857292) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.130575 s, 16.1 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 15:01:42 (1787857302) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa5:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15c:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x15b:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:01:51 (1787857311) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145231 s, 14.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:02:00 (1787857320) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.106538 s, 19.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15d:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:02:10 (1787857330) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.166093 s, 12.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15f:0x0] } PASS 204c (37s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:02:47 (1787857367) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg150-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103246 s, 10.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x160:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x161:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x160:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x160:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x161:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x160:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x161:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:03:40 (1787857420) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg150-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x162:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x163:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.130526 s, 16.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x162:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x163:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x162:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x162:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x163:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x162:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:04:38 (1787857478) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg150-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg150-server: Pool lustre.test_204f created Adding targets to pool oleg150-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg150-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.168112 s, 12.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg150-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg150-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg150-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:05:58 (1787857558) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:06:08 (1787857568) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:06:17 (1787857577) oleg150-server: Pool lustre.test_206 created Adding targets to pool oleg150-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg150-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg150-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg150-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg150-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:06:48 (1787857608) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0997406 s, 10.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537203 s, 19.5 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x1e1f310f Mirror 3: 0x1e1f310f PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:07:00 (1787857620) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:07:03 (1787857623) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:07:08 (1787857628) Starting client: oleg150-client.virtnet: -o user_xattr,flock 192.168.201.150@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.068021 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1aa0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x7b0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x7b0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x7b0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x16d0000 Read this (4 bytes): BBBB 192.168.201.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg150-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:07:50 (1787857670) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0117108 s, 700 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:07:58 (1787857678) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693488 s, 15.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc6 lmm_fid: [0x200000405:0xc6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc6 lmm_fid: [0x200000405:0xc6:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x16e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611334 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 15:08:00.000000000 -0400 Modify: 2026-08-27 15:08:00.000000000 -0400 Change: 2026-08-27 15:08:02.000000000 -0400 Birth: 2026-08-27 15:08:00.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:08:08 (1787857688) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511177 s, 20.5 MB/s oleg150-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg150-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg150-server: O/280000401/d15/16f: File not found by ext2_lookup PASS 210b (28s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:08:36 (1787857716) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.211548 s, 194 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0113782 s, 360 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 15:08 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xcb:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 15:08 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:08:44 (1787857724) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:08:53 (1787857733) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 2012 sec =========== 15:09:03 (1787857743) === sanity-flr: start cleanup 15:09:05 (1787857745) === === sanity-flr: finish cleanup 15:09:16 (1787857756) ===