-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 10:37:13 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg447-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 10:37:36 (1787841456) === oleg447-client.virtnet: executing check_config_client /mnt/lustre oleg447-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg447-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8f25c322d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8f25c322d800.idle_timeout=debug disable quota as required oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 10:38:19 (1787841499) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8221] == sanity-flr test 0a: lfs mirror create with -N option == 10:38:22 (1787841502) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:38:40 (1787841520) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:38:44 (1787841524) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:38:48 (1787841528) PASS 0d (30s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:39:18 (1787841558) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:39:22 (1787841562) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 10:39:26 (1787841566) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8f25c322d800.stats=clear osc.lustre-OST0001-osc-ffff8f25c322d800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:39:37 (1787841577) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:39:49 (1787841589) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:39:57 (1787841597) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:40:01 (1787841601) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:40:10 (1787841610) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 10:40:21 (1787841621) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:40:29 (1787841629) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:40:37 (1787841637) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:40:46 (1787841646) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:40:58 (1787841658) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:41:05 (1787841665) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 1.7338 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114308 s, 9.2 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:41:15 (1787841675) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.362533 s, 28.9 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:41:26 (1787841686) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:41:38 (1787841698) Stopping /mnt/lustre-ost1 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f25c322d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f25c322d800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg447-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f25c322d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f25c322d800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg447-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (82s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:43:00 (1787841780) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.30814 s, 20.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f25c322d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f25c322d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg447-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:43:43 (1787841823) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f25c322d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f25c322d800.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg447-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f25c322d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f25c322d800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg447-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (85s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:45:08 (1787841908) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.99625 s, 17.5 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 10:45 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.186s user 0m0.000s sys 0m0.201s Stopping /mnt/lustre-ost1 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8f25c322d800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8f25c322d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.403s user 0m0.000s sys 0m0.200s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg447-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg447-server oleg447-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8f25c322d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f25c322d800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.266s user 0m0.001s sys 0m0.195s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg447-server: oleg447-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg447-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg447-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f25c322d800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8f25c322d800.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:46:46 (1787842006) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:46:50 (1787842010) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:46:54 (1787842014) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 10:46:58 (1787842018) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 10:47:07 (1787842027) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.90661 s, 13.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:47:21 (1787842041) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218056 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465892 s, 22.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:47:29 (1787842049) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:47:38 (1787842058) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00182176 s, 562 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000871961 s, 1.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000967593 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0025296 s, 405 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 10:47:48 (1787842068) create test files with size 15749584 .. 29eb07d840e37bf857181637ebaf3564 - 9b468b9504c7e39a8c3dfbb7ae371c8f - b531832fe7a4da10773eb717f0227c43 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg447-client.virtnet /mnt/lustre (opts:) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 10:48:11 (1787842091) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:48:23 (1787842103) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:48:30 (1787842110) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693573 s, 15.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177903 s, 58.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 10:48:39 (1787842119) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.103949 s, 20.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.357343 s, 11.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.28448 s, 7.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 10:48:53 (1787842133) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 10:48:57 (1787842137) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:49:00 (1787842140) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0308274 s, 34.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 10:49:08 (1787842148) llite.lustre-ffff8f25c8745000.enable_compression=1 striped dir -i1 -c2 -H crush /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (8s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 10:49:16 (1787842156) llite.lustre-ffff8f25c8745000.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493334 s, 21.3 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677819 s, 15.5 MB/s **verify components PASS 43d (9s) == sanity-flr test 44a: lfs mirror split check =========== 10:49:25 (1787842165) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.394427 s, 8.0 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:49:36 (1787842176) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xce:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc6:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:49:46 (1787842186) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.486679 s, 21.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x9:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg447-client.virtnet /mnt/lustre (opts:) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:50:00 (1787842200) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8f25c5bf5800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.497677 s, 21.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg447-client.virtnet /mnt/lustre (opts:) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8f25d2b28000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:50:17 (1787842217) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (45s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:51:02 (1787842262) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:51:44 (1787842304) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 10:51:47 (1787842307) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.222495 s, 14.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.121412 s, 25.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x5782179f (nosync,stale) Mirror 2: 0x5782179f (nosync,stale) ** verify mirror 2 doesn't change original checksum: a13957401b1b25d42d358aac60a3659c - mirror 1 checksum: 5282b9cd156149448aa44eb4215bb485 - mirror 2 checksum: a13957401b1b25d42d358aac60a3659c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T10:51:51-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T10:51:51-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: a13957401b1b25d42d358aac60a3659c - mirror 1 checksum: 5282b9cd156149448aa44eb4215bb485 - mirror 2 checksum: 5282b9cd156149448aa44eb4215bb485 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T10:51:51-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T10:51:51-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } PASS 48 (13s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 10:52:00 (1787842320) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0286718 s, 36.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (10s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:52:10 (1787842330) Failing mds1 on oleg447-server Stopping /mnt/lustre-mds1 (opts:) on oleg447-server 10:52:22 (1787842342) shut down facet: mds1 facet_host: oleg447-server facet_failover_host: oleg447-server Failover mds1 to oleg447-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Started lustre-MDT0000 10:52:54 (1787842374) targets are mounted 10:52:54 (1787842374) facet_failover done oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727961 s, 14.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } PASS 50A (62s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:53:13 (1787842393) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0439954 s, 466 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0165934 s, 1.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:53:24 (1787842404) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0753079 s, 272 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0100353 s, 2.0 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:53:34 (1787842414) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00559113 s, 2.9 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:53:41 (1787842421) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.083626 s, 245 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:53:49 (1787842429) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00527574 s, 776 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0412063 s, 99.4 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:53:58 (1787842438) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (27s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:54:26 (1787842466) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:54:56 (1787842496) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:55:21 (1787842521) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:55:46 (1787842546) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 10:55:54 (1787842554) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 58867 mirror split pid 58868 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58867 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58868 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 10:57:03 (1787842623) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 2626 skipping zero size read 000002 1787842626.963821 MIRROR_EXTEND to 2 mirrors skipping zero size read 000004 1787842627.476225 mapwrite 0x027399 thru 0x036dc4 (0xfa2c bytes) 000005 1787842627.621386 mapread 0x00ee80 thru 0x01a5db (0xb75c bytes) 000006 1787842627.674022 write 0x038f91 thru 0x03ffff (0x706f bytes) truncating to largest ever: 0x2c72e 000007 1787842627.684025 trunc from 0x040000 to 0x02c72d (0x138d2 bytes) 000008 1787842627.738103 write 0x02c595 thru 0x0368df (0xa34b bytes) 000009 1787842627.740003 write 0x0014a5 thru 0x010876 (0xf3d2 bytes) 000010 1787842627.752790 trunc from 0x0368e0 to 0x0207ce (0x16111 bytes) 000011 1787842627.787980 mapwrite 0x0129a6 thru 0x0208d7 (0xdf32 bytes) 000012 1787842627.909974 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) 000013 1787842628.093750 trunc from 0x0208d8 to 0x01740f (0x94c8 bytes) 000014 1787842628.129108 mapwrite 0x03a27b thru 0x03ffff (0x5d85 bytes) 000016 1787842628.237639 trunc from 0x040000 to 0x008b1e (0x374e1 bytes) 000017 1787842628.298783 mapread 0x0082c9 thru 0x008b1e (0x856 bytes) 000018 1787842628.329496 mapwrite 0x020a41 thru 0x022f7a (0x253a bytes) 000019 1787842628.423515 read 0x018630 thru 0x020ebc (0x888d bytes) 000020 1787842628.431746 trunc from 0x022f7b to 0x008b2a (0x1a450 bytes) 000021 1787842628.467972 write 0x025b7b thru 0x0293c9 (0x384f bytes) 000022 1787842628.483001 write 0x00b9f0 thru 0x015cbd (0xa2ce bytes) 000023 1787842628.513573 read 0x006513 thru 0x011738 (0xb226 bytes) 000024 1787842628.524705 punch from 0x00be18 to 0x01a580 (0xe769 bytes) 000025 1787842628.564803 trunc from 0x0293ca to 0x00614a (0x2327f bytes) 000026 1787842628.616587 mapread 0x0041bd thru 0x00614a (0x1f8e bytes) 000027 1787842628.624540 fallocate 0x006b4d thru 0x009264 (0x2718 bytes) fallocating to largest ever: 0x381e7 000028 1787842628.633027 fallocate 0x030edf thru 0x0381e6 (0x7308 bytes) 000029 1787842628.640611 mapwrite 0x023ac1 thru 0x026b44 (0x3084 bytes) 000030 1787842628.714730 trunc from 0x01da91 to 0x029fa7 (0xc517 bytes) 000031 1787842628.721065 write 0x0067c8 thru 0x014288 (0xdac1 bytes) 000032 1787842628.745206 mapread 0x020cae thru 0x0248e2 (0x3c35 bytes) 000033 1787842628.799037 write 0x031c5c thru 0x031d5f (0x104 bytes) 000034 1787842628.800814 mapwrite 0x002d96 thru 0x0100be (0xd329 bytes) 000035 1787842628.893811 trunc from 0x02a169 to 0x034bbb (0xaa53 bytes) fallocating to largest ever: 0x40000 000037 1787842628.916215 fallocate 0x03f5ce thru 0x03ffff (0xa32 bytes) 000038 1787842628.925531 fallocate 0x00d021 thru 0x014902 (0x78e2 bytes) 000039 1787842628.934718 trunc from 0x040000 to 0x00bd47 (0x342b8 bytes) 000040 1787842629.018780 read 0x005156 thru 0x00881c (0x36c7 bytes) 000041 1787842629.020542 mapwrite 0x03b571 thru 0x03ffff (0x4a8f bytes) 000042 1787842629.093066 fallocate 0x025ad1 thru 0x02dae8 (0x8018 bytes) 000043 1787842629.103522 mapwrite 0x032a5e thru 0x037cec (0x528f bytes) 000044 1787842629.126774 mapread 0x03df89 thru 0x03ffff (0x2077 bytes) 000045 1787842629.127696 read 0x023347 thru 0x028333 (0x4fed bytes) 000046 1787842629.128337 mapwrite 0x03dc44 thru 0x03ffff (0x23bc bytes) 000047 1787842629.149219 trunc from 0x040000 to 0x036caa (0x9355 bytes) 000048 1787842629.168522 trunc from 0x036cab to 0x036ef8 (0x24e bytes) 000049 1787842629.208564 read 0x00e8aa thru 0x01162a (0x2d81 bytes) 000050 1787842629.210224 fallocate 0x00111d thru 0x00b67d (0xa561 bytes) 000051 1787842629.235611 fallocate 0x01cfd2 thru 0x028662 (0xb691 bytes) 000052 1787842629.250648 punch from 0x003227 to 0x00c8d7 (0x96b1 bytes) 000053 1787842629.269651 mapread 0x033ffa thru 0x036ef8 (0x2eff bytes) 000054 1787842629.274118 punch from 0x0127fb to 0x014d6c (0x2572 bytes) 000055 1787842629.280943 trunc from 0x036ef9 to 0x012ecb (0x2402d bytes) 000056 1787842629.351123 read 0x008f4f thru 0x012ecb (0x9f7d bytes) 000057 1787842629.384993 trunc from 0x005248 to 0x009983 (0x473c bytes) 000058 1787842629.401795 mapwrite 0x023126 thru 0x02427b (0x1156 bytes) 000059 1787842629.500815 punch from 0x000c4b to 0x0081ae (0x7564 bytes) 000060 1787842629.516690 trunc from 0x010f6c to 0x01a7b5 (0x984a bytes) 000061 1787842629.525794 trunc from 0x009b0b to 0x00ee29 (0x531f bytes) 000062 1787842629.531866 mapread 0x01e65d thru 0x0210ba (0x2a5e bytes) 000063 1787842629.533458 mapwrite 0x03d1eb thru 0x03ffff (0x2e15 bytes) 000064 1787842629.613079 write 0x008c54 thru 0x00db73 (0x4f20 bytes) 000065 1787842629.641450 trunc from 0x02160d to 0x030b25 (0xf519 bytes) 000066 1787842629.648066 trunc from 0x040000 to 0x005fff (0x3a000 bytes) 000067 1787842629.671817 write 0x021a54 thru 0x023f16 (0x24c3 bytes) 000068 1787842629.673350 punch from 0x00af1e to 0x014400 (0x94e3 bytes) 000069 1787842629.678249 read 0x01eb7a thru 0x023f16 (0x539d bytes) 000070 1787842629.690680 mapread 0x019563 thru 0x01a8bd (0x135b bytes) 000071 1787842629.692677 fallocate 0x02bc5a thru 0x0332c1 (0x7668 bytes) 000072 1787842629.703984 write 0x011023 thru 0x01172a (0x708 bytes) 000073 1787842629.714082 trunc from 0x003fe3 to 0x00b48d (0x74ab bytes) 000074 1787842629.723877 trunc from 0x00d701 to 0x0113ba (0x3cba bytes) 000075 1787842629.755550 read 0x01eb49 thru 0x0230dc (0x4594 bytes) 000076 1787842629.756867 read 0x0049c3 thru 0x00e3b9 (0x99f7 bytes) 000077 1787842629.779839 read 0x00e1eb thru 0x01c372 (0xe188 bytes) 000079 1787842629.786265 punch from 0x0097d2 to 0x014414 (0xac43 bytes) 000080 1787842629.800123 fallocate 0x03eb5d thru 0x03ffff (0x14a3 bytes) 000082 1787842629.808111 trunc from 0x040000 to 0x02e8be (0x11741 bytes) 000083 1787842629.842970 punch from 0x00154e to 0x00ffc1 (0xea74 bytes) 000084 1787842629.848714 punch from 0x02d6fb to 0x02e8be (0x11c4 bytes) 000085 1787842629.857583 MIRROR_EXTEND to 2 mirrors 000086 1787842630.688722 read 0x001d99 thru 0x00fef8 (0xe160 bytes) 000087 1787842630.703247 fallocate 0x01d029 thru 0x029c84 (0xcc5c bytes) 000088 1787842630.762194 mapread 0x014fd8 thru 0x0164f0 (0x1519 bytes) 000089 1787842630.796889 trunc from 0x02011c to 0x023c49 (0x3b2e bytes) 000090 1787842630.803355 fallocate 0x00bd73 thru 0x00e708 (0x2996 bytes) 000091 1787842630.811745 write 0x021d3d thru 0x02637e (0x4642 bytes) 000092 1787842630.846729 fallocate 0x0277c6 thru 0x0353a8 (0xdbe3 bytes) 000093 1787842630.880868 MIRROR_RESYNC 2 mirrors 000094 1787842632.238715 mapwrite 0x03da7b thru 0x03ffff (0x2585 bytes) 000096 1787842632.416715 trunc from 0x040000 to 0x032757 (0xd8a8 bytes) 000097 1787842632.451770 write 0x01a4eb thru 0x02350c (0x9022 bytes) 000100 1787842632.515396 read 0x01effe thru 0x024843 (0x5846 bytes) 000101 1787842632.519383 mapread 0x00982f thru 0x009a10 (0x1e2 bytes) 000102 1787842632.552817 punch from 0x01632a to 0x026312 (0xffe9 bytes) 000103 1787842632.617896 fallocate 0x0365c6 thru 0x03ffff (0x9a3a bytes) 000104 1787842632.640298 mapread 0x00eed9 thru 0x010513 (0x163b bytes) 000105 1787842632.653001 mapread 0x036529 thru 0x03ffff (0x9ad7 bytes) 000106 1787842632.691861 punch from 0x000f3a to 0x007e69 (0x6f30 bytes) 000107 1787842632.701071 read 0x03eae1 thru 0x03ffff (0x151f bytes) 000108 1787842632.701691 trunc from 0x040000 to 0x017fd6 (0x28029 bytes) 000110 1787842632.723537 fallocate 0x01e2b6 thru 0x026acd (0x8818 bytes) 000111 1787842632.734549 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x26ace): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26ace): Mirror 2: 0x208a50c7 (stale) CRC-32 checksum value for chunk [0, 0x26ace): Mirror 1: 0xdbbd995 Mirror 3: 0xdbbd995 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26ace: skipped 000112 1787842633.705610 read 0x0064a6 thru 0x00b45e (0x4fb9 bytes) 000113 1787842633.732466 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x26ace): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26ace): Mirror 2: 0x208a50c7 (stale) CRC-32 checksum value for chunk [0, 0x26ace): Mirror 1: 0xdbbd995 Mirror 3: 0xdbbd995 Mirror 4: 0xdbbd995 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26ace: skipped 000114 1787842635.076986 mapread 0x00a5f1 thru 0x00f5ca (0x4fda bytes) 000115 1787842635.115972 read 0x019036 thru 0x026acd (0xda98 bytes) 000117 1787842635.139857 fallocate 0x014a44 thru 0x01d95c (0x8f19 bytes) 000118 1787842635.235756 mapread 0x0106c8 thru 0x01c3de (0xbd17 bytes) 000119 1787842635.290786 punch from 0x019778 to 0x026acd (0xd356 bytes) 000120 1787842635.300390 fallocate 0x02c08c thru 0x036bb7 (0xab2c bytes) 000121 1787842635.317479 trunc from 0x026ace to 0x034a42 (0xdf75 bytes) 000122 1787842635.390332 mapread 0x002e7b thru 0x007d7b (0x4f01 bytes) 000123 1787842635.408878 punch from 0x023cf6 to 0x0320ec (0xe3f7 bytes) 000124 1787842635.423113 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x34a43): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34a43): Mirror 2: 0xe1bb1e9a (stale) Mirror 3: 0xadb5fcf0 (stale) Mirror 4: 0xadb5fcf0 (stale) CRC-32 checksum value for chunk [0, 0x34a43): Mirror 1: 0xadb5fcf0 Mirror 5: 0xadb5fcf0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34a43: skipped 000125 1787842636.520141 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34a43): Mirror 2: 0xe1bb1e9a (stale) Mirror 3: 0xadb5fcf0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34a43): Mirror 2: 0xe1bb1e9a (stale) Mirror 3: 0xadb5fcf0 (stale) CRC-32 checksum value for chunk [0, 0x34a43): Mirror 1: 0xadb5fcf0 Mirror 5: 0xadb5fcf0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34a43: skipped 000126 1787842637.414646 write 0x031cab thru 0x039be4 (0x7f3a bytes) 000127 1787842637.468590 mapwrite 0x03b442 thru 0x03ffff (0x4bbe bytes) 000128 1787842637.634698 mapread 0x0334e4 thru 0x033620 (0x13d bytes) 000130 1787842637.638794 mapwrite 0x0019aa thru 0x002167 (0x7be bytes) 000131 1787842637.696635 read 0x0254b8 thru 0x02d870 (0x83b9 bytes) 000133 1787842637.726688 trunc from 0x040000 to 0x02cb99 (0x13466 bytes) 000134 1787842637.752396 write 0x026535 thru 0x02ea0d (0x84d9 bytes) 000135 1787842637.754794 read 0x009cd6 thru 0x0161fa (0xc525 bytes) 000136 1787842637.767139 trunc from 0x02ea0e to 0x020381 (0xe68c bytes) 000138 1787842637.798637 mapwrite 0x036f93 thru 0x03c3dd (0x544b bytes) 000139 1787842637.920727 write 0x01edb2 thru 0x02b8db (0xcb2a bytes) 000140 1787842637.935062 mapread 0x02eb38 thru 0x03c3dd (0xd8a6 bytes) 000141 1787842637.959646 fallocate 0x00ce50 thru 0x01a728 (0xd8d9 bytes) 000142 1787842637.970547 fallocate 0x01b36b thru 0x0236be (0x8354 bytes) 000143 1787842637.986744 fallocate 0x038b17 thru 0x03b249 (0x2733 bytes) 000144 1787842638.001061 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c3de): Mirror 2: 0x10016d2e (stale) Mirror 5: 0xcbb228e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c3de): Mirror 2: 0x10016d2e (stale) Mirror 5: 0xcbb228e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c3de: skipped 000145 1787842638.425721 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c3de): Mirror 2: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c3de): Mirror 2: 0x10016d2e (stale) Mirror 5: 0xcbb228e1 (stale) CRC-32 checksum value for chunk [0, 0x3c3de): Mirror 1: 0xb1d07754 Mirror 6: 0xb1d07754 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c3de: skipped 000146 1787842639.688156 read 0x02c6e8 thru 0x03a750 (0xe069 bytes) 000148 1787842639.705735 fallocate 0x01775f thru 0x01d902 (0x61a4 bytes) 000150 1787842639.755420 read 0x01a2fa thru 0x026f62 (0xcc69 bytes) 000151 1787842639.766398 trunc from 0x03c3de to 0x014f5a (0x27483 bytes) 000152 1787842639.796552 trunc from 0x014f5b to 0x02839d (0x13443 bytes) 000153 1787842639.815374 mapread 0x00f634 thru 0x01eede (0xf8ab bytes) 000154 1787842639.873345 fallocate 0x022d7d thru 0x02527b (0x24ff bytes) 000155 1787842639.879593 trunc from 0x00f6ec to 0x01d127 (0xda3c bytes) 000156 1787842639.884649 trunc from 0x0188a2 to 0x0277d5 (0xef34 bytes) 000157 1787842639.891194 fallocate 0x0393f5 thru 0x03ffff (0x6c0b bytes) 000158 1787842639.897713 fallocate 0x0244e1 thru 0x030a45 (0xc565 bytes) 000159 1787842639.910185 MIRROR_RESYNC 4 mirrors 000160 1787842641.268797 mapwrite 0x0023f9 thru 0x00ebf4 (0xc7fc bytes) 000161 1787842641.484945 MIRROR_RESYNC 4 mirrors 000162 1787842642.654579 fallocate 0x01fa77 thru 0x02ce90 (0xd41a bytes) 000163 1787842642.704588 trunc from 0x004743 to 0x00c472 (0x7d30 bytes) 000164 1787842642.710538 trunc from 0x030a46 to 0x03ede5 (0xe3a0 bytes) 000165 1787842642.731470 fallocate 0x02f64f thru 0x034461 (0x4e13 bytes) 000166 1787842642.740723 mapread 0x027546 thru 0x029f7e (0x2a39 bytes) 000167 1787842642.765680 trunc from 0x03ede6 to 0x009d3c (0x350a9 bytes) 000168 1787842642.796895 mapwrite 0x007058 thru 0x01371b (0xc6c4 bytes) 000169 1787842642.867223 mapwrite 0x033beb thru 0x03a202 (0x6618 bytes) 000170 1787842642.908245 read 0x02c5be thru 0x035e2b (0x986e bytes) 000171 1787842642.909040 punch from 0x02d7f5 to 0x0335d9 (0x5de5 bytes) 000172 1787842642.919051 fallocate 0x00a99a thru 0x00e2a4 (0x390b bytes) 000173 1787842642.924464 punch from 0x000a14 to 0x00213a (0x1727 bytes) 000174 1787842642.933483 write 0x00b45a thru 0x017eac (0xca53 bytes) 000175 1787842642.952252 punch from 0x03576e to 0x039f5e (0x47f1 bytes) 000177 1787842642.967438 trunc from 0x0363f7 to 0x036f3b (0xb45 bytes) 000178 1787842642.984534 mapwrite 0x037172 thru 0x03e652 (0x74e1 bytes) 000179 1787842643.092346 trunc from 0x005d81 to 0x014f7d (0xf1fd bytes) 000180 1787842643.101196 mapwrite 0x012b37 thru 0x015031 (0x24fb bytes) 000181 1787842643.171204 MIRROR_RESYNC 4 mirrors 000182 1787842644.778248 mapwrite 0x02364d thru 0x0303da (0xcd8e bytes) 000183 1787842644.927368 MIRROR_RESYNC 4 mirrors 000184 1787842646.799058 trunc from 0x0202dc to 0x02871c (0x8441 bytes) 000185 1787842646.850826 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e653): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e653): Mirror 2: 0x85d51385 (stale) Mirror 5: 0x85d51385 (stale) Mirror 6: 0x85d51385 (stale) CRC-32 checksum value for chunk [0, 0x3e653): Mirror 1: 0xb7a7397 Mirror 7: 0xb7a7397 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e653: skipped 000186 1787842648.025879 read 0x0178c4 thru 0x020876 (0x8fb3 bytes) 000187 1787842648.048139 MIRROR_RESYNC 5 mirrors 000188 1787842649.873285 punch from 0x0071b8 to 0x01541e (0xe267 bytes) 000189 1787842649.955749 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3e653): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e653): Mirror 2: 0xb7a7397 (stale) Mirror 5: 0xb7a7397 (stale) Mirror 6: 0xb7a7397 (stale) Mirror 7: 0xb7a7397 (stale) CRC-32 checksum value for chunk [0, 0x3e653): Mirror 1: 0x3800c54 Mirror 8: 0x3800c54 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e653: skipped 000190 1787842651.249327 mapwrite 0x0109f2 thru 0x019e89 (0x9498 bytes) 000191 1787842651.372241 punch from 0x03cabb to 0x03cb22 (0x68 bytes) 000192 1787842651.379443 fallocate 0x01ccaa thru 0x022b47 (0x5e9e bytes) 000193 1787842651.396513 trunc from 0x03e653 to 0x0274f6 (0x1715c bytes) 000194 1787842651.457866 read 0x0274d6 thru 0x0274f6 (0x21 bytes) 000195 1787842651.474446 trunc from 0x0274f7 to 0x0038df (0x23c17 bytes) 000196 1787842651.503272 fallocate 0x033fbf thru 0x03e9da (0xaa1c bytes) 000198 1787842651.513843 fallocate 0x036a9b thru 0x038c1e (0x2184 bytes) 000199 1787842651.536347 mapwrite 0x01356d thru 0x01c580 (0x9014 bytes) 000200 1787842651.659464 read 0x019aea thru 0x01c4df (0x29f6 bytes) 000201 1787842651.660191 fallocate 0x02503c thru 0x025d0a (0xccf bytes) 000202 1787842651.674647 fallocate 0x025e6a thru 0x02d234 (0x73cb bytes) 000203 1787842651.697035 write 0x02e454 thru 0x03b5dd (0xd18a bytes) 000204 1787842651.713242 trunc from 0x03b5de to 0x030f0d (0xa6d0 bytes) 000205 1787842651.752459 read 0x00ae4d thru 0x013a78 (0x8c2c bytes) 000206 1787842651.785566 mapwrite 0x017331 thru 0x01c6af (0x537f bytes) 000207 1787842651.809178 write 0x00b61d thru 0x0165e0 (0xafc4 bytes) 000208 1787842651.813027 trunc from 0x030f0e to 0x01bf9f (0x14f6e bytes) 000209 1787842651.834007 read 0x009648 thru 0x011953 (0x830c bytes) 000210 1787842651.836052 write 0x03201e thru 0x03b73e (0x9721 bytes) 000211 1787842651.839122 mapread 0x027d59 thru 0x02bc8e (0x3f36 bytes) 000212 1787842651.847868 trunc from 0x003552 to 0x011cdc (0xe78b bytes) 000213 1787842651.864164 write 0x02da2e thru 0x038026 (0xa5f9 bytes) 000215 1787842651.867298 trunc from 0x036baa to 0x037eda (0x1331 bytes) 000216 1787842651.892755 fallocate 0x00f1d3 thru 0x01da3e (0xe86c bytes) 000217 1787842651.900675 trunc from 0x0112c0 to 0x017f2f (0x6c70 bytes) 000218 1787842651.908500 read 0x023518 thru 0x031b48 (0xe631 bytes) 000219 1787842651.912315 fallocate 0x017ab0 thru 0x024090 (0xc5e1 bytes) 000220 1787842651.920158 fallocate 0x012f4c thru 0x019677 (0x672c bytes) 000221 1787842651.931239 read 0x0080b6 thru 0x015f8e (0xded9 bytes) 000222 1787842651.944112 mapread 0x00c0cf thru 0x01437e (0x82b0 bytes) 000223 1787842651.953976 read 0x03a925 thru 0x03b73e (0xe1a bytes) 000224 1787842651.955214 read 0x029619 thru 0x02ebcc (0x55b4 bytes) 000225 1787842651.958097 mapread 0x019669 thru 0x01ff05 (0x689d bytes) 000226 1787842651.978144 MIRROR_RESYNC 6 mirrors 000227 1787842654.193297 MIRROR_EXTEND to 7 mirrors 000228 1787842655.343252 mapwrite 0x02f0ee thru 0x03eef0 (0xfe03 bytes) 000229 1787842655.496539 fallocate 0x02d953 thru 0x03beb8 (0xe566 bytes) 000231 1787842655.505462 trunc from 0x03eef1 to 0x012708 (0x2c7e8 bytes) 000232 1787842655.529771 trunc from 0x00b3d6 to 0x012708 (0x7333 bytes) 000233 1787842655.534353 trunc from 0x000e41 to 0x00ca61 (0xbc21 bytes) 000234 1787842655.544403 write 0x03ce4e thru 0x03fd6a (0x2f1d bytes) 000235 1787842655.545598 trunc from 0x03f8c8 to 0x03fd6a (0x4a3 bytes) 000236 1787842655.562795 write 0x016fd4 thru 0x018673 (0x16a0 bytes) 000237 1787842655.576680 fallocate 0x009c8d thru 0x00b9ea (0x1d5e bytes) 000240 1787842655.590171 trunc from 0x03fd6b to 0x03f5b0 (0x7ba bytes) 000241 1787842655.630155 mapread 0x004810 thru 0x00dea9 (0x969a bytes) 000243 1787842655.674523 MIRROR_RESYNC 7 mirrors 000244 1787842657.660202 mapwrite 0x02ce59 thru 0x02f35d (0x2505 bytes) 000245 1787842657.755335 write 0x022af1 thru 0x02eeab (0xc3bb bytes) 000246 1787842657.769864 punch from 0x035c10 to 0x03b809 (0x5bfa bytes) 000248 1787842657.783169 MIRROR_RESYNC 7 mirrors 000249 1787842659.700214 fallocate 0x03abc1 thru 0x03ffff (0x543f bytes) 000250 1787842659.762627 write 0x0038bc thru 0x00bd81 (0x84c6 bytes) 000251 1787842659.781983 trunc from 0x003663 to 0x0116f6 (0xe094 bytes) 000252 1787842659.795474 read 0x01429a thru 0x01b8f9 (0x7660 bytes) 000253 1787842659.806124 fallocate 0x00181f thru 0x00bd59 (0xa53b bytes) 000254 1787842659.811707 write 0x00d464 thru 0x01bab5 (0xe652 bytes) 000255 1787842659.823977 punch from 0x022222 to 0x02f648 (0xd427 bytes) 000256 1787842659.830058 trunc from 0x040000 to 0x02ab6c (0x15493 bytes) 000257 1787842659.856636 mapwrite 0x02c119 thru 0x034114 (0x7ffc bytes) 000259 1787842659.944028 read 0x02c2ad thru 0x02eed1 (0x2c25 bytes) 000260 1787842659.946088 read 0x00a50f thru 0x01288d (0x837f bytes) 000261 1787842659.965369 MIRROR_RESYNC 7 mirrors 000262 1787842662.849791 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) 000263 1787842663.472108 mapwrite 0x022daa thru 0x030de5 (0xe03c bytes) 000264 1787842663.682313 MIRROR_RESYNC 6 mirrors 000265 1787842665.890232 MIRROR_EXTEND to 7 mirrors 000266 1787842666.882358 write 0x01b8e0 thru 0x02a70c (0xee2d bytes) 000267 1787842666.948845 write 0x02e2e4 thru 0x03b0b3 (0xcdd0 bytes) 000268 1787842666.962524 write 0x02f972 thru 0x03d689 (0xdd18 bytes) 000269 1787842666.965578 mapwrite 0x0064cf thru 0x00f7b8 (0x92ea bytes) 000270 1787842667.019050 punch from 0x03190b to 0x032fb6 (0x16ac bytes) 000271 1787842667.025700 mapwrite 0x028337 thru 0x03656d (0xe237 bytes) 000272 1787842667.064487 mapwrite 0x000283 thru 0x001501 (0x127f bytes) 000273 1787842667.084896 read 0x030b76 thru 0x038a3c (0x7ec7 bytes) 000274 1787842667.086168 fallocate 0x006433 thru 0x00fb3f (0x970d bytes) 000275 1787842667.093154 fallocate 0x036784 thru 0x03ffff (0x987c bytes) 000276 1787842667.099822 read 0x02c44f thru 0x035235 (0x8de7 bytes) 000277 1787842667.101224 mapwrite 0x005878 thru 0x014d5c (0xf4e5 bytes) 000278 1787842667.140768 write 0x0171ed thru 0x021646 (0xa45a bytes) 000279 1787842667.142348 trunc from 0x03d68a to 0x023227 (0x1a462 bytes) 000280 1787842667.172787 trunc from 0x003b37 to 0x0043e4 (0x8ae bytes) 000281 1787842667.179446 trunc from 0x01685c to 0x023227 (0xc9cc bytes) 000282 1787842667.202455 trunc from 0x023228 to 0x0237ef (0x5c8 bytes) 000283 1787842667.222568 trunc from 0x01bda6 to 0x01d9be (0x1c19 bytes) 000284 1787842667.229582 punch from 0x000afd to 0x004237 (0x373b bytes) 000285 1787842667.237768 read 0x007895 thru 0x017667 (0xfdd3 bytes) 000286 1787842667.250015 write 0x037b6a thru 0x03b883 (0x3d1a bytes) 000287 1787842667.258871 MIRROR_RESYNC 7 mirrors 000288 1787842669.299819 mapread 0x0137ee thru 0x019f71 (0x6784 bytes) 000289 1787842669.324043 MIRROR_RESYNC 7 mirrors 000290 1787842669.869728 trunc from 0x0041eb to 0x00cabe (0x88d4 bytes) 000291 1787842669.992825 read 0x02bd9f thru 0x035b88 (0x9dea bytes) 000292 1787842670.004185 read 0x0021f7 thru 0x005c83 (0x3a8d bytes) 000293 1787842670.031753 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3b884): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b884): Mirror 2: 0x3135279b (stale) Mirror 5: 0x3135279b (stale) Mirror 6: 0x3135279b (stale) Mirror 7: 0x3135279b (stale) Mirror 9: 0x3135279b (stale) Mirror 10: 0x3135279b (stale) CRC-32 checksum value for chunk [0, 0x3b884): Mirror 1: 0xe974a793 Mirror 11: 0xe974a793 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b884: skipped 000294 1787842671.083770 punch from 0x02365e to 0x031ba7 (0xe54a bytes) 000295 1787842671.144455 fallocate 0x018e5b thru 0x01e0b1 (0x5257 bytes) 000297 1787842671.155417 read 0x004f68 thru 0x0067b9 (0x1852 bytes) 000298 1787842671.162055 read 0x032949 thru 0x03b883 (0x8f3b bytes) 000299 1787842671.173959 mapread 0x003bb2 thru 0x00b84d (0x7c9c bytes) 000301 1787842671.177025 write 0x03ea27 thru 0x03ffff (0x15d9 bytes) 000302 1787842671.183840 mapread 0x019c04 thru 0x024c84 (0xb081 bytes) 000303 1787842671.186862 mapread 0x03184c thru 0x035660 (0x3e15 bytes) 000304 1787842671.188810 punch from 0x012c32 to 0x015f40 (0x330f bytes) 000306 1787842671.196100 read 0x0178c3 thru 0x026464 (0xeba2 bytes) 000307 1787842671.196896 trunc from 0x00ddcd to 0x01188d (0x3ac1 bytes) 000308 1787842671.211354 mapwrite 0x0272f4 thru 0x03294e (0xb65b bytes) 000309 1787842671.303655 punch from 0x0020c0 to 0x00aa68 (0x89a9 bytes) 000310 1787842671.319668 punch from 0x01b29e to 0x02ac94 (0xf9f7 bytes) 000311 1787842671.339193 mapread 0x00b6a8 thru 0x00e084 (0x29dd bytes) 000312 1787842671.393588 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1cd5a0b3 (stale) Mirror 5: 0x1cd5a0b3 (stale) Mirror 6: 0x1cd5a0b3 (stale) Mirror 7: 0x1cd5a0b3 (stale) Mirror 9: 0x1cd5a0b3 (stale) Mirror 11: 0x4d288bf5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1cd5a0b3 (stale) Mirror 5: 0x1cd5a0b3 (stale) Mirror 6: 0x1cd5a0b3 (stale) Mirror 7: 0x1cd5a0b3 (stale) Mirror 9: 0x1cd5a0b3 (stale) Mirror 11: 0x4d288bf5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000313 1787842671.933866 fallocate 0x02f882 thru 0x0304e3 (0xc62 bytes) 000314 1787842671.942551 trunc from 0x040000 to 0x012ba0 (0x2d45f bytes) 000315 1787842671.971757 trunc from 0x012ba1 to 0x0326eb (0x1fb4b bytes) 000316 1787842671.995434 read 0x01800e thru 0x025245 (0xd238 bytes) 000317 1787842672.006105 mapread 0x000152 thru 0x00f0f9 (0xefa8 bytes) 000318 1787842672.060498 fallocate 0x00d82d thru 0x0183b8 (0xab8c bytes) 000320 1787842672.098069 mapread 0x032632 thru 0x0326eb (0xba bytes) 000321 1787842672.118880 trunc from 0x013503 to 0x020220 (0xcd1e bytes) 000322 1787842672.125680 fallocate 0x017ab9 thru 0x01e6a6 (0x6bee bytes) 000323 1787842672.144252 MIRROR_RESYNC 7 mirrors 000324 1787842674.724600 punch from 0x019d7f to 0x01df1a (0x419c bytes) 000325 1787842674.838957 mapread 0x02dd51 thru 0x0326eb (0x499b bytes) 000326 1787842674.902954 punch from 0x013d08 to 0x021c78 (0xdf71 bytes) 000327 1787842674.926793 punch from 0x00b0e0 to 0x0151d1 (0xa0f2 bytes) 000328 1787842674.939877 mapread 0x012ad5 thru 0x01ce8a (0xa3b6 bytes) 000329 1787842675.007837 read 0x018612 thru 0x026329 (0xdd18 bytes) 000330 1787842675.042768 mapread 0x017629 thru 0x019d8b (0x2763 bytes) 000331 1787842675.046709 fallocate 0x0268b3 thru 0x02e4a4 (0x7bf2 bytes) 000332 1787842675.067512 write 0x00c4c4 thru 0x00fb58 (0x3695 bytes) 000333 1787842675.068671 trunc from 0x0326ec to 0x00d30b (0x253e0 bytes) 000334 1787842675.126414 trunc from 0x00d30c to 0x004ef0 (0x841b bytes) 000335 1787842675.185950 fallocate 0x03c39d thru 0x03ffff (0x3c63 bytes) 000336 1787842675.196083 mapread 0x023fef thru 0x0247df (0x7f1 bytes) 000337 1787842675.229423 trunc from 0x02977e to 0x034214 (0xaa97 bytes) 000338 1787842675.236888 trunc from 0x040000 to 0x006870 (0x3978f bytes) 000340 1787842675.293247 read 0x0039f8 thru 0x006870 (0x2e79 bytes) 000341 1787842675.321549 trunc from 0x0004ed to 0x004306 (0x3e1a bytes) 000342 1787842675.329003 read 0x00455b thru 0x006870 (0x2316 bytes) 000343 1787842675.344547 trunc from 0x003c5d to 0x006870 (0x2c14 bytes) 000344 1787842675.352167 trunc from 0x006871 to 0x039a2e (0x331be bytes) 000345 1787842675.381956 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39a2f): Mirror 2: 0x8f52737f (stale) Mirror 5: 0x8f52737f (stale) Mirror 6: 0x8f52737f (stale) Mirror 7: 0x8f52737f (stale) Mirror 9: 0x8f52737f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a2f): Mirror 2: 0x8f52737f (stale) Mirror 5: 0x8f52737f (stale) Mirror 6: 0x8f52737f (stale) Mirror 7: 0x8f52737f (stale) Mirror 9: 0x8f52737f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a2f: skipped 000346 1787842676.287405 mapwrite 0x0363c3 thru 0x03ffff (0x9c3d bytes) 000348 1787842676.393183 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x773ffdc7 (stale) Mirror 5: 0x773ffdc7 (stale) Mirror 6: 0x773ffdc7 (stale) Mirror 7: 0x773ffdc7 (stale) Mirror 9: 0x773ffdc7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x84794af0 Mirror 10: 0x84794af0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000349 1787842677.535077 punch from 0x02243d to 0x02a6cf (0x8293 bytes) 000350 1787842677.605747 trunc from 0x040000 to 0x01531c (0x2ace3 bytes) 000351 1787842677.694345 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1531d): Mirror 2: 0x71d78ad9 (stale) Mirror 5: 0x71d78ad9 (stale) Mirror 6: 0x71d78ad9 (stale) Mirror 9: 0x71d78ad9 (stale) Mirror 10: 0x7097a855 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1531d): Mirror 2: 0x71d78ad9 (stale) Mirror 5: 0x71d78ad9 (stale) Mirror 6: 0x71d78ad9 (stale) Mirror 9: 0x71d78ad9 (stale) Mirror 10: 0x7097a855 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1531d: skipped 000352 1787842678.363517 mapwrite 0x01c25c thru 0x01fe56 (0x3bfb bytes) 000353 1787842678.478857 punch from 0x01a364 to 0x01fe56 (0x5af3 bytes) 000354 1787842678.494837 fallocate 0x02e6ce thru 0x034451 (0x5d84 bytes) 000355 1787842678.499983 trunc from 0x020f85 to 0x02e569 (0xd5e5 bytes) 000356 1787842678.510408 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34452): Mirror 2: 0xd9e884f1 (stale) Mirror 5: 0xd9e884f1 (stale) Mirror 6: 0xd9e884f1 (stale) Mirror 9: 0xd9e884f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34452): Mirror 2: 0xd9e884f1 (stale) Mirror 5: 0xd9e884f1 (stale) Mirror 6: 0xd9e884f1 (stale) Mirror 9: 0xd9e884f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34452: skipped 000357 1787842679.152243 fallocate 0x0212bc thru 0x02b956 (0xa69b bytes) 000358 1787842679.157483 write 0x02bff4 thru 0x0371a9 (0xb1b6 bytes) 000359 1787842679.170658 fallocate 0x015d33 thru 0x01f6ea (0x99b8 bytes) 000360 1787842679.176209 fallocate 0x0240b2 thru 0x025b83 (0x1ad2 bytes) 000362 1787842679.184303 punch from 0x00c6b6 to 0x00d613 (0xf5e bytes) 000363 1787842679.190280 trunc from 0x0371aa to 0x013722 (0x23a87 bytes) 000364 1787842679.215388 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x13723): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13723): Mirror 2: 0x15529966 (stale) Mirror 5: 0x15529966 (stale) Mirror 6: 0x15529966 (stale) Mirror 9: 0x15529966 (stale) CRC-32 checksum value for chunk [0, 0x13723): Mirror 1: 0xa10bbe57 Mirror 10: 0xa10bbe57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13723: skipped 000365 1787842680.574796 write 0x008dc3 thru 0x016f56 (0xe194 bytes) 000366 1787842680.635808 mapread 0x00a712 thru 0x00ab8b (0x47a bytes) 000367 1787842680.639082 trunc from 0x004b95 to 0x00aca4 (0x6110 bytes) 000368 1787842680.656738 mapread 0x014d91 thru 0x016f56 (0x21c6 bytes) 000369 1787842680.659561 read 0x01694a thru 0x016f56 (0x60d bytes) 000370 1787842680.660880 fallocate 0x007a62 thru 0x00fffb (0x859a bytes) 000372 1787842680.666588 punch from 0x002bb1 to 0x00aba6 (0x7ff6 bytes) 000373 1787842680.676709 punch from 0x00aeb9 to 0x00cc97 (0x1ddf bytes) 000374 1787842680.690801 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16f57): Mirror 5: 0x9ebb936d (stale) Mirror 6: 0x9ebb936d (stale) Mirror 9: 0x9ebb936d (stale) Mirror 10: 0x354b546c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16f57): Mirror 5: 0x9ebb936d (stale) Mirror 6: 0x9ebb936d (stale) Mirror 9: 0x9ebb936d (stale) Mirror 10: 0x354b546c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16f57: skipped 000375 1787842681.288860 read 0x00e660 thru 0x016f56 (0x88f7 bytes) 000377 1787842681.300485 trunc from 0x0062e1 to 0x012ab1 (0xc7d1 bytes) 000378 1787842681.305487 mapwrite 0x0334d0 thru 0x03ffff (0xcb30 bytes) 000379 1787842681.366871 mapread 0x01060f thru 0x013c97 (0x3689 bytes) 000380 1787842681.377427 punch from 0x02c6c7 to 0x03bd60 (0xf69a bytes) 000381 1787842681.388075 read 0x01f4bf thru 0x02c330 (0xce72 bytes) 000382 1787842681.397813 fallocate 0x0177b3 thru 0x021903 (0xa151 bytes) 000383 1787842681.402180 trunc from 0x040000 to 0x02e57e (0x11a81 bytes) 000384 1787842681.418531 read 0x023b51 thru 0x02ae56 (0x7306 bytes) 000385 1787842681.419163 read 0x007d68 thru 0x00b470 (0x3709 bytes) 000386 1787842681.420828 write 0x01f47f thru 0x027677 (0x81f9 bytes) 000387 1787842681.422717 read 0x0086c9 thru 0x009b52 (0x148a bytes) 000388 1787842681.423365 mapread 0x00878f thru 0x00a706 (0x1f78 bytes) 000389 1787842681.433953 read 0x00e53a thru 0x00ecea (0x7b1 bytes) 000390 1787842681.435627 trunc from 0x02e57f to 0x00d706 (0x20e78 bytes) 000391 1787842681.472484 write 0x01c29c thru 0x0271e7 (0xaf4c bytes) 000392 1787842681.476995 read 0x0260e6 thru 0x0271e7 (0x1102 bytes) 000393 1787842681.483739 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x271e8): Mirror 5: 0xe4285c3e (stale) Mirror 6: 0xe4285c3e (stale) Mirror 10: 0x1b0d0cae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x271e8): Mirror 5: 0xe4285c3e (stale) Mirror 6: 0xe4285c3e (stale) Mirror 10: 0x1b0d0cae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x271e8: skipped 000394 1787842682.051453 fallocate 0x033690 thru 0x03709d (0x3a0e bytes) 000395 1787842682.056390 fallocate 0x0059b9 thru 0x007a58 (0x20a0 bytes) 000396 1787842682.065070 write 0x020d2c thru 0x025ba1 (0x4e76 bytes) 000397 1787842682.086323 read 0x01a893 thru 0x01bb76 (0x12e4 bytes) 000398 1787842682.094286 read 0x026a14 thru 0x0271e7 (0x7d4 bytes) 000399 1787842682.117883 fallocate 0x015d19 thru 0x022408 (0xc6f0 bytes) 000400 1787842682.126110 mapwrite 0x033622 thru 0x03a4a7 (0x6e86 bytes) 000401 1787842682.180854 write 0x025b03 thru 0x030f37 (0xb435 bytes) 000403 1787842682.182739 mapread 0x038ef1 thru 0x03a4a7 (0x15b7 bytes) 000404 1787842682.185226 mapwrite 0x0199ea thru 0x0297e3 (0xfdfa bytes) 000405 1787842682.207802 mapwrite 0x027a8a thru 0x02bc4d (0x41c4 bytes) 000407 1787842682.259313 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a4a8): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a4a8): Mirror 5: 0xb9c2bdc (stale) Mirror 6: 0xb9c2bdc (stale) Mirror 10: 0xd295bbc2 (stale) CRC-32 checksum value for chunk [0, 0x3a4a8): Mirror 1: 0xd2eb1c4b Mirror 11: 0xd2eb1c4b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a4a8: skipped 000408 1787842683.454743 write 0x016874 thru 0x023c36 (0xd3c3 bytes) 000409 1787842683.513439 fallocate 0x02d96f thru 0x032909 (0x4f9b bytes) 000410 1787842683.518154 mapwrite 0x031ccf thru 0x037984 (0x5cb6 bytes) 000411 1787842683.652481 write 0x030309 thru 0x039d85 (0x9a7d bytes) 000412 1787842683.668508 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3a4a8): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a4a8): Mirror 5: 0xb9c2bdc (stale) Mirror 6: 0xb9c2bdc (stale) Mirror 10: 0xd295bbc2 (stale) Mirror 11: 0xd2eb1c4b (stale) CRC-32 checksum value for chunk [0, 0x3a4a8): Mirror 1: 0x68e69c02 Mirror 12: 0x68e69c02 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a4a8: skipped 000413 1787842685.469633 read 0x0292bc thru 0x02b429 (0x216e bytes) 000414 1787842685.477865 trunc from 0x021068 to 0x030ea6 (0xfe3f bytes) 000415 1787842685.541829 trunc from 0x01f4a9 to 0x029729 (0xa281 bytes) 000417 1787842685.549418 mapwrite 0x021cc1 thru 0x025a2d (0x3d6d bytes) 000418 1787842685.615097 trunc from 0x02664e to 0x035580 (0xef33 bytes) 000419 1787842685.628729 punch from 0x00394b to 0x00edf3 (0xb4a9 bytes) 000420 1787842685.652159 read 0x0185c9 thru 0x01c6af (0x40e7 bytes) 000421 1787842685.689086 write 0x019852 thru 0x01d8a8 (0x4057 bytes) 000422 1787842685.695086 read 0x0379ed thru 0x03a4a7 (0x2abb bytes) 000424 1787842685.696139 trunc from 0x0014e3 to 0x00b539 (0xa057 bytes) 000425 1787842685.712669 read 0x029021 thru 0x02e9a8 (0x5988 bytes) 000426 1787842685.714720 mapread 0x02964d thru 0x02c009 (0x29bd bytes) 000427 1787842685.716091 trunc from 0x00536c to 0x0115a0 (0xc235 bytes) 000428 1787842685.727855 fallocate 0x0364d1 thru 0x0368d8 (0x408 bytes) 000429 1787842685.740624 write 0x01612a thru 0x024f53 (0xee2a bytes) 000431 1787842685.754960 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3a4a8): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a4a8): Mirror 5: 0xb9c2bdc (stale) Mirror 6: 0xb9c2bdc (stale) Mirror 10: 0xd295bbc2 (stale) Mirror 11: 0xd2eb1c4b (stale) Mirror 12: 0x68e69c02 (stale) CRC-32 checksum value for chunk [0, 0x3a4a8): Mirror 1: 0xfb8e3bd2 Mirror 13: 0xfb8e3bd2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a4a8: skipped 000432 1787842687.539295 trunc from 0x03a4a8 to 0x01cd45 (0x1d762 bytes) 000433 1787842687.597479 trunc from 0x0170a5 to 0x01cd45 (0x5ca1 bytes) 000434 1787842687.605573 read 0x00bfab thru 0x019b0d (0xdb63 bytes) 000435 1787842687.614164 fallocate 0x022eb7 thru 0x024857 (0x19a1 bytes) 000436 1787842687.624886 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1cd46): Mirror 5: 0xd00e3441 (stale) Mirror 6: 0xd00e3441 (stale) Mirror 10: 0x2ef66891 (stale) Mirror 12: 0xb7baf4c8 (stale) Mirror 13: 0x59ac857e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cd46): Mirror 5: 0xd00e3441 (stale) Mirror 6: 0xd00e3441 (stale) Mirror 10: 0x2ef66891 (stale) Mirror 12: 0xb7baf4c8 (stale) Mirror 13: 0x59ac857e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cd46: skipped 000439 1787842688.088325 write 0x00dcd5 thru 0x010caf (0x2fdb bytes) 000440 1787842688.107705 trunc from 0x0119cf to 0x014cbd (0x32ef bytes) 000441 1787842688.117889 fallocate 0x00b485 thru 0x011497 (0x6013 bytes) 000442 1787842688.124592 write 0x01b03c thru 0x021630 (0x65f5 bytes) 000443 1787842688.132864 trunc from 0x01a0b9 to 0x01d430 (0x3378 bytes) 000445 1787842688.160515 read 0x005ef2 thru 0x00952b (0x363a bytes) 000446 1787842688.172628 trunc from 0x021631 to 0x02e26e (0xcc3e bytes) 000447 1787842688.207712 fallocate 0x026315 thru 0x0271d6 (0xec2 bytes) 000448 1787842688.214269 write 0x005ef5 thru 0x00d61b (0x7727 bytes) 000449 1787842688.229661 trunc from 0x021e2b to 0x022c52 (0xe28 bytes) 000450 1787842688.243327 trunc from 0x02e26f to 0x007318 (0x26f56 bytes) 000451 1787842688.283742 write 0x0306cf thru 0x034dd5 (0x4707 bytes) 000453 1787842688.290146 mapread 0x0132e5 thru 0x01e565 (0xb281 bytes) 000454 1787842688.351764 mapwrite 0x024cef thru 0x02d742 (0x8a54 bytes) 000456 1787842688.407515 read 0x020167 thru 0x02cc1e (0xcab8 bytes) 000457 1787842688.408343 fallocate 0x017953 thru 0x020a00 (0x90ae bytes) 000458 1787842688.416714 mapread 0x025f25 thru 0x0333c6 (0xd4a2 bytes) 000460 1787842688.422565 mapwrite 0x023ff9 thru 0x0256dc (0x16e4 bytes) 000461 1787842688.460557 mapread 0x01ad60 thru 0x0218e8 (0x6b89 bytes) 000462 1787842688.464325 punch from 0x0294a6 to 0x034dd5 (0xb930 bytes) 000463 1787842688.485982 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34dd6): Mirror 6: 0x59ba6d71 (stale) Mirror 10: 0xe8a68aef (stale) Mirror 12: 0x8a2d8cd5 (stale) Mirror 13: 0x84228368 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34dd6): Mirror 6: 0x59ba6d71 (stale) Mirror 10: 0xe8a68aef (stale) Mirror 12: 0x8a2d8cd5 (stale) Mirror 13: 0x84228368 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34dd6: skipped 000465 1787842689.256391 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34dd6): Mirror 10: 0xe8a68aef (stale) Mirror 12: 0x8a2d8cd5 (stale) Mirror 13: 0x84228368 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34dd6): Mirror 10: 0xe8a68aef (stale) Mirror 12: 0x8a2d8cd5 (stale) Mirror 13: 0x84228368 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34dd6: skipped 000466 1787842689.721569 mapread 0x01038a thru 0x0129a8 (0x261f bytes) 000467 1787842689.751611 punch from 0x006c5c to 0x0127ed (0xbb92 bytes) 000468 1787842689.760569 trunc from 0x022ec8 to 0x02363c (0x775 bytes) 000469 1787842689.767527 read 0x01eefe thru 0x02272a (0x382d bytes) 000470 1787842689.784100 trunc from 0x034dd6 to 0x029492 (0xb943 bytes) 000471 1787842689.820625 mapwrite 0x026078 thru 0x034a56 (0xe9df bytes) 000473 1787842689.972343 fallocate 0x012227 thru 0x01ef20 (0xccfa bytes) 000474 1787842689.977679 mapwrite 0x028f29 thru 0x02fdd4 (0x6eac bytes) 000475 1787842689.996550 punch from 0x019046 to 0x02488f (0xb84a bytes) 000476 1787842690.004123 punch from 0x013ec8 to 0x0237fc (0xf935 bytes) 000477 1787842690.015431 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34a57): Mirror 10: 0x55c25d4e (stale) Mirror 12: 0x8c5d962a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34a57): Mirror 10: 0x55c25d4e (stale) Mirror 12: 0x8c5d962a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34a57: skipped 000478 1787842690.371387 mapwrite 0x018e70 thru 0x0267ae (0xd93f bytes) 000479 1787842690.510878 read 0x01a3d1 thru 0x02640a (0xc03a bytes) 000480 1787842690.511999 fallocate 0x030211 thru 0x033aa4 (0x3894 bytes) 000481 1787842690.517691 read 0x0048be thru 0x00dede (0x9621 bytes) 000482 1787842690.542857 punch from 0x02814d to 0x02b0fd (0x2fb1 bytes) 000483 1787842690.566238 mapwrite 0x007371 thru 0x0136de (0xc36e bytes) 000484 1787842690.596331 trunc from 0x034a57 to 0x00e811 (0x26245 bytes) 000485 1787842690.630449 read 0x00794f thru 0x00e811 (0x6ec3 bytes) 000486 1787842690.631098 punch from 0x009b08 to 0x00e811 (0x4d0a bytes) 000487 1787842690.647931 mapwrite 0x003de8 thru 0x0065da (0x27f3 bytes) 000488 1787842690.673696 punch from 0x008946 to 0x00e811 (0x5ecc bytes) 000489 1787842690.683244 write 0x01f1ef thru 0x023c18 (0x4a2a bytes) 000490 1787842690.685792 mapwrite 0x00e973 thru 0x00f786 (0xe14 bytes) 000492 1787842690.719312 fallocate 0x02355e thru 0x032e8c (0xf92f bytes) 000493 1787842690.725011 write 0x004b7a thru 0x00def2 (0x9379 bytes) 000494 1787842690.729733 punch from 0x017ac6 to 0x01c7ed (0x4d28 bytes) 000495 1787842690.735734 mapread 0x019564 thru 0x022cf5 (0x9792 bytes) 000496 1787842690.758494 mapwrite 0x03c816 thru 0x03ffff (0x37ea bytes) 000497 1787842690.811702 read 0x02ce85 thru 0x0327b1 (0x592d bytes) 000498 1787842690.812715 mapwrite 0x0073c0 thru 0x00d2a1 (0x5ee2 bytes) 000499 1787842690.845852 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xee262dd8 (stale) Mirror 12: 0xb344e0c3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa08b929 Mirror 13: 0xa08b929 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000500 1787842692.235424 MIRROR_RESYNC 4 mirrors 000501 1787842693.732707 read 0x02c653 thru 0x032a43 (0x63f1 bytes) 000502 1787842693.762620 mapread 0x034449 thru 0x03ffff (0xbbb7 bytes) 000504 1787842693.831638 write 0x00a3fb thru 0x00e71e (0x4324 bytes) 000505 1787842693.953892 fallocate 0x02955f thru 0x034b83 (0xb625 bytes) 000506 1787842693.969576 punch from 0x014c07 to 0x0151b8 (0x5b2 bytes) 000507 1787842693.985645 trunc from 0x012ba1 to 0x01fcfb (0xd15b bytes) 000509 1787842694.012637 trunc from 0x02ec66 to 0x03c1e3 (0xd57e bytes) 000510 1787842694.028307 MIRROR_RESYNC 4 mirrors 000512 1787842695.879966 trunc from 0x040000 to 0x03e5f6 (0x1a09 bytes) 000513 1787842695.943050 MIRROR_RESYNC 4 mirrors 000514 1787842697.399203 mapwrite 0x038cbb thru 0x03ffff (0x7345 bytes) 000515 1787842697.639704 mapwrite 0x03aa1b thru 0x03eb54 (0x413a bytes) 000516 1787842697.696712 write 0x00f4c8 thru 0x01581a (0x6353 bytes) 000518 1787842697.720617 mapread 0x0188a4 thru 0x025bce (0xd32b bytes) 000519 1787842697.753384 mapread 0x0364ad thru 0x03b672 (0x51c6 bytes) 000520 1787842697.754448 mapread 0x032d32 thru 0x03d8f1 (0xabc0 bytes) 000521 1787842697.796879 read 0x02fdd2 thru 0x03e174 (0xe3a3 bytes) 000522 1787842697.797727 fallocate 0x03d519 thru 0x03ffff (0x2ae7 bytes) 000523 1787842697.805792 trunc from 0x040000 to 0x03e379 (0x1c86 bytes) 000524 1787842697.843346 trunc from 0x01586e to 0x025693 (0xfe26 bytes) 000525 1787842697.889560 MIRROR_RESYNC 4 mirrors 000526 1787842699.383315 fallocate 0x029873 thru 0x029f85 (0x713 bytes) 000527 1787842699.435586 fallocate 0x029386 thru 0x0381c3 (0xee3e bytes) 000529 1787842699.451584 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e37a): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e37a): Mirror 10: 0xff12035d (stale) Mirror 12: 0xff12035d (stale) Mirror 13: 0xff12035d (stale) CRC-32 checksum value for chunk [0, 0x3e37a): Mirror 1: 0xff12035d Mirror 14: 0xff12035d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e37a: skipped 000530 1787842700.515883 read 0x01cb7c thru 0x021414 (0x4899 bytes) 000531 1787842700.528978 write 0x018641 thru 0x026b88 (0xe548 bytes) 000532 1787842700.584232 fallocate 0x038f5c thru 0x03ffff (0x70a4 bytes) 000533 1787842700.589274 trunc from 0x03e37a to 0x038e89 (0x54f0 bytes) 000534 1787842700.622361 MIRROR_RESYNC 5 mirrors 000535 1787842702.510013 mapwrite 0x002713 thru 0x0066c7 (0x3fb5 bytes) 000536 1787842702.638302 trunc from 0x038e8a to 0x02a6df (0xe7aa bytes) 000537 1787842702.667838 punch from 0x028cc7 to 0x02a6df (0x1a19 bytes) 000538 1787842702.685033 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2a6e0): Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a6e0): Mirror 10: 0x62a2fc6d (stale) Mirror 12: 0x62a2fc6d (stale) Mirror 13: 0x62a2fc6d (stale) Mirror 14: 0x62a2fc6d (stale) CRC-32 checksum value for chunk [0, 0x2a6e0): Mirror 1: 0x14d8627c Mirror 15: 0x14d8627c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a6e0: skipped 000539 1787842703.793480 fallocate 0x013650 thru 0x01af45 (0x78f6 bytes) 000540 1787842703.902172 read 0x00ff05 thru 0x0170bc (0x71b8 bytes) 000541 1787842703.928553 trunc from 0x02a6e0 to 0x0092a1 (0x2143e bytes) 000542 1787842703.989248 trunc from 0x0092a2 to 0x01b1c1 (0x11f20 bytes) 000543 1787842704.040330 MIRROR_RESYNC 6 mirrors 000545 1787842705.578388 mapwrite 0x02fe31 thru 0x03e46a (0xe63a bytes) 000546 1787842705.736632 trunc from 0x0067ac to 0x011a25 (0xb27a bytes) 000547 1787842705.746275 trunc from 0x03e46b to 0x01857f (0x25eeb bytes) 000548 1787842705.777405 read 0x00a0d3 thru 0x013959 (0x9887 bytes) 000549 1787842705.809416 mapread 0x002d53 thru 0x0046f0 (0x199e bytes) 000550 1787842705.824423 trunc from 0x018580 to 0x022d26 (0xa7a7 bytes) 000551 1787842705.844419 fallocate 0x02d19b thru 0x03144c (0x42b2 bytes) 000552 1787842705.849232 trunc from 0x015f8c to 0x01c608 (0x667d bytes) 000553 1787842705.860242 trunc from 0x008438 to 0x013c4e (0xb817 bytes) 000554 1787842705.867339 trunc from 0x022d27 to 0x005a3f (0x1d2e7 bytes) 000555 1787842705.908031 fallocate 0x031793 thru 0x032e65 (0x16d3 bytes) 000558 1787842705.920508 punch from 0x0043c9 to 0x005a3f (0x1677 bytes) 000559 1787842705.932433 punch from 0x004faf to 0x005a3f (0xa91 bytes) 000560 1787842705.937430 mapwrite 0x0214a7 thru 0x02d908 (0xc462 bytes) 000561 1787842706.026354 punch from 0x0209bf to 0x0280c3 (0x7705 bytes) 000562 1787842706.033708 punch from 0x0280a5 to 0x02d908 (0x5864 bytes) 000563 1787842706.039829 fallocate 0x0365ec thru 0x037d9f (0x17b4 bytes) 000564 1787842706.044447 mapwrite 0x02c4c9 thru 0x02dbb9 (0x16f1 bytes) 000565 1787842706.113220 trunc from 0x02dbba to 0x01a1d8 (0x139e1 bytes) 000566 1787842706.139486 trunc from 0x008ea7 to 0x00c00e (0x3168 bytes) 000567 1787842706.152814 MIRROR_RESYNC 6 mirrors 000568 1787842707.712402 mapread 0x0090c2 thru 0x00af07 (0x1e46 bytes) 000569 1787842707.722107 read 0x00e1e5 thru 0x0177ce (0x95ea bytes) 000570 1787842707.732681 read 0x005d8f thru 0x00af88 (0x51fa bytes) 000571 1787842707.740932 mapread 0x00081b thru 0x006114 (0x58fa bytes) 000572 1787842707.741653 trunc from 0x002aa1 to 0x00f013 (0xc573 bytes) 000573 1787842707.776415 write 0x037af3 thru 0x03ffff (0x850d bytes) 000574 1787842707.782751 MIRROR_RESYNC 6 mirrors 000575 1787842709.233645 trunc from 0x040000 to 0x00bd85 (0x3427a bytes) 000577 1787842709.295347 read 0x009b9c thru 0x00bd85 (0x21ea bytes) 000578 1787842709.302711 punch from 0x00b1c1 to 0x00bd85 (0xbc5 bytes) 000579 1787842709.307533 mapwrite 0x0215d6 thru 0x023161 (0x1b8c bytes) 000580 1787842709.359653 trunc from 0x023162 to 0x037bff (0x14a9e bytes) 000581 1787842709.380024 read 0x02be11 thru 0x0339db (0x7bcb bytes) 000582 1787842709.400951 mapread 0x0363cf thru 0x0373cc (0xffe bytes) 000583 1787842709.402069 punch from 0x024859 to 0x0296c6 (0x4e6e bytes) 000584 1787842709.406947 write 0x033157 thru 0x038506 (0x53b0 bytes) 000585 1787842709.408513 mapwrite 0x021a50 thru 0x0286b1 (0x6c62 bytes) 000586 1787842709.446102 write 0x02411f thru 0x034045 (0xff27 bytes) 000587 1787842709.447624 mapwrite 0x039530 thru 0x03f068 (0x5b39 bytes) 000588 1787842709.499253 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f069): Mirror 12: 0x550c9e0b (stale) Mirror 13: 0x550c9e0b (stale) Mirror 14: 0x550c9e0b (stale) Mirror 15: 0x550c9e0b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f069): Mirror 12: 0x550c9e0b (stale) Mirror 13: 0x550c9e0b (stale) Mirror 14: 0x550c9e0b (stale) Mirror 15: 0x550c9e0b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f069: skipped 000589 1787842709.909374 write 0x01cceb thru 0x01e279 (0x158f bytes) 000590 1787842709.947415 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f069): Mirror 12: 0x550c9e0b (stale) Mirror 14: 0x550c9e0b (stale) Mirror 15: 0x550c9e0b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f069): Mirror 12: 0x550c9e0b (stale) Mirror 14: 0x550c9e0b (stale) Mirror 15: 0x550c9e0b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f069: skipped 000591 1787842710.377865 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f069): Mirror 14: 0x550c9e0b (stale) Mirror 15: 0x550c9e0b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f069): Mirror 14: 0x550c9e0b (stale) Mirror 15: 0x550c9e0b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f069: skipped 000592 1787842710.813804 MIRROR_RESYNC 3 mirrors 000593 1787842712.125216 write 0x010464 thru 0x018860 (0x83fd bytes) 000594 1787842712.181700 MIRROR_RESYNC 3 mirrors 000595 1787842713.601672 MIRROR_EXTEND to 4 mirrors 000596 1787842714.568547 trunc from 0x03f069 to 0x03c184 (0x2ee4 bytes) 000598 1787842714.633265 trunc from 0x03c185 to 0x00bd99 (0x303eb bytes) 000599 1787842714.658313 read 0x004388 thru 0x00bd99 (0x7a12 bytes) 000601 1787842714.675699 punch from 0x00867f to 0x00bd99 (0x371b bytes) 000603 1787842714.682843 read 0x007c4d thru 0x00bd99 (0x414d bytes) 000604 1787842714.693133 punch from 0x008c23 to 0x00bd99 (0x3177 bytes) 000605 1787842714.702158 MIRROR_RESYNC 4 mirrors 000606 1787842715.928231 write 0x00bdbe thru 0x015896 (0x9ad9 bytes) 000608 1787842715.989261 trunc from 0x0025a0 to 0x00e1b9 (0xbc1a bytes) 000609 1787842716.008720 read 0x00c391 thru 0x015896 (0x9506 bytes) 000610 1787842716.021595 mapread 0x0070b1 thru 0x015689 (0xe5d9 bytes) 000611 1787842716.052591 trunc from 0x015897 to 0x00ca2f (0x8e67 bytes) 000612 1787842716.124216 trunc from 0x00ca30 to 0x01c0c6 (0xf697 bytes) 000613 1787842716.172451 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1c0c7): Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c0c7): Mirror 14: 0xfb8618ea (stale) Mirror 15: 0xfb8618ea (stale) Mirror 16: 0xfb8618ea (stale) CRC-32 checksum value for chunk [0, 0x1c0c7): Mirror 1: 0x6e6f4205 Mirror 17: 0x6e6f4205 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c0c7: skipped 000614 1787842717.387669 mapwrite 0x0108e4 thru 0x01d5e3 (0xcd00 bytes) 000615 1787842717.546492 mapread 0x005606 thru 0x00cec4 (0x78bf bytes) 000616 1787842717.570363 trunc from 0x018839 to 0x01d5e3 (0x4dab bytes) 000617 1787842717.586446 write 0x009f60 thru 0x00ab4c (0xbed bytes) 000618 1787842717.588626 punch from 0x005caa to 0x00678f (0xae6 bytes) 000619 1787842717.598307 trunc from 0x016757 to 0x01cbd2 (0x647c bytes) 000621 1787842717.607332 mapread 0x001246 thru 0x00679e (0x5559 bytes) 000622 1787842717.614348 fallocate 0x03a110 thru 0x03ffff (0x5ef0 bytes) 000623 1787842717.620320 fallocate 0x03deeb thru 0x03e421 (0x537 bytes) 000624 1787842717.627408 mapwrite 0x0134c5 thru 0x01cb2a (0x9666 bytes) 000625 1787842717.661851 punch from 0x0170d0 to 0x017476 (0x3a7 bytes) 000627 1787842717.674558 MIRROR_RESYNC 5 mirrors 000628 1787842719.852417 write 0x02fd34 thru 0x031115 (0x13e2 bytes) 000630 1787842719.909072 write 0x00d3f7 thru 0x014c9f (0x78a9 bytes) 000631 1787842719.936119 write 0x00c0b2 thru 0x00d14c (0x109b bytes) 000632 1787842719.953049 read 0x02e49d thru 0x03c6f3 (0xe257 bytes) 000633 1787842719.982475 mapread 0x03037a thru 0x03d678 (0xd2ff bytes) 000634 1787842719.991326 trunc from 0x03bd5d to 0x03ffff (0x42a3 bytes) 000635 1787842720.004133 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xce831ff0 (stale) Mirror 16: 0xce831ff0 (stale) Mirror 17: 0xce831ff0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xce831ff0 (stale) Mirror 16: 0xce831ff0 (stale) Mirror 17: 0xce831ff0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000636 1787842720.466183 trunc from 0x000381 to 0x00d70e (0xd38e bytes) 000637 1787842720.473220 write 0x02a0c0 thru 0x02e7dd (0x471e bytes) 000638 1787842720.488735 punch from 0x003583 to 0x00d792 (0xa210 bytes) 000639 1787842720.494239 trunc from 0x025de6 to 0x0359b7 (0xfbd2 bytes) 000640 1787842720.511751 trunc from 0x03e722 to 0x03ffff (0x18de bytes) 000641 1787842720.530179 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xce831ff0 (stale) Mirror 16: 0xce831ff0 (stale) Mirror 17: 0xce831ff0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd9371811 Mirror 18: 0xd9371811 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000642 1787842721.654340 trunc from 0x0126fc to 0x021ac3 (0xf3c8 bytes) 000643 1787842721.706395 trunc from 0x040000 to 0x012858 (0x2d7a7 bytes) 000645 1787842721.754966 mapwrite 0x02243c thru 0x02d47c (0xb041 bytes) 000646 1787842721.859943 fallocate 0x01e5d3 thru 0x01fbd5 (0x1603 bytes) 000647 1787842721.865472 punch from 0x01b4e0 to 0x0248eb (0x940c bytes) 000648 1787842721.871524 trunc from 0x0095e2 to 0x0097e4 (0x203 bytes) 000649 1787842721.876666 fallocate 0x02d211 thru 0x0354ce (0x82be bytes) 000650 1787842721.881865 fallocate 0x00f503 thru 0x01b084 (0xbb82 bytes) 000651 1787842721.886884 trunc from 0x00f983 to 0x014767 (0x4de5 bytes) 000652 1787842721.891414 mapread 0x025885 thru 0x026534 (0xcb0 bytes) 000653 1787842721.892513 trunc from 0x02d47d to 0x01ce8a (0x105f2 bytes) 000654 1787842721.913013 trunc from 0x01ce8b to 0x02d013 (0x10189 bytes) 000655 1787842721.929418 trunc from 0x02d014 to 0x02ad95 (0x227e bytes) 000657 1787842721.946110 trunc from 0x001c36 to 0x00f468 (0xd833 bytes) 000658 1787842721.952165 punch from 0x015345 to 0x017eee (0x2baa bytes) 000659 1787842721.957130 read 0x015b9b thru 0x01f03e (0x94a4 bytes) 000660 1787842721.970691 MIRROR_RESYNC 5 mirrors 000661 1787842723.538183 MIRROR_EXTEND to 6 mirrors 000662 1787842724.529470 trunc from 0x02ad96 to 0x037ce4 (0xcf4f bytes) 000663 1787842724.622734 punch from 0x0124e3 to 0x012dac (0x8ca bytes) 000664 1787842724.627641 punch from 0x029240 to 0x02aec5 (0x1c86 bytes) 000666 1787842724.633333 write 0x031f9b thru 0x03e21d (0xc283 bytes) 000667 1787842724.654032 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e21e): Mirror 15: 0x8e371472 (stale) Mirror 17: 0x8e371472 (stale) Mirror 18: 0x8e371472 (stale) Mirror 19: 0x8e371472 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e21e): Mirror 15: 0x8e371472 (stale) Mirror 17: 0x8e371472 (stale) Mirror 18: 0x8e371472 (stale) Mirror 19: 0x8e371472 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e21e: skipped 000668 1787842725.233414 read 0x0375d8 thru 0x03e21d (0x6c46 bytes) 000669 1787842725.250175 punch from 0x03aec4 to 0x03e21d (0x335a bytes) 000670 1787842725.256677 punch from 0x039f19 to 0x03cbb0 (0x2c98 bytes) 000671 1787842725.267763 trunc from 0x03e21e to 0x014e34 (0x293e9 bytes) 000672 1787842725.290750 trunc from 0x004400 to 0x013bf2 (0xf7f3 bytes) 000673 1787842725.297372 trunc from 0x00b00e to 0x014e34 (0x9e27 bytes) 000675 1787842725.306031 write 0x02b8d7 thru 0x0353ee (0x9b18 bytes) 000676 1787842725.310328 fallocate 0x02da6c thru 0x02fd2d (0x22c2 bytes) 000677 1787842725.315929 write 0x00132e thru 0x00c3fc (0xb0cf bytes) 000678 1787842725.330764 write 0x02f2ba thru 0x038c89 (0x99d0 bytes) 000679 1787842725.334761 mapwrite 0x012898 thru 0x013f7e (0x16e7 bytes) 000680 1787842725.384855 write 0x02c697 thru 0x031d0b (0x5675 bytes) 000681 1787842725.393885 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38c8a): Mirror 15: 0xfbc66038 (stale) Mirror 17: 0xfbc66038 (stale) Mirror 19: 0xfbc66038 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c8a): Mirror 15: 0xfbc66038 (stale) Mirror 17: 0xfbc66038 (stale) Mirror 19: 0xfbc66038 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c8a: skipped 000682 1787842725.908248 trunc from 0x0322af to 0x038c89 (0x69db bytes) 000683 1787842725.916949 trunc from 0x038c8a to 0x038029 (0xc60 bytes) 000684 1787842725.960301 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3802a): Mirror 17: 0x801279c0 (stale) Mirror 19: 0x801279c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3802a): Mirror 17: 0x801279c0 (stale) Mirror 19: 0x801279c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3802a: skipped 000685 1787842726.373074 mapread 0x016da4 thru 0x023aea (0xcd47 bytes) 000686 1787842726.399856 write 0x02f79e thru 0x0301e6 (0xa49 bytes) 000687 1787842726.414002 trunc from 0x03802a to 0x03783c (0x7ed bytes) 000688 1787842726.427172 write 0x014e48 thru 0x01f751 (0xa90a bytes) 000689 1787842726.428632 trunc from 0x03783d to 0x01fe93 (0x179a9 bytes) 000690 1787842726.443624 trunc from 0x012b66 to 0x013d5d (0x11f8 bytes) 000691 1787842726.448832 read 0x01959b thru 0x01daa9 (0x450f bytes) 000692 1787842726.451826 mapwrite 0x0349a1 thru 0x0354a7 (0xb07 bytes) 000693 1787842726.506973 trunc from 0x0354a8 to 0x025fa0 (0xf507 bytes) 000694 1787842726.526691 read 0x005a29 thru 0x01076b (0xad43 bytes) 000695 1787842726.547005 trunc from 0x025fa1 to 0x021aac (0x44f4 bytes) 000696 1787842726.586128 trunc from 0x021aad to 0x016649 (0xb463 bytes) 000697 1787842726.608367 trunc from 0x01664a to 0x0318aa (0x1b261 bytes) 000699 1787842726.647103 fallocate 0x03ac80 thru 0x03be68 (0x11e9 bytes) 000700 1787842726.669763 fallocate 0x00dfcc thru 0x018550 (0xa585 bytes) 000701 1787842726.681778 punch from 0x015eeb to 0x022f55 (0xd06b bytes) 000702 1787842726.690546 read 0x024720 thru 0x029fa1 (0x5882 bytes) 000703 1787842726.714789 trunc from 0x00938b to 0x016e81 (0xdaf7 bytes) 000704 1787842726.723672 trunc from 0x01d2f0 to 0x024e5e (0x7b6f bytes) 000705 1787842726.730067 fallocate 0x02dff0 thru 0x03113f (0x3150 bytes) 000706 1787842726.737288 punch from 0x00f54e to 0x014115 (0x4bc8 bytes) 000707 1787842726.742863 fallocate 0x01b143 thru 0x0245c0 (0x947e bytes) 000708 1787842726.752778 punch from 0x002eb4 to 0x008ee0 (0x602d bytes) 000710 1787842726.770607 write 0x0187b0 thru 0x01c526 (0x3d77 bytes) 000711 1787842726.779055 read 0x0029fa thru 0x0094bc (0x6ac3 bytes) 000712 1787842726.805174 write 0x01b1b9 thru 0x01d873 (0x26bb bytes) 000713 1787842726.809031 write 0x00092b thru 0x002701 (0x1dd7 bytes) 000714 1787842726.810142 write 0x020ff3 thru 0x029217 (0x8225 bytes) 000717 1787842726.815196 punch from 0x030d26 to 0x0318aa (0xb85 bytes) 000718 1787842726.830341 trunc from 0x008f0a to 0x015626 (0xc71d bytes) 000719 1787842726.840864 fallocate 0x014fe5 thru 0x0164dc (0x14f8 bytes) 000720 1787842726.848499 trunc from 0x0318ab to 0x0267a3 (0xb107 bytes) 000721 1787842726.872887 trunc from 0x0267a4 to 0x023a78 (0x2d2b bytes) 000722 1787842726.898398 punch from 0x022068 to 0x02342a (0x13c3 bytes) 000723 1787842726.919775 read 0x018808 thru 0x023a78 (0xb271 bytes) 000728 1787842726.934792 write 0x01b933 thru 0x01d165 (0x1833 bytes) 000729 1787842726.936183 trunc from 0x01d5d9 to 0x023a78 (0x64a0 bytes) 000730 1787842726.951396 mapwrite 0x01f121 thru 0x02a017 (0xaef7 bytes) 000731 1787842727.032071 mapwrite 0x02f497 thru 0x0392ae (0x9e18 bytes) 000732 1787842727.072990 trunc from 0x017e5b to 0x019db7 (0x1f5d bytes) 000733 1787842727.079000 mapwrite 0x02dd9e thru 0x03703e (0x92a1 bytes) 000734 1787842727.099572 fallocate 0x032e4b thru 0x039a17 (0x6bcd bytes) 000735 1787842727.106745 mapwrite 0x017fe4 thru 0x020bef (0x8c0c bytes) 000736 1787842727.184619 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39a18): Mirror 17: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a18): Mirror 17: 0x1f1f13e4 (stale) Mirror 19: 0x1f1f13e4 (stale) CRC-32 checksum value for chunk [0, 0x39a18): Mirror 1: 0x369b39f9 Mirror 20: 0x369b39f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a18: skipped 000737 1787842728.527423 fallocate 0x01de84 thru 0x01f97e (0x1afb bytes) 000738 1787842728.566270 trunc from 0x039a18 to 0x01d630 (0x1c3e7 bytes) 000739 1787842728.624013 MIRROR_RESYNC 4 mirrors 000740 1787842730.083101 trunc from 0x01d631 to 0x033d36 (0x16706 bytes) 000741 1787842730.131944 trunc from 0x033d37 to 0x01e4d1 (0x15865 bytes) 000742 1787842730.173096 trunc from 0x01e4d2 to 0x0366fe (0x1822d bytes) 000743 1787842730.227370 punch from 0x027be2 to 0x0366fe (0xeb1d bytes) 000744 1787842730.240218 write 0x00e7b6 thru 0x015e3d (0x7688 bytes) 000745 1787842730.271535 write 0x01286d thru 0x020376 (0xdb0a bytes) 000746 1787842730.296322 punch from 0x033f52 to 0x0366fe (0x27ad bytes) 000747 1787842730.325029 mapread 0x009d26 thru 0x015b36 (0xbe11 bytes) 000748 1787842730.371753 fallocate 0x00fcee thru 0x01c6a9 (0xc9bc bytes) 000749 1787842730.389148 write 0x016fd5 thru 0x01dad3 (0x6aff bytes) 000750 1787842730.394131 write 0x031919 thru 0x0395a2 (0x7c8a bytes) 000751 1787842730.416333 trunc from 0x02d2f1 to 0x034ce5 (0x79f5 bytes) 000752 1787842730.452051 trunc from 0x013c2d to 0x01e7fe (0xabd2 bytes) 000753 1787842730.460234 trunc from 0x003979 to 0x00b639 (0x7cc1 bytes) 000754 1787842730.499781 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x395a3): Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x395a3): Mirror 17: 0xa1062c05 (stale) Mirror 19: 0xa1062c05 (stale) Mirror 20: 0xa1062c05 (stale) CRC-32 checksum value for chunk [0, 0x395a3): Mirror 1: 0xd45dcd1d Mirror 21: 0xd45dcd1d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x395a3: skipped 000755 1787842731.810484 write 0x0170e9 thru 0x01b243 (0x415b bytes) 000756 1787842731.916832 write 0x02535c thru 0x0350c6 (0xfd6b bytes) 000757 1787842731.951891 trunc from 0x0310fe to 0x0395a2 (0x84a5 bytes) 000758 1787842731.986576 fallocate 0x03a6d8 thru 0x03ffff (0x5928 bytes) 000759 1787842731.994942 trunc from 0x003c28 to 0x0082cc (0x46a5 bytes) 000760 1787842732.008397 punch from 0x02cc1c to 0x035305 (0x86ea bytes) 000761 1787842732.014290 write 0x036098 thru 0x03ffff (0x9f68 bytes) 000762 1787842732.024517 mapwrite 0x0233fc thru 0x02fdb4 (0xc9b9 bytes) 000763 1787842732.075803 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0x4faaadf4 (stale) Mirror 19: 0x4faaadf4 (stale) Mirror 20: 0x4faaadf4 (stale) Mirror 21: 0x8a15b528 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1464a5e6 Mirror 22: 0x1464a5e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000764 1787842733.185921 mapread 0x0101c6 thru 0x01f6d5 (0xf510 bytes) 000765 1787842733.229469 read 0x019021 thru 0x0195f4 (0x5d4 bytes) 000766 1787842733.230069 punch from 0x01605c to 0x01a3de (0x4383 bytes) 000767 1787842733.270082 mapwrite 0x011ee8 thru 0x01741e (0x5537 bytes) 000769 1787842733.296801 mapwrite 0x00f48b thru 0x01cec5 (0xda3b bytes) 000771 1787842733.329761 write 0x0051d4 thru 0x013c8f (0xeabc bytes) 000773 1787842733.342985 punch from 0x0246fd to 0x02fda4 (0xb6a8 bytes) 000774 1787842733.348890 read 0x02ecbc thru 0x0396b2 (0xa9f7 bytes) 000775 1787842733.362846 trunc from 0x040000 to 0x023bce (0x1c431 bytes) 000776 1787842733.407089 fallocate 0x0271b4 thru 0x02ef63 (0x7db0 bytes) 000777 1787842733.413037 mapread 0x023203 thru 0x023bce (0x9cc bytes) 000778 1787842733.414671 mapread 0x00ca7d thru 0x00e23a (0x17be bytes) 000779 1787842733.421538 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23bcf): Mirror 17: 0xafc2f577 (stale) Mirror 20: 0xafc2f577 (stale) Mirror 21: 0x6c68cc82 (stale) Mirror 22: 0xdb307395 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23bcf): Mirror 17: 0xafc2f577 (stale) Mirror 20: 0xafc2f577 (stale) Mirror 21: 0x6c68cc82 (stale) Mirror 22: 0xdb307395 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23bcf: skipped 000780 1787842733.938403 read 0x006b1d thru 0x00718e (0x672 bytes) 000781 1787842733.944763 read 0x00f3c8 thru 0x0108ed (0x1526 bytes) 000782 1787842733.971254 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x23bcf): Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23bcf): Mirror 17: 0xafc2f577 (stale) Mirror 20: 0xafc2f577 (stale) Mirror 21: 0x6c68cc82 (stale) Mirror 22: 0xdb307395 (stale) CRC-32 checksum value for chunk [0, 0x23bcf): Mirror 1: 0x137e3b79 Mirror 23: 0x137e3b79 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23bcf: skipped 000783 1787842735.401186 fallocate 0x02feee thru 0x03803a (0x814d bytes) 000784 1787842735.458504 MIRROR_RESYNC 6 mirrors 000785 1787842737.860781 write 0x03be16 thru 0x03ffff (0x41ea bytes) 000786 1787842737.911187 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xc75420db (stale) Mirror 20: 0xc75420db (stale) Mirror 21: 0xc75420db (stale) Mirror 23: 0xc75420db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xc75420db (stale) Mirror 20: 0xc75420db (stale) Mirror 21: 0xc75420db (stale) Mirror 23: 0xc75420db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000787 1787842738.473023 fallocate 0x0219d4 thru 0x0235bb (0x1be8 bytes) 000788 1787842738.482759 write 0x0118ab thru 0x0156df (0x3e35 bytes) 000789 1787842738.498423 read 0x01d84a thru 0x021f8e (0x4745 bytes) 000791 1787842738.509681 fallocate 0x010778 thru 0x013c32 (0x34bb bytes) 000792 1787842738.515538 read 0x024c56 thru 0x0316a7 (0xca52 bytes) 000793 1787842738.542812 write 0x010703 thru 0x013a53 (0x3351 bytes) 000794 1787842738.545352 mapwrite 0x02a68d thru 0x02be8d (0x1801 bytes) 000795 1787842738.602014 write 0x03f023 thru 0x03ffff (0xfdd bytes) 000796 1787842738.603157 write 0x01e934 thru 0x020a97 (0x2164 bytes) 000797 1787842738.622317 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xc75420db (stale) Mirror 20: 0xc75420db (stale) Mirror 21: 0xc75420db (stale) Mirror 23: 0xc75420db (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2293e370 Mirror 24: 0x2293e370 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000798 1787842740.274934 fallocate 0x015b89 thru 0x024c46 (0xf0be bytes) 000799 1787842740.370774 punch from 0x02bd7f to 0x02cd04 (0xf86 bytes) 000801 1787842740.386534 write 0x034510 thru 0x03ffff (0xbaf0 bytes) 000802 1787842740.410517 trunc from 0x040000 to 0x002e97 (0x3d168 bytes) 000803 1787842740.435789 read 0x001c7b thru 0x002e97 (0x121d bytes) 000806 1787842740.444102 punch from 0x0018de to 0x002e97 (0x15ba bytes) 000807 1787842740.449283 punch from 0x0001f9 to 0x001547 (0x134f bytes) 000808 1787842740.453791 trunc from 0x002e98 to 0x005215 (0x237e bytes) 000809 1787842740.471080 mapwrite 0x00677f thru 0x00a8b4 (0x4136 bytes) 000810 1787842740.524846 read 0x001b73 thru 0x00a8b4 (0x8d42 bytes) 000811 1787842740.525832 trunc from 0x00a8b5 to 0x01dc47 (0x13393 bytes) 000812 1787842740.543103 trunc from 0x01dc48 to 0x03fd9e (0x22157 bytes) 000813 1787842740.557047 punch from 0x03c6cd to 0x03fd9e (0x36d2 bytes) 000815 1787842740.563266 write 0x0090ed thru 0x018811 (0xf725 bytes) 000816 1787842740.573418 write 0x020130 thru 0x02844a (0x831b bytes) 000817 1787842740.585933 trunc from 0x00e1ea to 0x015c2a (0x7a41 bytes) 000818 1787842740.619771 read 0x0274ad thru 0x027b5c (0x6b0 bytes) 000819 1787842740.620813 write 0x011dcb thru 0x012fe5 (0x121b bytes) 000820 1787842740.633814 read 0x031dc9 thru 0x032209 (0x441 bytes) 000822 1787842740.639935 trunc from 0x03fd9f to 0x007109 (0x38c95 bytes) 000823 1787842740.665408 trunc from 0x00710a to 0x0011ee (0x5f1b bytes) 000824 1787842740.689530 fallocate 0x03f351 thru 0x03ffff (0xcaf bytes) 000825 1787842740.695471 mapread 0x0329a8 thru 0x035b81 (0x31da bytes) 000826 1787842740.741338 mapread 0x033d17 thru 0x03c9a3 (0x8c8d bytes) 000827 1787842740.742656 mapwrite 0x01d665 thru 0x02bae7 (0xe483 bytes) 000828 1787842740.805350 MIRROR_RESYNC 6 mirrors 000829 1787842742.480516 trunc from 0x040000 to 0x01b212 (0x24ded bytes) 000830 1787842742.544594 trunc from 0x013680 to 0x01b212 (0x7b93 bytes) 000833 1787842742.554489 mapwrite 0x000236 thru 0x0091ae (0x8f79 bytes) 000834 1787842742.611001 trunc from 0x0158cc to 0x01b212 (0x5947 bytes) 000835 1787842742.619501 trunc from 0x01b213 to 0x023701 (0x84ef bytes) 000836 1787842742.679103 punch from 0x0105d4 to 0x01cea4 (0xc8d1 bytes) 000837 1787842742.693704 fallocate 0x039aa2 thru 0x03ffff (0x655e bytes) 000838 1787842742.713854 mapwrite 0x0141e8 thru 0x0181e3 (0x3ffc bytes) 000839 1787842742.779122 write 0x031d56 thru 0x03ffff (0xe2aa bytes) 000840 1787842742.807650 mapread 0x007c7a thru 0x00d559 (0x58e0 bytes) 000841 1787842742.813342 write 0x030d39 thru 0x0390fd (0x83c5 bytes) 000842 1787842742.827332 trunc from 0x040000 to 0x00ad3d (0x352c2 bytes) 000843 1787842742.847336 fallocate 0x013199 thru 0x01ac71 (0x7ad9 bytes) 000844 1787842742.852998 read 0x0072aa thru 0x0091f6 (0x1f4d bytes) 000845 1787842742.854097 read 0x0063f3 thru 0x00fa86 (0x9694 bytes) 000846 1787842742.875790 MIRROR_RESYNC 6 mirrors 000848 1787842744.352395 mapread 0x00c45b thru 0x010078 (0x3c1e bytes) 000849 1787842744.384037 mapwrite 0x035f6d thru 0x03ffff (0xa093 bytes) 000850 1787842744.514854 write 0x02b63f thru 0x032151 (0x6b13 bytes) 000852 1787842744.526349 write 0x0135f7 thru 0x01c789 (0x9193 bytes) 000853 1787842744.539525 mapwrite 0x015334 thru 0x023230 (0xdefd bytes) 000854 1787842744.644888 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xa0ea06f0 (stale) Mirror 20: 0xa0ea06f0 (stale) Mirror 21: 0xa0ea06f0 (stale) Mirror 23: 0xa0ea06f0 (stale) Mirror 24: 0xa0ea06f0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xbf8c2acf Mirror 25: 0xbf8c2acf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000855 1787842745.963440 read 0x02d2c4 thru 0x02f9a5 (0x26e2 bytes) 000856 1787842745.980673 mapread 0x01fac4 thru 0x021fac (0x24e9 bytes) 000857 1787842745.999809 read 0x004b17 thru 0x00e259 (0x9743 bytes) 000858 1787842746.009441 punch from 0x0355e1 to 0x03f628 (0xa048 bytes) 000859 1787842746.067623 punch from 0x022834 to 0x025f08 (0x36d5 bytes) 000860 1787842746.072284 punch from 0x034612 to 0x03ffff (0xb9ee bytes) 000861 1787842746.077646 fallocate 0x00ef28 thru 0x01e93d (0xfa16 bytes) 000862 1787842746.082241 trunc from 0x00839b to 0x0131e4 (0xae4a bytes) 000863 1787842746.088190 read 0x01837e thru 0x01e4f6 (0x6179 bytes) 000864 1787842746.118262 write 0x02bbae thru 0x02e8cb (0x2d1e bytes) 000865 1787842746.123672 punch from 0x03c4b3 to 0x03ffff (0x3b4d bytes) 000867 1787842746.129699 mapwrite 0x0235e3 thru 0x0320c3 (0xeae1 bytes) 000868 1787842746.151162 fallocate 0x01f8a1 thru 0x021835 (0x1f95 bytes) 000869 1787842746.159731 punch from 0x0004ee to 0x0062ac (0x5dbf bytes) 000870 1787842746.166829 read 0x020360 thru 0x028af9 (0x879a bytes) 000871 1787842746.167704 write 0x006e1d thru 0x00ee75 (0x8059 bytes) 000872 1787842746.177384 write 0x02b7bb thru 0x03428c (0x8ad2 bytes) 000873 1787842746.178575 fallocate 0x024fe2 thru 0x033cfb (0xed1a bytes) 000875 1787842746.187679 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xa0ea06f0 (stale) Mirror 20: 0xa0ea06f0 (stale) Mirror 21: 0xa0ea06f0 (stale) Mirror 23: 0xa0ea06f0 (stale) Mirror 24: 0xa0ea06f0 (stale) Mirror 25: 0xbf8c2acf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x70540714 Mirror 26: 0x70540714 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000876 1787842747.396196 mapwrite 0x028af9 thru 0x0358d6 (0xcdde bytes) 000877 1787842747.549693 punch from 0x03b245 to 0x03ffff (0x4dbb bytes) 000878 1787842747.557877 mapread 0x03f0b4 thru 0x03ffff (0xf4c bytes) 000879 1787842747.569350 mapread 0x0158ee thru 0x01a769 (0x4e7c bytes) 000880 1787842747.589294 punch from 0x01ea15 to 0x02a82f (0xbe1b bytes) 000881 1787842747.601500 mapread 0x01f283 thru 0x026ac5 (0x7843 bytes) 000882 1787842747.613373 read 0x027219 thru 0x02e3dc (0x71c4 bytes) 000883 1787842747.619706 mapwrite 0x012553 thru 0x018e66 (0x6914 bytes) 000884 1787842747.646030 trunc from 0x03ea59 to 0x03ffff (0x15a7 bytes) 000886 1787842747.653186 mapread 0x037a55 thru 0x03ffff (0x85ab bytes) 000887 1787842747.665559 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xa0ea06f0 (stale) Mirror 20: 0xa0ea06f0 (stale) Mirror 21: 0xa0ea06f0 (stale) Mirror 23: 0xa0ea06f0 (stale) Mirror 24: 0xa0ea06f0 (stale) Mirror 25: 0xbf8c2acf (stale) Mirror 26: 0x70540714 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x57eb485e Mirror 27: 0x57eb485e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000888 1787842748.714520 fallocate 0x014e56 thru 0x01802d (0x31d8 bytes) 000889 1787842748.782585 trunc from 0x040000 to 0x01b477 (0x24b88 bytes) 000891 1787842748.855340 punch from 0x015b69 to 0x01b477 (0x590f bytes) 000892 1787842748.870080 mapwrite 0x03876e thru 0x03ffff (0x7892 bytes) 000894 1787842748.993170 fallocate 0x00f8a9 thru 0x017d81 (0x84d9 bytes) 000896 1787842749.005969 fallocate 0x0085a7 thru 0x0147c3 (0xc21d bytes) 000897 1787842749.013565 mapread 0x031e88 thru 0x033afa (0x1c73 bytes) 000899 1787842749.030944 fallocate 0x025210 thru 0x032069 (0xce5a bytes) 000900 1787842749.039026 write 0x03bcc4 thru 0x03ffff (0x433c bytes) 000901 1787842749.050891 MIRROR_RESYNC 9 mirrors 000902 1787842751.587883 mapwrite 0x00e682 thru 0x018e19 (0xa798 bytes) 000903 1787842751.705448 trunc from 0x040000 to 0x036df8 (0x9207 bytes) 000904 1787842751.725486 mapread 0x03155e thru 0x032fb7 (0x1a5a bytes) 000906 1787842751.736247 write 0x0087ee thru 0x015757 (0xcf6a bytes) 000907 1787842751.738401 write 0x03d25d thru 0x03f828 (0x25cc bytes) 000909 1787842751.740829 fallocate 0x01b307 thru 0x01fbee (0x48e8 bytes) 000910 1787842751.746457 trunc from 0x02f0b9 to 0x031a5a (0x29a2 bytes) 000911 1787842751.753436 write 0x02e231 thru 0x0331fe (0x4fce bytes) 000912 1787842751.762473 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f829): Mirror 17: 0x39806b09 (stale) Mirror 20: 0x39806b09 (stale) Mirror 21: 0x39806b09 (stale) Mirror 23: 0x39806b09 (stale) Mirror 24: 0x39806b09 (stale) Mirror 26: 0x39806b09 (stale) Mirror 27: 0x39806b09 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f829): Mirror 17: 0x39806b09 (stale) Mirror 20: 0x39806b09 (stale) Mirror 21: 0x39806b09 (stale) Mirror 23: 0x39806b09 (stale) Mirror 24: 0x39806b09 (stale) Mirror 26: 0x39806b09 (stale) Mirror 27: 0x39806b09 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f829: skipped 000915 1787842752.570712 trunc from 0x039a48 to 0x03f828 (0x5de1 bytes) 000916 1787842752.584050 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3f829): Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f829): Mirror 17: 0x39806b09 (stale) Mirror 20: 0x39806b09 (stale) Mirror 21: 0x39806b09 (stale) Mirror 23: 0x39806b09 (stale) Mirror 24: 0x39806b09 (stale) Mirror 26: 0x39806b09 (stale) Mirror 27: 0x39806b09 (stale) CRC-32 checksum value for chunk [0, 0x3f829): Mirror 1: 0xd60392ba Mirror 28: 0xd60392ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f829: skipped 000917 1787842754.084278 trunc from 0x00ded0 to 0x00f3e6 (0x1517 bytes) 000918 1787842754.185718 mapread 0x030a45 thru 0x03f1f8 (0xe7b4 bytes) 000919 1787842754.259099 mapwrite 0x0207e9 thru 0x02a652 (0x9e6a bytes) 000920 1787842754.298113 write 0x02b210 thru 0x02e32f (0x3120 bytes) 000921 1787842754.304113 write 0x02b377 thru 0x02d911 (0x259b bytes) 000922 1787842754.305053 fallocate 0x02216c thru 0x0261ad (0x4042 bytes) 000923 1787842754.312816 fallocate 0x014727 thru 0x024663 (0xff3d bytes) 000924 1787842754.319647 trunc from 0x03f829 to 0x02f10b (0x1071d bytes) 000925 1787842754.348668 MIRROR_RESYNC 9 mirrors 000926 1787842757.418620 write 0x03dcec thru 0x03ffff (0x2314 bytes) 000927 1787842757.471743 read 0x01c1a2 thru 0x023f10 (0x7d6f bytes) 000928 1787842757.481410 punch from 0x013b3a to 0x01938b (0x5852 bytes) 000929 1787842757.488188 punch from 0x00a707 to 0x0192eb (0xebe5 bytes) 000930 1787842757.502153 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xf620a44b (stale) Mirror 20: 0xf620a44b (stale) Mirror 21: 0xf620a44b (stale) Mirror 23: 0xf620a44b (stale) Mirror 24: 0xf620a44b (stale) Mirror 26: 0xf620a44b (stale) Mirror 28: 0xf620a44b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xf620a44b (stale) Mirror 20: 0xf620a44b (stale) Mirror 21: 0xf620a44b (stale) Mirror 23: 0xf620a44b (stale) Mirror 24: 0xf620a44b (stale) Mirror 26: 0xf620a44b (stale) Mirror 28: 0xf620a44b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000931 1787842758.243006 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xf620a44b (stale) Mirror 21: 0xf620a44b (stale) Mirror 23: 0xf620a44b (stale) Mirror 24: 0xf620a44b (stale) Mirror 26: 0xf620a44b (stale) Mirror 28: 0xf620a44b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xf620a44b (stale) Mirror 21: 0xf620a44b (stale) Mirror 23: 0xf620a44b (stale) Mirror 24: 0xf620a44b (stale) Mirror 26: 0xf620a44b (stale) Mirror 28: 0xf620a44b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000932 1787842759.050376 trunc from 0x040000 to 0x01a162 (0x25e9d bytes) 000933 1787842759.080602 fallocate 0x00ea0c thru 0x0162de (0x78d3 bytes) 000934 1787842759.091372 mapread 0x00002f thru 0x00e1b0 (0xe182 bytes) 000936 1787842759.157716 mapread 0x00c4d1 thru 0x012f32 (0x6a62 bytes) 000937 1787842759.223982 trunc from 0x014356 to 0x01908e (0x4d39 bytes) 000938 1787842759.253053 MIRROR_RESYNC 7 mirrors 000939 1787842761.235417 mapwrite 0x0139cf thru 0x0183df (0x4a11 bytes) 000940 1787842761.335141 trunc from 0x015c4d to 0x0170ec (0x14a0 bytes) 000941 1787842761.342166 mapwrite 0x00ade3 thru 0x010e86 (0x60a4 bytes) 000942 1787842761.370680 mapwrite 0x02fd17 thru 0x03bd8d (0xc077 bytes) 000943 1787842761.436124 read 0x017fbb thru 0x026c35 (0xec7b bytes) 000944 1787842761.449275 read 0x01e2a3 thru 0x02420c (0x5f6a bytes) 000945 1787842761.450344 trunc from 0x01c8e1 to 0x01f0cf (0x27ef bytes) 000946 1787842761.461221 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x167:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bd8e): Mirror 21: 0x6d5f289a (stale) Mirror 23: 0x6d5f289a (stale) Mirror 24: 0x6d5f289a (stale) Mirror 26: 0x6d5f289a (stale) Mirror 28: 0x6d5f289a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd8e): Mirror 21: 0x6d5f289a (stale) Mirror 23: 0x6d5f289a (stale) Mirror 24: 0x6d5f289a (stale) Mirror 26: 0x6d5f289a (stale) Mirror 28: 0x6d5f289a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd8e: skipped 000947 1787842761.935027 mapread 0x01cf71 thru 0x0201d4 (0x3264 bytes) 000948 1787842761.972194 punch from 0x0168a2 to 0x0258b2 (0xf011 bytes) 000949 1787842761.979823 fallocate 0x00cdb6 thru 0x01a747 (0xd992 bytes) 000950 1787842761.992376 trunc from 0x01044f to 0x018262 (0x7e14 bytes) 000951 1787842762.002535 punch from 0x00d6c3 to 0x015ac8 (0x8406 bytes) 000952 1787842762.013734 write 0x032170 thru 0x038098 (0x5f29 bytes) 000953 1787842762.053583 punch from 0x027f50 to 0x0348c7 (0xc978 bytes) 000954 1787842762.091402 fallocate 0x0186cb thru 0x01b893 (0x31c9 bytes) 000955 1787842762.104458 trunc from 0x03bd8e to 0x0162a5 (0x25ae8 bytes) 000956 1787842762.167025 fallocate 0x03f1af thru 0x03ffff (0xe51 bytes) 000957 1787842762.190423 trunc from 0x002ea6 to 0x004455 (0x15b0 bytes) 000958 1787842762.210961 write 0x01269d thru 0x01f91d (0xd281 bytes) 000959 1787842762.234011 trunc from 0x01f91e to 0x019022 (0x68fb bytes) 000960 1787842762.280864 read 0x00839e thru 0x014d1b (0xc97e bytes) 000961 1787842762.301516 mapread 0x00d3cd thru 0x019022 (0xbc56 bytes) 000962 1787842762.307893 mapwrite 0x036fce thru 0x03ffff (0x9032 bytes) 000963 1787842762.440453 mapwrite 0x02dc44 thru 0x036ba2 (0x8f5f bytes) 000964 1787842762.475658 write 0x038623 thru 0x03ffff (0x79dd bytes) 000965 1787842762.488389 MIRROR_RESYNC 6 mirrors 000966 1787842764.829793 fallocate 0x03fd0d thru 0x03ffff (0x2f3 bytes) 000967 1787842764.870655 mapread 0x023256 thru 0x024f02 (0x1cad bytes) 000968 1787842764.881283 trunc from 0x040000 to 0x00b7ad (0x34852 bytes) 000969 1787842764.906787 trunc from 0x0001dc to 0x00b7ad (0xb5d2 bytes) 000970 1787842764.913762 fallocate 0x03843b thru 0x03ffff (0x7bc5 bytes) 000971 1787842764.918582 write 0x022578 thru 0x029310 (0x6d99 bytes) 000973 1787842764.920295 punch from 0x024320 to 0x029310 (0x4ff1 bytes) 000974 1787842764.937885 punch from 0x017576 to 0x01ac99 (0x3724 bytes) 000975 1787842764.942756 trunc from 0x029311 to 0x00b84a (0x1dac6 bytes) 000976 1787842764.963332 read 0x0040a2 thru 0x00877b (0x46da bytes) 000978 1787842764.974676 fallocate 0x00aba7 thru 0x019613 (0xea6d bytes) 000979 1787842764.981413 mapread 0x00e959 thru 0x019613 (0xacbb bytes) 000980 1787842764.995775 trunc from 0x019614 to 0x011c20 (0x79f3 bytes) 000981 1787842765.015638 fallocate 0x00ec96 thru 0x014e2c (0x6197 bytes) 000982 1787842765.021864 mapread 0x00e8ca thru 0x011c20 (0x3357 bytes) 000983 1787842765.023165 write 0x02c130 thru 0x03bafd (0xf9ce bytes) 000986 1787842765.030342 write 0x0127c6 thru 0x0217d0 (0xf00b bytes) 000987 1787842765.044656 MIRROR_RESYNC 6 mirrors 000990 1787842767.012022 mapwrite 0x02aee8 thru 0x031617 (0x6730 bytes) 000991 1787842767.152780 write 0x001ac6 thru 0x01001a (0xe555 bytes) 000993 1787842767.169321 trunc from 0x03bafe to 0x0281d5 (0x13928 bytes) 000994 1787842767.213211 mapwrite 0x007d9f thru 0x016c89 (0xeeeb bytes) 000995 1787842767.256186 write 0x02444d thru 0x031b49 (0xd6fd bytes) 000996 1787842767.261153 read 0x0159c5 thru 0x01d516 (0x7b52 bytes) 000997 1787842767.269368 read 0x0149eb thru 0x014e88 (0x49e bytes) 000998 1787842767.270331 trunc from 0x031b4a to 0x00e4de (0x2366b bytes) 001000 1787842767.300645 mapwrite 0x0015c8 thru 0x0047fa (0x3233 bytes) All operations completed A-OK! PASS 70a (149s) == sanity-flr test 71: check mirror extend parent fid ==== 10:59:33 (1787842773) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1a7:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 10:59:44 (1787842784) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.25054 s, 8.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129273 s, 8.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 10:59:54 (1787842794) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7506368 .. Extending file size to 4611296 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7506176 .. Extending file size to 8729824 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8047744 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8652160 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8986848 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62621 62622 62625 62626 62628 done 192.168.204.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre2 (opts:) 192.168.204.147@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 11:01:15 (1787842875) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2463520 .. Extending file size to 1720512 .. done Extending file size to 6631520 .. Extending file size to 8288288 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7947520 .. Extending file size to 8423488 .. Extending file size to 8804320 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8853952 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8875648 .. Extending file size to 9097472 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9034272 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 64299 64300 64302 64303 64305 192.168.204.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre2 (opts:) 192.168.204.147@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 11:02:32 (1787842952) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2324 1268980 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24548 3582472 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18732 3588288 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43280 7170760 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2324 1268980 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24548 3582472 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18732 3588288 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43280 7170760 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138465 s, 15.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.20074 s, 10.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xfcf9dfa8 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.204.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 11:02:54 (1787842974) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.162664 s, 12.9 MB/s PASS 202 (11s) == sanity-flr test 203: mirror file preserve mirror ID === 11:03:05 (1787842985) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } PASS 203 (11s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 11:03:16 (1787842996) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.1145 s, 18.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x204:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x204:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x205:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 11:03:25 (1787843005) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.158141 s, 13.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 11:03:34 (1787843014) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.133494 s, 15.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x205:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x206:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x205:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x206:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x207:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 11:04:08 (1787843048) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg447-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10821 s, 9.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x206:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x209:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x208:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x206:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x209:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x208:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x207:0x0] } PASS 204d (51s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 11:04:59 (1787843099) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg447-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x208:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x209:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102561 s, 20.4 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x208:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x209:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x208:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x209:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 11:05:47 (1787843147) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg447-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg447-server: Pool lustre.test_204f created Adding targets to pool oleg447-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg447-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129746 s, 16.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x20a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x20b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg447-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg447-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg447-server: Pool lustre.test_204f destroyed PASS 204f (88s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 11:07:14 (1787843234) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 11:07:23 (1787843243) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 11:07:33 (1787843253) oleg447-server: Pool lustre.test_206 created Adding targets to pool oleg447-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg447-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg447-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg447-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg447-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 11:08:03 (1787843283) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766153 s, 13.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565046 s, 18.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd2508903 Mirror 3: 0xd2508903 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 11:08:14 (1787843294) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 11:08:19 (1787843299) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 11:08:24 (1787843304) Starting client: oleg447-client.virtnet: -o user_xattr,flock 192.168.204.147@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0395868 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xc80000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xcf0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xcf0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xcf0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xce0000 Read this (4 bytes): BBBB 192.168.204.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg447-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 11:09:04 (1787843344) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0449418 s, 182 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 11:09:13 (1787843353) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0653701 s, 16.0 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7a lmm_fid: [0x200000405:0x7a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7a lmm_fid: [0x200000405:0x7a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x140:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611258 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 11:09:14.000000000 -0400 Modify: 2026-08-27 11:09:14.000000000 -0400 Change: 2026-08-27 11:09:16.000000000 -0400 Birth: 2026-08-27 11:09:16.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 11:09:22 (1787843362) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811959 s, 12.9 MB/s oleg447-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg447-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg447-server: O/280000401/d27/13b: File not found by ext2_lookup PASS 210b (28s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 11:09:50 (1787843390) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.318207 s, 129 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0242535 s, 169 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 11:09 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x7f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 11:09 /mnt/lustre/f211.sanity-flr PASS 211 (11s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 11:10:01 (1787843401) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 11:10:11 (1787843411) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1987 sec =========== 11:10:22 (1787843422) === sanity-flr: start cleanup 11:10:23 (1787843423) === === sanity-flr: finish cleanup 11:10:33 (1787843433) ===