-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 07:48:45 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg212-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 07:49:07 (1787831347) === oleg212-client.virtnet: executing check_config_client /mnt/lustre oleg212-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg212-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9c05d0644800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9c05d0644800.idle_timeout=debug disable quota as required oleg212-server: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 07:49:49 (1787831389) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8241] == sanity-flr test 0a: lfs mirror create with -N option == 07:49:51 (1787831391) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 07:50:06 (1787831406) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 07:50:09 (1787831409) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 07:50:12 (1787831412) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 07:50:34 (1787831434) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 07:50:37 (1787831437) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 07:50:41 (1787831441) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9c05d0644800.stats=clear osc.lustre-OST0001-osc-ffff9c05d0644800.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 07:50:48 (1787831448) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 07:50:59 (1787831459) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 07:51:06 (1787831466) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 07:51:10 (1787831470) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 07:51:20 (1787831480) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 07:51:29 (1787831489) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 07:51:36 (1787831496) PASS 3 (6s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 07:51:43 (1787831503) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 07:51:51 (1787831511) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 07:52:03 (1787831523) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 07:52:09 (1787831529) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.097388 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646833 s, 16.2 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 07:52:18 (1787831538) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.464752 s, 27.1 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 07:52:29 (1787831549) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 07:52:40 (1787831560) Stopping /mnt/lustre-ost1 (opts:) on oleg212-server oleg212-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c05d0644800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c05d0644800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg212-server: oleg212-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg212-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg212-server oleg212-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c05d0644800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c05d0644800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg212-server: oleg212-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg212-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (73s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 07:53:53 (1787831633) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.396019 s, 18.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg212-server oleg212-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c05d0644800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c05d0644800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg212-server: oleg212-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg212-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 07:54:37 (1787831677) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg212-server oleg212-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c05d0644800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c05d0644800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg212-server: oleg212-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg212-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg212-server oleg212-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c05d0644800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c05d0644800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg212-server: oleg212-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg212-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (86s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 07:56:02 (1787831762) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.91681 s, 18.0 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 07:56 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.961s user 0m0.000s sys 0m0.156s Stopping /mnt/lustre-ost1 (opts:) on oleg212-server oleg212-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9c05d0644800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9c05d0644800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.866s user 0m0.000s sys 0m0.175s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg212-server: oleg212-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg212-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg212-server oleg212-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9c05d0644800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c05d0644800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.085s user 0m0.000s sys 0m0.174s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg212-server: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg212-server: oleg212-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg212-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg212-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9c05d0644800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9c05d0644800.ost_server_uuid in FULL state after 0 sec PASS 33b (99s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 07:57:42 (1787831862) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 07:57:45 (1787831865) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 07:57:49 (1787831869) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 07:57:52 (1787831872) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 07:57:59 (1787831879) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.423023 s, 29.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 07:58:11 (1787831891) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301693 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313852 s, 33.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 07:58:19 (1787831899) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 07:58:27 (1787831907) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00162864 s, 629 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00122149 s, 838 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00108578 s, 943 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00125463 s, 816 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 07:58:37 (1787831917) create test files with size 15747001 .. 360e144674f9ccce42ec24f601fc01f5 - d57d84ecd73d75f9db4fed53781be8a3 - 0b77d4db219d1cae2f2e8b7249463f89 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg212-client.virtnet /mnt/lustre (opts:) Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 07:59:00 (1787831940) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 07:59:15 (1787831955) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 07:59:22 (1787831962) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585433 s, 17.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xca:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659604 s, 15.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb3:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb4:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 07:59:31 (1787831971) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0573879 s, 36.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.265258 s, 15.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105912 s, 19.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 07:59:44 (1787831984) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 07:59:47 (1787831987) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 07:59:50 (1787831990) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0274669 s, 38.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 43c: read prefer uncompressed mirror == 07:59:56 (1787831996) llite.lustre-ffff9c05e09ef000.enable_compression=1 striped dir -i1 -c2 -H crush2 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (7s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 08:00:03 (1787832003) llite.lustre-ffff9c05e09ef000.enable_compression=1 striped dir -i1 -c2 -H crush2 /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422249 s, 24.8 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0313803 s, 33.4 MB/s **verify components PASS 43d (8s) == sanity-flr test 44a: lfs mirror split check =========== 08:00:11 (1787832011) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.305936 s, 10.3 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:00:22 (1787832022) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:00:31 (1787832031) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.237006 s, 44.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x9:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg212-client.virtnet /mnt/lustre (opts:) Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:00:42 (1787832042) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9c05e10e4000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.406257 s, 25.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg212-client.virtnet /mnt/lustre (opts:) Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9c05d00b0800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:00:58 (1787832058) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:01:38 (1787832098) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:02:16 (1787832136) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 08:02:21 (1787832141) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.303615 s, 10.4 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.140413 s, 22.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x6748368b (nosync,stale) Mirror 2: 0x6748368b (nosync,stale) ** verify mirror 2 doesn't change original checksum: 03be0b01e4741d3739977e3cb4e2d3d5 - mirror 1 checksum: e127fa794a72ea58ce03615df3dc607f - mirror 2 checksum: 03be0b01e4741d3739977e3cb4e2d3d5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T08:02:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T08:02:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 03be0b01e4741d3739977e3cb4e2d3d5 - mirror 1 checksum: e127fa794a72ea58ce03615df3dc607f - mirror 2 checksum: e127fa794a72ea58ce03615df3dc607f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T08:02:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T08:02:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcd:0x0] } PASS 48 (13s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 08:02:33 (1787832153) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284634 s, 36.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:02:40 (1787832160) Failing mds1 on oleg212-server Stopping /mnt/lustre-mds1 (opts:) on oleg212-server 08:02:51 (1787832171) shut down facet: mds1 facet_host: oleg212-server facet_failover_host: oleg212-server Failover mds1 to oleg212-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg212-server: oleg212-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Started lustre-MDT0000 08:03:14 (1787832194) targets are mounted 08:03:14 (1787832194) facet_failover done oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041414 s, 25.3 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } PASS 50A (54s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:03:34 (1787832214) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0275179 s, 744 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0210496 s, 973 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:03:45 (1787832225) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0690873 s, 296 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0117072 s, 1.7 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:03:55 (1787832235) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0128601 s, 1.3 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:04:04 (1787832244) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0543002 s, 377 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:04:13 (1787832253) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00663619 s, 617 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0358896 s, 114 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:04:21 (1787832261) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:04:40 (1787832280) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:05:10 (1787832310) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:05:33 (1787832333) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:05:59 (1787832359) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 08:06:07 (1787832367) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 58865 mirror split pid 58866 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58865 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58866 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 08:07:15 (1787832435) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 2438 skipping zero size read skipping zero length punch hole skipping zero size read fallocating to largest ever: 0x1e860 000005 1787832438.480072 fallocate 0x018d6a thru 0x01e85f (0x5af6 bytes) 000006 1787832438.489913 punch from 0x019dac to 0x01e85f (0x4ab4 bytes) 000007 1787832438.496090 write 0x02305b thru 0x0252ca (0x2270 bytes) 000008 1787832438.497711 mapwrite 0x008871 thru 0x00fb91 (0x7321 bytes) 000009 1787832438.599618 punch from 0x00eca8 to 0x018c33 (0x9f8c bytes) 000010 1787832438.606823 trunc from 0x006d88 to 0x00ab4d (0x3dc6 bytes) 000012 1787832438.616089 read 0x0017b9 thru 0x006085 (0x48cd bytes) 000013 1787832438.626977 trunc from 0x0252cb to 0x004cde (0x205ec bytes) 000014 1787832438.643118 read 0x00437f thru 0x004cde (0x960 bytes) 000015 1787832438.645112 punch from 0x00224f to 0x004cde (0x2a90 bytes) 000016 1787832438.653556 write 0x017834 thru 0x01c85f (0x502c bytes) 000018 1787832438.655065 mapread 0x009ea4 thru 0x00f7a9 (0x5906 bytes) fallocating to largest ever: 0x40000 000019 1787832438.665089 fallocate 0x0392bd thru 0x03ffff (0x6d43 bytes) 000020 1787832438.670666 mapread 0x001a9f thru 0x007f69 (0x64cb bytes) 000021 1787832438.672096 write 0x02dbb7 thru 0x02df3e (0x388 bytes) 000023 1787832438.675883 trunc from 0x009944 to 0x0165f5 (0xccb2 bytes) 000024 1787832438.681821 mapread 0x018716 thru 0x019ac6 (0x13b1 bytes) 000025 1787832438.683674 trunc from 0x0237b9 to 0x02df3e (0xa786 bytes) 000026 1787832438.699387 read 0x00ffd2 thru 0x017b81 (0x7bb0 bytes) 000027 1787832438.716020 mapwrite 0x03f630 thru 0x03ffff (0x9d0 bytes) 000029 1787832438.757640 punch from 0x033c14 to 0x0364b7 (0x28a4 bytes) 000030 1787832438.761975 mapread 0x0232de thru 0x02a61b (0x733e bytes) 000031 1787832438.763654 trunc from 0x03a9df to 0x03e8a5 (0x3ec7 bytes) 000032 1787832438.768975 read 0x0029ed thru 0x005640 (0x2c54 bytes) 000033 1787832438.769887 read 0x02bd0f thru 0x02be9b (0x18d bytes) 000034 1787832438.774755 fallocate 0x01192c thru 0x013ae1 (0x21b6 bytes) 000036 1787832438.781113 read 0x030d8f thru 0x039e1d (0x908f bytes) 000037 1787832438.793069 fallocate 0x00d4f1 thru 0x00f762 (0x2272 bytes) 000038 1787832438.797871 punch from 0x007f6c to 0x00ce06 (0x4e9b bytes) 000040 1787832438.804023 trunc from 0x040000 to 0x00bd81 (0x3427e bytes) 000041 1787832438.821590 mapwrite 0x021495 thru 0x0309e7 (0xf553 bytes) 000043 1787832438.943534 fallocate 0x01287e thru 0x015d4b (0x34ce bytes) 000044 1787832438.947620 fallocate 0x028e20 thru 0x02a40a (0x15eb bytes) 000045 1787832438.952337 write 0x00f2ad thru 0x013944 (0x4698 bytes) 000046 1787832438.962991 punch from 0x010dcd to 0x0177f3 (0x6a27 bytes) 000047 1787832438.978005 mapwrite 0x01acb4 thru 0x022578 (0x78c5 bytes) 000048 1787832439.062070 trunc from 0x008e81 to 0x012bed (0x9d6d bytes) 000049 1787832439.067438 fallocate 0x038d0e thru 0x03ffff (0x72f2 bytes) 000050 1787832439.072072 mapread 0x0180c9 thru 0x022509 (0xa441 bytes) 000051 1787832439.082913 mapwrite 0x001276 thru 0x0076d3 (0x645e bytes) 000052 1787832439.104824 trunc from 0x040000 to 0x020d75 (0x1f28a bytes) 000053 1787832439.127655 write 0x005649 thru 0x00c439 (0x6df1 bytes) 000054 1787832439.134948 write 0x008d86 thru 0x017099 (0xe314 bytes) 000057 1787832439.144615 read 0x0105ee thru 0x01ab58 (0xa56b bytes) 000058 1787832439.145712 fallocate 0x0018c0 thru 0x0021c0 (0x901 bytes) 000059 1787832439.150699 mapread 0x01a9f4 thru 0x020d75 (0x6382 bytes) 000060 1787832439.152005 read 0x005f91 thru 0x00b611 (0x5681 bytes) 000061 1787832439.152969 fallocate 0x01b082 thru 0x020a39 (0x59b8 bytes) 000062 1787832439.157873 trunc from 0x01631a to 0x01769d (0x1384 bytes) 000063 1787832439.172654 read 0x00a387 thru 0x00ebbb (0x4835 bytes) 000064 1787832439.173398 mapread 0x00e697 thru 0x010b33 (0x249d bytes) 000066 1787832439.176438 write 0x00943c thru 0x0136af (0xa274 bytes) 000067 1787832439.177877 trunc from 0x020d76 to 0x023872 (0x2afd bytes) 000068 1787832439.194885 mapwrite 0x00b99a thru 0x00eb24 (0x318b bytes) 000069 1787832439.225990 write 0x03133e thru 0x036cad (0x5970 bytes) 000070 1787832439.227447 trunc from 0x036cae to 0x029c6d (0xd040 bytes) 000071 1787832439.243492 trunc from 0x029c6e to 0x02a2c2 (0x655 bytes) 000072 1787832439.260395 fallocate 0x01806c thru 0x01ca9a (0x4a2f bytes) 000073 1787832439.265555 punch from 0x01e72b to 0x02a2c2 (0xbb98 bytes) 000076 1787832439.272818 read 0x01e8d1 thru 0x026f05 (0x8635 bytes) 000077 1787832439.283929 write 0x031cc9 thru 0x03f5ab (0xd8e3 bytes) 000078 1787832439.286692 trunc from 0x0087a4 to 0x00a8e7 (0x2144 bytes) 000079 1787832439.291729 trunc from 0x03f5ac to 0x037364 (0x8247 bytes) 000081 1787832439.308722 MIRROR_EXTEND to 2 mirrors 000082 1787832440.146400 MIRROR_RESYNC 2 mirrors 000083 1787832440.477869 trunc from 0x028d0c to 0x03061b (0x7910 bytes) 000084 1787832440.517697 trunc from 0x037085 to 0x037364 (0x2e0 bytes) 000085 1787832440.523757 trunc from 0x037365 to 0x01f1cf (0x18195 bytes) 000086 1787832440.577901 trunc from 0x01f1d0 to 0x03a806 (0x1b637 bytes) 000087 1787832440.601410 mapwrite 0x00cba2 thru 0x01c559 (0xf9b8 bytes) 000088 1787832440.666751 read 0x025488 thru 0x034514 (0xf08d bytes) 000089 1787832440.679065 punch from 0x005a3c to 0x00e312 (0x88d7 bytes) 000090 1787832440.686923 mapwrite 0x0312ef thru 0x03f3af (0xe0c1 bytes) 000091 1787832440.746712 mapwrite 0x02e940 thru 0x03da4a (0xf10b bytes) 000092 1787832440.776315 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) 000093 1787832440.913673 trunc from 0x03f3b0 to 0x0303d5 (0xefda bytes) 000094 1787832440.941555 mapwrite 0x03c473 thru 0x03ffff (0x3b8d bytes) 000095 1787832440.986983 punch from 0x020e0a to 0x026e55 (0x604c bytes) 000096 1787832440.991827 fallocate 0x01fa93 thru 0x02ec2a (0xf198 bytes) 000097 1787832440.996689 write 0x00dc58 thru 0x015a69 (0x7e12 bytes) 000098 1787832441.006167 mapwrite 0x00f59d thru 0x01db0a (0xe56e bytes) 000099 1787832441.031765 trunc from 0x00af99 to 0x010ec1 (0x5f29 bytes) 000100 1787832441.036987 write 0x02539e thru 0x030c8c (0xb8ef bytes) 000102 1787832441.048343 trunc from 0x040000 to 0x01d63b (0x229c4 bytes) 000103 1787832441.067862 fallocate 0x028331 thru 0x02f4cb (0x719b bytes) 000104 1787832441.072912 write 0x01d270 thru 0x0258da (0x866b bytes) 000105 1787832441.078492 trunc from 0x012702 to 0x017782 (0x5081 bytes) 000106 1787832441.083883 trunc from 0x02f4cc to 0x00429f (0x2b22c bytes) 000107 1787832441.098476 trunc from 0x0042a0 to 0x038133 (0x33e94 bytes) 000108 1787832441.116003 mapwrite 0x03bc8b thru 0x03ffff (0x4375 bytes) 000109 1787832441.165049 read 0x0355d7 thru 0x03d5b0 (0x7fda bytes) 000110 1787832441.165716 mapwrite 0x01a428 thru 0x026e33 (0xca0c bytes) 000111 1787832441.202854 write 0x036e1d thru 0x03f245 (0x8429 bytes) 000112 1787832441.204960 fallocate 0x0284cb thru 0x03799b (0xf4d1 bytes) 000113 1787832441.209698 read 0x00cd62 thru 0x0116ee (0x498d bytes) 000114 1787832441.219004 trunc from 0x040000 to 0x00bb04 (0x344fb bytes) 000115 1787832441.234037 punch from 0x009282 to 0x00bb04 (0x2883 bytes) 000116 1787832441.239426 punch from 0x002c76 to 0x007828 (0x4bb3 bytes) 000118 1787832441.245578 write 0x019fc6 thru 0x024b6f (0xabaa bytes) 000121 1787832441.250020 read 0x00cd9a thru 0x01aa87 (0xdcee bytes) 000122 1787832441.263842 trunc from 0x024b70 to 0x00d38d (0x177e2 bytes) 000123 1787832441.278621 fallocate 0x026785 thru 0x03114d (0xa9c9 bytes) 000124 1787832441.283424 mapwrite 0x013fa1 thru 0x016fbf (0x301f bytes) 000125 1787832441.326346 trunc from 0x03114e to 0x02b016 (0x6137 bytes) 000126 1787832441.342917 mapwrite 0x002ac9 thru 0x009866 (0x6d9e bytes) 000127 1787832441.370435 fallocate 0x023b48 thru 0x02f85a (0xbd13 bytes) 000130 1787832441.380062 mapwrite 0x0213b2 thru 0x02c402 (0xb051 bytes) 000131 1787832441.456051 mapwrite 0x00bedd thru 0x01267e (0x67a2 bytes) 000132 1787832441.547027 mapread 0x0261eb thru 0x02a695 (0x44ab bytes) 000134 1787832441.556483 mapread 0x006b99 thru 0x00d8ab (0x6d13 bytes) 000135 1787832441.584392 trunc from 0x02c403 to 0x02990b (0x2af7 bytes) 000136 1787832441.611159 trunc from 0x02990c to 0x03a0ea (0x107df bytes) 000137 1787832441.634418 read 0x037762 thru 0x03a0ea (0x2989 bytes) 000138 1787832441.646824 mapread 0x024c75 thru 0x030352 (0xb6de bytes) 000139 1787832441.648587 write 0x01acd2 thru 0x020a33 (0x5d62 bytes) 000140 1787832441.650932 write 0x038912 thru 0x03ffff (0x76ee bytes) 000141 1787832441.652988 read 0x020a1f thru 0x02dbc2 (0xd1a4 bytes) 000142 1787832441.654036 trunc from 0x03eb32 to 0x03ffff (0x14ce bytes) 000143 1787832441.672477 mapread 0x02381f thru 0x02b3ea (0x7bcc bytes) 000145 1787832441.673876 punch from 0x01053e to 0x01ce00 (0xc8c3 bytes) 000146 1787832441.683038 read 0x035ac1 thru 0x03a98c (0x4ecc bytes) 000147 1787832441.685551 punch from 0x010c17 to 0x01b216 (0xa600 bytes) 000148 1787832441.703867 trunc from 0x040000 to 0x00a151 (0x35eae bytes) 000149 1787832441.729008 read 0x008ac9 thru 0x00a151 (0x1689 bytes) 000150 1787832441.730887 fallocate 0x002228 thru 0x0102e4 (0xe0bd bytes) 000151 1787832441.736267 mapwrite 0x003087 thru 0x00ca88 (0x9a02 bytes) 000152 1787832441.792783 write 0x00ee86 thru 0x0184f4 (0x966f bytes) 000153 1787832441.794697 trunc from 0x0184f5 to 0x02c3c9 (0x13ed5 bytes) 000154 1787832441.817991 read 0x0105b0 thru 0x0195d3 (0x9024 bytes) 000155 1787832441.833515 read 0x009468 thru 0x00a7ee (0x1387 bytes) 000156 1787832441.835964 trunc from 0x023b76 to 0x0266d2 (0x2b5d bytes) 000157 1787832441.843009 trunc from 0x000021 to 0x00594d (0x592d bytes) 000158 1787832441.852646 trunc from 0x02c3ca to 0x00d219 (0x1f1b0 bytes) 000159 1787832441.881054 mapread 0x00ad31 thru 0x00d219 (0x24e9 bytes) 000160 1787832441.882135 mapwrite 0x016d8f thru 0x0218b8 (0xab2a bytes) 000163 1787832441.928553 mapwrite 0x026112 thru 0x034426 (0xe315 bytes) 000164 1787832441.985502 trunc from 0x034427 to 0x0153b8 (0x1f06e bytes) 000165 1787832442.013801 fallocate 0x024484 thru 0x02ed74 (0xa8f1 bytes) 000166 1787832442.019462 punch from 0x006a1a to 0x009231 (0x2818 bytes) 000167 1787832442.027434 mapwrite 0x029b8b thru 0x038f03 (0xf379 bytes) 000168 1787832442.177792 trunc from 0x038f04 to 0x023b91 (0x15372 bytes) 000169 1787832442.199929 MIRROR_EXTEND to 2 mirrors 000170 1787832442.844523 fallocate 0x020b0b thru 0x02ff11 (0xf407 bytes) 000171 1787832442.885919 mapread 0x001cd1 thru 0x007b3f (0x5e6f bytes) 000172 1787832442.910194 write 0x025b8f thru 0x02d9b6 (0x7e28 bytes) 000173 1787832442.915458 trunc from 0x02d9b7 to 0x02a8a2 (0x3114 bytes) 000174 1787832442.938207 mapread 0x0077c4 thru 0x00e7c7 (0x7004 bytes) 000175 1787832442.939035 mapread 0x01e2ec thru 0x028213 (0x9f28 bytes) 000176 1787832442.952806 trunc from 0x02a8a3 to 0x00a44c (0x20456 bytes) 000178 1787832442.972418 trunc from 0x00a44d to 0x01e78b (0x1433f bytes) 000180 1787832442.986880 trunc from 0x01e78c to 0x0333fd (0x14c72 bytes) 000181 1787832442.998507 mapwrite 0x00e295 thru 0x01a03e (0xbdaa bytes) 000182 1787832443.027409 trunc from 0x026989 to 0x032373 (0xb9eb bytes) 000185 1787832443.033292 fallocate 0x037344 thru 0x03ffff (0x8cbc bytes) 000186 1787832443.038861 mapwrite 0x003601 thru 0x007fed (0x49ed bytes) 000187 1787832443.061999 MIRROR_RESYNC 2 mirrors 000188 1787832443.898641 fallocate 0x02679d thru 0x0288fc (0x2160 bytes) 000189 1787832443.972108 trunc from 0x0333fe to 0x026049 (0xd3b4 bytes) 000190 1787832444.042903 MIRROR_RESYNC 2 mirrors 000191 1787832444.688670 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) 000193 1787832444.863706 read 0x01953f thru 0x01d404 (0x3ec6 bytes) 000194 1787832444.883998 trunc from 0x01b7f6 to 0x025806 (0xa011 bytes) 000196 1787832444.911516 fallocate 0x03796b thru 0x03ffff (0x8695 bytes) 000199 1787832444.922939 mapread 0x017539 thru 0x02433a (0xce02 bytes) 000200 1787832444.965939 punch from 0x025e5f to 0x026049 (0x1eb bytes) 000201 1787832444.972995 mapread 0x023ed6 thru 0x02435c (0x487 bytes) 000202 1787832444.976166 write 0x0356ce thru 0x03cdfd (0x7730 bytes) 000203 1787832444.984198 read 0x004f93 thru 0x00a60f (0x567d bytes) 000205 1787832445.004821 trunc from 0x03cdfe to 0x0279be (0x1543f bytes) 000207 1787832445.031959 mapread 0x00c97c thru 0x01653d (0x9bc2 bytes) 000208 1787832445.032969 trunc from 0x0279bf to 0x00452f (0x2348f bytes) 000209 1787832445.064976 MIRROR_EXTEND to 2 mirrors 000210 1787832445.719131 fallocate 0x03d411 thru 0x03ffff (0x2bef bytes) 000211 1787832445.776579 trunc from 0x004530 to 0x03b05e (0x36b2f bytes) 000212 1787832445.820030 mapwrite 0x01b7ad thru 0x02926f (0xdac3 bytes) 000213 1787832445.877748 mapwrite 0x01c17b thru 0x01cbcb (0xa51 bytes) 000214 1787832445.891555 trunc from 0x020ccb to 0x027678 (0x69ae bytes) 000215 1787832445.896543 trunc from 0x027318 to 0x03534c (0xe035 bytes) 000216 1787832445.901558 mapwrite 0x00f5c4 thru 0x015f47 (0x6984 bytes) 000217 1787832445.923803 read 0x00f01d thru 0x015e45 (0x6e29 bytes) 000218 1787832445.924535 trunc from 0x03b05f to 0x0256d6 (0x15988 bytes) 000219 1787832445.941620 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) 000220 1787832446.061498 mapwrite 0x020c6f thru 0x0288e1 (0x7c73 bytes) 000221 1787832446.120567 mapread 0x00851f thru 0x00f6c3 (0x71a5 bytes) 000222 1787832446.129646 trunc from 0x0288e2 to 0x03bedd (0x135fc bytes) 000223 1787832446.141710 trunc from 0x03bede to 0x002361 (0x39b7c bytes) 000226 1787832446.158699 read 0x00044a thru 0x002361 (0x1f18 bytes) 000227 1787832446.163752 fallocate 0x0326ec thru 0x034973 (0x2288 bytes) 000228 1787832446.168639 trunc from 0x01fc8b to 0x02e700 (0xea76 bytes) 000229 1787832446.172651 trunc from 0x034974 to 0x03f46e (0xaafb bytes) 000230 1787832446.188662 punch from 0x033647 to 0x03667a (0x3034 bytes) 000231 1787832446.192405 read 0x03eb1b thru 0x03f46e (0x954 bytes) 000232 1787832446.220319 mapread 0x01e5f9 thru 0x02121c (0x2c24 bytes) 000233 1787832446.221362 read 0x033dbf thru 0x03f46e (0xb6b0 bytes) 000234 1787832446.222333 fallocate 0x026f96 thru 0x035809 (0xe874 bytes) 000235 1787832446.226435 fallocate 0x014161 thru 0x0196e1 (0x5581 bytes) 000236 1787832446.230864 trunc from 0x03f46f to 0x039c1d (0x5851 bytes) 000237 1787832446.246429 fallocate 0x010953 thru 0x01dcde (0xd38c bytes) 000239 1787832446.251841 read 0x00e12b thru 0x01a9f2 (0xc8c8 bytes) 000240 1787832446.252875 trunc from 0x01b2a1 to 0x02b173 (0xfed3 bytes) 000241 1787832446.258597 fallocate 0x035313 thru 0x038014 (0x2d02 bytes) 000242 1787832446.262877 mapwrite 0x010bde thru 0x017291 (0x66b4 bytes) 000243 1787832446.283929 MIRROR_EXTEND to 2 mirrors 000244 1787832446.815895 trunc from 0x027d34 to 0x0365c4 (0xe891 bytes) 000245 1787832446.877001 punch from 0x01073d to 0x014fa4 (0x4868 bytes) 000246 1787832446.885974 trunc from 0x039c1e to 0x0111e5 (0x28a38 bytes) 000247 1787832446.946370 read 0x008297 thru 0x00af80 (0x2cea bytes) 000248 1787832446.953657 write 0x024600 thru 0x024dd6 (0x7d7 bytes) 000249 1787832446.956087 mapwrite 0x0247b7 thru 0x030389 (0xbbd3 bytes) 000250 1787832447.031483 read 0x027ceb thru 0x02b871 (0x3b87 bytes) 000251 1787832447.032105 trunc from 0x02311f to 0x030389 (0xd26b bytes) 000252 1787832447.044455 MIRROR_RESYNC 2 mirrors 000253 1787832447.689457 mapwrite 0x00689c thru 0x00c5aa (0x5d0f bytes) 000254 1787832447.786951 trunc from 0x01cef3 to 0x01fba1 (0x2caf bytes) 000255 1787832447.792638 trunc from 0x03038a to 0x00e6c9 (0x21cc0 bytes) 000256 1787832447.837641 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) 000257 1787832448.028684 mapread 0x00577f thru 0x00e6c9 (0x8f4b bytes) 000258 1787832448.070112 trunc from 0x00e6ca to 0x006a49 (0x7c80 bytes) 000259 1787832448.148882 mapwrite 0x003ad5 thru 0x005410 (0x193c bytes) 000260 1787832448.190897 trunc from 0x006a4a to 0x019058 (0x1260f bytes) 000261 1787832448.230808 mapwrite 0x0261da thru 0x035a0a (0xf831 bytes) 000263 1787832448.405937 punch from 0x022be0 to 0x02481f (0x1c40 bytes) 000264 1787832448.412001 trunc from 0x035a0b to 0x0293f6 (0xc614 bytes) 000265 1787832448.430423 fallocate 0x03d153 thru 0x03ffff (0x2ead bytes) 000266 1787832448.435461 trunc from 0x00a9d1 to 0x0186df (0xdd0f bytes) 000267 1787832448.439937 read 0x03d0de thru 0x03ffff (0x2f22 bytes) 000268 1787832448.446861 write 0x035d47 thru 0x03a3e6 (0x46a0 bytes) 000269 1787832448.456920 trunc from 0x040000 to 0x00b5aa (0x34a55 bytes) 000270 1787832448.478113 fallocate 0x00a354 thru 0x0165bb (0xc268 bytes) 000271 1787832448.483000 mapwrite 0x00e10f thru 0x0198d9 (0xb7cb bytes) 000272 1787832448.570459 fallocate 0x01bba6 thru 0x01f0e3 (0x353e bytes) 000273 1787832448.591188 MIRROR_EXTEND to 2 mirrors 000274 1787832449.333833 read 0x01aeca thru 0x01f0e3 (0x421a bytes) 000276 1787832449.346844 MIRROR_RESYNC 2 mirrors 000277 1787832449.702606 mapread 0x015639 thru 0x01eb58 (0x9520 bytes) 000280 1787832449.734517 mapwrite 0x00ff01 thru 0x011c91 (0x1d91 bytes) 000281 1787832449.811647 trunc from 0x01f0e4 to 0x01f2b9 (0x1d6 bytes) 000282 1787832449.839629 trunc from 0x01f2ba to 0x003fb6 (0x1b303 bytes) 000283 1787832449.856408 fallocate 0x014978 thru 0x01a68b (0x5d14 bytes) 000284 1787832449.861863 punch from 0x0005a0 to 0x001e3b (0x189c bytes) 000285 1787832449.867719 mapwrite 0x03c3fb thru 0x03f1e9 (0x2def bytes) 000287 1787832449.904744 punch from 0x00dd19 to 0x015f64 (0x824c bytes) 000288 1787832449.910687 trunc from 0x03d6cf to 0x03f1e9 (0x1b1b bytes) 000289 1787832449.915831 trunc from 0x03d597 to 0x03f1e9 (0x1c53 bytes) 000290 1787832449.921413 mapread 0x031f9d thru 0x03f1e9 (0xd24d bytes) 000291 1787832449.935726 punch from 0x0328f9 to 0x03cf9c (0xa6a4 bytes) 000292 1787832449.944891 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f1ea): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f1ea): Mirror 2: 0x3c221635 (stale) CRC-32 checksum value for chunk [0, 0x3f1ea): Mirror 1: 0xdb0549ae Mirror 3: 0xdb0549ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f1ea: skipped 000293 1787832450.738543 write 0x02f6c3 thru 0x03ceb8 (0xd7f6 bytes) 000294 1787832450.790428 fallocate 0x019a08 thru 0x02781d (0xde16 bytes) 000295 1787832450.796253 mapread 0x00b63a thru 0x00d17d (0x1b44 bytes) 000296 1787832450.819383 mapwrite 0x018811 thru 0x01d5cd (0x4dbd bytes) 000297 1787832450.848695 mapread 0x02bba1 thru 0x02e172 (0x25d2 bytes) 000298 1787832450.861517 punch from 0x036158 to 0x03f1e9 (0x9092 bytes) 000299 1787832450.866806 mapwrite 0x00b09d thru 0x00e119 (0x307d bytes) 000300 1787832450.882123 fallocate 0x00c914 thru 0x010c07 (0x42f4 bytes) 000301 1787832450.887580 write 0x02688f thru 0x02d323 (0x6a95 bytes) 000303 1787832450.891334 mapread 0x02b031 thru 0x038dfd (0xddcd bytes) 000305 1787832450.908133 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f1ea): Mirror 3: 0xdb0549ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f1ea): Mirror 3: 0xdb0549ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f1ea: skipped 000306 1787832451.179826 mapread 0x0354a5 thru 0x038910 (0x346c bytes) 000307 1787832451.203263 fallocate 0x037a6a thru 0x039878 (0x1e0f bytes) 000308 1787832451.208306 trunc from 0x02137c to 0x02b334 (0x9fb9 bytes) 000309 1787832451.213819 mapread 0x00f18a thru 0x01333d (0x41b4 bytes) 000310 1787832451.224291 mapwrite 0x018fcb thru 0x020d15 (0x7d4b bytes) 000311 1787832451.252454 write 0x006ed0 thru 0x010762 (0x9893 bytes) 000312 1787832451.262912 mapwrite 0x00ef6a thru 0x01981d (0xa8b4 bytes) 000313 1787832451.281879 trunc from 0x03e00a to 0x03f1e9 (0x11e0 bytes) 000314 1787832451.287438 write 0x0303e4 thru 0x03e55d (0xe17a bytes) 000316 1787832451.293823 trunc from 0x03f1ea to 0x037c98 (0x7551 bytes) 000317 1787832451.319105 MIRROR_RESYNC 2 mirrors 000318 1787832451.979658 read 0x01f1c6 thru 0x01f2c9 (0x104 bytes) 000319 1787832451.992686 read 0x001766 thru 0x0024c0 (0xd5b bytes) 000320 1787832452.016331 mapread 0x0130df thru 0x01d7af (0xa6d1 bytes) 000321 1787832452.017970 fallocate 0x00ecfa thru 0x01bad8 (0xcddf bytes) 000322 1787832452.059579 read 0x02187c thru 0x025b76 (0x42fb bytes) 000323 1787832452.073840 fallocate 0x021372 thru 0x02ec78 (0xd907 bytes) 000324 1787832452.079742 read 0x00f5e1 thru 0x01191b (0x233b bytes) 000325 1787832452.084865 MIRROR_RESYNC 2 mirrors 000326 1787832452.779704 mapread 0x034775 thru 0x035c7c (0x1508 bytes) 000327 1787832452.794726 trunc from 0x037c99 to 0x0119ee (0x262aa bytes) 000328 1787832452.847796 MIRROR_RESYNC 2 mirrors 000329 1787832453.373428 trunc from 0x009799 to 0x0119ee (0x8256 bytes) 000330 1787832453.416339 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x119ef): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x119ef): Mirror 3: 0x7ba25e (stale) CRC-32 checksum value for chunk [0, 0x119ef): Mirror 1: 0x1055099d Mirror 4: 0x1055099d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x119ef: skipped 000331 1787832454.203504 write 0x022987 thru 0x02320c (0x886 bytes) 000332 1787832454.247707 read 0x011dcc thru 0x0176d7 (0x590c bytes) 000334 1787832454.259556 write 0x00242b thru 0x00cc91 (0xa867 bytes) 000335 1787832454.277038 read 0x004a20 thru 0x012af5 (0xe0d6 bytes) 000337 1787832454.289553 read 0x007250 thru 0x009d21 (0x2ad2 bytes) 000338 1787832454.290901 write 0x02806f thru 0x02bd17 (0x3ca9 bytes) 000339 1787832454.292768 read 0x01457f thru 0x01fabf (0xb541 bytes) 000340 1787832454.293828 punch from 0x0275ed to 0x02bd17 (0x472b bytes) 000341 1787832454.316654 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2bd18): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bd18): Mirror 3: 0x5fd7b999 (stale) Mirror 4: 0xd8c556ba (stale) CRC-32 checksum value for chunk [0, 0x2bd18): Mirror 1: 0xf170c0c5 Mirror 5: 0xf170c0c5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bd18: skipped 000342 1787832455.110857 punch from 0x023923 to 0x02bd17 (0x83f5 bytes) 000343 1787832455.164150 mapwrite 0x00786b thru 0x009cb5 (0x244b bytes) 000344 1787832455.210790 trunc from 0x01db5b to 0x01fe8d (0x2333 bytes) 000345 1787832455.221148 fallocate 0x024b33 thru 0x02917b (0x4649 bytes) 000346 1787832455.229449 mapwrite 0x00402c thru 0x005b6e (0x1b43 bytes) 000347 1787832455.244933 write 0x008ed3 thru 0x00992e (0xa5c bytes) 000348 1787832455.252178 trunc from 0x02bd18 to 0x0302f2 (0x45db bytes) 000349 1787832455.324915 trunc from 0x002684 to 0x004408 (0x1d85 bytes) 000350 1787832455.332820 punch from 0x01028e to 0x0142ae (0x4021 bytes) 000351 1787832455.339915 trunc from 0x0302f3 to 0x02d914 (0x29de bytes) 000352 1787832455.371819 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d915): Mirror 4: 0xe2515418 (stale) Mirror 5: 0x7a0e8442 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d915): Mirror 4: 0xe2515418 (stale) Mirror 5: 0x7a0e8442 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d915: skipped 000353 1787832455.869328 mapwrite 0x029867 thru 0x030dc1 (0x755b bytes) 000354 1787832455.984210 write 0x037988 thru 0x03ecd9 (0x7352 bytes) 000355 1787832455.986012 write 0x03fc89 thru 0x03ffff (0x377 bytes) 000356 1787832455.995714 write 0x011698 thru 0x016d30 (0x5699 bytes) 000357 1787832456.023888 trunc from 0x011244 to 0x018dd8 (0x7b95 bytes) 000358 1787832456.052024 mapread 0x006df3 thru 0x00ff47 (0x9155 bytes) 000359 1787832456.068518 mapwrite 0x024cfa thru 0x03223b (0xd542 bytes) 000360 1787832456.139455 trunc from 0x040000 to 0x030b7c (0xf483 bytes) 000361 1787832456.175768 MIRROR_RESYNC 3 mirrors 000362 1787832457.049922 trunc from 0x007b5b to 0x0097c2 (0x1c68 bytes) 000363 1787832457.091383 mapwrite 0x0034df thru 0x007549 (0x406b bytes) 000364 1787832457.122356 trunc from 0x030b7d to 0x01d69f (0x134dd bytes) 000365 1787832457.138712 mapwrite 0x03a604 thru 0x03ffff (0x59fc bytes) 000366 1787832457.179384 mapread 0x00b214 thru 0x0120ec (0x6ed9 bytes) 000367 1787832457.188636 fallocate 0x003e0b thru 0x009422 (0x5618 bytes) 000368 1787832457.193342 mapwrite 0x03c9e8 thru 0x03ffff (0x3618 bytes) 000369 1787832457.207606 fallocate 0x03c2ab thru 0x03ffff (0x3d55 bytes) 000370 1787832457.212089 trunc from 0x040000 to 0x03a30f (0x5cf0 bytes) 000371 1787832457.228728 mapwrite 0x025778 thru 0x02ea3b (0x92c4 bytes) 000372 1787832457.265892 mapwrite 0x03cf08 thru 0x03ffff (0x30f8 bytes) 000373 1787832457.301047 trunc from 0x040000 to 0x00799f (0x38660 bytes) 000374 1787832457.325674 trunc from 0x001a24 to 0x00799f (0x5f7c bytes) 000375 1787832457.333166 trunc from 0x002dbd to 0x00799f (0x4be3 bytes) 000376 1787832457.338204 write 0x03ef91 thru 0x03ffff (0x106f bytes) 000377 1787832457.339593 trunc from 0x040000 to 0x029279 (0x16d86 bytes) 000378 1787832457.354908 mapread 0x017b54 thru 0x01b718 (0x3bc5 bytes) 000379 1787832457.368760 mapwrite 0x0000fa thru 0x005231 (0x5138 bytes) 000380 1787832457.401037 write 0x030596 thru 0x036c5c (0x66c7 bytes) 000381 1787832457.405725 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36c5d): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36c5d): Mirror 4: 0x1b4d49e4 (stale) Mirror 5: 0x1b4d49e4 (stale) CRC-32 checksum value for chunk [0, 0x36c5d): Mirror 1: 0x35444370 Mirror 6: 0x35444370 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36c5d: skipped 000382 1787832458.308141 trunc from 0x036c5d to 0x027504 (0xf758 bytes) 000383 1787832458.392303 fallocate 0x00807d thru 0x00a9dd (0x2961 bytes) 000384 1787832458.401981 punch from 0x00524f to 0x01515a (0xff0c bytes) 000385 1787832458.407053 mapread 0x01d95b thru 0x026eba (0x9560 bytes) 000386 1787832458.452921 read 0x00c217 thru 0x011d2d (0x5b17 bytes) 000388 1787832458.485948 punch from 0x005da7 to 0x007271 (0x14cb bytes) 000389 1787832458.499380 trunc from 0x027505 to 0x009c39 (0x1d8cb bytes) 000390 1787832458.566525 mapread 0x008795 thru 0x009bb6 (0x1422 bytes) 000391 1787832458.627850 read 0x003478 thru 0x009c39 (0x67c2 bytes) 000392 1787832458.629536 write 0x02f8bb thru 0x033824 (0x3f6a bytes) 000393 1787832458.631082 fallocate 0x0084b7 thru 0x00f633 (0x717d bytes) 000394 1787832458.642173 fallocate 0x015a6c thru 0x0226fc (0xcc91 bytes) 000395 1787832458.661653 trunc from 0x02d9d2 to 0x02ef16 (0x1545 bytes) 000396 1787832458.684602 write 0x01cfc7 thru 0x02179c (0x47d6 bytes) 000397 1787832458.706764 trunc from 0x033825 to 0x0236ed (0x10137 bytes) 000398 1787832458.725468 mapread 0x01eace thru 0x0236ed (0x4c20 bytes) 000399 1787832458.734388 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x236ee): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x236ee): Mirror 4: 0x1503a953 (stale) Mirror 5: 0x1503a953 (stale) Mirror 6: 0x72bf1d7e (stale) CRC-32 checksum value for chunk [0, 0x236ee): Mirror 1: 0xdfd1b226 Mirror 7: 0xdfd1b226 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x236ee: skipped 000400 1787832459.979203 trunc from 0x0236ee to 0x02a938 (0x724b bytes) 000401 1787832460.029537 fallocate 0x01f8b8 thru 0x01fd82 (0x4cb bytes) 000402 1787832460.037856 MIRROR_RESYNC 5 mirrors 000403 1787832460.938356 punch from 0x019d44 to 0x0261a1 (0xc45e bytes) 000404 1787832460.966394 read 0x00cc9a thru 0x01c95f (0xfcc6 bytes) 000405 1787832460.975562 fallocate 0x01252f thru 0x0178b8 (0x538a bytes) 000406 1787832460.979869 mapread 0x018a2f thru 0x01a869 (0x1e3b bytes) 000407 1787832460.980713 trunc from 0x021422 to 0x02276e (0x134d bytes) 000408 1787832460.985262 punch from 0x01fab5 to 0x027649 (0x7b95 bytes) 000409 1787832460.989251 trunc from 0x0159ce to 0x018447 (0x2a7a bytes) 000410 1787832460.993377 mapread 0x00698f thru 0x00c77a (0x5dec bytes) 000411 1787832461.009489 mapwrite 0x02daa5 thru 0x03d3d3 (0xf92f bytes) 000412 1787832461.073353 read 0x03032b thru 0x03d3d3 (0xd0a9 bytes) 000413 1787832461.074693 write 0x03d51b thru 0x03ffff (0x2ae5 bytes) 000414 1787832461.075797 mapwrite 0x03f44f thru 0x03ffff (0xbb1 bytes) 000415 1787832461.086693 mapread 0x00d32a thru 0x01007f (0x2d56 bytes) 000416 1787832461.087581 mapread 0x0013ec thru 0x009f34 (0x8b49 bytes) 000417 1787832461.088448 write 0x03d80b thru 0x03ffff (0x27f5 bytes) 000418 1787832461.090252 mapread 0x001908 thru 0x00ae83 (0x957c bytes) 000419 1787832461.095409 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xaa96b079 (stale) Mirror 6: 0xaa96b079 (stale) Mirror 7: 0xaa96b079 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xaa96b079 (stale) Mirror 6: 0xaa96b079 (stale) Mirror 7: 0xaa96b079 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000420 1787832461.399890 trunc from 0x040000 to 0x036124 (0x9edb bytes) 000421 1787832461.418731 write 0x030355 thru 0x0359cc (0x5678 bytes) 000422 1787832461.427736 punch from 0x0199d0 to 0x0204e9 (0x6b1a bytes) 000423 1787832461.435882 fallocate 0x03cb6a thru 0x03ffff (0x3496 bytes) 000424 1787832461.440674 read 0x00534e thru 0x005673 (0x326 bytes) 000425 1787832461.444900 write 0x018968 thru 0x021921 (0x8fba bytes) 000426 1787832461.453552 punch from 0x02a0de to 0x02b932 (0x1855 bytes) 000427 1787832461.460558 MIRROR_RESYNC 4 mirrors 000429 1787832462.151255 trunc from 0x036125 to 0x018fe7 (0x1d13d bytes) 000430 1787832462.216180 trunc from 0x00fc8c to 0x0101a2 (0x517 bytes) 000431 1787832462.220085 read 0x018ee6 thru 0x018fe7 (0x102 bytes) 000432 1787832462.225809 read 0x00170f thru 0x00e25c (0xcb4e bytes) 000433 1787832462.234699 read 0x01602e thru 0x01817c (0x214f bytes) 000434 1787832462.235350 write 0x00dda3 thru 0x0118b9 (0x3b17 bytes) 000435 1787832462.239951 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x18fe8): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18fe8): Mirror 4: 0x6b36b924 (stale) Mirror 6: 0x6b36b924 (stale) Mirror 7: 0x6b36b924 (stale) CRC-32 checksum value for chunk [0, 0x18fe8): Mirror 1: 0xbe6bd868 Mirror 8: 0xbe6bd868 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18fe8: skipped 000436 1787832462.844571 mapread 0x01184e thru 0x018fe7 (0x779a bytes) 000437 1787832462.864196 trunc from 0x011280 to 0x018fe7 (0x7d68 bytes) 000438 1787832462.903018 punch from 0x003f9e to 0x0070c2 (0x3125 bytes) 000439 1787832462.907691 fallocate 0x025a20 thru 0x02bd5b (0x633c bytes) 000440 1787832462.913246 mapread 0x004a63 thru 0x006ad4 (0x2072 bytes) 000441 1787832462.922042 mapwrite 0x01df0e thru 0x02a0b2 (0xc1a5 bytes) 000442 1787832462.953459 trunc from 0x02bd5c to 0x03ee09 (0x130ae bytes) 000443 1787832462.998802 write 0x006994 thru 0x00946c (0x2ad9 bytes) 000445 1787832463.001695 trunc from 0x005b45 to 0x010abf (0xaf7b bytes) 000446 1787832463.013411 fallocate 0x031420 thru 0x035883 (0x4464 bytes) 000447 1787832463.021788 MIRROR_RESYNC 5 mirrors 000448 1787832464.153492 trunc from 0x03ee0a to 0x00255d (0x3c8ac bytes) 000449 1787832464.200557 mapwrite 0x008527 thru 0x00b5e8 (0x30c2 bytes) 000451 1787832464.247380 mapwrite 0x00a127 thru 0x0126ba (0x8594 bytes) 000452 1787832464.304464 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x126bb): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x126bb): Mirror 4: 0xa1db3554 (stale) Mirror 6: 0xa1db3554 (stale) Mirror 7: 0xa1db3554 (stale) Mirror 8: 0xa1db3554 (stale) CRC-32 checksum value for chunk [0, 0x126bb): Mirror 1: 0xfe9c0f89 Mirror 9: 0xfe9c0f89 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x126bb: skipped 000454 1787832464.993577 trunc from 0x0126bb to 0x03541f (0x22d65 bytes) 000456 1787832465.081171 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35420): Mirror 4: 0xaca9a649 (stale) Mirror 7: 0xaca9a649 (stale) Mirror 8: 0xaca9a649 (stale) Mirror 9: 0x2add67f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35420): Mirror 4: 0xaca9a649 (stale) Mirror 7: 0xaca9a649 (stale) Mirror 8: 0xaca9a649 (stale) Mirror 9: 0x2add67f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35420: skipped 000457 1787832465.528111 read 0x00d05e thru 0x010039 (0x2fdc bytes) 000458 1787832465.547399 trunc from 0x035420 to 0x009d69 (0x2b6b6 bytes) 000459 1787832465.611431 mapwrite 0x00883b thru 0x018140 (0xf906 bytes) 000460 1787832465.694665 punch from 0x01270f to 0x018140 (0x5a32 bytes) 000462 1787832465.704123 trunc from 0x006c2c to 0x010936 (0x9d0b bytes) 000463 1787832465.710478 punch from 0x010c45 to 0x018140 (0x74fc bytes) 000464 1787832465.729518 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18141): Mirror 4: 0xf963ca9 (stale) Mirror 8: 0xf963ca9 (stale) Mirror 9: 0xf9677c57 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18141): Mirror 4: 0xf963ca9 (stale) Mirror 8: 0xf963ca9 (stale) Mirror 9: 0xf9677c57 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18141: skipped 000465 1787832466.031702 trunc from 0x00fe4b to 0x018140 (0x82f6 bytes) 000467 1787832466.037875 write 0x007fce thru 0x010b55 (0x8b88 bytes) 000469 1787832466.053488 write 0x034910 thru 0x03badc (0x71cd bytes) 000470 1787832466.054950 mapwrite 0x0134aa thru 0x01751a (0x4071 bytes) 000471 1787832466.088178 punch from 0x0360f5 to 0x03badc (0x59e8 bytes) 000472 1787832466.093810 trunc from 0x03badd to 0x02a9ac (0x11130 bytes) 000473 1787832466.108511 mapread 0x007825 thru 0x00e9bf (0x719b bytes) 000474 1787832466.109771 fallocate 0x039fbd thru 0x03ffff (0x6043 bytes) 000475 1787832466.114444 mapwrite 0x027c91 thru 0x02e3fc (0x676c bytes) 000476 1787832466.138283 mapread 0x0290a6 thru 0x02f474 (0x63cf bytes) 000477 1787832466.142575 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf812bd56 (stale) Mirror 8: 0xf812bd56 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xf812bd56 (stale) Mirror 8: 0xf812bd56 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000478 1787832466.409550 mapread 0x01ef72 thru 0x021f39 (0x2fc8 bytes) 000479 1787832466.428121 mapwrite 0x00aa4b thru 0x017f8e (0xd544 bytes) 000480 1787832466.459265 write 0x01f1fe thru 0x01ffa2 (0xda5 bytes) 000481 1787832466.463825 read 0x002f70 thru 0x00e5e8 (0xb679 bytes) 000482 1787832466.473707 MIRROR_RESYNC 3 mirrors 000483 1787832467.343350 mapread 0x03222b thru 0x03df89 (0xbd5f bytes) 000484 1787832467.372558 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) 000485 1787832467.654774 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) 000486 1787832467.777172 MIRROR_EXTEND to 2 mirrors 000487 1787832468.335642 fallocate 0x029d70 thru 0x0319b7 (0x7c48 bytes) 000488 1787832468.374506 write 0x01411f thru 0x0216a2 (0xd584 bytes) 000489 1787832468.389088 fallocate 0x02627e thru 0x02a0c6 (0x3e49 bytes) 000490 1787832468.393707 trunc from 0x040000 to 0x0370f8 (0x8f07 bytes) 000491 1787832468.414547 read 0x00a128 thru 0x017878 (0xd751 bytes) 000492 1787832468.423606 read 0x0296c4 thru 0x035d2d (0xc66a bytes) 000493 1787832468.432978 trunc from 0x02f048 to 0x0331b0 (0x4169 bytes) 000495 1787832468.438164 mapwrite 0x0231b4 thru 0x02440f (0x125c bytes) 000496 1787832468.459107 fallocate 0x013b62 thru 0x017571 (0x3a10 bytes) 000497 1787832468.464429 mapwrite 0x00dd83 thru 0x016b91 (0x8e0f bytes) 000499 1787832468.488627 write 0x0173f8 thru 0x018442 (0x104b bytes) 000500 1787832468.489764 mapwrite 0x022d1e thru 0x02ddcc (0xb0af bytes) 000501 1787832468.516616 punch from 0x00d2c3 to 0x00fb49 (0x2887 bytes) 000502 1787832468.521779 punch from 0x019ebc to 0x025138 (0xb27d bytes) 000503 1787832468.534136 read 0x02641b thru 0x02ba6d (0x5653 bytes) 000504 1787832468.535189 mapwrite 0x03d003 thru 0x03ffff (0x2ffd bytes) 000505 1787832468.579273 punch from 0x032cac to 0x03dfea (0xb33f bytes) 000506 1787832468.585499 write 0x00019f thru 0x008fd7 (0x8e39 bytes) 000507 1787832468.587710 trunc from 0x040000 to 0x03549f (0xab60 bytes) 000508 1787832468.608624 mapwrite 0x026696 thru 0x030da9 (0xa714 bytes) 000509 1787832468.627212 mapwrite 0x03597e thru 0x03ffff (0xa682 bytes) 000510 1787832468.721316 write 0x0321ec thru 0x03a37a (0x818f bytes) 000511 1787832468.730804 mapwrite 0x018a7f thru 0x022d84 (0xa306 bytes) 000512 1787832468.748612 trunc from 0x024693 to 0x02505d (0x9cb bytes) 000514 1787832468.754211 write 0x01dec5 thru 0x021802 (0x393e bytes) 000516 1787832468.755729 read 0x03c657 thru 0x03ffff (0x39a9 bytes) 000517 1787832468.757074 write 0x036877 thru 0x03ffff (0x9789 bytes) 000518 1787832468.763337 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8954742c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4129132 Mirror 3: 0x4129132 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000519 1787832469.585717 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) 000520 1787832469.846630 fallocate 0x001285 thru 0x00f2a3 (0xe01f bytes) 000521 1787832469.896384 fallocate 0x00ca70 thru 0x01213b (0x56cc bytes) 000522 1787832469.902710 trunc from 0x040000 to 0x02b396 (0x14c69 bytes) 000523 1787832469.931487 trunc from 0x02b397 to 0x02eb48 (0x37b2 bytes) 000524 1787832469.953729 mapread 0x01e8e9 thru 0x02807c (0x9794 bytes) 000525 1787832469.975393 fallocate 0x020ac7 thru 0x02e27f (0xd7b9 bytes) 000526 1787832469.985004 MIRROR_RESYNC 2 mirrors 000527 1787832470.557407 read 0x001066 thru 0x00977c (0x8717 bytes) 000528 1787832470.567252 mapread 0x02975c thru 0x02eb48 (0x53ed bytes) 000529 1787832470.585638 trunc from 0x02eb49 to 0x02254f (0xc5f9 bytes) 000530 1787832470.628148 mapwrite 0x039e93 thru 0x03e072 (0x41e0 bytes) 000531 1787832470.676589 fallocate 0x004c90 thru 0x0116d6 (0xca47 bytes) 000532 1787832470.682175 mapread 0x02eda4 thru 0x03daae (0xed0b bytes) 000533 1787832470.693773 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e073): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e073): Mirror 3: 0xa9f0f12 (stale) CRC-32 checksum value for chunk [0, 0x3e073): Mirror 1: 0x533f2f34 Mirror 4: 0x533f2f34 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e073: skipped 000534 1787832471.394480 read 0x0112f5 thru 0x01dee7 (0xcbf3 bytes) 000535 1787832471.402837 write 0x011310 thru 0x015a83 (0x4774 bytes) 000537 1787832471.451458 trunc from 0x01591a to 0x01b767 (0x5e4e bytes) 000538 1787832471.463800 punch from 0x020af6 to 0x02e71b (0xdc26 bytes) 000539 1787832471.469737 write 0x03bbc7 thru 0x03ffff (0x4439 bytes) 000540 1787832471.474653 punch from 0x030b96 to 0x039087 (0x84f2 bytes) 000541 1787832471.480411 read 0x009808 thru 0x0101de (0x69d7 bytes) 000542 1787832471.491377 trunc from 0x040000 to 0x03b2c5 (0x4d3a bytes) 000543 1787832471.510172 write 0x02cf90 thru 0x034509 (0x757a bytes) 000544 1787832471.511713 punch from 0x0073ae to 0x00e900 (0x7553 bytes) 000545 1787832471.525159 MIRROR_RESYNC 3 mirrors 000546 1787832472.461922 read 0x0040d5 thru 0x00aefe (0x6e2a bytes) 000547 1787832472.479128 trunc from 0x03b2c6 to 0x03f301 (0x403c bytes) 000548 1787832472.560098 trunc from 0x034c3a to 0x03f301 (0xa6c8 bytes) 000549 1787832472.570937 punch from 0x0396ea to 0x03f301 (0x5c18 bytes) 000550 1787832472.575535 punch from 0x02f9cb to 0x03b90b (0xbf41 bytes) 000552 1787832472.581488 fallocate 0x02054f thru 0x0295cc (0x907e bytes) 000553 1787832472.591266 trunc from 0x036610 to 0x03f301 (0x8cf2 bytes) 000554 1787832472.597676 write 0x0199b5 thru 0x01c1ff (0x284b bytes) 000555 1787832472.615757 read 0x00801d thru 0x00803e (0x22 bytes) 000558 1787832472.654250 write 0x0349e7 thru 0x03c2a2 (0x78bc bytes) 000559 1787832472.655599 mapwrite 0x025742 thru 0x028d6a (0x3629 bytes) 000560 1787832472.689594 punch from 0x004534 to 0x006b2c (0x25f9 bytes) 000561 1787832472.698750 trunc from 0x03f302 to 0x00c885 (0x32a7c bytes) 000562 1787832472.727459 punch from 0x00b80e to 0x00c885 (0x1078 bytes) 000563 1787832472.732107 trunc from 0x008161 to 0x00c885 (0x4725 bytes) 000565 1787832472.740102 write 0x03f2a6 thru 0x03ffff (0xd5a bytes) 000566 1787832472.752485 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x35f1dbec (stale) Mirror 4: 0x35f1dbec (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x44f8173c Mirror 5: 0x44f8173c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000567 1787832473.554909 MIRROR_RESYNC 4 mirrors 000568 1787832474.725028 trunc from 0x034c71 to 0x03ffff (0xb38f bytes) 000569 1787832474.767592 read 0x00afc0 thru 0x015e24 (0xae65 bytes) 000570 1787832474.776515 read 0x0063be thru 0x012cf0 (0xc933 bytes) 000571 1787832474.786394 fallocate 0x023a02 thru 0x02af4f (0x754e bytes) 000572 1787832474.790949 trunc from 0x040000 to 0x021700 (0x1e8ff bytes) 000574 1787832474.817745 mapwrite 0x01be36 thru 0x0227f9 (0x69c4 bytes) 000576 1787832474.857887 mapwrite 0x02aecb thru 0x03a1f4 (0xf32a bytes) 000577 1787832474.962886 punch from 0x002da3 to 0x00827a (0x54d8 bytes) 000578 1787832474.968331 mapread 0x00f7a9 thru 0x010c34 (0x148c bytes) 000579 1787832474.969516 mapwrite 0x00c79c thru 0x019f60 (0xd7c5 bytes) 000580 1787832474.993731 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a1f5): Mirror 3: 0x699b142e (stale) Mirror 5: 0x699b142e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a1f5): Mirror 3: 0x699b142e (stale) Mirror 5: 0x699b142e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a1f5: skipped 000581 1787832475.303603 write 0x025dea thru 0x029d1f (0x3f36 bytes) 000582 1787832475.322022 trunc from 0x03a1f5 to 0x00c90f (0x2d8e5 bytes) 000583 1787832475.343531 fallocate 0x03ddce thru 0x03ffff (0x2232 bytes) 000584 1787832475.349727 write 0x017e2a thru 0x019707 (0x18de bytes) 000585 1787832475.351399 fallocate 0x01cff5 thru 0x02320d (0x6219 bytes) 000586 1787832475.357629 punch from 0x0108e7 to 0x016f13 (0x662d bytes) 000587 1787832475.363300 mapwrite 0x0037f3 thru 0x010e95 (0xd6a3 bytes) 000588 1787832475.433603 mapread 0x007393 thru 0x00fc4f (0x88bd bytes) 000589 1787832475.448361 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2320e): Mirror 3: 0x8006926f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2320e): Mirror 3: 0x8006926f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2320e: skipped 000590 1787832475.719946 mapread 0x0222aa thru 0x02320d (0xf64 bytes) 000591 1787832475.737456 punch from 0x008283 to 0x017250 (0xefce bytes) 000592 1787832475.742904 punch from 0x00d898 to 0x019f35 (0xc69e bytes) 000593 1787832475.750141 punch from 0x01dc94 to 0x02320d (0x557a bytes) 000595 1787832475.759531 MIRROR_RESYNC 2 mirrors 000596 1787832476.301592 punch from 0x002dd8 to 0x008099 (0x52c2 bytes) 000599 1787832476.342368 trunc from 0x02320e to 0x0107ff (0x12a0e bytes) 000600 1787832476.368371 fallocate 0x01211a thru 0x0159a1 (0x3888 bytes) 000601 1787832476.374108 read 0x0047f2 thru 0x00a086 (0x5895 bytes) 000602 1787832476.384069 mapwrite 0x022ae9 thru 0x02b02c (0x8544 bytes) 000604 1787832476.441365 fallocate 0x01739a thru 0x01a0ee (0x2d55 bytes) 000605 1787832476.447465 mapwrite 0x023e6c thru 0x02cbbf (0x8d54 bytes) 000606 1787832476.502987 mapread 0x015227 thru 0x01d1fe (0x7fd8 bytes) 000607 1787832476.510285 write 0x03db00 thru 0x03ffff (0x2500 bytes) 000608 1787832476.511450 trunc from 0x040000 to 0x012fe0 (0x2d01f bytes) 000609 1787832476.525483 write 0x027837 thru 0x036c04 (0xf3ce bytes) 000610 1787832476.527306 fallocate 0x00f9dd thru 0x012e13 (0x3437 bytes) 000611 1787832476.532552 write 0x03db59 thru 0x03ffff (0x24a7 bytes) 000612 1787832476.534754 write 0x009c5e thru 0x011fd0 (0x8373 bytes) 000613 1787832476.536371 write 0x03c43b thru 0x03ffff (0x3bc5 bytes) 000614 1787832476.542345 fallocate 0x00c3a5 thru 0x0128cf (0x652b bytes) 000615 1787832476.548495 mapwrite 0x03c27e thru 0x03dd0d (0x1a90 bytes) 000616 1787832476.567289 write 0x0283be thru 0x02ec38 (0x687b bytes) 000617 1787832476.568712 mapread 0x00b413 thru 0x014680 (0x926e bytes) 000618 1787832476.581307 write 0x0046bb thru 0x004a6c (0x3b2 bytes) 000619 1787832476.582843 write 0x02b30c thru 0x031996 (0x668b bytes) 000620 1787832476.588479 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xfa2e7951 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x37f7524d Mirror 4: 0x37f7524d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000621 1787832477.510710 write 0x007c8b thru 0x00d710 (0x5a86 bytes) 000622 1787832477.609767 write 0x02626f thru 0x02d694 (0x7426 bytes) 000623 1787832477.620727 write 0x023800 thru 0x02908a (0x588b bytes) 000624 1787832477.625089 punch from 0x030aed to 0x03d976 (0xce8a bytes) 000625 1787832477.630089 punch from 0x0112a3 to 0x015536 (0x4294 bytes) 000626 1787832477.634736 fallocate 0x03e6d8 thru 0x03ffff (0x1928 bytes) 000627 1787832477.642075 trunc from 0x040000 to 0x025d56 (0x1a2a9 bytes) 000628 1787832477.665123 trunc from 0x025d57 to 0x023930 (0x2426 bytes) 000629 1787832477.688527 trunc from 0x023931 to 0x02bcbd (0x838d bytes) 000630 1787832477.703759 fallocate 0x0145ec thru 0x01e4be (0x9ed3 bytes) 000631 1787832477.709140 trunc from 0x022d1a to 0x02b51e (0x8805 bytes) 000632 1787832477.723706 punch from 0x0265d5 to 0x02bcbd (0x56e9 bytes) 000633 1787832477.728595 mapread 0x00c636 thru 0x01365c (0x7027 bytes) 000634 1787832477.753341 MIRROR_RESYNC 3 mirrors 000635 1787832479.020998 punch from 0x01fcc0 to 0x02bcbd (0xbffe bytes) 000637 1787832479.056472 mapwrite 0x02893a thru 0x03557b (0xcc42 bytes) 000638 1787832479.129525 MIRROR_RESYNC 3 mirrors 000639 1787832480.220682 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) 000640 1787832480.618581 fallocate 0x021710 thru 0x02be1e (0xa70f bytes) 000641 1787832480.720628 fallocate 0x03729e thru 0x0399ae (0x2711 bytes) 000642 1787832480.728440 mapwrite 0x03f42f thru 0x03ffff (0xbd1 bytes) 000643 1787832480.817093 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) 000644 1787832480.979164 read 0x0232a4 thru 0x02972e (0x648b bytes) 000645 1787832480.992551 punch from 0x026bb8 to 0x032124 (0xb56d bytes) 000646 1787832480.999580 trunc from 0x014b85 to 0x016259 (0x16d5 bytes) 000647 1787832481.005628 punch from 0x018a5f to 0x01e754 (0x5cf6 bytes) 000648 1787832481.011349 trunc from 0x040000 to 0x030f82 (0xf07d bytes) 000649 1787832481.038407 punch from 0x01f11e to 0x02efbd (0xfea0 bytes) 000650 1787832481.043713 fallocate 0x0000a2 thru 0x004a51 (0x49b0 bytes) 000651 1787832481.049044 trunc from 0x030f83 to 0x03d7c6 (0xc844 bytes) 000652 1787832481.069546 trunc from 0x013832 to 0x01fa48 (0xc217 bytes) 000653 1787832481.075606 trunc from 0x036c3a to 0x03a62f (0x39f6 bytes) 000655 1787832481.082523 trunc from 0x03d7c7 to 0x018802 (0x24fc4 bytes) 000656 1787832481.101712 punch from 0x001d01 to 0x00f4d0 (0xd7d0 bytes) 000657 1787832481.107662 write 0x02222e thru 0x0309b4 (0xe787 bytes) 000658 1787832481.109406 trunc from 0x0309b5 to 0x01c9b8 (0x13ffc bytes) 000660 1787832481.130463 mapwrite 0x0234da thru 0x02db3d (0xa664 bytes) 000661 1787832481.193872 fallocate 0x0090bd thru 0x00cf05 (0x3e49 bytes) 000662 1787832481.198955 mapread 0x028507 thru 0x02b8fd (0x33f7 bytes) 000663 1787832481.201077 read 0x00d738 thru 0x0109a0 (0x3269 bytes) 000664 1787832481.214329 mapwrite 0x03f710 thru 0x03ffff (0x8f0 bytes) 000665 1787832481.266685 trunc from 0x040000 to 0x03f9d9 (0x626 bytes) 000666 1787832481.285474 trunc from 0x03f9da to 0x016a91 (0x28f48 bytes) 000667 1787832481.312590 read 0x005704 thru 0x00c6e0 (0x6fdd bytes) 000669 1787832481.314562 mapwrite 0x02a30d thru 0x032cd9 (0x89cd bytes) 000670 1787832481.369578 write 0x003e0a thru 0x008869 (0x4a60 bytes) 000671 1787832481.376546 MIRROR_EXTEND to 2 mirrors 000673 1787832482.101319 punch from 0x01bb48 to 0x024632 (0x8aeb bytes) 000674 1787832482.150062 read 0x0038f5 thru 0x004617 (0xd23 bytes) 000675 1787832482.170262 MIRROR_RESYNC 2 mirrors 000676 1787832482.815943 mapwrite 0x034f63 thru 0x03b4e1 (0x657f bytes) 000677 1787832482.891546 mapread 0x00f113 thru 0x0177f0 (0x86de bytes) 000678 1787832482.900630 write 0x009d79 thru 0x010fee (0x7276 bytes) 000679 1787832482.905943 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b4e2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b4e2): Mirror 2: 0x8ff13a6 (stale) CRC-32 checksum value for chunk [0, 0x3b4e2): Mirror 1: 0xf3ff0dc6 Mirror 3: 0xf3ff0dc6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b4e2: skipped 000680 1787832483.980986 trunc from 0x00e800 to 0x016044 (0x7845 bytes) 000681 1787832484.028673 trunc from 0x03b4e2 to 0x0149e3 (0x26afe bytes) 000682 1787832484.062602 fallocate 0x00973d thru 0x01677a (0xd03e bytes) 000683 1787832484.070686 trunc from 0x01677b to 0x019e4b (0x36d1 bytes) 000684 1787832484.111611 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x19e4c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19e4c): Mirror 2: 0xc56edc3f (stale) Mirror 3: 0x96a03f53 (stale) CRC-32 checksum value for chunk [0, 0x19e4c): Mirror 1: 0x62e20d47 Mirror 4: 0x62e20d47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19e4c: skipped 000685 1787832485.148142 punch from 0x0118f8 to 0x017a95 (0x619e bytes) 000686 1787832485.193903 read 0x00878a thru 0x00c9b3 (0x422a bytes) 000687 1787832485.202523 punch from 0x00fd71 to 0x019e4b (0xa0db bytes) 000688 1787832485.208483 mapread 0x016ebd thru 0x019e4b (0x2f8f bytes) 000689 1787832485.240536 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x19e4c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19e4c): Mirror 2: 0xc56edc3f (stale) Mirror 3: 0x96a03f53 (stale) Mirror 4: 0x62e20d47 (stale) CRC-32 checksum value for chunk [0, 0x19e4c): Mirror 1: 0x62e20d47 Mirror 5: 0x62e20d47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19e4c: skipped 000690 1787832486.220301 mapwrite 0x00f218 thru 0x01347b (0x4264 bytes) 000692 1787832486.412362 mapwrite 0x01099b thru 0x0189b8 (0x801e bytes) 000693 1787832486.456142 mapread 0x010146 thru 0x0130af (0x2f6a bytes) 000694 1787832486.458492 punch from 0x00904b to 0x009347 (0x2fd bytes) 000695 1787832486.471416 punch from 0x014a06 to 0x019e4b (0x5446 bytes) 000696 1787832486.515164 mapwrite 0x0116f0 thru 0x0118d7 (0x1e8 bytes) 000697 1787832486.551824 mapwrite 0x027fb1 thru 0x02c443 (0x4493 bytes) 000698 1787832486.691825 trunc from 0x01bcab to 0x025cc6 (0xa01c bytes) 000699 1787832486.709135 MIRROR_RESYNC 5 mirrors 000701 1787832488.451037 read 0x017b9b thru 0x01ed3d (0x71a3 bytes) 000702 1787832488.480172 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) 000704 1787832488.890507 read 0x02a15e thru 0x02c443 (0x22e6 bytes) 000705 1787832488.920747 trunc from 0x02c444 to 0x036274 (0x9e31 bytes) 000706 1787832489.042133 mapwrite 0x037355 thru 0x03eabe (0x776a bytes) 000707 1787832489.144123 read 0x033ea9 thru 0x034c91 (0xde9 bytes) 000708 1787832489.146054 read 0x03403d thru 0x03509e (0x1062 bytes) 000709 1787832489.154556 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eabf): Mirror 2: 0xb2bf4c23 (stale) Mirror 4: 0xb2bf4c23 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eabf): Mirror 2: 0xb2bf4c23 (stale) Mirror 4: 0xb2bf4c23 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eabf: skipped 000710 1787832489.568071 punch from 0x037653 to 0x03a796 (0x3144 bytes) 000711 1787832489.573479 mapwrite 0x027bdd thru 0x02f3c0 (0x77e4 bytes) 000712 1787832489.658440 punch from 0x02231c to 0x022f16 (0xbfb bytes) 000713 1787832489.664535 fallocate 0x026693 thru 0x02b9fd (0x536b bytes) 000714 1787832489.671753 write 0x038d03 thru 0x03cb61 (0x3e5f bytes) 000715 1787832489.698423 punch from 0x037318 to 0x038ffe (0x1ce7 bytes) 000716 1787832489.722804 mapwrite 0x03d2c4 thru 0x03e8cc (0x1609 bytes) 000717 1787832489.751538 fallocate 0x00ad54 thru 0x01070a (0x59b7 bytes) 000718 1787832489.768690 read 0x0268ea thru 0x0287a1 (0x1eb8 bytes) 000719 1787832489.771576 trunc from 0x03eabf to 0x02d4e5 (0x115d9 bytes) 000720 1787832489.803612 trunc from 0x017096 to 0x01c03d (0x4fa8 bytes) 000721 1787832489.812610 punch from 0x00d3f8 to 0x015427 (0x8030 bytes) 000722 1787832489.827432 read 0x026f24 thru 0x02d4e5 (0x65c2 bytes) 000723 1787832489.828206 trunc from 0x02d4e6 to 0x02ce5c (0x689 bytes) 000725 1787832489.865770 punch from 0x00078b to 0x0056c9 (0x4f3f bytes) 000726 1787832489.878153 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ce5d): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ce5d): Mirror 2: 0x49f6e67 (stale) Mirror 4: 0x49f6e67 (stale) CRC-32 checksum value for chunk [0, 0x2ce5d): Mirror 1: 0x6e8a6c12 Mirror 5: 0x6e8a6c12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ce5d: skipped 000727 1787832490.821861 fallocate 0x0264c6 thru 0x02c5f0 (0x612b bytes) 000728 1787832490.894044 trunc from 0x02ce5d to 0x0173aa (0x15ab2 bytes) 000729 1787832490.942486 fallocate 0x039bae thru 0x03b2fa (0x174d bytes) 000730 1787832490.951496 trunc from 0x001e89 to 0x010728 (0xe8a0 bytes) 000731 1787832490.958029 trunc from 0x00c8e1 to 0x00e0b5 (0x17d5 bytes) 000732 1787832490.973576 MIRROR_RESYNC 4 mirrors 000733 1787832492.307383 trunc from 0x016653 to 0x0173aa (0xd58 bytes) 000734 1787832492.355168 punch from 0x0164e2 to 0x0173aa (0xec9 bytes) 000735 1787832492.366019 write 0x0164bb thru 0x024883 (0xe3c9 bytes) 000736 1787832492.375360 trunc from 0x0165fb to 0x01879d (0x21a3 bytes) 000737 1787832492.391658 write 0x015476 thru 0x01a7a8 (0x5333 bytes) 000738 1787832492.397276 mapwrite 0x03583b thru 0x03ffff (0xa7c5 bytes) 000739 1787832492.455887 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x925d006f (stale) Mirror 4: 0x925d006f (stale) Mirror 5: 0x925d006f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8496d1b Mirror 6: 0x8496d1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000740 1787832493.400098 read 0x0242bb thru 0x0321cd (0xdf13 bytes) 000741 1787832493.414135 punch from 0x03463f to 0x0363e5 (0x1da7 bytes) 000742 1787832493.482182 fallocate 0x03a152 thru 0x03d6ff (0x35ae bytes) 000744 1787832493.498814 trunc from 0x040000 to 0x014219 (0x2bde6 bytes) 000745 1787832493.583156 write 0x01d62d thru 0x02c3d9 (0xedad bytes) 000746 1787832493.584979 write 0x00e177 thru 0x00f666 (0x14f0 bytes) 000747 1787832493.603013 fallocate 0x03f3af thru 0x03ffff (0xc51 bytes) 000748 1787832493.611583 trunc from 0x040000 to 0x01db55 (0x224aa bytes) 000749 1787832493.655062 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1db56): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1db56): Mirror 2: 0xdc4fa7d2 (stale) Mirror 4: 0xdc4fa7d2 (stale) Mirror 5: 0xdc4fa7d2 (stale) Mirror 6: 0xc0f3e156 (stale) CRC-32 checksum value for chunk [0, 0x1db56): Mirror 1: 0x67f39c40 Mirror 7: 0x67f39c40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1db56: skipped 000750 1787832494.685463 punch from 0x00ebac to 0x015199 (0x65ee bytes) 000751 1787832494.722887 mapread 0x000815 thru 0x001bd5 (0x13c1 bytes) 000752 1787832494.732840 write 0x03ed18 thru 0x03ffff (0x12e8 bytes) 000753 1787832494.738196 mapwrite 0x029518 thru 0x02e78d (0x5276 bytes) 000754 1787832494.763809 mapread 0x03138e thru 0x0354cd (0x4140 bytes) 000755 1787832494.773596 fallocate 0x02e954 thru 0x03c1d1 (0xd87e bytes) 000756 1787832494.778499 write 0x01cb3a thru 0x022291 (0x5758 bytes) 000757 1787832494.788422 write 0x024590 thru 0x032fe7 (0xea58 bytes) 000758 1787832494.790568 mapread 0x030f64 thru 0x03574a (0x47e7 bytes) 000759 1787832494.792116 mapwrite 0x025ac4 thru 0x02bf14 (0x6451 bytes) 000760 1787832494.819990 trunc from 0x0038de to 0x0050e3 (0x1806 bytes) 000761 1787832494.829028 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x925d006f (stale) Mirror 4: 0x925d006f (stale) Mirror 5: 0x925d006f (stale) Mirror 7: 0x72f1c7f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x925d006f (stale) Mirror 4: 0x925d006f (stale) Mirror 5: 0x925d006f (stale) Mirror 7: 0x72f1c7f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000763 1787832495.240863 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x925d006f (stale) Mirror 5: 0x925d006f (stale) Mirror 7: 0x72f1c7f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x925d006f (stale) Mirror 5: 0x925d006f (stale) Mirror 7: 0x72f1c7f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000764 1787832495.833457 mapwrite 0x036d06 thru 0x03ffff (0x92fa bytes) 000765 1787832495.868373 mapread 0x020dda thru 0x02adc3 (0x9fea bytes) 000766 1787832495.888205 MIRROR_RESYNC 4 mirrors 000767 1787832497.151755 trunc from 0x040000 to 0x00aeee (0x35111 bytes) 000768 1787832497.231905 mapread 0x000c0f thru 0x00aeee (0xa2e0 bytes) 000769 1787832497.279699 write 0x036e49 thru 0x03ffff (0x91b7 bytes) 000770 1787832497.281177 fallocate 0x02be24 thru 0x02ee29 (0x3006 bytes) 000771 1787832497.291203 write 0x026d82 thru 0x0356e3 (0xe962 bytes) 000772 1787832497.313501 trunc from 0x040000 to 0x03bc85 (0x437a bytes) 000773 1787832497.341628 punch from 0x032a7d to 0x03aeea (0x846e bytes) 000774 1787832497.370509 read 0x02f244 thru 0x03bc85 (0xca42 bytes) 000775 1787832497.385546 mapwrite 0x02ad78 thru 0x03528c (0xa515 bytes) 000777 1787832497.419316 trunc from 0x021939 to 0x0264e7 (0x4baf bytes) 000779 1787832497.428506 trunc from 0x03bc86 to 0x011a98 (0x2a1ed bytes) 000780 1787832497.473091 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x11a99): Mirror 5: 0xa78853a8 (stale) Mirror 7: 0xa78853a8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11a99): Mirror 5: 0xa78853a8 (stale) Mirror 7: 0xa78853a8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11a99: skipped 000781 1787832497.947539 trunc from 0x001ba2 to 0x010e8a (0xf2e9 bytes) 000782 1787832497.954687 trunc from 0x009f92 to 0x011a98 (0x7b07 bytes) 000783 1787832497.958950 mapwrite 0x02885c thru 0x036ce0 (0xe485 bytes) 000784 1787832498.036408 write 0x0058ff thru 0x00cd35 (0x7437 bytes) 000785 1787832498.056790 punch from 0x008f02 to 0x0133ce (0xa4cd bytes) 000786 1787832498.083351 trunc from 0x036ce1 to 0x00f85d (0x27483 bytes) 000787 1787832498.123261 trunc from 0x006862 to 0x00f85d (0x8ffc bytes) 000788 1787832498.135393 punch from 0x00f122 to 0x00f85d (0x73c bytes) 000790 1787832498.149108 trunc from 0x00f85e to 0x03f3a0 (0x2fb43 bytes) 000791 1787832498.174392 trunc from 0x0319b0 to 0x03d8df (0xbf30 bytes) 000792 1787832498.179305 trunc from 0x01aeff to 0x020f4a (0x604c bytes) 000793 1787832498.187736 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f3a1): Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f3a1): Mirror 5: 0x213f568c (stale) Mirror 7: 0x213f568c (stale) CRC-32 checksum value for chunk [0, 0x3f3a1): Mirror 1: 0x2a05f731 Mirror 8: 0x2a05f731 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f3a1: skipped 000795 1787832499.142068 read 0x035359 thru 0x03f3a0 (0xa048 bytes) 000796 1787832499.150990 mapwrite 0x02aa27 thru 0x031c25 (0x71ff bytes) 000797 1787832499.283439 read 0x032537 thru 0x03f3a0 (0xce6a bytes) 000798 1787832499.312535 write 0x00c19d thru 0x00d3a0 (0x1204 bytes) 000800 1787832499.323442 mapwrite 0x029e83 thru 0x030b59 (0x6cd7 bytes) 000801 1787832499.353415 mapwrite 0x0282f2 thru 0x032452 (0xa161 bytes) 000803 1787832499.404036 trunc from 0x03f3a1 to 0x019797 (0x25c09 bytes) 000804 1787832499.427251 trunc from 0x01141c to 0x013ce5 (0x28ca bytes) 000805 1787832499.432880 write 0x02a5df thru 0x0379c6 (0xd3e8 bytes) 000806 1787832499.435788 read 0x0308d9 thru 0x0379c6 (0x70ee bytes) 000807 1787832499.441080 MIRROR_RESYNC 4 mirrors 000808 1787832500.672155 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) 000809 1787832501.004648 punch from 0x015ccb to 0x01890f (0x2c45 bytes) 000810 1787832501.078494 mapread 0x00bcee thru 0x010c66 (0x4f79 bytes) 000811 1787832501.106130 trunc from 0x017407 to 0x026f7d (0xfb77 bytes) 000812 1787832501.111874 fallocate 0x039d13 thru 0x03ffff (0x62ed bytes) 000813 1787832501.120866 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x379c7): Mirror 8: 0xa884eb2f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x379c7): Mirror 8: 0xa884eb2f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x379c7: skipped 000815 1787832501.416900 mapwrite 0x0004aa thru 0x00f211 (0xed68 bytes) 000816 1787832501.457658 mapread 0x03701d thru 0x0379c6 (0x9aa bytes) 000817 1787832501.469769 mapread 0x02d1e1 thru 0x02f967 (0x2787 bytes) 000818 1787832501.484760 mapread 0x02d4e0 thru 0x0333d8 (0x5ef9 bytes) 000819 1787832501.488857 read 0x0215f3 thru 0x02795d (0x636b bytes) 000820 1787832501.496146 write 0x039391 thru 0x03e3b9 (0x5029 bytes) 000821 1787832501.501478 trunc from 0x00586d to 0x00efe6 (0x977a bytes) 000822 1787832501.508519 mapwrite 0x008abf thru 0x01427f (0xb7c1 bytes) 000823 1787832501.558249 write 0x022f24 thru 0x02dcb2 (0xad8f bytes) 000824 1787832501.564196 trunc from 0x024453 to 0x02c8c6 (0x8474 bytes) 000825 1787832501.613064 fallocate 0x018ad8 thru 0x018ccd (0x1f6 bytes) 000826 1787832501.619516 fallocate 0x018883 thru 0x027c0d (0xf38b bytes) 000827 1787832501.629265 mapwrite 0x01cff5 thru 0x026734 (0x9740 bytes) 000829 1787832501.676374 trunc from 0x003262 to 0x0121bc (0xef5b bytes) 000830 1787832501.695182 mapread 0x026535 thru 0x028bf9 (0x26c5 bytes) 000831 1787832501.696126 trunc from 0x03e3ba to 0x03fed7 (0x1b1e bytes) 000832 1787832501.737889 write 0x03585f thru 0x03709c (0x183e bytes) 000833 1787832501.747006 MIRROR_RESYNC 2 mirrors 000834 1787832502.480503 mapread 0x0186e6 thru 0x025b94 (0xd4af bytes) 000835 1787832502.511951 punch from 0x01de30 to 0x02a766 (0xc937 bytes) 000836 1787832502.549562 trunc from 0x007a06 to 0x012cb9 (0xb2b4 bytes) 000837 1787832502.558955 punch from 0x02c5ea to 0x032205 (0x5c1c bytes) 000838 1787832502.563528 mapread 0x0192e6 thru 0x02871e (0xf439 bytes) 000839 1787832502.582121 punch from 0x03adae to 0x03f6aa (0x48fd bytes) 000840 1787832502.589346 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) 000841 1787832502.728799 trunc from 0x026223 to 0x032ad3 (0xc8b1 bytes) 000843 1787832502.735443 trunc from 0x03fed8 to 0x01f4a5 (0x20a32 bytes) 000844 1787832502.755119 write 0x01a262 thru 0x01a38b (0x12a bytes) 000845 1787832502.761010 mapread 0x00e4c1 thru 0x012272 (0x3db2 bytes) 000846 1787832502.779815 mapread 0x013fd4 thru 0x01af57 (0x6f84 bytes) 000847 1787832502.789398 write 0x010213 thru 0x01feac (0xfc9a bytes) 000849 1787832502.791606 mapwrite 0x00af36 thru 0x01a459 (0xf524 bytes) 000850 1787832502.814187 mapread 0x01174a thru 0x01da3f (0xc2f6 bytes) 000851 1787832502.816441 read 0x01d30a thru 0x01feac (0x2ba3 bytes) 000852 1787832502.817398 punch from 0x0106de to 0x0111a4 (0xac7 bytes) 000853 1787832502.822402 trunc from 0x01fead to 0x037110 (0x17264 bytes) 000854 1787832502.842058 mapwrite 0x00c2c1 thru 0x00cb3b (0x87b bytes) 000856 1787832502.854029 mapwrite 0x01cd48 thru 0x029f4e (0xd207 bytes) 000857 1787832502.899049 mapwrite 0x027211 thru 0x02b69c (0x448c bytes) 000859 1787832502.913498 write 0x03a12b thru 0x03ffff (0x5ed5 bytes) 000860 1787832502.915738 mapread 0x03769e thru 0x03e75a (0x70bd bytes) 000862 1787832502.927811 mapwrite 0x03369d thru 0x034d93 (0x16f7 bytes) 000863 1787832502.954500 MIRROR_EXTEND to 2 mirrors 000864 1787832503.971744 mapwrite 0x03078f thru 0x03e7e9 (0xe05b bytes) 000865 1787832504.050351 trunc from 0x01e669 to 0x02ab9e (0xc536 bytes) 000866 1787832504.056340 fallocate 0x01c818 thru 0x022c86 (0x646f bytes) 000867 1787832504.061987 fallocate 0x006c07 thru 0x00cf81 (0x637b bytes) 000868 1787832504.065058 mapwrite 0x0099ab thru 0x01482c (0xae82 bytes) 000869 1787832504.113260 read 0x02aef0 thru 0x035af3 (0xac04 bytes) 000870 1787832504.119205 read 0x0090bb thru 0x00dc79 (0x4bbf bytes) 000871 1787832504.119804 mapread 0x00592a thru 0x00784e (0x1f25 bytes) 000872 1787832504.120627 punch from 0x00c3cb to 0x0134f7 (0x712d bytes) 000873 1787832504.131489 fallocate 0x01d366 thru 0x022712 (0x53ad bytes) 000874 1787832504.139369 mapread 0x00547d thru 0x013c6e (0xe7f2 bytes) 000875 1787832504.157858 read 0x011226 thru 0x011de0 (0xbbb bytes) 000876 1787832504.158498 read 0x01a818 thru 0x025a3e (0xb227 bytes) 000877 1787832504.161924 trunc from 0x040000 to 0x03e1e6 (0x1e19 bytes) 000878 1787832504.205900 trunc from 0x03e1e7 to 0x038494 (0x5d52 bytes) 000879 1787832504.224980 trunc from 0x009d43 to 0x013bb5 (0x9e73 bytes) 000880 1787832504.234655 fallocate 0x012858 thru 0x021d21 (0xf4ca bytes) 000881 1787832504.238680 mapwrite 0x02b7ef thru 0x033f40 (0x8752 bytes) 000882 1787832504.260684 trunc from 0x038495 to 0x023cd6 (0x147be bytes) 000883 1787832504.302093 trunc from 0x008bc3 to 0x015819 (0xcc57 bytes) 000884 1787832504.311013 trunc from 0x0057db to 0x012bc4 (0xd3ea bytes) 000885 1787832504.326962 write 0x02ea0a thru 0x03d710 (0xed07 bytes) 000886 1787832504.332882 punch from 0x0395d4 to 0x03d710 (0x413d bytes) 000887 1787832504.349786 read 0x030447 thru 0x03c373 (0xbf2d bytes) 000888 1787832504.363660 MIRROR_RESYNC 2 mirrors 000889 1787832504.956782 write 0x0073b7 thru 0x00deb3 (0x6afd bytes) 000890 1787832504.992855 read 0x009187 thru 0x00a1df (0x1059 bytes) 000891 1787832504.994210 punch from 0x03ba25 to 0x03d710 (0x1cec bytes) 000894 1787832505.000811 trunc from 0x024e66 to 0x033fd9 (0xf174 bytes) 000895 1787832505.008322 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d711): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d711): Mirror 2: 0xf5857512 (stale) CRC-32 checksum value for chunk [0, 0x3d711): Mirror 1: 0xf695aaad Mirror 3: 0xf695aaad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d711: skipped 000896 1787832506.133135 write 0x03b4a6 thru 0x03ffff (0x4b5a bytes) 000897 1787832506.186862 mapread 0x018496 thru 0x019eeb (0x1a56 bytes) 000898 1787832506.198366 punch from 0x031efa to 0x0348d7 (0x29de bytes) 000899 1787832506.204733 trunc from 0x00c457 to 0x0118d0 (0x547a bytes) 000900 1787832506.212945 trunc from 0x040000 to 0x0151a1 (0x2ae5e bytes) 000901 1787832506.244902 punch from 0x002305 to 0x004fe3 (0x2cdf bytes) 000903 1787832506.253705 write 0x01901e thru 0x01eff3 (0x5fd6 bytes) 000904 1787832506.259937 write 0x020d8a thru 0x02f05e (0xe2d5 bytes) 000905 1787832506.266869 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f05f): Mirror 3: 0xc4cd9ea5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f05f): Mirror 3: 0xc4cd9ea5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f05f: skipped 000906 1787832506.590727 mapread 0x023ef6 thru 0x02445a (0x565 bytes) 000907 1787832506.605881 write 0x01cab0 thru 0x01d947 (0xe98 bytes) 000908 1787832506.631103 read 0x010505 thru 0x0181e9 (0x7ce5 bytes) 000909 1787832506.643079 punch from 0x01d86a to 0x0233c6 (0x5b5d bytes) 000910 1787832506.656596 mapread 0x00e168 thru 0x017fbb (0x9e54 bytes) 000911 1787832506.670629 punch from 0x01927e to 0x02148d (0x8210 bytes) 000912 1787832506.679606 mapread 0x021a2f thru 0x02758e (0x5b60 bytes) 000913 1787832506.691923 mapwrite 0x024b19 thru 0x028256 (0x373e bytes) 000914 1787832506.713938 trunc from 0x02dc1f to 0x02f05e (0x1440 bytes) 000916 1787832506.724718 punch from 0x02a19d to 0x02f05e (0x4ec2 bytes) 000917 1787832506.731201 fallocate 0x01eeb7 thru 0x02e5a0 (0xf6ea bytes) 000918 1787832506.745175 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2f05f): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f05f): Mirror 3: 0xc4cd9ea5 (stale) CRC-32 checksum value for chunk [0, 0x2f05f): Mirror 1: 0x46244945 Mirror 4: 0x46244945 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f05f: skipped 000919 1787832507.982133 trunc from 0x024d4e to 0x028e22 (0x40d5 bytes) 000920 1787832508.023131 trunc from 0x022317 to 0x02f05e (0xcd48 bytes) 000921 1787832508.032114 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2f05f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f05f): Mirror 3: 0xc4cd9ea5 (stale) Mirror 4: 0x46244945 (stale) CRC-32 checksum value for chunk [0, 0x2f05f): Mirror 1: 0xd166d659 Mirror 5: 0xd166d659 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f05f: skipped 000922 1787832508.939639 mapwrite 0x007611 thru 0x00be91 (0x4881 bytes) 000923 1787832509.014259 trunc from 0x02f05f to 0x02f49f (0x441 bytes) 000924 1787832509.039986 trunc from 0x02f4a0 to 0x03007a (0xbdb bytes) 000925 1787832509.088956 punch from 0x003d38 to 0x0124ff (0xe7c8 bytes) 000926 1787832509.099985 fallocate 0x02d533 thru 0x03238b (0x4e59 bytes) 000927 1787832509.109846 punch from 0x02504b to 0x02edbf (0x9d75 bytes) 000928 1787832509.120042 write 0x003f68 thru 0x00857e (0x4617 bytes) 000929 1787832509.132804 write 0x019e6e thru 0x021a65 (0x7bf8 bytes) 000930 1787832509.150951 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x168:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3007b): Mirror 3: 0x168e8fbc (stale) Mirror 5: 0x812d70e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3007b): Mirror 3: 0x168e8fbc (stale) Mirror 5: 0x812d70e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3007b: skipped 000931 1787832509.524468 trunc from 0x017284 to 0x017d1f (0xa9c bytes) 000932 1787832509.530931 mapread 0x02fb00 thru 0x03007a (0x57b bytes) 000933 1787832509.541145 read 0x02f256 thru 0x03007a (0xe25 bytes) 000934 1787832509.546144 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3007b): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3007b): Mirror 3: 0x168e8fbc (stale) Mirror 5: 0x812d70e9 (stale) CRC-32 checksum value for chunk [0, 0x3007b): Mirror 1: 0xe7e417fa Mirror 6: 0xe7e417fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3007b: skipped 000935 1787832510.284007 mapwrite 0x03affc thru 0x03ffff (0x5004 bytes) 000936 1787832510.394797 punch from 0x007c98 to 0x012fd0 (0xb339 bytes) 000937 1787832510.400982 write 0x02c922 thru 0x031dda (0x54b9 bytes) 000938 1787832510.411393 punch from 0x03b493 to 0x03ffff (0x4b6d bytes) 000939 1787832510.417433 write 0x002528 thru 0x007f9b (0x5a74 bytes) 000940 1787832510.430183 punch from 0x007ab8 to 0x0166f9 (0xec42 bytes) 000941 1787832510.443826 mapwrite 0x005aee thru 0x012a3d (0xcf50 bytes) 000942 1787832510.492290 trunc from 0x015e36 to 0x01931f (0x34ea bytes) 000943 1787832510.497829 trunc from 0x040000 to 0x02ba5a (0x145a5 bytes) 000944 1787832510.517830 trunc from 0x02ba5b to 0x0326b7 (0x6c5d bytes) 000945 1787832510.530360 trunc from 0x00679f to 0x00a19d (0x39ff bytes) 000946 1787832510.535751 punch from 0x01a84a to 0x021fb4 (0x776b bytes) 000947 1787832510.540404 punch from 0x02dd09 to 0x0301b1 (0x24a9 bytes) 000948 1787832510.544698 punch from 0x01ad3f to 0x028526 (0xd7e8 bytes) 000950 1787832510.549960 mapwrite 0x010591 thru 0x011ef4 (0x1964 bytes) 000951 1787832510.561858 trunc from 0x0326b8 to 0x0360d2 (0x3a1b bytes) 000952 1787832510.579727 write 0x02b927 thru 0x0329f8 (0x70d2 bytes) 000953 1787832510.589198 trunc from 0x027bce to 0x02d5b6 (0x59e9 bytes) 000954 1787832510.601953 trunc from 0x03482c to 0x0360d2 (0x18a7 bytes) 000957 1787832510.607080 write 0x02eb57 thru 0x02f063 (0x50d bytes) 000958 1787832510.608825 fallocate 0x0353b0 thru 0x03ffff (0xac50 bytes) 000959 1787832510.616258 trunc from 0x019bb0 to 0x025ab1 (0xbf02 bytes) 000960 1787832510.628363 read 0x031ea3 thru 0x0360d2 (0x4230 bytes) 000961 1787832510.637125 read 0x02f521 thru 0x03162e (0x210e bytes) 000962 1787832510.638414 punch from 0x01538b to 0x0173d9 (0x204f bytes) 000963 1787832510.643309 read 0x01ca62 thru 0x02776e (0xad0d bytes) 000965 1787832510.659989 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x360d3): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x360d3): Mirror 3: 0xa4d4f9af (stale) Mirror 5: 0x2bf74120 (stale) Mirror 6: 0x6e583e2c (stale) CRC-32 checksum value for chunk [0, 0x360d3): Mirror 1: 0x96186060 Mirror 7: 0x96186060 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x360d3: skipped 000966 1787832511.859926 punch from 0x0106ca to 0x011208 (0xb3f bytes) 000967 1787832511.903441 mapwrite 0x027b6a thru 0x037b32 (0xffc9 bytes) 000968 1787832511.985749 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x37b33): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37b33): Mirror 3: 0x50e5347f (stale) Mirror 5: 0x7385c96 (stale) Mirror 6: 0xd5f1fa02 (stale) Mirror 7: 0x1af761c5 (stale) CRC-32 checksum value for chunk [0, 0x37b33): Mirror 1: 0x7a041a2f Mirror 8: 0x7a041a2f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37b33: skipped 000969 1787832513.156894 mapwrite 0x001b34 thru 0x004665 (0x2b32 bytes) 000970 1787832513.229352 fallocate 0x0157e4 thru 0x0196ee (0x3f0b bytes) 000972 1787832513.234158 trunc from 0x016f3e to 0x018b58 (0x1c1b bytes) 000974 1787832513.238746 fallocate 0x005869 thru 0x008cb7 (0x344f bytes) 000975 1787832513.241985 read 0x0136a4 thru 0x016085 (0x29e2 bytes) 000976 1787832513.246608 trunc from 0x037b33 to 0x0291f6 (0xe93c bytes) 000977 1787832513.261853 mapread 0x01edd7 thru 0x029172 (0xa39c bytes) 000978 1787832513.271403 trunc from 0x0291f7 to 0x01fb85 (0x9671 bytes) 000979 1787832513.290802 write 0x0390d3 thru 0x03ffff (0x6f2d bytes) 000982 1787832513.295111 mapwrite 0x037572 thru 0x03b9c9 (0x4458 bytes) 000983 1787832513.323718 read 0x03dea6 thru 0x03ffff (0x215a bytes) 000984 1787832513.324124 read 0x01813c thru 0x022912 (0xa7d7 bytes) 000985 1787832513.334596 trunc from 0x040000 to 0x004f48 (0x3b0b7 bytes) 000986 1787832513.344069 trunc from 0x004f49 to 0x008b5f (0x3c17 bytes) 000988 1787832513.356420 trunc from 0x007afe to 0x008b5f (0x1062 bytes) 000989 1787832513.360967 fallocate 0x02fede thru 0x033550 (0x3673 bytes) 000991 1787832513.366641 punch from 0x001a9f to 0x008b5f (0x70c1 bytes) 000993 1787832513.372722 read 0x0064e7 thru 0x008b5f (0x2679 bytes) 000994 1787832513.380332 write 0x025455 thru 0x02f09d (0x9c49 bytes) 000995 1787832513.381859 trunc from 0x02f09e to 0x01b2a7 (0x13df6 bytes) 000996 1787832513.396199 read 0x015c01 thru 0x01664d (0xa4d bytes) 000997 1787832513.405982 MIRROR_RESYNC 6 mirrors 000998 1787832514.391819 trunc from 0x002dc2 to 0x00aea8 (0x80e7 bytes) 000999 1787832514.434139 mapread 0x007bed thru 0x00e93f (0x6d53 bytes) 001000 1787832514.449163 fallocate 0x031b39 thru 0x034201 (0x26c9 bytes) All operations completed A-OK! PASS 70a (83s) == sanity-flr test 71: check mirror extend parent fid ==== 08:08:38 (1787832518) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1a8:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 08:08:49 (1787832529) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x108:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12393 s, 16.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xeb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x108:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875516 s, 12.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xeb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x109:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x108:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xeb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x109:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x108:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 08:08:59 (1787832539) Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre2 Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5545408 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6232000 .. Extending file size to 7055872 .. Extending file size to 8544992 .. Extending file size to 6372352 .. Extending file size to 6784416 .. Extending file size to 8583488 .. Extending file size to 8719520 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8653120 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62472 62473 62475 62478 62480 done 192.168.202.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg212-client.virtnet /mnt/lustre2 (opts:) 192.168.202.112@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg212-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 08:10:22 (1787832622) Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre2 Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2522944 .. Extending file size to 7413280 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6279360 .. Extending file size to 7036096 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7820928 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8133184 .. Extending file size to 8062720 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8187744 .. Extending file size to 9081248 .. Extending file size to 8354752 .. Extending file size to 9143296 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 64067 64068 64069 64071 64073 done 192.168.202.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg212-client.virtnet /mnt/lustre2 (opts:) 192.168.202.112@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg212-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 08:11:44 (1787832704) Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2284 1269020 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2124 1269180 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17800 3589220 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25756 3581264 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43556 7170484 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2284 1269020 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2124 1269180 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17800 3589220 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25756 3581264 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43556 7170484 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111435 s, 18.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.202296 s, 10.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x5310ceee (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.202.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg212-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 08:12:05 (1787832725) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.110635 s, 19.0 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 08:12:13 (1787832733) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x36:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x125:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x126:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x107:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 08:12:22 (1787832742) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0621786 s, 33.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 08:12:31 (1787832751) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141401 s, 14.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x108:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x127:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x128:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x108:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x109:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 08:12:40 (1787832760) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0849821 s, 24.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d2:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d6:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d7:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 08:13:15 (1787832795) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg212-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144968 s, 7.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d3:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d3:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d4:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 08:14:11 (1787832851) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg212-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.147296 s, 14.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (53s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 08:15:04 (1787832904) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg212-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg212-server: Pool lustre.test_204f created Adding targets to pool oleg212-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg212-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.148571 s, 14.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x10b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg212-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg212-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg212-server: Pool lustre.test_204f destroyed PASS 204f (71s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 08:16:15 (1787832975) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 08:16:23 (1787832983) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 08:16:33 (1787832993) oleg212-server: Pool lustre.test_206 created Adding targets to pool oleg212-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg212-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg212-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg212-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg212-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 08:17:03 (1787833023) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746847 s, 14.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356981 s, 29.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x6facc6c0 Mirror 3: 0x6facc6c0 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 08:17:16 (1787833036) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 08:17:21 (1787833041) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 08:17:26 (1787833046) Starting client: oleg212-client.virtnet: -o user_xattr,flock 192.168.202.112@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0691592 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x8d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1200000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1200000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1200000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1ae0000 Read this (4 bytes): BBBB 192.168.202.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg212-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 08:18:09 (1787833089) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0288091 s, 284 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 08:18:18 (1787833098) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585907 s, 17.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x55 lmm_fid: [0x200000405:0x55:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x55 lmm_fid: [0x200000405:0x55:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611221 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 08:18:20.000000000 -0400 Modify: 2026-08-27 08:18:20.000000000 -0400 Change: 2026-08-27 08:18:21.000000000 -0400 Birth: 2026-08-27 08:18:20.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 08:18:26 (1787833106) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118065 s, 8.9 MB/s oleg212-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg212-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg212-server: O/280000401/d22/136: File not found by ext2_lookup PASS 210b (25s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 08:18:51 (1787833131) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.234821 s, 174 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0218548 s, 187 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 08:18 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5a:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 08:18 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 08:19:01 (1787833141) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 08:19:11 (1787833151) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1835 sec =========== 08:19:22 (1787833162) === sanity-flr: start cleanup 08:19:24 (1787833164) === === sanity-flr: finish cleanup 08:19:33 (1787833173) ===