-----============= acceptance-small: sanity-flr ============----- Mon Aug 24 04:35:07 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg354-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 04:35:26 (1787560526) === oleg354-client.virtnet: executing check_config_client /mnt/lustre oleg354-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg354-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9d9546574000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9d9546574000.idle_timeout=debug disable quota as required oleg354-server: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 04:36:01 (1787560561) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7641] == sanity-flr test 0a: lfs mirror create with -N option == 04:36:04 (1787560564) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:36:20 (1787560580) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:36:23 (1787560583) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:36:27 (1787560587) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:36:51 (1787560611) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:36:54 (1787560614) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 04:36:57 (1787560617) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9d9546574000.stats=clear osc.lustre-OST0001-osc-ffff9d9546574000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:37:04 (1787560624) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:37:15 (1787560635) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:37:23 (1787560643) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 04:37:27 (1787560647) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:37:35 (1787560655) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 04:37:44 (1787560664) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:37:50 (1787560670) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:37:54 (1787560674) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:38:02 (1787560682) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:38:15 (1787560695) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:38:21 (1787560701) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.24644 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149647 s, 7.0 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:38:32 (1787560712) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304504 s, 34.4 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 04:38:40 (1787560720) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:38:51 (1787560731) Stopping /mnt/lustre-ost1 (opts:) on oleg354-server oleg354-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d9546574000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d9546574000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg354-server: oleg354-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg354-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg354-server oleg354-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9d9546574000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d9546574000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg354-server: oleg354-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg354-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (77s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:40:08 (1787560808) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.488875 s, 12.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg354-server oleg354-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d9546574000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d9546574000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 6291456 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg354-server: oleg354-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg354-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:40:54 (1787560854) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg354-server oleg354-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d9546574000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d9546574000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg354-server: oleg354-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg354-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg354-server oleg354-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9d9546574000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d9546574000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg354-server: oleg354-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg354-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (77s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:42:11 (1787560931) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.61603 s, 11.4 MB/s -rw-r--r-- 1 root root 52428800 Aug 24 04:42 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.915s user 0m0.000s sys 0m0.187s Stopping /mnt/lustre-ost1 (opts:) on oleg354-server oleg354-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9d9546574000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9d9546574000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.031s user 0m0.001s sys 0m0.175s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg354-server: oleg354-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg354-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg354-server oleg354-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9d9546574000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d9546574000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.300s user 0m0.000s sys 0m0.191s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg354-server: oleg354-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg354-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg354-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9d9546574000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9d9546574000.ost_server_uuid in FULL state after 0 sec PASS 33b (88s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:43:39 (1787561019) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:43:42 (1787561022) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:43:46 (1787561026) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 04:43:50 (1787561030) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 04:43:56 (1787561036) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.403875 s, 31.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:44:08 (1787561048) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473903 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385265 s, 27.2 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:44:15 (1787561055) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:44:22 (1787561062) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00102392 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00312438 s, 328 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00102305 s, 1.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00142787 s, 717 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 04:44:32 (1787561072) create test files with size 15737119 .. bf1119df764cda78a98c9947c5837ae0 - c8c357830942ffd1bd69c9a148ba921a - d92146d4034cd64807dde44d67c834f9 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg354-client.virtnet /mnt/lustre (opts:) Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 04:44:57 (1787561097) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (17s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:45:15 (1787561115) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:45:22 (1787561122) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119035 s, 8.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318772 s, 32.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 04:45:31 (1787561131) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.16656 s, 12.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.374295 s, 11.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.281764 s, 7.4 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 04:45:49 (1787561149) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 04:45:53 (1787561153) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:45:57 (1787561157) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420792 s, 24.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (12s) == sanity-flr test 44a: lfs mirror split check =========== 04:46:10 (1787561170) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (5s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:46:14 (1787561174) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:46:24 (1787561184) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.271749 s, 38.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg354-client.virtnet /mnt/lustre (opts:) Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:46:35 (1787561195) llite.lustre-ffff9d9551b5e800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.336573 s, 31.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg354-client.virtnet /mnt/lustre (opts:) Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9d9545dfb000.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:46:50 (1787561210) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (35s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:47:25 (1787561245) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:47:58 (1787561278) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 04:48:02 (1787561282) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.209365 s, 15.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.135042 s, 23.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x8ec2d8fb (nosync,stale) Mirror 2: 0x8ec2d8fb (nosync,stale) ** verify mirror 2 doesn't change original checksum: 84ffd6a65f1a7633335ce17d73417200 - mirror 1 checksum: accca3937850a745a16fe16d45464616 - mirror 2 checksum: 84ffd6a65f1a7633335ce17d73417200 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T04:48:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-24T04:48:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 84ffd6a65f1a7633335ce17d73417200 - mirror 1 checksum: accca3937850a745a16fe16d45464616 - mirror 2 checksum: accca3937850a745a16fe16d45464616 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T04:48:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xda:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-24T04:48:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdb:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:48:13 (1787561293) Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 04:48:19 (1787561299) shut down facet: mds1 facet_host: oleg354-server facet_failover_host: oleg354-server Failover mds1 to oleg354-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 04:48:41 (1787561321) targets are mounted 04:48:41 (1787561321) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525046 s, 20.0 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } PASS 50A (47s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:49:00 (1787561340) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:49:04 (1787561344) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:49:08 (1787561348) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0210099 s, 780 kB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:49:15 (1787561355) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:49:19 (1787561359) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0177033 s, 231 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.117235 s, 34.9 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:49:27 (1787561367) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (31s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:49:58 (1787561398) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:50:27 (1787561427) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:50:53 (1787561453) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:51:17 (1787561477) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 04:51:25 (1787561485) mirror create pid 54318 mirror split pid 54320 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54318 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 54320 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 04:52:33 (1787561553) Seed set to 1555 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000004 1787561555.693498 mapwrite 0x02dfc5 thru 0x0379d1 (0x9a0d bytes) 000010 1787561555.799364 mapwrite 0x01f4a0 thru 0x021524 (0x2085 bytes) 000011 1787561555.856711 write 0x015589 thru 0x01f7eb (0xa263 bytes) truncating to largest ever: 0x16508 000013 1787561555.867010 trunc from 0x0379d2 to 0x016507 (0x214ca bytes) 000015 1787561555.886512 mapread 0x00f00a thru 0x010fc0 (0x1fb7 bytes) 000020 1787561555.915857 write 0x007700 thru 0x00c3ef (0x4cf0 bytes) truncating to largest ever: 0x22005 000021 1787561555.931093 trunc from 0x016508 to 0x022004 (0xbafd bytes) 000022 1787561555.959811 trunc from 0x022005 to 0x01b2a0 (0x6d64 bytes) 000025 1787561555.988799 MIRROR_EXTEND to 2 mirrors 000027 1787561556.967974 trunc from 0x01b2a1 to 0x00178e (0x19b12 bytes) 000028 1787561557.142478 read 0x0011d1 thru 0x00178e (0x5be bytes) 000029 1787561557.158314 mapread 0x000766 thru 0x00178e (0x1029 bytes) 000032 1787561557.186156 MIRROR_RESYNC 2 mirrors 000033 1787561558.170584 write 0x01b227 thru 0x020f60 (0x5d3a bytes) 000034 1787561558.233019 mapread 0x001677 thru 0x0072f4 (0x5c7e bytes) 000036 1787561558.256956 read 0x01222b thru 0x020aa2 (0xe878 bytes) 000037 1787561558.274733 read 0x011022 thru 0x01e1d7 (0xd1b6 bytes) 000039 1787561558.287605 mapwrite 0x03460f thru 0x03d1da (0x8bcc bytes) 000040 1787561558.445370 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d1db): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1db): Mirror 2: 0xc68f1bcb (stale) CRC-32 checksum value for chunk [0, 0x3d1db): Mirror 1: 0x2130fc5a Mirror 3: 0x2130fc5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1db: skipped 000041 1787561559.589215 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x2e0c5 000042 1787561560.217299 trunc from 0x03d1db to 0x02e0c4 (0xf116 bytes) 000043 1787561560.316515 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x2ef5d 000044 1787561561.336947 trunc from 0x02e0c5 to 0x02ef5c (0xe98 bytes) 000046 1787561561.400389 mapread 0x006076 thru 0x00d78b (0x7716 bytes) 000047 1787561561.428960 MIRROR_RESYNC 3 mirrors 000048 1787561562.279046 mapwrite 0x00ac10 thru 0x00f09c (0x448d bytes) 000049 1787561562.416217 trunc from 0x02ef5d to 0x01f497 (0xfac5 bytes) 000052 1787561562.453366 mapread 0x0154ce thru 0x01f497 (0x9fca bytes) 000053 1787561562.486599 write 0x024b57 thru 0x030944 (0xbdee bytes) truncating to largest ever: 0x3c82d 000054 1787561562.488146 trunc from 0x030945 to 0x03c82c (0xbee8 bytes) 000057 1787561562.528140 trunc from 0x03c82d to 0x002cd0 (0x39b5c bytes) 000058 1787561562.560475 write 0x03c57d thru 0x03eabb (0x253f bytes) 000060 1787561562.563013 mapwrite 0x00a658 thru 0x0182a5 (0xdc4e bytes) 000062 1787561562.703449 write 0x035368 thru 0x03ffff (0xac98 bytes) 000063 1787561562.714454 read 0x006170 thru 0x00fd5b (0x9bec bytes) 000064 1787561562.725477 MIRROR_RESYNC 3 mirrors 000066 1787561564.681303 read 0x03d799 thru 0x03ffff (0x2867 bytes) 000068 1787561564.698496 mapread 0x0044b7 thru 0x0065b1 (0x20fb bytes) 000075 1787561564.737396 trunc from 0x040000 to 0x013a6a (0x2c595 bytes) 000076 1787561564.889103 trunc from 0x013a6b to 0x0283c9 (0x1495f bytes) 000077 1787561564.950009 write 0x001d40 thru 0x001e64 (0x125 bytes) 000078 1787561564.963957 read 0x01e09d thru 0x0283c9 (0xa32d bytes) 000079 1787561564.998638 mapread 0x009dd9 thru 0x00df4e (0x4176 bytes) 000080 1787561565.026646 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x283ca): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x283ca): Mirror 2: 0xd2109550 (stale) Mirror 3: 0xd2109550 (stale) CRC-32 checksum value for chunk [0, 0x283ca): Mirror 1: 0x1d80b8b0 Mirror 4: 0x1d80b8b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x283ca: skipped 000082 1787561567.056265 write 0x03f174 thru 0x03ffff (0xe8c bytes) 000083 1787561567.262315 MIRROR_RESYNC 4 mirrors 000084 1787561568.933110 mapwrite 0x01a473 thru 0x01db95 (0x3723 bytes) 000085 1787561569.107705 mapwrite 0x00ebf7 thru 0x015ef2 (0x72fc bytes) 000086 1787561569.210471 write 0x035e78 thru 0x03ffff (0xa188 bytes) 000088 1787561569.229319 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x80b57400 (stale) Mirror 4: 0x80b57400 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x80b57400 (stale) Mirror 4: 0x80b57400 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000089 1787561569.676856 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x80b57400 (stale) Mirror 4: 0x80b57400 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x544e9b77 Mirror 5: 0x544e9b77 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000090 1787561571.680060 trunc from 0x040000 to 0x000ee6 (0x3f119 bytes) 000092 1787561571.779358 mapwrite 0x002f9a thru 0x009b25 (0x6b8c bytes) 000094 1787561571.885505 mapwrite 0x00b9b5 thru 0x013fec (0x8638 bytes) 000095 1787561572.002638 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x13fed): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13fed): Mirror 3: 0x67d8df57 (stale) Mirror 4: 0x67d8df57 (stale) Mirror 5: 0xd721001d (stale) CRC-32 checksum value for chunk [0, 0x13fed): Mirror 1: 0x6e50be57 Mirror 6: 0x6e50be57 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13fed: skipped 000099 1787561573.102906 mapwrite 0x02a17c thru 0x02f1e2 (0x5067 bytes) 000100 1787561573.228472 trunc from 0x02f1e3 to 0x02d7bf (0x1a23 bytes) 000102 1787561573.249320 mapwrite 0x014128 thru 0x01be16 (0x7cef bytes) 000103 1787561573.295847 read 0x019798 thru 0x02463e (0xaea7 bytes) 000108 1787561573.305907 trunc from 0x02d7c0 to 0x0184ce (0x152f1 bytes) 000111 1787561573.329764 write 0x007338 thru 0x009016 (0x1cdf bytes) 000112 1787561573.348245 write 0x008010 thru 0x016d27 (0xed18 bytes) 000113 1787561573.349912 trunc from 0x0184cf to 0x03bff9 (0x23b2b bytes) 000117 1787561573.370790 mapwrite 0x03c886 thru 0x03ffff (0x377a bytes) 000119 1787561573.450279 mapread 0x0081ac thru 0x017a12 (0xf867 bytes) 000121 1787561573.452680 mapread 0x02f2bd thru 0x0318e9 (0x262d bytes) 000123 1787561573.465828 trunc from 0x040000 to 0x037f8b (0x8074 bytes) 000124 1787561573.482283 write 0x024f5d thru 0x02f562 (0xa606 bytes) 000125 1787561573.490876 mapwrite 0x011c16 thru 0x019432 (0x781d bytes) 000126 1787561573.576363 read 0x01db90 thru 0x0208e5 (0x2d56 bytes) 000128 1787561573.586229 write 0x0105a8 thru 0x0159de (0x5437 bytes) 000129 1787561573.595381 trunc from 0x037f8c to 0x02de1a (0xa171 bytes) 000130 1787561573.645412 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2de1b): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2de1b): Mirror 3: 0xc6ec31a9 (stale) Mirror 4: 0xc6ec31a9 (stale) Mirror 5: 0x5946d54d (stale) Mirror 6: 0x7eb26677 (stale) CRC-32 checksum value for chunk [0, 0x2de1b): Mirror 1: 0x987547f8 Mirror 7: 0x987547f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2de1b: skipped 000131 1787561574.988509 write 0x02e6f0 thru 0x03ad7e (0xc68f bytes) 000133 1787561575.148223 read 0x028da2 thru 0x02a178 (0x13d7 bytes) 000135 1787561575.167165 mapread 0x025770 thru 0x0323e7 (0xcc78 bytes) 000136 1787561575.234275 trunc from 0x03ad7f to 0x037865 (0x3519 bytes) 000137 1787561575.274065 trunc from 0x037866 to 0x0197eb (0x1e07a bytes) 000138 1787561575.340867 write 0x021433 thru 0x0259b7 (0x4585 bytes) 000140 1787561575.354387 read 0x01c6a9 thru 0x020a88 (0x43e0 bytes) 000141 1787561575.381896 mapread 0x01ea08 thru 0x0259b7 (0x6fb0 bytes) 000144 1787561575.390404 write 0x023162 thru 0x02d562 (0xa401 bytes) 000145 1787561575.406838 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d563): Mirror 3: 0xbaf882a5 (stale) Mirror 4: 0xbaf882a5 (stale) Mirror 6: 0xa092fa09 (stale) Mirror 7: 0xe630d09 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d563): Mirror 3: 0xbaf882a5 (stale) Mirror 4: 0xbaf882a5 (stale) Mirror 6: 0xa092fa09 (stale) Mirror 7: 0xe630d09 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d563: skipped 000147 1787561575.951634 mapread 0x014223 thru 0x0205bd (0xc39b bytes) 000148 1787561575.995376 write 0x03c2ce thru 0x03ffff (0x3d32 bytes) 000149 1787561575.996768 read 0x03e263 thru 0x03ffff (0x1d9d bytes) 000150 1787561576.000034 mapread 0x02f56f thru 0x032329 (0x2dbb bytes) 000151 1787561576.027996 write 0x00a54e thru 0x00b259 (0xd0c bytes) 000153 1787561576.068845 MIRROR_RESYNC 5 mirrors 000154 1787561577.757740 read 0x01d5bc thru 0x020a34 (0x3479 bytes) 000156 1787561577.788225 read 0x0371b2 thru 0x03e2da (0x7129 bytes) 000157 1787561577.814273 mapread 0x015beb thru 0x01e50f (0x8925 bytes) 000158 1787561577.815464 read 0x013c0d thru 0x019e11 (0x6205 bytes) 000159 1787561577.816205 mapread 0x010354 thru 0x019d2a (0x99d7 bytes) 000160 1787561577.816814 mapwrite 0x01814e thru 0x020e9e (0x8d51 bytes) 000161 1787561578.000461 read 0x0356c8 thru 0x038e3d (0x3776 bytes) 000168 1787561578.027540 trunc from 0x040000 to 0x022559 (0x1daa6 bytes) 000171 1787561578.074124 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2255a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2255a): Mirror 3: 0xa9048249 (stale) Mirror 4: 0xa9048249 (stale) Mirror 6: 0xa9048249 (stale) Mirror 7: 0xa9048249 (stale) CRC-32 checksum value for chunk [0, 0x2255a): Mirror 1: 0x501fcaa4 Mirror 8: 0x501fcaa4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2255a: skipped 000172 1787561579.376759 trunc from 0x02255a to 0x025ff4 (0x3a9b bytes) 000178 1787561579.516132 mapread 0x01e875 thru 0x025ff4 (0x7780 bytes) 000180 1787561579.536658 trunc from 0x025ff5 to 0x036e11 (0x10e1d bytes) 000181 1787561579.556508 mapread 0x0123f8 thru 0x01f22d (0xce36 bytes) 000182 1787561579.568446 trunc from 0x036e12 to 0x01a461 (0x1c9b0 bytes) 000183 1787561579.584640 trunc from 0x01a462 to 0x02d0df (0x12c7e bytes) 000185 1787561579.616848 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2d0e0): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d0e0): Mirror 3: 0xc8a7bf36 (stale) Mirror 4: 0xc8a7bf36 (stale) Mirror 6: 0xc8a7bf36 (stale) Mirror 7: 0xc8a7bf36 (stale) Mirror 8: 0x95ac956a (stale) CRC-32 checksum value for chunk [0, 0x2d0e0): Mirror 1: 0xbc45b601 Mirror 9: 0xbc45b601 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d0e0: skipped 000186 1787561581.768580 write 0x01eeb5 thru 0x029ae3 (0xac2f bytes) 000187 1787561581.897692 write 0x03daad thru 0x03ffff (0x2553 bytes) 000188 1787561581.898875 trunc from 0x040000 to 0x021f5f (0x1e0a0 bytes) 000190 1787561581.960384 mapread 0x009d85 thru 0x019066 (0xf2e2 bytes) 000192 1787561582.018491 write 0x00a2f0 thru 0x00c8f9 (0x260a bytes) 000193 1787561582.028333 mapread 0x0204b8 thru 0x021f5f (0x1aa8 bytes) 000194 1787561582.029828 mapwrite 0x022547 thru 0x0253ba (0x2e74 bytes) 000195 1787561582.222149 write 0x02d9fe thru 0x0381f4 (0xa7f7 bytes) 000198 1787561582.231774 trunc from 0x0381f5 to 0x016619 (0x21bdb bytes) 000199 1787561582.268117 mapwrite 0x03e8f6 thru 0x03ffff (0x170a bytes) 000200 1787561582.453479 write 0x01f2f4 thru 0x02cf5f (0xdc6c bytes) 000202 1787561582.479234 read 0x0381c8 thru 0x03dcb7 (0x5af0 bytes) 000203 1787561582.479888 write 0x03c74b thru 0x03ffff (0x38b5 bytes) 000204 1787561582.480794 trunc from 0x040000 to 0x015f69 (0x2a096 bytes) 000206 1787561582.516429 mapwrite 0x01ab64 thru 0x024cb5 (0xa152 bytes) 000208 1787561582.677054 mapread 0x018dcc thru 0x024cb5 (0xbeea bytes) 000210 1787561582.679792 read 0x015ed6 thru 0x01ebc4 (0x8cef bytes) 000211 1787561582.681044 mapwrite 0x028543 thru 0x02c030 (0x3aee bytes) 000212 1787561582.828531 mapwrite 0x0273be thru 0x033a69 (0xc6ac bytes) 000213 1787561583.007382 mapread 0x00e52a thru 0x01b15f (0xcc36 bytes) 000214 1787561583.008371 read 0x0040c9 thru 0x00b92d (0x7865 bytes) 000215 1787561583.008976 write 0x011239 thru 0x01c804 (0xb5cc bytes) 000217 1787561583.016768 mapread 0x00e5b4 thru 0x0134ce (0x4f1b bytes) 000219 1787561583.023291 mapread 0x01b3b8 thru 0x021a4c (0x6695 bytes) 000221 1787561583.029356 trunc from 0x033a6a to 0x03c488 (0x8a1f bytes) 000222 1787561583.067922 trunc from 0x03c489 to 0x01c3d2 (0x200b6 bytes) 000224 1787561583.094728 write 0x00ed18 thru 0x0115be (0x28a7 bytes) 000226 1787561583.096188 trunc from 0x01c3d3 to 0x02a291 (0xdebf bytes) 000227 1787561583.134430 read 0x00a6ba thru 0x00b69c (0xfe3 bytes) 000229 1787561583.135824 read 0x026b80 thru 0x0292de (0x275f bytes) 000231 1787561583.159076 mapwrite 0x035680 thru 0x03ffff (0xa980 bytes) 000232 1787561583.307928 mapread 0x015b58 thru 0x019181 (0x362a bytes) 000236 1787561583.325512 mapread 0x011d7f thru 0x013313 (0x1595 bytes) 000238 1787561583.338661 mapread 0x03d5a9 thru 0x03ffff (0x2a57 bytes) 000240 1787561583.349598 mapwrite 0x02d5a0 thru 0x0397f2 (0xc253 bytes) 000241 1787561583.515624 write 0x021286 thru 0x031071 (0xfdec bytes) 000244 1787561583.551547 mapread 0x039a05 thru 0x03fbc2 (0x61be bytes) 000245 1787561583.558630 trunc from 0x040000 to 0x02c25f (0x13da0 bytes) 000247 1787561583.602506 trunc from 0x02c260 to 0x02acfa (0x1565 bytes) 000249 1787561583.627185 read 0x01f02e thru 0x02046c (0x143f bytes) 000250 1787561583.628360 read 0x010795 thru 0x016406 (0x5c72 bytes) 000252 1787561583.639875 trunc from 0x02acfb to 0x039467 (0xe76d bytes) 000253 1787561583.687722 write 0x0102a9 thru 0x012bea (0x2942 bytes) 000254 1787561583.697150 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39468): Mirror 3: 0xd1bc6036 (stale) Mirror 6: 0xd1bc6036 (stale) Mirror 7: 0xd1bc6036 (stale) Mirror 8: 0xc6bd45b9 (stale) Mirror 9: 0xce08bc1c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39468): Mirror 3: 0xd1bc6036 (stale) Mirror 6: 0xd1bc6036 (stale) Mirror 7: 0xd1bc6036 (stale) Mirror 8: 0xc6bd45b9 (stale) Mirror 9: 0xce08bc1c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39468: skipped 000257 1787561584.348883 trunc from 0x039468 to 0x036e03 (0x2664 bytes) 000258 1787561584.375927 mapread 0x03630a thru 0x036e03 (0xafa bytes) 000259 1787561584.385786 read 0x005b2a thru 0x0096a4 (0x3b7b bytes) 000260 1787561584.398307 mapread 0x031fec thru 0x036e03 (0x4e18 bytes) 000261 1787561584.416434 write 0x03651b thru 0x03ffff (0x9ae5 bytes) 000262 1787561584.428816 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x42271c6f (stale) Mirror 6: 0x42271c6f (stale) Mirror 7: 0x42271c6f (stale) Mirror 8: 0x5c1836ca (stale) Mirror 9: 0x77ede67c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdf1065a3 Mirror 10: 0xdf1065a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000263 1787561586.045967 write 0x02e409 thru 0x031afb (0x36f3 bytes) 000265 1787561586.346816 mapread 0x01443f thru 0x018582 (0x4144 bytes) 000266 1787561586.376480 write 0x021875 thru 0x0269ed (0x5179 bytes) 000268 1787561586.388876 trunc from 0x040000 to 0x00791b (0x386e4 bytes) 000270 1787561586.442536 mapwrite 0x03dcea thru 0x03ffff (0x2316 bytes) 000273 1787561586.577310 mapwrite 0x01afe8 thru 0x026922 (0xb93b bytes) 000274 1787561586.723178 read 0x022708 thru 0x02552f (0x2e28 bytes) 000275 1787561586.728544 trunc from 0x040000 to 0x022211 (0x1ddee bytes) 000276 1787561586.766607 mapread 0x009c49 thru 0x00d0a4 (0x345c bytes) 000278 1787561586.781118 mapread 0x0024ce thru 0x007602 (0x5135 bytes) 000279 1787561586.781929 trunc from 0x022212 to 0x017a6f (0xa7a2 bytes) 000282 1787561586.823279 mapwrite 0x0069f1 thru 0x01086e (0x9e7e bytes) 000284 1787561586.971738 MIRROR_RESYNC 7 mirrors 000286 1787561588.776128 read 0x013756 thru 0x017a6f (0x431a bytes) 000289 1787561588.788074 mapread 0x00abcb thru 0x015d4f (0xb185 bytes) 000292 1787561588.814884 write 0x02d606 thru 0x039949 (0xc344 bytes) 000297 1787561589.032515 mapwrite 0x01cb3c thru 0x01e1f7 (0x16bc bytes) 000298 1787561589.149620 mapread 0x002701 thru 0x00dfd4 (0xb8d4 bytes) 000299 1787561589.161368 mapwrite 0x02aa16 thru 0x0342b5 (0x98a0 bytes) 000300 1787561589.236476 mapwrite 0x011351 thru 0x012a5b (0x170b bytes) 000302 1787561589.287879 mapwrite 0x021f5f thru 0x030c7c (0xed1e bytes) 000303 1787561589.329180 read 0x018da4 thru 0x022839 (0x9a96 bytes) 000305 1787561589.330308 trunc from 0x03994a to 0x00a092 (0x2f8b7 bytes) 000307 1787561589.362492 MIRROR_RESYNC 7 mirrors 000308 1787561591.366059 MIRROR_EXTEND to 8 mirrors 000309 1787561592.585703 mapwrite 0x030df9 thru 0x03acb7 (0x9ebf bytes) 000310 1787561592.928113 read 0x027a41 thru 0x027dde (0x39e bytes) 000313 1787561592.938472 mapwrite 0x03b927 thru 0x03ffff (0x46d9 bytes) 000314 1787561593.040176 write 0x000c63 thru 0x00e619 (0xd9b7 bytes) 000316 1787561593.057060 mapwrite 0x00e9ce thru 0x01983a (0xae6d bytes) 000317 1787561593.275539 mapwrite 0x028cf3 thru 0x0328bb (0x9bc9 bytes) 000318 1787561593.362668 read 0x02123d thru 0x02d88b (0xc64f bytes) 000319 1787561593.363367 read 0x0039d2 thru 0x00f787 (0xbdb6 bytes) 000328 1787561593.393713 mapwrite 0x03e772 thru 0x03ffff (0x188e bytes) 000332 1787561593.535980 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x133ce91d (stale) Mirror 6: 0x133ce91d (stale) Mirror 7: 0x133ce91d (stale) Mirror 8: 0x133ce91d (stale) Mirror 9: 0x133ce91d (stale) Mirror 10: 0x133ce91d (stale) Mirror 11: 0x133ce91d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe928ce34 Mirror 12: 0xe928ce34 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped truncating to largest ever: 0x3e45b 000333 1787561594.653231 trunc from 0x040000 to 0x03e45a (0x1ba5 bytes) 000336 1787561594.771218 mapwrite 0x03f2c9 thru 0x03ffff (0xd37 bytes) 000339 1787561594.835659 mapread 0x01e7ec thru 0x02e230 (0xfa45 bytes) 000340 1787561594.857212 write 0x021129 thru 0x02d1aa (0xc082 bytes) 000341 1787561594.858962 mapread 0x0157b0 thru 0x01bcb0 (0x6501 bytes) 000342 1787561594.869040 write 0x00f591 thru 0x01d514 (0xdf84 bytes) 000345 1787561594.882674 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x133ce91d (stale) Mirror 6: 0x133ce91d (stale) Mirror 7: 0x133ce91d (stale) Mirror 8: 0x133ce91d (stale) Mirror 9: 0x133ce91d (stale) Mirror 10: 0x133ce91d (stale) Mirror 11: 0x133ce91d (stale) Mirror 12: 0xe928ce34 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x84a9a149 Mirror 13: 0x84a9a149 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000346 1787561596.403961 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x133ce91d (stale) Mirror 6: 0x133ce91d (stale) Mirror 7: 0x133ce91d (stale) Mirror 8: 0x133ce91d (stale) Mirror 9: 0x133ce91d (stale) Mirror 10: 0x133ce91d (stale) Mirror 11: 0x133ce91d (stale) Mirror 12: 0xe928ce34 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x84a9a149 Mirror 13: 0x84a9a149 Mirror 14: 0x84a9a149 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000350 1787561597.399302 write 0x021860 thru 0x02bd59 (0xa4fa bytes) 000351 1787561597.565862 write 0x035e8a thru 0x03ffff (0xa176 bytes) 000353 1787561597.597594 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x133ce91d (stale) Mirror 6: 0x133ce91d (stale) Mirror 7: 0x133ce91d (stale) Mirror 8: 0x133ce91d (stale) Mirror 9: 0x133ce91d (stale) Mirror 10: 0x133ce91d (stale) Mirror 11: 0x133ce91d (stale) Mirror 12: 0xe928ce34 (stale) Mirror 13: 0x84a9a149 (stale) Mirror 14: 0x84a9a149 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x10757737 Mirror 15: 0x10757737 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000354 1787561599.268823 trunc from 0x040000 to 0x022800 (0x1d7ff bytes) 000355 1787561599.503335 MIRROR_SPLIT mirror 6 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x22801): Mirror 3: 0x5dd5d6f0 (stale) Mirror 6: 0x5dd5d6f0 (stale) Mirror 7: 0x5dd5d6f0 (stale) Mirror 8: 0x5dd5d6f0 (stale) Mirror 9: 0x5dd5d6f0 (stale) Mirror 11: 0x5dd5d6f0 (stale) Mirror 12: 0xbeff55bd (stale) Mirror 13: 0x5a316479 (stale) Mirror 14: 0x5a316479 (stale) Mirror 15: 0x32320132 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22801): Mirror 3: 0x5dd5d6f0 (stale) Mirror 6: 0x5dd5d6f0 (stale) Mirror 7: 0x5dd5d6f0 (stale) Mirror 8: 0x5dd5d6f0 (stale) Mirror 9: 0x5dd5d6f0 (stale) Mirror 11: 0x5dd5d6f0 (stale) Mirror 12: 0xbeff55bd (stale) Mirror 13: 0x5a316479 (stale) Mirror 14: 0x5a316479 (stale) Mirror 15: 0x32320132 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22801: skipped 000356 1787561600.403603 mapread 0x01b4bf thru 0x01c62a (0x116c bytes) 000357 1787561600.417535 write 0x01a773 thru 0x028a6e (0xe2fc bytes) 000358 1787561600.431850 MIRROR_RESYNC 11 mirrors 000359 1787561602.950511 write 0x03b089 thru 0x03ffff (0x4f77 bytes) 000362 1787561603.047287 trunc from 0x040000 to 0x027155 (0x18eaa bytes) 000366 1787561603.070282 mapwrite 0x004977 thru 0x013fc3 (0xf64d bytes) 000367 1787561603.150806 mapwrite 0x003823 thru 0x00bb3b (0x8319 bytes) 000368 1787561603.196064 read 0x026fc4 thru 0x027155 (0x192 bytes) 000369 1787561603.201479 read 0x0008d3 thru 0x00c43a (0xbb68 bytes) 000373 1787561603.212530 read 0x01042c thru 0x017960 (0x7535 bytes) 000375 1787561603.214345 trunc from 0x027156 to 0x008085 (0x1f0d0 bytes) 000376 1787561603.231975 mapread 0x002d67 thru 0x008085 (0x531f bytes) 000378 1787561603.234368 trunc from 0x008086 to 0x03cf07 (0x34e82 bytes) 000379 1787561603.248630 mapwrite 0x00477a thru 0x011fbf (0xd846 bytes) 000380 1787561603.330835 trunc from 0x03cf08 to 0x00b920 (0x315e7 bytes) 000387 1787561603.367292 trunc from 0x00b921 to 0x031c9c (0x2637c bytes) 000388 1787561603.392471 write 0x0023ea thru 0x0045af (0x21c6 bytes) 000389 1787561603.393832 trunc from 0x031c9d to 0x03e404 (0xc768 bytes) 000390 1787561603.419722 read 0x01b26d thru 0x026bd9 (0xb96d bytes) 000391 1787561603.439846 mapwrite 0x011f64 thru 0x020146 (0xe1e3 bytes) 000392 1787561603.536995 mapwrite 0x024567 thru 0x032e53 (0xe8ed bytes) 000393 1787561603.636722 mapwrite 0x03a0f4 thru 0x03b1a8 (0x10b5 bytes) 000394 1787561603.689938 mapwrite 0x012d7f thru 0x01b3ac (0x862e bytes) 000398 1787561603.763500 MIRROR_RESYNC 11 mirrors 000400 1787561605.538697 mapread 0x00ec51 thru 0x0181fa (0x95aa bytes) 000401 1787561605.559666 mapread 0x029b21 thru 0x02e23e (0x471e bytes) 000403 1787561605.575537 MIRROR_RESYNC 11 mirrors 000404 1787561606.221456 mapwrite 0x0072e6 thru 0x00dcce (0x69e9 bytes) 000405 1787561606.445894 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e405): Mirror 3: 0x960dcfe0 (stale) Mirror 6: 0x960dcfe0 (stale) Mirror 8: 0x960dcfe0 (stale) Mirror 9: 0x960dcfe0 (stale) Mirror 11: 0x960dcfe0 (stale) Mirror 12: 0x960dcfe0 (stale) Mirror 13: 0x960dcfe0 (stale) Mirror 14: 0x960dcfe0 (stale) Mirror 15: 0x960dcfe0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e405): Mirror 3: 0x960dcfe0 (stale) Mirror 6: 0x960dcfe0 (stale) Mirror 8: 0x960dcfe0 (stale) Mirror 9: 0x960dcfe0 (stale) Mirror 11: 0x960dcfe0 (stale) Mirror 12: 0x960dcfe0 (stale) Mirror 13: 0x960dcfe0 (stale) Mirror 14: 0x960dcfe0 (stale) Mirror 15: 0x960dcfe0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e405: skipped 000409 1787561607.112245 MIRROR_RESYNC 10 mirrors 000411 1787561609.019691 MIRROR_RESYNC 10 mirrors 000412 1787561609.725053 mapwrite 0x024083 thru 0x0292e9 (0x5267 bytes) 000413 1787561609.930416 trunc from 0x03e405 to 0x01beb9 (0x2254b bytes) 000416 1787561609.978102 trunc from 0x01beba to 0x02a6f2 (0xe839 bytes) 000419 1787561610.011231 mapread 0x001554 thru 0x001867 (0x314 bytes) 000421 1787561610.034699 trunc from 0x02a6f3 to 0x0028a9 (0x27e49 bytes) 000422 1787561610.064534 trunc from 0x0028aa to 0x0018ce (0xfdb bytes) 000423 1787561610.082523 mapwrite 0x002cd0 thru 0x00569c (0x29cd bytes) 000424 1787561610.167427 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x569d): Mirror 6: 0x3685d6a7 (stale) Mirror 8: 0x3685d6a7 (stale) Mirror 9: 0x3685d6a7 (stale) Mirror 11: 0x3685d6a7 (stale) Mirror 12: 0x3685d6a7 (stale) Mirror 13: 0x3685d6a7 (stale) Mirror 14: 0x3685d6a7 (stale) Mirror 15: 0x3685d6a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x569d): Mirror 6: 0x3685d6a7 (stale) Mirror 8: 0x3685d6a7 (stale) Mirror 9: 0x3685d6a7 (stale) Mirror 11: 0x3685d6a7 (stale) Mirror 12: 0x3685d6a7 (stale) Mirror 13: 0x3685d6a7 (stale) Mirror 14: 0x3685d6a7 (stale) Mirror 15: 0x3685d6a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x569d: skipped 000425 1787561610.912039 trunc from 0x00569d to 0x00bd00 (0x6664 bytes) 000426 1787561610.929793 mapwrite 0x002c8c thru 0x00eda6 (0xc11b bytes) 000428 1787561611.091333 mapread 0x002874 thru 0x00c2d5 (0x9a62 bytes) 000434 1787561611.097668 mapwrite 0x025f7d thru 0x02f181 (0x9205 bytes) 000437 1787561611.174307 mapwrite 0x01e3c0 thru 0x01fad1 (0x1712 bytes) 000438 1787561611.213968 read 0x026eee thru 0x02ade2 (0x3ef5 bytes) 000440 1787561611.223470 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f182): Mirror 6: 0x2348bec7 (stale) Mirror 8: 0x2348bec7 (stale) Mirror 9: 0x2348bec7 (stale) Mirror 11: 0x2348bec7 (stale) Mirror 13: 0x2348bec7 (stale) Mirror 14: 0x2348bec7 (stale) Mirror 15: 0x2348bec7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f182): Mirror 6: 0x2348bec7 (stale) Mirror 8: 0x2348bec7 (stale) Mirror 9: 0x2348bec7 (stale) Mirror 11: 0x2348bec7 (stale) Mirror 13: 0x2348bec7 (stale) Mirror 14: 0x2348bec7 (stale) Mirror 15: 0x2348bec7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f182: skipped 000441 1787561611.676690 mapread 0x02a760 thru 0x02f181 (0x4a22 bytes) 000442 1787561611.698837 mapwrite 0x02774d thru 0x02a503 (0x2db7 bytes) 000443 1787561611.731213 trunc from 0x02f182 to 0x02fe7c (0xcfb bytes) 000444 1787561611.745080 mapread 0x005645 thru 0x01321a (0xdbd6 bytes) 000445 1787561611.767412 mapread 0x008ce1 thru 0x013017 (0xa337 bytes) 000446 1787561611.769946 read 0x008a73 thru 0x00c8fb (0x3e89 bytes) 000447 1787561611.770936 trunc from 0x02fe7d to 0x02fe22 (0x5a bytes) 000448 1787561611.788075 mapread 0x023501 thru 0x02fe22 (0xc922 bytes) 000449 1787561611.796628 mapwrite 0x00ab23 thru 0x013060 (0x853e bytes) 000450 1787561611.842432 mapread 0x020558 thru 0x026723 (0x61cc bytes) 000451 1787561611.844670 trunc from 0x02fe23 to 0x004c6a (0x2b1b8 bytes) 000456 1787561611.873126 read 0x004a5e thru 0x004c6a (0x20d bytes) 000459 1787561611.877098 read 0x0038cf thru 0x004c6a (0x139c bytes) 000460 1787561611.883307 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x4c6b): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4c6b): Mirror 6: 0x71444b8e (stale) Mirror 8: 0x71444b8e (stale) Mirror 9: 0x71444b8e (stale) Mirror 11: 0x71444b8e (stale) Mirror 13: 0x71444b8e (stale) Mirror 14: 0x71444b8e (stale) Mirror 15: 0x71444b8e (stale) CRC-32 checksum value for chunk [0, 0x4c6b): Mirror 1: 0xa8e6ff0f Mirror 16: 0xa8e6ff0f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4c6b: skipped 000462 1787561613.097594 trunc from 0x004c6b to 0x020bd6 (0x1bf6c bytes) 000465 1787561613.206695 write 0x0286ce thru 0x02b768 (0x309b bytes) 000469 1787561613.228856 write 0x0235a5 thru 0x02d9e5 (0xa441 bytes) 000470 1787561613.246110 trunc from 0x02d9e6 to 0x035c3b (0x8256 bytes) truncating to largest ever: 0x3f4c2 000472 1787561613.273530 trunc from 0x035c3c to 0x03f4c1 (0x9886 bytes) 000473 1787561613.314752 mapread 0x036891 thru 0x03c7bf (0x5f2f bytes) 000474 1787561613.348683 trunc from 0x03f4c2 to 0x017c82 (0x2783f bytes) 000476 1787561613.363223 mapwrite 0x00c7c7 thru 0x011a56 (0x5290 bytes) 000477 1787561613.460144 mapwrite 0x00b3f5 thru 0x010cf7 (0x5903 bytes) 000478 1787561613.547174 read 0x014365 thru 0x017c82 (0x391e bytes) 000480 1787561613.553518 read 0x003c23 thru 0x007053 (0x3431 bytes) 000482 1787561613.561603 write 0x0393ad thru 0x03bea3 (0x2af7 bytes) 000484 1787561613.563985 trunc from 0x03bea4 to 0x0223fa (0x19aa9 bytes) 000485 1787561613.575720 write 0x001738 thru 0x004926 (0x31ef bytes) 000486 1787561613.584342 MIRROR_RESYNC 9 mirrors 000487 1787561615.650886 read 0x01f57d thru 0x0223fa (0x2e7e bytes) 000490 1787561615.663722 MIRROR_RESYNC 9 mirrors 000492 1787561616.441931 trunc from 0x0223fb to 0x00a38f (0x1806b bytes) 000493 1787561616.601491 mapwrite 0x0205c3 thru 0x02fea5 (0xf8e3 bytes) 000494 1787561616.766510 read 0x003a97 thru 0x00847c (0x49e6 bytes) 000498 1787561616.781971 write 0x03f295 thru 0x03ffff (0xd6b bytes) 000500 1787561616.784466 trunc from 0x040000 to 0x02c519 (0x13ae6 bytes) 000501 1787561616.800067 read 0x0209b9 thru 0x02b234 (0xa87c bytes) 000502 1787561616.801192 write 0x00bb01 thru 0x015b12 (0xa012 bytes) 000503 1787561616.812640 trunc from 0x02c51a to 0x013772 (0x18da7 bytes) 000504 1787561616.827892 read 0x008e8e thru 0x013772 (0xa8e5 bytes) 000507 1787561616.831485 mapwrite 0x0164d3 thru 0x025bd3 (0xf701 bytes) 000509 1787561616.917935 trunc from 0x025bd4 to 0x027206 (0x1633 bytes) 000510 1787561616.937398 mapwrite 0x009de3 thru 0x00a998 (0xbb6 bytes) 000513 1787561616.985320 mapread 0x015538 thru 0x0167b8 (0x1281 bytes) 000514 1787561616.994359 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x27207): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27207): Mirror 6: 0xd006f759 (stale) Mirror 8: 0xd006f759 (stale) Mirror 9: 0xd006f759 (stale) Mirror 11: 0xd006f759 (stale) Mirror 13: 0xd006f759 (stale) Mirror 14: 0xd006f759 (stale) Mirror 15: 0xd006f759 (stale) Mirror 16: 0xd006f759 (stale) CRC-32 checksum value for chunk [0, 0x27207): Mirror 1: 0xdd57dee6 Mirror 17: 0xdd57dee6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27207: skipped 000515 1787561618.394693 write 0x023e5b thru 0x028216 (0x43bc bytes) 000517 1787561618.546607 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28217): Mirror 6: 0xbaf9ea3e (stale) Mirror 8: 0xbaf9ea3e (stale) Mirror 9: 0xbaf9ea3e (stale) Mirror 11: 0xbaf9ea3e (stale) Mirror 13: 0xbaf9ea3e (stale) Mirror 14: 0xbaf9ea3e (stale) Mirror 16: 0xbaf9ea3e (stale) Mirror 17: 0x5db0b921 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28217): Mirror 6: 0xbaf9ea3e (stale) Mirror 8: 0xbaf9ea3e (stale) Mirror 9: 0xbaf9ea3e (stale) Mirror 11: 0xbaf9ea3e (stale) Mirror 13: 0xbaf9ea3e (stale) Mirror 14: 0xbaf9ea3e (stale) Mirror 16: 0xbaf9ea3e (stale) Mirror 17: 0x5db0b921 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28217: skipped 000519 1787561619.412944 mapread 0x0046a2 thru 0x00de20 (0x977f bytes) 000520 1787561619.470443 trunc from 0x028217 to 0x00acd5 (0x1d541 bytes) 000521 1787561619.491953 read 0x00287d thru 0x00aa6f (0x81f3 bytes) 000522 1787561619.493467 trunc from 0x00acd6 to 0x009667 (0x166e bytes) 000523 1787561619.528361 mapwrite 0x03f80f thru 0x03ff79 (0x76b bytes) 000524 1787561619.646990 mapread 0x01d73e thru 0x02ccfd (0xf5c0 bytes) 000525 1787561619.679982 write 0x03a460 thru 0x03ffff (0x5ba0 bytes) 000527 1787561619.681490 read 0x038cd1 thru 0x03fdcb (0x70fb bytes) 000528 1787561619.682102 mapwrite 0x01abaa thru 0x029a97 (0xeeee bytes) 000529 1787561619.757756 mapread 0x001165 thru 0x0046fc (0x3598 bytes) 000530 1787561619.758620 write 0x035a19 thru 0x0368fb (0xee3 bytes) 000531 1787561619.776334 trunc from 0x040000 to 0x03371a (0xc8e5 bytes) 000532 1787561619.802830 mapwrite 0x039ab6 thru 0x03ffff (0x654a bytes) 000535 1787561620.053665 mapwrite 0x026919 thru 0x0323f8 (0xbae0 bytes) 000539 1787561620.132281 MIRROR_RESYNC 9 mirrors 000542 1787561622.269259 mapwrite 0x03fa60 thru 0x03ffff (0x5a0 bytes) 000545 1787561622.719166 trunc from 0x040000 to 0x013523 (0x2cadc bytes) 000547 1787561622.782832 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x13524): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13524): Mirror 6: 0xa69a8198 (stale) Mirror 8: 0xa69a8198 (stale) Mirror 9: 0xa69a8198 (stale) Mirror 11: 0xa69a8198 (stale) Mirror 13: 0xa69a8198 (stale) Mirror 14: 0xa69a8198 (stale) Mirror 16: 0xa69a8198 (stale) Mirror 17: 0xa69a8198 (stale) CRC-32 checksum value for chunk [0, 0x13524): Mirror 1: 0xa69a8198 Mirror 18: 0xa69a8198 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13524: skipped 000548 1787561624.304707 read 0x001a38 thru 0x00420a (0x27d3 bytes) 000549 1787561624.324406 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x13524): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13524): Mirror 6: 0xa69a8198 (stale) Mirror 8: 0xa69a8198 (stale) Mirror 9: 0xa69a8198 (stale) Mirror 11: 0xa69a8198 (stale) Mirror 13: 0xa69a8198 (stale) Mirror 14: 0xa69a8198 (stale) Mirror 16: 0xa69a8198 (stale) Mirror 17: 0xa69a8198 (stale) CRC-32 checksum value for chunk [0, 0x13524): Mirror 1: 0xa69a8198 Mirror 18: 0xa69a8198 Mirror 19: 0xa69a8198 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13524: skipped 000550 1787561626.386507 read 0x000f4d thru 0x00ba66 (0xab1a bytes) 000551 1787561626.397625 trunc from 0x013524 to 0x017b35 (0x4612 bytes) 000552 1787561626.616337 read 0x00dc95 thru 0x013ceb (0x6057 bytes) 000553 1787561626.651729 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x17b36): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17b36): Mirror 6: 0xe133e9be (stale) Mirror 8: 0xe133e9be (stale) Mirror 9: 0xe133e9be (stale) Mirror 11: 0xe133e9be (stale) Mirror 13: 0xe133e9be (stale) Mirror 14: 0xe133e9be (stale) Mirror 16: 0xe133e9be (stale) Mirror 17: 0xe133e9be (stale) Mirror 18: 0xe133e9be (stale) Mirror 19: 0xe133e9be (stale) CRC-32 checksum value for chunk [0, 0x17b36): Mirror 1: 0xe133e9be Mirror 20: 0xe133e9be lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17b36: skipped 000555 1787561627.972735 write 0x00a89c thru 0x0178f8 (0xd05d bytes) 000558 1787561628.105065 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17b36): Mirror 8: 0xe133e9be (stale) Mirror 9: 0xe133e9be (stale) Mirror 11: 0xe133e9be (stale) Mirror 13: 0xe133e9be (stale) Mirror 14: 0xe133e9be (stale) Mirror 16: 0xe133e9be (stale) Mirror 17: 0xe133e9be (stale) Mirror 18: 0xe133e9be (stale) Mirror 19: 0xe133e9be (stale) Mirror 20: 0xe133e9be (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17b36): Mirror 8: 0xe133e9be (stale) Mirror 9: 0xe133e9be (stale) Mirror 11: 0xe133e9be (stale) Mirror 13: 0xe133e9be (stale) Mirror 14: 0xe133e9be (stale) Mirror 16: 0xe133e9be (stale) Mirror 17: 0xe133e9be (stale) Mirror 18: 0xe133e9be (stale) Mirror 19: 0xe133e9be (stale) Mirror 20: 0xe133e9be (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17b36: skipped 000559 1787561628.712850 read 0x002d21 thru 0x0102d3 (0xd5b3 bytes) 000560 1787561628.724338 read 0x01747f thru 0x017b35 (0x6b7 bytes) 000561 1787561628.737886 trunc from 0x017b36 to 0x022ae2 (0xafad bytes) 000562 1787561628.762016 read 0x0093f8 thru 0x00f2d2 (0x5edb bytes) 000563 1787561628.762671 trunc from 0x022ae3 to 0x015da4 (0xcd3e bytes) 000565 1787561628.791862 mapwrite 0x0027e6 thru 0x0049ac (0x21c7 bytes) 000567 1787561628.958587 mapwrite 0x03a3f2 thru 0x03ffff (0x5c0e bytes) 000569 1787561629.170908 write 0x029e98 thru 0x030116 (0x627f bytes) 000571 1787561629.179305 write 0x026fff thru 0x027a5a (0xa5c bytes) 000572 1787561629.181380 write 0x0257a5 thru 0x02c132 (0x698e bytes) 000574 1787561629.186549 write 0x02ac47 thru 0x035f7e (0xb338 bytes) 000575 1787561629.197962 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x21cbe159 (stale) Mirror 9: 0x21cbe159 (stale) Mirror 11: 0x21cbe159 (stale) Mirror 13: 0x21cbe159 (stale) Mirror 14: 0x21cbe159 (stale) Mirror 16: 0x21cbe159 (stale) Mirror 17: 0x21cbe159 (stale) Mirror 19: 0xebd18042 (stale) Mirror 20: 0xebd18042 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x21cbe159 (stale) Mirror 9: 0x21cbe159 (stale) Mirror 11: 0x21cbe159 (stale) Mirror 13: 0x21cbe159 (stale) Mirror 14: 0x21cbe159 (stale) Mirror 16: 0x21cbe159 (stale) Mirror 17: 0x21cbe159 (stale) Mirror 19: 0xebd18042 (stale) Mirror 20: 0xebd18042 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000576 1787561629.988779 write 0x007a3d thru 0x012fef (0xb5b3 bytes) 000577 1787561630.008209 MIRROR_RESYNC 10 mirrors 000580 1787561632.438804 trunc from 0x040000 to 0x001c72 (0x3e38d bytes) 000582 1787561632.567534 write 0x012168 thru 0x01db03 (0xb99c bytes) 000583 1787561632.568996 trunc from 0x01db04 to 0x032d52 (0x1524f bytes) 000584 1787561632.603085 mapwrite 0x02f536 thru 0x03e5e3 (0xf0ae bytes) 000585 1787561632.739197 trunc from 0x03e5e4 to 0x006a08 (0x37bdb bytes) 000586 1787561632.764455 MIRROR_RESYNC 10 mirrors 000591 1787561635.041831 read 0x002e78 thru 0x006a08 (0x3b91 bytes) 000592 1787561635.054848 mapread 0x004905 thru 0x006a08 (0x2104 bytes) 000593 1787561635.055692 trunc from 0x006a09 to 0x004cb8 (0x1d50 bytes) 000594 1787561635.238629 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4cb9): Mirror 9: 0x5b3f6f67 (stale) Mirror 11: 0x5b3f6f67 (stale) Mirror 13: 0x5b3f6f67 (stale) Mirror 14: 0x5b3f6f67 (stale) Mirror 16: 0x5b3f6f67 (stale) Mirror 17: 0x5b3f6f67 (stale) Mirror 19: 0x5b3f6f67 (stale) Mirror 20: 0x5b3f6f67 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4cb9): Mirror 9: 0x5b3f6f67 (stale) Mirror 11: 0x5b3f6f67 (stale) Mirror 13: 0x5b3f6f67 (stale) Mirror 14: 0x5b3f6f67 (stale) Mirror 16: 0x5b3f6f67 (stale) Mirror 17: 0x5b3f6f67 (stale) Mirror 19: 0x5b3f6f67 (stale) Mirror 20: 0x5b3f6f67 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4cb9: skipped 000595 1787561635.897977 mapwrite 0x00fd6d thru 0x01f2e3 (0xf577 bytes) 000596 1787561636.070706 read 0x003de0 thru 0x005cb9 (0x1eda bytes) 000597 1787561636.082010 write 0x037828 thru 0x03ffff (0x87d8 bytes) 000600 1787561636.097487 trunc from 0x040000 to 0x02fc9e (0x10361 bytes) 000602 1787561636.151090 trunc from 0x02fc9f to 0x0388c1 (0x8c23 bytes) 000604 1787561636.206503 mapread 0x01cf53 thru 0x0248d0 (0x797e bytes) 000605 1787561636.232810 mapwrite 0x01de72 thru 0x020fa3 (0x3132 bytes) 000606 1787561636.307359 mapwrite 0x015fc5 thru 0x01845e (0x249a bytes) 000608 1787561636.365218 mapread 0x022cb3 thru 0x02e2c3 (0xb611 bytes) 000609 1787561636.385930 mapwrite 0x0388d4 thru 0x038dce (0x4fb bytes) 000610 1787561636.513850 mapread 0x003e39 thru 0x00e34c (0xa514 bytes) 000612 1787561636.573035 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x38dcf): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38dcf): Mirror 9: 0x3eb85ad7 (stale) Mirror 11: 0x3eb85ad7 (stale) Mirror 13: 0x3eb85ad7 (stale) Mirror 14: 0x3eb85ad7 (stale) Mirror 16: 0x3eb85ad7 (stale) Mirror 17: 0x3eb85ad7 (stale) Mirror 19: 0x3eb85ad7 (stale) Mirror 20: 0x3eb85ad7 (stale) CRC-32 checksum value for chunk [0, 0x38dcf): Mirror 1: 0x9d98871 Mirror 21: 0x9d98871 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38dcf: skipped 000613 1787561637.959495 write 0x022cde thru 0x031157 (0xe47a bytes) 000614 1787561638.181217 mapwrite 0x00cfb3 thru 0x010305 (0x3353 bytes) 000616 1787561638.269401 mapread 0x0203d8 thru 0x02c2c3 (0xbeec bytes) 000617 1787561638.281530 mapread 0x01a7d1 thru 0x01f063 (0x4893 bytes) 000619 1787561638.283026 write 0x004a79 thru 0x009df6 (0x537e bytes) 000620 1787561638.297220 read 0x02e026 thru 0x0369dc (0x89b7 bytes) 000621 1787561638.310321 read 0x011eca thru 0x01d719 (0xb850 bytes) 000622 1787561638.330957 read 0x01adcd thru 0x024a10 (0x9c44 bytes) 000623 1787561638.332257 mapread 0x035a5c thru 0x038dce (0x3373 bytes) 000626 1787561638.339252 write 0x016118 thru 0x0189bb (0x28a4 bytes) 000628 1787561638.345090 trunc from 0x038dcf to 0x021fca (0x16e04 bytes) 000629 1787561638.402314 read 0x01a83e thru 0x021fca (0x778d bytes) 000630 1787561638.403002 write 0x014e21 thru 0x02229f (0xd47f bytes) 000634 1787561638.409305 mapread 0x00476e thru 0x0133e0 (0xec73 bytes) 000638 1787561638.414722 trunc from 0x0222a0 to 0x009d53 (0x1854c bytes) 000639 1787561638.490400 MIRROR_RESYNC 10 mirrors 000641 1787561640.725193 read 0x0078ef thru 0x009d36 (0x2448 bytes) 000642 1787561640.730476 write 0x03cc6e thru 0x03ffff (0x3392 bytes) 000644 1787561640.806412 mapread 0x020c8c thru 0x02de3b (0xd1b0 bytes) 000645 1787561640.872359 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x528657b9 (stale) Mirror 11: 0x528657b9 (stale) Mirror 13: 0x528657b9 (stale) Mirror 14: 0x528657b9 (stale) Mirror 16: 0x528657b9 (stale) Mirror 17: 0x528657b9 (stale) Mirror 19: 0x528657b9 (stale) Mirror 20: 0x528657b9 (stale) Mirror 21: 0x528657b9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8fee81ad Mirror 22: 0x8fee81ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000646 1787561642.436489 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x528657b9 (stale) Mirror 11: 0x528657b9 (stale) Mirror 13: 0x528657b9 (stale) Mirror 14: 0x528657b9 (stale) Mirror 16: 0x528657b9 (stale) Mirror 17: 0x528657b9 (stale) Mirror 19: 0x528657b9 (stale) Mirror 20: 0x528657b9 (stale) Mirror 21: 0x528657b9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8fee81ad Mirror 22: 0x8fee81ad Mirror 23: 0x8fee81ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000647 1787561643.907398 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x528657b9 (stale) Mirror 11: 0x528657b9 (stale) Mirror 14: 0x528657b9 (stale) Mirror 16: 0x528657b9 (stale) Mirror 17: 0x528657b9 (stale) Mirror 19: 0x528657b9 (stale) Mirror 20: 0x528657b9 (stale) Mirror 21: 0x528657b9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x528657b9 (stale) Mirror 11: 0x528657b9 (stale) Mirror 14: 0x528657b9 (stale) Mirror 16: 0x528657b9 (stale) Mirror 17: 0x528657b9 (stale) Mirror 19: 0x528657b9 (stale) Mirror 20: 0x528657b9 (stale) Mirror 21: 0x528657b9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8fee81ad Mirror 22: 0x8fee81ad Mirror 23: 0x8fee81ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000648 1787561644.918196 mapread 0x000771 thru 0x0090ae (0x893e bytes) 000651 1787561644.974084 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x528657b9 (stale) Mirror 11: 0x528657b9 (stale) Mirror 14: 0x528657b9 (stale) Mirror 16: 0x528657b9 (stale) Mirror 17: 0x528657b9 (stale) Mirror 19: 0x528657b9 (stale) Mirror 20: 0x528657b9 (stale) Mirror 21: 0x528657b9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8fee81ad Mirror 22: 0x8fee81ad Mirror 23: 0x8fee81ad Mirror 24: 0x8fee81ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000653 1787561646.211537 mapwrite 0x01d4e9 thru 0x0245bc (0x70d4 bytes) 000655 1787561646.521675 mapread 0x03f5c8 thru 0x03ffff (0xa38 bytes) 000657 1787561646.535041 read 0x030a8e thru 0x03582c (0x4d9f bytes) 000658 1787561646.549472 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x528657b9 (stale) Mirror 11: 0x528657b9 (stale) Mirror 14: 0x528657b9 (stale) Mirror 16: 0x528657b9 (stale) Mirror 17: 0x528657b9 (stale) Mirror 19: 0x528657b9 (stale) Mirror 20: 0x528657b9 (stale) Mirror 21: 0x528657b9 (stale) Mirror 22: 0x8fee81ad (stale) Mirror 23: 0x8fee81ad (stale) Mirror 24: 0x8fee81ad (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xab209520 Mirror 25: 0xab209520 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000659 1787561647.649745 write 0x02fbd4 thru 0x030a02 (0xe2f bytes) 000662 1787561647.883829 trunc from 0x040000 to 0x01c4fe (0x23b01 bytes) 000664 1787561647.933716 mapwrite 0x011f5a thru 0x01ddd8 (0xbe7f bytes) 000665 1787561648.127533 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x1ddd9): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ddd9): Mirror 9: 0x7fbe101e (stale) Mirror 11: 0x7fbe101e (stale) Mirror 14: 0x7fbe101e (stale) Mirror 16: 0x7fbe101e (stale) Mirror 17: 0x7fbe101e (stale) Mirror 19: 0x7fbe101e (stale) Mirror 20: 0x7fbe101e (stale) Mirror 21: 0x7fbe101e (stale) Mirror 22: 0x7fbe101e (stale) Mirror 23: 0x7fbe101e (stale) Mirror 24: 0x7fbe101e (stale) Mirror 25: 0x82a54b15 (stale) CRC-32 checksum value for chunk [0, 0x1ddd9): Mirror 1: 0xef2973b8 Mirror 26: 0xef2973b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ddd9: skipped 000668 1787561649.453054 mapwrite 0x004568 thru 0x01404c (0xfae5 bytes) 000670 1787561649.785770 trunc from 0x01ddd9 to 0x00c6b7 (0x11721 bytes) 000671 1787561649.870165 trunc from 0x00c6b8 to 0x023537 (0x16e80 bytes) 000672 1787561649.899352 trunc from 0x023538 to 0x00f634 (0x13f03 bytes) 000673 1787561649.939094 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf635): Mirror 11: 0xec976b76 (stale) Mirror 14: 0xec976b76 (stale) Mirror 16: 0xec976b76 (stale) Mirror 17: 0xec976b76 (stale) Mirror 19: 0xec976b76 (stale) Mirror 20: 0xec976b76 (stale) Mirror 21: 0xec976b76 (stale) Mirror 22: 0xec976b76 (stale) Mirror 23: 0xec976b76 (stale) Mirror 24: 0xec976b76 (stale) Mirror 25: 0xec976b76 (stale) Mirror 26: 0xec976b76 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf635): Mirror 11: 0xec976b76 (stale) Mirror 14: 0xec976b76 (stale) Mirror 16: 0xec976b76 (stale) Mirror 17: 0xec976b76 (stale) Mirror 19: 0xec976b76 (stale) Mirror 20: 0xec976b76 (stale) Mirror 21: 0xec976b76 (stale) Mirror 22: 0xec976b76 (stale) Mirror 23: 0xec976b76 (stale) Mirror 24: 0xec976b76 (stale) Mirror 25: 0xec976b76 (stale) Mirror 26: 0xec976b76 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf635: skipped 000677 1787561650.651699 trunc from 0x00f635 to 0x02395a (0x14326 bytes) 000678 1787561650.667743 mapwrite 0x0122bf thru 0x01bd58 (0x9a9a bytes) truncating to largest ever: 0x3f6ab 000679 1787561650.755996 trunc from 0x02395b to 0x03f6aa (0x1bd50 bytes) 000680 1787561650.776377 trunc from 0x03f6ab to 0x00d09e (0x3260c bytes) 000681 1787561650.802691 MIRROR_SPLIT mirror 7 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd09f): Mirror 11: 0x4a7f83ef (stale) Mirror 14: 0x4a7f83ef (stale) Mirror 16: 0x4a7f83ef (stale) Mirror 17: 0x4a7f83ef (stale) Mirror 19: 0x4a7f83ef (stale) Mirror 20: 0x4a7f83ef (stale) Mirror 22: 0x4a7f83ef (stale) Mirror 23: 0x4a7f83ef (stale) Mirror 24: 0x4a7f83ef (stale) Mirror 25: 0x4a7f83ef (stale) Mirror 26: 0x4a7f83ef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd09f): Mirror 11: 0x4a7f83ef (stale) Mirror 14: 0x4a7f83ef (stale) Mirror 16: 0x4a7f83ef (stale) Mirror 17: 0x4a7f83ef (stale) Mirror 19: 0x4a7f83ef (stale) Mirror 20: 0x4a7f83ef (stale) Mirror 22: 0x4a7f83ef (stale) Mirror 23: 0x4a7f83ef (stale) Mirror 24: 0x4a7f83ef (stale) Mirror 25: 0x4a7f83ef (stale) Mirror 26: 0x4a7f83ef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd09f: skipped 000682 1787561651.516771 trunc from 0x00d09f to 0x021ba0 (0x14b02 bytes) 000684 1787561651.532383 trunc from 0x021ba1 to 0x018936 (0x926a bytes) 000686 1787561651.564863 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x18937): Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18937): Mirror 11: 0x17b763ce (stale) Mirror 14: 0x17b763ce (stale) Mirror 16: 0x17b763ce (stale) Mirror 17: 0x17b763ce (stale) Mirror 19: 0x17b763ce (stale) Mirror 20: 0x17b763ce (stale) Mirror 22: 0x17b763ce (stale) Mirror 23: 0x17b763ce (stale) Mirror 24: 0x17b763ce (stale) Mirror 25: 0x17b763ce (stale) Mirror 26: 0x5e0995d6 (stale) CRC-32 checksum value for chunk [0, 0x18937): Mirror 1: 0x59aeb828 Mirror 27: 0x59aeb828 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18937: skipped 000687 1787561652.789778 trunc from 0x018937 to 0x02473c (0xbe06 bytes) 000689 1787561652.971506 MIRROR_SPLIT mirror 3 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2473d): Mirror 11: 0x24dd1155 (stale) Mirror 14: 0x24dd1155 (stale) Mirror 17: 0x24dd1155 (stale) Mirror 19: 0x24dd1155 (stale) Mirror 20: 0x24dd1155 (stale) Mirror 22: 0x24dd1155 (stale) Mirror 23: 0x24dd1155 (stale) Mirror 24: 0x24dd1155 (stale) Mirror 25: 0x4e7a3e16 (stale) Mirror 26: 0x7075a8ea (stale) Mirror 27: 0x618e71e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2473d): Mirror 11: 0x24dd1155 (stale) Mirror 14: 0x24dd1155 (stale) Mirror 17: 0x24dd1155 (stale) Mirror 19: 0x24dd1155 (stale) Mirror 20: 0x24dd1155 (stale) Mirror 22: 0x24dd1155 (stale) Mirror 23: 0x24dd1155 (stale) Mirror 24: 0x24dd1155 (stale) Mirror 25: 0x4e7a3e16 (stale) Mirror 26: 0x7075a8ea (stale) Mirror 27: 0x618e71e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2473d: skipped 000690 1787561653.754612 read 0x016b00 thru 0x021cb4 (0xb1b5 bytes) 000691 1787561653.772379 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2473d): Mirror 14: 0x24dd1155 (stale) Mirror 17: 0x24dd1155 (stale) Mirror 19: 0x24dd1155 (stale) Mirror 20: 0x24dd1155 (stale) Mirror 22: 0x24dd1155 (stale) Mirror 23: 0x24dd1155 (stale) Mirror 24: 0x24dd1155 (stale) Mirror 25: 0x4e7a3e16 (stale) Mirror 26: 0x7075a8ea (stale) Mirror 27: 0x618e71e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2473d): Mirror 14: 0x24dd1155 (stale) Mirror 17: 0x24dd1155 (stale) Mirror 19: 0x24dd1155 (stale) Mirror 20: 0x24dd1155 (stale) Mirror 22: 0x24dd1155 (stale) Mirror 23: 0x24dd1155 (stale) Mirror 24: 0x24dd1155 (stale) Mirror 25: 0x4e7a3e16 (stale) Mirror 26: 0x7075a8ea (stale) Mirror 27: 0x618e71e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2473d: skipped 000692 1787561654.661862 MIRROR_RESYNC 11 mirrors 000697 1787561657.934659 mapwrite 0x036392 thru 0x03b221 (0x4e90 bytes) 000698 1787561658.267035 mapread 0x0322db thru 0x03776c (0x5492 bytes) 000699 1787561658.268000 read 0x004648 thru 0x010879 (0xc232 bytes) 000701 1787561658.288252 mapread 0x03a994 thru 0x03b221 (0x88e bytes) 000703 1787561658.298170 MIRROR_RESYNC 11 mirrors 000704 1787561662.970717 mapwrite 0x034136 thru 0x03ffff (0xbeca bytes) 000706 1787561663.583290 mapwrite 0x00a1c4 thru 0x01166e (0x74ab bytes) 000707 1787561663.859702 write 0x027f64 thru 0x031349 (0x93e6 bytes) 000709 1787561663.874256 write 0x02f857 thru 0x037a8c (0x8236 bytes) 000710 1787561663.876100 mapwrite 0x02b2bc thru 0x033498 (0x81dd bytes) 000713 1787561663.995808 write 0x00bd17 thru 0x010db4 (0x509e bytes) 000716 1787561663.998966 read 0x023505 thru 0x0266a2 (0x319e bytes) 000717 1787561664.017473 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xe175b570 (stale) Mirror 17: 0xe175b570 (stale) Mirror 20: 0xe175b570 (stale) Mirror 22: 0xe175b570 (stale) Mirror 23: 0xe175b570 (stale) Mirror 24: 0xe175b570 (stale) Mirror 25: 0xe175b570 (stale) Mirror 26: 0xe175b570 (stale) Mirror 27: 0xe175b570 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xe175b570 (stale) Mirror 17: 0xe175b570 (stale) Mirror 20: 0xe175b570 (stale) Mirror 22: 0xe175b570 (stale) Mirror 23: 0xe175b570 (stale) Mirror 24: 0xe175b570 (stale) Mirror 25: 0xe175b570 (stale) Mirror 26: 0xe175b570 (stale) Mirror 27: 0xe175b570 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000718 1787561664.882236 write 0x03fb46 thru 0x03ffff (0x4ba bytes) 000720 1787561664.890779 read 0x01453a thru 0x016767 (0x222e bytes) 000721 1787561664.900061 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xe175b570 (stale) Mirror 17: 0xe175b570 (stale) Mirror 20: 0xe175b570 (stale) Mirror 22: 0xe175b570 (stale) Mirror 23: 0xe175b570 (stale) Mirror 24: 0xe175b570 (stale) Mirror 25: 0xe175b570 (stale) Mirror 26: 0xe175b570 (stale) Mirror 27: 0xe175b570 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x86b23c59 Mirror 28: 0x86b23c59 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000722 1787561666.083547 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xe175b570 (stale) Mirror 17: 0xe175b570 (stale) Mirror 20: 0xe175b570 (stale) Mirror 22: 0xe175b570 (stale) Mirror 23: 0xe175b570 (stale) Mirror 24: 0xe175b570 (stale) Mirror 25: 0xe175b570 (stale) Mirror 26: 0xe175b570 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xe175b570 (stale) Mirror 17: 0xe175b570 (stale) Mirror 20: 0xe175b570 (stale) Mirror 22: 0xe175b570 (stale) Mirror 23: 0xe175b570 (stale) Mirror 24: 0xe175b570 (stale) Mirror 25: 0xe175b570 (stale) Mirror 26: 0xe175b570 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x86b23c59 Mirror 28: 0x86b23c59 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000724 1787561667.610628 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xe175b570 (stale) Mirror 17: 0xe175b570 (stale) Mirror 20: 0xe175b570 (stale) Mirror 22: 0xe175b570 (stale) Mirror 23: 0xe175b570 (stale) Mirror 24: 0xe175b570 (stale) Mirror 26: 0xe175b570 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xe175b570 (stale) Mirror 17: 0xe175b570 (stale) Mirror 20: 0xe175b570 (stale) Mirror 22: 0xe175b570 (stale) Mirror 23: 0xe175b570 (stale) Mirror 24: 0xe175b570 (stale) Mirror 26: 0xe175b570 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x86b23c59 Mirror 28: 0x86b23c59 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000726 1787561668.735778 trunc from 0x040000 to 0x036bd0 (0x942f bytes) 000727 1787561668.884458 read 0x012e13 thru 0x01fc44 (0xce32 bytes) 000728 1787561668.898283 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x36bd1): Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36bd1): Mirror 14: 0xd45d9851 (stale) Mirror 17: 0xd45d9851 (stale) Mirror 20: 0xd45d9851 (stale) Mirror 22: 0xd45d9851 (stale) Mirror 23: 0xd45d9851 (stale) Mirror 24: 0xd45d9851 (stale) Mirror 26: 0xd45d9851 (stale) Mirror 28: 0x3899a5a7 (stale) CRC-32 checksum value for chunk [0, 0x36bd1): Mirror 1: 0x3899a5a7 Mirror 29: 0x3899a5a7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36bd1: skipped 000730 1787561670.278547 trunc from 0x036bd1 to 0x00f7f3 (0x273dd bytes) 000733 1787561670.435404 MIRROR_RESYNC 10 mirrors 000734 1787561672.249278 mapread 0x001de7 thru 0x00f7f3 (0xda0d bytes) 000735 1787561672.282002 MIRROR_RESYNC 10 mirrors 000736 1787561672.959017 mapread 0x00e315 thru 0x00f7f3 (0x14df bytes) 000740 1787561672.962408 mapwrite 0x00caa5 thru 0x013247 (0x67a3 bytes) 000742 1787561673.123633 mapwrite 0x02dc93 thru 0x034780 (0x6aee bytes) 000744 1787561673.255243 write 0x00f05b thru 0x015ff2 (0x6f98 bytes) 000745 1787561673.267993 mapwrite 0x0314f0 thru 0x037bcb (0x66dc bytes) 000746 1787561673.363610 write 0x02ddb9 thru 0x03cda4 (0xefec bytes) 000752 1787561673.398669 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3cda5): Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cda5): Mirror 14: 0x86bdd0e6 (stale) Mirror 17: 0x86bdd0e6 (stale) Mirror 20: 0x86bdd0e6 (stale) Mirror 22: 0x86bdd0e6 (stale) Mirror 23: 0x86bdd0e6 (stale) Mirror 24: 0x86bdd0e6 (stale) Mirror 26: 0x86bdd0e6 (stale) Mirror 28: 0x86bdd0e6 (stale) Mirror 29: 0x86bdd0e6 (stale) CRC-32 checksum value for chunk [0, 0x3cda5): Mirror 1: 0x50aa8336 Mirror 30: 0x50aa8336 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cda5: skipped 000753 1787561674.602723 MIRROR_RESYNC 11 mirrors 000754 1787561676.678091 trunc from 0x03cda5 to 0x03ef9b (0x21f7 bytes) 000756 1787561676.843318 mapwrite 0x01a597 thru 0x023e31 (0x989b bytes) 000757 1787561676.926906 read 0x010f40 thru 0x0197b8 (0x8879 bytes) 000759 1787561676.937175 mapwrite 0x025fdb thru 0x02c09b (0x60c1 bytes) 000763 1787561677.001688 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3ef9c): Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef9c): Mirror 14: 0x2ca0621c (stale) Mirror 17: 0x2ca0621c (stale) Mirror 20: 0x2ca0621c (stale) Mirror 22: 0x2ca0621c (stale) Mirror 23: 0x2ca0621c (stale) Mirror 24: 0x2ca0621c (stale) Mirror 26: 0x2ca0621c (stale) Mirror 28: 0x2ca0621c (stale) Mirror 29: 0x2ca0621c (stale) Mirror 30: 0x2ca0621c (stale) CRC-32 checksum value for chunk [0, 0x3ef9c): Mirror 1: 0xf0ea8481 Mirror 31: 0xf0ea8481 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef9c: skipped 000765 1787561678.376111 trunc from 0x03ef9c to 0x013fb5 (0x2afe6 bytes) 000766 1787561678.518975 read 0x002f07 thru 0x007478 (0x4572 bytes) 000768 1787561678.545293 trunc from 0x013fb6 to 0x012bf4 (0x13c1 bytes) 000770 1787561678.576688 mapwrite 0x007dbf thru 0x010645 (0x8887 bytes) 000772 1787561678.672282 mapwrite 0x03fbf0 thru 0x03ffff (0x410 bytes) 000773 1787561678.767402 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x337cba0f (stale) Mirror 17: 0x337cba0f (stale) Mirror 20: 0x337cba0f (stale) Mirror 22: 0x337cba0f (stale) Mirror 23: 0x337cba0f (stale) Mirror 24: 0x337cba0f (stale) Mirror 26: 0x337cba0f (stale) Mirror 28: 0x337cba0f (stale) Mirror 29: 0x337cba0f (stale) Mirror 30: 0x337cba0f (stale) Mirror 31: 0x2f650ce0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x721fcf75 Mirror 32: 0x721fcf75 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000774 1787561680.304936 write 0x0028e1 thru 0x003b9d (0x12bd bytes) 000775 1787561680.436512 MIRROR_RESYNC 13 mirrors 000776 1787561683.975547 trunc from 0x040000 to 0x01f337 (0x20cc8 bytes) 000778 1787561684.100413 write 0x02eaa4 thru 0x033b99 (0x50f6 bytes) 000779 1787561684.107028 write 0x03bebd thru 0x03ffff (0x4143 bytes) 000780 1787561684.116330 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xab78f221 (stale) Mirror 17: 0xab78f221 (stale) Mirror 20: 0xab78f221 (stale) Mirror 22: 0xab78f221 (stale) Mirror 23: 0xab78f221 (stale) Mirror 24: 0xab78f221 (stale) Mirror 26: 0xab78f221 (stale) Mirror 28: 0xab78f221 (stale) Mirror 30: 0xab78f221 (stale) Mirror 31: 0xab78f221 (stale) Mirror 32: 0xab78f221 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xab78f221 (stale) Mirror 17: 0xab78f221 (stale) Mirror 20: 0xab78f221 (stale) Mirror 22: 0xab78f221 (stale) Mirror 23: 0xab78f221 (stale) Mirror 24: 0xab78f221 (stale) Mirror 26: 0xab78f221 (stale) Mirror 28: 0xab78f221 (stale) Mirror 30: 0xab78f221 (stale) Mirror 31: 0xab78f221 (stale) Mirror 32: 0xab78f221 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000782 1787561685.085971 write 0x024508 thru 0x02a205 (0x5cfe bytes) 000784 1787561685.097878 mapread 0x03a713 thru 0x03ffff (0x58ed bytes) 000785 1787561685.118246 mapread 0x03a369 thru 0x03ffff (0x5c97 bytes) 000786 1787561685.121086 trunc from 0x040000 to 0x02e27d (0x11d82 bytes) 000787 1787561685.139030 mapwrite 0x014553 thru 0x016fdc (0x2a8a bytes) 000788 1787561685.193175 read 0x010e90 thru 0x01c041 (0xb1b2 bytes) 000789 1787561685.193963 write 0x03f3a2 thru 0x03ffff (0xc5e bytes) 000796 1787561685.199937 mapread 0x030fc9 thru 0x03a665 (0x969d bytes) 000797 1787561685.220875 mapread 0x019c7a thru 0x02899c (0xed23 bytes) 000798 1787561685.232065 mapread 0x00cfd8 thru 0x00d618 (0x641 bytes) 000799 1787561685.242531 write 0x021e6a thru 0x0313a6 (0xf53d bytes) 000800 1787561685.250074 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xab78f221 (stale) Mirror 17: 0xab78f221 (stale) Mirror 20: 0xab78f221 (stale) Mirror 23: 0xab78f221 (stale) Mirror 24: 0xab78f221 (stale) Mirror 26: 0xab78f221 (stale) Mirror 28: 0xab78f221 (stale) Mirror 30: 0xab78f221 (stale) Mirror 31: 0xab78f221 (stale) Mirror 32: 0xab78f221 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xab78f221 (stale) Mirror 17: 0xab78f221 (stale) Mirror 20: 0xab78f221 (stale) Mirror 23: 0xab78f221 (stale) Mirror 24: 0xab78f221 (stale) Mirror 26: 0xab78f221 (stale) Mirror 28: 0xab78f221 (stale) Mirror 30: 0xab78f221 (stale) Mirror 31: 0xab78f221 (stale) Mirror 32: 0xab78f221 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000802 1787561685.912985 trunc from 0x040000 to 0x00f452 (0x30bad bytes) 000804 1787561685.931188 mapread 0x00ebce thru 0x00f452 (0x885 bytes) 000805 1787561685.944227 mapwrite 0x028df5 thru 0x0307f2 (0x79fe bytes) 000806 1787561686.036054 MIRROR_RESYNC 11 mirrors 000807 1787561689.324034 trunc from 0x0307f3 to 0x03355e (0x2d6c bytes) 000809 1787561689.435075 mapwrite 0x02e1ba thru 0x032b88 (0x49cf bytes) 000810 1787561689.501630 MIRROR_RESYNC 11 mirrors 000813 1787561692.078962 mapread 0x018be6 thru 0x023bce (0xafe9 bytes) 000814 1787561692.104087 write 0x01ccf3 thru 0x01fdfb (0x3109 bytes) 000816 1787561692.186930 write 0x031029 thru 0x0324e0 (0x14b8 bytes) 000820 1787561692.203307 MIRROR_RESYNC 11 mirrors 000821 1787561694.095780 mapread 0x0258de thru 0x02da91 (0x81b4 bytes) 000822 1787561694.121486 mapread 0x02c5e0 thru 0x03355e (0x6f7f bytes) 000824 1787561694.136966 trunc from 0x03355f to 0x005302 (0x2e25c bytes) 000825 1787561694.224896 mapwrite 0x0047b8 thru 0x00f0a9 (0xa8f2 bytes) 000826 1787561694.369245 mapread 0x007c7d thru 0x00f0a9 (0x742d bytes) 000827 1787561694.381160 MIRROR_RESYNC 11 mirrors 000828 1787561696.263999 read 0x0055b1 thru 0x00cf7a (0x79ca bytes) 000829 1787561696.287803 trunc from 0x00f0aa to 0x03b28d (0x2c1e4 bytes) 000830 1787561696.424033 mapwrite 0x0184f3 thru 0x0265ba (0xe0c8 bytes) 000831 1787561696.541451 mapwrite 0x03ab2d thru 0x03ffff (0x54d3 bytes) 000832 1787561696.668881 write 0x01a401 thru 0x026b3b (0xc73b bytes) 000833 1787561696.672431 read 0x02d0c3 thru 0x037c36 (0xab74 bytes) 000835 1787561696.693388 mapwrite 0x02a64f thru 0x034075 (0x9a27 bytes) 000838 1787561696.762224 mapwrite 0x03e713 thru 0x03ffff (0x18ed bytes) 000839 1787561696.819366 mapwrite 0x012743 thru 0x012fc9 (0x887 bytes) 000842 1787561696.894758 trunc from 0x040000 to 0x00c959 (0x336a6 bytes) 000843 1787561696.943538 mapwrite 0x0327c9 thru 0x03c20b (0x9a43 bytes) 000844 1787561697.131405 read 0x0335fa thru 0x0388b0 (0x52b7 bytes) 000847 1787561697.135388 mapwrite 0x03496a thru 0x03522a (0x8c1 bytes) 000848 1787561697.199658 mapread 0x03b5b6 thru 0x03c20b (0xc56 bytes) 000849 1787561697.200477 trunc from 0x03c20c to 0x001a37 (0x3a7d4 bytes) 000850 1787561697.248920 mapwrite 0x02e05c thru 0x030b67 (0x2b0c bytes) 000851 1787561697.383739 trunc from 0x030b68 to 0x037eed (0x7386 bytes) 000852 1787561697.400019 mapwrite 0x002c26 thru 0x0100b7 (0xd492 bytes) 000853 1787561697.446623 read 0x00f79e thru 0x01b183 (0xb9e6 bytes) 000854 1787561697.452313 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37eee): Mirror 14: 0x43bac373 (stale) Mirror 17: 0x43bac373 (stale) Mirror 20: 0x43bac373 (stale) Mirror 23: 0x43bac373 (stale) Mirror 24: 0x43bac373 (stale) Mirror 26: 0x43bac373 (stale) Mirror 30: 0x43bac373 (stale) Mirror 31: 0x43bac373 (stale) Mirror 32: 0x43bac373 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37eee): Mirror 14: 0x43bac373 (stale) Mirror 17: 0x43bac373 (stale) Mirror 20: 0x43bac373 (stale) Mirror 23: 0x43bac373 (stale) Mirror 24: 0x43bac373 (stale) Mirror 26: 0x43bac373 (stale) Mirror 30: 0x43bac373 (stale) Mirror 31: 0x43bac373 (stale) Mirror 32: 0x43bac373 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37eee: skipped 000857 1787561698.018473 mapread 0x032734 thru 0x037eed (0x57ba bytes) 000858 1787561698.049296 trunc from 0x037eee to 0x005133 (0x32dba bytes) 000863 1787561698.085814 trunc from 0x005134 to 0x01746e (0x1233b bytes) 000864 1787561698.111898 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x1746f): Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1746f): Mirror 14: 0xcee04128 (stale) Mirror 17: 0xcee04128 (stale) Mirror 20: 0xcee04128 (stale) Mirror 23: 0xcee04128 (stale) Mirror 24: 0xcee04128 (stale) Mirror 26: 0xcee04128 (stale) Mirror 30: 0xcee04128 (stale) Mirror 31: 0xcee04128 (stale) Mirror 32: 0xcee04128 (stale) CRC-32 checksum value for chunk [0, 0x1746f): Mirror 1: 0xd58d5ff2 Mirror 33: 0xd58d5ff2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1746f: skipped 000865 1787561699.425538 mapwrite 0x00ba75 thru 0x01036d (0x48f9 bytes) 000868 1787561699.626860 mapwrite 0x01f275 thru 0x02b9e4 (0xc770 bytes) 000870 1787561699.743178 mapwrite 0x02b910 thru 0x031295 (0x5986 bytes) 000871 1787561699.831837 trunc from 0x031296 to 0x01cfc6 (0x142cf bytes) 000872 1787561699.857233 mapread 0x0194d1 thru 0x01cfc6 (0x3af6 bytes) 000876 1787561699.863078 write 0x02b4b2 thru 0x030fe5 (0x5b34 bytes) 000878 1787561699.871458 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x30fe6): Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30fe6): Mirror 14: 0x5621a4f9 (stale) Mirror 17: 0x5621a4f9 (stale) Mirror 20: 0x5621a4f9 (stale) Mirror 23: 0x5621a4f9 (stale) Mirror 24: 0x5621a4f9 (stale) Mirror 26: 0x5621a4f9 (stale) Mirror 30: 0x5621a4f9 (stale) Mirror 31: 0x5621a4f9 (stale) Mirror 32: 0x5621a4f9 (stale) Mirror 33: 0x8a0870f1 (stale) CRC-32 checksum value for chunk [0, 0x30fe6): Mirror 1: 0xc0edd5f7 Mirror 34: 0xc0edd5f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30fe6: skipped 000881 1787561701.413100 mapwrite 0x019844 thru 0x023281 (0x9a3e bytes) 000883 1787561701.714855 mapwrite 0x01123d thru 0x0169fa (0x57be bytes) 000884 1787561701.782261 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x195:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30fe6): Mirror 14: 0x5621a4f9 (stale) Mirror 17: 0x5621a4f9 (stale) Mirror 23: 0x5621a4f9 (stale) Mirror 24: 0x5621a4f9 (stale) Mirror 26: 0x5621a4f9 (stale) Mirror 30: 0x5621a4f9 (stale) Mirror 31: 0x5621a4f9 (stale) Mirror 32: 0x5621a4f9 (stale) Mirror 33: 0x8a0870f1 (stale) Mirror 34: 0xc0edd5f7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30fe6): Mirror 14: 0x5621a4f9 (stale) Mirror 17: 0x5621a4f9 (stale) Mirror 23: 0x5621a4f9 (stale) Mirror 24: 0x5621a4f9 (stale) Mirror 26: 0x5621a4f9 (stale) Mirror 30: 0x5621a4f9 (stale) Mirror 31: 0x5621a4f9 (stale) Mirror 32: 0x5621a4f9 (stale) Mirror 33: 0x8a0870f1 (stale) Mirror 34: 0xc0edd5f7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30fe6: skipped 000885 1787561702.350818 read 0x02c005 thru 0x030fe5 (0x4fe1 bytes) 000886 1787561702.361525 mapwrite 0x02d56d thru 0x033630 (0x60c4 bytes) 000887 1787561702.520463 write 0x029823 thru 0x02aff6 (0x17d4 bytes) 000888 1787561702.555356 mapwrite 0x027e89 thru 0x02c6a1 (0x4819 bytes) 000891 1787561702.620548 write 0x011ccf thru 0x01d4c2 (0xb7f4 bytes) 000892 1787561702.630859 mapread 0x014ad1 thru 0x0214f3 (0xca23 bytes) 000895 1787561702.638216 mapread 0x0287c3 thru 0x02d627 (0x4e65 bytes) 000898 1787561702.640378 read 0x0172af thru 0x020751 (0x94a3 bytes) 000900 1787561702.642224 read 0x02ed2c thru 0x033630 (0x4905 bytes) 000902 1787561702.643797 read 0x02dfec thru 0x032d58 (0x4d6d bytes) 000904 1787561702.645396 write 0x01d2ab thru 0x02bc2b (0xe981 bytes) 000907 1787561702.648005 mapread 0x020d77 thru 0x0245a0 (0x382a bytes) 000908 1787561702.649756 write 0x01b2ab thru 0x02a8bc (0xf612 bytes) 000909 1787561702.652815 write 0x032276 thru 0x038e2d (0x6bb8 bytes) 000910 1787561702.654495 mapread 0x037969 thru 0x038e2d (0x14c5 bytes) 000911 1787561702.656041 read 0x01ad9d thru 0x025bdd (0xae41 bytes) 000913 1787561702.658452 mapwrite 0x03ef85 thru 0x03ffff (0x107b bytes) 000914 1787561702.744792 write 0x01ec7d thru 0x02d80a (0xeb8e bytes) 000917 1787561702.749519 mapwrite 0x005612 thru 0x009261 (0x3c50 bytes) 000919 1787561702.797255 trunc from 0x040000 to 0x02f1d6 (0x10e29 bytes) 000923 1787561702.819833 trunc from 0x02f1d7 to 0x01e4f9 (0x10cdd bytes) 000924 1787561702.841893 mapwrite 0x01d7ed thru 0x026175 (0x8989 bytes) 000928 1787561702.941801 read 0x00c1a5 thru 0x01b9ed (0xf849 bytes) 000929 1787561702.943905 read 0x01c6f5 thru 0x026175 (0x9a81 bytes) 000931 1787561702.945498 trunc from 0x026176 to 0x0219a6 (0x47cf bytes) 000933 1787561702.965417 write 0x01868d thru 0x02014c (0x7ac0 bytes) 000937 1787561702.969724 mapwrite 0x035607 thru 0x03ce4b (0x7845 bytes) 000938 1787561703.041416 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3ce4c): Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ce4c): Mirror 14: 0x50b4458d (stale) Mirror 17: 0x50b4458d (stale) Mirror 23: 0x50b4458d (stale) Mirror 24: 0x50b4458d (stale) Mirror 26: 0x50b4458d (stale) Mirror 30: 0x50b4458d (stale) Mirror 31: 0x50b4458d (stale) Mirror 32: 0x50b4458d (stale) Mirror 33: 0xd5e74607 (stale) Mirror 34: 0x4a77a098 (stale) CRC-32 checksum value for chunk [0, 0x3ce4c): Mirror 1: 0xe9f5061 Mirror 35: 0xe9f5061 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ce4c: skipped 000941 1787561704.199350 mapwrite 0x012654 thru 0x017dcc (0x5779 bytes) 000942 1787561704.418848 mapread 0x02e51d thru 0x03c46e (0xdf52 bytes) 000944 1787561704.453197 mapread 0x004dcc thru 0x0131ba (0xe3ef bytes) 000945 1787561704.467429 mapwrite 0x03a3a6 thru 0x03c3fc (0x2057 bytes) 000946 1787561704.548195 trunc from 0x03ce4c to 0x01aa3e (0x2240d bytes) 000947 1787561704.580123 trunc from 0x01aa3f to 0x038cfc (0x1e2be bytes) 000949 1787561704.609450 read 0x028278 thru 0x02cf92 (0x4d1b bytes) 000950 1787561704.636236 read 0x0096a7 thru 0x01149b (0x7df5 bytes) 000951 1787561704.643025 write 0x002ed8 thru 0x0047e0 (0x1909 bytes) 000952 1787561704.658814 MIRROR_RESYNC 12 mirrors 000954 1787561707.572180 write 0x00b166 thru 0x016051 (0xaeec bytes) 000957 1787561707.674464 trunc from 0x038cfd to 0x02b017 (0xdce5 bytes) 000958 1787561707.713845 read 0x01fdc5 thru 0x028348 (0x8584 bytes) 000959 1787561707.737469 trunc from 0x02b018 to 0x00a0fc (0x20f1b bytes) 000960 1787561707.772343 mapwrite 0x003c66 thru 0x0135c8 (0xf963 bytes) 000964 1787561707.964736 write 0x00e6aa thru 0x0130cd (0x4a24 bytes) 000965 1787561707.966182 mapwrite 0x0131db thru 0x016473 (0x3299 bytes) 000968 1787561708.061710 mapwrite 0x0220e8 thru 0x02c31d (0xa236 bytes) 000969 1787561708.166623 trunc from 0x02c31e to 0x028493 (0x3e8a bytes) 000971 1787561708.192830 write 0x02b794 thru 0x032dd9 (0x7646 bytes) 000972 1787561708.199621 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x32dda): Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32dda): Mirror 14: 0xc882f73 (stale) Mirror 17: 0xc882f73 (stale) Mirror 23: 0xc882f73 (stale) Mirror 24: 0xc882f73 (stale) Mirror 26: 0xc882f73 (stale) Mirror 30: 0xc882f73 (stale) Mirror 31: 0xc882f73 (stale) Mirror 32: 0xc882f73 (stale) Mirror 33: 0xc882f73 (stale) Mirror 34: 0xc882f73 (stale) Mirror 35: 0xc882f73 (stale) CRC-32 checksum value for chunk [0, 0x32dda): Mirror 1: 0x6539d7d8 Mirror 36: 0x6539d7d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32dda: skipped 000975 1787561709.891990 mapread 0x0150eb thru 0x01b64a (0x6560 bytes) 000977 1787561709.993551 mapwrite 0x0225cb thru 0x02a815 (0x824b bytes) 000980 1787561710.211526 mapread 0x007e98 thru 0x00d5e6 (0x574f bytes) 000983 1787561710.229473 write 0x03989d thru 0x03ffff (0x6763 bytes) 000984 1787561710.232849 write 0x036b0a thru 0x038b31 (0x2028 bytes) 000986 1787561710.245487 trunc from 0x040000 to 0x01148b (0x2eb74 bytes) 000988 1787561710.281539 mapwrite 0x00e681 thru 0x018572 (0x9ef2 bytes) 000989 1787561710.360501 trunc from 0x018573 to 0x02637f (0xde0d bytes) truncating to largest ever: 0x3fb94 000992 1787561710.389547 trunc from 0x026380 to 0x03fb93 (0x19814 bytes) 000995 1787561710.426775 MIRROR_RESYNC 13 mirrors 000997 1787561713.002532 mapread 0x02dd91 thru 0x037012 (0x9282 bytes) 001000 1787561713.042360 trunc from 0x03fb94 to 0x023dde (0x1bdb5 bytes) All operations completed A-OK! PASS 70a (165s) == sanity-flr test 71: check mirror extend parent fid ==== 04:55:18 (1787561718) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 04:55:22 (1787561722) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.381302 s, 5.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231264 s, 4.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2de:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 04:55:33 (1787561733) Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre2 Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3635520 .. Extending file size to 4728960 .. done Extending file size to 5690528 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8746144 .. Extending file size to 7222816 .. Extending file size to 8663936 .. Extending file size to 8996896 .. Extending file size to 8684000 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8781472 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9055232 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57408 57409 57411 57413 57415 done 192.168.203.154@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg354-client.virtnet /mnt/lustre2 (opts:) 192.168.203.154@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg354-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (91s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 04:57:04 (1787561824) Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre2 Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2783680 .. Extending file size to 898592 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6416288 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6957920 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6799200 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7181792 .. Extending file size to 7817504 .. Extending file size to 7950464 .. Extending file size to 8388416 .. Extending file size to 8588064 .. Extending file size to 7993792 .. Extending file size to 8351456 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8602592 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8390272 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58890 58891 58892 58894 58895 done 192.168.203.154@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg354-client.virtnet /mnt/lustre2 (opts:) 192.168.203.154@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg354-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (84s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 04:58:28 (1787561908) Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139881 s, 15.0 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.594003 s, 3.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x5346b389 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.154@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg354-client.virtnet /mnt/lustre2 (opts:) PASS 200c (27s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 04:58:57 (1787561937) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0953137 s, 22.0 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 04:59:05 (1787561945) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e0:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f4:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f5:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x2d8:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 04:59:13 (1787561953) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.386667 s, 5.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 04:59:23 (1787561963) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.354929 s, 5.9 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2db:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dc:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 04:59:34 (1787561974) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.340378 s, 6.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2dd:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2de:0x0] } PASS 204c (39s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:00:14 (1787562014) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg354-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162795 s, 6.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fa:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2df:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fb:0x0] } PASS 204d (61s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:01:15 (1787562075) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg354-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.181465 s, 11.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e1:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (65s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:02:20 (1787562140) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg354-server mds-ost sync done. Creating new pool test_204f oleg354-server: Pool lustre.test_204f created Adding targets to pool oleg354-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg354-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.351268 s, 6.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2fe:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2e2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg354-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg354-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg354-server: Pool lustre.test_204f destroyed PASS 204f (86s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:03:46 (1787562226) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:03:54 (1787562234) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:04:04 (1787562244) oleg354-server: Pool lustre.test_206 created Adding targets to pool oleg354-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg354-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg354-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg354-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg354-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:04:30 (1787562270) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644951 s, 16.3 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435411 s, 24.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x9727c405 Mirror 3: 0x9727c405 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:04:42 (1787562282) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:04:45 (1787562285) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:04:49 (1787562289) Starting client: oleg354-client.virtnet: -o user_xattr,flock 192.168.203.154@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.119345 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x750000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2410000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2410000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2410000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2470000 Read this (4 bytes): BBBB 192.168.203.154@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg354-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:05:29 (1787562329) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0477489 s, 172 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:05:36 (1787562336) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579069 s, 18.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x205 lmm_fid: [0x200000404:0x205:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x205 lmm_fid: [0x200000404:0x205:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834437 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-24 05:05:38.000000000 -0400 Modify: 2026-08-24 05:05:38.000000000 -0400 Change: 2026-08-24 05:05:39.000000000 -0400 Birth: 2026-08-24 05:05:38.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:05:45 (1787562345) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:05:49 (1787562349) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 1.0245 s, 40.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0965723 s, 42.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 24 05:05 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x207:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 24 05:05 /mnt/lustre/f211.sanity-flr PASS 211 (13s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 05:06:02 (1787562362) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 05:06:12 (1787562372) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1874 sec =========== 05:06:23 (1787562383) === sanity-flr: start cleanup 05:06:25 (1787562385) === === sanity-flr: finish cleanup 05:06:32 (1787562392) ===