-----============= acceptance-small: sanity-flr ============----- Mon May 25 01:39:01 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg250-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 01:39:22 (1779687562) === oleg250-client.virtnet: executing check_config_client /mnt/lustre oleg250-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg250-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b500aa91800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b500aa91800.idle_timeout=debug disable quota as required oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 01:39:57 (1779687597) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7564] == sanity-flr test 0a: lfs mirror create with -N option == 01:40:00 (1779687600) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:40:21 (1779687621) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:40:25 (1779687625) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:40:30 (1779687630) PASS 0d (32s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:41:01 (1779687661) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:41:05 (1779687665) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 01:41:09 (1779687669) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b500aa91800.stats=clear osc.lustre-OST0001-osc-ffff9b500aa91800.stats=clear PASS 0g (11s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:41:20 (1779687680) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:41:34 (1779687694) PASS 0j (12s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:41:45 (1779687705) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:41:49 (1779687709) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:41:59 (1779687719) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 01:42:09 (1779687729) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:42:17 (1779687737) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:42:21 (1779687741) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:42:30 (1779687750) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:42:42 (1779687762) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:42:50 (1779687770) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 2.27412 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162206 s, 6.5 MB/s PASS 21 (14s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:43:04 (1779687784) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.631661 s, 33.2 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 01:43:15 (1779687795) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:43:26 (1779687806) Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b500aa91800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b500aa91800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b500aa91800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b500aa91800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg250-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (85s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:44:51 (1779687891) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.158428 s, 13.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b500aa91800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b500aa91800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:45:34 (1779687934) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b500aa91800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b500aa91800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b500aa91800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b500aa91800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg250-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (81s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:46:55 (1779688015) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 6.16536 s, 8.5 MB/s -rw-r--r-- 1 root root 52428800 May 25 01:47 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.219s user 0m0.001s sys 0m0.182s Stopping /mnt/lustre-ost1 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b500aa91800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b500aa91800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.259s user 0m0.002s sys 0m0.159s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg250-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg250-server oleg250-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b500aa91800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b500aa91800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m2.036s user 0m0.001s sys 0m0.192s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg250-server: oleg250-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg250-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg250-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b500aa91800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b500aa91800.ost_server_uuid in FULL state after 0 sec PASS 33b (103s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:48:38 (1779688118) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:48:42 (1779688122) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:48:47 (1779688127) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (5s) == sanity-flr test 35: allow to write to mirrored files == 01:48:51 (1779688131) PASS 35 (10s) == sanity-flr test 36a: write to mirrored files ========== 01:49:01 (1779688141) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.7996 s, 15.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:49:17 (1779688157) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624424 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0511449 s, 20.5 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:49:26 (1779688166) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:49:36 (1779688176) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00104173 s, 983 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00155538 s, 658 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00118198 s, 866 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00235633 s, 435 kB/s flv=10 olv=10 PASS 36d (14s) == sanity-flr test 37: mirror I/O API verification ======= 01:49:50 (1779688190) create test files with size 15756739 .. fe6bdc940f0e412e52b6bb431c621774 - 489d7a46273aca75310d6d54a3173acd - 697db8824498ed114c46beac637c3f2a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg250-client.virtnet /mnt/lustre (opts:) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre PASS 37 (31s) == sanity-flr test 38: resync ============================ 01:50:21 (1779688221) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:50:40 (1779688240) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:50:48 (1779688248) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158232 s, 6.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0380154 s, 27.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 01:50:57 (1779688257) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138844 s, 15.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.401428 s, 10.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.353468 s, 5.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 01:51:17 (1779688277) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 01:51:20 (1779688280) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:51:24 (1779688284) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321596 s, 32.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 01:51:35 (1779688295) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:51:39 (1779688299) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:51:52 (1779688312) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.366646 s, 28.6 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg250-client.virtnet /mnt/lustre (opts:) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (16s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:52:08 (1779688328) llite.lustre-ffff9b500b644000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.159219 s, 65.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg250-client.virtnet /mnt/lustre (opts:) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9b50067ec000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:52:23 (1779688343) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (43s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:53:06 (1779688386) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (36s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:53:43 (1779688423) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 01:53:46 (1779688426) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.266159 s, 11.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.213105 s, 14.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa6c019ef (nosync,stale) Mirror 2: 0xa6c019ef (nosync,stale) ** verify mirror 2 doesn't change original checksum: bd141bc9d7ad4b132e4894b24d640ed4 - mirror 1 checksum: c183349ec9e184ff7889ec635700c3ad - mirror 2 checksum: bd141bc9d7ad4b132e4894b24d640ed4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon May 25 01:53:49 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon May 25 01:53:49 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: bd141bc9d7ad4b132e4894b24d640ed4 - mirror 1 checksum: c183349ec9e184ff7889ec635700c3ad - mirror 2 checksum: c183349ec9e184ff7889ec635700c3ad - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon May 25 01:53:49 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon May 25 01:53:49 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:54:00 (1779688440) Failing mds1 on oleg250-server Stopping /mnt/lustre-mds1 (opts:) on oleg250-server 01:54:07 (1779688447) shut down facet: mds1 facet_host: oleg250-server facet_failover_host: oleg250-server Failover mds1 to oleg250-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg250-server: oleg250-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg250-client: oleg250-server: ssh exited with exit code 1 Started lustre-MDT0000 01:54:31 (1779688471) targets are mounted 01:54:31 (1779688471) facet_failover done oleg250-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0460269 s, 22.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:54:49 (1779688489) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:54:54 (1779688494) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:54:58 (1779688498) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0092845 s, 1.8 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:55:07 (1779688507) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:55:11 (1779688511) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:55:15 (1779688515) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (31s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:55:45 (1779688545) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:56:14 (1779688574) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:56:39 (1779688599) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:57:06 (1779688626) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 01:57:13 (1779688633) mirror create pid 53539 mirror split pid 53540 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53539 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 53540 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 01:58:22 (1779688702) Seed set to 8704 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000005 1779688704.539096 mapwrite 0x022cfd thru 0x02f848 (0xcb4c bytes) 000006 1779688704.694823 read 0x0282ec thru 0x02f848 (0x755d bytes) truncating to largest ever: 0x2ee4 000007 1779688704.696697 trunc from 0x02f849 to 0x002ee3 (0x2c965 bytes) 000009 1779688704.718942 mapread 0x0003d0 thru 0x002ee3 (0x2b14 bytes) 000010 1779688704.730869 write 0x0198b1 thru 0x01b8dc (0x202c bytes) 000012 1779688704.734226 read 0x014a73 thru 0x01b8dc (0x6e6a bytes) 000014 1779688704.743215 mapread 0x004403 thru 0x010d94 (0xc992 bytes) 000015 1779688704.744612 mapwrite 0x00ee9b thru 0x018583 (0x96e9 bytes) truncating to largest ever: 0xd244 000016 1779688704.802644 trunc from 0x01b8dd to 0x00d243 (0xe699 bytes) 000017 1779688704.837581 mapread 0x002bca thru 0x00bd2b (0x9162 bytes) 000018 1779688704.839297 mapwrite 0x00c109 thru 0x012983 (0x687b bytes) 000019 1779688704.943558 write 0x034501 thru 0x034a24 (0x524 bytes) 000021 1779688704.945899 read 0x010c75 thru 0x01a3fa (0x9786 bytes) 000022 1779688704.960597 MIRROR_EXTEND to 2 mirrors 000023 1779688705.718985 mapread 0x01cdd1 thru 0x024fe1 (0x8211 bytes) 000027 1779688705.742207 MIRROR_RESYNC 2 mirrors 000028 1779688706.328454 MIRROR_EXTEND to 3 mirrors truncating to largest ever: 0x23454 000029 1779688706.860417 trunc from 0x034a25 to 0x023453 (0x115d1 bytes) 000033 1779688706.964465 read 0x009e54 thru 0x014b94 (0xad41 bytes) 000035 1779688706.976829 read 0x0032d8 thru 0x0046f8 (0x1421 bytes) 000037 1779688706.990961 write 0x03209f thru 0x038770 (0x66d2 bytes) 000040 1779688706.995766 mapread 0x018d61 thru 0x023aad (0xad4d bytes) 000041 1779688707.003915 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38771): Mirror 2: 0x4011b91c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38771): Mirror 2: 0x4011b91c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38771: skipped 000043 1779688707.307330 MIRROR_RESYNC 2 mirrors 000045 1779688708.119847 read 0x0358b8 thru 0x036f38 (0x1681 bytes) 000046 1779688708.124926 trunc from 0x038771 to 0x0202ed (0x18483 bytes) 000048 1779688708.247234 mapwrite 0x002da9 thru 0x012d67 (0xffbf bytes) 000049 1779688708.450565 mapwrite 0x03cfeb thru 0x03ffff (0x3015 bytes) 000050 1779688708.608771 trunc from 0x040000 to 0x004390 (0x3bc6f bytes) 000053 1779688708.637522 read 0x00269a thru 0x004390 (0x1cf7 bytes) 000055 1779688708.638805 mapread 0x003ee9 thru 0x004390 (0x4a8 bytes) 000056 1779688708.644713 write 0x03d3f6 thru 0x03ffff (0x2c0a bytes) 000057 1779688708.646560 mapread 0x012e2c thru 0x017fd7 (0x51ac bytes) 000060 1779688708.659354 write 0x02bc39 thru 0x03a360 (0xe728 bytes) 000062 1779688708.667035 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x15dae5a0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x65c04602 Mirror 3: 0x65c04602 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000064 1779688709.567992 mapwrite 0x016ab8 thru 0x023f89 (0xd4d2 bytes) 000065 1779688709.849542 write 0x01a209 thru 0x020e7e (0x6c76 bytes) 000067 1779688709.853044 mapwrite 0x017f48 thru 0x01a109 (0x21c2 bytes) 000069 1779688709.994042 mapwrite 0x0259ec thru 0x0274c1 (0x1ad6 bytes) 000071 1779688710.156070 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x15dae5a0 (stale) Mirror 3: 0x65c04602 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x214956fa Mirror 4: 0x214956fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000073 1779688711.146012 trunc from 0x040000 to 0x01528e (0x2ad71 bytes) 000074 1779688711.207502 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1528f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1528f): Mirror 2: 0x7cc6ec7e (stale) Mirror 3: 0xdb9407b9 (stale) Mirror 4: 0xdb9407b9 (stale) CRC-32 checksum value for chunk [0, 0x1528f): Mirror 1: 0xdb9407b9 Mirror 5: 0xdb9407b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1528f: skipped 000076 1779688712.248786 mapread 0x00ff3e thru 0x01528e (0x5351 bytes) 000077 1779688712.335054 mapwrite 0x00aa39 thru 0x015f8a (0xb552 bytes) 000078 1779688712.601110 mapread 0x00aa19 thru 0x00ed1f (0x4307 bytes) 000080 1779688712.602711 read 0x0092ad thru 0x00e9dd (0x5731 bytes) 000081 1779688712.608851 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x15f8b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15f8b): Mirror 2: 0x275eefa1 (stale) Mirror 3: 0x6ee9d2c7 (stale) Mirror 4: 0x6ee9d2c7 (stale) Mirror 5: 0x6ee9d2c7 (stale) CRC-32 checksum value for chunk [0, 0x15f8b): Mirror 1: 0xd1018f46 Mirror 6: 0xd1018f46 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15f8b: skipped 000086 1779688713.751834 trunc from 0x015f8b to 0x00852b (0xda5f bytes) 000087 1779688713.886662 mapwrite 0x0223a3 thru 0x02c800 (0xa45e bytes) 000090 1779688714.092496 mapread 0x01d62f thru 0x01d94c (0x31e bytes) 000091 1779688714.103375 trunc from 0x02c801 to 0x0167d9 (0x16027 bytes) 000092 1779688714.149970 read 0x01053e thru 0x0167d9 (0x629c bytes) 000093 1779688714.180766 write 0x026fad thru 0x03547b (0xe4cf bytes) 000095 1779688714.202841 MIRROR_RESYNC 6 mirrors 000097 1779688715.638813 mapread 0x0206f8 thru 0x021620 (0xf29 bytes) 000098 1779688715.650633 mapread 0x02bdfa thru 0x02c3b2 (0x5b9 bytes) 000100 1779688715.661134 trunc from 0x03547c to 0x017096 (0x1e3e5 bytes) 000102 1779688715.752057 mapread 0x0013e5 thru 0x001c27 (0x843 bytes) 000103 1779688715.763693 mapwrite 0x0183eb thru 0x01f78c (0x73a2 bytes) 000105 1779688715.842061 read 0x015c7d thru 0x01f78c (0x9b10 bytes) 000106 1779688715.843038 mapwrite 0x02e01f thru 0x031f4e (0x3f30 bytes) 000114 1779688715.939059 MIRROR_RESYNC 6 mirrors 000117 1779688717.031138 mapread 0x02a69a thru 0x031f4e (0x78b5 bytes) 000119 1779688717.069039 read 0x028296 thru 0x031f4e (0x9cb9 bytes) 000121 1779688717.070641 read 0x00e19c thru 0x013d35 (0x5b9a bytes) 000122 1779688717.098528 mapread 0x015931 thru 0x021fe2 (0xc6b2 bytes) 000123 1779688717.099809 read 0x026f55 thru 0x02b9b4 (0x4a60 bytes) 000125 1779688717.122543 MIRROR_EXTEND to 7 mirrors truncating to largest ever: 0x31bfb 000127 1779688718.041943 trunc from 0x031f4f to 0x031bfa (0x354 bytes) 000128 1779688718.133655 mapread 0x024fbe thru 0x03062f (0xb672 bytes) 000129 1779688718.166684 trunc from 0x031bfb to 0x02cfb7 (0x4c43 bytes) 000130 1779688718.189018 read 0x016325 thru 0x0218ca (0xb5a6 bytes) 000131 1779688718.203091 read 0x0030c9 thru 0x0082b8 (0x51f0 bytes) 000133 1779688718.214837 trunc from 0x02cfb8 to 0x01d844 (0xf773 bytes) 000134 1779688718.236820 mapread 0x017e0b thru 0x019c90 (0x1e86 bytes) 000136 1779688718.241850 mapwrite 0x006a0a thru 0x009740 (0x2d37 bytes) truncating to largest ever: 0x32d44 000137 1779688718.298298 trunc from 0x01d845 to 0x032d43 (0x154ff bytes) 000142 1779688718.342363 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x32d44): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32d44): Mirror 2: 0x5da8b3a0 (stale) Mirror 3: 0x5da8b3a0 (stale) Mirror 4: 0x5da8b3a0 (stale) Mirror 5: 0x5da8b3a0 (stale) Mirror 6: 0x5da8b3a0 (stale) Mirror 7: 0x5da8b3a0 (stale) CRC-32 checksum value for chunk [0, 0x32d44): Mirror 1: 0xbf1c39ad Mirror 8: 0xbf1c39ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32d44: skipped 000146 1779688719.953756 write 0x004200 thru 0x00aed8 (0x6cd9 bytes) 000147 1779688720.040210 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x32d44): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32d44): Mirror 2: 0x5da8b3a0 (stale) Mirror 3: 0x5da8b3a0 (stale) Mirror 4: 0x5da8b3a0 (stale) Mirror 5: 0x5da8b3a0 (stale) Mirror 6: 0x5da8b3a0 (stale) Mirror 7: 0x5da8b3a0 (stale) Mirror 8: 0xbf1c39ad (stale) CRC-32 checksum value for chunk [0, 0x32d44): Mirror 1: 0x79012b3a Mirror 9: 0x79012b3a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32d44: skipped 000148 1779688721.718052 MIRROR_RESYNC 9 mirrors 000150 1779688723.393868 mapwrite 0x03660b thru 0x0396ac (0x30a2 bytes) 000153 1779688723.618019 mapwrite 0x035ff4 thru 0x03ffff (0xa00c bytes) truncating to largest ever: 0x37f41 000154 1779688723.723700 trunc from 0x040000 to 0x037f40 (0x80bf bytes) 000155 1779688723.747827 read 0x01044a thru 0x019c69 (0x9820 bytes) 000156 1779688723.760037 trunc from 0x037f41 to 0x017792 (0x207ae bytes) 000157 1779688723.783896 write 0x01691a thru 0x01ff1b (0x9602 bytes) 000158 1779688723.786302 mapwrite 0x013821 thru 0x013ae2 (0x2c2 bytes) 000159 1779688723.849061 read 0x007fae thru 0x008432 (0x485 bytes) 000160 1779688723.858978 mapwrite 0x03f809 thru 0x03ffff (0x7f7 bytes) 000161 1779688723.933848 read 0x00a2bf thru 0x01769d (0xd3df bytes) 000164 1779688723.939387 write 0x033563 thru 0x039911 (0x63af bytes) 000166 1779688723.942341 mapwrite 0x011b57 thru 0x0167ed (0x4c97 bytes) 000167 1779688724.000135 mapread 0x02ec3e thru 0x037263 (0x8626 bytes) 000168 1779688724.003534 write 0x02f785 thru 0x030b9d (0x1419 bytes) 000169 1779688724.006881 mapread 0x0342f6 thru 0x0346a8 (0x3b3 bytes) 000171 1779688724.009962 trunc from 0x040000 to 0x028d60 (0x1729f bytes) 000176 1779688724.044047 read 0x018cbe thru 0x01b202 (0x2545 bytes) 000178 1779688724.046088 trunc from 0x028d61 to 0x01dca5 (0xb0bb bytes) 000181 1779688724.076872 write 0x0141c5 thru 0x01aa50 (0x688c bytes) 000183 1779688724.080826 read 0x01699a thru 0x01a378 (0x39df bytes) 000184 1779688724.082071 read 0x00aad1 thru 0x01579f (0xaccf bytes) 000185 1779688724.082889 mapread 0x0044a1 thru 0x00b559 (0x70b9 bytes) 000186 1779688724.090479 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dca6): Mirror 2: 0xff268cb8 (stale) Mirror 3: 0xff268cb8 (stale) Mirror 4: 0xff268cb8 (stale) Mirror 5: 0xff268cb8 (stale) Mirror 6: 0xff268cb8 (stale) Mirror 7: 0xff268cb8 (stale) Mirror 8: 0xff268cb8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dca6): Mirror 2: 0xff268cb8 (stale) Mirror 3: 0xff268cb8 (stale) Mirror 4: 0xff268cb8 (stale) Mirror 5: 0xff268cb8 (stale) Mirror 6: 0xff268cb8 (stale) Mirror 7: 0xff268cb8 (stale) Mirror 8: 0xff268cb8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dca6: skipped 000187 1779688724.696043 MIRROR_RESYNC 8 mirrors 000188 1779688726.228920 read 0x00eb40 thru 0x01749c (0x895d bytes) 000190 1779688726.240895 mapwrite 0x03097b thru 0x034a6c (0x40f2 bytes) 000192 1779688726.388941 write 0x02a2c6 thru 0x039a2a (0xf765 bytes) 000195 1779688726.404293 mapwrite 0x0058af thru 0x00cd9e (0x74f0 bytes) 000197 1779688726.563622 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x39a2b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a2b): Mirror 2: 0xa3c965 (stale) Mirror 3: 0xa3c965 (stale) Mirror 4: 0xa3c965 (stale) Mirror 5: 0xa3c965 (stale) Mirror 6: 0xa3c965 (stale) Mirror 7: 0xa3c965 (stale) Mirror 8: 0xa3c965 (stale) CRC-32 checksum value for chunk [0, 0x39a2b): Mirror 1: 0xa3286e2b Mirror 9: 0xa3286e2b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a2b: skipped 000198 1779688727.622003 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x39a2b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a2b): Mirror 2: 0xa3c965 (stale) Mirror 3: 0xa3c965 (stale) Mirror 4: 0xa3c965 (stale) Mirror 5: 0xa3c965 (stale) Mirror 6: 0xa3c965 (stale) Mirror 7: 0xa3c965 (stale) Mirror 8: 0xa3c965 (stale) CRC-32 checksum value for chunk [0, 0x39a2b): Mirror 1: 0xa3286e2b Mirror 9: 0xa3286e2b Mirror 10: 0xa3286e2b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a2b: skipped 000203 1779688728.917533 mapwrite 0x027326 thru 0x035687 (0xe362 bytes) 000204 1779688729.197949 mapread 0x00b451 thru 0x016190 (0xad40 bytes) 000205 1779688729.268710 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x39a2b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a2b): Mirror 2: 0xa3c965 (stale) Mirror 3: 0xa3c965 (stale) Mirror 4: 0xa3c965 (stale) Mirror 5: 0xa3c965 (stale) Mirror 6: 0xa3c965 (stale) Mirror 7: 0xa3c965 (stale) Mirror 8: 0xa3c965 (stale) Mirror 9: 0xa3286e2b (stale) Mirror 10: 0xa3286e2b (stale) CRC-32 checksum value for chunk [0, 0x39a2b): Mirror 1: 0xd2ee09ef Mirror 11: 0xd2ee09ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a2b: skipped 000206 1779688730.708143 trunc from 0x039a2b to 0x0056bf (0x3436b bytes) 000207 1779688730.888406 write 0x03bc24 thru 0x03c049 (0x426 bytes) 000208 1779688730.901042 MIRROR_RESYNC 11 mirrors 000209 1779688733.196727 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) 000210 1779688733.651060 mapread 0x0271fb thru 0x02c72e (0x5534 bytes) 000212 1779688733.677714 trunc from 0x03c04a to 0x0220e0 (0x19f69 bytes) 000213 1779688733.782219 mapwrite 0x0240e6 thru 0x02a459 (0x6374 bytes) 000215 1779688733.861839 mapread 0x018237 thru 0x01abae (0x2978 bytes) 000216 1779688733.871556 read 0x00e7f4 thru 0x00fed8 (0x16e5 bytes) 000217 1779688733.877587 mapread 0x00b9c3 thru 0x0102b2 (0x48f0 bytes) 000218 1779688733.887967 mapread 0x02577f thru 0x02a459 (0x4cdb bytes) 000220 1779688733.890642 read 0x01e8f4 thru 0x027ae2 (0x91ef bytes) 000221 1779688733.891587 mapread 0x01553c thru 0x0162dd (0xda2 bytes) 000222 1779688733.893591 read 0x0113da thru 0x013951 (0x2578 bytes) 000223 1779688733.894485 read 0x0264b1 thru 0x026841 (0x391 bytes) 000225 1779688733.896508 trunc from 0x02a45a to 0x0262fa (0x415f bytes) 000226 1779688733.917060 mapwrite 0x03476b thru 0x03ffff (0xb895 bytes) 000228 1779688734.005841 write 0x0373b6 thru 0x03ffff (0x8c4a bytes) 000231 1779688734.010816 read 0x03e6c1 thru 0x03ffff (0x193f bytes) 000238 1779688734.018748 mapwrite 0x024543 thru 0x02927b (0x4d39 bytes) 000239 1779688734.072671 trunc from 0x040000 to 0x023d54 (0x1c2ab bytes) 000240 1779688734.089908 read 0x0114d5 thru 0x01e11e (0xcc4a bytes) truncating to largest ever: 0x3a350 000241 1779688734.090854 trunc from 0x023d55 to 0x03a34f (0x165fb bytes) 000244 1779688734.117749 read 0x02cc11 thru 0x0314fa (0x48ea bytes) 000245 1779688734.126056 write 0x0318ce thru 0x03ffff (0xe732 bytes) 000246 1779688734.128680 mapwrite 0x0183b4 thru 0x020d9a (0x89e7 bytes) 000248 1779688734.194566 mapwrite 0x0389b4 thru 0x03ffff (0x764c bytes) 000249 1779688734.234968 mapwrite 0x025b74 thru 0x02f83d (0x9cca bytes) 000250 1779688734.296751 mapread 0x02bd39 thru 0x033e2b (0x80f3 bytes) 000251 1779688734.297998 mapwrite 0x00b631 thru 0x00f04b (0x3a1b bytes) 000253 1779688734.346012 read 0x035e82 thru 0x038292 (0x2411 bytes) 000254 1779688734.347747 trunc from 0x040000 to 0x028efb (0x17104 bytes) 000255 1779688734.363336 mapwrite 0x0236f6 thru 0x028176 (0x4a81 bytes) 000256 1779688734.406521 mapwrite 0x0329d1 thru 0x03ffff (0xd62f bytes) 000257 1779688734.475998 read 0x024934 thru 0x025ce0 (0x13ad bytes) 000258 1779688734.477726 mapread 0x016afb thru 0x0212e1 (0xa7e7 bytes) 000259 1779688734.480702 trunc from 0x040000 to 0x00fbd9 (0x30426 bytes) 000261 1779688734.498926 trunc from 0x00fbda to 0x00f1d4 (0xa05 bytes) 000262 1779688734.522889 trunc from 0x00f1d5 to 0x030088 (0x20eb4 bytes) 000264 1779688734.548611 mapread 0x0116e2 thru 0x012bdd (0x14fc bytes) 000266 1779688734.561872 trunc from 0x030089 to 0x010583 (0x1fb05 bytes) 000269 1779688734.582015 mapwrite 0x02adc1 thru 0x02e27d (0x34bd bytes) 000271 1779688734.662891 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2e27e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e27e): Mirror 2: 0xc924f02c (stale) Mirror 3: 0xc924f02c (stale) Mirror 4: 0xc924f02c (stale) Mirror 6: 0xc924f02c (stale) Mirror 7: 0xc924f02c (stale) Mirror 8: 0xc924f02c (stale) Mirror 9: 0xc924f02c (stale) Mirror 10: 0xc924f02c (stale) Mirror 11: 0xc924f02c (stale) CRC-32 checksum value for chunk [0, 0x2e27e): Mirror 1: 0x138c5c5b Mirror 12: 0x138c5c5b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e27e: skipped 000272 1779688735.920917 mapwrite 0x006d4d thru 0x00e27f (0x7533 bytes) 000273 1779688736.136117 trunc from 0x02e27e to 0x025d28 (0x8555 bytes) 000275 1779688736.160907 trunc from 0x025d29 to 0x027a06 (0x1cde bytes) 000278 1779688736.192888 read 0x02471a thru 0x027a06 (0x32ed bytes) 000279 1779688736.211679 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27a07): Mirror 3: 0x6306dff9 (stale) Mirror 4: 0x6306dff9 (stale) Mirror 6: 0x6306dff9 (stale) Mirror 7: 0x6306dff9 (stale) Mirror 8: 0x6306dff9 (stale) Mirror 9: 0x6306dff9 (stale) Mirror 10: 0x6306dff9 (stale) Mirror 11: 0x6306dff9 (stale) Mirror 12: 0x969df35f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27a07): Mirror 3: 0x6306dff9 (stale) Mirror 4: 0x6306dff9 (stale) Mirror 6: 0x6306dff9 (stale) Mirror 7: 0x6306dff9 (stale) Mirror 8: 0x6306dff9 (stale) Mirror 9: 0x6306dff9 (stale) Mirror 10: 0x6306dff9 (stale) Mirror 11: 0x6306dff9 (stale) Mirror 12: 0x969df35f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27a07: skipped 000282 1779688736.800556 MIRROR_RESYNC 10 mirrors 000284 1779688738.955324 write 0x03e2cd thru 0x03ffff (0x1d33 bytes) 000285 1779688739.056075 mapwrite 0x00e2c6 thru 0x019a42 (0xb77d bytes) 000286 1779688739.279725 write 0x0245f3 thru 0x025ebf (0x18cd bytes) 000287 1779688739.313861 MIRROR_RESYNC 10 mirrors 000288 1779688741.286710 mapwrite 0x034586 thru 0x03a9d8 (0x6453 bytes) 000290 1779688741.546425 write 0x006981 thru 0x00fc95 (0x9315 bytes) 000292 1779688741.562363 mapwrite 0x028c8f thru 0x02e570 (0x58e2 bytes) 000294 1779688741.611642 read 0x0216da thru 0x02699c (0x52c3 bytes) 000296 1779688741.612681 trunc from 0x040000 to 0x01ff0b (0x200f4 bytes) 000298 1779688741.636704 write 0x0072de thru 0x0104b3 (0x91d6 bytes) 000301 1779688741.639333 trunc from 0x01ff0c to 0x014be3 (0xb328 bytes) 000302 1779688741.656931 mapread 0x0075ea thru 0x01477c (0xd193 bytes) 000303 1779688741.662106 mapwrite 0x013750 thru 0x019ef5 (0x67a6 bytes) 000304 1779688741.837298 write 0x0099c2 thru 0x00e875 (0x4eb4 bytes) 000306 1779688741.843194 write 0x016c64 thru 0x01e760 (0x7afd bytes) 000307 1779688741.851842 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e761): Mirror 3: 0xdbc01353 (stale) Mirror 4: 0xdbc01353 (stale) Mirror 6: 0xdbc01353 (stale) Mirror 7: 0xdbc01353 (stale) Mirror 8: 0xdbc01353 (stale) Mirror 9: 0xdbc01353 (stale) Mirror 11: 0xdbc01353 (stale) Mirror 12: 0xdbc01353 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e761): Mirror 3: 0xdbc01353 (stale) Mirror 4: 0xdbc01353 (stale) Mirror 6: 0xdbc01353 (stale) Mirror 7: 0xdbc01353 (stale) Mirror 8: 0xdbc01353 (stale) Mirror 9: 0xdbc01353 (stale) Mirror 11: 0xdbc01353 (stale) Mirror 12: 0xdbc01353 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e761: skipped 000308 1779688742.701996 MIRROR_RESYNC 9 mirrors 000311 1779688744.447549 mapwrite 0x00a222 thru 0x00aaf3 (0x8d2 bytes) 000313 1779688744.579502 mapread 0x002388 thru 0x006c48 (0x48c1 bytes) 000314 1779688744.591650 mapwrite 0x03acbf thru 0x03ffff (0x5341 bytes) 000315 1779688744.669901 trunc from 0x040000 to 0x00bc03 (0x343fc bytes) 000316 1779688744.688850 trunc from 0x00bc04 to 0x019755 (0xdb52 bytes) 000317 1779688744.709767 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x19756): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19756): Mirror 3: 0x13d98acc (stale) Mirror 4: 0x13d98acc (stale) Mirror 6: 0x13d98acc (stale) Mirror 7: 0x13d98acc (stale) Mirror 8: 0x13d98acc (stale) Mirror 9: 0x13d98acc (stale) Mirror 11: 0x13d98acc (stale) Mirror 12: 0x13d98acc (stale) CRC-32 checksum value for chunk [0, 0x19756): Mirror 1: 0x8d58c62d Mirror 13: 0x8d58c62d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19756: skipped 000318 1779688746.165658 read 0x00b80f thru 0x00b837 (0x29 bytes) 000319 1779688746.193286 mapread 0x014b57 thru 0x019755 (0x4bff bytes) 000323 1779688746.326634 mapread 0x019170 thru 0x019755 (0x5e6 bytes) 000324 1779688746.344809 mapwrite 0x03b1cc thru 0x03ffff (0x4e34 bytes) 000326 1779688746.518738 mapread 0x02bc0c thru 0x02d52d (0x1922 bytes) 000328 1779688746.535114 trunc from 0x040000 to 0x00c073 (0x33f8c bytes) 000329 1779688746.563741 mapread 0x00bc7a thru 0x00c073 (0x3fa bytes) 000330 1779688746.578075 mapread 0x008c94 thru 0x00c073 (0x33e0 bytes) 000332 1779688746.585887 MIRROR_RESYNC 10 mirrors 000333 1779688748.396438 write 0x01d320 thru 0x01d50a (0x1eb bytes) 000334 1779688748.472627 mapwrite 0x00ca55 thru 0x01b9db (0xef87 bytes) 000338 1779688748.593071 trunc from 0x01d50b to 0x00c3a5 (0x11165 bytes) 000339 1779688748.615766 mapread 0x00756b thru 0x00c3a5 (0x4e3b bytes) 000341 1779688748.617879 read 0x000105 thru 0x0082c9 (0x81c5 bytes) 000342 1779688748.632436 write 0x03ebc9 thru 0x03ffff (0x1437 bytes) 000343 1779688748.634444 write 0x00bc7f thru 0x016620 (0xa9a2 bytes) 000344 1779688748.641248 trunc from 0x040000 to 0x0166f9 (0x29906 bytes) 000346 1779688748.662835 mapwrite 0x033f0f thru 0x03b5b8 (0x76aa bytes) 000347 1779688748.746616 write 0x03e12e thru 0x03ffff (0x1ed2 bytes) 000348 1779688748.751220 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x6437941f (stale) Mirror 4: 0x6437941f (stale) Mirror 6: 0x6437941f (stale) Mirror 7: 0x6437941f (stale) Mirror 8: 0x6437941f (stale) Mirror 9: 0x6437941f (stale) Mirror 11: 0x6437941f (stale) Mirror 12: 0x6437941f (stale) Mirror 13: 0x6437941f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfea3626a Mirror 14: 0xfea3626a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000351 1779688750.362118 read 0x00b9a5 thru 0x00eb74 (0x31d0 bytes) 000355 1779688750.440575 write 0x020223 thru 0x02eb8e (0xe96c bytes) 000356 1779688750.672478 trunc from 0x040000 to 0x0241c5 (0x1be3a bytes) 000357 1779688750.696796 mapwrite 0x00c3a4 thru 0x00e6cd (0x232a bytes) 000359 1779688750.845573 trunc from 0x0241c6 to 0x02ede1 (0xac1c bytes) 000361 1779688750.881822 MIRROR_RESYNC 11 mirrors 000362 1779688752.661670 mapread 0x01ef3f thru 0x025fce (0x7090 bytes) 000363 1779688752.681513 read 0x01ccb8 thru 0x01e14e (0x1497 bytes) 000364 1779688752.682133 mapwrite 0x00c85b thru 0x01c404 (0xfbaa bytes) 000366 1779688752.863668 mapwrite 0x03b311 thru 0x03ffff (0x4cef bytes) 000368 1779688752.936114 read 0x01f6a1 thru 0x02606e (0x69ce bytes) 000369 1779688752.946297 mapwrite 0x03f8c7 thru 0x03ffff (0x739 bytes) 000371 1779688752.985408 trunc from 0x040000 to 0x02b57c (0x14a83 bytes) 000372 1779688753.013762 read 0x0228e0 thru 0x02b57c (0x8c9d bytes) 000374 1779688753.015879 read 0x01ef13 thru 0x02ae63 (0xbf51 bytes) 000376 1779688753.018424 read 0x017b7b thru 0x01a9d3 (0x2e59 bytes) 000377 1779688753.019428 mapwrite 0x00a3e0 thru 0x016901 (0xc522 bytes) 000379 1779688753.088395 mapread 0x015f37 thru 0x016578 (0x642 bytes) 000380 1779688753.090584 read 0x014fc5 thru 0x02143a (0xc476 bytes) 000382 1779688753.092835 read 0x00b5ce thru 0x00f033 (0x3a66 bytes) 000383 1779688753.097334 mapread 0x00cca3 thru 0x011773 (0x4ad1 bytes) 000385 1779688753.099539 mapwrite 0x01bbce thru 0x021f26 (0x6359 bytes) 000387 1779688753.160969 trunc from 0x02b57d to 0x002554 (0x29028 bytes) 000388 1779688753.182069 write 0x024ae5 thru 0x027162 (0x267e bytes) 000389 1779688753.183556 mapread 0x023f2f thru 0x027162 (0x3234 bytes) 000391 1779688753.195607 trunc from 0x027163 to 0x008301 (0x1ee61 bytes) 000392 1779688753.219000 MIRROR_RESYNC 11 mirrors 000394 1779688754.974405 trunc from 0x008302 to 0x021765 (0x19464 bytes) 000397 1779688755.049792 trunc from 0x021766 to 0x010fe9 (0x1077c bytes) 000400 1779688755.072260 mapwrite 0x02393e thru 0x026e85 (0x3548 bytes) 000402 1779688755.185112 mapwrite 0x006368 thru 0x008a6a (0x2703 bytes) 000403 1779688755.227444 read 0x00a441 thru 0x00fb73 (0x5733 bytes) 000405 1779688755.229237 mapread 0x009ec6 thru 0x00a754 (0x88f bytes) 000406 1779688755.234771 MIRROR_RESYNC 11 mirrors 000409 1779688756.715096 trunc from 0x026e86 to 0x0257e8 (0x169d bytes) 000410 1779688756.821213 trunc from 0x0257e9 to 0x01068b (0x1515d bytes) 000411 1779688756.849511 mapwrite 0x005048 thru 0x005a7c (0xa35 bytes) 000412 1779688756.908391 read 0x00c6c7 thru 0x01068b (0x3fc5 bytes) 000413 1779688756.918943 mapread 0x00500c thru 0x01068b (0xb680 bytes) 000414 1779688756.936449 write 0x006cf3 thru 0x00b65e (0x496c bytes) 000415 1779688756.939404 write 0x03c6cb thru 0x03ec6e (0x25a4 bytes) 000416 1779688756.941457 trunc from 0x03ec6f to 0x029c8e (0x14fe0 bytes) 000418 1779688756.963129 read 0x01588a thru 0x01b084 (0x57fb bytes) 000420 1779688756.976664 trunc from 0x029c8f to 0x035f13 (0xc285 bytes) 000422 1779688756.997937 read 0x01de88 thru 0x021410 (0x3589 bytes) 000423 1779688757.002955 mapread 0x001083 thru 0x00868b (0x7609 bytes) 000425 1779688757.007326 trunc from 0x035f14 to 0x02ee83 (0x7090 bytes) 000426 1779688757.029873 write 0x03a06b thru 0x03ffff (0x5f95 bytes) 000427 1779688757.038673 MIRROR_RESYNC 11 mirrors 000431 1779688758.605517 write 0x01bf55 thru 0x0299a6 (0xda52 bytes) 000432 1779688758.701097 read 0x00dde4 thru 0x01dce7 (0xff04 bytes) 000434 1779688758.712445 trunc from 0x040000 to 0x02064e (0x1f9b1 bytes) 000436 1779688758.739704 mapread 0x01104f thru 0x02064e (0xf600 bytes) 000438 1779688758.745994 trunc from 0x02064f to 0x026454 (0x5e06 bytes) 000439 1779688758.778253 read 0x01c29b thru 0x026454 (0xa1ba bytes) 000440 1779688758.810117 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x26455): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26455): Mirror 3: 0x7cde6fcf (stale) Mirror 4: 0x7cde6fcf (stale) Mirror 6: 0x7cde6fcf (stale) Mirror 7: 0x7cde6fcf (stale) Mirror 8: 0x7cde6fcf (stale) Mirror 9: 0x7cde6fcf (stale) Mirror 11: 0x7cde6fcf (stale) Mirror 12: 0x7cde6fcf (stale) Mirror 13: 0x7cde6fcf (stale) Mirror 14: 0x7cde6fcf (stale) CRC-32 checksum value for chunk [0, 0x26455): Mirror 1: 0x17cd4df4 Mirror 15: 0x17cd4df4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26455: skipped 000441 1779688760.249109 read 0x020c88 thru 0x026454 (0x57cd bytes) 000445 1779688760.290837 read 0x01f9bd thru 0x026454 (0x6a98 bytes) 000446 1779688760.303916 trunc from 0x026455 to 0x032e3c (0xc9e8 bytes) 000448 1779688760.399328 read 0x0205cc thru 0x02dff7 (0xda2c bytes) 000449 1779688760.418837 trunc from 0x032e3d to 0x001277 (0x31bc5 bytes) 000450 1779688760.464095 MIRROR_RESYNC 12 mirrors 000451 1779688762.503386 read 0x001148 thru 0x001277 (0x130 bytes) 000452 1779688762.508723 mapwrite 0x023aa4 thru 0x025c8d (0x21ea bytes) 000453 1779688762.750439 write 0x03a59e thru 0x03ffff (0x5a62 bytes) 000454 1779688762.754313 trunc from 0x040000 to 0x019bf0 (0x2640f bytes) 000455 1779688762.779027 mapwrite 0x03d6c4 thru 0x03de45 (0x782 bytes) 000456 1779688762.867056 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3de46): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3de46): Mirror 3: 0x556a6f8 (stale) Mirror 4: 0x556a6f8 (stale) Mirror 6: 0x556a6f8 (stale) Mirror 7: 0x556a6f8 (stale) Mirror 8: 0x556a6f8 (stale) Mirror 9: 0x556a6f8 (stale) Mirror 11: 0x556a6f8 (stale) Mirror 12: 0x556a6f8 (stale) Mirror 13: 0x556a6f8 (stale) Mirror 14: 0x556a6f8 (stale) Mirror 15: 0x556a6f8 (stale) CRC-32 checksum value for chunk [0, 0x3de46): Mirror 1: 0x9fe1917d Mirror 16: 0x9fe1917d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3de46: skipped 000459 1779688764.841971 read 0x01d293 thru 0x02850c (0xb27a bytes) 000460 1779688764.875268 mapread 0x0150d8 thru 0x023faf (0xeed8 bytes) 000461 1779688764.940032 read 0x02489b thru 0x029304 (0x4a6a bytes) 000465 1779688765.025316 read 0x0160d6 thru 0x0213db (0xb306 bytes) 000466 1779688765.039479 trunc from 0x03de46 to 0x0301df (0xdc66 bytes) 000468 1779688765.258290 mapwrite 0x010321 thru 0x01ed7d (0xea5d bytes) 000470 1779688765.392060 MIRROR_SPLIT mirror 11 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x301e0): Mirror 3: 0x5be61ed9 (stale) Mirror 4: 0x5be61ed9 (stale) Mirror 6: 0x5be61ed9 (stale) Mirror 7: 0x5be61ed9 (stale) Mirror 8: 0x5be61ed9 (stale) Mirror 9: 0x5be61ed9 (stale) Mirror 11: 0x5be61ed9 (stale) Mirror 12: 0x5be61ed9 (stale) Mirror 13: 0x5be61ed9 (stale) Mirror 14: 0x5be61ed9 (stale) Mirror 16: 0x5be61ed9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x301e0): Mirror 3: 0x5be61ed9 (stale) Mirror 4: 0x5be61ed9 (stale) Mirror 6: 0x5be61ed9 (stale) Mirror 7: 0x5be61ed9 (stale) Mirror 8: 0x5be61ed9 (stale) Mirror 9: 0x5be61ed9 (stale) Mirror 11: 0x5be61ed9 (stale) Mirror 12: 0x5be61ed9 (stale) Mirror 13: 0x5be61ed9 (stale) Mirror 14: 0x5be61ed9 (stale) Mirror 16: 0x5be61ed9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x301e0: skipped 000472 1779688766.277741 trunc from 0x0301e0 to 0x0127e9 (0x1d9f6 bytes) 000473 1779688766.297445 write 0x00a378 thru 0x017e2a (0xdab3 bytes) 000474 1779688766.305227 mapwrite 0x03ff6a thru 0x03ffff (0x96 bytes) 000476 1779688766.393866 mapwrite 0x0047e1 thru 0x00d210 (0x8a30 bytes) 000477 1779688766.451628 trunc from 0x040000 to 0x0105ec (0x2fa13 bytes) 000478 1779688766.471089 mapread 0x005989 thru 0x0105ec (0xac64 bytes) 000479 1779688766.476821 mapwrite 0x015bf5 thru 0x015ce3 (0xef bytes) 000481 1779688766.544243 trunc from 0x015ce4 to 0x003594 (0x1274f bytes) 000482 1779688766.569438 MIRROR_RESYNC 12 mirrors 000484 1779688768.122738 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) 000485 1779688768.508837 trunc from 0x003595 to 0x02ed47 (0x2b7b3 bytes) 000486 1779688768.615436 write 0x028eea thru 0x02c50e (0x3625 bytes) 000487 1779688768.628495 trunc from 0x02ed48 to 0x02508b (0x9cbc bytes) 000488 1779688768.654129 read 0x011a36 thru 0x011a55 (0x20 bytes) 000490 1779688768.661826 write 0x02d193 thru 0x02eeb0 (0x1d1e bytes) 000492 1779688768.668634 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2eeb1): Mirror 3: 0xcd3e9086 (stale) Mirror 4: 0xcd3e9086 (stale) Mirror 6: 0xcd3e9086 (stale) Mirror 7: 0xcd3e9086 (stale) Mirror 8: 0xcd3e9086 (stale) Mirror 9: 0xcd3e9086 (stale) Mirror 11: 0xcd3e9086 (stale) Mirror 14: 0xcd3e9086 (stale) Mirror 16: 0xcd3e9086 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eeb1): Mirror 3: 0xcd3e9086 (stale) Mirror 4: 0xcd3e9086 (stale) Mirror 6: 0xcd3e9086 (stale) Mirror 7: 0xcd3e9086 (stale) Mirror 8: 0xcd3e9086 (stale) Mirror 9: 0xcd3e9086 (stale) Mirror 11: 0xcd3e9086 (stale) Mirror 14: 0xcd3e9086 (stale) Mirror 16: 0xcd3e9086 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eeb1: skipped 000495 1779688769.190596 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2eeb1): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2eeb1): Mirror 3: 0xcd3e9086 (stale) Mirror 4: 0xcd3e9086 (stale) Mirror 6: 0xcd3e9086 (stale) Mirror 7: 0xcd3e9086 (stale) Mirror 8: 0xcd3e9086 (stale) Mirror 9: 0xcd3e9086 (stale) Mirror 11: 0xcd3e9086 (stale) Mirror 14: 0xcd3e9086 (stale) Mirror 16: 0xcd3e9086 (stale) CRC-32 checksum value for chunk [0, 0x2eeb1): Mirror 1: 0x353dd4aa Mirror 17: 0x353dd4aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2eeb1: skipped truncating to largest ever: 0x3d03c 000496 1779688770.252049 trunc from 0x02eeb1 to 0x03d03b (0xe18b bytes) 000498 1779688770.359387 read 0x0317d0 thru 0x03d03b (0xb86c bytes) 000503 1779688770.372136 mapread 0x000bfd thru 0x006e0c (0x6210 bytes) 000504 1779688770.398040 read 0x00b325 thru 0x00bfe4 (0xcc0 bytes) 000506 1779688770.399572 mapwrite 0x0259d6 thru 0x02f4f9 (0x9b24 bytes) 000508 1779688770.466232 write 0x01911e thru 0x01ed51 (0x5c34 bytes) 000510 1779688770.469339 mapread 0x00289e thru 0x00c182 (0x98e5 bytes) 000512 1779688770.475993 write 0x03e37b thru 0x03ffff (0x1c85 bytes) 000513 1779688770.478324 write 0x00e1de thru 0x01144b (0x326e bytes) 000514 1779688770.484571 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 14: 0xe20eea22 (stale) Mirror 16: 0xe20eea22 (stale) Mirror 17: 0x2e73b8c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) Mirror 11: 0xe20eea22 (stale) Mirror 14: 0xe20eea22 (stale) Mirror 16: 0xe20eea22 (stale) Mirror 17: 0x2e73b8c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000515 1779688770.983004 mapread 0x02018c thru 0x023a6e (0x38e3 bytes) 000519 1779688771.004797 read 0x02e2cf thru 0x02f62b (0x135d bytes) 000520 1779688771.010545 trunc from 0x040000 to 0x0343d4 (0xbc2b bytes) 000524 1779688771.031772 mapread 0x02be25 thru 0x0343d4 (0x85b0 bytes) 000526 1779688771.047367 mapread 0x03238b thru 0x032bc4 (0x83a bytes) 000527 1779688771.048989 read 0x0104ce thru 0x0192c5 (0x8df8 bytes) 000529 1779688771.060303 trunc from 0x0343d5 to 0x01d26c (0x17168 bytes) 000530 1779688771.077548 mapwrite 0x01ef95 thru 0x0299ef (0xaa5b bytes) 000531 1779688771.217112 mapwrite 0x019bb9 thru 0x01d6b3 (0x3afb bytes) 000533 1779688771.269779 read 0x001b6b thru 0x010602 (0xea98 bytes) 000539 1779688771.280567 trunc from 0x0299f0 to 0x00fa0f (0x19fe0 bytes) 000540 1779688771.303579 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0xfa10): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfa10): Mirror 3: 0x90c08239 (stale) Mirror 6: 0x90c08239 (stale) Mirror 7: 0x90c08239 (stale) Mirror 8: 0x90c08239 (stale) Mirror 9: 0x90c08239 (stale) Mirror 11: 0x90c08239 (stale) Mirror 14: 0x90c08239 (stale) Mirror 16: 0x90c08239 (stale) Mirror 17: 0x90c08239 (stale) CRC-32 checksum value for chunk [0, 0xfa10): Mirror 1: 0x17787329 Mirror 18: 0x17787329 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfa10: skipped 000541 1779688772.136549 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0xfa10): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfa10): Mirror 3: 0x90c08239 (stale) Mirror 6: 0x90c08239 (stale) Mirror 7: 0x90c08239 (stale) Mirror 8: 0x90c08239 (stale) Mirror 9: 0x90c08239 (stale) Mirror 11: 0x90c08239 (stale) Mirror 14: 0x90c08239 (stale) Mirror 16: 0x90c08239 (stale) Mirror 17: 0x90c08239 (stale) CRC-32 checksum value for chunk [0, 0xfa10): Mirror 1: 0x17787329 Mirror 18: 0x17787329 Mirror 19: 0x17787329 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfa10: skipped 000542 1779688772.986404 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xfa10): Mirror 3: 0x90c08239 (stale) Mirror 6: 0x90c08239 (stale) Mirror 7: 0x90c08239 (stale) Mirror 8: 0x90c08239 (stale) Mirror 9: 0x90c08239 (stale) Mirror 11: 0x90c08239 (stale) Mirror 16: 0x90c08239 (stale) Mirror 17: 0x90c08239 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfa10): Mirror 3: 0x90c08239 (stale) Mirror 6: 0x90c08239 (stale) Mirror 7: 0x90c08239 (stale) Mirror 8: 0x90c08239 (stale) Mirror 9: 0x90c08239 (stale) Mirror 11: 0x90c08239 (stale) Mirror 16: 0x90c08239 (stale) Mirror 17: 0x90c08239 (stale) CRC-32 checksum value for chunk [0, 0xfa10): Mirror 1: 0x17787329 Mirror 18: 0x17787329 Mirror 19: 0x17787329 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfa10: skipped 000543 1779688773.649767 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xfa10): Mirror 3: 0x90c08239 (stale) Mirror 6: 0x90c08239 (stale) Mirror 8: 0x90c08239 (stale) Mirror 9: 0x90c08239 (stale) Mirror 11: 0x90c08239 (stale) Mirror 16: 0x90c08239 (stale) Mirror 17: 0x90c08239 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xfa10): Mirror 3: 0x90c08239 (stale) Mirror 6: 0x90c08239 (stale) Mirror 8: 0x90c08239 (stale) Mirror 9: 0x90c08239 (stale) Mirror 11: 0x90c08239 (stale) Mirror 16: 0x90c08239 (stale) Mirror 17: 0x90c08239 (stale) CRC-32 checksum value for chunk [0, 0xfa10): Mirror 1: 0x17787329 Mirror 18: 0x17787329 Mirror 19: 0x17787329 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xfa10: skipped 000544 1779688774.250047 MIRROR_RESYNC 10 mirrors 000547 1779688775.455573 mapwrite 0x03f7da thru 0x03ffff (0x826 bytes) 000550 1779688775.654903 mapread 0x011f4f thru 0x0179e3 (0x5a95 bytes) 000554 1779688775.673088 mapwrite 0x03088f thru 0x031b90 (0x1302 bytes) 000555 1779688775.725509 read 0x039654 thru 0x03ffff (0x69ac bytes) 000556 1779688775.734744 read 0x0034e7 thru 0x00eed8 (0xb9f2 bytes) 000557 1779688775.749666 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1788726f (stale) Mirror 6: 0x1788726f (stale) Mirror 8: 0x1788726f (stale) Mirror 11: 0x1788726f (stale) Mirror 16: 0x1788726f (stale) Mirror 17: 0x1788726f (stale) Mirror 18: 0x1788726f (stale) Mirror 19: 0x1788726f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1788726f (stale) Mirror 6: 0x1788726f (stale) Mirror 8: 0x1788726f (stale) Mirror 11: 0x1788726f (stale) Mirror 16: 0x1788726f (stale) Mirror 17: 0x1788726f (stale) Mirror 18: 0x1788726f (stale) Mirror 19: 0x1788726f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000560 1779688776.257643 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1788726f (stale) Mirror 6: 0x1788726f (stale) Mirror 8: 0x1788726f (stale) Mirror 11: 0x1788726f (stale) Mirror 16: 0x1788726f (stale) Mirror 17: 0x1788726f (stale) Mirror 18: 0x1788726f (stale) Mirror 19: 0x1788726f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2ff58486 Mirror 20: 0x2ff58486 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000562 1779688777.396536 write 0x0141f8 thru 0x023b6c (0xf975 bytes) 000563 1779688777.473806 write 0x031253 thru 0x03f930 (0xe6de bytes) 000566 1779688777.492916 mapread 0x02a4c1 thru 0x02af00 (0xa40 bytes) 000567 1779688777.501542 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x1788726f (stale) Mirror 6: 0x1788726f (stale) Mirror 8: 0x1788726f (stale) Mirror 11: 0x1788726f (stale) Mirror 16: 0x1788726f (stale) Mirror 17: 0x1788726f (stale) Mirror 18: 0x1788726f (stale) Mirror 19: 0x1788726f (stale) Mirror 20: 0x2ff58486 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x154e5e2c Mirror 21: 0x154e5e2c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000570 1779688778.489306 trunc from 0x040000 to 0x0054d8 (0x3ab27 bytes) 000572 1779688778.728445 read 0x000887 thru 0x003a71 (0x31eb bytes) 000573 1779688778.736717 mapread 0x004067 thru 0x0054d8 (0x1472 bytes) 000574 1779688778.751921 mapwrite 0x03ed39 thru 0x03ffff (0x12c7 bytes) 000576 1779688778.849259 read 0x03f502 thru 0x03ffff (0xafe bytes) 000577 1779688778.861887 MIRROR_RESYNC 11 mirrors 000578 1779688780.892382 write 0x01742a thru 0x0185db (0x11b2 bytes) 000579 1779688780.970631 mapread 0x014762 thru 0x01637d (0x1c1c bytes) 000580 1779688780.994012 write 0x01ef7e thru 0x024bc7 (0x5c4a bytes) 000581 1779688781.005013 trunc from 0x040000 to 0x01ec3a (0x213c5 bytes) 000582 1779688781.029453 mapwrite 0x018b91 thru 0x024ed1 (0xc341 bytes) 000583 1779688781.113598 mapread 0x01b3c4 thru 0x01ef7f (0x3bbc bytes) 000584 1779688781.114966 trunc from 0x024ed2 to 0x022156 (0x2d7b bytes) 000585 1779688781.153937 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x22157): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22157): Mirror 3: 0x335bd057 (stale) Mirror 6: 0x335bd057 (stale) Mirror 8: 0x335bd057 (stale) Mirror 11: 0x335bd057 (stale) Mirror 16: 0x335bd057 (stale) Mirror 17: 0x335bd057 (stale) Mirror 18: 0x335bd057 (stale) Mirror 19: 0x335bd057 (stale) Mirror 20: 0x335bd057 (stale) Mirror 21: 0x335bd057 (stale) CRC-32 checksum value for chunk [0, 0x22157): Mirror 1: 0x5e97f7d9 Mirror 22: 0x5e97f7d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22157: skipped 000586 1779688782.512854 trunc from 0x022157 to 0x0220a6 (0xb0 bytes) 000587 1779688782.650363 mapwrite 0x01f7ab thru 0x020037 (0x88d bytes) 000589 1779688782.764418 trunc from 0x0220a7 to 0x012335 (0xfd71 bytes) 000590 1779688782.833266 mapwrite 0x02051b thru 0x02f972 (0xf458 bytes) 000591 1779688783.030325 MIRROR_SPLIT mirror 7 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f973): Mirror 3: 0xfbb3ddc6 (stale) Mirror 6: 0xfbb3ddc6 (stale) Mirror 8: 0xfbb3ddc6 (stale) Mirror 11: 0xfbb3ddc6 (stale) Mirror 16: 0xfbb3ddc6 (stale) Mirror 17: 0xfbb3ddc6 (stale) Mirror 19: 0xfbb3ddc6 (stale) Mirror 20: 0xfbb3ddc6 (stale) Mirror 21: 0xfbb3ddc6 (stale) Mirror 22: 0x71e06ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f973): Mirror 3: 0xfbb3ddc6 (stale) Mirror 6: 0xfbb3ddc6 (stale) Mirror 8: 0xfbb3ddc6 (stale) Mirror 11: 0xfbb3ddc6 (stale) Mirror 16: 0xfbb3ddc6 (stale) Mirror 17: 0xfbb3ddc6 (stale) Mirror 19: 0xfbb3ddc6 (stale) Mirror 20: 0xfbb3ddc6 (stale) Mirror 21: 0xfbb3ddc6 (stale) Mirror 22: 0x71e06ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f973: skipped 000595 1779688784.327911 write 0x023911 thru 0x02a2ac (0x699c bytes) 000596 1779688784.343474 write 0x0186dd thru 0x019b61 (0x1485 bytes) 000597 1779688784.364828 read 0x024ea7 thru 0x02dd0b (0x8e65 bytes) 000598 1779688784.376611 MIRROR_RESYNC 11 mirrors 000599 1779688786.381861 write 0x03917d thru 0x03ffff (0x6e83 bytes) 000600 1779688786.445132 trunc from 0x040000 to 0x037e68 (0x8197 bytes) 000602 1779688786.473763 read 0x019899 thru 0x022403 (0x8b6b bytes) 000607 1779688786.486000 mapread 0x024cd0 thru 0x028588 (0x38b9 bytes) 000609 1779688786.507100 mapwrite 0x015751 thru 0x021dc4 (0xc674 bytes) 000610 1779688786.573088 trunc from 0x037e69 to 0x00793b (0x3052d bytes) 000611 1779688786.595905 mapread 0x0051ef thru 0x00793b (0x274d bytes) 000613 1779688786.607704 write 0x01b73f thru 0x025f80 (0xa842 bytes) 000618 1779688786.613423 mapwrite 0x001dc8 thru 0x009a8e (0x7cc7 bytes) 000620 1779688786.676007 read 0x00f6f8 thru 0x01d257 (0xdb60 bytes) 000623 1779688786.690576 write 0x01173a thru 0x021512 (0xfdd9 bytes) 000625 1779688786.692406 trunc from 0x025f81 to 0x0315ef (0xb66f bytes) 000627 1779688786.725713 write 0x002f19 thru 0x009f6c (0x7054 bytes) 000628 1779688786.727276 trunc from 0x0315f0 to 0x029e7d (0x7772 bytes) 000629 1779688786.754568 mapread 0x01079f thru 0x01da41 (0xd2a3 bytes) 000630 1779688786.756429 mapread 0x026c1f thru 0x029e7d (0x325f bytes) 000631 1779688786.779937 mapread 0x02873e thru 0x029e7d (0x1740 bytes) 000633 1779688786.781386 mapwrite 0x01eedb thru 0x028165 (0x928b bytes) 000634 1779688786.850289 mapread 0x0084c1 thru 0x00dbc8 (0x5708 bytes) 000635 1779688786.851385 trunc from 0x029e7e to 0x021f10 (0x7f6d bytes) 000637 1779688786.878355 mapread 0x00a750 thru 0x015c16 (0xb4c7 bytes) 000639 1779688786.888803 MIRROR_RESYNC 11 mirrors 000640 1779688788.860290 trunc from 0x021f11 to 0x02f3f9 (0xd4e9 bytes) 000641 1779688788.938781 mapwrite 0x00629e thru 0x012a69 (0xc7cc bytes) 000643 1779688789.022975 mapwrite 0x01c4f5 thru 0x02c03a (0xfb46 bytes) 000644 1779688789.096237 mapread 0x028e9a thru 0x02c8a2 (0x3a09 bytes) 000647 1779688789.099970 trunc from 0x02f3fa to 0x0388e2 (0x94e9 bytes) 000649 1779688789.136934 mapread 0x02cbd3 thru 0x02df4d (0x137b bytes) 000650 1779688789.138250 mapwrite 0x01a7f7 thru 0x026088 (0xb892 bytes) 000651 1779688789.199015 mapwrite 0x00edcf thru 0x01cd7a (0xdfac bytes) 000652 1779688789.249504 write 0x01bddc thru 0x01c912 (0xb37 bytes) 000653 1779688789.250739 trunc from 0x0388e3 to 0x026591 (0x12351 bytes) 000655 1779688789.275139 mapread 0x0229f0 thru 0x026591 (0x3ba2 bytes) 000657 1779688789.278129 mapwrite 0x031f83 thru 0x03dc78 (0xbcf6 bytes) 000659 1779688789.387581 read 0x038b8a thru 0x03b5c5 (0x2a3c bytes) 000660 1779688789.402528 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dc79): Mirror 6: 0x38978e23 (stale) Mirror 8: 0x38978e23 (stale) Mirror 11: 0x38978e23 (stale) Mirror 16: 0x38978e23 (stale) Mirror 17: 0x38978e23 (stale) Mirror 19: 0x38978e23 (stale) Mirror 20: 0x38978e23 (stale) Mirror 21: 0x38978e23 (stale) Mirror 22: 0x38978e23 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc79): Mirror 6: 0x38978e23 (stale) Mirror 8: 0x38978e23 (stale) Mirror 11: 0x38978e23 (stale) Mirror 16: 0x38978e23 (stale) Mirror 17: 0x38978e23 (stale) Mirror 19: 0x38978e23 (stale) Mirror 20: 0x38978e23 (stale) Mirror 21: 0x38978e23 (stale) Mirror 22: 0x38978e23 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc79: skipped 000661 1779688790.062218 trunc from 0x03dc79 to 0x010f65 (0x2cd13 bytes) 000662 1779688790.087399 write 0x03982a thru 0x03d5a0 (0x3d77 bytes) 000663 1779688790.088702 trunc from 0x03d5a1 to 0x00110e (0x3c492 bytes) 000664 1779688790.107655 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x110f): Mirror 6: 0x48bcb50e (stale) Mirror 8: 0x48bcb50e (stale) Mirror 11: 0x48bcb50e (stale) Mirror 16: 0x48bcb50e (stale) Mirror 17: 0x48bcb50e (stale) Mirror 19: 0x48bcb50e (stale) Mirror 20: 0x48bcb50e (stale) Mirror 21: 0x48bcb50e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x110f): Mirror 6: 0x48bcb50e (stale) Mirror 8: 0x48bcb50e (stale) Mirror 11: 0x48bcb50e (stale) Mirror 16: 0x48bcb50e (stale) Mirror 17: 0x48bcb50e (stale) Mirror 19: 0x48bcb50e (stale) Mirror 20: 0x48bcb50e (stale) Mirror 21: 0x48bcb50e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x110f: skipped 000667 1779688790.533788 mapread 0x000d39 thru 0x00110e (0x3d6 bytes) 000668 1779688790.565559 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x110f): Mirror 6: 0x48bcb50e (stale) Mirror 8: 0x48bcb50e (stale) Mirror 11: 0x48bcb50e (stale) Mirror 16: 0x48bcb50e (stale) Mirror 17: 0x48bcb50e (stale) Mirror 19: 0x48bcb50e (stale) Mirror 21: 0x48bcb50e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x110f): Mirror 6: 0x48bcb50e (stale) Mirror 8: 0x48bcb50e (stale) Mirror 11: 0x48bcb50e (stale) Mirror 16: 0x48bcb50e (stale) Mirror 17: 0x48bcb50e (stale) Mirror 19: 0x48bcb50e (stale) Mirror 21: 0x48bcb50e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x110f: skipped 000669 1779688790.984282 mapwrite 0x02aaca thru 0x033d20 (0x9257 bytes) 000670 1779688791.079771 read 0x028370 thru 0x033d20 (0xb9b1 bytes) 000672 1779688791.087292 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x33d21): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33d21): Mirror 6: 0xfdafbe72 (stale) Mirror 8: 0xfdafbe72 (stale) Mirror 11: 0xfdafbe72 (stale) Mirror 16: 0xfdafbe72 (stale) Mirror 17: 0xfdafbe72 (stale) Mirror 19: 0xfdafbe72 (stale) Mirror 21: 0xfdafbe72 (stale) CRC-32 checksum value for chunk [0, 0x33d21): Mirror 1: 0x72f9e7e2 Mirror 22: 0x72f9e7e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33d21: skipped 000673 1779688792.072875 mapwrite 0x0121d7 thru 0x016fe1 (0x4e0b bytes) 000676 1779688792.223597 mapread 0x03221c thru 0x033d20 (0x1b05 bytes) 000678 1779688792.245980 write 0x0324fd thru 0x03e0e7 (0xbbeb bytes) truncating to largest ever: 0x3fb0d 000681 1779688792.251982 trunc from 0x03e0e8 to 0x03fb0c (0x1a25 bytes) 000683 1779688792.307639 MIRROR_RESYNC 9 mirrors 000686 1779688794.073011 mapwrite 0x00703b thru 0x00756d (0x533 bytes) 000687 1779688794.238769 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3fb0d): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb0d): Mirror 6: 0x1d1e680b (stale) Mirror 8: 0x1d1e680b (stale) Mirror 11: 0x1d1e680b (stale) Mirror 16: 0x1d1e680b (stale) Mirror 17: 0x1d1e680b (stale) Mirror 19: 0x1d1e680b (stale) Mirror 21: 0x1d1e680b (stale) Mirror 22: 0x1d1e680b (stale) CRC-32 checksum value for chunk [0, 0x3fb0d): Mirror 1: 0x53c26fd8 Mirror 23: 0x53c26fd8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb0d: skipped 000690 1779688795.598452 mapwrite 0x00d90b thru 0x01b926 (0xe01c bytes) 000691 1779688795.817041 read 0x00e95c thru 0x010082 (0x1727 bytes) 000692 1779688795.817686 trunc from 0x03fb0d to 0x0019d5 (0x3e137 bytes) 000693 1779688795.862596 read 0x000606 thru 0x0019d5 (0x13d0 bytes) 000694 1779688795.873024 write 0x0026e0 thru 0x011590 (0xeeb1 bytes) 000698 1779688795.895499 mapread 0x0040de thru 0x00a8da (0x67fd bytes) 000699 1779688795.898436 trunc from 0x011591 to 0x00afd1 (0x65bf bytes) 000703 1779688795.942316 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0xafd2): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xafd2): Mirror 6: 0xce675fa5 (stale) Mirror 8: 0xce675fa5 (stale) Mirror 11: 0xce675fa5 (stale) Mirror 16: 0xce675fa5 (stale) Mirror 17: 0xce675fa5 (stale) Mirror 19: 0xce675fa5 (stale) Mirror 21: 0xce675fa5 (stale) Mirror 22: 0xce675fa5 (stale) Mirror 23: 0x15253d29 (stale) CRC-32 checksum value for chunk [0, 0xafd2): Mirror 1: 0xd6a72caf Mirror 24: 0xd6a72caf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xafd2: skipped 000707 1779688797.301952 mapwrite 0x02528d thru 0x03049b (0xb20f bytes) 000708 1779688797.508310 read 0x012b8e thru 0x0201ee (0xd661 bytes) 000712 1779688797.526271 mapread 0x001d08 thru 0x01122b (0xf524 bytes) 000713 1779688797.541422 write 0x019d63 thru 0x01db9f (0x3e3d bytes) 000714 1779688797.542963 mapread 0x014518 thru 0x0182e1 (0x3dca bytes) 000716 1779688797.547985 write 0x01255f thru 0x01a63d (0x80df bytes) 000717 1779688797.556411 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3049c): Mirror 6: 0x1189fd4b (stale) Mirror 8: 0x1189fd4b (stale) Mirror 11: 0x1189fd4b (stale) Mirror 16: 0x1189fd4b (stale) Mirror 17: 0x1189fd4b (stale) Mirror 19: 0x1189fd4b (stale) Mirror 22: 0x1189fd4b (stale) Mirror 23: 0xa3669164 (stale) Mirror 24: 0x7eeb73b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3049c): Mirror 6: 0x1189fd4b (stale) Mirror 8: 0x1189fd4b (stale) Mirror 11: 0x1189fd4b (stale) Mirror 16: 0x1189fd4b (stale) Mirror 17: 0x1189fd4b (stale) Mirror 19: 0x1189fd4b (stale) Mirror 22: 0x1189fd4b (stale) Mirror 23: 0xa3669164 (stale) Mirror 24: 0x7eeb73b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3049c: skipped 000718 1779688798.188895 write 0x00fc19 thru 0x01743a (0x7822 bytes) 000721 1779688798.211125 MIRROR_RESYNC 10 mirrors 000722 1779688799.862836 mapread 0x027e21 thru 0x02c126 (0x4306 bytes) 000724 1779688799.886991 trunc from 0x03049c to 0x029090 (0x740b bytes) 000725 1779688799.979141 mapread 0x027547 thru 0x029090 (0x1b4a bytes) 000726 1779688799.991999 mapread 0x024768 thru 0x029090 (0x4929 bytes) 000727 1779688799.994565 mapread 0x01f094 thru 0x025297 (0x6204 bytes) 000728 1779688800.005814 mapwrite 0x0209a0 thru 0x02a16f (0x97d0 bytes) 000731 1779688800.097042 trunc from 0x02a170 to 0x03a74d (0x105de bytes) 000735 1779688800.128866 read 0x01fab7 thru 0x025521 (0x5a6b bytes) 000736 1779688800.144635 MIRROR_RESYNC 10 mirrors 000737 1779688801.752849 write 0x01603c thru 0x01e861 (0x8826 bytes) 000739 1779688801.901672 mapread 0x0195fc thru 0x01e024 (0x4a29 bytes) 000740 1779688801.902852 mapwrite 0x02b1ec thru 0x02d695 (0x24aa bytes) 000741 1779688801.958788 write 0x014e3b thru 0x015d37 (0xefd bytes) 000743 1779688801.965753 read 0x0010b2 thru 0x001866 (0x7b5 bytes) 000744 1779688801.972353 MIRROR_RESYNC 10 mirrors 000745 1779688803.573463 mapwrite 0x018ab2 thru 0x01de92 (0x53e1 bytes) 000746 1779688803.814607 write 0x01dfdb thru 0x0247d4 (0x67fa bytes) 000748 1779688803.822773 mapread 0x017e70 thru 0x01d930 (0x5ac1 bytes) 000749 1779688803.828082 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a74e): Mirror 6: 0xc01d6ae0 (stale) Mirror 8: 0xc01d6ae0 (stale) Mirror 11: 0xc01d6ae0 (stale) Mirror 16: 0xc01d6ae0 (stale) Mirror 17: 0xc01d6ae0 (stale) Mirror 19: 0xc01d6ae0 (stale) Mirror 22: 0xc01d6ae0 (stale) Mirror 23: 0xc01d6ae0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a74e): Mirror 6: 0xc01d6ae0 (stale) Mirror 8: 0xc01d6ae0 (stale) Mirror 11: 0xc01d6ae0 (stale) Mirror 16: 0xc01d6ae0 (stale) Mirror 17: 0xc01d6ae0 (stale) Mirror 19: 0xc01d6ae0 (stale) Mirror 22: 0xc01d6ae0 (stale) Mirror 23: 0xc01d6ae0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a74e: skipped 000750 1779688804.376822 mapread 0x013808 thru 0x015d06 (0x24ff bytes) 000752 1779688804.405982 trunc from 0x03a74e to 0x0376da (0x3073 bytes) 000756 1779688804.437111 trunc from 0x0376db to 0x00a01d (0x2d6bd bytes) 000757 1779688804.457783 mapread 0x008160 thru 0x00a01d (0x1ebe bytes) 000758 1779688804.470573 mapread 0x007554 thru 0x00a01d (0x2aca bytes) 000759 1779688804.472059 mapwrite 0x033509 thru 0x03ffff (0xcaf7 bytes) 000760 1779688804.561080 trunc from 0x040000 to 0x003382 (0x3cc7d bytes) 000761 1779688804.580052 read 0x00131b thru 0x003382 (0x2068 bytes) 000762 1779688804.582556 mapread 0x000621 thru 0x003382 (0x2d62 bytes) 000763 1779688804.590687 mapread 0x0009ad thru 0x003382 (0x29d6 bytes) 000765 1779688804.592948 trunc from 0x003383 to 0x0341c7 (0x30e45 bytes) 000766 1779688804.615035 read 0x008bff thru 0x00fe61 (0x7263 bytes) 000768 1779688804.629093 trunc from 0x0341c8 to 0x033237 (0xf90 bytes) 000769 1779688804.650185 mapwrite 0x012a55 thru 0x01f62e (0xcbda bytes) 000770 1779688804.754915 trunc from 0x033238 to 0x039488 (0x6251 bytes) 000772 1779688804.809382 trunc from 0x039489 to 0x00c7ab (0x2ccdd bytes) 000773 1779688804.833497 trunc from 0x00c7ac to 0x003cab (0x8b00 bytes) 000774 1779688804.869012 trunc from 0x003cac to 0x007f09 (0x425e bytes) 000781 1779688804.904649 write 0x02ec70 thru 0x0359e2 (0x6d73 bytes) 000783 1779688804.908568 read 0x027e04 thru 0x02c1c3 (0x43c0 bytes) 000784 1779688804.918696 mapwrite 0x0267fb thru 0x03501f (0xe825 bytes) 000785 1779688804.999117 mapwrite 0x025d7f thru 0x02f0c6 (0x9348 bytes) 000786 1779688805.053921 mapwrite 0x0038df thru 0x0053eb (0x1b0d bytes) 000787 1779688805.113861 read 0x01995e thru 0x023006 (0x96a9 bytes) 000788 1779688805.114716 write 0x03b621 thru 0x03ffff (0x49df bytes) 000790 1779688805.122735 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x849f0529 (stale) Mirror 11: 0x849f0529 (stale) Mirror 16: 0x849f0529 (stale) Mirror 17: 0x849f0529 (stale) Mirror 19: 0x849f0529 (stale) Mirror 22: 0x849f0529 (stale) Mirror 23: 0x849f0529 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x849f0529 (stale) Mirror 11: 0x849f0529 (stale) Mirror 16: 0x849f0529 (stale) Mirror 17: 0x849f0529 (stale) Mirror 19: 0x849f0529 (stale) Mirror 22: 0x849f0529 (stale) Mirror 23: 0x849f0529 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000791 1779688805.756091 trunc from 0x040000 to 0x01f0b4 (0x20f4b bytes) 000793 1779688805.779991 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f0b5): Mirror 8: 0x1605ee09 (stale) Mirror 11: 0x1605ee09 (stale) Mirror 16: 0x1605ee09 (stale) Mirror 17: 0x1605ee09 (stale) Mirror 19: 0x1605ee09 (stale) Mirror 22: 0x1605ee09 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f0b5): Mirror 8: 0x1605ee09 (stale) Mirror 11: 0x1605ee09 (stale) Mirror 16: 0x1605ee09 (stale) Mirror 17: 0x1605ee09 (stale) Mirror 19: 0x1605ee09 (stale) Mirror 22: 0x1605ee09 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f0b5: skipped 000794 1779688806.209334 MIRROR_RESYNC 7 mirrors 000795 1779688807.547669 mapwrite 0x00bf4e thru 0x017ba6 (0xbc59 bytes) 000796 1779688807.721788 mapwrite 0x035b4f thru 0x035d7e (0x230 bytes) 000797 1779688807.818970 read 0x002692 thru 0x01218b (0xfafa bytes) 000799 1779688807.834095 read 0x014aa3 thru 0x01f9f3 (0xaf51 bytes) 000800 1779688807.849876 mapwrite 0x003ce5 thru 0x00b249 (0x7565 bytes) 000801 1779688807.909595 write 0x01c4d2 thru 0x0223a9 (0x5ed8 bytes) 000802 1779688807.910790 mapwrite 0x00a4eb thru 0x0189e1 (0xe4f7 bytes) 000804 1779688807.972477 trunc from 0x035d7f to 0x012bce (0x231b0 bytes) 000805 1779688807.995903 trunc from 0x012bcf to 0x02d96e (0x1ada0 bytes) 000807 1779688808.018825 mapread 0x009b98 thru 0x017feb (0xe454 bytes) 000808 1779688808.051556 write 0x035b3e thru 0x036264 (0x727 bytes) 000810 1779688808.053767 write 0x012771 thru 0x018416 (0x5ca6 bytes) 000811 1779688808.060778 read 0x01eda1 thru 0x01efbf (0x21f bytes) 000813 1779688808.072677 mapread 0x005356 thru 0x0056df (0x38a bytes) 000815 1779688808.075643 mapread 0x011992 thru 0x0136d7 (0x1d46 bytes) 000816 1779688808.076849 trunc from 0x036265 to 0x033b94 (0x26d0 bytes) 000817 1779688808.111668 write 0x015788 thru 0x01d78f (0x8008 bytes) 000821 1779688808.116784 read 0x032431 thru 0x033b94 (0x1764 bytes) 000824 1779688808.119912 trunc from 0x033b95 to 0x02906d (0xab27 bytes) 000825 1779688808.143896 read 0x026e41 thru 0x028155 (0x1315 bytes) 000826 1779688808.145022 trunc from 0x02906e to 0x0081b0 (0x20ebd bytes) 000827 1779688808.171022 trunc from 0x0081b1 to 0x023765 (0x1b5b5 bytes) 000829 1779688808.202264 mapread 0x01c058 thru 0x023765 (0x770e bytes) 000830 1779688808.214849 mapwrite 0x011e98 thru 0x01adec (0x8f55 bytes) 000831 1779688808.262894 trunc from 0x023766 to 0x013629 (0x1013c bytes) 000832 1779688808.278108 trunc from 0x01362a to 0x031757 (0x1e12e bytes) 000833 1779688808.309111 trunc from 0x031758 to 0x033cae (0x2557 bytes) 000836 1779688808.333502 read 0x01ec2a thru 0x02b1d1 (0xc5a8 bytes) 000838 1779688808.344793 write 0x0009c8 thru 0x00b3c7 (0xaa00 bytes) 000840 1779688808.347311 read 0x001ef8 thru 0x003592 (0x169b bytes) 000841 1779688808.352798 MIRROR_RESYNC 7 mirrors 000844 1779688809.581872 mapread 0x022081 thru 0x024001 (0x1f81 bytes) 000845 1779688809.625614 mapread 0x0044bc thru 0x013163 (0xeca8 bytes) 000846 1779688809.658976 mapread 0x012f16 thru 0x019cce (0x6db9 bytes) 000847 1779688809.663418 mapread 0x02c33e thru 0x033cae (0x7971 bytes) 000848 1779688809.679618 trunc from 0x033caf to 0x000a6f (0x3323f bytes) 000850 1779688809.810214 trunc from 0x000a70 to 0x031a90 (0x31021 bytes) 000851 1779688809.836434 mapwrite 0x02ef8c thru 0x033ef5 (0x4f6a bytes) 000853 1779688809.915769 read 0x0142c0 thru 0x01c4ba (0x81fb bytes) 000857 1779688809.928761 write 0x03ba48 thru 0x03ffff (0x45b8 bytes) 000858 1779688809.931801 mapread 0x020e62 thru 0x029e41 (0x8fe0 bytes) 000859 1779688809.941777 mapwrite 0x011636 thru 0x0145d7 (0x2fa2 bytes) 000860 1779688810.007985 MIRROR_RESYNC 7 mirrors 000864 1779688811.470856 write 0x02de1b thru 0x03bad5 (0xdcbb bytes) 000865 1779688811.630435 read 0x03e53e thru 0x03ffff (0x1ac2 bytes) 000866 1779688811.644185 mapwrite 0x028180 thru 0x036076 (0xdef7 bytes) 000869 1779688811.718088 trunc from 0x040000 to 0x01b4be (0x24b41 bytes) 000871 1779688811.746248 read 0x000ce0 thru 0x00a385 (0x96a6 bytes) 000872 1779688811.761909 mapwrite 0x01638d thru 0x0207aa (0xa41e bytes) 000873 1779688811.856875 mapwrite 0x02294f thru 0x02d6cb (0xad7d bytes) 000875 1779688812.006910 mapread 0x0065a5 thru 0x015c5c (0xf6b8 bytes) 000876 1779688812.011373 trunc from 0x02d6cc to 0x02b862 (0x1e69 bytes) 000877 1779688812.043942 mapread 0x01d8c2 thru 0x02b862 (0xdfa1 bytes) 000878 1779688812.046120 mapread 0x01ea2d thru 0x01fb30 (0x1104 bytes) 000879 1779688812.070021 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2b863): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b863): Mirror 8: 0x4c695ecf (stale) Mirror 11: 0x4c695ecf (stale) Mirror 16: 0x4c695ecf (stale) Mirror 17: 0x4c695ecf (stale) Mirror 19: 0x4c695ecf (stale) Mirror 22: 0x4c695ecf (stale) CRC-32 checksum value for chunk [0, 0x2b863): Mirror 1: 0x7ce3e15a Mirror 23: 0x7ce3e15a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b863: skipped 000881 1779688813.391565 trunc from 0x02b863 to 0x03bec1 (0x1065f bytes) 000882 1779688813.551820 trunc from 0x03bec2 to 0x01be27 (0x2009a bytes) 000884 1779688813.580886 mapread 0x007a46 thru 0x00e1ef (0x67aa bytes) 000885 1779688813.606222 write 0x023da5 thru 0x02c2fd (0x8559 bytes) 000886 1779688813.608943 mapread 0x001bf5 thru 0x006541 (0x494d bytes) 000887 1779688813.616734 mapwrite 0x0209a1 thru 0x0226c0 (0x1d20 bytes) 000888 1779688813.718404 write 0x03beb6 thru 0x03ffff (0x414a bytes) 000889 1779688813.720911 trunc from 0x040000 to 0x011ca0 (0x2e35f bytes) 000891 1779688813.766049 mapread 0x00c125 thru 0x00d744 (0x1620 bytes) 000892 1779688813.767836 write 0x01c70d thru 0x029d11 (0xd605 bytes) 000893 1779688813.771488 mapread 0x00e84c thru 0x012c04 (0x43b9 bytes) 000894 1779688813.783716 write 0x01f7d2 thru 0x02a122 (0xa951 bytes) 000897 1779688813.786826 write 0x0083e5 thru 0x010ca6 (0x88c2 bytes) 000898 1779688813.788431 write 0x016847 thru 0x017c2b (0x13e5 bytes) 000899 1779688813.790827 mapwrite 0x00453e thru 0x0045b1 (0x74 bytes) 000900 1779688813.839972 write 0x012293 thru 0x01e2d5 (0xc043 bytes) 000901 1779688813.841511 trunc from 0x02a123 to 0x01daba (0xc668 bytes) 000902 1779688813.868448 MIRROR_RESYNC 8 mirrors 000903 1779688815.660694 mapwrite 0x03091d thru 0x0377a4 (0x6e88 bytes) 000904 1779688815.815637 write 0x0388fa thru 0x03a3c5 (0x1acc bytes) 000905 1779688815.816852 read 0x013717 thru 0x01743c (0x3d26 bytes) 000907 1779688815.824292 trunc from 0x03a3c6 to 0x020770 (0x19c55 bytes) 000908 1779688815.844117 MIRROR_RESYNC 8 mirrors 000910 1779688817.452838 trunc from 0x020771 to 0x03a72c (0x19fbc bytes) 000911 1779688817.575840 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3a72d): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a72d): Mirror 8: 0x42542b0c (stale) Mirror 11: 0x42542b0c (stale) Mirror 16: 0x42542b0c (stale) Mirror 17: 0x42542b0c (stale) Mirror 19: 0x42542b0c (stale) Mirror 22: 0x42542b0c (stale) Mirror 23: 0x42542b0c (stale) CRC-32 checksum value for chunk [0, 0x3a72d): Mirror 1: 0x42542b0c Mirror 24: 0x42542b0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a72d: skipped 000912 1779688818.624768 write 0x00a933 thru 0x014076 (0x9744 bytes) 000914 1779688818.706724 trunc from 0x03a72d to 0x00b7a2 (0x2ef8a bytes) 000917 1779688818.725868 read 0x00b42a thru 0x00b7a2 (0x379 bytes) 000918 1779688818.726738 read 0x00108a thru 0x00b7a2 (0xa719 bytes) 000919 1779688818.732996 read 0x000dd7 thru 0x00a0c3 (0x92ed bytes) 000922 1779688818.734883 trunc from 0x00b7a3 to 0x00b0bf (0x6e3 bytes) 000923 1779688818.755944 write 0x00d8bd thru 0x01b96e (0xe0b2 bytes) 000924 1779688818.757823 read 0x001464 thru 0x008dba (0x7957 bytes) 000925 1779688818.759778 mapwrite 0x02acfa thru 0x030df3 (0x60fa bytes) 000927 1779688818.858016 read 0x008611 thru 0x01855c (0xff4c bytes) 000928 1779688818.859507 read 0x020140 thru 0x020925 (0x7e6 bytes) 000931 1779688818.861646 write 0x022c86 thru 0x028867 (0x5be2 bytes) 000932 1779688818.863610 read 0x02383f thru 0x030df3 (0xd5b5 bytes) 000933 1779688818.864446 write 0x009886 thru 0x010524 (0x6c9f bytes) 000935 1779688818.870776 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x30df4): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30df4): Mirror 8: 0x76f1d146 (stale) Mirror 11: 0x76f1d146 (stale) Mirror 16: 0x76f1d146 (stale) Mirror 17: 0x76f1d146 (stale) Mirror 19: 0x76f1d146 (stale) Mirror 22: 0x76f1d146 (stale) Mirror 23: 0x76f1d146 (stale) Mirror 24: 0x76f1d146 (stale) CRC-32 checksum value for chunk [0, 0x30df4): Mirror 1: 0xf5116a9e Mirror 25: 0xf5116a9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30df4: skipped 000936 1779688819.803025 read 0x03078a thru 0x030df3 (0x66a bytes) 000938 1779688819.833852 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x30df4): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30df4): Mirror 8: 0x76f1d146 (stale) Mirror 11: 0x76f1d146 (stale) Mirror 16: 0x76f1d146 (stale) Mirror 17: 0x76f1d146 (stale) Mirror 19: 0x76f1d146 (stale) Mirror 22: 0x76f1d146 (stale) Mirror 23: 0x76f1d146 (stale) Mirror 24: 0x76f1d146 (stale) CRC-32 checksum value for chunk [0, 0x30df4): Mirror 1: 0xf5116a9e Mirror 25: 0xf5116a9e Mirror 26: 0xf5116a9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30df4: skipped 000939 1779688820.702769 write 0x021f3f thru 0x02fd11 (0xddd3 bytes) 000940 1779688820.777595 MIRROR_RESYNC 11 mirrors 000943 1779688822.512129 read 0x013a2e thru 0x01abf9 (0x71cc bytes) 000944 1779688822.526336 read 0x011c85 thru 0x01483a (0x2bb6 bytes) 000946 1779688822.536694 mapwrite 0x007823 thru 0x008ac9 (0x12a7 bytes) 000947 1779688822.683930 read 0x023b43 thru 0x0253cf (0x188d bytes) 000951 1779688822.704389 read 0x0298a0 thru 0x030df3 (0x7554 bytes) 000952 1779688822.710260 MIRROR_RESYNC 11 mirrors 000953 1779688824.262933 read 0x015d9d thru 0x01b30e (0x5572 bytes) 000954 1779688824.282669 mapwrite 0x014186 thru 0x01ed56 (0xabd1 bytes) 000956 1779688824.539368 read 0x027290 thru 0x030df3 (0x9b64 bytes) 000957 1779688824.552510 mapwrite 0x01150c thru 0x01dbd7 (0xc6cc bytes) 000961 1779688824.617569 write 0x030600 thru 0x038ecc (0x88cd bytes) 000962 1779688824.620384 mapwrite 0x007dad thru 0x008e09 (0x105d bytes) 000964 1779688824.722792 mapread 0x008de7 thru 0x00c0c2 (0x32dc bytes) 000965 1779688824.723909 mapwrite 0x025e6b thru 0x0335db (0xd771 bytes) 000966 1779688824.794468 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x38ecd): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ecd): Mirror 8: 0x31e04460 (stale) Mirror 11: 0x31e04460 (stale) Mirror 16: 0x31e04460 (stale) Mirror 17: 0x31e04460 (stale) Mirror 19: 0x31e04460 (stale) Mirror 22: 0x31e04460 (stale) Mirror 23: 0x31e04460 (stale) Mirror 24: 0x31e04460 (stale) Mirror 25: 0x31e04460 (stale) Mirror 26: 0x31e04460 (stale) CRC-32 checksum value for chunk [0, 0x38ecd): Mirror 1: 0x2df8c4fa Mirror 27: 0x2df8c4fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ecd: skipped 000967 1779688826.447934 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x38ecd): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ecd): Mirror 8: 0x31e04460 (stale) Mirror 11: 0x31e04460 (stale) Mirror 16: 0x31e04460 (stale) Mirror 17: 0x31e04460 (stale) Mirror 19: 0x31e04460 (stale) Mirror 22: 0x31e04460 (stale) Mirror 23: 0x31e04460 (stale) Mirror 24: 0x31e04460 (stale) Mirror 25: 0x31e04460 (stale) Mirror 26: 0x31e04460 (stale) CRC-32 checksum value for chunk [0, 0x38ecd): Mirror 1: 0x2df8c4fa Mirror 27: 0x2df8c4fa Mirror 28: 0x2df8c4fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ecd: skipped 000968 1779688827.631452 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38ecd): Mirror 8: 0x31e04460 (stale) Mirror 11: 0x31e04460 (stale) Mirror 16: 0x31e04460 (stale) Mirror 19: 0x31e04460 (stale) Mirror 22: 0x31e04460 (stale) Mirror 23: 0x31e04460 (stale) Mirror 24: 0x31e04460 (stale) Mirror 25: 0x31e04460 (stale) Mirror 26: 0x31e04460 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ecd): Mirror 8: 0x31e04460 (stale) Mirror 11: 0x31e04460 (stale) Mirror 16: 0x31e04460 (stale) Mirror 19: 0x31e04460 (stale) Mirror 22: 0x31e04460 (stale) Mirror 23: 0x31e04460 (stale) Mirror 24: 0x31e04460 (stale) Mirror 25: 0x31e04460 (stale) Mirror 26: 0x31e04460 (stale) CRC-32 checksum value for chunk [0, 0x38ecd): Mirror 1: 0x2df8c4fa Mirror 27: 0x2df8c4fa Mirror 28: 0x2df8c4fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ecd: skipped 000970 1779688828.515516 trunc from 0x038ecd to 0x00a804 (0x2e6c8 bytes) 000972 1779688828.650358 trunc from 0x00a805 to 0x03ecba (0x344b6 bytes) 000973 1779688828.697670 trunc from 0x03ecbb to 0x038bf7 (0x60c3 bytes) 000974 1779688828.717954 mapread 0x013168 thru 0x01f6cb (0xc564 bytes) 000977 1779688828.773742 mapwrite 0x010c11 thru 0x01c8ad (0xbc9d bytes) 000978 1779688828.836060 trunc from 0x038bf8 to 0x023ac1 (0x15136 bytes) 000979 1779688828.871295 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x182:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23ac2): Mirror 8: 0x86014cf9 (stale) Mirror 11: 0x86014cf9 (stale) Mirror 16: 0x86014cf9 (stale) Mirror 19: 0x86014cf9 (stale) Mirror 22: 0x86014cf9 (stale) Mirror 23: 0x86014cf9 (stale) Mirror 24: 0x86014cf9 (stale) Mirror 25: 0x86014cf9 (stale) Mirror 26: 0x86014cf9 (stale) Mirror 28: 0x6d549fd3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23ac2): Mirror 8: 0x86014cf9 (stale) Mirror 11: 0x86014cf9 (stale) Mirror 16: 0x86014cf9 (stale) Mirror 19: 0x86014cf9 (stale) Mirror 22: 0x86014cf9 (stale) Mirror 23: 0x86014cf9 (stale) Mirror 24: 0x86014cf9 (stale) Mirror 25: 0x86014cf9 (stale) Mirror 26: 0x86014cf9 (stale) Mirror 28: 0x6d549fd3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23ac2: skipped 000980 1779688829.569126 write 0x00abd8 thru 0x00c3a1 (0x17ca bytes) 000982 1779688829.595427 trunc from 0x023ac2 to 0x00f637 (0x1448a bytes) 000983 1779688829.625576 mapread 0x00e897 thru 0x00f637 (0xda1 bytes) 000984 1779688829.638749 mapwrite 0x035166 thru 0x03da82 (0x891d bytes) 000986 1779688829.771598 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3da83): Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da83): Mirror 8: 0x4bade71f (stale) Mirror 11: 0x4bade71f (stale) Mirror 16: 0x4bade71f (stale) Mirror 19: 0x4bade71f (stale) Mirror 22: 0x4bade71f (stale) Mirror 23: 0x4bade71f (stale) Mirror 24: 0x4bade71f (stale) Mirror 25: 0x4bade71f (stale) Mirror 26: 0x4bade71f (stale) Mirror 28: 0x80a7f8f1 (stale) CRC-32 checksum value for chunk [0, 0x3da83): Mirror 1: 0xaac98473 Mirror 29: 0xaac98473 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da83: skipped 000989 1779688830.948323 read 0x025bbe thru 0x0315e4 (0xba27 bytes) 000990 1779688830.970670 mapwrite 0x015ea9 thru 0x0232d5 (0xd42d bytes) 000992 1779688831.186813 trunc from 0x03da83 to 0x01e0f3 (0x1f98f bytes) 000994 1779688831.232677 trunc from 0x01e0f4 to 0x021e53 (0x3d60 bytes) 000995 1779688831.253508 trunc from 0x021e54 to 0x032640 (0x107ed bytes) 000996 1779688831.273256 read 0x003b6d thru 0x00a724 (0x6bb8 bytes) 000998 1779688831.289585 mapwrite 0x0394ba thru 0x03ffff (0x6b46 bytes) 000999 1779688831.368183 read 0x0154f1 thru 0x020edd (0xb9ed bytes) All operations completed A-OK! PASS 70a (135s) == sanity-flr test 71: check mirror extend parent fid ==== 02:00:37 (1779688837) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 02:00:41 (1779688841) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.366552 s, 5.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21539 s, 4.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 02:00:51 (1779688851) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2735520 .. Extending file size to 7866304 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8535584 .. failed Extending file size to 2902432 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3174880 .. Extending file size to 6590560 .. Extending file size to 7840544 .. Extending file size to 8558400 .. Extending file size to 8960416 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 56604 56605 56606 56607 56609 done 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) 192.168.202.150@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:02:15 (1779688935) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6342272 .. Extending file size to 1638976 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7383776 .. Extending file size to 8610048 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8658848 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8710464 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7457600 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8372384 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8455360 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8957088 .. Waiting 58238 58239 58240 58242 58245 done 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) 192.168.202.150@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:03:36 (1779689016) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.122571 s, 17.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.349696 s, 6.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x37645438 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:03:57 (1779689037) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.107157 s, 19.6 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 02:04:05 (1779689045) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ca:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2df:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:04:13 (1779689053) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.281448 s, 7.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (11s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:04:24 (1779689064) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.334945 s, 6.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e2:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e3:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:04:34 (1779689074) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.423289 s, 5.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e4:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e5:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:05:09 (1779689109) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg250-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249252 s, 4.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:06:08 (1779689168) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg250-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.346746 s, 6.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (59s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:07:08 (1779689228) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg250-server mds-ost sync done. Creating new pool test_204f oleg250-server: Pool lustre.test_204f created Adding targets to pool oleg250-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg250-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.203228 s, 10.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg250-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg250-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg250-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:08:27 (1779689307) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:08:35 (1779689315) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:08:43 (1779689323) oleg250-server: Pool lustre.test_206 created Adding targets to pool oleg250-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg250-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg250-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg250-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg250-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:09:10 (1779689350) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0944722 s, 11.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496586 s, 21.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf2431237 Mirror 3: 0xf2431237 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:09:23 (1779689363) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:09:27 (1779689367) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:09:31 (1779689371) Starting client: oleg250-client.virtnet: -o user_xattr,flock 192.168.202.150@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.341452 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2600000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1540000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1540000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1540000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x6a0000 Read this (4 bytes): BBBB 192.168.202.150@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg250-client.virtnet /mnt/lustre2 (opts:) PASS 209a (47s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:10:18 (1779689418) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.019348 s, 423 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:10:26 (1779689426) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493851 s, 21.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1ef lmm_fid: [0x200000404:0x1ef:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1ef lmm_fid: [0x200000404:0x1ef:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834415 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-05-25 02:10:28.000000000 -0400 Modify: 2026-05-25 02:10:28.000000000 -0400 Change: 2026-05-25 02:10:29.000000000 -0400 Birth: 2026-05-25 02:10:27.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:10:35 (1779689435) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:10:39 (1779689439) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.485718 s, 84.3 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0457219 s, 89.6 kB/s size after second write -rw-r--r-- 1 root root 4096 May 25 02:10 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1f1:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 May 25 02:10 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:10:48 (1779689448) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:10:57 (1779689457) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1924 sec =========== 02:11:07 (1779689467) === sanity-flr: start cleanup 02:11:09 (1779689469) === === sanity-flr: finish cleanup 02:11:17 (1779689477) ===