-----============= acceptance-small: sanity-flr ============----- Thu Jun 11 22:07:48 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg360-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 22:08:07 (1781230087) === oleg360-client.virtnet: executing check_config_client /mnt/lustre oleg360-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg360-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9773d2861000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9773d2861000.idle_timeout=debug disable quota as required oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 22:08:38 (1781230118) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7577] == sanity-flr test 0a: lfs mirror create with -N option == 22:08:40 (1781230120) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:08:57 (1781230137) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:09:01 (1781230141) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:09:04 (1781230144) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:09:29 (1781230169) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:09:32 (1781230172) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 22:09:36 (1781230176) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9773d2861000.stats=clear osc.lustre-OST0001-osc-ffff9773d2861000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:09:44 (1781230184) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--compress|-Z] [--compress-chunk=SIZE] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:09:56 (1781230196) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:10:04 (1781230204) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:10:07 (1781230207) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:10:15 (1781230215) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 22:10:24 (1781230224) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:10:31 (1781230231) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:10:34 (1781230234) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:10:41 (1781230241) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:10:51 (1781230251) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:10:57 (1781230257) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 2.1852 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132808 s, 7.9 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:11:08 (1781230268) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.322504 s, 35.8 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:11:16 (1781230276) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:11:27 (1781230287) Stopping /mnt/lustre-ost1 (opts:) on oleg360-server oleg360-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9773d2861000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9773d2861000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg360-server: oleg360-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg360-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg360-server oleg360-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9773d2861000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9773d2861000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg360-server: oleg360-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg360-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (59s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:12:26 (1781230346) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.392829 s, 18.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg360-server oleg360-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9773d2861000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9773d2861000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg360-server: oleg360-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg360-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:13:06 (1781230386) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg360-server oleg360-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9773d2861000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9773d2861000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg360-server: oleg360-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg360-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg360-server oleg360-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9773d2861000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9773d2861000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg360-server: oleg360-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg360-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (79s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:14:25 (1781230465) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 6.26404 s, 8.4 MB/s -rw-r--r-- 1 root root 52428800 Jun 11 22:14 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.264s user 0m0.000s sys 0m0.192s Stopping /mnt/lustre-ost1 (opts:) on oleg360-server oleg360-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9773d2861000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9773d2861000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.592s user 0m0.002s sys 0m0.202s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg360-server: oleg360-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg360-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg360-server oleg360-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9773d2861000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9773d2861000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.123s user 0m0.001s sys 0m0.182s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg360-server: oleg360-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg360-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg360-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9773d2861000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9773d2861000.ost_server_uuid in FULL state after 0 sec PASS 33b (91s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:15:56 (1781230556) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:16:00 (1781230560) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:16:03 (1781230563) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 22:16:07 (1781230567) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 22:16:15 (1781230575) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.537105 s, 23.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:16:27 (1781230587) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0401883 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347585 s, 30.2 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:16:35 (1781230595) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:16:44 (1781230604) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0073049 s, 140 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0114125 s, 89.7 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00176224 s, 581 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00177492 s, 577 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 22:16:56 (1781230616) create test files with size 15733366 .. aec0b1c76c7b94a2bec5d87feb2acf4c - fdf5323bda1308888f74b27305adfacc - 7c2fc3ff28847dae4317b5781ce3a7e6 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg360-client.virtnet /mnt/lustre (opts:) Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre PASS 37 (24s) == sanity-flr test 38: resync ============================ 22:17:20 (1781230640) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:17:38 (1781230658) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:17:45 (1781230665) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174544 s, 6.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xae:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xaf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xed:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346453 s, 30.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xee:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xef:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xb0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 22:17:54 (1781230674) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.206454 s, 10.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.389931 s, 10.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.305424 s, 6.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 22:18:11 (1781230691) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 22:18:14 (1781230694) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:18:18 (1781230698) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0549926 s, 19.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 43c: read prefer uncompressed mirror == 22:18:26 (1781230706) llite.lustre-ffff9773d00bd800.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (8s) == sanity-flr test 44a: lfs mirror split check =========== 22:18:33 (1781230713) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:18:37 (1781230717) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xf7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb8:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xb8:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:18:47 (1781230727) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.237448 s, 44.2 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1a:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg360-client.virtnet /mnt/lustre (opts:) Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:18:57 (1781230737) llite.lustre-ffff9773c6159000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.311769 s, 33.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg360-client.virtnet /mnt/lustre (opts:) Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9773d2759000.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:19:11 (1781230751) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:19:48 (1781230788) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:20:25 (1781230825) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 22:20:28 (1781230828) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.297008 s, 10.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.146187 s, 21.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x8dcdc226 (nosync,stale) Mirror 2: 0x8dcdc226 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 093bb48ac22ae61a987243fbadc70ce5 - mirror 1 checksum: c68e0a4d14d8cb8b51d9b624a47affb9 - mirror 2 checksum: 093bb48ac22ae61a987243fbadc70ce5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 22:20:31 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 22:20:31 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xce:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 093bb48ac22ae61a987243fbadc70ce5 - mirror 1 checksum: c68e0a4d14d8cb8b51d9b624a47affb9 - mirror 2 checksum: c68e0a4d14d8cb8b51d9b624a47affb9 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 22:20:31 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 22:20:31 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xce:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:20:40 (1781230840) Failing mds1 on oleg360-server Stopping /mnt/lustre-mds1 (opts:) on oleg360-server 22:20:48 (1781230848) shut down facet: mds1 facet_host: oleg360-server facet_failover_host: oleg360-server Failover mds1 to oleg360-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg360-server: oleg360-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg360-client: oleg360-server: ssh exited with exit code 1 Started lustre-MDT0000 22:21:19 (1781230879) targets are mounted 22:21:19 (1781230879) facet_failover done oleg360-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.044466 s, 23.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xcf:0x0] } PASS 50A (58s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:21:39 (1781230899) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:21:42 (1781230902) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:21:45 (1781230905) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0105386 s, 1.6 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:21:53 (1781230913) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:21:57 (1781230917) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:22:00 (1781230920) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:22:24 (1781230944) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:22:54 (1781230974) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:23:19 (1781230999) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:23:43 (1781231023) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 22:23:50 (1781231030) mirror create pid 54125 mirror split pid 54126 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 54125 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 54126 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 22:24:58 (1781231098) Seed set to 1101 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000005 1781231101.646106 write 0x02d710 thru 0x039e22 (0xc713 bytes) 000006 1781231101.649940 read 0x00e159 thru 0x014f3e (0x6de6 bytes) truncating to largest ever: 0x35c7e 000008 1781231101.659947 trunc from 0x039e23 to 0x035c7d (0x41a5 bytes) 000010 1781231101.694991 mapread 0x00c6bf thru 0x010820 (0x4162 bytes) 000012 1781231101.711803 read 0x02c30b thru 0x02d976 (0x166c bytes) 000013 1781231101.734191 read 0x006b18 thru 0x012f88 (0xc471 bytes) 000016 1781231101.736507 mapwrite 0x00d1d7 thru 0x0196b6 (0xc4e0 bytes) 000017 1781231101.917853 read 0x00a4a4 thru 0x016522 (0xc07f bytes) 000018 1781231101.921066 mapwrite 0x01a582 thru 0x0215f7 (0x7076 bytes) 000020 1781231102.032492 mapread 0x0074e7 thru 0x00f6ab (0x81c5 bytes) 000021 1781231102.035390 mapread 0x02c0cc thru 0x035c7d (0x9bb2 bytes) 000022 1781231102.036776 trunc from 0x035c7e to 0x02bd7e (0x9eff bytes) 000024 1781231102.070489 write 0x038653 thru 0x03ffff (0x79ad bytes) 000027 1781231102.081027 trunc from 0x040000 to 0x02d99e (0x12661 bytes) 000029 1781231102.158254 mapwrite 0x017944 thru 0x020254 (0x8911 bytes) 000033 1781231102.254919 MIRROR_EXTEND to 2 mirrors 000034 1781231103.231691 mapread 0x0268de thru 0x02d99e (0x70c1 bytes) 000036 1781231103.291565 read 0x000a11 thru 0x005eb3 (0x54a3 bytes) 000037 1781231103.318655 write 0x01196b thru 0x01a7c3 (0x8e59 bytes) 000039 1781231103.561666 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) 000040 1781231103.831829 read 0x00c866 thru 0x011922 (0x50bd bytes) 000041 1781231103.850998 read 0x018af0 thru 0x019d2a (0x123b bytes) 000045 1781231103.868909 write 0x02aba3 thru 0x033ed9 (0x9337 bytes) 000047 1781231103.871244 mapwrite 0x024dcf thru 0x0270f3 (0x2325 bytes) 000048 1781231104.031885 write 0x0120b9 thru 0x01b6a9 (0x95f1 bytes) 000052 1781231104.041688 trunc from 0x033eda to 0x002505 (0x319d4 bytes) truncating to largest ever: 0x379f6 000053 1781231104.123131 trunc from 0x002506 to 0x0379f5 (0x354f0 bytes) 000054 1781231104.158274 read 0x0264da thru 0x031f6e (0xba95 bytes) 000055 1781231104.211952 MIRROR_EXTEND to 2 mirrors 000056 1781231105.068636 mapread 0x032bef thru 0x0379f5 (0x4e07 bytes) 000060 1781231105.100360 mapread 0x01cd67 thru 0x02bbe4 (0xee7e bytes) 000061 1781231105.126850 trunc from 0x0379f6 to 0x02aa9c (0xcf59 bytes) 000064 1781231105.383349 mapwrite 0x017b76 thru 0x0181cf (0x65a bytes) 000069 1781231105.469695 trunc from 0x02aa9d to 0x005dbd (0x24cdf bytes) 000071 1781231105.509611 write 0x03930e thru 0x03b8f9 (0x25ec bytes) 000073 1781231105.514751 read 0x033bef thru 0x03b8f9 (0x7d0b bytes) 000074 1781231105.527143 mapwrite 0x032fc4 thru 0x0362a0 (0x32dd bytes) 000077 1781231105.643226 MIRROR_RESYNC 2 mirrors 000079 1781231106.858969 read 0x02c7b2 thru 0x03a4c7 (0xdd16 bytes) 000080 1781231106.914966 MIRROR_RESYNC 2 mirrors 000081 1781231107.439497 write 0x03794c thru 0x03ffff (0x86b4 bytes) 000082 1781231107.590021 mapread 0x022111 thru 0x024b76 (0x2a66 bytes) 000083 1781231107.642592 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x393ae1a6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x662aefcc Mirror 3: 0x662aefcc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000084 1781231108.760502 mapread 0x002c2e thru 0x00d402 (0xa7d5 bytes) 000085 1781231108.803005 write 0x020b89 thru 0x02993e (0x8db6 bytes) 000088 1781231108.974307 read 0x02cdde thru 0x030df1 (0x4014 bytes) 000092 1781231108.994340 mapread 0x03ed92 thru 0x03f946 (0xbb5 bytes) 000093 1781231109.026522 trunc from 0x040000 to 0x00e15d (0x31ea2 bytes) 000094 1781231109.084514 read 0x003409 thru 0x008d87 (0x597f bytes) 000100 1781231109.119996 trunc from 0x00e15e to 0x014f3f (0x6de2 bytes) 000103 1781231109.158646 mapwrite 0x01dd91 thru 0x0220a7 (0x4317 bytes) 000104 1781231109.395574 write 0x02e591 thru 0x03dec9 (0xf939 bytes) 000106 1781231109.411515 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3deca): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3deca): Mirror 2: 0xa94c53bb (stale) Mirror 3: 0xa99ec73f (stale) CRC-32 checksum value for chunk [0, 0x3deca): Mirror 1: 0x8e56224b Mirror 4: 0x8e56224b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3deca: skipped 000108 1781231111.139997 write 0x03f583 thru 0x03ffff (0xa7d bytes) 000111 1781231111.342517 trunc from 0x040000 to 0x018941 (0x276be bytes) 000112 1781231111.381812 trunc from 0x018942 to 0x00471f (0x14222 bytes) 000113 1781231111.417160 write 0x003c4b thru 0x007dac (0x4162 bytes) 000114 1781231111.434473 mapwrite 0x012dcd thru 0x016f27 (0x415b bytes) 000116 1781231111.575395 MIRROR_RESYNC 4 mirrors 000117 1781231113.058971 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) 000118 1781231113.552455 write 0x00dd2d thru 0x0127f7 (0x4acb bytes) 000120 1781231113.805206 trunc from 0x016f28 to 0x013f36 (0x2ff1 bytes) 000121 1781231113.841666 trunc from 0x013f37 to 0x031e03 (0x1decd bytes) 000122 1781231113.881145 trunc from 0x031e04 to 0x009354 (0x28aaf bytes) 000126 1781231113.918207 mapwrite 0x022be9 thru 0x023a22 (0xe3a bytes) 000128 1781231114.054136 mapwrite 0x02b312 thru 0x03080f (0x54fe bytes) 000131 1781231114.304077 trunc from 0x030810 to 0x021cb3 (0xeb5c bytes) 000133 1781231114.342535 read 0x0124b0 thru 0x021cb3 (0xf804 bytes) 000136 1781231114.365857 trunc from 0x021cb4 to 0x01ba32 (0x6281 bytes) 000137 1781231114.405675 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1ba33): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ba33): Mirror 3: 0x6f682663 (stale) Mirror 4: 0x6f682663 (stale) CRC-32 checksum value for chunk [0, 0x1ba33): Mirror 1: 0x9d62b86b Mirror 5: 0x9d62b86b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ba33: skipped 000139 1781231115.775245 mapwrite 0x021fc3 thru 0x0294ee (0x752c bytes) 000141 1781231116.010372 mapwrite 0x019194 thru 0x027667 (0xe4d4 bytes) 000142 1781231116.081239 mapwrite 0x01532b thru 0x01c581 (0x7257 bytes) 000144 1781231116.136418 mapwrite 0x028339 thru 0x029ea1 (0x1b69 bytes) 000147 1781231116.210973 mapread 0x0106fc thru 0x01bea3 (0xb7a8 bytes) 000150 1781231116.238370 mapwrite 0x022229 thru 0x028ee0 (0x6cb8 bytes) 000152 1781231116.346293 trunc from 0x029ea2 to 0x017441 (0x12a60 bytes) 000153 1781231116.380071 read 0x011fd2 thru 0x013c1e (0x1c4d bytes) 000154 1781231116.381346 mapread 0x008542 thru 0x00df95 (0x5a54 bytes) 000155 1781231116.398614 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x17442): Mirror 3: 0x5332c9ad (stale) Mirror 5: 0xb5df0451 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17442): Mirror 3: 0x5332c9ad (stale) Mirror 5: 0xb5df0451 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17442: skipped 000156 1781231116.959191 write 0x033b13 thru 0x03b86b (0x7d59 bytes) 000157 1781231116.965200 trunc from 0x03b86c to 0x014c22 (0x26c49 bytes) 000158 1781231117.016069 mapwrite 0x00846d thru 0x00e73a (0x62ce bytes) 000159 1781231117.167099 read 0x010c17 thru 0x014c22 (0x400c bytes) 000160 1781231117.182029 trunc from 0x014c23 to 0x0279c9 (0x12da7 bytes) 000161 1781231117.218303 read 0x0031ef thru 0x00979c (0x65ae bytes) 000164 1781231117.224125 write 0x02902f thru 0x0323a3 (0x9375 bytes) 000166 1781231117.239856 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x323a4): Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x323a4): Mirror 3: 0x2efb3c3e (stale) Mirror 5: 0x8141a015 (stale) CRC-32 checksum value for chunk [0, 0x323a4): Mirror 1: 0x73517506 Mirror 6: 0x73517506 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x323a4: skipped 000167 1781231119.464013 read 0x02bce7 thru 0x0323a3 (0x66bd bytes) 000168 1781231119.554576 MIRROR_RESYNC 4 mirrors 000175 1781231120.834851 trunc from 0x0323a4 to 0x029782 (0x8c21 bytes) 000176 1781231121.079272 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x29783): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29783): Mirror 3: 0x88bee01f (stale) Mirror 5: 0x88bee01f (stale) Mirror 6: 0x88bee01f (stale) CRC-32 checksum value for chunk [0, 0x29783): Mirror 1: 0x88bee01f Mirror 7: 0x88bee01f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29783: skipped 000178 1781231122.701563 mapwrite 0x004e8d thru 0x009e08 (0x4f7c bytes) 000179 1781231123.068916 mapread 0x0026e1 thru 0x0040b9 (0x19d9 bytes) 000181 1781231123.079475 read 0x01dbb6 thru 0x024d81 (0x71cc bytes) 000184 1781231123.098974 mapread 0x0282f4 thru 0x029782 (0x148f bytes) 000186 1781231123.137697 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29783): Mirror 3: 0x88bee01f (stale) Mirror 6: 0x88bee01f (stale) Mirror 7: 0x88bee01f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29783): Mirror 3: 0x88bee01f (stale) Mirror 6: 0x88bee01f (stale) Mirror 7: 0x88bee01f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29783: skipped 000190 1781231123.788936 MIRROR_RESYNC 4 mirrors 000191 1781231125.267008 write 0x02cc6c thru 0x038bad (0xbf42 bytes) 000192 1781231125.408117 MIRROR_RESYNC 4 mirrors 000193 1781231126.881444 MIRROR_EXTEND to 5 mirrors 000194 1781231128.220334 mapwrite 0x0109a5 thru 0x0142fd (0x3959 bytes) 000196 1781231128.500964 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x38bae): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38bae): Mirror 3: 0xddcc3ac1 (stale) Mirror 6: 0xddcc3ac1 (stale) Mirror 7: 0xddcc3ac1 (stale) Mirror 8: 0xddcc3ac1 (stale) CRC-32 checksum value for chunk [0, 0x38bae): Mirror 1: 0xd0145201 Mirror 9: 0xd0145201 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38bae: skipped 000201 1781231129.777955 trunc from 0x038bae to 0x032af0 (0x60bd bytes) 000202 1781231129.984010 MIRROR_RESYNC 6 mirrors 000203 1781231131.580478 read 0x006222 thru 0x01152c (0xb30b bytes) 000204 1781231131.602586 write 0x0206df thru 0x0291e7 (0x8b09 bytes) 000205 1781231131.839346 write 0x036fbf thru 0x03e910 (0x7952 bytes) 000208 1781231131.845026 mapread 0x03b358 thru 0x03e910 (0x35b9 bytes) 000209 1781231131.847019 mapwrite 0x03c1d4 thru 0x03ffff (0x3e2c bytes) 000210 1781231131.995190 write 0x00e37c thru 0x01501d (0x6ca2 bytes) 000212 1781231132.028498 trunc from 0x040000 to 0x0077cc (0x38833 bytes) 000215 1781231132.058171 mapwrite 0x022438 thru 0x023f03 (0x1acc bytes) 000216 1781231132.287576 write 0x011369 thru 0x017ded (0x6a85 bytes) 000217 1781231132.328001 mapwrite 0x02da13 thru 0x03a39f (0xc98d bytes) 000218 1781231132.547376 MIRROR_RESYNC 6 mirrors 000219 1781231134.532828 trunc from 0x03a3a0 to 0x00388a (0x36b15 bytes) 000221 1781231134.683470 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x388b): Mirror 6: 0xce70b177 (stale) Mirror 7: 0xce70b177 (stale) Mirror 8: 0xce70b177 (stale) Mirror 9: 0xce70b177 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x388b): Mirror 6: 0xce70b177 (stale) Mirror 7: 0xce70b177 (stale) Mirror 8: 0xce70b177 (stale) Mirror 9: 0xce70b177 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x388b: skipped 000222 1781231135.262610 read 0x001586 thru 0x00388a (0x2305 bytes) 000225 1781231135.272281 trunc from 0x00388b to 0x01bdd4 (0x1854a bytes) 000226 1781231135.317976 mapwrite 0x018354 thru 0x01edd5 (0x6a82 bytes) 000227 1781231135.564224 read 0x00037b thru 0x00165f (0x12e5 bytes) 000229 1781231135.585710 write 0x012364 thru 0x01dd0c (0xb9a9 bytes) 000230 1781231135.587145 trunc from 0x01edd6 to 0x017682 (0x7753 bytes) 000233 1781231135.619054 write 0x03dc14 thru 0x03ffff (0x23ec bytes) 000234 1781231135.627211 MIRROR_RESYNC 5 mirrors 000235 1781231137.139995 write 0x025b20 thru 0x02d89e (0x7d7f bytes) 000236 1781231137.395068 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf6d72ac5 (stale) Mirror 7: 0xf6d72ac5 (stale) Mirror 9: 0xf6d72ac5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf6d72ac5 (stale) Mirror 7: 0xf6d72ac5 (stale) Mirror 9: 0xf6d72ac5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000238 1781231137.968221 mapwrite 0x012a51 thru 0x0224f0 (0xfaa0 bytes) 000239 1781231138.156279 read 0x02c426 thru 0x035033 (0x8c0e bytes) 000241 1781231138.179211 write 0x035d57 thru 0x037b8d (0x1e37 bytes) 000243 1781231138.219838 mapwrite 0x03adce thru 0x03ffff (0x5232 bytes) 000244 1781231138.323587 write 0x00392b thru 0x00e978 (0xb04e bytes) 000246 1781231138.356555 write 0x0006ff thru 0x0075ce (0x6ed0 bytes) 000248 1781231138.386461 mapread 0x02e5b6 thru 0x03ad1d (0xc768 bytes) 000249 1781231138.393678 read 0x011c4f thru 0x014fef (0x33a1 bytes) 000250 1781231138.395354 trunc from 0x040000 to 0x00f454 (0x30bab bytes) 000252 1781231138.441448 read 0x00081c thru 0x005493 (0x4c78 bytes) 000253 1781231138.444670 mapwrite 0x03b80e thru 0x03ffff (0x47f2 bytes) 000254 1781231138.571638 mapread 0x00a735 thru 0x0115e6 (0x6eb2 bytes) 000255 1781231138.576774 mapread 0x01c252 thru 0x021a78 (0x5827 bytes) 000256 1781231138.584211 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf6d72ac5 (stale) Mirror 7: 0xf6d72ac5 (stale) Mirror 9: 0xf6d72ac5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2aaec3e1 Mirror 10: 0x2aaec3e1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000257 1781231140.265455 mapread 0x0221cc thru 0x027638 (0x546d bytes) 000259 1781231140.298279 mapwrite 0x0100fd thru 0x01a9b0 (0xa8b4 bytes) 000262 1781231140.666306 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf6d72ac5 (stale) Mirror 7: 0xf6d72ac5 (stale) Mirror 9: 0xf6d72ac5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf6d72ac5 (stale) Mirror 7: 0xf6d72ac5 (stale) Mirror 9: 0xf6d72ac5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000263 1781231141.223375 mapwrite 0x0183e1 thru 0x019769 (0x1389 bytes) 000265 1781231141.399576 mapread 0x0388df thru 0x03ffff (0x7721 bytes) 000266 1781231141.444443 mapread 0x02c9d7 thru 0x02e712 (0x1d3c bytes) 000267 1781231141.484267 mapwrite 0x00c817 thru 0x012405 (0x5bef bytes) 000268 1781231141.686784 read 0x03afa8 thru 0x03ffff (0x5058 bytes) 000269 1781231141.689090 mapwrite 0x03be81 thru 0x03ffff (0x417f bytes) 000270 1781231141.753267 mapwrite 0x00a0d3 thru 0x00c2fd (0x222b bytes) 000271 1781231141.818058 mapwrite 0x00abff thru 0x00d12f (0x2531 bytes) 000274 1781231141.917125 write 0x02e59e thru 0x030d5a (0x27bd bytes) 000278 1781231141.927378 mapwrite 0x008f41 thru 0x00d83c (0x48fc bytes) 000281 1781231142.034511 mapread 0x016c29 thru 0x01bba3 (0x4f7b bytes) 000282 1781231142.049640 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf6d72ac5 (stale) Mirror 7: 0xf6d72ac5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xf6d72ac5 (stale) Mirror 7: 0xf6d72ac5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000283 1781231142.442519 trunc from 0x040000 to 0x0331f6 (0xce09 bytes) 000284 1781231142.464323 mapwrite 0x00f213 thru 0x01effa (0xfde8 bytes) 000285 1781231142.638412 MIRROR_RESYNC 3 mirrors 000286 1781231143.607895 mapread 0x01e4b3 thru 0x028774 (0xa2c2 bytes) 000287 1781231143.631613 write 0x032315 thru 0x0360a1 (0x3d8d bytes) 000288 1781231143.753266 trunc from 0x0360a2 to 0x0028ec (0x337b5 bytes) 000289 1781231143.794395 write 0x0212e4 thru 0x02aa9c (0x97b9 bytes) 000290 1781231143.795859 mapread 0x000a3a thru 0x00e2ca (0xd891 bytes) 000291 1781231143.821556 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2aa9d): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aa9d): Mirror 6: 0x4d85d18c (stale) Mirror 7: 0x4d85d18c (stale) CRC-32 checksum value for chunk [0, 0x2aa9d): Mirror 1: 0xbc845b89 Mirror 8: 0xbc845b89 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aa9d: skipped 000292 1781231145.159172 mapwrite 0x029bf0 thru 0x0351da (0xb5eb bytes) 000293 1781231145.419919 mapwrite 0x0348cb thru 0x03e9bb (0xa0f1 bytes) 000295 1781231145.627999 read 0x032e16 thru 0x03e9bb (0xbba6 bytes) 000297 1781231145.629290 mapread 0x0220bf thru 0x0318c6 (0xf808 bytes) 000301 1781231145.673094 mapwrite 0x01ac26 thru 0x01bfe6 (0x13c1 bytes) 000302 1781231145.874919 write 0x00579d thru 0x008366 (0x2bca bytes) 000303 1781231145.893790 mapwrite 0x03b2c7 thru 0x03ffff (0x4d39 bytes) 000304 1781231146.087832 write 0x007a06 thru 0x00807f (0x67a bytes) 000305 1781231146.094491 write 0x0339c4 thru 0x038ac9 (0x5106 bytes) 000306 1781231146.104875 read 0x007b32 thru 0x0136ea (0xbbb9 bytes) 000310 1781231146.146164 MIRROR_RESYNC 4 mirrors 000312 1781231147.692090 mapread 0x009a0c thru 0x01518b (0xb780 bytes) 000313 1781231147.719252 write 0x029292 thru 0x030d16 (0x7a85 bytes) 000317 1781231147.845042 mapread 0x02af45 thru 0x0324aa (0x7566 bytes) 000318 1781231147.856611 read 0x03ac2b thru 0x03ffff (0x53d5 bytes) 000320 1781231147.866920 mapwrite 0x015817 thru 0x0192aa (0x3a94 bytes) 000321 1781231147.970660 trunc from 0x040000 to 0x0132cb (0x2cd34 bytes) 000323 1781231148.012962 mapread 0x010db5 thru 0x0132cb (0x2517 bytes) 000324 1781231148.014093 write 0x0037ee thru 0x009762 (0x5f75 bytes) 000325 1781231148.034662 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x132cc): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x132cc): Mirror 6: 0xcdad63d1 (stale) Mirror 7: 0xcdad63d1 (stale) Mirror 8: 0xcdad63d1 (stale) CRC-32 checksum value for chunk [0, 0x132cc): Mirror 1: 0x59db0ad7 Mirror 9: 0x59db0ad7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x132cc: skipped 000327 1781231149.240579 read 0x001de2 thru 0x007677 (0x5896 bytes) 000329 1781231149.257659 trunc from 0x0132cc to 0x0093dc (0x9eef bytes) 000332 1781231149.470822 write 0x03c02f thru 0x03ffff (0x3fd1 bytes) 000333 1781231149.473341 read 0x00926c thru 0x011938 (0x86cd bytes) 000334 1781231149.483897 write 0x03b12a thru 0x03ffff (0x4ed6 bytes) 000335 1781231149.485068 write 0x020444 thru 0x02e592 (0xe14f bytes) 000336 1781231149.490179 trunc from 0x040000 to 0x024974 (0x1b68b bytes) 000337 1781231149.520181 mapread 0x00199f thru 0x011812 (0xfe74 bytes) 000339 1781231149.527084 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x24975): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24975): Mirror 6: 0x30665ebc (stale) Mirror 7: 0x30665ebc (stale) Mirror 8: 0x30665ebc (stale) Mirror 9: 0xb5092ae1 (stale) CRC-32 checksum value for chunk [0, 0x24975): Mirror 1: 0xae9e80db Mirror 10: 0xae9e80db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24975: skipped 000341 1781231150.415215 MIRROR_RESYNC 6 mirrors 000343 1781231151.696835 write 0x0240a0 thru 0x0248c0 (0x821 bytes) 000344 1781231151.883564 trunc from 0x024975 to 0x022623 (0x2351 bytes) 000345 1781231151.960847 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x22624): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22624): Mirror 6: 0xd7966372 (stale) Mirror 7: 0xd7966372 (stale) Mirror 8: 0xd7966372 (stale) Mirror 9: 0xd7966372 (stale) Mirror 10: 0xd7966372 (stale) CRC-32 checksum value for chunk [0, 0x22624): Mirror 1: 0xd7966372 Mirror 11: 0xd7966372 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22624: skipped 000349 1781231153.233569 mapwrite 0x014a5d thru 0x01c3cf (0x7973 bytes) 000353 1781231153.526393 MIRROR_RESYNC 7 mirrors 000355 1781231154.929070 mapread 0x01c708 thru 0x0209f2 (0x42eb bytes) 000358 1781231154.968446 read 0x01b159 thru 0x01e859 (0x3701 bytes) 000359 1781231154.969122 mapwrite 0x02ba54 thru 0x035d5e (0xa30b bytes) 000361 1781231155.194390 mapread 0x013607 thru 0x02142c (0xde26 bytes) 000362 1781231155.226104 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x35d5f): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35d5f): Mirror 6: 0x58079d1d (stale) Mirror 7: 0x58079d1d (stale) Mirror 8: 0x58079d1d (stale) Mirror 9: 0x58079d1d (stale) Mirror 10: 0x58079d1d (stale) Mirror 11: 0x58079d1d (stale) CRC-32 checksum value for chunk [0, 0x35d5f): Mirror 1: 0xc0a7d1cf Mirror 12: 0xc0a7d1cf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35d5f: skipped 000363 1781231157.010855 trunc from 0x035d5f to 0x00bdfb (0x29f63 bytes) 000364 1781231157.317864 read 0x000620 thru 0x004e32 (0x4813 bytes) 000365 1781231157.331675 read 0x004127 thru 0x00bdfb (0x7cd5 bytes) 000367 1781231157.345367 read 0x007c29 thru 0x00bdfb (0x41d3 bytes) 000370 1781231157.362575 read 0x001cb6 thru 0x00ada9 (0x90f4 bytes) 000372 1781231157.387237 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xbdfc): Mirror 6: 0xc39f33fa (stale) Mirror 7: 0xc39f33fa (stale) Mirror 8: 0xc39f33fa (stale) Mirror 9: 0xc39f33fa (stale) Mirror 10: 0xc39f33fa (stale) Mirror 12: 0xc39f33fa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbdfc): Mirror 6: 0xc39f33fa (stale) Mirror 7: 0xc39f33fa (stale) Mirror 8: 0xc39f33fa (stale) Mirror 9: 0xc39f33fa (stale) Mirror 10: 0xc39f33fa (stale) Mirror 12: 0xc39f33fa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbdfc: skipped 000374 1781231158.183206 read 0x008950 thru 0x00bdfb (0x34ac bytes) 000375 1781231158.202136 mapwrite 0x00e74e thru 0x0186b4 (0x9f67 bytes) 000377 1781231158.324155 mapwrite 0x013e3e thru 0x020109 (0xc2cc bytes) 000378 1781231158.468094 write 0x02ae69 thru 0x03272b (0x78c3 bytes) 000379 1781231158.471273 mapread 0x001449 thru 0x00b580 (0xa138 bytes) 000380 1781231158.495319 mapwrite 0x02a5c1 thru 0x02c4c0 (0x1f00 bytes) 000381 1781231158.696241 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3272c): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3272c): Mirror 6: 0x8e6b2247 (stale) Mirror 7: 0x8e6b2247 (stale) Mirror 8: 0x8e6b2247 (stale) Mirror 9: 0x8e6b2247 (stale) Mirror 10: 0x8e6b2247 (stale) Mirror 12: 0x54debace (stale) CRC-32 checksum value for chunk [0, 0x3272c): Mirror 1: 0xfd97d73e Mirror 13: 0xfd97d73e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3272c: skipped 000382 1781231160.616153 trunc from 0x03272c to 0x036f0d (0x47e2 bytes) 000384 1781231160.889301 trunc from 0x036f0e to 0x0340dd (0x2e30 bytes) 000385 1781231160.943632 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x340de): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x340de): Mirror 6: 0x5e0baec4 (stale) Mirror 7: 0x5e0baec4 (stale) Mirror 8: 0x5e0baec4 (stale) Mirror 9: 0x5e0baec4 (stale) Mirror 10: 0x5e0baec4 (stale) Mirror 12: 0x5f2222fe (stale) Mirror 13: 0xf8bb8318 (stale) CRC-32 checksum value for chunk [0, 0x340de): Mirror 1: 0xf8bb8318 Mirror 14: 0xf8bb8318 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x340de: skipped 000388 1781231162.179166 write 0x00db65 thru 0x013e1a (0x62b6 bytes) 000391 1781231162.372682 read 0x033931 thru 0x0340dd (0x7ad bytes) 000393 1781231162.387226 mapwrite 0x029564 thru 0x038791 (0xf22e bytes) 000394 1781231162.682798 write 0x021410 thru 0x0262f0 (0x4ee1 bytes) 000395 1781231162.702612 write 0x03136f thru 0x03a51f (0x91b1 bytes) 000397 1781231162.709808 write 0x00c12d thru 0x01753f (0xb413 bytes) 000399 1781231162.749112 write 0x0323a2 thru 0x03ffff (0xdc5e bytes) 000401 1781231162.761794 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x112b18bb (stale) Mirror 8: 0x112b18bb (stale) Mirror 9: 0x112b18bb (stale) Mirror 10: 0x112b18bb (stale) Mirror 12: 0xae6a9237 (stale) Mirror 13: 0x9c9d82d6 (stale) Mirror 14: 0x9c9d82d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x112b18bb (stale) Mirror 8: 0x112b18bb (stale) Mirror 9: 0x112b18bb (stale) Mirror 10: 0x112b18bb (stale) Mirror 12: 0xae6a9237 (stale) Mirror 13: 0x9c9d82d6 (stale) Mirror 14: 0x9c9d82d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000404 1781231163.648640 read 0x02701e thru 0x031af0 (0xaad3 bytes) 000405 1781231163.689223 read 0x009bd3 thru 0x00ef7f (0x53ad bytes) 000407 1781231163.715925 mapwrite 0x01dc5e thru 0x02c401 (0xe7a4 bytes) 000409 1781231163.821406 mapread 0x015462 thru 0x0250c8 (0xfc67 bytes) 000410 1781231163.823599 write 0x01d76c thru 0x021dd5 (0x466a bytes) 000411 1781231163.829840 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x112b18bb (stale) Mirror 9: 0x112b18bb (stale) Mirror 10: 0x112b18bb (stale) Mirror 12: 0xae6a9237 (stale) Mirror 13: 0x9c9d82d6 (stale) Mirror 14: 0x9c9d82d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x112b18bb (stale) Mirror 9: 0x112b18bb (stale) Mirror 10: 0x112b18bb (stale) Mirror 12: 0xae6a9237 (stale) Mirror 13: 0x9c9d82d6 (stale) Mirror 14: 0x9c9d82d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000415 1781231164.586062 mapread 0x0304d3 thru 0x039db8 (0x98e6 bytes) 000416 1781231164.658605 read 0x015a68 thru 0x021808 (0xbda1 bytes) 000417 1781231164.686386 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x112b18bb (stale) Mirror 9: 0x112b18bb (stale) Mirror 10: 0x112b18bb (stale) Mirror 12: 0xae6a9237 (stale) Mirror 13: 0x9c9d82d6 (stale) Mirror 14: 0x9c9d82d6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xad75c4ef Mirror 15: 0xad75c4ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000418 1781231165.877011 trunc from 0x040000 to 0x0295e1 (0x16a1e bytes) 000419 1781231166.040575 mapwrite 0x036736 thru 0x03ffff (0x98ca bytes) 000420 1781231166.300069 mapread 0x01c1ce thru 0x01dcd9 (0x1b0c bytes) 000423 1781231166.332782 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x112b18bb (stale) Mirror 9: 0x112b18bb (stale) Mirror 10: 0x112b18bb (stale) Mirror 12: 0xae6a9237 (stale) Mirror 13: 0x9c9d82d6 (stale) Mirror 15: 0xad75c4ef (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x112b18bb (stale) Mirror 9: 0x112b18bb (stale) Mirror 10: 0x112b18bb (stale) Mirror 12: 0xae6a9237 (stale) Mirror 13: 0x9c9d82d6 (stale) Mirror 15: 0xad75c4ef (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000425 1781231166.955423 mapwrite 0x030d8e thru 0x03305f (0x22d2 bytes) 000428 1781231167.108425 write 0x02db5b thru 0x03b1ca (0xd670 bytes) 000429 1781231167.133764 mapwrite 0x021a97 thru 0x025829 (0x3d93 bytes) 000432 1781231167.284383 read 0x0115d3 thru 0x01e790 (0xd1be bytes) 000433 1781231167.307530 mapwrite 0x02453a thru 0x02c19f (0x7c66 bytes) 000435 1781231167.474899 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x112b18bb (stale) Mirror 9: 0x112b18bb (stale) Mirror 10: 0x112b18bb (stale) Mirror 12: 0xae6a9237 (stale) Mirror 13: 0x9c9d82d6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x112b18bb (stale) Mirror 9: 0x112b18bb (stale) Mirror 10: 0x112b18bb (stale) Mirror 12: 0xae6a9237 (stale) Mirror 13: 0x9c9d82d6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000436 1781231168.221300 write 0x002193 thru 0x0081cd (0x603b bytes) 000439 1781231168.252031 read 0x033cd7 thru 0x034b0c (0xe36 bytes) 000440 1781231168.270779 mapread 0x008bff thru 0x017007 (0xe409 bytes) 000441 1781231168.307594 mapread 0x031894 thru 0x03a8e2 (0x904f bytes) 000442 1781231168.341490 write 0x00bf09 thru 0x013f0b (0x8003 bytes) 000443 1781231168.342820 read 0x02f90e thru 0x03262f (0x2d22 bytes) 000447 1781231168.348680 write 0x0062eb thru 0x0104ae (0xa1c4 bytes) 000448 1781231168.350427 trunc from 0x040000 to 0x002068 (0x3df97 bytes) 000450 1781231168.377556 read 0x00006a thru 0x002068 (0x1fff bytes) 000451 1781231168.387669 mapread 0x001b40 thru 0x002068 (0x529 bytes) 000452 1781231168.389928 write 0x037183 thru 0x03c02f (0x4ead bytes) 000454 1781231168.400446 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3c030): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c030): Mirror 8: 0x38ac1381 (stale) Mirror 9: 0x38ac1381 (stale) Mirror 10: 0x38ac1381 (stale) Mirror 12: 0xaf2e7d7d (stale) Mirror 13: 0x976aa869 (stale) CRC-32 checksum value for chunk [0, 0x3c030): Mirror 1: 0xfea7f225 Mirror 14: 0xfea7f225 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c030: skipped 000456 1781231170.442463 mapread 0x002517 thru 0x00d45c (0xaf46 bytes) 000457 1781231170.511470 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c030): Mirror 9: 0x38ac1381 (stale) Mirror 10: 0x38ac1381 (stale) Mirror 12: 0xaf2e7d7d (stale) Mirror 13: 0x976aa869 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c030): Mirror 9: 0x38ac1381 (stale) Mirror 10: 0x38ac1381 (stale) Mirror 12: 0xaf2e7d7d (stale) Mirror 13: 0x976aa869 (stale) CRC-32 checksum value for chunk [0, 0x3c030): Mirror 1: 0xfea7f225 Mirror 14: 0xfea7f225 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c030: skipped 000458 1781231171.674674 mapwrite 0x025371 thru 0x031c72 (0xc902 bytes) 000460 1781231171.982656 trunc from 0x03c030 to 0x010961 (0x2b6ce bytes) 000461 1781231172.025788 write 0x036f3e thru 0x0370dd (0x1a0 bytes) 000464 1781231172.041206 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x370de): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x370de): Mirror 9: 0x166531ba (stale) Mirror 10: 0x166531ba (stale) Mirror 12: 0xed500b5a (stale) Mirror 13: 0x77a32192 (stale) Mirror 14: 0xe0e10c16 (stale) CRC-32 checksum value for chunk [0, 0x370de): Mirror 1: 0x2de7e3b5 Mirror 15: 0x2de7e3b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x370de: skipped 000469 1781231173.814512 mapwrite 0x024397 thru 0x02f366 (0xafd0 bytes) 000471 1781231174.179397 MIRROR_RESYNC 7 mirrors 000472 1781231176.024013 MIRROR_EXTEND to 8 mirrors 000477 1781231177.131547 mapwrite 0x01cc08 thru 0x023b69 (0x6f62 bytes) 000478 1781231177.619652 mapread 0x019a69 thru 0x0283cc (0xe964 bytes) 000479 1781231177.640465 mapwrite 0x02ebd7 thru 0x03e5a3 (0xf9cd bytes) 000481 1781231177.863998 trunc from 0x03e5a4 to 0x02695c (0x17c47 bytes) 000483 1781231177.899575 mapread 0x01016a thru 0x0172e8 (0x717f bytes) 000484 1781231177.920763 mapread 0x01d59c thru 0x01f5e3 (0x2048 bytes) 000489 1781231177.936311 trunc from 0x02695d to 0x02d6a2 (0x6d46 bytes) 000490 1781231177.973214 mapwrite 0x01c4da thru 0x02580d (0x9334 bytes) 000492 1781231178.070341 mapwrite 0x02ef61 thru 0x035a4d (0x6aed bytes) 000493 1781231178.264208 read 0x009956 thru 0x00d6fb (0x3da6 bytes) 000494 1781231178.268113 mapread 0x01abdd thru 0x028be5 (0xe009 bytes) 000497 1781231178.281270 read 0x026d9e thru 0x029a20 (0x2c83 bytes) 000498 1781231178.285849 mapwrite 0x038f87 thru 0x03ffff (0x7079 bytes) 000499 1781231178.418188 MIRROR_RESYNC 8 mirrors 000500 1781231180.975474 MIRROR_EXTEND to 9 mirrors 000505 1781231182.382601 read 0x02318c thru 0x030b29 (0xd99e bytes) 000506 1781231182.413400 write 0x02a0da thru 0x039860 (0xf787 bytes) 000510 1781231182.639434 write 0x03708f thru 0x03ffff (0x8f71 bytes) 000511 1781231182.646125 trunc from 0x040000 to 0x033f1f (0xc0e0 bytes) 000512 1781231182.681442 mapwrite 0x022073 thru 0x02c8fc (0xa88a bytes) 000514 1781231182.824091 mapwrite 0x012b53 thru 0x015198 (0x2646 bytes) 000516 1781231182.957798 read 0x03015b thru 0x033f1f (0x3dc5 bytes) 000517 1781231182.975076 MIRROR_RESYNC 9 mirrors 000519 1781231184.879802 mapwrite 0x0126f4 thru 0x0192e9 (0x6bf6 bytes) 000520 1781231185.100706 mapread 0x00203e thru 0x0086c9 (0x668c bytes) 000521 1781231185.130017 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33f20): Mirror 9: 0x40120ce1 (stale) Mirror 10: 0x40120ce1 (stale) Mirror 12: 0x40120ce1 (stale) Mirror 13: 0x40120ce1 (stale) Mirror 15: 0x40120ce1 (stale) Mirror 16: 0x40120ce1 (stale) Mirror 17: 0x40120ce1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33f20): Mirror 9: 0x40120ce1 (stale) Mirror 10: 0x40120ce1 (stale) Mirror 12: 0x40120ce1 (stale) Mirror 13: 0x40120ce1 (stale) Mirror 15: 0x40120ce1 (stale) Mirror 16: 0x40120ce1 (stale) Mirror 17: 0x40120ce1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33f20: skipped 000522 1781231185.690788 write 0x0342cf thru 0x03f467 (0xb199 bytes) 000526 1781231185.703415 write 0x01928a thru 0x0197c8 (0x53f bytes) 000527 1781231185.711325 mapread 0x0145da thru 0x022095 (0xdabc bytes) 000528 1781231185.769479 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3f468): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f468): Mirror 9: 0x3dd15647 (stale) Mirror 10: 0x3dd15647 (stale) Mirror 12: 0x3dd15647 (stale) Mirror 13: 0x3dd15647 (stale) Mirror 15: 0x3dd15647 (stale) Mirror 16: 0x3dd15647 (stale) Mirror 17: 0x3dd15647 (stale) CRC-32 checksum value for chunk [0, 0x3f468): Mirror 1: 0xf2696c7c Mirror 18: 0xf2696c7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f468: skipped 000529 1781231187.398956 mapwrite 0x02fce4 thru 0x03ef71 (0xf28e bytes) 000530 1781231187.717096 mapread 0x01ebd7 thru 0x02cc0f (0xe039 bytes) 000534 1781231187.773343 trunc from 0x03f468 to 0x036850 (0x8c17 bytes) 000536 1781231187.840305 read 0x02b85d thru 0x036850 (0xaff4 bytes) 000537 1781231187.844874 mapwrite 0x015a9f thru 0x0178ae (0x1e10 bytes) 000538 1781231187.959056 mapread 0x0182cd thru 0x01dd66 (0x5a9a bytes) 000541 1781231187.976336 write 0x0089bf thru 0x01215c (0x979e bytes) 000543 1781231188.006443 trunc from 0x036851 to 0x010104 (0x2674c bytes) 000544 1781231188.060235 write 0x00f51e thru 0x018345 (0x8e28 bytes) 000545 1781231188.070981 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x18346): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18346): Mirror 9: 0x3ae3ab31 (stale) Mirror 10: 0x3ae3ab31 (stale) Mirror 12: 0x3ae3ab31 (stale) Mirror 13: 0x3ae3ab31 (stale) Mirror 15: 0x3ae3ab31 (stale) Mirror 16: 0x3ae3ab31 (stale) Mirror 17: 0x3ae3ab31 (stale) Mirror 18: 0xb066cc4b (stale) CRC-32 checksum value for chunk [0, 0x18346): Mirror 1: 0xd1ee03dd Mirror 19: 0xd1ee03dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18346: skipped 000548 1781231189.474895 write 0x00a006 thru 0x00fea2 (0x5e9d bytes) 000550 1781231189.608836 mapwrite 0x03ad16 thru 0x03ffff (0x52ea bytes) 000551 1781231189.789111 write 0x03e7bf thru 0x03ffff (0x1841 bytes) 000552 1781231189.790980 trunc from 0x040000 to 0x028f38 (0x170c7 bytes) 000553 1781231189.826196 mapwrite 0x00408c thru 0x004c5e (0xbd3 bytes) 000554 1781231189.968657 write 0x002652 thru 0x0089f9 (0x63a8 bytes) 000555 1781231189.979783 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x28f39): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28f39): Mirror 9: 0xd355543b (stale) Mirror 10: 0xd355543b (stale) Mirror 12: 0xd355543b (stale) Mirror 13: 0xd355543b (stale) Mirror 15: 0xd355543b (stale) Mirror 16: 0xd355543b (stale) Mirror 17: 0xd355543b (stale) Mirror 18: 0xa2baa8bc (stale) Mirror 19: 0x58edd92b (stale) CRC-32 checksum value for chunk [0, 0x28f39): Mirror 1: 0x328f1562 Mirror 20: 0x328f1562 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28f39: skipped 000557 1781231191.481855 write 0x012d76 thru 0x01cc99 (0x9f24 bytes) 000558 1781231191.613055 mapread 0x009b65 thru 0x011a5c (0x7ef8 bytes) 000560 1781231191.629154 trunc from 0x028f39 to 0x02e5f3 (0x56bb bytes) 000561 1781231191.673530 mapwrite 0x023637 thru 0x026d33 (0x36fd bytes) 000562 1781231191.788587 write 0x004a01 thru 0x00f861 (0xae61 bytes) 000563 1781231191.793047 write 0x022aa0 thru 0x030b40 (0xe0a1 bytes) 000564 1781231191.807737 trunc from 0x030b41 to 0x02aa61 (0x60df bytes) 000565 1781231191.845883 mapwrite 0x00e4d3 thru 0x01d389 (0xeeb7 bytes) 000568 1781231191.920253 write 0x0137d3 thru 0x02355b (0xfd89 bytes) 000569 1781231191.921797 read 0x010127 thru 0x01f91f (0xf7f9 bytes) 000571 1781231191.930697 mapread 0x024e12 thru 0x02aa61 (0x5c50 bytes) 000572 1781231191.934499 mapwrite 0x01e7b7 thru 0x022556 (0x3da0 bytes) 000573 1781231192.032275 mapwrite 0x00526f thru 0x0091d0 (0x3f62 bytes) 000576 1781231192.128786 mapwrite 0x03d48b thru 0x03ffff (0x2b75 bytes) 000578 1781231192.272413 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xa7e1ef49 (stale) Mirror 10: 0xa7e1ef49 (stale) Mirror 12: 0xa7e1ef49 (stale) Mirror 13: 0xa7e1ef49 (stale) Mirror 15: 0xa7e1ef49 (stale) Mirror 16: 0xa7e1ef49 (stale) Mirror 17: 0xa7e1ef49 (stale) Mirror 18: 0x831f4206 (stale) Mirror 19: 0xb26e5196 (stale) Mirror 20: 0x299d80db (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x64f709fd Mirror 21: 0x64f709fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000582 1781231193.700107 trunc from 0x040000 to 0x0095c9 (0x36a36 bytes) 000583 1781231193.902566 mapwrite 0x03ea05 thru 0x03ffff (0x15fb bytes) 000584 1781231194.010937 MIRROR_RESYNC 12 mirrors 000585 1781231196.546349 mapread 0x01e4d7 thru 0x01e7c9 (0x2f3 bytes) 000586 1781231196.552365 trunc from 0x040000 to 0x009f15 (0x360ea bytes) 000590 1781231196.673305 trunc from 0x009f16 to 0x013343 (0x942e bytes) 000591 1781231196.720834 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x13344): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13344): Mirror 9: 0xeac72d2f (stale) Mirror 10: 0xeac72d2f (stale) Mirror 12: 0xeac72d2f (stale) Mirror 13: 0xeac72d2f (stale) Mirror 15: 0xeac72d2f (stale) Mirror 16: 0xeac72d2f (stale) Mirror 17: 0xeac72d2f (stale) Mirror 18: 0xeac72d2f (stale) Mirror 19: 0xeac72d2f (stale) Mirror 20: 0xeac72d2f (stale) Mirror 21: 0xeac72d2f (stale) CRC-32 checksum value for chunk [0, 0x13344): Mirror 1: 0xeac72d2f Mirror 22: 0xeac72d2f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13344: skipped 000592 1781231198.051275 trunc from 0x013344 to 0x01ae4e (0x7b0b bytes) 000593 1781231198.206484 MIRROR_RESYNC 13 mirrors 000594 1781231200.763374 trunc from 0x01ae4f to 0x026392 (0xb544 bytes) 000595 1781231200.885327 MIRROR_RESYNC 13 mirrors 000596 1781231203.391545 mapwrite 0x02a1af thru 0x033ddb (0x9c2d bytes) 000598 1781231203.710018 mapwrite 0x0297dd thru 0x036884 (0xd0a8 bytes) 000599 1781231203.883795 mapwrite 0x02639f thru 0x02916d (0x2dcf bytes) 000600 1781231203.988824 read 0x001c0c thru 0x002256 (0x64b bytes) 000601 1781231203.998439 mapwrite 0x019f8e thru 0x01ed24 (0x4d97 bytes) 000604 1781231204.110950 trunc from 0x036885 to 0x004301 (0x32583 bytes) 000606 1781231204.149467 mapread 0x002fb6 thru 0x004301 (0x134c bytes) 000607 1781231204.156842 mapread 0x00267f thru 0x004301 (0x1c83 bytes) 000608 1781231204.165781 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x4302): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4302): Mirror 9: 0x5166ef4 (stale) Mirror 10: 0x5166ef4 (stale) Mirror 12: 0x5166ef4 (stale) Mirror 13: 0x5166ef4 (stale) Mirror 15: 0x5166ef4 (stale) Mirror 16: 0x5166ef4 (stale) Mirror 17: 0x5166ef4 (stale) Mirror 18: 0x5166ef4 (stale) Mirror 19: 0x5166ef4 (stale) Mirror 20: 0x5166ef4 (stale) Mirror 21: 0x5166ef4 (stale) Mirror 22: 0x5166ef4 (stale) CRC-32 checksum value for chunk [0, 0x4302): Mirror 1: 0x5166ef4 Mirror 23: 0x5166ef4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4302: skipped 000609 1781231205.115335 trunc from 0x004302 to 0x034c29 (0x30928 bytes) 000611 1781231205.270429 mapwrite 0x00c33f thru 0x0116df (0x53a1 bytes) 000612 1781231205.359212 mapread 0x01388b thru 0x013b94 (0x30a bytes) 000614 1781231205.364923 mapwrite 0x010d6c thru 0x01ec15 (0xdeaa bytes) 000616 1781231205.524137 trunc from 0x034c2a to 0x000ef5 (0x33d34 bytes) 000617 1781231205.572616 mapread 0x000593 thru 0x000ef5 (0x963 bytes) 000620 1781231205.613793 read 0x000c79 thru 0x000ef5 (0x27d bytes) 000623 1781231205.625403 read 0x000ed0 thru 0x000ef5 (0x26 bytes) 000624 1781231205.627958 read 0x000b9c thru 0x000ef5 (0x35a bytes) 000626 1781231205.632577 trunc from 0x000ef6 to 0x0368bb (0x359c6 bytes) 000627 1781231205.671805 write 0x00b6b4 thru 0x00f5a2 (0x3eef bytes) 000630 1781231205.696381 mapwrite 0x00a55a thru 0x016c4a (0xc6f1 bytes) 000633 1781231205.784363 mapwrite 0x036266 thru 0x03ffff (0x9d9a bytes) 000634 1781231205.876322 mapread 0x003197 thru 0x010051 (0xcebb bytes) 000637 1781231205.885308 trunc from 0x040000 to 0x00335d (0x3cca2 bytes) 000638 1781231205.909356 trunc from 0x00335e to 0x003393 (0x36 bytes) 000642 1781231205.937095 write 0x03ba09 thru 0x03ffff (0x45f7 bytes) 000643 1781231205.939417 write 0x009071 thru 0x009d3a (0xcca bytes) 000644 1781231205.945136 trunc from 0x040000 to 0x0357a2 (0xa85d bytes) 000645 1781231205.962289 mapwrite 0x0073e4 thru 0x00d1c5 (0x5de2 bytes) 000647 1781231206.044215 mapread 0x0121ea thru 0x012b0e (0x925 bytes) 000652 1781231206.055069 mapwrite 0x008ebd thru 0x014b10 (0xbc54 bytes) 000653 1781231206.137314 write 0x01a39d thru 0x021a18 (0x767c bytes) 000655 1781231206.158500 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x357a3): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x357a3): Mirror 9: 0x29cbfe06 (stale) Mirror 10: 0x29cbfe06 (stale) Mirror 12: 0x29cbfe06 (stale) Mirror 13: 0x29cbfe06 (stale) Mirror 15: 0x29cbfe06 (stale) Mirror 16: 0x29cbfe06 (stale) Mirror 17: 0x29cbfe06 (stale) Mirror 18: 0x29cbfe06 (stale) Mirror 19: 0x29cbfe06 (stale) Mirror 20: 0x29cbfe06 (stale) Mirror 21: 0x29cbfe06 (stale) Mirror 22: 0x29cbfe06 (stale) Mirror 23: 0x58d68417 (stale) CRC-32 checksum value for chunk [0, 0x357a3): Mirror 1: 0xe67d375c Mirror 24: 0xe67d375c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x357a3: skipped 000659 1781231207.721119 read 0x02ed28 thru 0x0357a2 (0x6a7b bytes) 000665 1781231207.747059 mapread 0x01d0a4 thru 0x029119 (0xc076 bytes) 000666 1781231207.788640 mapwrite 0x01e919 thru 0x024a89 (0x6171 bytes) 000667 1781231208.076622 MIRROR_SPLIT mirror 7 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x357a3): Mirror 9: 0x29cbfe06 (stale) Mirror 10: 0x29cbfe06 (stale) Mirror 12: 0x29cbfe06 (stale) Mirror 13: 0x29cbfe06 (stale) Mirror 15: 0x29cbfe06 (stale) Mirror 16: 0x29cbfe06 (stale) Mirror 18: 0x29cbfe06 (stale) Mirror 19: 0x29cbfe06 (stale) Mirror 20: 0x29cbfe06 (stale) Mirror 21: 0x29cbfe06 (stale) Mirror 22: 0x29cbfe06 (stale) Mirror 23: 0x58d68417 (stale) Mirror 24: 0xe67d375c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x357a3): Mirror 9: 0x29cbfe06 (stale) Mirror 10: 0x29cbfe06 (stale) Mirror 12: 0x29cbfe06 (stale) Mirror 13: 0x29cbfe06 (stale) Mirror 15: 0x29cbfe06 (stale) Mirror 16: 0x29cbfe06 (stale) Mirror 18: 0x29cbfe06 (stale) Mirror 19: 0x29cbfe06 (stale) Mirror 20: 0x29cbfe06 (stale) Mirror 21: 0x29cbfe06 (stale) Mirror 22: 0x29cbfe06 (stale) Mirror 23: 0x58d68417 (stale) Mirror 24: 0xe67d375c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x357a3: skipped 000668 1781231208.982952 trunc from 0x0357a3 to 0x005e48 (0x2f95a bytes) 000670 1781231209.069358 read 0x0020ce thru 0x005e48 (0x3d7b bytes) 000671 1781231209.078785 mapwrite 0x01df22 thru 0x027565 (0x9644 bytes) 000672 1781231209.206217 MIRROR_RESYNC 14 mirrors 000673 1781231211.463518 read 0x023728 thru 0x027565 (0x3e3e bytes) 000675 1781231211.474186 read 0x015506 thru 0x02412a (0xec25 bytes) 000676 1781231211.488300 trunc from 0x027566 to 0x009eec (0x1d679 bytes) 000677 1781231211.612701 read 0x00287b thru 0x00991b (0x70a1 bytes) 000682 1781231211.627601 read 0x000135 thru 0x007895 (0x7761 bytes) 000684 1781231211.630125 write 0x00fe11 thru 0x00ffbc (0x1ac bytes) 000690 1781231211.635472 mapwrite 0x02733f thru 0x035435 (0xe0f7 bytes) 000691 1781231211.757607 mapread 0x024369 thru 0x02f0ab (0xad43 bytes) 000692 1781231211.760469 write 0x01a71c thru 0x01eaaf (0x4394 bytes) 000695 1781231211.764438 mapwrite 0x033a92 thru 0x03f034 (0xb5a3 bytes) 000697 1781231211.919264 MIRROR_RESYNC 14 mirrors 000698 1781231214.342819 read 0x026cf3 thru 0x02f82d (0x8b3b bytes) 000699 1781231214.351201 trunc from 0x03f035 to 0x01c634 (0x22a00 bytes) 000700 1781231214.468325 mapwrite 0x030228 thru 0x03712f (0x6f08 bytes) 000701 1781231214.575870 write 0x009bd6 thru 0x017024 (0xd44f bytes) 000704 1781231214.591409 mapread 0x00ae0d thru 0x0169e8 (0xbbdc bytes) 000706 1781231214.593230 trunc from 0x037130 to 0x016696 (0x20a99 bytes) 000708 1781231214.619163 mapread 0x0027fc thru 0x00288e (0x93 bytes) 000709 1781231214.620665 write 0x029377 thru 0x034c49 (0xb8d3 bytes) 000710 1781231214.622402 write 0x00b758 thru 0x014556 (0x8dff bytes) 000712 1781231214.625858 trunc from 0x034c4a to 0x0049a7 (0x302a2 bytes) 000716 1781231214.655150 MIRROR_SPLIT mirror 3 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x49a8): Mirror 9: 0x395f5ee2 (stale) Mirror 10: 0x395f5ee2 (stale) Mirror 13: 0x395f5ee2 (stale) Mirror 15: 0x395f5ee2 (stale) Mirror 16: 0x395f5ee2 (stale) Mirror 18: 0x395f5ee2 (stale) Mirror 19: 0x395f5ee2 (stale) Mirror 20: 0x395f5ee2 (stale) Mirror 21: 0x395f5ee2 (stale) Mirror 22: 0x395f5ee2 (stale) Mirror 23: 0x395f5ee2 (stale) Mirror 24: 0x395f5ee2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x49a8): Mirror 9: 0x395f5ee2 (stale) Mirror 10: 0x395f5ee2 (stale) Mirror 13: 0x395f5ee2 (stale) Mirror 15: 0x395f5ee2 (stale) Mirror 16: 0x395f5ee2 (stale) Mirror 18: 0x395f5ee2 (stale) Mirror 19: 0x395f5ee2 (stale) Mirror 20: 0x395f5ee2 (stale) Mirror 21: 0x395f5ee2 (stale) Mirror 22: 0x395f5ee2 (stale) Mirror 23: 0x395f5ee2 (stale) Mirror 24: 0x395f5ee2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x49a8: skipped 000717 1781231215.306561 MIRROR_RESYNC 13 mirrors 000718 1781231217.143156 trunc from 0x0049a8 to 0x00aac9 (0x6122 bytes) 000719 1781231217.273370 write 0x0298db thru 0x03973c (0xfe62 bytes) 000722 1781231217.283487 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3973d): Mirror 10: 0xd6658ce9 (stale) Mirror 13: 0xd6658ce9 (stale) Mirror 15: 0xd6658ce9 (stale) Mirror 16: 0xd6658ce9 (stale) Mirror 18: 0xd6658ce9 (stale) Mirror 19: 0xd6658ce9 (stale) Mirror 20: 0xd6658ce9 (stale) Mirror 21: 0xd6658ce9 (stale) Mirror 22: 0xd6658ce9 (stale) Mirror 23: 0xd6658ce9 (stale) Mirror 24: 0xd6658ce9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3973d): Mirror 10: 0xd6658ce9 (stale) Mirror 13: 0xd6658ce9 (stale) Mirror 15: 0xd6658ce9 (stale) Mirror 16: 0xd6658ce9 (stale) Mirror 18: 0xd6658ce9 (stale) Mirror 19: 0xd6658ce9 (stale) Mirror 20: 0xd6658ce9 (stale) Mirror 21: 0xd6658ce9 (stale) Mirror 22: 0xd6658ce9 (stale) Mirror 23: 0xd6658ce9 (stale) Mirror 24: 0xd6658ce9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3973d: skipped 000723 1781231218.269001 write 0x03e6a5 thru 0x03ffff (0x195b bytes) 000724 1781231218.274509 MIRROR_SPLIT mirror 9 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xacb115a6 (stale) Mirror 13: 0xacb115a6 (stale) Mirror 15: 0xacb115a6 (stale) Mirror 16: 0xacb115a6 (stale) Mirror 18: 0xacb115a6 (stale) Mirror 19: 0xacb115a6 (stale) Mirror 20: 0xacb115a6 (stale) Mirror 21: 0xacb115a6 (stale) Mirror 23: 0xacb115a6 (stale) Mirror 24: 0xacb115a6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0xacb115a6 (stale) Mirror 13: 0xacb115a6 (stale) Mirror 15: 0xacb115a6 (stale) Mirror 16: 0xacb115a6 (stale) Mirror 18: 0xacb115a6 (stale) Mirror 19: 0xacb115a6 (stale) Mirror 20: 0xacb115a6 (stale) Mirror 21: 0xacb115a6 (stale) Mirror 23: 0xacb115a6 (stale) Mirror 24: 0xacb115a6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000725 1781231218.987661 write 0x035754 thru 0x038809 (0x30b6 bytes) 000726 1781231219.006621 write 0x029f27 thru 0x03337a (0x9454 bytes) 000730 1781231219.029428 read 0x021676 thru 0x025d59 (0x46e4 bytes) 000732 1781231219.040403 read 0x00e5bb thru 0x01578d (0x71d3 bytes) 000733 1781231219.083337 write 0x01f785 thru 0x0276e0 (0x7f5c bytes) 000735 1781231219.091009 mapwrite 0x036fc0 thru 0x03b97a (0x49bb bytes) 000736 1781231219.193588 write 0x03929d thru 0x03ffff (0x6d63 bytes) 000737 1781231219.195001 mapread 0x02a97c thru 0x03052e (0x5bb3 bytes) 000740 1781231219.201161 read 0x0042b8 thru 0x008644 (0x438d bytes) 000741 1781231219.202418 mapread 0x01159b thru 0x01b6e5 (0xa14b bytes) 000742 1781231219.208442 read 0x00b299 thru 0x01481b (0x9583 bytes) 000743 1781231219.212300 mapread 0x014192 thru 0x022d18 (0xeb87 bytes) 000748 1781231219.222619 mapwrite 0x006982 thru 0x00b340 (0x49bf bytes) 000749 1781231219.370624 read 0x0178ad thru 0x023389 (0xbadd bytes) 000754 1781231219.377608 write 0x00197d thru 0x00c1b1 (0xa835 bytes) 000755 1781231219.381691 mapread 0x022fcc thru 0x02f52d (0xc562 bytes) 000756 1781231219.384546 trunc from 0x040000 to 0x024e46 (0x1b1b9 bytes) 000757 1781231219.435504 read 0x020f68 thru 0x024e46 (0x3edf bytes) 000759 1781231219.444119 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x24e47): Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24e47): Mirror 10: 0xa6ab27a (stale) Mirror 13: 0xa6ab27a (stale) Mirror 15: 0xa6ab27a (stale) Mirror 16: 0xa6ab27a (stale) Mirror 18: 0xa6ab27a (stale) Mirror 19: 0xa6ab27a (stale) Mirror 20: 0xa6ab27a (stale) Mirror 21: 0xa6ab27a (stale) Mirror 23: 0xa6ab27a (stale) Mirror 24: 0xa6ab27a (stale) CRC-32 checksum value for chunk [0, 0x24e47): Mirror 1: 0x4d5e594a Mirror 25: 0x4d5e594a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24e47: skipped 000760 1781231221.405640 trunc from 0x024e47 to 0x0086b0 (0x1c796 bytes) 000761 1781231221.531903 trunc from 0x0086b1 to 0x024787 (0x1c0d7 bytes) 000762 1781231221.592275 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24788): Mirror 13: 0x6c6ab921 (stale) Mirror 15: 0x6c6ab921 (stale) Mirror 16: 0x6c6ab921 (stale) Mirror 18: 0x6c6ab921 (stale) Mirror 19: 0x6c6ab921 (stale) Mirror 20: 0x6c6ab921 (stale) Mirror 21: 0x6c6ab921 (stale) Mirror 23: 0x6c6ab921 (stale) Mirror 24: 0x6c6ab921 (stale) Mirror 25: 0x82614f5a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24788): Mirror 13: 0x6c6ab921 (stale) Mirror 15: 0x6c6ab921 (stale) Mirror 16: 0x6c6ab921 (stale) Mirror 18: 0x6c6ab921 (stale) Mirror 19: 0x6c6ab921 (stale) Mirror 20: 0x6c6ab921 (stale) Mirror 21: 0x6c6ab921 (stale) Mirror 23: 0x6c6ab921 (stale) Mirror 24: 0x6c6ab921 (stale) Mirror 25: 0x82614f5a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24788: skipped 000764 1781231222.451132 mapread 0x01ca40 thru 0x020b52 (0x4113 bytes) 000767 1781231222.495476 read 0x01a875 thru 0x01e4ed (0x3c79 bytes) 000768 1781231222.496181 mapwrite 0x0302e1 thru 0x03c3cb (0xc0eb bytes) 000769 1781231222.659005 write 0x02b0c7 thru 0x02b91b (0x855 bytes) 000770 1781231222.667358 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3c3cc): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c3cc): Mirror 13: 0xe5e5ba7a (stale) Mirror 15: 0xe5e5ba7a (stale) Mirror 16: 0xe5e5ba7a (stale) Mirror 18: 0xe5e5ba7a (stale) Mirror 19: 0xe5e5ba7a (stale) Mirror 20: 0xe5e5ba7a (stale) Mirror 21: 0xe5e5ba7a (stale) Mirror 23: 0xe5e5ba7a (stale) Mirror 24: 0xe5e5ba7a (stale) Mirror 25: 0xaccc9e5d (stale) CRC-32 checksum value for chunk [0, 0x3c3cc): Mirror 1: 0xe1770557 Mirror 26: 0xe1770557 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c3cc: skipped 000771 1781231224.162456 trunc from 0x03c3cc to 0x01e32e (0x1e09d bytes) 000772 1781231224.255648 write 0x03f35e thru 0x03ffff (0xca2 bytes) 000773 1781231224.257032 write 0x009475 thru 0x013657 (0xa1e3 bytes) 000774 1781231224.269618 read 0x0270fc thru 0x02cf5d (0x5e62 bytes) 000777 1781231224.284495 mapwrite 0x02a875 thru 0x0362b9 (0xba45 bytes) 000779 1781231224.363846 mapread 0x0370ad thru 0x03b07a (0x3fce bytes) 000780 1781231224.365702 mapread 0x010b62 thru 0x01aeaf (0xa34e bytes) 000781 1781231224.380283 read 0x026e40 thru 0x028870 (0x1a31 bytes) 000783 1781231224.382268 write 0x030f7d thru 0x03495e (0x39e2 bytes) 000784 1781231224.385214 mapread 0x01fd8e thru 0x020547 (0x7ba bytes) 000785 1781231224.407928 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xacb115a6 (stale) Mirror 15: 0xacb115a6 (stale) Mirror 16: 0xacb115a6 (stale) Mirror 18: 0xacb115a6 (stale) Mirror 19: 0xacb115a6 (stale) Mirror 20: 0xacb115a6 (stale) Mirror 21: 0xacb115a6 (stale) Mirror 23: 0xacb115a6 (stale) Mirror 24: 0xacb115a6 (stale) Mirror 25: 0x28922c1b (stale) Mirror 26: 0xbd169fd2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x25afcc7d Mirror 27: 0x25afcc7d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000786 1781231225.714537 mapread 0x0114bb thru 0x01fb76 (0xe6bc bytes) 000787 1781231225.748976 write 0x02fbcc thru 0x0338d1 (0x3d06 bytes) 000788 1781231225.912944 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xacb115a6 (stale) Mirror 15: 0xacb115a6 (stale) Mirror 16: 0xacb115a6 (stale) Mirror 19: 0xacb115a6 (stale) Mirror 20: 0xacb115a6 (stale) Mirror 21: 0xacb115a6 (stale) Mirror 23: 0xacb115a6 (stale) Mirror 24: 0xacb115a6 (stale) Mirror 25: 0x28922c1b (stale) Mirror 26: 0xbd169fd2 (stale) Mirror 27: 0x25afcc7d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xacb115a6 (stale) Mirror 15: 0xacb115a6 (stale) Mirror 16: 0xacb115a6 (stale) Mirror 19: 0xacb115a6 (stale) Mirror 20: 0xacb115a6 (stale) Mirror 21: 0xacb115a6 (stale) Mirror 23: 0xacb115a6 (stale) Mirror 24: 0xacb115a6 (stale) Mirror 25: 0x28922c1b (stale) Mirror 26: 0xbd169fd2 (stale) Mirror 27: 0x25afcc7d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000789 1781231227.013203 MIRROR_RESYNC 12 mirrors 000790 1781231230.151480 mapread 0x01d04e thru 0x027ae7 (0xaa9a bytes) 000793 1781231230.205353 trunc from 0x040000 to 0x007a78 (0x38587 bytes) 000794 1781231230.432131 mapwrite 0x03acfb thru 0x03e453 (0x3759 bytes) 000795 1781231230.686246 read 0x01f6b9 thru 0x02584a (0x6192 bytes) 000797 1781231230.720546 read 0x00a651 thru 0x013725 (0x90d5 bytes) 000798 1781231230.744118 read 0x03c478 thru 0x03e453 (0x1fdc bytes) 000799 1781231230.754704 trunc from 0x03e454 to 0x032c59 (0xb7fa bytes) truncating to largest ever: 0x39a56 000800 1781231230.791785 trunc from 0x032c5a to 0x039a55 (0x6dfc bytes) 000801 1781231230.830608 mapwrite 0x02912b thru 0x02a5bb (0x1491 bytes) 000802 1781231230.934028 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x39a56): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a56): Mirror 13: 0x9a85360 (stale) Mirror 15: 0x9a85360 (stale) Mirror 16: 0x9a85360 (stale) Mirror 19: 0x9a85360 (stale) Mirror 20: 0x9a85360 (stale) Mirror 21: 0x9a85360 (stale) Mirror 23: 0x9a85360 (stale) Mirror 24: 0x9a85360 (stale) Mirror 25: 0x9a85360 (stale) Mirror 26: 0x9a85360 (stale) Mirror 27: 0x9a85360 (stale) CRC-32 checksum value for chunk [0, 0x39a56): Mirror 1: 0x60259695 Mirror 28: 0x60259695 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a56: skipped 000803 1781231232.805142 MIRROR_RESYNC 13 mirrors 000804 1781231235.589186 mapread 0x027312 thru 0x02a8bb (0x35aa bytes) 000805 1781231235.612276 mapwrite 0x02bd7a thru 0x03683f (0xaac6 bytes) 000806 1781231235.928569 trunc from 0x039a56 to 0x035a39 (0x401c bytes) 000809 1781231235.956191 read 0x024eaf thru 0x02fc15 (0xad67 bytes) 000810 1781231235.958027 trunc from 0x035a3a to 0x022118 (0x13921 bytes) 000811 1781231236.004435 write 0x01a31a thru 0x01d51f (0x3206 bytes) 000814 1781231236.025944 write 0x032e1e thru 0x03c68f (0x9872 bytes) 000815 1781231236.031304 read 0x034642 thru 0x03c68f (0x804e bytes) 000816 1781231236.033952 write 0x023b0a thru 0x02dc76 (0xa16d bytes) 000817 1781231236.054120 trunc from 0x03c690 to 0x02db10 (0xeb7f bytes) 000818 1781231236.088814 MIRROR_RESYNC 13 mirrors 000819 1781231238.850901 trunc from 0x02db11 to 0x0099a3 (0x2416d bytes) 000820 1781231238.961476 write 0x02abf7 thru 0x02c4c9 (0x18d3 bytes) 000821 1781231238.962660 read 0x023ece thru 0x02c4c9 (0x85fc bytes) 000825 1781231238.978935 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x2c4ca): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c4ca): Mirror 13: 0x7a0642f5 (stale) Mirror 15: 0x7a0642f5 (stale) Mirror 16: 0x7a0642f5 (stale) Mirror 19: 0x7a0642f5 (stale) Mirror 20: 0x7a0642f5 (stale) Mirror 21: 0x7a0642f5 (stale) Mirror 23: 0x7a0642f5 (stale) Mirror 24: 0x7a0642f5 (stale) Mirror 25: 0x7a0642f5 (stale) Mirror 26: 0x7a0642f5 (stale) Mirror 27: 0x7a0642f5 (stale) Mirror 28: 0x7a0642f5 (stale) CRC-32 checksum value for chunk [0, 0x2c4ca): Mirror 1: 0x199dd702 Mirror 29: 0x199dd702 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c4ca: skipped 000827 1781231241.251188 mapwrite 0x00a07d thru 0x010033 (0x5fb7 bytes) 000829 1781231241.720403 mapwrite 0x03f201 thru 0x03ffff (0xdff bytes) 000830 1781231241.894539 write 0x003ca2 thru 0x00f6b4 (0xba13 bytes) 000831 1781231241.907194 read 0x01ba67 thru 0x022c6d (0x7207 bytes) 000834 1781231241.932025 mapread 0x0290a8 thru 0x02c0b7 (0x3010 bytes) 000835 1781231241.953518 mapread 0x0027b3 thru 0x010a35 (0xe283 bytes) 000837 1781231241.972849 MIRROR_RESYNC 14 mirrors 000841 1781231245.011183 mapread 0x03c954 thru 0x03ffff (0x36ac bytes) 000844 1781231245.034083 read 0x024dce thru 0x029388 (0x45bb bytes) 000846 1781231245.044610 write 0x03c398 thru 0x03ffff (0x3c68 bytes) 000847 1781231245.179175 mapread 0x016c32 thru 0x01e658 (0x7a27 bytes) 000849 1781231245.197596 trunc from 0x040000 to 0x02c918 (0x136e7 bytes) 000850 1781231245.235333 MIRROR_SPLIT mirror 11 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c919): Mirror 13: 0xdccea9e0 (stale) Mirror 15: 0xdccea9e0 (stale) Mirror 16: 0xdccea9e0 (stale) Mirror 19: 0xdccea9e0 (stale) Mirror 20: 0xdccea9e0 (stale) Mirror 21: 0xdccea9e0 (stale) Mirror 23: 0xdccea9e0 (stale) Mirror 24: 0xdccea9e0 (stale) Mirror 25: 0xdccea9e0 (stale) Mirror 26: 0xdccea9e0 (stale) Mirror 28: 0xdccea9e0 (stale) Mirror 29: 0xdccea9e0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c919): Mirror 13: 0xdccea9e0 (stale) Mirror 15: 0xdccea9e0 (stale) Mirror 16: 0xdccea9e0 (stale) Mirror 19: 0xdccea9e0 (stale) Mirror 20: 0xdccea9e0 (stale) Mirror 21: 0xdccea9e0 (stale) Mirror 23: 0xdccea9e0 (stale) Mirror 24: 0xdccea9e0 (stale) Mirror 25: 0xdccea9e0 (stale) Mirror 26: 0xdccea9e0 (stale) Mirror 28: 0xdccea9e0 (stale) Mirror 29: 0xdccea9e0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c919: skipped 000852 1781231246.123243 write 0x03f259 thru 0x03ffff (0xda7 bytes) 000855 1781231246.126467 write 0x010ea4 thru 0x020255 (0xf3b2 bytes) 000856 1781231246.142465 MIRROR_RESYNC 13 mirrors 000857 1781231248.360769 trunc from 0x040000 to 0x020fab (0x1f054 bytes) 000859 1781231248.503914 mapwrite 0x02ff80 thru 0x03a43d (0xa4be bytes) 000861 1781231248.573536 trunc from 0x03a43e to 0x01195b (0x28ae2 bytes) 000862 1781231248.605397 MIRROR_RESYNC 13 mirrors 000863 1781231251.102417 MIRROR_RESYNC 13 mirrors 000864 1781231252.008824 read 0x001f72 thru 0x01014f (0xe1de bytes) 000865 1781231252.029774 trunc from 0x01195c to 0x022c0c (0x112b1 bytes) 000866 1781231252.239813 write 0x0162a2 thru 0x02214c (0xbeab bytes) 000867 1781231252.282773 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x22c0d): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22c0d): Mirror 13: 0x69051eb0 (stale) Mirror 15: 0x69051eb0 (stale) Mirror 16: 0x69051eb0 (stale) Mirror 19: 0x69051eb0 (stale) Mirror 20: 0x69051eb0 (stale) Mirror 21: 0x69051eb0 (stale) Mirror 23: 0x69051eb0 (stale) Mirror 24: 0x69051eb0 (stale) Mirror 25: 0x69051eb0 (stale) Mirror 26: 0x69051eb0 (stale) Mirror 28: 0x69051eb0 (stale) Mirror 29: 0x69051eb0 (stale) CRC-32 checksum value for chunk [0, 0x22c0d): Mirror 1: 0x65afdc7a Mirror 30: 0x65afdc7a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22c0d: skipped 000869 1781231253.617316 read 0x00134a thru 0x00166a (0x321 bytes) 000870 1781231253.624500 write 0x021387 thru 0x02c633 (0xb2ad bytes) 000871 1781231253.803197 mapwrite 0x001a9c thru 0x0107fa (0xed5f bytes) 000876 1781231254.026213 mapread 0x011750 thru 0x01806b (0x691c bytes) 000879 1781231254.082315 MIRROR_RESYNC 14 mirrors 000881 1781231256.569108 mapread 0x0021de thru 0x004da9 (0x2bcc bytes) 000882 1781231256.590126 write 0x00d9fe thru 0x014653 (0x6c56 bytes) 000886 1781231256.726372 mapread 0x0186d9 thru 0x01883e (0x166 bytes) 000887 1781231256.741255 mapwrite 0x01fb36 thru 0x02344d (0x3918 bytes) 000888 1781231256.828572 trunc from 0x02c634 to 0x012f80 (0x196b3 bytes) 000889 1781231256.858100 trunc from 0x012f81 to 0x01c112 (0x9192 bytes) 000890 1781231256.887684 mapwrite 0x03b7ca thru 0x03f305 (0x3b3c bytes) 000892 1781231256.996570 read 0x03ebf2 thru 0x03f305 (0x714 bytes) 000893 1781231256.998052 mapwrite 0x00a3b0 thru 0x01a393 (0xffe4 bytes) 000897 1781231257.072845 mapwrite 0x024097 thru 0x027aa2 (0x3a0c bytes) 000898 1781231257.177056 MIRROR_RESYNC 14 mirrors 000899 1781231260.158732 mapwrite 0x0142bf thru 0x020d7b (0xcabd bytes) 000900 1781231260.411914 write 0x01af22 thru 0x020744 (0x5823 bytes) 000902 1781231260.420975 read 0x033d88 thru 0x034a1a (0xc93 bytes) 000903 1781231260.441823 MIRROR_SPLIT mirror 12 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f306): Mirror 13: 0x25472fe8 (stale) Mirror 15: 0x25472fe8 (stale) Mirror 16: 0x25472fe8 (stale) Mirror 19: 0x25472fe8 (stale) Mirror 20: 0x25472fe8 (stale) Mirror 21: 0x25472fe8 (stale) Mirror 23: 0x25472fe8 (stale) Mirror 24: 0x25472fe8 (stale) Mirror 25: 0x25472fe8 (stale) Mirror 26: 0x25472fe8 (stale) Mirror 28: 0x25472fe8 (stale) Mirror 30: 0x25472fe8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f306): Mirror 13: 0x25472fe8 (stale) Mirror 15: 0x25472fe8 (stale) Mirror 16: 0x25472fe8 (stale) Mirror 19: 0x25472fe8 (stale) Mirror 20: 0x25472fe8 (stale) Mirror 21: 0x25472fe8 (stale) Mirror 23: 0x25472fe8 (stale) Mirror 24: 0x25472fe8 (stale) Mirror 25: 0x25472fe8 (stale) Mirror 26: 0x25472fe8 (stale) Mirror 28: 0x25472fe8 (stale) Mirror 30: 0x25472fe8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f306: skipped 000904 1781231261.229121 mapwrite 0x00cbf9 thru 0x00e39b (0x17a3 bytes) 000906 1781231261.409303 write 0x0052d4 thru 0x009c24 (0x4951 bytes) truncating to largest ever: 0x3d283 000907 1781231261.419354 trunc from 0x03f306 to 0x03d282 (0x2083 bytes) 000908 1781231261.450610 mapread 0x02cbc0 thru 0x03742a (0xa86b bytes) 000911 1781231261.469619 trunc from 0x03d283 to 0x017cf2 (0x25590 bytes) 000913 1781231261.493317 write 0x015051 thru 0x01f0f2 (0xa0a2 bytes) 000914 1781231261.496045 mapread 0x00a470 thru 0x00b143 (0xcd4 bytes) 000915 1781231261.497845 mapwrite 0x00b391 thru 0x0104bc (0x512c bytes) 000918 1781231261.563330 mapwrite 0x019733 thru 0x024e1d (0xb6eb bytes) 000921 1781231261.712569 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x24e1e): Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24e1e): Mirror 13: 0x836960cd (stale) Mirror 15: 0x836960cd (stale) Mirror 16: 0x836960cd (stale) Mirror 19: 0x836960cd (stale) Mirror 20: 0x836960cd (stale) Mirror 21: 0x836960cd (stale) Mirror 23: 0x836960cd (stale) Mirror 24: 0x836960cd (stale) Mirror 25: 0x836960cd (stale) Mirror 26: 0x836960cd (stale) Mirror 28: 0x836960cd (stale) Mirror 30: 0x836960cd (stale) CRC-32 checksum value for chunk [0, 0x24e1e): Mirror 1: 0xe09edfa9 Mirror 31: 0xe09edfa9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24e1e: skipped 000922 1781231263.340296 trunc from 0x024e1e to 0x010005 (0x14e18 bytes) 000923 1781231263.506677 write 0x0287ed thru 0x02bf55 (0x3769 bytes) 000924 1781231263.513932 mapread 0x000b06 thru 0x003d34 (0x322f bytes) 000925 1781231263.594294 mapread 0x025ce6 thru 0x026a13 (0xd2e bytes) 000926 1781231263.617297 mapread 0x001717 thru 0x007296 (0x5b80 bytes) 000929 1781231263.620614 write 0x0077d5 thru 0x00bae7 (0x4313 bytes) 000931 1781231263.629430 write 0x03f584 thru 0x03f8f4 (0x371 bytes) 000933 1781231263.632242 mapwrite 0x02944c thru 0x03543e (0xbff3 bytes) 000934 1781231263.755313 mapread 0x0028b1 thru 0x00ea14 (0xc164 bytes) 000935 1781231263.758055 mapread 0x01a02d thru 0x02652b (0xc4ff bytes) 000936 1781231263.769303 mapwrite 0x005483 thru 0x00af2e (0x5aac bytes) 000937 1781231263.853825 write 0x0212f4 thru 0x030385 (0xf092 bytes) 000938 1781231263.859385 read 0x03a79b thru 0x03f8f4 (0x515a bytes) 000939 1781231263.871443 MIRROR_RESYNC 14 mirrors 000940 1781231266.930176 write 0x000b63 thru 0x00d35b (0xc7f9 bytes) 000941 1781231267.022431 write 0x02962d thru 0x02c7eb (0x31bf bytes) 000942 1781231267.036621 read 0x0386a3 thru 0x03b6ab (0x3009 bytes) 000944 1781231267.044113 read 0x0173d2 thru 0x01fc71 (0x88a0 bytes) 000945 1781231267.057064 mapwrite 0x0191dc thru 0x0267e3 (0xd608 bytes) 000947 1781231267.131257 read 0x03bddb thru 0x03f645 (0x386b bytes) 000948 1781231267.137661 MIRROR_RESYNC 14 mirrors 000949 1781231269.754296 trunc from 0x03f8f5 to 0x01a06d (0x25887 bytes) 000950 1781231269.881622 mapwrite 0x0151c7 thru 0x019cbc (0x4af6 bytes) 000952 1781231269.985975 trunc from 0x01a06e to 0x016488 (0x3be5 bytes) 000953 1781231270.022622 mapwrite 0x03dd73 thru 0x03ffff (0x228d bytes) 000954 1781231270.095002 MIRROR_SPLIT mirror 2 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x68982ef4 (stale) Mirror 16: 0x68982ef4 (stale) Mirror 19: 0x68982ef4 (stale) Mirror 20: 0x68982ef4 (stale) Mirror 21: 0x68982ef4 (stale) Mirror 23: 0x68982ef4 (stale) Mirror 24: 0x68982ef4 (stale) Mirror 25: 0x68982ef4 (stale) Mirror 26: 0x68982ef4 (stale) Mirror 28: 0x68982ef4 (stale) Mirror 30: 0x68982ef4 (stale) Mirror 31: 0x68982ef4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x68982ef4 (stale) Mirror 16: 0x68982ef4 (stale) Mirror 19: 0x68982ef4 (stale) Mirror 20: 0x68982ef4 (stale) Mirror 21: 0x68982ef4 (stale) Mirror 23: 0x68982ef4 (stale) Mirror 24: 0x68982ef4 (stale) Mirror 25: 0x68982ef4 (stale) Mirror 26: 0x68982ef4 (stale) Mirror 28: 0x68982ef4 (stale) Mirror 30: 0x68982ef4 (stale) Mirror 31: 0x68982ef4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000956 1781231271.078948 mapwrite 0x02053a thru 0x02e41b (0xdee2 bytes) 000957 1781231271.213337 trunc from 0x040000 to 0x011c7f (0x2e380 bytes) 000961 1781231271.240101 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x11c80): Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11c80): Mirror 13: 0xcb9ce33b (stale) Mirror 16: 0xcb9ce33b (stale) Mirror 19: 0xcb9ce33b (stale) Mirror 20: 0xcb9ce33b (stale) Mirror 21: 0xcb9ce33b (stale) Mirror 23: 0xcb9ce33b (stale) Mirror 24: 0xcb9ce33b (stale) Mirror 25: 0xcb9ce33b (stale) Mirror 26: 0xcb9ce33b (stale) Mirror 28: 0xcb9ce33b (stale) Mirror 30: 0xcb9ce33b (stale) Mirror 31: 0xcb9ce33b (stale) CRC-32 checksum value for chunk [0, 0x11c80): Mirror 1: 0xcb9ce33b Mirror 32: 0xcb9ce33b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11c80: skipped 000962 1781231272.620034 write 0x00cd7a thru 0x00ee84 (0x210b bytes) 000963 1781231272.750543 write 0x02fe27 thru 0x03d0b3 (0xd28d bytes) 000964 1781231272.770954 MIRROR_RESYNC 14 mirrors 000965 1781231275.511122 mapwrite 0x00d893 thru 0x0162c0 (0x8a2e bytes) 000966 1781231275.778958 trunc from 0x03d0b4 to 0x036d4d (0x6366 bytes) 000967 1781231275.802697 trunc from 0x036d4e to 0x00ee94 (0x27eb9 bytes) 000968 1781231275.839768 write 0x02c522 thru 0x03b93c (0xf41b bytes) 000969 1781231275.843224 trunc from 0x03b93d to 0x01bc2f (0x1fd0d bytes) 000971 1781231275.873706 mapread 0x000175 thru 0x001d4d (0x1bd9 bytes) 000972 1781231275.885976 mapread 0x005626 thru 0x00cf34 (0x790f bytes) 000973 1781231275.887967 mapwrite 0x019f92 thru 0x021923 (0x7992 bytes) 000974 1781231275.978777 mapwrite 0x0281ee thru 0x03083d (0x8650 bytes) 000975 1781231276.160090 mapread 0x00de0a thru 0x019f68 (0xc15f bytes) 000978 1781231276.173041 mapwrite 0x020476 thru 0x025cc0 (0x584b bytes) 000979 1781231276.261961 trunc from 0x03083e to 0x01e816 (0x12027 bytes) 000981 1781231276.300826 trunc from 0x01e817 to 0x02ab81 (0xc36b bytes) 000984 1781231276.344378 trunc from 0x02ab82 to 0x002da3 (0x27dde bytes) 000985 1781231276.377963 mapread 0x002464 thru 0x002da3 (0x940 bytes) 000987 1781231276.382076 mapread 0x0027e3 thru 0x002da3 (0x5c1 bytes) 000990 1781231276.405993 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x2da4): Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2da4): Mirror 13: 0xe708ed1 (stale) Mirror 16: 0xe708ed1 (stale) Mirror 19: 0xe708ed1 (stale) Mirror 20: 0xe708ed1 (stale) Mirror 21: 0xe708ed1 (stale) Mirror 23: 0xe708ed1 (stale) Mirror 24: 0xe708ed1 (stale) Mirror 25: 0xe708ed1 (stale) Mirror 26: 0xe708ed1 (stale) Mirror 28: 0xe708ed1 (stale) Mirror 30: 0xe708ed1 (stale) Mirror 31: 0xe708ed1 (stale) Mirror 32: 0xe708ed1 (stale) CRC-32 checksum value for chunk [0, 0x2da4): Mirror 1: 0xe708ed1 Mirror 33: 0xe708ed1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2da4: skipped 000991 1781231277.655262 mapread 0x000c11 thru 0x002da3 (0x2193 bytes) 000993 1781231277.680809 read 0x001599 thru 0x002da3 (0x180b bytes) 000995 1781231277.701918 MIRROR_SPLIT mirror 12 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2da4): Mirror 13: 0xe708ed1 (stale) Mirror 16: 0xe708ed1 (stale) Mirror 19: 0xe708ed1 (stale) Mirror 20: 0xe708ed1 (stale) Mirror 21: 0xe708ed1 (stale) Mirror 23: 0xe708ed1 (stale) Mirror 24: 0xe708ed1 (stale) Mirror 25: 0xe708ed1 (stale) Mirror 26: 0xe708ed1 (stale) Mirror 28: 0xe708ed1 (stale) Mirror 30: 0xe708ed1 (stale) Mirror 32: 0xe708ed1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2da4): Mirror 13: 0xe708ed1 (stale) Mirror 16: 0xe708ed1 (stale) Mirror 19: 0xe708ed1 (stale) Mirror 20: 0xe708ed1 (stale) Mirror 21: 0xe708ed1 (stale) Mirror 23: 0xe708ed1 (stale) Mirror 24: 0xe708ed1 (stale) Mirror 25: 0xe708ed1 (stale) Mirror 26: 0xe708ed1 (stale) Mirror 28: 0xe708ed1 (stale) Mirror 30: 0xe708ed1 (stale) Mirror 32: 0xe708ed1 (stale) CRC-32 checksum value for chunk [0, 0x2da4): Mirror 1: 0xe708ed1 Mirror 33: 0xe708ed1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2da4: skipped 000997 1781231278.456568 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2da4): Mirror 16: 0xe708ed1 (stale) Mirror 19: 0xe708ed1 (stale) Mirror 20: 0xe708ed1 (stale) Mirror 21: 0xe708ed1 (stale) Mirror 23: 0xe708ed1 (stale) Mirror 24: 0xe708ed1 (stale) Mirror 25: 0xe708ed1 (stale) Mirror 26: 0xe708ed1 (stale) Mirror 28: 0xe708ed1 (stale) Mirror 30: 0xe708ed1 (stale) Mirror 32: 0xe708ed1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2da4): Mirror 16: 0xe708ed1 (stale) Mirror 19: 0xe708ed1 (stale) Mirror 20: 0xe708ed1 (stale) Mirror 21: 0xe708ed1 (stale) Mirror 23: 0xe708ed1 (stale) Mirror 24: 0xe708ed1 (stale) Mirror 25: 0xe708ed1 (stale) Mirror 26: 0xe708ed1 (stale) Mirror 28: 0xe708ed1 (stale) Mirror 30: 0xe708ed1 (stale) Mirror 32: 0xe708ed1 (stale) CRC-32 checksum value for chunk [0, 0x2da4): Mirror 1: 0xe708ed1 Mirror 33: 0xe708ed1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2da4: skipped 000998 1781231279.244336 trunc from 0x002da4 to 0x018181 (0x153de bytes) 000999 1781231279.427222 read 0x00482c thru 0x012095 (0xd86a bytes) 001000 1781231279.454590 read 0x011964 thru 0x011ea6 (0x543 bytes) All operations completed A-OK! PASS 70a (186s) == sanity-flr test 71: check mirror extend parent fid ==== 22:28:04 (1781231284) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 22:28:07 (1781231287) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.208241 s, 10.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224439 s, 4.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26e:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x26f:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 22:28:17 (1781231297) Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre2 Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2532448 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7065344 .. Extending file size to 7884672 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8151616 .. Extending file size to 8075136 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8374560 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8512064 .. Extending file size to 8615840 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8627776 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8837376 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9078368 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57233 57234 57236 57238 57239 done 192.168.203.160@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg360-client.virtnet /mnt/lustre2 (opts:) 192.168.203.160@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg360-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:29:40 (1781231380) Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre2 Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2829888 .. Extending file size to 4038656 .. done Extending file size to 8791904 .. Extending file size to 5097344 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9121408 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6270848 .. Extending file size to 6967872 .. Extending file size to 7613408 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8054112 .. Extending file size to 8071200 .. Extending file size to 8370592 .. Extending file size to 8965984 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9316768 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58738 58739 58740 58743 58744 done 192.168.203.160@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg360-client.virtnet /mnt/lustre2 (opts:) 192.168.203.160@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg360-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:31:02 (1781231462) Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145254 s, 14.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.573038 s, 3.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x8a366f84 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.203.160@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg360-client.virtnet /mnt/lustre2 (opts:) PASS 200c (22s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:31:26 (1781231486) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0821425 s, 25.5 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 22:31:33 (1781231493) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1a4:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e6:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e7:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x28a:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:31:41 (1781231501) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.256758 s, 8.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:31:50 (1781231510) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.159308 s, 13.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2e9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ea:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:32:00 (1781231520) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.296324 s, 7.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x290:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:32:33 (1781231553) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg360-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141812 s, 7.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x292:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x291:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ec:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x292:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x291:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ed:0x0] } PASS 204d (60s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:33:33 (1781231613) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg360-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.335376 s, 6.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ee:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ef:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x293:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (62s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:34:35 (1781231675) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg360-server mds-ost sync done. Creating new pool test_204f oleg360-server: Pool lustre.test_204f created Adding targets to pool oleg360-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg360-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.382469 s, 5.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x294:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2f0:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x294:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg360-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg360-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg360-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:35:57 (1781231757) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:36:05 (1781231765) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:36:12 (1781231772) oleg360-server: Pool lustre.test_206 created Adding targets to pool oleg360-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg360-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg360-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg360-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg360-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:36:37 (1781231797) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0919413 s, 11.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421074 s, 24.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xfd4c9e2 Mirror 3: 0xfd4c9e2 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:36:49 (1781231809) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:36:53 (1781231813) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:36:56 (1781231816) Starting client: oleg360-client.virtnet: -o user_xattr,flock 192.168.203.160@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0790946 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xd20000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1140000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1140000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1140000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x19d0000 Read this (4 bytes): BBBB 192.168.203.160@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg360-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:37:32 (1781231852) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0255548 s, 321 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:37:40 (1781231860) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859783 s, 12.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1c9 lmm_fid: [0x200000404:0x1c9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1c9 lmm_fid: [0x200000404:0x1c9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x29f:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834377 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-11 22:37:41.000000000 -0400 Modify: 2026-06-11 22:37:41.000000000 -0400 Change: 2026-06-11 22:37:43.000000000 -0400 Birth: 2026-06-11 22:37:41.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:37:49 (1781231869) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:37:52 (1781231872) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.631026 s, 64.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0786091 s, 52.1 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 11 22:37 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1cb:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 11 22:37 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:38:01 (1781231881) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:38:08 (1781231888) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1829 sec =========== 22:38:18 (1781231898) === sanity-flr: start cleanup 22:38:20 (1781231900) === === sanity-flr: finish cleanup 22:38:26 (1781231906) ===