-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 05:41:35 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg151-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 05:41:56 (1787823716) === oleg151-client.virtnet: executing check_config_client /mnt/lustre oleg151-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg151-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8a55d0121000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a55d0121000.idle_timeout=debug disable quota as required oleg151-server: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 05:42:39 (1787823759) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8242] == sanity-flr test 0a: lfs mirror create with -N option == 05:42:41 (1787823761) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 05:42:59 (1787823779) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 05:43:02 (1787823782) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 05:43:06 (1787823786) PASS 0d (19s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 05:43:25 (1787823805) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 05:43:28 (1787823808) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 05:43:33 (1787823813) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8a55d0121000.stats=clear osc.lustre-OST0001-osc-ffff8a55d0121000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 05:43:40 (1787823820) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 05:43:50 (1787823830) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 05:43:59 (1787823839) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 05:44:02 (1787823842) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 05:44:09 (1787823849) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 05:44:18 (1787823858) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 05:44:25 (1787823865) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 05:44:32 (1787823872) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 05:44:40 (1787823880) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 05:44:49 (1787823889) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 05:44:55 (1787823895) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.487614 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102683 s, 10.2 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 05:45:02 (1787823902) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.475484 s, 33.1 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 05:45:09 (1787823909) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 05:45:19 (1787823919) Stopping /mnt/lustre-ost1 (opts:) on oleg151-server oleg151-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a55d0121000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a55d0121000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg151-server: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg151-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg151-server oleg151-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a55d0121000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a55d0121000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg151-server: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg151-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (61s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 05:46:20 (1787823980) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.407195 s, 12.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg151-server oleg151-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a55d0121000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a55d0121000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg151-server: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg151-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 05:47:00 (1787824020) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg151-server oleg151-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a55d0121000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a55d0121000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg151-server: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg151-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg151-server oleg151-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a55d0121000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a55d0121000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg151-server: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg151-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (74s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 05:48:14 (1787824094) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.30994 s, 15.8 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 05:48 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.855s user 0m0.000s sys 0m0.184s Stopping /mnt/lustre-ost1 (opts:) on oleg151-server oleg151-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8a55d0121000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a55d0121000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.988s user 0m0.000s sys 0m0.171s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg151-server: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg151-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg151-server oleg151-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8a55d0121000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a55d0121000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.081s user 0m0.000s sys 0m0.185s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg151-server: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg151-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg151-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a55d0121000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a55d0121000.ost_server_uuid in FULL state after 0 sec PASS 33b (80s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 05:49:34 (1787824174) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 05:49:37 (1787824177) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 05:49:41 (1787824181) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 05:49:44 (1787824184) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 05:49:52 (1787824192) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.35655 s, 35.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (10s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 05:50:02 (1787824202) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162187 s, 64.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368139 s, 28.5 MB/s PASS 36b (5s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 05:50:07 (1787824207) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 05:50:14 (1787824214) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000954508 s, 1.1 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0011992 s, 854 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000969743 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00303265 s, 338 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 05:50:22 (1787824222) create test files with size 15744734 .. 817730de22af3cdba62a342af96959d0 - 595928a9dd18644bac6876705e6e0fd2 - ef4ff58ebc13632e4ebc68516a868342 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg151-client.virtnet /mnt/lustre (opts:) Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre PASS 37 (18s) == sanity-flr test 38: resync ============================ 05:50:40 (1787824240) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (11s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 05:50:51 (1787824251) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 05:50:57 (1787824257) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811352 s, 12.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408149 s, 25.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 05:51:05 (1787824265) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0512017 s, 41.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.219201 s, 19.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.130768 s, 16.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (12s) == sanity-flr test 42: lfs mirror verify ================= 05:51:18 (1787824278) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 05:51:21 (1787824281) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 05:51:23 (1787824283) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219577 s, 47.8 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 05:51:31 (1787824291) llite.lustre-ffff8a55c9383000.enable_compression=1 striped dir -i1 -c2 -H crush2 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (7s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 05:51:38 (1787824298) llite.lustre-ffff8a55c9383000.enable_compression=1 striped dir -i1 -c2 -H all_char /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.018461 s, 56.8 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0254532 s, 41.2 MB/s **verify components PASS 43d (7s) == sanity-flr test 44a: lfs mirror split check =========== 05:51:45 (1787824305) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.237943 s, 13.2 MB/s PASS 44a (8s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 05:51:53 (1787824313) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcc:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 05:52:03 (1787824323) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.324746 s, 32.3 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0xd:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg151-client.virtnet /mnt/lustre (opts:) Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 05:52:13 (1787824333) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8a55eca6e000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.409409 s, 25.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg151-client.virtnet /mnt/lustre (opts:) Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8a55d1e3f000.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 05:52:31 (1787824351) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (37s) == sanity-flr test 46: Verify setstripe --copy option ==== 05:53:08 (1787824388) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 05:53:45 (1787824425) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (2s) == sanity-flr test 48: Verify snapshot mirror ============ 05:53:48 (1787824428) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.262682 s, 12.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.107068 s, 29.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x928f6745 (nosync,stale) Mirror 2: 0x928f6745 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 3a903b315cfe215dac58939b417df909 - mirror 1 checksum: 0cbf6cd5300a498b7f9365391ce6325a - mirror 2 checksum: 3a903b315cfe215dac58939b417df909 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T05:53:50-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T05:53:50-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 3a903b315cfe215dac58939b417df909 - mirror 1 checksum: 0cbf6cd5300a498b7f9365391ce6325a - mirror 2 checksum: 0cbf6cd5300a498b7f9365391ce6325a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T05:53:50-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T05:53:50-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } PASS 48 (8s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 05:53:56 (1787824436) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228997 s, 45.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 314368.. 315391: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (6s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 05:54:02 (1787824442) Failing mds1 on oleg151-server Stopping /mnt/lustre-mds1 (opts:) on oleg151-server 05:54:14 (1787824454) shut down facet: mds1 facet_host: oleg151-server facet_failover_host: oleg151-server Failover mds1 to oleg151-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0000 05:54:47 (1787824487) targets are mounted 05:54:47 (1787824487) facet_failover done oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0228327 s, 45.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd5:0x0] } PASS 50A (63s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 05:55:05 (1787824505) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0601596 s, 340 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0204657 s, 1.0 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 05:55:14 (1787824514) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0602125 s, 340 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0228668 s, 896 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 05:55:24 (1787824524) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00613896 s, 2.7 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 05:55:32 (1787824532) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.042768 s, 479 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 05:55:39 (1787824539) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0148488 s, 276 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0248706 s, 165 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 05:55:46 (1787824546) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (18s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 05:56:04 (1787824564) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 05:56:31 (1787824591) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 05:56:55 (1787824615) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 05:57:20 (1787824640) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 05:57:27 (1787824647) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 58841 mirror split pid 58842 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7044: 58842 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7044: 58842 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58841 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58842 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 05:58:35 (1787824715) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 4717 skipping zero size read skipping zero length zero range skipping zero length zero range skipping zero length punch hole truncating to largest ever: 0x17f47 000007 1787824717.351854 trunc from 00000000 to 0x017f46 (0x17f47 bytes) truncating to largest ever: 0x314a7 000008 1787824717.364838 trunc from 0x017f47 to 0x0314a6 (0x19560 bytes) 000009 1787824717.376747 punch from 0x0043f1 to 0x01066b (0xc27b bytes) 000010 1787824717.381159 trunc from 0x02e96f to 0x02fe2a (0x14bc bytes) 000011 1787824717.388912 MIRROR_EXTEND to 2 mirrors 000012 1787824717.887002 mapwrite 0x028617 thru 0x02f032 (0x6a1c bytes) 000013 1787824717.967126 trunc from 0x018cde to 0x01c9ce (0x3cf1 bytes) 000014 1787824717.972723 write 0x035d36 thru 0x03ffff (0xa2ca bytes) 000015 1787824717.978833 mapread 0x010b97 thru 0x015acb (0x4f35 bytes) fallocating to largest ever: 0x31950 000016 1787824717.988542 fallocate 0x02d420 thru 0x03194f (0x4530 bytes) 000017 1787824717.992974 mapwrite 0x011a59 thru 0x0166c0 (0x4c68 bytes) 000018 1787824718.009767 fallocate 0x01d80c thru 0x022748 (0x4f3d bytes) 000019 1787824718.014139 punch from 0x03ebf7 to 0x03ffff (0x1409 bytes) 000020 1787824718.018248 write 0x025529 thru 0x02949e (0x3f76 bytes) 000021 1787824718.020240 punch from 0x014c70 to 0x022b74 (0xdf05 bytes) 000023 1787824718.026083 fallocate 0x013501 thru 0x0218cf (0xe3cf bytes) zero_range to largest ever: 0x33dd3 000024 1787824718.030800 trunc from 0x02a7af to 0x033dd2 (0x9624 bytes) 000026 1787824718.035987 read 0x03f9e2 thru 0x03ffff (0x61e bytes) 000027 1787824718.040514 trunc from 0x031552 to 0x03b433 (0x9ee2 bytes) 000028 1787824718.044750 mapwrite 0x020732 thru 0x0273a7 (0x6c76 bytes) truncating to largest ever: 0x3e120 000029 1787824718.067175 trunc from 0x040000 to 0x03e11f (0x1ee0 bytes) 000030 1787824718.086070 punch from 0x0323fe to 0x03b47f (0x9082 bytes) 000031 1787824718.094621 MIRROR_RESYNC 2 mirrors 000032 1787824718.567631 fallocate 0x0191fb thru 0x0274ea (0xe2f0 bytes) 000033 1787824718.598722 read 0x02cd00 thru 0x034bb6 (0x7eb7 bytes) 000034 1787824718.607986 trunc from 0x021b3c to 0x0258da (0x3d9f bytes) 000035 1787824718.612634 trunc from 0x017e6d to 0x019bff (0x1d93 bytes) 000036 1787824718.619143 mapwrite 0x0173cf thru 0x01e30e (0x6f40 bytes) 000037 1787824718.648907 mapwrite 0x010922 thru 0x01b482 (0xab61 bytes) 000038 1787824718.675875 write 0x033503 thru 0x035001 (0x1aff bytes) 000041 1787824718.682691 trunc from 0x03e120 to 0x010542 (0x2dbdd bytes) 000042 1787824718.747496 mapread 0x004553 thru 0x010542 (0xbff0 bytes) 000043 1787824718.760006 write 0x02f629 thru 0x033a34 (0x440c bytes) 000045 1787824718.766964 MIRROR_RESYNC 2 mirrors fallocating to largest ever: 0x40000 000046 1787824719.210057 fallocate 0x03b9c0 thru 0x03ffff (0x4640 bytes) 000047 1787824719.238269 punch from 0x011844 to 0x0206d1 (0xee8e bytes) 000048 1787824719.242666 read 0x0305dd thru 0x03774f (0x7173 bytes) 000049 1787824719.249697 trunc from 0x0336de to 0x036ae7 (0x340a bytes) 000050 1787824719.258783 trunc from 0x040000 to 0x025ec1 (0x1a13e bytes) 000051 1787824719.277941 mapwrite 0x03d622 thru 0x03ffff (0x29de bytes) 000053 1787824719.318018 mapwrite 0x016b2c thru 0x019fa0 (0x3475 bytes) 000054 1787824719.339044 punch from 0x02bbb3 to 0x0376ab (0xbaf9 bytes) 000055 1787824719.342861 read 0x019d40 thru 0x022ffc (0x92bd bytes) 000056 1787824719.351934 read 0x002ed0 thru 0x005c7c (0x2dad bytes) 000057 1787824719.352736 mapwrite 0x000886 thru 0x00edbc (0xe537 bytes) 000058 1787824719.371002 read 0x01c500 thru 0x01f4d5 (0x2fd6 bytes) 000059 1787824719.371810 write 0x00d345 thru 0x01c886 (0xf542 bytes) 000060 1787824719.374563 punch from 0x01e0fc to 0x0228ea (0x47ef bytes) 000061 1787824719.378481 read 0x035e23 thru 0x03cacc (0x6caa bytes) 000062 1787824719.379156 mapread 0x02bedb thru 0x03b0cb (0xf1f1 bytes) 000063 1787824719.380681 read 0x018d60 thru 0x01f63a (0x68db bytes) 000065 1787824719.389933 read 0x03a552 thru 0x03ffff (0x5aae bytes) 000066 1787824719.390761 trunc from 0x02b533 to 0x037b6c (0xc63a bytes) 000067 1787824719.395782 read 0x02446b thru 0x02b10c (0x6ca2 bytes) 000068 1787824719.403817 fallocate 0x01db5a thru 0x01edc4 (0x126b bytes) 000069 1787824719.409441 fallocate 0x0281b8 thru 0x0374ba (0xf303 bytes) 000070 1787824719.414924 trunc from 0x040000 to 0x036b82 (0x947d bytes) 000071 1787824719.428253 mapwrite 0x028925 thru 0x02d9b4 (0x5090 bytes) 000072 1787824719.444281 punch from 0x0257d3 to 0x034455 (0xec83 bytes) 000073 1787824719.448475 fallocate 0x02f948 thru 0x03edb8 (0xf471 bytes) 000074 1787824719.452324 trunc from 0x03edb9 to 0x02f687 (0xf731 bytes) 000075 1787824719.471647 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000076 1787824719.591131 mapread 0x02c274 thru 0x02d832 (0x15bf bytes) 000077 1787824719.598776 read 0x023102 thru 0x024624 (0x1523 bytes) 000078 1787824719.602725 fallocate 0x03eac7 thru 0x03ffff (0x1539 bytes) 000079 1787824719.607056 punch from 0x030536 to 0x03c736 (0xc201 bytes) 000081 1787824719.613818 fallocate 0x00a72b thru 0x00c87b (0x2151 bytes) 000083 1787824719.620078 write 0x014aaa thru 0x0166aa (0x1c01 bytes) 000084 1787824719.630801 mapwrite 0x02e3e6 thru 0x03d6b0 (0xf2cb bytes) 000085 1787824719.663987 MIRROR_EXTEND to 2 mirrors 000086 1787824720.280056 trunc from 0x003040 to 0x00ea71 (0xba32 bytes) 000088 1787824720.323969 write 0x017e11 thru 0x01a90c (0x2afc bytes) 000089 1787824720.348244 punch from 0x002ae5 to 0x00b6d0 (0x8bec bytes) 000090 1787824720.367773 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000091 1787824720.515583 write 0x032658 thru 0x03ffff (0xd9a8 bytes) 000092 1787824720.526236 mapwrite 0x03d310 thru 0x03ffff (0x2cf0 bytes) 000093 1787824720.549891 fallocate 0x028d32 thru 0x02d7b6 (0x4a85 bytes) 000094 1787824720.554306 read 0x02cb69 thru 0x02dcee (0x1186 bytes) 000095 1787824720.567040 mapwrite 0x0247cf thru 0x02efbc (0xa7ee bytes) 000096 1787824720.626666 mapwrite 0x018731 thru 0x0249b1 (0xc281 bytes) 000097 1787824720.666175 MIRROR_EXTEND to 2 mirrors 000098 1787824721.162274 mapread 0x0381fc thru 0x03ffff (0x7e04 bytes) 000099 1787824721.176571 trunc from 0x01e8e6 to 0x02cb4e (0xe269 bytes) 000100 1787824721.208768 write 0x02a248 thru 0x034c32 (0xa9eb bytes) 000101 1787824721.223091 write 0x02ae62 thru 0x039e5a (0xeff9 bytes) 000103 1787824721.228857 trunc from 0x0078db to 0x011249 (0x996f bytes) 000104 1787824721.235304 mapwrite 0x0249f8 thru 0x033e7b (0xf484 bytes) 000105 1787824721.269610 write 0x0223e2 thru 0x02b1d7 (0x8df6 bytes) 000106 1787824721.271582 write 0x0266e5 thru 0x028da5 (0x26c1 bytes) 000108 1787824721.273887 mapread 0x015d05 thru 0x02183c (0xbb38 bytes) 000109 1787824721.282733 trunc from 0x040000 to 0x01d818 (0x227e7 bytes) 000110 1787824721.323552 mapread 0x009602 thru 0x00ca83 (0x3482 bytes) 000111 1787824721.335440 trunc from 0x0120c3 to 0x0142ce (0x220c bytes) 000112 1787824721.341062 trunc from 0x001d0b to 0x00bc78 (0x9f6e bytes) 000113 1787824721.345897 trunc from 0x01d819 to 0x004092 (0x19786 bytes) 000114 1787824721.365065 read 0x002735 thru 0x004092 (0x195e bytes) 000115 1787824721.371657 trunc from 0x004093 to 0x00a5be (0x652c bytes) 000117 1787824721.386997 punch from 0x005c49 to 0x00a5be (0x4976 bytes) 000120 1787824721.396730 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000121 1787824721.515664 fallocate 0x01c086 thru 0x01caa0 (0xa1b bytes) 000122 1787824721.519902 punch from 0x0046f0 to 0x0069a8 (0x22b9 bytes) 000123 1787824721.525543 write 0x0015de thru 0x00eb21 (0xd544 bytes) 000124 1787824721.538117 fallocate 0x00f9b2 thru 0x014479 (0x4ac8 bytes) 000126 1787824721.543712 write 0x03c0bc thru 0x03ffff (0x3f44 bytes) 000127 1787824721.545044 mapwrite 0x00b9f2 thru 0x00d377 (0x1986 bytes) 000128 1787824721.560972 trunc from 0x040000 to 0x02df65 (0x1209a bytes) 000129 1787824721.573956 write 0x020de8 thru 0x030a02 (0xfc1b bytes) 000130 1787824721.579825 read 0x02bd6c thru 0x030a02 (0x4c97 bytes) 000131 1787824721.580584 write 0x013e2b thru 0x016775 (0x294b bytes) 000133 1787824721.598829 trunc from 0x000760 to 0x00764f (0x6ef0 bytes) 000134 1787824721.604044 trunc from 0x0069eb to 0x00d9fb (0x7011 bytes) 000135 1787824721.608596 mapwrite 0x016bc3 thru 0x01f3dc (0x881a bytes) 000137 1787824721.645338 mapwrite 0x010104 thru 0x01cfab (0xcea8 bytes) 000138 1787824721.675439 fallocate 0x027ce4 thru 0x02cfcb (0x52e8 bytes) 000139 1787824721.680181 read 0x023dcb thru 0x024e98 (0x10ce bytes) 000140 1787824721.681052 trunc from 0x030a03 to 0x028d81 (0x7c81 bytes) 000142 1787824721.701957 MIRROR_EXTEND to 2 mirrors 000143 1787824722.305611 MIRROR_RESYNC 2 mirrors 000144 1787824722.626626 write 0x0104fa thru 0x0168a3 (0x63aa bytes) 000145 1787824722.681688 trunc from 0x01d96e to 0x028d81 (0xb414 bytes) 000147 1787824722.687564 trunc from 0x028b39 to 0x028d81 (0x249 bytes) 000148 1787824722.692529 trunc from 0x00d889 to 0x011150 (0x38c8 bytes) 000149 1787824722.706417 punch from 0x007dc7 to 0x007e0d (0x47 bytes) 000151 1787824722.714917 read 0x00ea8a thru 0x01466a (0x5be1 bytes) 000152 1787824722.725584 MIRROR_RESYNC 2 mirrors 000153 1787824723.248961 trunc from 0x028d82 to 0x039b2c (0x10dab bytes) 000154 1787824723.294801 write 0x0112a0 thru 0x01f2d6 (0xe037 bytes) 000155 1787824723.306878 punch from 0x016e08 to 0x01b93f (0x4b38 bytes) 000156 1787824723.330767 mapread 0x00d8ef thru 0x019f92 (0xc6a4 bytes) 000157 1787824723.367614 trunc from 0x0060b3 to 0x00cb0a (0x6a58 bytes) 000159 1787824723.374733 mapread 0x02b4cc thru 0x030cd6 (0x580b bytes) 000160 1787824723.392059 mapread 0x00d73f thru 0x01a6f6 (0xcfb8 bytes) 000161 1787824723.393259 mapread 0x0387fd thru 0x039b2c (0x1330 bytes) 000162 1787824723.402590 mapread 0x032c3d thru 0x0396d3 (0x6a97 bytes) 000163 1787824723.408850 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000164 1787824723.545640 punch from 0x0041f6 to 0x005cb0 (0x1abb bytes) 000165 1787824723.550109 read 0x031b46 thru 0x039b2c (0x7fe7 bytes) 000166 1787824723.562481 mapread 0x038b6a thru 0x039b2c (0xfc3 bytes) 000167 1787824723.563332 trunc from 0x039b2d to 0x010c62 (0x28eca bytes) 000168 1787824723.580109 trunc from 0x006d6b to 0x010c62 (0x9ef8 bytes) 000169 1787824723.586648 trunc from 0x010c63 to 0x03c42f (0x2b7cd bytes) 000170 1787824723.603618 mapread 0x031a35 thru 0x03c42f (0xa9fb bytes) 000171 1787824723.629747 mapread 0x0237fa thru 0x02de63 (0xa66a bytes) 000172 1787824723.637871 trunc from 0x03c430 to 0x024b99 (0x17896 bytes) 000173 1787824723.650505 read 0x01fe34 thru 0x0219e0 (0x1bad bytes) 000174 1787824723.651193 trunc from 0x012178 to 0x013f39 (0x1dc2 bytes) 000176 1787824723.656902 read 0x0242b0 thru 0x024b99 (0x8ea bytes) 000177 1787824723.657996 mapread 0x009045 thru 0x00fa8b (0x6a47 bytes) 000178 1787824723.666854 read 0x0146a6 thru 0x01dbc5 (0x9520 bytes) 000179 1787824723.668082 trunc from 0x02161d to 0x024b99 (0x357d bytes) 000181 1787824723.676728 MIRROR_EXTEND to 2 mirrors 000182 1787824724.263746 mapwrite 0x032d70 thru 0x03ffff (0xd290 bytes) 000183 1787824724.505963 write 0x003089 thru 0x003227 (0x19f bytes) 000184 1787824724.516760 mapwrite 0x01088c thru 0x01de02 (0xd577 bytes) 000185 1787824724.574158 trunc from 0x040000 to 0x038bbb (0x7444 bytes) 000186 1787824724.650433 trunc from 0x019b03 to 0x01bc9c (0x219a bytes) 000187 1787824724.669604 MIRROR_RESYNC 2 mirrors 000188 1787824725.669357 trunc from 0x0238bd to 0x02c660 (0x8da4 bytes) 000189 1787824725.729179 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38bbc): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38bbc): Mirror 2: 0x8d4727f9 (stale) CRC-32 checksum value for chunk [0, 0x38bbc): Mirror 1: 0x8d4727f9 Mirror 3: 0x8d4727f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38bbc: skipped 000190 1787824726.697353 trunc from 0x038bbc to 0x002349 (0x36872 bytes) 000191 1787824726.772698 punch from 0x000452 to 0x002349 (0x1ef8 bytes) 000192 1787824726.782971 trunc from 0x00234a to 0x02a3f3 (0x280aa bytes) 000193 1787824726.804734 fallocate 0x02b084 thru 0x03667e (0xb5fb bytes) 000194 1787824726.812090 fallocate 0x003b13 thru 0x010244 (0xc732 bytes) 000195 1787824726.817799 punch from 0x00943d to 0x00cb6c (0x3730 bytes) 000196 1787824726.826108 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a3f4): Mirror 2: 0xb3f85e9a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a3f4): Mirror 2: 0xb3f85e9a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a3f4: skipped 000197 1787824727.180732 trunc from 0x02a3f4 to 0x011d8d (0x18666 bytes) 000198 1787824727.228318 write 0x004243 thru 0x008328 (0x40e6 bytes) 000199 1787824727.248015 trunc from 0x00ed78 to 0x011d8d (0x3016 bytes) 000200 1787824727.259630 mapwrite 0x03c062 thru 0x03c325 (0x2c4 bytes) 000201 1787824727.315052 write 0x004bb5 thru 0x00f795 (0xabe1 bytes) 000202 1787824727.327914 fallocate 0x03ee86 thru 0x03ffff (0x117a bytes) 000203 1787824727.343898 fallocate 0x01a16c thru 0x027b85 (0xda1a bytes) 000204 1787824727.351587 punch from 0x039307 to 0x03ffff (0x6cf9 bytes) 000206 1787824727.355277 punch from 0x00b5d4 to 0x0135be (0x7feb bytes) 000207 1787824727.363302 trunc from 0x040000 to 0x0143f4 (0x2bc0b bytes) 000208 1787824727.375918 fallocate 0x033eef thru 0x0397d3 (0x58e5 bytes) 000209 1787824727.381361 fallocate 0x03713f thru 0x03ffff (0x8ec1 bytes) 000210 1787824727.390924 MIRROR_RESYNC 2 mirrors 000211 1787824727.924451 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000212 1787824728.041716 read 0x0249c3 thru 0x03074c (0xbd8a bytes) 000213 1787824728.057640 write 0x00508b thru 0x00b006 (0x5f7c bytes) 000214 1787824728.080920 trunc from 0x0397d4 to 0x01d111 (0x1c6c2 bytes) 000215 1787824728.094806 mapwrite 0x00a18a thru 0x0103e9 (0x6260 bytes) 000216 1787824728.135139 mapread 0x01c9b6 thru 0x01d111 (0x75c bytes) 000217 1787824728.146746 write 0x0177f4 thru 0x022e39 (0xb646 bytes) 000218 1787824728.147968 fallocate 0x004f2b thru 0x009bd1 (0x4ca7 bytes) 000219 1787824728.156579 punch from 0x005cf4 to 0x00b0fb (0x5408 bytes) 000221 1787824728.163373 mapread 0x007325 thru 0x010500 (0x91dc bytes) 000222 1787824728.178937 mapread 0x01ddbb thru 0x022e39 (0x507f bytes) 000223 1787824728.181568 write 0x025dcf thru 0x02c0d7 (0x6309 bytes) 000224 1787824728.182951 write 0x0129ab thru 0x0203a7 (0xd9fd bytes) 000225 1787824728.185923 trunc from 0x0037df to 0x00e316 (0xab38 bytes) 000226 1787824728.193375 punch from 0x027310 to 0x02c0d7 (0x4dc8 bytes) 000227 1787824728.206376 fallocate 0x00d0ee thru 0x00ec92 (0x1ba5 bytes) 000228 1787824728.211899 write 0x023d0d thru 0x02dbdb (0x9ecf bytes) 000229 1787824728.217327 punch from 0x0238b8 to 0x02550c (0x1c55 bytes) 000230 1787824728.236323 fallocate 0x036ba7 thru 0x03d403 (0x685d bytes) 000232 1787824728.241318 write 0x03c642 thru 0x03ffff (0x39be bytes) 000233 1787824728.242982 write 0x01839b thru 0x019860 (0x14c6 bytes) 000234 1787824728.244238 fallocate 0x037a33 thru 0x038595 (0xb63 bytes) 000235 1787824728.248598 read 0x020b7b thru 0x02831f (0x77a5 bytes) 000236 1787824728.257518 trunc from 0x010e0e to 0x016023 (0x5216 bytes) 000237 1787824728.265176 MIRROR_EXTEND to 2 mirrors 000238 1787824728.819651 trunc from 0x040000 to 0x017e48 (0x281b7 bytes) 000239 1787824728.885909 trunc from 0x017e49 to 0x01cdd7 (0x4f8f bytes) 000240 1787824728.902440 punch from 0x01cd38 to 0x01cdd7 (0xa0 bytes) 000241 1787824728.910381 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000243 1787824729.033879 punch from 0x01c83f to 0x01cdd7 (0x599 bytes) 000244 1787824729.037884 trunc from 0x01cdd8 to 0x03f4a4 (0x226cd bytes) 000245 1787824729.052626 write 0x0255ad thru 0x0312fd (0xbd51 bytes) 000246 1787824729.062689 mapread 0x0237f2 thru 0x02de5b (0xa66a bytes) 000247 1787824729.072417 punch from 0x00eb60 to 0x01bf2e (0xd3cf bytes) 000248 1787824729.077610 trunc from 0x006c95 to 0x0119a2 (0xad0e bytes) 000249 1787824729.085020 punch from 0x0076ae to 0x00bfd7 (0x492a bytes) 000250 1787824729.094849 write 0x007c2f thru 0x0128f0 (0xacc2 bytes) 000253 1787824729.105538 mapwrite 0x018cc4 thru 0x022af5 (0x9e32 bytes) 000254 1787824729.135933 write 0x005a9d thru 0x0139af (0xdf13 bytes) 000255 1787824729.143113 write 0x003c28 thru 0x00be91 (0x826a bytes) 000256 1787824729.148341 fallocate 0x020ca8 thru 0x029140 (0x8499 bytes) 000259 1787824729.154555 mapwrite 0x02a2d5 thru 0x037939 (0xd665 bytes) 000260 1787824729.183593 fallocate 0x007ca4 thru 0x00db6c (0x5ec9 bytes) 000261 1787824729.187242 trunc from 0x02c9ed to 0x033404 (0x6a18 bytes) 000262 1787824729.190634 trunc from 0x03f4a5 to 0x014a0e (0x2aa96 bytes) 000263 1787824729.204270 read 0x00210d thru 0x007fa3 (0x5e97 bytes) 000264 1787824729.208221 trunc from 0x014a0f to 0x000094 (0x1497a bytes) 000265 1787824729.221434 mapwrite 0x02bba9 thru 0x03774d (0xbba5 bytes) 000266 1787824729.270285 write 0x018ca2 thru 0x022edd (0xa23c bytes) 000267 1787824729.279087 trunc from 0x03774e to 0x017ede (0x1f86f bytes) 000270 1787824729.290919 trunc from 0x015626 to 0x017ede (0x28b9 bytes) 000271 1787824729.296335 read 0x00a9fb thru 0x015dfe (0xb404 bytes) 000272 1787824729.313592 punch from 0x01063a to 0x014b18 (0x44df bytes) 000273 1787824729.317769 read 0x002391 thru 0x00d24c (0xaebc bytes) 000274 1787824729.318658 punch from 0x00fd3d to 0x015837 (0x5afb bytes) 000275 1787824729.322874 trunc from 0x017edf to 0x0013fb (0x16ae3 bytes) 000277 1787824729.337656 mapread 0x000671 thru 0x0013fb (0xd8b bytes) 000278 1787824729.338893 trunc from 0x000b38 to 0x0013fb (0x8c4 bytes) 000279 1787824729.344305 mapwrite 0x01eec8 thru 0x024c15 (0x5d4e bytes) 000280 1787824729.384835 trunc from 0x013759 to 0x01fe24 (0xc6cc bytes) 000281 1787824729.392117 trunc from 0x007a8d to 0x00b0ba (0x362e bytes) 000282 1787824729.397150 mapwrite 0x02c984 thru 0x0332b7 (0x6934 bytes) 000283 1787824729.429875 trunc from 0x0332b8 to 0x02536d (0xdf4a bytes) 000284 1787824729.443819 trunc from 0x008725 to 0x01303a (0xa916 bytes) 000285 1787824729.453602 MIRROR_EXTEND to 2 mirrors 000286 1787824729.904809 fallocate 0x031525 thru 0x034697 (0x3173 bytes) 000287 1787824729.930122 punch from 0x01422b to 0x01cf83 (0x8d59 bytes) 000288 1787824729.935155 read 0x012d01 thru 0x01d0c4 (0xa3c4 bytes) 000289 1787824729.946844 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2536e): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2536e): Mirror 2: 0xd530aae (stale) CRC-32 checksum value for chunk [0, 0x2536e): Mirror 1: 0xd530aae Mirror 3: 0xd530aae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2536e: skipped 000290 1787824730.631331 mapwrite 0x02476f thru 0x02d9d7 (0x9269 bytes) 000291 1787824730.711236 trunc from 0x0141ae to 0x0171d1 (0x3024 bytes) 000292 1787824730.714676 trunc from 0x012911 to 0x019454 (0x6b44 bytes) 000293 1787824730.720590 fallocate 0x033a58 thru 0x03c07d (0x8626 bytes) 000294 1787824730.724350 mapwrite 0x026c2b thru 0x0367d7 (0xfbad bytes) 000295 1787824730.780460 mapread 0x02dd99 thru 0x0367d7 (0x8a3f bytes) 000296 1787824730.782503 trunc from 0x0367d8 to 0x02cb4b (0x9c8c bytes) 000297 1787824730.798776 trunc from 0x02cb4c to 0x01d892 (0xf2b9 bytes) 000298 1787824730.815797 write 0x01ed5e thru 0x02ea4e (0xfcf1 bytes) 000299 1787824730.817731 write 0x01314e thru 0x0150ea (0x1f9d bytes) 000300 1787824730.827225 trunc from 0x02ea4f to 0x0324d9 (0x3a8b bytes) 000301 1787824730.843225 fallocate 0x00e129 thru 0x0160af (0x7f87 bytes) 000302 1787824730.848709 trunc from 0x021616 to 0x02960c (0x7ff7 bytes) 000303 1787824730.861837 write 0x0266a9 thru 0x02eec9 (0x8821 bytes) 000304 1787824730.868318 write 0x005c05 thru 0x005e80 (0x27c bytes) 000305 1787824730.873417 fallocate 0x021656 thru 0x02970a (0x80b5 bytes) 000306 1787824730.877774 fallocate 0x01d711 thru 0x02d3f1 (0xfce1 bytes) 000307 1787824730.882751 read 0x02d7a3 thru 0x03082f (0x308d bytes) 000309 1787824730.888475 punch from 0x00134a to 0x00b70f (0xa3c6 bytes) 000310 1787824730.902645 trunc from 0x0324da to 0x027068 (0xb471 bytes) 000311 1787824730.914475 trunc from 0x027069 to 0x018ab3 (0xe5b5 bytes) 000312 1787824730.926043 trunc from 0x0084ca to 0x01615a (0xdc91 bytes) 000313 1787824730.931367 trunc from 0x018ab4 to 0x01f9bb (0x6f08 bytes) 000314 1787824730.943423 trunc from 0x01f9bc to 0x013602 (0xc3b9 bytes) 000315 1787824730.967128 fallocate 0x00ab3a thru 0x013120 (0x85e7 bytes) 000316 1787824730.972088 punch from 0x005006 to 0x0075f7 (0x25f2 bytes) 000317 1787824730.979654 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x13603): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13603): Mirror 2: 0x9abfc0ab (stale) Mirror 3: 0x9abfc0ab (stale) CRC-32 checksum value for chunk [0, 0x13603): Mirror 1: 0x9abfc0ab Mirror 4: 0x9abfc0ab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13603: skipped 000318 1787824731.821455 write 0x02357e thru 0x0334c6 (0xff49 bytes) 000319 1787824731.861833 punch from 0x0269e1 to 0x0334c6 (0xcae6 bytes) 000320 1787824731.880772 MIRROR_RESYNC 4 mirrors 000322 1787824732.741136 punch from 0x01b565 to 0x01e1f2 (0x2c8e bytes) 000323 1787824732.778293 fallocate 0x035d06 thru 0x038398 (0x2693 bytes) 000325 1787824732.784310 trunc from 0x0334c7 to 0x03aa3f (0x7579 bytes) 000326 1787824732.804171 mapwrite 0x011e2d thru 0x013ceb (0x1ebf bytes) 000327 1787824732.852690 mapread 0x01d0f5 thru 0x01f4a3 (0x23af bytes) 000328 1787824732.865817 read 0x034f58 thru 0x03aa3f (0x5ae8 bytes) 000329 1787824732.876209 punch from 0x0111e2 to 0x014c0d (0x3a2c bytes) 000330 1787824732.880846 mapread 0x0021ef thru 0x00b7b6 (0x95c8 bytes) 000331 1787824732.890564 trunc from 0x03aa40 to 0x002f1a (0x37b25 bytes) 000332 1787824732.911503 fallocate 0x01ba59 thru 0x01ec8d (0x3235 bytes) 000333 1787824732.916564 fallocate 0x006d1a thru 0x00997f (0x2c66 bytes) 000334 1787824732.921453 write 0x03799d thru 0x03ffff (0x8663 bytes) 000336 1787824732.929952 fallocate 0x0016e3 thru 0x008a66 (0x7384 bytes) 000337 1787824732.935587 mapread 0x018db6 thru 0x01adcd (0x2018 bytes) 000338 1787824732.946713 read 0x00931b thru 0x00cf94 (0x3c7a bytes) 000339 1787824732.961187 write 0x03fc8d thru 0x03ffff (0x373 bytes) 000340 1787824732.961905 trunc from 0x040000 to 0x01757c (0x28a83 bytes) 000341 1787824732.977749 write 0x0396fb thru 0x03ffff (0x6905 bytes) 000342 1787824732.979650 fallocate 0x012ba9 thru 0x01571b (0x2b73 bytes) 000343 1787824732.985505 fallocate 0x00891c thru 0x015fc0 (0xd6a5 bytes) 000344 1787824732.992158 trunc from 0x040000 to 0x037f95 (0x806a bytes) 000345 1787824733.024931 mapread 0x024775 thru 0x02899d (0x4229 bytes) 000346 1787824733.082245 mapread 0x02db56 thru 0x037f95 (0xa440 bytes) 000347 1787824733.175800 trunc from 0x037f96 to 0x0373e5 (0xbb0 bytes) 000348 1787824733.212483 read 0x021ccc thru 0x030d50 (0xf085 bytes) 000349 1787824733.231879 trunc from 0x0373e6 to 0x025da6 (0x1163f bytes) 000350 1787824733.250509 trunc from 0x025da7 to 0x029b5c (0x3db6 bytes) 000351 1787824733.291687 trunc from 0x029b5d to 0x016a4b (0x13111 bytes) 000352 1787824733.318806 trunc from 0x016a4c to 0x023a5c (0xd011 bytes) 000353 1787824733.348753 trunc from 0x008974 to 0x01111e (0x87ab bytes) 000354 1787824733.352776 read 0x004662 thru 0x0136bf (0xf05e bytes) 000355 1787824733.360527 mapread 0x012c62 thru 0x01962e (0x69cd bytes) 000356 1787824733.361505 write 0x00b32e thru 0x01525f (0x9f32 bytes) 000357 1787824733.362732 trunc from 0x023a5d to 0x037e39 (0x143dd bytes) 000358 1787824733.377306 mapwrite 0x01e703 thru 0x02ad62 (0xc660 bytes) 000359 1787824733.393763 mapwrite 0x02c24c thru 0x03b36f (0xf124 bytes) 000360 1787824733.453001 punch from 0x00f100 to 0x016291 (0x7192 bytes) 000361 1787824733.461527 mapread 0x02221c thru 0x0315ad (0xf392 bytes) 000362 1787824733.463494 punch from 0x00c91a to 0x01832c (0xba13 bytes) 000363 1787824733.470300 fallocate 0x00146c thru 0x0096bd (0x8252 bytes) 000364 1787824733.480464 trunc from 0x03b370 to 0x028f32 (0x1243d bytes) 000365 1787824733.503701 read 0x023c2e thru 0x0262ae (0x2681 bytes) 000366 1787824733.504308 mapread 0x00fa25 thru 0x01268b (0x2c67 bytes) 000367 1787824733.516913 punch from 0x026114 to 0x028f32 (0x2e1f bytes) 000368 1787824733.526681 fallocate 0x02530a thru 0x0289f0 (0x36e7 bytes) 000370 1787824733.535888 fallocate 0x00dfd5 thru 0x010c60 (0x2c8c bytes) 000371 1787824733.539833 fallocate 0x03ea1a thru 0x03ffff (0x15e6 bytes) 000372 1787824733.543694 read 0x00b630 thru 0x016500 (0xaed1 bytes) 000373 1787824733.544624 trunc from 0x028f33 to 0x00a48a (0x1eaa8 bytes) 000374 1787824733.567343 punch from 0x001914 to 0x003a00 (0x20ed bytes) 000375 1787824733.577873 mapread 0x003fcd thru 0x00a48a (0x64be bytes) 000376 1787824733.585543 punch from 0x0079cd to 0x00a48a (0x2abe bytes) 000377 1787824733.592968 fallocate 0x01fece thru 0x02909d (0x91d0 bytes) 000378 1787824733.600035 write 0x02a0b0 thru 0x02b647 (0x1598 bytes) 000379 1787824733.602457 mapread 0x00b33d thru 0x011969 (0x662d bytes) 000380 1787824733.625160 trunc from 0x01cab1 to 0x021409 (0x4959 bytes) 000381 1787824733.633327 trunc from 0x026346 to 0x02b647 (0x5302 bytes) 000382 1787824733.642790 punch from 0x01e13a to 0x02b647 (0xd50e bytes) 000383 1787824733.649061 mapwrite 0x033a1a thru 0x03636e (0x2955 bytes) 000384 1787824733.693972 fallocate 0x0028c8 thru 0x00ffcc (0xd705 bytes) 000385 1787824733.712599 fallocate 0x03da80 thru 0x03ffff (0x2580 bytes) 000387 1787824733.724322 trunc from 0x008109 to 0x00a203 (0x20fb bytes) 000389 1787824733.734493 trunc from 0x003c5a to 0x00682c (0x2bd3 bytes) 000390 1787824733.741715 mapwrite 0x02f88d thru 0x030e85 (0x15f9 bytes) 000391 1787824733.759597 mapwrite 0x03472b thru 0x034c2e (0x504 bytes) 000392 1787824733.770853 mapwrite 0x00d1b4 thru 0x0112bb (0x4108 bytes) 000393 1787824733.785294 mapwrite 0x01b7bf thru 0x029806 (0xe048 bytes) 000394 1787824733.860587 punch from 0x00265a to 0x00b39f (0x8d46 bytes) 000395 1787824733.864697 punch from 0x00e098 to 0x0140bc (0x6025 bytes) 000396 1787824733.873012 read 0x030fc0 thru 0x03636e (0x53af bytes) 000397 1787824733.873973 fallocate 0x032ec2 thru 0x03ffff (0xd13e bytes) 000398 1787824733.878374 trunc from 0x03636f to 0x0157be (0x20bb0 bytes) 000399 1787824733.895811 trunc from 0x00e6cf to 0x0157be (0x70f0 bytes) 000400 1787824733.901440 read 0x00d094 thru 0x0157be (0x872b bytes) 000401 1787824733.915056 mapread 0x00678e thru 0x00b2ff (0x4b72 bytes) 000402 1787824733.916150 punch from 0x014a62 to 0x0157be (0xd5d bytes) 000403 1787824733.920565 trunc from 0x0157bf to 0x022464 (0xcca6 bytes) 000404 1787824733.933711 fallocate 0x0169f6 thru 0x01b0ec (0x46f7 bytes) 000405 1787824733.938293 write 0x032921 thru 0x032f67 (0x647 bytes) 000406 1787824733.939264 write 0x007e9e thru 0x00dc03 (0x5d66 bytes) 000407 1787824733.949486 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x32f68): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32f68): Mirror 2: 0xb799e391 (stale) Mirror 3: 0xb799e391 (stale) Mirror 4: 0xb799e391 (stale) CRC-32 checksum value for chunk [0, 0x32f68): Mirror 1: 0x467403ce Mirror 5: 0x467403ce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32f68: skipped 000408 1787824734.625173 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x32f68): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32f68): Mirror 2: 0xb799e391 (stale) Mirror 3: 0xb799e391 (stale) Mirror 4: 0xb799e391 (stale) CRC-32 checksum value for chunk [0, 0x32f68): Mirror 1: 0x467403ce Mirror 5: 0x467403ce Mirror 6: 0x467403ce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32f68: skipped 000410 1787824735.423565 write 0x008b83 thru 0x00d982 (0x4e00 bytes) 000411 1787824735.479024 fallocate 0x023ad9 thru 0x0335bd (0xfae5 bytes) 000412 1787824735.485752 punch from 0x029793 to 0x02d355 (0x3bc3 bytes) 000413 1787824735.494610 read 0x02d440 thru 0x032f67 (0x5b28 bytes) 000414 1787824735.505826 punch from 0x0248f7 to 0x032376 (0xda80 bytes) 000415 1787824735.511526 trunc from 0x00e817 to 0x015914 (0x70fe bytes) 000416 1787824735.519627 punch from 0x007982 to 0x01489a (0xcf19 bytes) 000417 1787824735.539110 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32f68): Mirror 3: 0xb799e391 (stale) Mirror 4: 0xb799e391 (stale) Mirror 5: 0x467403ce (stale) Mirror 6: 0x467403ce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32f68): Mirror 3: 0xb799e391 (stale) Mirror 4: 0xb799e391 (stale) Mirror 5: 0x467403ce (stale) Mirror 6: 0x467403ce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32f68: skipped 000418 1787824735.908220 mapwrite 0x02bac9 thru 0x02f710 (0x3c48 bytes) 000419 1787824735.937910 write 0x03b215 thru 0x03ffff (0x4deb bytes) 000420 1787824735.941814 fallocate 0x0252d7 thru 0x033d93 (0xeabd bytes) 000421 1787824735.951048 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x8d3f9938 (stale) Mirror 4: 0x8d3f9938 (stale) Mirror 5: 0xf1488e2d (stale) Mirror 6: 0xf1488e2d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x93800a47 Mirror 7: 0x93800a47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000422 1787824736.853755 MIRROR_RESYNC 6 mirrors 000423 1787824738.288484 punch from 0x03f5e5 to 0x03ffff (0xa1b bytes) 000424 1787824738.328074 punch from 0x001a9d to 0x00b7cb (0x9d2f bytes) 000425 1787824738.332590 punch from 0x00a8eb to 0x01050b (0x5c21 bytes) 000426 1787824738.336382 mapwrite 0x01d0aa thru 0x02a820 (0xd777 bytes) 000427 1787824738.385533 mapwrite 0x025c40 thru 0x02a674 (0x4a35 bytes) 000428 1787824738.406333 fallocate 0x0082b1 thru 0x00f199 (0x6ee9 bytes) 000429 1787824738.424891 MIRROR_RESYNC 6 mirrors 000430 1787824739.686716 write 0x006f04 thru 0x00ffb6 (0x90b3 bytes) 000431 1787824739.784823 trunc from 0x040000 to 0x01e784 (0x2187b bytes) 000432 1787824739.818885 read 0x007bc9 thru 0x00d8fa (0x5d32 bytes) 000433 1787824739.819522 trunc from 0x01e785 to 0x02652c (0x7da8 bytes) 000434 1787824739.845139 punch from 0x005d25 to 0x00a26a (0x4546 bytes) 000435 1787824739.865084 MIRROR_RESYNC 6 mirrors 000436 1787824741.332635 mapwrite 0x03fe5c thru 0x03ffff (0x1a4 bytes) 000437 1787824741.402892 fallocate 0x00d24e thru 0x011225 (0x3fd8 bytes) 000438 1787824741.408539 mapwrite 0x00f727 thru 0x011d70 (0x264a bytes) 000439 1787824741.436182 mapread 0x02a70a thru 0x03439c (0x9c93 bytes) 000440 1787824741.451313 mapwrite 0x01850a thru 0x018663 (0x15a bytes) 000441 1787824741.460189 mapwrite 0x0047b6 thru 0x00c089 (0x78d4 bytes) 000442 1787824741.476911 fallocate 0x0299a6 thru 0x034599 (0xabf4 bytes) 000443 1787824741.480360 punch from 0x039641 to 0x03ffff (0x69bf bytes) 000444 1787824741.483550 punch from 0x0173ab to 0x01c8ef (0x5545 bytes) 000445 1787824741.489073 trunc from 0x03320e to 0x03af29 (0x7d1c bytes) 000446 1787824741.493934 mapread 0x00abed thru 0x00c03a (0x144e bytes) 000448 1787824741.495567 fallocate 0x03363f thru 0x035378 (0x1d3a bytes) 000450 1787824741.499652 fallocate 0x000886 thru 0x00b8b8 (0xb033 bytes) 000451 1787824741.503170 mapwrite 0x02b7a3 thru 0x02be02 (0x660 bytes) 000452 1787824741.512503 mapread 0x001814 thru 0x0054af (0x3c9c bytes) 000453 1787824741.513357 trunc from 0x040000 to 0x018dc5 (0x2723a bytes) 000454 1787824741.532594 read 0x010367 thru 0x013cd7 (0x3971 bytes) 000455 1787824741.533217 punch from 0x003f06 to 0x00b4ec (0x75e7 bytes) 000456 1787824741.542490 mapread 0x00a2f6 thru 0x013daa (0x9ab5 bytes) 000458 1787824741.557495 mapread 0x0141ee thru 0x01432a (0x13d bytes) 000459 1787824741.558364 mapread 0x016790 thru 0x018dc5 (0x2636 bytes) 000460 1787824741.563727 trunc from 0x0066d0 to 0x00d895 (0x71c6 bytes) 000462 1787824741.570851 write 0x00ae47 thru 0x018b9b (0xdd55 bytes) 000463 1787824741.582155 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18dc6): Mirror 4: 0xe8dc4df9 (stale) Mirror 5: 0xe8dc4df9 (stale) Mirror 6: 0xe8dc4df9 (stale) Mirror 7: 0xe8dc4df9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18dc6): Mirror 4: 0xe8dc4df9 (stale) Mirror 5: 0xe8dc4df9 (stale) Mirror 6: 0xe8dc4df9 (stale) Mirror 7: 0xe8dc4df9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18dc6: skipped 000464 1787824741.852085 read 0x005aa7 thru 0x0121d8 (0xc732 bytes) 000465 1787824741.857582 fallocate 0x03fa6c thru 0x03ffff (0x594 bytes) 000466 1787824741.860697 mapread 0x0047e1 thru 0x007efc (0x371c bytes) 000467 1787824741.863812 read 0x01500b thru 0x016083 (0x1079 bytes) 000468 1787824741.868957 punch from 0x0045d9 to 0x00f89c (0xb2c4 bytes) 000470 1787824741.872066 fallocate 0x033cf0 thru 0x03d96d (0x9c7e bytes) 000471 1787824741.874743 write 0x0053a0 thru 0x0122cf (0xcf30 bytes) 000472 1787824741.880447 mapread 0x016750 thru 0x018dc5 (0x2676 bytes) 000473 1787824741.881258 fallocate 0x03af57 thru 0x03ffff (0x50a9 bytes) 000474 1787824741.884226 punch from 0x0034ee to 0x005155 (0x1c68 bytes) 000475 1787824741.893280 write 0x00b169 thru 0x00fed6 (0x4d6e bytes) 000476 1787824741.894106 read 0x003dd6 thru 0x007bcc (0x3df7 bytes) 000477 1787824741.900411 write 0x004100 thru 0x004c0c (0xb0d bytes) 000478 1787824741.901117 write 0x0365c9 thru 0x03ca54 (0x648c bytes) 000479 1787824741.901742 punch from 0x03c1db to 0x03ffff (0x3e25 bytes) 000480 1787824741.909370 trunc from 0x03894d to 0x03ffff (0x76b3 bytes) 000481 1787824741.916556 punch from 0x00512e to 0x00c5c7 (0x749a bytes) 000482 1787824741.921354 mapread 0x020d27 thru 0x023926 (0x2c00 bytes) 000483 1787824741.922168 trunc from 0x040000 to 0x03dcb8 (0x2347 bytes) 000484 1787824741.933759 punch from 0x01f8f7 to 0x020031 (0x73b bytes) 000485 1787824741.941466 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dcb9): Mirror 4: 0x80afe23b (stale) Mirror 5: 0x80afe23b (stale) Mirror 7: 0x80afe23b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dcb9): Mirror 4: 0x80afe23b (stale) Mirror 5: 0x80afe23b (stale) Mirror 7: 0x80afe23b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dcb9: skipped 000486 1787824742.294111 fallocate 0x01566c thru 0x0194ac (0x3e41 bytes) 000487 1787824742.302294 mapwrite 0x030515 thru 0x036867 (0x6353 bytes) 000488 1787824742.348474 trunc from 0x03dcb9 to 0x009cbd (0x33ffb bytes) 000490 1787824742.368964 mapread 0x001466 thru 0x00937b (0x7f16 bytes) 000491 1787824742.376630 trunc from 0x008346 to 0x009cbd (0x1978 bytes) 000492 1787824742.380029 mapwrite 0x0004fb thru 0x00bad4 (0xb5da bytes) 000493 1787824742.424235 write 0x035cea thru 0x03ffff (0xa316 bytes) 000494 1787824742.426641 fallocate 0x01f58b thru 0x02cbd3 (0xd649 bytes) 000495 1787824742.435958 trunc from 0x037d9f to 0x03bde6 (0x4048 bytes) 000496 1787824742.449451 read 0x00f2eb thru 0x015aec (0x6802 bytes) 000497 1787824742.456442 trunc from 0x02157e to 0x0231b6 (0x1c39 bytes) 000498 1787824742.460366 punch from 0x03897a to 0x03ffff (0x7686 bytes) 000499 1787824742.465534 read 0x001706 thru 0x00bce1 (0xa5dc bytes) 000500 1787824742.466486 mapread 0x03ab4a thru 0x03b853 (0xd0a bytes) 000501 1787824742.481136 fallocate 0x01129d thru 0x018b3b (0x789f bytes) 000502 1787824742.486032 mapwrite 0x01cb1a thru 0x024be1 (0x80c8 bytes) 000503 1787824742.503006 punch from 0x02f331 to 0x03d822 (0xe4f2 bytes) 000505 1787824742.507150 punch from 0x03dee6 to 0x03ffff (0x211a bytes) 000506 1787824742.511045 mapwrite 0x0014ec thru 0x0089ab (0x74c0 bytes) 000507 1787824742.524254 trunc from 0x040000 to 0x039f0e (0x60f1 bytes) 000508 1787824742.534153 fallocate 0x0026d0 thru 0x006dff (0x4730 bytes) 000509 1787824742.541458 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39f0f): Mirror 5: 0xc3737b20 (stale) Mirror 7: 0xc3737b20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39f0f): Mirror 5: 0xc3737b20 (stale) Mirror 7: 0xc3737b20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39f0f: skipped 000510 1787824742.805046 mapread 0x0229b3 thru 0x0305cb (0xdc19 bytes) 000511 1787824742.830569 write 0x015d5a thru 0x01c46a (0x6711 bytes) 000513 1787824742.843746 mapread 0x00ccd0 thru 0x01a158 (0xd489 bytes) 000514 1787824742.851917 punch from 0x02a371 to 0x02ba0c (0x169c bytes) 000515 1787824742.856497 fallocate 0x03b9bf thru 0x03ffff (0x4641 bytes) 000516 1787824742.860904 mapread 0x013d70 thru 0x021626 (0xd8b7 bytes) 000517 1787824742.862484 read 0x021c08 thru 0x024243 (0x263c bytes) 000518 1787824742.862919 punch from 0x021bd8 to 0x02b32d (0x9756 bytes) 000519 1787824742.867573 trunc from 0x001201 to 0x010ba8 (0xf9a8 bytes) 000520 1787824742.871753 fallocate 0x002d2e thru 0x00d1e4 (0xa4b7 bytes) 000522 1787824742.875590 mapread 0x03b924 thru 0x03c60c (0xce9 bytes) 000523 1787824742.882560 read 0x029e12 thru 0x0387ab (0xe99a bytes) 000524 1787824742.888795 read 0x00a444 thru 0x013e9d (0x9a5a bytes) 000526 1787824742.889520 punch from 0x02a7f3 to 0x037563 (0xcd71 bytes) 000527 1787824742.892514 trunc from 0x009e33 to 0x015885 (0xba53 bytes) 000528 1787824742.901526 read 0x02cf16 thru 0x02fc7f (0x2d6a bytes) 000529 1787824742.908334 mapread 0x01c664 thru 0x024061 (0x79fe bytes) 000530 1787824742.909169 trunc from 0x040000 to 0x036930 (0x96cf bytes) 000532 1787824742.916653 fallocate 0x021af1 thru 0x02555e (0x3a6e bytes) 000533 1787824742.920230 fallocate 0x00e366 thru 0x01df3f (0xfbda bytes) 000534 1787824742.924305 mapread 0x014705 thru 0x01eba5 (0xa4a1 bytes) 000535 1787824742.925460 punch from 0x005994 to 0x0073a6 (0x1a13 bytes) 000536 1787824742.928992 write 0x02eaf8 thru 0x02f55c (0xa65 bytes) 000537 1787824742.929671 punch from 0x02d12b to 0x036930 (0x9806 bytes) 000538 1787824742.938497 fallocate 0x008be8 thru 0x01512b (0xc544 bytes) 000539 1787824742.942799 read 0x02e38c thru 0x036930 (0x85a5 bytes) 000540 1787824742.952380 write 0x022d30 thru 0x02718a (0x445b bytes) 000541 1787824742.953767 read 0x002ddd thru 0x010e0d (0xe031 bytes) 000542 1787824742.954785 punch from 0x00a247 to 0x0129c4 (0x877e bytes) 000543 1787824742.959636 write 0x01ba83 thru 0x01d5ab (0x1b29 bytes) 000545 1787824742.966203 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36931): Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36931): Mirror 5: 0x47dbfac5 (stale) Mirror 7: 0x47dbfac5 (stale) CRC-32 checksum value for chunk [0, 0x36931): Mirror 1: 0xa7cce6ba Mirror 8: 0xa7cce6ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36931: skipped 000546 1787824743.724225 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36931): Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36931): Mirror 5: 0x47dbfac5 (stale) Mirror 7: 0x47dbfac5 (stale) CRC-32 checksum value for chunk [0, 0x36931): Mirror 1: 0xa7cce6ba Mirror 8: 0xa7cce6ba Mirror 9: 0xa7cce6ba lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36931: skipped 000547 1787824744.547381 trunc from 0x036931 to 0x013616 (0x2331a bytes) 000548 1787824744.607446 write 0x01a0cc thru 0x01b818 (0x174d bytes) 000549 1787824744.608890 mapread 0x0150be thru 0x019ce1 (0x4c24 bytes) 000550 1787824744.645055 write 0x019ed4 thru 0x01edee (0x4f1b bytes) 000551 1787824744.647650 mapread 0x01d02a thru 0x01edee (0x1dc5 bytes) 000553 1787824744.648885 mapwrite 0x02f6be thru 0x033ffa (0x493d bytes) 000554 1787824744.685073 read 0x006774 thru 0x00c560 (0x5ded bytes) 000555 1787824744.694233 trunc from 0x0100e8 to 0x01732e (0x7247 bytes) 000556 1787824744.699065 mapwrite 0x03e829 thru 0x03ffff (0x17d7 bytes) 000557 1787824744.733813 mapread 0x01bfdd thru 0x02a56d (0xe591 bytes) 000558 1787824744.744888 trunc from 0x040000 to 0x01d158 (0x22ea7 bytes) 000559 1787824744.764158 mapread 0x002f06 thru 0x005db3 (0x2eae bytes) 000560 1787824744.773131 mapwrite 0x00dbc1 thru 0x01bb3d (0xdf7d bytes) 000561 1787824744.812620 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d159): Mirror 7: 0x51c98d73 (stale) Mirror 8: 0x53321693 (stale) Mirror 9: 0x53321693 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d159): Mirror 7: 0x51c98d73 (stale) Mirror 8: 0x53321693 (stale) Mirror 9: 0x53321693 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d159: skipped 000562 1787824745.097028 mapwrite 0x030dca thru 0x03ff04 (0xf13b bytes) 000563 1787824745.172091 fallocate 0x0179aa thru 0x0210bb (0x9712 bytes) 000564 1787824745.180376 MIRROR_RESYNC 4 mirrors 000565 1787824745.940555 mapwrite 0x03d0f2 thru 0x03ffff (0x2f0e bytes) 000566 1787824746.027052 write 0x03e71c thru 0x03fece (0x17b3 bytes) 000567 1787824746.028136 write 0x01ba7b thru 0x0269da (0xaf60 bytes) 000568 1787824746.047467 trunc from 0x040000 to 0x015cf5 (0x2a30a bytes) 000569 1787824746.065758 trunc from 0x00cfe6 to 0x015cf5 (0x8d10 bytes) 000570 1787824746.075043 MIRROR_RESYNC 4 mirrors 000571 1787824746.652066 punch from 0x009335 to 0x013ddb (0xaaa7 bytes) 000572 1787824746.687089 trunc from 0x0117ff to 0x015cf5 (0x44f7 bytes) 000573 1787824746.691829 trunc from 0x01127a to 0x015cf5 (0x4a7c bytes) 000574 1787824746.700544 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15cf6): Mirror 7: 0x24c42fe5 (stale) Mirror 9: 0x24c42fe5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15cf6): Mirror 7: 0x24c42fe5 (stale) Mirror 9: 0x24c42fe5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15cf6: skipped 000575 1787824746.968099 trunc from 0x015cf6 to 0x00d03f (0x8cb6 bytes) 000576 1787824746.985529 read 0x007562 thru 0x00a330 (0x2dcf bytes) 000577 1787824746.991301 mapwrite 0x011668 thru 0x0128fa (0x1293 bytes) 000578 1787824747.022764 trunc from 0x0128fb to 0x011c61 (0xc99 bytes) 000579 1787824747.040340 punch from 0x00d666 to 0x011c61 (0x45fc bytes) 000580 1787824747.046039 fallocate 0x03eaf5 thru 0x03ffff (0x150b bytes) 000581 1787824747.050742 punch from 0x026ba9 to 0x0277f3 (0xc4b bytes) 000582 1787824747.056197 trunc from 0x040000 to 0x006058 (0x39fa7 bytes) 000583 1787824747.077350 trunc from 0x004510 to 0x006058 (0x1b49 bytes) 000584 1787824747.082585 punch from 0x004daf to 0x006058 (0x12aa bytes) 000587 1787824747.091544 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x6059): Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6059): Mirror 7: 0x271e9519 (stale) Mirror 9: 0x271e9519 (stale) CRC-32 checksum value for chunk [0, 0x6059): Mirror 1: 0x271e9519 Mirror 10: 0x271e9519 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6059: skipped 000588 1787824747.713454 mapwrite 0x0324f2 thru 0x0334af (0xfbe bytes) 000590 1787824747.793564 read 0x0215a8 thru 0x027403 (0x5e5c bytes) 000593 1787824747.803157 read 0x022f9e thru 0x02a81b (0x787e bytes) 000594 1787824747.812513 mapread 0x01db8b thru 0x02269b (0x4b11 bytes) 000595 1787824747.813376 read 0x0063c0 thru 0x012a98 (0xc6d9 bytes) 000596 1787824747.814006 trunc from 0x023681 to 0x027f16 (0x4896 bytes) 000597 1787824747.818644 write 0x0295c1 thru 0x02ae09 (0x1849 bytes) 000598 1787824747.819744 read 0x02e512 thru 0x030962 (0x2451 bytes) 000599 1787824747.825369 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x334b0): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x334b0): Mirror 7: 0x137d21d9 (stale) Mirror 9: 0x137d21d9 (stale) Mirror 10: 0x137d21d9 (stale) CRC-32 checksum value for chunk [0, 0x334b0): Mirror 1: 0x92289118 Mirror 11: 0x92289118 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x334b0: skipped 000600 1787824748.606902 mapread 0x027e75 thru 0x0334af (0xb63b bytes) 000601 1787824748.627936 trunc from 0x01a02f to 0x02478b (0xa75d bytes) 000602 1787824748.676054 trunc from 0x014dd4 to 0x01be50 (0x707d bytes) 000603 1787824748.684278 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x334b0): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x334b0): Mirror 7: 0x137d21d9 (stale) Mirror 9: 0x137d21d9 (stale) Mirror 10: 0x137d21d9 (stale) Mirror 11: 0x92289118 (stale) CRC-32 checksum value for chunk [0, 0x334b0): Mirror 1: 0x92289118 Mirror 12: 0x92289118 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x334b0: skipped 000604 1787824749.370314 trunc from 0x0334b0 to 0x033e31 (0x982 bytes) 000605 1787824749.437812 mapwrite 0x0244e0 thru 0x028312 (0x3e33 bytes) 000606 1787824749.489833 fallocate 0x016438 thru 0x01930b (0x2ed4 bytes) 000607 1787824749.494784 mapread 0x001ddd thru 0x01137c (0xf5a0 bytes) 000608 1787824749.518996 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x33e32): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33e32): Mirror 7: 0x665827d9 (stale) Mirror 9: 0x665827d9 (stale) Mirror 10: 0x665827d9 (stale) Mirror 11: 0x9e2e5801 (stale) Mirror 12: 0x9e2e5801 (stale) CRC-32 checksum value for chunk [0, 0x33e32): Mirror 1: 0x746938d5 Mirror 13: 0x746938d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33e32: skipped 000609 1787824750.282013 read 0x021751 thru 0x02590f (0x41bf bytes) 000610 1787824750.288873 write 0x021f7b thru 0x03059b (0xe621 bytes) 000611 1787824750.337747 trunc from 0x022523 to 0x025d4c (0x382a bytes) 000612 1787824750.356078 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x33e32): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33e32): Mirror 7: 0x665827d9 (stale) Mirror 9: 0x665827d9 (stale) Mirror 10: 0x665827d9 (stale) Mirror 11: 0x9e2e5801 (stale) Mirror 12: 0x9e2e5801 (stale) Mirror 13: 0x746938d5 (stale) CRC-32 checksum value for chunk [0, 0x33e32): Mirror 1: 0xa0632729 Mirror 14: 0xa0632729 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33e32: skipped 000613 1787824751.134797 read 0x00218b thru 0x00550d (0x3383 bytes) 000614 1787824751.139324 punch from 0x01cd81 to 0x02ba40 (0xecc0 bytes) 000615 1787824751.182977 trunc from 0x033e32 to 0x03cede (0x90ad bytes) 000616 1787824751.204324 mapread 0x014c3b thru 0x014c3c (0x2 bytes) 000618 1787824751.210868 trunc from 0x03cedf to 0x00e236 (0x2eca8 bytes) 000619 1787824751.230886 trunc from 0x003e44 to 0x0080d7 (0x4294 bytes) 000620 1787824751.237911 write 0x029586 thru 0x02ba14 (0x248f bytes) 000621 1787824751.238983 fallocate 0x03377f thru 0x03508c (0x190e bytes) 000623 1787824751.253121 trunc from 0x03508d to 0x020515 (0x14b77 bytes) 000624 1787824751.266787 trunc from 0x014995 to 0x020515 (0xbb81 bytes) 000625 1787824751.273201 read 0x01f5e2 thru 0x020515 (0xf34 bytes) 000626 1787824751.284386 mapread 0x00fd7c thru 0x0164ed (0x6772 bytes) 000627 1787824751.286330 trunc from 0x020516 to 0x003902 (0x1cc13 bytes) 000628 1787824751.309301 read 0x000e8f thru 0x002c93 (0x1e05 bytes) 000629 1787824751.313620 mapread 0x000c83 thru 0x003902 (0x2c80 bytes) 000630 1787824751.315324 punch from 0x0010a9 to 0x003902 (0x285a bytes) 000631 1787824751.321344 mapwrite 0x010851 thru 0x01c8f0 (0xc0a0 bytes) 000632 1787824751.414216 mapwrite 0x005d22 thru 0x00d920 (0x7bff bytes) 000633 1787824751.448131 mapwrite 0x019a56 thru 0x01aed1 (0x147c bytes) 000634 1787824751.464384 mapwrite 0x032697 thru 0x03b7d6 (0x9140 bytes) 000635 1787824751.535929 punch from 0x02c4ce to 0x03a54f (0xe082 bytes) 000636 1787824751.546170 mapread 0x01cd9b thru 0x02b131 (0xe397 bytes) 000637 1787824751.549012 mapread 0x03a1c2 thru 0x03b7d6 (0x1615 bytes) 000638 1787824751.564312 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b7d7): Mirror 9: 0x8e0131cc (stale) Mirror 10: 0x8e0131cc (stale) Mirror 11: 0xb69326a7 (stale) Mirror 12: 0xb69326a7 (stale) Mirror 13: 0x826632db (stale) Mirror 14: 0xa61e118b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b7d7): Mirror 9: 0x8e0131cc (stale) Mirror 10: 0x8e0131cc (stale) Mirror 11: 0xb69326a7 (stale) Mirror 12: 0xb69326a7 (stale) Mirror 13: 0x826632db (stale) Mirror 14: 0xa61e118b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b7d7: skipped 000639 1787824752.045921 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3b7d7): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b7d7): Mirror 9: 0x8e0131cc (stale) Mirror 10: 0x8e0131cc (stale) Mirror 11: 0xb69326a7 (stale) Mirror 12: 0xb69326a7 (stale) Mirror 13: 0x826632db (stale) Mirror 14: 0xa61e118b (stale) CRC-32 checksum value for chunk [0, 0x3b7d7): Mirror 1: 0xd333118c Mirror 15: 0xd333118c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b7d7: skipped 000640 1787824753.058302 mapread 0x01f8aa thru 0x02e4ec (0xec43 bytes) 000641 1787824753.086361 write 0x00ee83 thru 0x01c3d2 (0xd550 bytes) 000643 1787824753.139347 punch from 0x03a133 to 0x03b7d6 (0x16a4 bytes) 000644 1787824753.143175 mapwrite 0x012115 thru 0x01d3f0 (0xb2dc bytes) 000645 1787824753.191110 trunc from 0x032ded to 0x033bd5 (0xde9 bytes) 000646 1787824753.206915 fallocate 0x036011 thru 0x03ffff (0x9fef bytes) 000647 1787824753.222318 trunc from 0x03b7d7 to 0x008e4b (0x3298b bytes) 000648 1787824753.245652 read 0x0020cd thru 0x008e4b (0x6d7f bytes) 000649 1787824753.262606 mapwrite 0x034994 thru 0x03ffff (0xb66c bytes) 000650 1787824753.336492 punch from 0x01c709 to 0x023a7b (0x7373 bytes) 000651 1787824753.341830 write 0x01e011 thru 0x0201fa (0x21ea bytes) 000652 1787824753.351723 fallocate 0x03896f thru 0x03ffff (0x7691 bytes) 000653 1787824753.359268 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xbcd3347d (stale) Mirror 10: 0xbcd3347d (stale) Mirror 11: 0x1becf64c (stale) Mirror 12: 0x1becf64c (stale) Mirror 13: 0x22a2a797 (stale) Mirror 14: 0x68fc5184 (stale) Mirror 15: 0x52836150 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x71d6a950 Mirror 16: 0x71d6a950 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000654 1787824754.744237 read 0x01270c thru 0x017f66 (0x585b bytes) 000655 1787824754.772032 mapread 0x01f6f5 thru 0x02867f (0x8f8b bytes) 000656 1787824754.808365 write 0x013d08 thru 0x01c372 (0x866b bytes) 000657 1787824754.879039 fallocate 0x01321a thru 0x01ed8c (0xbb73 bytes) 000660 1787824754.887458 fallocate 0x029b3c thru 0x02c0c8 (0x258d bytes) 000661 1787824754.894135 write 0x03389e thru 0x0391de (0x5941 bytes) 000662 1787824754.906032 punch from 0x01606b to 0x0200b7 (0xa04d bytes) 000663 1787824754.921864 trunc from 0x032f96 to 0x03ecec (0xbd57 bytes) 000666 1787824754.930685 trunc from 0x00bf98 to 0x013b06 (0x7b6f bytes) 000667 1787824754.939216 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xbcd3347d (stale) Mirror 10: 0xbcd3347d (stale) Mirror 12: 0x1becf64c (stale) Mirror 13: 0x22a2a797 (stale) Mirror 14: 0x68fc5184 (stale) Mirror 15: 0x52836150 (stale) Mirror 16: 0x71d6a950 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xbcd3347d (stale) Mirror 10: 0xbcd3347d (stale) Mirror 12: 0x1becf64c (stale) Mirror 13: 0x22a2a797 (stale) Mirror 14: 0x68fc5184 (stale) Mirror 15: 0x52836150 (stale) Mirror 16: 0x71d6a950 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000668 1787824755.691448 punch from 0x027c44 to 0x0363f1 (0xe7ae bytes) 000669 1787824755.699312 mapwrite 0x0168a7 thru 0x025ca0 (0xf3fa bytes) 000671 1787824755.824162 trunc from 0x025bd3 to 0x02f5f8 (0x9a26 bytes) 000672 1787824755.849358 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xbcd3347d (stale) Mirror 10: 0xbcd3347d (stale) Mirror 12: 0x1becf64c (stale) Mirror 13: 0x22a2a797 (stale) Mirror 14: 0x68fc5184 (stale) Mirror 15: 0x52836150 (stale) Mirror 16: 0x71d6a950 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb8c4b354 Mirror 17: 0xb8c4b354 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000674 1787824757.186288 punch from 0x02a2fd to 0x03579f (0xb4a3 bytes) 000675 1787824757.242279 fallocate 0x028d04 thru 0x02e806 (0x5b03 bytes) 000676 1787824757.248080 read 0x02e450 thru 0x0303d2 (0x1f83 bytes) 000677 1787824757.257010 mapread 0x009726 thru 0x01441a (0xacf5 bytes) 000678 1787824757.293765 mapwrite 0x038f90 thru 0x03a181 (0x11f2 bytes) 000679 1787824757.326280 write 0x01143c thru 0x01ae49 (0x9a0e bytes) 000680 1787824757.341429 write 0x00c32e thru 0x018a85 (0xc758 bytes) 000681 1787824757.352254 fallocate 0x002d11 thru 0x0050e5 (0x23d5 bytes) 000682 1787824757.380498 punch from 0x025f4f to 0x0282e0 (0x2392 bytes) 000683 1787824757.384248 trunc from 0x040000 to 0x02fd83 (0x1027c bytes) 000684 1787824757.448985 mapwrite 0x029a95 thru 0x039273 (0xf7df bytes) 000685 1787824757.575385 mapwrite 0x01a4c8 thru 0x020830 (0x6369 bytes) 000686 1787824757.626538 write 0x033c6d thru 0x0397fb (0x5b8f bytes) 000687 1787824757.630885 write 0x01266f thru 0x017a32 (0x53c4 bytes) 000688 1787824757.636341 read 0x0216df thru 0x02fb7c (0xe49e bytes) 000689 1787824757.637222 trunc from 0x023ca1 to 0x02f07d (0xb3dd bytes) 000690 1787824757.645709 fallocate 0x009d40 thru 0x00cedc (0x319d bytes) 000691 1787824757.665155 mapread 0x0314a1 thru 0x0397fb (0x835b bytes) 000692 1787824757.666618 mapwrite 0x00fd5f thru 0x01b386 (0xb628 bytes) 000693 1787824757.704622 fallocate 0x03d2dc thru 0x03f6e6 (0x240b bytes) 000694 1787824757.715281 read 0x027287 thru 0x0292e8 (0x2062 bytes) 000695 1787824757.724866 trunc from 0x013704 to 0x021bd1 (0xe4ce bytes) 000696 1787824757.733031 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x397fc): Mirror 9: 0x1e0f385a (stale) Mirror 10: 0x1e0f385a (stale) Mirror 12: 0x64b37ee6 (stale) Mirror 13: 0x9b5c359f (stale) Mirror 14: 0x45fa188a (stale) Mirror 15: 0xc67bfea5 (stale) Mirror 16: 0x88e305b5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x397fc): Mirror 9: 0x1e0f385a (stale) Mirror 10: 0x1e0f385a (stale) Mirror 12: 0x64b37ee6 (stale) Mirror 13: 0x9b5c359f (stale) Mirror 14: 0x45fa188a (stale) Mirror 15: 0xc67bfea5 (stale) Mirror 16: 0x88e305b5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x397fc: skipped 000697 1787824758.106695 mapwrite 0x035e5a thru 0x038ebb (0x3062 bytes) 000698 1787824758.141029 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x397fc): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x397fc): Mirror 9: 0x1e0f385a (stale) Mirror 10: 0x1e0f385a (stale) Mirror 12: 0x64b37ee6 (stale) Mirror 13: 0x9b5c359f (stale) Mirror 14: 0x45fa188a (stale) Mirror 15: 0xc67bfea5 (stale) Mirror 16: 0x88e305b5 (stale) CRC-32 checksum value for chunk [0, 0x397fc): Mirror 1: 0xf25c7d9c Mirror 17: 0xf25c7d9c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x397fc: skipped 000699 1787824759.165130 punch from 0x029651 to 0x02eb23 (0x54d3 bytes) 000700 1787824759.222806 MIRROR_RESYNC 9 mirrors 000702 1787824762.929871 fallocate 0x007d48 thru 0x0091bd (0x1476 bytes) 000704 1787824762.971776 write 0x00ff9a thru 0x018071 (0x80d8 bytes) 000705 1787824762.982696 mapwrite 0x013cca thru 0x020496 (0xc7cd bytes) 000706 1787824763.034181 trunc from 0x00a61d to 0x012486 (0x7e6a bytes) 000707 1787824763.040326 fallocate 0x0149ef thru 0x016656 (0x1c68 bytes) 000708 1787824763.049293 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x397fc): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x397fc): Mirror 9: 0xf25c7d9c (stale) Mirror 10: 0xf25c7d9c (stale) Mirror 12: 0xf25c7d9c (stale) Mirror 13: 0xf25c7d9c (stale) Mirror 14: 0xf25c7d9c (stale) Mirror 15: 0xf25c7d9c (stale) Mirror 16: 0xf25c7d9c (stale) Mirror 17: 0xf25c7d9c (stale) CRC-32 checksum value for chunk [0, 0x397fc): Mirror 1: 0xf7f6e528 Mirror 18: 0xf7f6e528 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x397fc: skipped 000709 1787824764.437744 trunc from 0x0397fc to 0x00aa48 (0x2edb3 bytes) 000710 1787824764.497924 fallocate 0x007dc2 thru 0x008a7f (0xcbe bytes) 000711 1787824764.504535 mapwrite 0x024126 thru 0x0323b5 (0xe290 bytes) 000713 1787824764.653306 punch from 0x0152b9 to 0x015404 (0x14c bytes) 000714 1787824764.658967 fallocate 0x0202f1 thru 0x02a53d (0xa24d bytes) 000715 1787824764.665611 trunc from 0x02957f to 0x0323b5 (0x8e37 bytes) 000716 1787824764.671106 mapread 0x02e291 thru 0x0323b5 (0x4125 bytes) 000717 1787824764.680087 trunc from 0x0323b6 to 0x02195d (0x10a58 bytes) 000718 1787824764.696368 trunc from 0x02195e to 0x015d86 (0xbbd7 bytes) 000719 1787824764.718986 write 0x01d639 thru 0x022bc3 (0x558b bytes) 000720 1787824764.721651 fallocate 0x01155e thru 0x013ee3 (0x2986 bytes) 000723 1787824764.737060 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x22bc4): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22bc4): Mirror 9: 0x48dbd942 (stale) Mirror 10: 0x48dbd942 (stale) Mirror 12: 0x48dbd942 (stale) Mirror 13: 0x48dbd942 (stale) Mirror 14: 0x48dbd942 (stale) Mirror 15: 0x48dbd942 (stale) Mirror 16: 0x48dbd942 (stale) Mirror 17: 0x48dbd942 (stale) Mirror 18: 0x76d1ad8e (stale) CRC-32 checksum value for chunk [0, 0x22bc4): Mirror 1: 0x5a9a4525 Mirror 19: 0x5a9a4525 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22bc4: skipped 000724 1787824765.979129 read 0x004b5b thru 0x011cc6 (0xd16c bytes) 000725 1787824765.987610 trunc from 0x022bc4 to 0x012420 (0x107a3 bytes) 000726 1787824766.042241 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x12421): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12421): Mirror 9: 0x6ba65391 (stale) Mirror 10: 0x6ba65391 (stale) Mirror 12: 0x6ba65391 (stale) Mirror 13: 0x6ba65391 (stale) Mirror 14: 0x6ba65391 (stale) Mirror 15: 0x6ba65391 (stale) Mirror 16: 0x6ba65391 (stale) Mirror 17: 0x6ba65391 (stale) Mirror 18: 0x7cf56452 (stale) Mirror 19: 0x7cf56452 (stale) CRC-32 checksum value for chunk [0, 0x12421): Mirror 1: 0x7cf56452 Mirror 20: 0x7cf56452 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12421: skipped 000727 1787824766.789262 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x12421): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12421): Mirror 9: 0x6ba65391 (stale) Mirror 10: 0x6ba65391 (stale) Mirror 12: 0x6ba65391 (stale) Mirror 13: 0x6ba65391 (stale) Mirror 14: 0x6ba65391 (stale) Mirror 15: 0x6ba65391 (stale) Mirror 16: 0x6ba65391 (stale) Mirror 17: 0x6ba65391 (stale) Mirror 18: 0x7cf56452 (stale) Mirror 19: 0x7cf56452 (stale) CRC-32 checksum value for chunk [0, 0x12421): Mirror 1: 0x7cf56452 Mirror 20: 0x7cf56452 Mirror 21: 0x7cf56452 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12421: skipped 000728 1787824767.672786 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x12421): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12421): Mirror 9: 0x6ba65391 (stale) Mirror 10: 0x6ba65391 (stale) Mirror 12: 0x6ba65391 (stale) Mirror 13: 0x6ba65391 (stale) Mirror 14: 0x6ba65391 (stale) Mirror 15: 0x6ba65391 (stale) Mirror 16: 0x6ba65391 (stale) Mirror 17: 0x6ba65391 (stale) Mirror 18: 0x7cf56452 (stale) Mirror 19: 0x7cf56452 (stale) CRC-32 checksum value for chunk [0, 0x12421): Mirror 1: 0x7cf56452 Mirror 20: 0x7cf56452 Mirror 21: 0x7cf56452 Mirror 22: 0x7cf56452 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12421: skipped 000729 1787824768.343294 write 0x02e936 thru 0x03d00f (0xe6da bytes) 000730 1787824768.388170 trunc from 0x02b231 to 0x034fff (0x9dcf bytes) 000731 1787824768.403130 write 0x00fbd1 thru 0x01ef2c (0xf35c bytes) 000732 1787824768.411694 write 0x00a39e thru 0x015245 (0xaea8 bytes) 000733 1787824768.417293 trunc from 0x01b46f to 0x01ceb7 (0x1a49 bytes) 000734 1787824768.429234 trunc from 0x03d010 to 0x03a3f4 (0x2c1b bytes) 000735 1787824768.447794 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3a3f5): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a3f5): Mirror 9: 0x15fdf14c (stale) Mirror 10: 0x15fdf14c (stale) Mirror 12: 0x15fdf14c (stale) Mirror 13: 0x15fdf14c (stale) Mirror 14: 0x15fdf14c (stale) Mirror 15: 0x15fdf14c (stale) Mirror 16: 0x15fdf14c (stale) Mirror 17: 0x15fdf14c (stale) Mirror 18: 0x15ee3e8c (stale) Mirror 19: 0x9815723c (stale) Mirror 20: 0x32bab384 (stale) Mirror 21: 0x32bab384 (stale) Mirror 22: 0x32bab384 (stale) CRC-32 checksum value for chunk [0, 0x3a3f5): Mirror 1: 0xe57841fd Mirror 23: 0xe57841fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a3f5: skipped 000736 1787824769.460115 trunc from 0x03a3f5 to 0x01070e (0x29ce6 bytes) 000737 1787824769.515090 read 0x005355 thru 0x009b9a (0x4846 bytes) 000738 1787824769.521043 trunc from 0x00e70d to 0x01070e (0x2002 bytes) 000739 1787824769.524593 fallocate 0x0173f5 thru 0x018ae9 (0x16f5 bytes) 000740 1787824769.528876 trunc from 0x01070f to 0x02667a (0x15f6c bytes) 000741 1787824769.548378 read 0x025b27 thru 0x02667a (0xb54 bytes) 000742 1787824769.557386 fallocate 0x03403f thru 0x03ffff (0xbfc1 bytes) 000743 1787824769.561315 trunc from 0x00f338 to 0x01b0a2 (0xbd6b bytes) 000744 1787824769.565644 mapread 0x000aff thru 0x00722f (0x6731 bytes) 000745 1787824769.566865 read 0x00f6e2 thru 0x01a97c (0xb29b bytes) 000746 1787824769.574286 mapread 0x001701 thru 0x0109cf (0xf2cf bytes) 000747 1787824769.576072 fallocate 0x004396 thru 0x0121c5 (0xde30 bytes) 000748 1787824769.580101 trunc from 0x040000 to 0x003826 (0x3c7d9 bytes) 000750 1787824769.597139 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3827): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3827): Mirror 9: 0x29913e7d (stale) Mirror 10: 0x29913e7d (stale) Mirror 12: 0x29913e7d (stale) Mirror 13: 0x29913e7d (stale) Mirror 14: 0x29913e7d (stale) Mirror 15: 0x29913e7d (stale) Mirror 16: 0x29913e7d (stale) Mirror 17: 0x29913e7d (stale) Mirror 18: 0x29913e7d (stale) Mirror 19: 0x29913e7d (stale) Mirror 20: 0x29913e7d (stale) Mirror 21: 0x29913e7d (stale) Mirror 22: 0x29913e7d (stale) Mirror 23: 0x29913e7d (stale) CRC-32 checksum value for chunk [0, 0x3827): Mirror 1: 0x29913e7d Mirror 24: 0x29913e7d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3827: skipped 000751 1787824770.202037 mapread 0x001db2 thru 0x003826 (0x1a75 bytes) 000752 1787824770.217379 trunc from 0x0028a0 to 0x003826 (0xf87 bytes) 000754 1787824770.258969 fallocate 0x022492 thru 0x02696b (0x44da bytes) 000755 1787824770.263895 mapwrite 0x03ec5c thru 0x03ffff (0x13a4 bytes) 000756 1787824770.314688 trunc from 0x014636 to 0x016b89 (0x2554 bytes) 000757 1787824770.322288 MIRROR_SPLIT mirror 15 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x3b044eb1 (stale) Mirror 10: 0x3b044eb1 (stale) Mirror 12: 0x3b044eb1 (stale) Mirror 13: 0x3b044eb1 (stale) Mirror 14: 0x3b044eb1 (stale) Mirror 15: 0x3b044eb1 (stale) Mirror 16: 0x3b044eb1 (stale) Mirror 17: 0x3b044eb1 (stale) Mirror 18: 0x582b403 (stale) Mirror 19: 0x4346e730 (stale) Mirror 20: 0xc18d4d70 (stale) Mirror 21: 0xc18d4d70 (stale) Mirror 22: 0xc18d4d70 (stale) Mirror 23: 0x99a160e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x3b044eb1 (stale) Mirror 10: 0x3b044eb1 (stale) Mirror 12: 0x3b044eb1 (stale) Mirror 13: 0x3b044eb1 (stale) Mirror 14: 0x3b044eb1 (stale) Mirror 15: 0x3b044eb1 (stale) Mirror 16: 0x3b044eb1 (stale) Mirror 17: 0x3b044eb1 (stale) Mirror 18: 0x582b403 (stale) Mirror 19: 0x4346e730 (stale) Mirror 20: 0xc18d4d70 (stale) Mirror 21: 0xc18d4d70 (stale) Mirror 22: 0xc18d4d70 (stale) Mirror 23: 0x99a160e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000758 1787824770.963514 trunc from 0x040000 to 0x019a8a (0x26575 bytes) 000759 1787824770.979796 punch from 0x002a76 to 0x00b3b9 (0x8944 bytes) 000760 1787824770.983016 trunc from 0x019a8b to 0x014e2e (0x4c5c bytes) 000761 1787824770.993075 trunc from 0x014e2f to 0x02594d (0x10b1f bytes) 000762 1787824771.006105 punch from 0x006588 to 0x0095c0 (0x3039 bytes) 000763 1787824771.009750 read 0x020e7a thru 0x02594d (0x4ad4 bytes) 000764 1787824771.016002 trunc from 0x02594e to 0x010824 (0x15129 bytes) 000765 1787824771.025047 mapwrite 0x009f6e thru 0x0173b6 (0xd449 bytes) 000766 1787824771.065696 mapread 0x00dee5 thru 0x01110a (0x3226 bytes) 000769 1787824771.067212 write 0x00e5e3 thru 0x0162a9 (0x7cc7 bytes) 000770 1787824771.068644 fallocate 0x006b47 thru 0x00f130 (0x85ea bytes) 000771 1787824771.072739 trunc from 0x0173b7 to 0x01d1d9 (0x5e23 bytes) 000772 1787824771.081773 trunc from 0x01d1da to 0x009db4 (0x13425 bytes) 000773 1787824771.091583 fallocate 0x03c9c8 thru 0x03ffff (0x3638 bytes) 000774 1787824771.095376 trunc from 0x009db5 to 0x0090ac (0xd08 bytes) 000775 1787824771.108824 fallocate 0x005598 thru 0x00b4c1 (0x5f2a bytes) 000776 1787824771.112603 trunc from 0x0090ad to 0x02d53e (0x24492 bytes) 000777 1787824771.123733 punch from 0x002bce to 0x007f60 (0x5393 bytes) 000778 1787824771.127142 mapwrite 0x00088b thru 0x000998 (0x10e bytes) 000779 1787824771.140889 fallocate 0x00c1be thru 0x010176 (0x3fb9 bytes) 000780 1787824771.144609 fallocate 0x0396f8 thru 0x03ffff (0x6908 bytes) 000781 1787824771.148059 trunc from 0x006c4d to 0x015efa (0xf2ae bytes) 000784 1787824771.152123 mapread 0x018a61 thru 0x020ad9 (0x8079 bytes) 000785 1787824771.157977 write 0x03b7c1 thru 0x03c6db (0xf1b bytes) 000786 1787824771.159055 trunc from 0x022ca2 to 0x026704 (0x3a63 bytes) 000787 1787824771.162516 trunc from 0x03c6dc to 0x02871d (0x13fbe bytes) 000788 1787824771.172699 punch from 0x01cb34 to 0x02221e (0x56eb bytes) 000789 1787824771.176625 trunc from 0x01254b to 0x021020 (0xead6 bytes) 000790 1787824771.179770 trunc from 0x0235aa to 0x02871d (0x5174 bytes) 000791 1787824771.185421 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x2871e): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2871e): Mirror 9: 0xe9db4d32 (stale) Mirror 10: 0xe9db4d32 (stale) Mirror 12: 0xe9db4d32 (stale) Mirror 13: 0xe9db4d32 (stale) Mirror 14: 0xe9db4d32 (stale) Mirror 15: 0xe9db4d32 (stale) Mirror 16: 0xe9db4d32 (stale) Mirror 17: 0xe9db4d32 (stale) Mirror 18: 0x5b83c5fd (stale) Mirror 19: 0x30bfca8a (stale) Mirror 20: 0xec00015a (stale) Mirror 21: 0xec00015a (stale) Mirror 22: 0xec00015a (stale) Mirror 23: 0xba317c06 (stale) CRC-32 checksum value for chunk [0, 0x2871e): Mirror 1: 0xe3815f5b Mirror 24: 0xe3815f5b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2871e: skipped 000794 1787824771.790670 write 0x02fe85 thru 0x039a4b (0x9bc7 bytes) 000795 1787824771.824545 MIRROR_RESYNC 16 mirrors 000797 1787824773.562379 mapwrite 0x02128a thru 0x02c596 (0xb30d bytes) 000798 1787824773.671784 trunc from 0x005c77 to 0x00f522 (0x98ac bytes) 000799 1787824773.687923 MIRROR_SPLIT mirror 9 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39a4c): Mirror 9: 0x5706c9f9 (stale) Mirror 10: 0x5706c9f9 (stale) Mirror 12: 0x5706c9f9 (stale) Mirror 13: 0x5706c9f9 (stale) Mirror 14: 0x5706c9f9 (stale) Mirror 15: 0x5706c9f9 (stale) Mirror 16: 0x5706c9f9 (stale) Mirror 17: 0x5706c9f9 (stale) Mirror 19: 0x5706c9f9 (stale) Mirror 20: 0x5706c9f9 (stale) Mirror 21: 0x5706c9f9 (stale) Mirror 22: 0x5706c9f9 (stale) Mirror 23: 0x5706c9f9 (stale) Mirror 24: 0x5706c9f9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a4c): Mirror 9: 0x5706c9f9 (stale) Mirror 10: 0x5706c9f9 (stale) Mirror 12: 0x5706c9f9 (stale) Mirror 13: 0x5706c9f9 (stale) Mirror 14: 0x5706c9f9 (stale) Mirror 15: 0x5706c9f9 (stale) Mirror 16: 0x5706c9f9 (stale) Mirror 17: 0x5706c9f9 (stale) Mirror 19: 0x5706c9f9 (stale) Mirror 20: 0x5706c9f9 (stale) Mirror 21: 0x5706c9f9 (stale) Mirror 22: 0x5706c9f9 (stale) Mirror 23: 0x5706c9f9 (stale) Mirror 24: 0x5706c9f9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a4c: skipped 000801 1787824774.354574 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x39a4c): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39a4c): Mirror 9: 0x5706c9f9 (stale) Mirror 10: 0x5706c9f9 (stale) Mirror 12: 0x5706c9f9 (stale) Mirror 13: 0x5706c9f9 (stale) Mirror 14: 0x5706c9f9 (stale) Mirror 15: 0x5706c9f9 (stale) Mirror 16: 0x5706c9f9 (stale) Mirror 17: 0x5706c9f9 (stale) Mirror 19: 0x5706c9f9 (stale) Mirror 20: 0x5706c9f9 (stale) Mirror 21: 0x5706c9f9 (stale) Mirror 22: 0x5706c9f9 (stale) Mirror 23: 0x5706c9f9 (stale) Mirror 24: 0x5706c9f9 (stale) CRC-32 checksum value for chunk [0, 0x39a4c): Mirror 1: 0x1af20111 Mirror 25: 0x1af20111 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39a4c: skipped 000802 1787824775.388047 mapwrite 0x01d2d1 thru 0x022d65 (0x5a95 bytes) 000803 1787824775.469992 write 0x02f911 thru 0x0384a8 (0x8b98 bytes) 000804 1787824775.485584 mapread 0x039761 thru 0x039a4b (0x2eb bytes) 000805 1787824775.491708 write 0x013d42 thru 0x01c1de (0x849d bytes) 000807 1787824775.500076 write 0x01c012 thru 0x0256b3 (0x96a2 bytes) 000810 1787824775.505598 trunc from 0x039a4c to 0x014443 (0x25608 bytes) 000811 1787824775.527790 fallocate 0x0016ca thru 0x00dd5f (0xc696 bytes) 000812 1787824775.531441 fallocate 0x00656c thru 0x009ec2 (0x3957 bytes) 000814 1787824775.537798 read 0x00a134 thru 0x014443 (0xa310 bytes) 000815 1787824775.543984 punch from 0x00e617 to 0x014443 (0x5e2d bytes) 000816 1787824775.554962 mapwrite 0x020510 thru 0x0259fa (0x54eb bytes) 000818 1787824775.597735 fallocate 0x005d15 thru 0x0147b2 (0xea9e bytes) 000819 1787824775.602372 write 0x01ee88 thru 0x0237a0 (0x4919 bytes) 000820 1787824775.603437 punch from 0x007b16 to 0x016e31 (0xf31c bytes) 000821 1787824775.609708 MIRROR_SPLIT mirror 13 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x259fb): Mirror 9: 0x8f6f0949 (stale) Mirror 10: 0x8f6f0949 (stale) Mirror 12: 0x8f6f0949 (stale) Mirror 13: 0x8f6f0949 (stale) Mirror 14: 0x8f6f0949 (stale) Mirror 15: 0x8f6f0949 (stale) Mirror 16: 0x8f6f0949 (stale) Mirror 17: 0x8f6f0949 (stale) Mirror 19: 0x8f6f0949 (stale) Mirror 20: 0x8f6f0949 (stale) Mirror 21: 0x8f6f0949 (stale) Mirror 22: 0x8f6f0949 (stale) Mirror 24: 0x8f6f0949 (stale) Mirror 25: 0xcd8df51 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x259fb): Mirror 9: 0x8f6f0949 (stale) Mirror 10: 0x8f6f0949 (stale) Mirror 12: 0x8f6f0949 (stale) Mirror 13: 0x8f6f0949 (stale) Mirror 14: 0x8f6f0949 (stale) Mirror 15: 0x8f6f0949 (stale) Mirror 16: 0x8f6f0949 (stale) Mirror 17: 0x8f6f0949 (stale) Mirror 19: 0x8f6f0949 (stale) Mirror 20: 0x8f6f0949 (stale) Mirror 21: 0x8f6f0949 (stale) Mirror 22: 0x8f6f0949 (stale) Mirror 24: 0x8f6f0949 (stale) Mirror 25: 0xcd8df51 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x259fb: skipped 000822 1787824776.345559 mapread 0x005c1a thru 0x00b6b3 (0x5a9a bytes) 000823 1787824776.382527 MIRROR_RESYNC 15 mirrors 000824 1787824778.568752 trunc from 0x0259fb to 0x01759e (0xe45c bytes) 000825 1787824778.656633 trunc from 0x017545 to 0x01759e (0x5a bytes) 000826 1787824778.668684 mapwrite 0x028673 thru 0x02a559 (0x1ee7 bytes) 000827 1787824778.766685 write 0x031cb3 thru 0x03a237 (0x8585 bytes) 000828 1787824778.771741 mapwrite 0x0017ec thru 0x003f2a (0x273f bytes) 000829 1787824778.829062 trunc from 0x03a238 to 0x02095e (0x198d9 bytes) 000830 1787824778.848079 trunc from 0x02095f to 0x03d68a (0x1cd2c bytes) 000831 1787824778.887662 mapwrite 0x00c0d2 thru 0x011b3d (0x5a6c bytes) 000832 1787824778.918954 write 0x02772e thru 0x03493c (0xd20f bytes) 000833 1787824778.931102 mapwrite 0x03659b thru 0x03fb90 (0x95f6 bytes) 000834 1787824778.994787 write 0x0316a2 thru 0x0316ab (0xa bytes) 000835 1787824778.996552 mapwrite 0x0226cb thru 0x02443d (0x1d73 bytes) 000836 1787824779.022090 write 0x002d56 thru 0x005c08 (0x2eb3 bytes) 000837 1787824779.023091 punch from 0x0280a4 to 0x036f19 (0xee76 bytes) 000838 1787824779.029099 trunc from 0x03fb91 to 0x0076b3 (0x384dd bytes) 000840 1787824779.047539 fallocate 0x005b5b thru 0x00bbba (0x6060 bytes) 000841 1787824779.052763 mapread 0x007fd3 thru 0x00bbba (0x3be8 bytes) 000843 1787824779.059362 punch from 0x00ba11 to 0x00bbba (0x1aa bytes) 000845 1787824779.067816 mapread 0x003119 thru 0x00a75e (0x7646 bytes) 000846 1787824779.069354 read 0x006946 thru 0x00b411 (0x4acc bytes) 000847 1787824779.074224 mapwrite 0x01487d thru 0x022714 (0xde98 bytes) 000848 1787824779.144733 mapread 0x00c18d thru 0x0115c7 (0x543b bytes) 000849 1787824779.153550 trunc from 0x0159b2 to 0x022714 (0xcd63 bytes) 000850 1787824779.158535 mapwrite 0x001918 thru 0x00a0cb (0x87b4 bytes) 000851 1787824779.177623 trunc from 0x022715 to 0x03816b (0x15a57 bytes) 000852 1787824779.192884 mapread 0x02d778 thru 0x0325c5 (0x4e4e bytes) 000854 1787824779.204787 trunc from 0x03816c to 0x02c82f (0xb93c bytes) 000855 1787824779.221798 fallocate 0x01cbf6 thru 0x02461d (0x7a28 bytes) 000856 1787824779.228112 read 0x0130ad thru 0x022e74 (0xfdc8 bytes) 000857 1787824779.238617 punch from 0x0101ec to 0x014ea3 (0x4cb8 bytes) 000858 1787824779.243836 mapread 0x005673 thru 0x00b49f (0x5e2d bytes) 000859 1787824779.244964 fallocate 0x01522e thru 0x01f365 (0xa138 bytes) 000860 1787824779.249699 trunc from 0x026c0c to 0x02c82f (0x5c24 bytes) 000861 1787824779.255053 write 0x0274dc thru 0x02dde0 (0x6905 bytes) 000862 1787824779.260127 fallocate 0x02c9fe thru 0x033e34 (0x7437 bytes) 000863 1787824779.268954 MIRROR_SPLIT mirror 2 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dde1): Mirror 9: 0xb1c257f6 (stale) Mirror 12: 0xb1c257f6 (stale) Mirror 13: 0xb1c257f6 (stale) Mirror 14: 0xb1c257f6 (stale) Mirror 15: 0xb1c257f6 (stale) Mirror 16: 0xb1c257f6 (stale) Mirror 17: 0xb1c257f6 (stale) Mirror 19: 0xb1c257f6 (stale) Mirror 20: 0xb1c257f6 (stale) Mirror 21: 0xb1c257f6 (stale) Mirror 22: 0xb1c257f6 (stale) Mirror 24: 0xb1c257f6 (stale) Mirror 25: 0xb1c257f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dde1): Mirror 9: 0xb1c257f6 (stale) Mirror 12: 0xb1c257f6 (stale) Mirror 13: 0xb1c257f6 (stale) Mirror 14: 0xb1c257f6 (stale) Mirror 15: 0xb1c257f6 (stale) Mirror 16: 0xb1c257f6 (stale) Mirror 17: 0xb1c257f6 (stale) Mirror 19: 0xb1c257f6 (stale) Mirror 20: 0xb1c257f6 (stale) Mirror 21: 0xb1c257f6 (stale) Mirror 22: 0xb1c257f6 (stale) Mirror 24: 0xb1c257f6 (stale) Mirror 25: 0xb1c257f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dde1: skipped 000864 1787824779.922788 mapread 0x01a5aa thru 0x01bf8f (0x19e6 bytes) 000865 1787824779.936727 write 0x003ef2 thru 0x0085bb (0x46ca bytes) 000866 1787824779.951783 write 0x00ee78 thru 0x01e1b2 (0xf33b bytes) 000867 1787824779.963527 trunc from 0x008622 to 0x00d9dc (0x53bb bytes) 000869 1787824779.986051 mapread 0x014a51 thru 0x02459c (0xfb4c bytes) 000870 1787824779.997357 fallocate 0x03c7c3 thru 0x03ffff (0x383d bytes) 000871 1787824780.008177 trunc from 0x02aafa to 0x038324 (0xd82b bytes) 000873 1787824780.013746 trunc from 0x040000 to 0x01cf03 (0x230fc bytes) 000874 1787824780.032438 trunc from 0x01cf04 to 0x03ba45 (0x1eb42 bytes) 000875 1787824780.051363 read 0x02a258 thru 0x03a06c (0xfe15 bytes) 000876 1787824780.066562 punch from 0x028a51 to 0x030f74 (0x8524 bytes) 000877 1787824780.072498 trunc from 0x02a10a to 0x0370f2 (0xcfe9 bytes) 000878 1787824780.078782 mapwrite 0x0319ee thru 0x03e814 (0xce27 bytes) 000879 1787824780.146663 fallocate 0x0330a9 thru 0x03f7ec (0xc744 bytes) 000880 1787824780.152630 punch from 0x037192 to 0x03f7ec (0x865b bytes) 000881 1787824780.159648 punch from 0x00b3d2 to 0x0141a9 (0x8dd8 bytes) 000882 1787824780.165613 punch from 0x01fe31 to 0x021713 (0x18e3 bytes) 000884 1787824780.171696 write 0x0052c6 thru 0x00a134 (0x4e6f bytes) 000885 1787824780.178046 punch from 0x00dd4c to 0x01043f (0x26f4 bytes) 000886 1787824780.183036 read 0x00988b thru 0x00efe2 (0x5758 bytes) 000887 1787824780.194518 mapread 0x01f301 thru 0x02ee89 (0xfb89 bytes) 000888 1787824780.195549 trunc from 0x00fe6e to 0x01840c (0x859f bytes) 000889 1787824780.202185 fallocate 0x00f55c thru 0x012752 (0x31f7 bytes) 000890 1787824780.209654 trunc from 0x03f7ed to 0x03f5a5 (0x247 bytes) 000891 1787824780.238573 mapwrite 0x01e867 thru 0x020f14 (0x26ae bytes) 000895 1787824780.263521 punch from 0x00694c to 0x0092f7 (0x29ac bytes) 000897 1787824780.273998 MIRROR_RESYNC 14 mirrors 000898 1787824785.348382 write 0x00a1f6 thru 0x015906 (0xb711 bytes) 000899 1787824785.421089 write 0x02a9db thru 0x03a681 (0xfca7 bytes) 000900 1787824785.438299 trunc from 0x017954 to 0x020ba6 (0x9253 bytes) 000901 1787824785.444747 mapwrite 0x00f89f thru 0x01346b (0x3bcd bytes) 000902 1787824785.463865 punch from 0x019f9c to 0x025e2d (0xbe92 bytes) 000903 1787824785.468869 mapread 0x0225cb thru 0x02bc14 (0x964a bytes) 000905 1787824785.489419 read 0x032c04 thru 0x03e4f9 (0xb8f6 bytes) 000906 1787824785.495944 write 0x0368f1 thru 0x037d15 (0x1425 bytes) 000907 1787824785.496879 write 0x03a858 thru 0x03bcc7 (0x1470 bytes) 000908 1787824785.497719 trunc from 0x01d142 to 0x021dbf (0x4c7e bytes) 000909 1787824785.503112 fallocate 0x01f6cd thru 0x021f0e (0x2842 bytes) 000910 1787824785.511947 MIRROR_SPLIT mirror 6 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f5a6): Mirror 9: 0x4e59d219 (stale) Mirror 12: 0x4e59d219 (stale) Mirror 13: 0x4e59d219 (stale) Mirror 14: 0x4e59d219 (stale) Mirror 15: 0x4e59d219 (stale) Mirror 17: 0x4e59d219 (stale) Mirror 19: 0x4e59d219 (stale) Mirror 20: 0x4e59d219 (stale) Mirror 21: 0x4e59d219 (stale) Mirror 22: 0x4e59d219 (stale) Mirror 24: 0x4e59d219 (stale) Mirror 25: 0x4e59d219 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f5a6): Mirror 9: 0x4e59d219 (stale) Mirror 12: 0x4e59d219 (stale) Mirror 13: 0x4e59d219 (stale) Mirror 14: 0x4e59d219 (stale) Mirror 15: 0x4e59d219 (stale) Mirror 17: 0x4e59d219 (stale) Mirror 19: 0x4e59d219 (stale) Mirror 20: 0x4e59d219 (stale) Mirror 21: 0x4e59d219 (stale) Mirror 22: 0x4e59d219 (stale) Mirror 24: 0x4e59d219 (stale) Mirror 25: 0x4e59d219 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f5a6: skipped 000911 1787824786.291357 read 0x017cb7 thru 0x0222dc (0xa626 bytes) 000912 1787824786.309462 write 0x026dc1 thru 0x033c88 (0xcec8 bytes) 000913 1787824786.331572 write 0x02db5c thru 0x03ac9e (0xd143 bytes) 000914 1787824786.342167 read 0x010023 thru 0x013bad (0x3b8b bytes) 000915 1787824786.357787 punch from 0x0281d3 to 0x02f250 (0x707e bytes) 000916 1787824786.381162 mapread 0x03b69d thru 0x03f5a5 (0x3f09 bytes) 000919 1787824786.385241 trunc from 0x03f5a6 to 0x0014d0 (0x3e0d5 bytes) 000920 1787824786.417146 MIRROR_RESYNC 13 mirrors 000921 1787824787.530288 punch from 0x0002d6 to 0x000a7b (0x7a6 bytes) 000922 1787824787.569284 write 0x01e926 thru 0x02acf3 (0xc3ce bytes) 000923 1787824787.572514 trunc from 0x02acf4 to 0x02f76a (0x4a77 bytes) 000924 1787824787.596742 MIRROR_RESYNC 13 mirrors 000925 1787824790.054106 mapwrite 0x00d81e thru 0x00f7a0 (0x1f83 bytes) 000926 1787824790.169522 mapread 0x00164e thru 0x00cab6 (0xb469 bytes) 000927 1787824790.186870 write 0x014911 thru 0x015ea8 (0x1598 bytes) 000928 1787824790.187849 trunc from 0x00e256 to 0x01c8dc (0xe687 bytes) 000929 1787824790.200862 read 0x001191 thru 0x010ec7 (0xfd37 bytes) 000930 1787824790.216889 mapread 0x02f033 thru 0x02f76a (0x738 bytes) 000932 1787824790.248134 MIRROR_SPLIT mirror 3 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f76b): Mirror 9: 0xb7c4fcb8 (stale) Mirror 12: 0xb7c4fcb8 (stale) Mirror 14: 0xb7c4fcb8 (stale) Mirror 15: 0xb7c4fcb8 (stale) Mirror 17: 0xb7c4fcb8 (stale) Mirror 19: 0xb7c4fcb8 (stale) Mirror 20: 0xb7c4fcb8 (stale) Mirror 21: 0xb7c4fcb8 (stale) Mirror 22: 0xb7c4fcb8 (stale) Mirror 24: 0xb7c4fcb8 (stale) Mirror 25: 0xb7c4fcb8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f76b): Mirror 9: 0xb7c4fcb8 (stale) Mirror 12: 0xb7c4fcb8 (stale) Mirror 14: 0xb7c4fcb8 (stale) Mirror 15: 0xb7c4fcb8 (stale) Mirror 17: 0xb7c4fcb8 (stale) Mirror 19: 0xb7c4fcb8 (stale) Mirror 20: 0xb7c4fcb8 (stale) Mirror 21: 0xb7c4fcb8 (stale) Mirror 22: 0xb7c4fcb8 (stale) Mirror 24: 0xb7c4fcb8 (stale) Mirror 25: 0xb7c4fcb8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f76b: skipped 000933 1787824790.957753 MIRROR_RESYNC 12 mirrors 000934 1787824793.601307 read 0x02bc1e thru 0x02c513 (0x8f6 bytes) 000935 1787824793.608957 trunc from 0x02f76b to 0x031059 (0x18ef bytes) 000936 1787824793.665661 trunc from 0x03105a to 0x0322b7 (0x125e bytes) 000937 1787824793.711890 trunc from 0x0322b8 to 0x01c3f7 (0x15ec0 bytes) 000938 1787824793.747601 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x1c3f8): Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c3f8): Mirror 9: 0x278789fd (stale) Mirror 12: 0x278789fd (stale) Mirror 14: 0x278789fd (stale) Mirror 15: 0x278789fd (stale) Mirror 17: 0x278789fd (stale) Mirror 19: 0x278789fd (stale) Mirror 20: 0x278789fd (stale) Mirror 21: 0x278789fd (stale) Mirror 22: 0x278789fd (stale) Mirror 24: 0x278789fd (stale) Mirror 25: 0x278789fd (stale) CRC-32 checksum value for chunk [0, 0x1c3f8): Mirror 1: 0x278789fd Mirror 26: 0x278789fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c3f8: skipped 000939 1787824794.802046 trunc from 0x008601 to 0x010fe4 (0x89e4 bytes) 000940 1787824794.859713 read 0x00aaf0 thru 0x014956 (0x9e67 bytes) 000941 1787824794.868835 fallocate 0x00e21c thru 0x011bf2 (0x39d7 bytes) 000942 1787824794.877744 MIRROR_RESYNC 13 mirrors 000943 1787824797.059228 read 0x0135b8 thru 0x01c3f7 (0x8e40 bytes) 000944 1787824797.071572 fallocate 0x038c0a thru 0x03eb48 (0x5f3f bytes) 000945 1787824797.128973 punch from 0x01c10e to 0x01c3f7 (0x2ea bytes) 000946 1787824797.139326 mapwrite 0x009d20 thru 0x010c46 (0x6f27 bytes) 000947 1787824797.206905 MIRROR_RESYNC 13 mirrors 000948 1787824799.394930 write 0x0259b7 thru 0x033d58 (0xe3a2 bytes) 000949 1787824799.429198 write 0x029a4c thru 0x02adc7 (0x137c bytes) 000950 1787824799.435988 MIRROR_SPLIT mirror 7 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33d59): Mirror 9: 0x5a270fba (stale) Mirror 12: 0x5a270fba (stale) Mirror 14: 0x5a270fba (stale) Mirror 15: 0x5a270fba (stale) Mirror 17: 0x5a270fba (stale) Mirror 19: 0x5a270fba (stale) Mirror 21: 0x5a270fba (stale) Mirror 22: 0x5a270fba (stale) Mirror 24: 0x5a270fba (stale) Mirror 25: 0x5a270fba (stale) Mirror 26: 0x5a270fba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33d59): Mirror 9: 0x5a270fba (stale) Mirror 12: 0x5a270fba (stale) Mirror 14: 0x5a270fba (stale) Mirror 15: 0x5a270fba (stale) Mirror 17: 0x5a270fba (stale) Mirror 19: 0x5a270fba (stale) Mirror 21: 0x5a270fba (stale) Mirror 22: 0x5a270fba (stale) Mirror 24: 0x5a270fba (stale) Mirror 25: 0x5a270fba (stale) Mirror 26: 0x5a270fba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33d59: skipped 000951 1787824800.167283 trunc from 0x0336ad to 0x033d58 (0x6ac bytes) 000952 1787824800.172559 mapwrite 0x01d024 thru 0x02a491 (0xd46e bytes) 000953 1787824800.221288 MIRROR_RESYNC 12 mirrors 000954 1787824803.042308 read 0x020d62 thru 0x02ed4d (0xdfec bytes) 000956 1787824803.050540 read 0x017ad9 thru 0x020598 (0x8ac0 bytes) 000957 1787824803.093351 write 0x03e3e2 thru 0x03ffff (0x1c1e bytes) 000958 1787824803.134579 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xfde959ff (stale) Mirror 14: 0xfde959ff (stale) Mirror 15: 0xfde959ff (stale) Mirror 17: 0xfde959ff (stale) Mirror 19: 0xfde959ff (stale) Mirror 21: 0xfde959ff (stale) Mirror 22: 0xfde959ff (stale) Mirror 24: 0xfde959ff (stale) Mirror 25: 0xfde959ff (stale) Mirror 26: 0xfde959ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xfde959ff (stale) Mirror 14: 0xfde959ff (stale) Mirror 15: 0xfde959ff (stale) Mirror 17: 0xfde959ff (stale) Mirror 19: 0xfde959ff (stale) Mirror 21: 0xfde959ff (stale) Mirror 22: 0xfde959ff (stale) Mirror 24: 0xfde959ff (stale) Mirror 25: 0xfde959ff (stale) Mirror 26: 0xfde959ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000959 1787824803.819671 trunc from 0x000421 to 0x00bf31 (0xbb11 bytes) 000960 1787824803.830812 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xfde959ff (stale) Mirror 14: 0xfde959ff (stale) Mirror 15: 0xfde959ff (stale) Mirror 17: 0xfde959ff (stale) Mirror 19: 0xfde959ff (stale) Mirror 21: 0xfde959ff (stale) Mirror 22: 0xfde959ff (stale) Mirror 24: 0xfde959ff (stale) Mirror 25: 0xfde959ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xfde959ff (stale) Mirror 14: 0xfde959ff (stale) Mirror 15: 0xfde959ff (stale) Mirror 17: 0xfde959ff (stale) Mirror 19: 0xfde959ff (stale) Mirror 21: 0xfde959ff (stale) Mirror 22: 0xfde959ff (stale) Mirror 24: 0xfde959ff (stale) Mirror 25: 0xfde959ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000961 1787824804.357714 mapread 0x021e83 thru 0x02a552 (0x86d0 bytes) 000962 1787824804.384919 trunc from 0x040000 to 0x02711a (0x18ee5 bytes) 000964 1787824804.407829 write 0x01be51 thru 0x029e8f (0xe03f bytes) 000965 1787824804.415220 trunc from 0x029e90 to 0x020908 (0x9587 bytes) 000966 1787824804.441745 fallocate 0x030308 thru 0x0383ba (0x80b3 bytes) 000968 1787824804.455090 MIRROR_RESYNC 10 mirrors 000969 1787824807.452728 trunc from 0x00d274 to 0x00efd3 (0x1d60 bytes) 000970 1787824807.532000 fallocate 0x02cfc3 thru 0x031b00 (0x4b3e bytes) 000971 1787824807.539141 read 0x020ee9 thru 0x02d086 (0xc19e bytes) 000972 1787824807.556511 read 0x011f0c thru 0x019a2a (0x7b1f bytes) 000973 1787824807.587658 trunc from 0x0257f9 to 0x029fab (0x47b3 bytes) 000974 1787824807.602114 trunc from 0x031b01 to 0x01d9fd (0x14103 bytes) 000975 1787824807.658870 read 0x01bc9a thru 0x01d9fd (0x1d64 bytes) 000976 1787824807.667120 MIRROR_RESYNC 10 mirrors 000977 1787824810.310455 mapread 0x0058d8 thru 0x013297 (0xd9c0 bytes) 000978 1787824810.348206 read 0x00d3ff thru 0x0186f2 (0xb2f4 bytes) 000979 1787824810.348741 trunc from 0x01d9fe to 0x005af3 (0x17f0a bytes) 000980 1787824810.394717 trunc from 0x000107 to 0x004387 (0x4281 bytes) 000981 1787824810.404190 MIRROR_RESYNC 10 mirrors 000982 1787824811.803171 mapread 0x0039d8 thru 0x003d45 (0x36e bytes) 000983 1787824811.811188 fallocate 0x012129 thru 0x013641 (0x1519 bytes) 000984 1787824811.850382 trunc from 0x013642 to 0x02fec4 (0x1c883 bytes) 000985 1787824811.866024 fallocate 0x004174 thru 0x00a82a (0x66b7 bytes) 000986 1787824811.869828 fallocate 0x0269be thru 0x02cf76 (0x65b9 bytes) 000987 1787824811.874053 mapwrite 0x007b3f thru 0x00aab5 (0x2f77 bytes) 000988 1787824811.896467 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fec5): Mirror 12: 0x313857b8 (stale) Mirror 14: 0x313857b8 (stale) Mirror 15: 0x313857b8 (stale) Mirror 17: 0x313857b8 (stale) Mirror 19: 0x313857b8 (stale) Mirror 22: 0x313857b8 (stale) Mirror 24: 0x313857b8 (stale) Mirror 25: 0x313857b8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fec5): Mirror 12: 0x313857b8 (stale) Mirror 14: 0x313857b8 (stale) Mirror 15: 0x313857b8 (stale) Mirror 17: 0x313857b8 (stale) Mirror 19: 0x313857b8 (stale) Mirror 22: 0x313857b8 (stale) Mirror 24: 0x313857b8 (stale) Mirror 25: 0x313857b8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fec5: skipped 000990 1787824812.409972 fallocate 0x020bda thru 0x028b11 (0x7f38 bytes) 000991 1787824812.416381 write 0x0108c3 thru 0x010db2 (0x4f0 bytes) 000992 1787824812.428112 fallocate 0x038407 thru 0x03c7ec (0x43e6 bytes) 000993 1787824812.436199 mapwrite 0x007c27 thru 0x015f37 (0xe311 bytes) 000994 1787824812.498578 read 0x039e8e thru 0x03c7ec (0x295f bytes) 000995 1787824812.518160 write 0x034a5f thru 0x03fc58 (0xb1fa bytes) 000996 1787824812.533015 punch from 0x01acef to 0x01ed19 (0x402b bytes) 000997 1787824812.542480 punch from 0x037587 to 0x03e0be (0x6b38 bytes) 000998 1787824812.585927 MIRROR_RESYNC 9 mirrors 000999 1787824815.875503 trunc from 0x03fc59 to 0x00fa01 (0x30257 bytes) 001000 1787824815.932586 punch from 0x007a1f to 0x00fa01 (0x7fe3 bytes) All operations completed A-OK! PASS 70a (107s) == sanity-flr test 71: check mirror extend parent fid ==== 06:00:22 (1787824822) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x95:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 06:00:32 (1787824832) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.138661 s, 15.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116834 s, 9.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13c:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 06:00:41 (1787824841) Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre2 Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3712608 .. Extending file size to 2039968 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8589792 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5636064 .. Extending file size to 8268256 .. Extending file size to 8952800 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8552704 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8990304 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9169184 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9292256 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 62831 62833 62834 62835 62836 192.168.201.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg151-client.virtnet /mnt/lustre2 (opts:) 192.168.201.151@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg151-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (78s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 06:01:59 (1787824919) Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre2 Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4690176 .. Extending file size to 6111136 .. failed Extending file size to 6192224 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6494912 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8149984 .. Extending file size to 7740384 .. Extending file size to 8272608 .. Extending file size to 8357760 .. Extending file size to 7890272 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8787136 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8539552 .. Extending file size to 9022528 .. Extending file size to 9247040 .. done Waiting 64571 64572 64573 64575 64578 192.168.201.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg151-client.virtnet /mnt/lustre2 (opts:) 192.168.201.151@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg151-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (84s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 06:03:23 (1787825003) Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2532 1268772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2092 1269212 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24424 3582596 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18672 3588348 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43096 7170944 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2532 1268772 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2092 1269212 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24424 3582596 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18672 3588348 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43096 7170944 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.172426 s, 12.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.17006 s, 12.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x257af63c (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.201.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg151-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 06:03:43 (1787825023) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0708594 s, 29.6 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 06:03:50 (1787825030) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa4:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x159:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x159:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15a:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 06:03:57 (1787825037) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.115896 s, 18.1 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x262:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x263:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x264:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 06:04:05 (1787825045) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0938011 s, 22.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x159:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 06:04:12 (1787825052) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195267 s, 10.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15b:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 06:04:45 (1787825085) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg151-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852811 s, 12.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x263:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x266:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x265:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x263:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x266:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x265:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x264:0x0] } PASS 204d (49s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 06:05:34 (1787825134) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg151-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117525 s, 17.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (50s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 06:06:24 (1787825184) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg151-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg151-server: Pool lustre.test_204f created Adding targets to pool oleg151-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg151-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.165231 s, 12.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg151-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg151-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg151-server: Pool lustre.test_204f destroyed PASS 204f (73s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 06:07:37 (1787825257) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 06:07:44 (1787825264) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 06:07:52 (1787825272) oleg151-server: Pool lustre.test_206 created Adding targets to pool oleg151-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg151-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg151-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg151-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg151-server: Pool lustre.test_206 destroyed PASS 206 (28s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 06:08:20 (1787825300) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0992403 s, 10.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310455 s, 33.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xcd38637 Mirror 3: 0xcd38637 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 06:08:30 (1787825310) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 06:08:34 (1787825314) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 06:08:39 (1787825319) Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0836873 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1370000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1890000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1890000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1890000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xac0000 Read this (4 bytes): BBBB 192.168.201.151@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg151-client.virtnet /mnt/lustre2 (opts:) PASS 209a (44s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 06:09:22 (1787825362) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0655329 s, 125 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 06:09:29 (1787825369) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246923 s, 42.5 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc5 lmm_fid: [0x200000405:0xc5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xc5 lmm_fid: [0x200000405:0xc5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x16a:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611333 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 06:09:30.000000000 -0400 Modify: 2026-08-27 06:09:30.000000000 -0400 Change: 2026-08-27 06:09:31.000000000 -0400 Birth: 2026-08-27 06:09:30.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 06:09:36 (1787825376) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535671 s, 19.6 MB/s oleg151-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg151-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg151-server: O/280000401/d8/168: File not found by ext2_lookup PASS 210b (20s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 06:09:56 (1787825396) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.268379 s, 153 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0225006 s, 182 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 06:09 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xca:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 06:09 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 06:10:03 (1787825403) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 06:10:12 (1787825412) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1723 sec =========== 06:10:20 (1787825420) === sanity-flr: start cleanup 06:10:22 (1787825422) === === sanity-flr: finish cleanup 06:10:28 (1787825428) ===