-----============= acceptance-small: sanity-flr ============----- Fri Jul 3 23:32:28 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg412-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 23:32:50 (1783135970) === oleg412-client.virtnet: executing check_config_client /mnt/lustre oleg412-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg412-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8b498c879000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8b498c879000.idle_timeout=debug disable quota as required oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 23:33:36 (1783136016) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8254] == sanity-flr test 0a: lfs mirror create with -N option == 23:33:39 (1783136019) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:33:58 (1783136038) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:34:02 (1783136042) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:34:06 (1783136046) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:34:35 (1783136075) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:34:38 (1783136078) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 23:34:42 (1783136082) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8b498c879000.stats=clear osc.lustre-OST0001-osc-ffff8b498c879000.stats=clear PASS 0g (11s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:34:53 (1783136093) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:35:04 (1783136104) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:35:13 (1783136113) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 23:35:17 (1783136117) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:35:28 (1783136128) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 23:35:39 (1783136139) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:35:46 (1783136146) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:35:54 (1783136154) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:36:03 (1783136163) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:36:15 (1783136175) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:36:21 (1783136181) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.870243 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102066 s, 10.3 MB/s PASS 21 (8s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:36:30 (1783136190) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.398038 s, 39.5 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 23:36:38 (1783136198) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:36:48 (1783136208) Stopping /mnt/lustre-ost1 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b498c879000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b498c879000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg412-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b498c879000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b498c879000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg412-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (67s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:37:55 (1783136275) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.498013 s, 23.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b498c879000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b498c879000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg412-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (40s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:38:36 (1783136316) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b498c879000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b498c879000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg412-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b498c879000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b498c879000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg412-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:39:57 (1783136397) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.0779 s, 17.0 MB/s -rw-r--r-- 1 root root 52428800 Jul 3 23:40 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.009s user 0m0.004s sys 0m0.191s Stopping /mnt/lustre-ost1 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8b498c879000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8b498c879000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.897s user 0m0.001s sys 0m0.177s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg412-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg412-server oleg412-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8b498c879000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b498c879000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.221s user 0m0.001s sys 0m0.209s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg412-server: oleg412-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg412-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg412-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b498c879000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8b498c879000.ost_server_uuid in FULL state after 0 sec PASS 33b (90s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:41:27 (1783136487) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:41:30 (1783136490) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:41:34 (1783136494) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 23:41:37 (1783136497) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 23:41:44 (1783136504) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.641439 s, 19.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:41:56 (1783136516) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437834 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280561 s, 37.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:42:03 (1783136523) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:42:11 (1783136531) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00131189 s, 781 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00134798 s, 760 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000867874 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00155474 s, 659 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 23:42:21 (1783136541) create test files with size 15756714 .. b6e74e181a79bf3670ebfda9275580e9 - 6b8c19d19792ae93a363ddbdcf6c5a83 - ce50398a1bc5d1ccd343cef98f434a36 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg412-client.virtnet /mnt/lustre (opts:) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 23:42:42 (1783136562) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:42:55 (1783136575) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:43:03 (1783136583) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913818 s, 11.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336225 s, 31.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 23:43:14 (1783136594) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0972155 s, 21.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.339436 s, 12.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.15408 s, 13.6 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 23:43:29 (1783136609) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 23:43:33 (1783136613) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:43:37 (1783136617) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0248851 s, 42.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 23:43:45 (1783136625) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.252282 s, 12.5 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:43:56 (1783136636) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:44:06 (1783136646) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.353478 s, 29.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg412-client.virtnet /mnt/lustre (opts:) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:44:18 (1783136658) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8b49914c2800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.396749 s, 26.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg412-client.virtnet /mnt/lustre (opts:) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8b498b712800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:44:35 (1783136675) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (35s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:45:09 (1783136709) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:45:42 (1783136742) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 23:45:45 (1783136745) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.251393 s, 12.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.135931 s, 23.1 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xbe749523 (nosync,stale) Mirror 2: 0xbe749523 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 710c4c4d7cd2d349649564a912d2666c - mirror 1 checksum: d303840779e6c02c70f7a46859a4f35e - mirror 2 checksum: 710c4c4d7cd2d349649564a912d2666c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-03T23:45:47-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-03T23:45:47-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 710c4c4d7cd2d349649564a912d2666c - mirror 1 checksum: d303840779e6c02c70f7a46859a4f35e - mirror 2 checksum: d303840779e6c02c70f7a46859a4f35e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-03T23:45:47-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-03T23:45:47-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 23:45:54 (1783136754) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287856 s, 36.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (6s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:46:00 (1783136760) Failing mds1 on oleg412-server Stopping /mnt/lustre-mds1 (opts:) on oleg412-server 23:46:11 (1783136771) shut down facet: mds1 facet_host: oleg412-server facet_failover_host: oleg412-server Failover mds1 to oleg412-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg412-server: oleg412-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Started lustre-MDT0000 23:46:34 (1783136794) targets are mounted 23:46:34 (1783136794) facet_failover done oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0409118 s, 25.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 50A (50s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:46:50 (1783136810) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0356112 s, 575 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0299282 s, 684 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:46:59 (1783136819) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0385819 s, 531 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0101241 s, 2.0 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:47:08 (1783136828) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00641889 s, 2.6 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:47:16 (1783136836) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0373435 s, 548 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:47:23 (1783136843) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00464427 s, 882 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0336906 s, 122 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:47:30 (1783136850) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1c:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:47:53 (1783136873) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:48:21 (1783136901) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x15:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:48:44 (1783136924) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:49:08 (1783136948) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 23:49:17 (1783136957) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57401 mirror split pid 57403 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57401 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57403 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 23:50:25 (1783137025) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 7027 skipping zero size read skipping zero length punch hole 000003 1783137027.702103 mapwrite 0x019000 thru 0x027662 (0xe663 bytes) 000004 1783137027.778892 mapwrite 0x036062 thru 0x03ffff (0x9f9e bytes) 000005 1783137027.849750 mapwrite 0x01968b thru 0x024f34 (0xb8aa bytes) 000006 1783137027.897343 trunc from 0x02d065 to 0x031133 (0x40cf bytes) truncating to largest ever: 0x1bc2b 000007 1783137027.905078 trunc from 0x040000 to 0x01bc2a (0x243d5 bytes) 000008 1783137027.943670 write 0x001f96 thru 0x001fa1 (0xc bytes) 000009 1783137027.988504 mapread 0x007da7 thru 0x013ab2 (0xbd0c bytes) 000010 1783137027.996355 mapwrite 0x0005a5 thru 0x00ec70 (0xe6cc bytes) fallocating to largest ever: 0x2f620 000011 1783137028.051316 fallocate 0x026dc8 thru 0x02f61f (0x8858 bytes) 000012 1783137028.061420 read 0x0068d4 thru 0x00a4bf (0x3bec bytes) 000013 1783137028.063013 mapread 0x0155d8 thru 0x01bc2a (0x6653 bytes) 000014 1783137028.064840 mapwrite 0x009b50 thru 0x012e7f (0x9330 bytes) 000016 1783137028.105365 write 0x008651 thru 0x014d5d (0xc70d bytes) 000017 1783137028.112207 write 0x01ed6c thru 0x02d09c (0xe331 bytes) 000018 1783137028.117347 trunc from 0x01a7b8 to 0x026699 (0xbee2 bytes) 000019 1783137028.162684 fallocate 0x001283 thru 0x00415a (0x2ed8 bytes) fallocating to largest ever: 0x3b145 000020 1783137028.168666 fallocate 0x0302c5 thru 0x03b144 (0xae80 bytes) 000022 1783137028.180968 mapread 0x009260 thru 0x017e22 (0xebc3 bytes) 000023 1783137028.182730 trunc from 0x03b145 to 0x0190d9 (0x2206b bytes) 000024 1783137028.219882 mapread 0x00ff93 thru 0x016128 (0x6196 bytes) 000025 1783137028.221308 punch from 0x00825f to 0x00ce71 (0x4c13 bytes) 000026 1783137028.232506 trunc from 0x00fe27 to 0x018d1c (0x8ef6 bytes) 000027 1783137028.245176 write 0x025966 thru 0x02df12 (0x85ad bytes) 000028 1783137028.248157 write 0x03d277 thru 0x03ffff (0x2d89 bytes) 000029 1783137028.250432 punch from 0x00eb46 to 0x0154d3 (0x698e bytes) 000030 1783137028.264237 punch from 0x01c09c to 0x022e5c (0x6dc1 bytes) 000031 1783137028.271367 punch from 0x02c347 to 0x039778 (0xd432 bytes) 000032 1783137028.283533 write 0x0190e6 thru 0x020e30 (0x7d4b bytes) 000034 1783137028.294950 mapwrite 0x010751 thru 0x01c6c4 (0xbf74 bytes) 000036 1783137028.345130 trunc from 0x040000 to 0x034293 (0xbd6c bytes) 000038 1783137028.370136 punch from 0x00280c to 0x00473a (0x1f2f bytes) 000039 1783137028.376735 fallocate 0x00fefc thru 0x010a71 (0xb76 bytes) 000040 1783137028.382901 mapread 0x006e3b thru 0x00a875 (0x3a3b bytes) 000041 1783137028.393421 mapread 0x02f550 thru 0x032f52 (0x3a03 bytes) 000042 1783137028.404472 mapwrite 0x003c25 thru 0x0128fd (0xecd9 bytes) 000044 1783137028.428415 mapread 0x01f764 thru 0x02510b (0x59a8 bytes) 000045 1783137028.436413 mapread 0x0323c6 thru 0x034293 (0x1ece bytes) 000046 1783137028.437682 punch from 0x0200cf to 0x026c5b (0x6b8d bytes) 000047 1783137028.443739 read 0x0267cd thru 0x034293 (0xdac7 bytes) 000048 1783137028.452690 mapwrite 0x0017ee thru 0x00b8ce (0xa0e1 bytes) 000050 1783137028.472769 trunc from 0x025abf to 0x02810a (0x264c bytes) 000051 1783137028.478055 trunc from 0x00df55 to 0x00f3d4 (0x1480 bytes) 000052 1783137028.484259 read 0x020ea6 thru 0x025400 (0x455b bytes) 000053 1783137028.495443 mapwrite 0x0354da thru 0x03cf22 (0x7a49 bytes) 000054 1783137028.539669 trunc from 0x005735 to 0x00f785 (0xa051 bytes) 000055 1783137028.544597 mapwrite 0x002b15 thru 0x00b608 (0x8af4 bytes) 000056 1783137028.620279 fallocate 0x01a9f5 thru 0x028875 (0xde81 bytes) 000057 1783137028.627276 write 0x00b263 thru 0x01550a (0xa2a8 bytes) 000058 1783137028.629962 write 0x01d5f7 thru 0x026011 (0x8a1b bytes) 000059 1783137028.632116 trunc from 0x03cf23 to 0x017b26 (0x253fc bytes) 000060 1783137028.653922 trunc from 0x0021b2 to 0x00a3e4 (0x8233 bytes) 000061 1783137028.660419 trunc from 0x009d27 to 0x00bb82 (0x1e5c bytes) 000062 1783137028.693421 MIRROR_EXTEND to 2 mirrors 000063 1783137029.528400 punch from 0x0040a6 to 0x011379 (0xd2d4 bytes) 000064 1783137029.577086 read 0x016899 thru 0x017b26 (0x128e bytes) 000065 1783137029.583298 punch from 0x00bf40 to 0x00df4d (0x200e bytes) 000067 1783137029.589874 trunc from 0x017b27 to 0x01f56d (0x7a47 bytes) 000068 1783137029.620151 write 0x025ab7 thru 0x02b0ef (0x5639 bytes) 000069 1783137029.621497 mapread 0x012d07 thru 0x01ef66 (0xc260 bytes) 000070 1783137029.655868 fallocate 0x01f55b thru 0x02e874 (0xf31a bytes) 000071 1783137029.668527 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) 000072 1783137029.858500 write 0x016706 thru 0x01eb51 (0x844c bytes) 000073 1783137029.881432 trunc from 0x018561 to 0x01c12c (0x3bcc bytes) 000074 1783137029.906024 punch from 0x017535 to 0x01b25b (0x3d27 bytes) 000075 1783137029.913236 read 0x00f54e thru 0x0107ea (0x129d bytes) 000077 1783137029.923466 write 0x03c017 thru 0x03ffff (0x3fe9 bytes) 000078 1783137029.926238 mapread 0x0139b0 thru 0x020d5e (0xd3af bytes) 000079 1783137029.966331 trunc from 0x040000 to 0x0200b1 (0x1ff4e bytes) 000080 1783137029.989877 trunc from 0x0200b2 to 0x022575 (0x24c4 bytes) 000081 1783137030.010100 read 0x017dc8 thru 0x0186a9 (0x8e2 bytes) 000082 1783137030.011051 write 0x02bd31 thru 0x037b02 (0xbdd2 bytes) 000084 1783137030.015499 read 0x023664 thru 0x026cc0 (0x365d bytes) 000085 1783137030.027207 mapwrite 0x013c4c thru 0x01c9c6 (0x8d7b bytes) 000087 1783137030.065254 read 0x0061d4 thru 0x01517c (0xefa9 bytes) 000088 1783137030.085930 MIRROR_EXTEND to 2 mirrors 000089 1783137031.072361 punch from 0x01c433 to 0x026662 (0xa230 bytes) 000090 1783137031.156441 read 0x01b5f6 thru 0x0237bd (0x81c8 bytes) 000092 1783137031.199665 punch from 0x00ca12 to 0x010afb (0x40ea bytes) 000093 1783137031.222327 write 0x02a266 thru 0x033a17 (0x97b2 bytes) 000094 1783137031.271296 trunc from 0x037b03 to 0x014ac6 (0x2303c bytes) 000095 1783137031.378446 punch from 0x003f78 to 0x006dd1 (0x2e5a bytes) 000096 1783137031.390816 mapread 0x00dabc thru 0x014ac6 (0x700b bytes) 000097 1783137031.434747 punch from 0x009a1e to 0x01325e (0x9841 bytes) 000098 1783137031.448028 fallocate 0x00364d thru 0x00590b (0x22bf bytes) 000099 1783137031.456624 write 0x016d19 thru 0x016e1d (0x105 bytes) 000100 1783137031.464013 mapwrite 0x017738 thru 0x01cb52 (0x541b bytes) 000101 1783137031.599783 punch from 0x007442 to 0x00a67b (0x323a bytes) 000102 1783137031.608713 mapwrite 0x00fa9e thru 0x01c650 (0xcbb3 bytes) 000103 1783137031.651938 read 0x01b928 thru 0x01cb52 (0x122b bytes) 000104 1783137031.652852 mapread 0x006fbc thru 0x00d5ee (0x6633 bytes) 000105 1783137031.700136 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) fallocating to largest ever: 0x40000 000107 1783137031.912736 fallocate 0x03f3e1 thru 0x03ffff (0xc1f bytes) 000108 1783137031.918568 punch from 0x0344b7 to 0x03ac7d (0x67c7 bytes) 000109 1783137031.928117 MIRROR_EXTEND to 2 mirrors 000110 1783137033.065558 fallocate 0x036a39 thru 0x03df81 (0x7549 bytes) 000111 1783137033.124850 punch from 0x004e08 to 0x0067b8 (0x19b1 bytes) 000113 1783137033.140021 trunc from 0x018bdf to 0x01e8ce (0x5cf0 bytes) 000114 1783137033.158053 trunc from 0x040000 to 0x017d6f (0x28290 bytes) 000115 1783137033.214035 punch from 0x00a18e to 0x0143bc (0xa22f bytes) 000117 1783137033.227445 mapwrite 0x016eeb thru 0x024864 (0xd97a bytes) 000118 1783137033.352501 read 0x010e3e thru 0x01c0e1 (0xb2a4 bytes) 000119 1783137033.367647 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x24865): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24865): Mirror 2: 0x594352c6 (stale) CRC-32 checksum value for chunk [0, 0x24865): Mirror 1: 0xca73730d Mirror 3: 0xca73730d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24865: skipped 000120 1783137034.719878 MIRROR_RESYNC 3 mirrors 000121 1783137035.434954 read 0x022593 thru 0x024864 (0x22d2 bytes) 000122 1783137035.443109 write 0x01ebf0 thru 0x022935 (0x3d46 bytes) 000123 1783137035.496765 write 0x02f69a thru 0x033fcb (0x4932 bytes) 000124 1783137035.506735 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33fcc): Mirror 3: 0x3ca054db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33fcc): Mirror 3: 0x3ca054db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33fcc: skipped 000125 1783137035.886814 punch from 0x013919 to 0x021807 (0xdeef bytes) 000126 1783137035.932492 mapread 0x030a3d thru 0x033fcb (0x358f bytes) 000127 1783137035.978244 trunc from 0x02ae0b to 0x033fcb (0x91c1 bytes) 000128 1783137035.985532 mapwrite 0x02e799 thru 0x03d481 (0xece9 bytes) 000129 1783137036.112185 punch from 0x023f77 to 0x0249bb (0xa45 bytes) 000130 1783137036.137783 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) 000132 1783137036.422978 mapwrite 0x031c37 thru 0x0329cb (0xd95 bytes) 000133 1783137036.475677 trunc from 0x03d482 to 0x02d579 (0xff08 bytes) 000134 1783137036.513693 write 0x024087 thru 0x024d6a (0xce4 bytes) 000135 1783137036.523916 mapread 0x018e3d thru 0x0283d2 (0xf596 bytes) 000136 1783137036.568246 trunc from 0x003fd3 to 0x0085e4 (0x4612 bytes) 000137 1783137036.583633 punch from 0x0111ac to 0x015f68 (0x4dbd bytes) 000138 1783137036.633720 punch from 0x0276ec to 0x02d579 (0x5e8e bytes) 000140 1783137036.669492 fallocate 0x002b16 thru 0x00d767 (0xac52 bytes) 000141 1783137036.694520 punch from 0x0137be to 0x01a4d3 (0x6d16 bytes) 000142 1783137036.712360 mapwrite 0x013f43 thru 0x01dee6 (0x9fa4 bytes) 000143 1783137036.807445 trunc from 0x02d57a to 0x026efe (0x667b bytes) 000145 1783137036.846083 trunc from 0x01ebd4 to 0x025a4d (0x6e7a bytes) 000147 1783137036.868777 fallocate 0x00cbb0 thru 0x00daef (0xf40 bytes) 000148 1783137036.876632 trunc from 0x026eff to 0x03cee6 (0x15fe8 bytes) 000149 1783137036.903830 mapread 0x0057a9 thru 0x00ee93 (0x96eb bytes) 000150 1783137036.933657 read 0x00bfe0 thru 0x0189dd (0xc9fe bytes) 000151 1783137036.934979 punch from 0x01e9f2 to 0x026a35 (0x8044 bytes) 000152 1783137036.953198 read 0x0029f2 thru 0x01289d (0xfeac bytes) 000153 1783137036.955000 fallocate 0x0181d5 thru 0x0249eb (0xc817 bytes) 000154 1783137036.964804 read 0x011f48 thru 0x02107d (0xf136 bytes) 000156 1783137036.991229 trunc from 0x03cee7 to 0x00a41c (0x32aca bytes) 000157 1783137037.022834 punch from 0x0026dd to 0x006de8 (0x470c bytes) 000158 1783137037.030743 read 0x004265 thru 0x00a41c (0x61b8 bytes) 000160 1783137037.045904 punch from 0x005186 to 0x00a41c (0x5297 bytes) 000161 1783137037.053344 mapwrite 0x025b3a thru 0x02daa6 (0x7f6d bytes) 000162 1783137037.156975 write 0x028409 thru 0x032e07 (0xa9ff bytes) 000163 1783137037.165693 write 0x007433 thru 0x00ce0a (0x59d8 bytes) 000165 1783137037.186771 mapwrite 0x00c541 thru 0x018825 (0xc2e5 bytes) 000166 1783137037.262014 trunc from 0x032e08 to 0x029936 (0x94d1 bytes) 000167 1783137037.326712 punch from 0x00273d to 0x008a63 (0x6327 bytes) 000168 1783137037.341993 mapread 0x008324 thru 0x00dc1a (0x58f7 bytes) 000169 1783137037.351209 trunc from 0x029937 to 0x00a389 (0x1f5ad bytes) 000170 1783137037.395977 mapread 0x001877 thru 0x00a389 (0x8b13 bytes) 000174 1783137037.529412 mapwrite 0x0354ce thru 0x03ffff (0xab32 bytes) 000175 1783137037.659099 read 0x01350d thru 0x01d73a (0xa22e bytes) 000176 1783137037.686454 mapwrite 0x0111f9 thru 0x01af2c (0x9d34 bytes) 000177 1783137037.741310 punch from 0x03f1dc to 0x03ffff (0xe24 bytes) 000178 1783137037.751025 trunc from 0x008def to 0x00f815 (0x6a27 bytes) 000179 1783137037.770764 fallocate 0x02c92b thru 0x038c9a (0xc370 bytes) 000180 1783137037.787709 punch from 0x00d8ef to 0x0188f7 (0xb009 bytes) 000182 1783137037.820934 mapwrite 0x019584 thru 0x02252c (0x8fa9 bytes) 000183 1783137037.857316 write 0x03f189 thru 0x03ffff (0xe77 bytes) 000184 1783137037.869295 mapread 0x021279 thru 0x02d65d (0xc3e5 bytes) 000185 1783137037.871863 trunc from 0x03438e to 0x03edb1 (0xaa24 bytes) 000187 1783137037.881880 punch from 0x021a10 to 0x02e8be (0xceaf bytes) 000188 1783137037.888149 fallocate 0x0391b8 thru 0x03ffff (0x6e48 bytes) 000189 1783137037.894877 trunc from 0x040000 to 0x03c846 (0x37b9 bytes) 000190 1783137037.913428 mapread 0x00dad9 thru 0x010229 (0x2751 bytes) 000191 1783137037.937827 mapwrite 0x0274b2 thru 0x02da4f (0x659e bytes) 000193 1783137038.100036 write 0x0037a5 thru 0x00ed1e (0xb57a bytes) 000194 1783137038.117534 MIRROR_EXTEND to 2 mirrors 000195 1783137039.551022 trunc from 0x02367c to 0x024623 (0xfa8 bytes) 000196 1783137039.662627 trunc from 0x03c847 to 0x039414 (0x3432 bytes) 000197 1783137039.729073 write 0x003dda thru 0x013b04 (0xfd2b bytes) 000198 1783137039.772914 trunc from 0x0264ff to 0x02f9b7 (0x94b9 bytes) 000199 1783137039.789959 fallocate 0x02b02e thru 0x02c65d (0x1630 bytes) 000200 1783137039.806053 punch from 0x0219ab to 0x0297e9 (0x7e3f bytes) 000201 1783137039.819259 punch from 0x01c1eb to 0x01c210 (0x26 bytes) 000202 1783137039.825105 mapread 0x028774 thru 0x032a0f (0xa29c bytes) 000203 1783137039.894245 mapread 0x01df33 thru 0x0236b9 (0x5787 bytes) 000206 1783137039.936720 trunc from 0x034be3 to 0x039414 (0x4832 bytes) 000207 1783137039.952021 read 0x0074c0 thru 0x00e71e (0x725f bytes) 000208 1783137039.953843 mapwrite 0x01dedd thru 0x02c176 (0xe29a bytes) 000210 1783137040.057677 mapread 0x0208e1 thru 0x02b823 (0xaf43 bytes) 000211 1783137040.062739 mapwrite 0x012162 thru 0x01ba2a (0x98c9 bytes) 000212 1783137040.110296 mapread 0x01d4bf thru 0x02762c (0xa16e bytes) 000213 1783137040.111489 mapread 0x033ae4 thru 0x0362ea (0x2807 bytes) 000214 1783137040.131871 punch from 0x0366e5 to 0x039414 (0x2d30 bytes) 000215 1783137040.137789 read 0x002ec1 thru 0x005585 (0x26c5 bytes) 000216 1783137040.162659 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39415): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39415): Mirror 2: 0x11ceb40e (stale) CRC-32 checksum value for chunk [0, 0x39415): Mirror 1: 0xc76bdce0 Mirror 3: 0xc76bdce0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39415: skipped 000217 1783137041.316486 write 0x01e428 thru 0x029682 (0xb25b bytes) 000218 1783137041.414993 punch from 0x02d1aa to 0x038f83 (0xbdda bytes) 000219 1783137041.450659 write 0x02111e thru 0x024d55 (0x3c38 bytes) 000220 1783137041.452873 mapwrite 0x03cd6a thru 0x03ffff (0x3296 bytes) 000221 1783137041.643264 fallocate 0x01a473 thru 0x0208a3 (0x6431 bytes) 000222 1783137041.654266 read 0x01e311 thru 0x023033 (0x4d23 bytes) 000223 1783137041.655912 punch from 0x00a170 to 0x00ade9 (0xc7a bytes) 000224 1783137041.663998 trunc from 0x024393 to 0x030351 (0xbfbf bytes) 000225 1783137041.682312 MIRROR_RESYNC 3 mirrors 000226 1783137042.657234 write 0x003bc3 thru 0x004625 (0xa63 bytes) 000227 1783137042.731761 trunc from 0x040000 to 0x03e470 (0x1b8f bytes) 000228 1783137042.770341 read 0x00be8e thru 0x012c72 (0x6de5 bytes) 000229 1783137042.787545 read 0x0088aa thru 0x017a9c (0xf1f3 bytes) 000230 1783137042.805460 mapwrite 0x02cbe6 thru 0x037c7b (0xb096 bytes) 000233 1783137042.846759 trunc from 0x02a49d to 0x02c409 (0x1f6d bytes) 000234 1783137042.856324 mapread 0x001b2c thru 0x00e1aa (0xc67f bytes) 000235 1783137042.862492 mapwrite 0x027685 thru 0x028628 (0xfa4 bytes) 000236 1783137042.899006 write 0x02a608 thru 0x032c6b (0x8664 bytes) 000237 1783137042.924811 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e471): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e471): Mirror 2: 0x3b9c551b (stale) Mirror 3: 0x3b9c551b (stale) CRC-32 checksum value for chunk [0, 0x3e471): Mirror 1: 0x5485423 Mirror 4: 0x5485423 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e471: skipped 000238 1783137044.139499 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3e471): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e471): Mirror 2: 0x3b9c551b (stale) Mirror 3: 0x3b9c551b (stale) CRC-32 checksum value for chunk [0, 0x3e471): Mirror 1: 0x5485423 Mirror 4: 0x5485423 Mirror 5: 0x5485423 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e471: skipped 000240 1783137045.574167 punch from 0x02bb47 to 0x02bb75 (0x2f bytes) 000242 1783137045.627904 trunc from 0x03e471 to 0x031d05 (0xc76b bytes) 000243 1783137045.664555 trunc from 0x02bf69 to 0x02e27b (0x2313 bytes) 000244 1783137045.676954 punch from 0x02580e to 0x02ebcd (0x93c0 bytes) 000245 1783137045.693048 trunc from 0x031d06 to 0x02e85c (0x34a9 bytes) 000246 1783137045.718206 trunc from 0x02e85d to 0x00c077 (0x227e5 bytes) 000247 1783137045.750788 fallocate 0x00467b thru 0x008b45 (0x44cb bytes) 000248 1783137045.761479 read 0x00a488 thru 0x00bb72 (0x16eb bytes) 000249 1783137045.770519 fallocate 0x003572 thru 0x00e50c (0xaf9b bytes) 000250 1783137045.779798 trunc from 0x00e50d to 0x003c15 (0xa8f7 bytes) 000251 1783137045.799893 trunc from 0x001c4f to 0x003c15 (0x1fc7 bytes) 000252 1783137045.808674 punch from 0x002148 to 0x003c15 (0x1ace bytes) 000253 1783137045.828499 mapwrite 0x012532 thru 0x022324 (0xfdf3 bytes) 000254 1783137045.931486 mapread 0x002112 thru 0x0080f0 (0x5fdf bytes) 000255 1783137045.950615 trunc from 0x01ba71 to 0x022324 (0x68b4 bytes) 000257 1783137045.958664 trunc from 0x00eb8d to 0x01e6b9 (0xfb2d bytes) 000258 1783137045.965653 mapread 0x01f529 thru 0x022324 (0x2dfc bytes) 000259 1783137045.976338 mapwrite 0x0011fe thru 0x00fd5b (0xeb5e bytes) 000260 1783137046.016830 fallocate 0x03b958 thru 0x03ffff (0x46a8 bytes) 000262 1783137046.040771 fallocate 0x00bded thru 0x015d49 (0x9f5d bytes) 000263 1783137046.064539 MIRROR_RESYNC 5 mirrors 000264 1783137047.633558 trunc from 0x02de2b to 0x033c52 (0x5e28 bytes) 000265 1783137047.671255 trunc from 0x040000 to 0x03f12f (0xed0 bytes) 000266 1783137047.701707 mapread 0x02dec6 thru 0x031a88 (0x3bc3 bytes) 000267 1783137047.732911 read 0x02a43b thru 0x02dabf (0x3685 bytes) 000268 1783137047.733705 write 0x03ee8d thru 0x03ffff (0x1173 bytes) 000269 1783137047.740020 punch from 0x000d8b to 0x00c9b3 (0xbc29 bytes) 000270 1783137047.747898 fallocate 0x03ed15 thru 0x03ffff (0x12eb bytes) 000271 1783137047.754631 fallocate 0x030c56 thru 0x03ddea (0xd195 bytes) 000272 1783137047.761051 fallocate 0x034992 thru 0x03ffff (0xb66e bytes) 000273 1783137047.767661 trunc from 0x00149a to 0x00fd72 (0xe8d9 bytes) 000274 1783137047.775074 write 0x025dba thru 0x034aee (0xed35 bytes) 000275 1783137047.783913 trunc from 0x040000 to 0x024534 (0x1bacb bytes) 000276 1783137047.808397 fallocate 0x000120 thru 0x008d53 (0x8c34 bytes) 000277 1783137047.813902 mapread 0x0187a0 thru 0x024534 (0xbd95 bytes) 000278 1783137047.832478 read 0x007d24 thru 0x0163bb (0xe698 bytes) 000279 1783137047.843440 mapwrite 0x02ae58 thru 0x0383a3 (0xd54c bytes) 000280 1783137047.913111 write 0x03946f thru 0x03dbd6 (0x4768 bytes) 000281 1783137047.915853 punch from 0x00e854 to 0x012a2b (0x41d8 bytes) 000282 1783137047.923807 write 0x03dc86 thru 0x03ffff (0x237a bytes) 000283 1783137047.925636 mapread 0x02cf57 thru 0x034149 (0x71f3 bytes) 000284 1783137047.930121 mapread 0x03457b thru 0x03ffff (0xba85 bytes) 000285 1783137047.932887 trunc from 0x0000aa to 0x008d52 (0x8ca9 bytes) 000286 1783137047.945658 trunc from 0x027226 to 0x029b19 (0x28f4 bytes) 000287 1783137047.955756 trunc from 0x040000 to 0x00880e (0x377f1 bytes) 000288 1783137047.983105 fallocate 0x012a80 thru 0x020d19 (0xe29a bytes) 000289 1783137048.001865 mapread 0x003d27 thru 0x00880e (0x4ae8 bytes) 000290 1783137048.062891 mapread 0x0068e4 thru 0x00880e (0x1f2b bytes) 000291 1783137048.064078 mapwrite 0x02aacd thru 0x030adf (0x6013 bytes) 000292 1783137048.143058 punch from 0x02b8ce to 0x030adf (0x5212 bytes) 000293 1783137048.161249 write 0x003955 thru 0x003ca5 (0x351 bytes) 000294 1783137048.164396 punch from 0x00a1f0 to 0x0124cf (0x82e0 bytes) 000295 1783137048.175645 fallocate 0x011e8b thru 0x01bffd (0xa173 bytes) 000296 1783137048.191349 read 0x0218ce thru 0x029194 (0x78c7 bytes) 000297 1783137048.213017 trunc from 0x030ae0 to 0x02d5f4 (0x34eb bytes) 000298 1783137048.240770 read 0x0245ee thru 0x02cdc4 (0x87d7 bytes) 000299 1783137048.244731 read 0x01c4e1 thru 0x02754b (0xb06b bytes) 000300 1783137048.247926 write 0x03825d thru 0x03ffff (0x7da3 bytes) 000301 1783137048.258905 mapwrite 0x019e85 thru 0x028593 (0xe70f bytes) 000302 1783137048.305565 punch from 0x0181e2 to 0x025ead (0xdccc bytes) 000303 1783137048.316303 punch from 0x015d23 to 0x019415 (0x36f3 bytes) 000304 1783137048.326502 read 0x00703c thru 0x016160 (0xf125 bytes) 000305 1783137048.344903 read 0x0365df thru 0x03ffff (0x9a21 bytes) 000306 1783137048.346694 mapwrite 0x00cd05 thru 0x0100a7 (0x33a3 bytes) 000307 1783137048.404622 read 0x023e50 thru 0x02ffc8 (0xc179 bytes) 000308 1783137048.412653 mapread 0x012e4b thru 0x013225 (0x3db bytes) 000309 1783137048.415339 read 0x005cc5 thru 0x012fd2 (0xd30e bytes) 000310 1783137048.416904 read 0x0146aa thru 0x016aef (0x2446 bytes) 000311 1783137048.418250 trunc from 0x038afa to 0x03ffff (0x7506 bytes) 000313 1783137048.428411 trunc from 0x03572a to 0x0390a9 (0x3980 bytes) 000314 1783137048.437020 mapread 0x006ed6 thru 0x00ca66 (0x5b91 bytes) 000315 1783137048.439677 write 0x039ed8 thru 0x03a375 (0x49e bytes) 000316 1783137048.452545 trunc from 0x040000 to 0x03579f (0xa860 bytes) 000317 1783137048.482838 fallocate 0x0087cb thru 0x00a273 (0x1aa9 bytes) 000319 1783137048.493126 mapread 0x02a2cd thru 0x02a405 (0x139 bytes) 000321 1783137048.496356 read 0x02f2dc thru 0x03579f (0x64c4 bytes) 000322 1783137048.497654 fallocate 0x02f0a1 thru 0x038c91 (0x9bf1 bytes) 000323 1783137048.509026 write 0x0368ce thru 0x03eb68 (0x829b bytes) 000324 1783137048.526741 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eb69): Mirror 3: 0xc8a853e8 (stale) Mirror 4: 0xc8a853e8 (stale) Mirror 5: 0xc8a853e8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb69): Mirror 3: 0xc8a853e8 (stale) Mirror 4: 0xc8a853e8 (stale) Mirror 5: 0xc8a853e8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb69: skipped 000325 1783137049.002181 write 0x022f21 thru 0x03184a (0xe92a bytes) 000326 1783137049.071864 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3eb69): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eb69): Mirror 3: 0xc8a853e8 (stale) Mirror 4: 0xc8a853e8 (stale) Mirror 5: 0xc8a853e8 (stale) CRC-32 checksum value for chunk [0, 0x3eb69): Mirror 1: 0xd504ce58 Mirror 6: 0xd504ce58 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eb69: skipped 000327 1783137050.851052 fallocate 0x0245e8 thru 0x0274a6 (0x2ebf bytes) 000328 1783137050.905083 trunc from 0x03eb69 to 0x029f48 (0x14c20 bytes) 000329 1783137050.930020 mapwrite 0x00600b thru 0x0141d5 (0xe1cb bytes) 000331 1783137051.000067 write 0x035131 thru 0x03fd56 (0xac26 bytes) 000332 1783137051.006481 write 0x0224e1 thru 0x027b92 (0x56b2 bytes) 000334 1783137051.020311 fallocate 0x0356ac thru 0x03d09c (0x79f1 bytes) 000335 1783137051.029224 mapread 0x010453 thru 0x01d561 (0xd10f bytes) 000336 1783137051.041383 trunc from 0x03fd57 to 0x0291a6 (0x16bb0 bytes) 000338 1783137051.063777 fallocate 0x034dc6 thru 0x038f5c (0x4197 bytes) 000339 1783137051.069885 punch from 0x0123af to 0x014f72 (0x2bc4 bytes) 000340 1783137051.093210 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38f5d): Mirror 4: 0x49a2b7e7 (stale) Mirror 5: 0x49a2b7e7 (stale) Mirror 6: 0x3802e77 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f5d): Mirror 4: 0x49a2b7e7 (stale) Mirror 5: 0x49a2b7e7 (stale) Mirror 6: 0x3802e77 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f5d: skipped 000341 1783137051.531675 write 0x02da67 thru 0x038f22 (0xb4bc bytes) 000343 1783137051.610444 write 0x017602 thru 0x020e65 (0x9864 bytes) 000344 1783137051.628875 trunc from 0x038f5d to 0x002ec1 (0x3609b bytes) 000345 1783137051.675880 trunc from 0x000702 to 0x000a80 (0x37f bytes) 000346 1783137051.695958 trunc from 0x002ec2 to 0x013106 (0x10245 bytes) 000347 1783137051.714494 trunc from 0x013107 to 0x0276d4 (0x145ce bytes) 000348 1783137051.754929 mapwrite 0x03842e thru 0x03b259 (0x2e2c bytes) 000349 1783137051.863842 punch from 0x008b13 to 0x009482 (0x970 bytes) 000350 1783137051.869650 trunc from 0x03b25a to 0x008fbf (0x3229a bytes) 000351 1783137051.896766 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8fc0): Mirror 5: 0xd1e8f669 (stale) Mirror 6: 0xd0a5710f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8fc0): Mirror 5: 0xd1e8f669 (stale) Mirror 6: 0xd0a5710f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8fc0: skipped 000352 1783137052.180780 punch from 0x0035c0 to 0x008fbf (0x5a00 bytes) 000353 1783137052.187754 mapwrite 0x0096ad thru 0x00c03a (0x298e bytes) 000354 1783137052.266188 write 0x030265 thru 0x037d93 (0x7b2f bytes) 000355 1783137052.270462 punch from 0x01831d to 0x022746 (0xa42a bytes) 000356 1783137052.278243 mapread 0x00f09d thru 0x013361 (0x42c5 bytes) 000357 1783137052.292362 mapwrite 0x00b86c thru 0x01a83b (0xefd0 bytes) 000358 1783137052.386943 MIRROR_RESYNC 3 mirrors 000359 1783137053.377771 write 0x01fc42 thru 0x02845e (0x881d bytes) 000360 1783137053.492001 read 0x001166 thru 0x00122a (0xc5 bytes) 000361 1783137053.502205 trunc from 0x037d94 to 0x01013d (0x27c56 bytes) 000362 1783137053.547466 trunc from 0x00de4b to 0x01013d (0x22f3 bytes) 000363 1783137053.559023 trunc from 0x00a05a to 0x01013d (0x60e4 bytes) 000365 1783137053.568913 read 0x00f01c thru 0x01013d (0x1122 bytes) 000366 1783137053.588267 mapread 0x000209 thru 0x00f4d8 (0xf2d0 bytes) 000367 1783137053.590902 fallocate 0x024d0f thru 0x02ea73 (0x9d65 bytes) 000368 1783137053.599080 trunc from 0x01013e to 0x00591b (0xa822 bytes) 000369 1783137053.625426 punch from 0x005863 to 0x00591b (0xb9 bytes) 000370 1783137053.632698 write 0x037054 thru 0x03ffff (0x8fac bytes) 000372 1783137053.636930 trunc from 0x040000 to 0x0228f3 (0x1d70c bytes) 000373 1783137053.668478 MIRROR_RESYNC 3 mirrors 000375 1783137054.498312 mapwrite 0x01b6d8 thru 0x01dcfe (0x2627 bytes) 000376 1783137054.626922 mapwrite 0x02dd43 thru 0x02ee39 (0x10f7 bytes) 000377 1783137054.677406 trunc from 0x02babf to 0x02ee39 (0x337b bytes) 000378 1783137054.690049 mapwrite 0x02e687 thru 0x02f74b (0x10c5 bytes) 000379 1783137054.737125 punch from 0x001a61 to 0x001dbd (0x35d bytes) 000381 1783137054.746370 mapread 0x026c10 thru 0x02f74b (0x8b3c bytes) 000382 1783137054.748395 trunc from 0x02f74c to 0x00a59c (0x251af bytes) 000383 1783137054.779324 trunc from 0x0039d5 to 0x00a59c (0x6bc8 bytes) 000384 1783137054.789308 punch from 0x007e43 to 0x00a59c (0x275a bytes) 000385 1783137054.795480 punch from 0x0068ee to 0x00a59c (0x3caf bytes) 000386 1783137054.800681 trunc from 0x004211 to 0x004787 (0x577 bytes) 000387 1783137054.806507 punch from 0x004a93 to 0x00a59c (0x5b0a bytes) 000388 1783137054.818004 trunc from 0x00a59d to 0x01653f (0xbfa3 bytes) 000389 1783137054.842121 mapread 0x010a01 thru 0x015b42 (0x5142 bytes) 000390 1783137054.859996 punch from 0x009777 to 0x0108a5 (0x712f bytes) 000391 1783137054.866382 fallocate 0x019c70 thru 0x020b54 (0x6ee5 bytes) 000392 1783137054.873092 mapwrite 0x0151f0 thru 0x0159e4 (0x7f5 bytes) 000393 1783137054.893360 trunc from 0x001a30 to 0x00bb6c (0xa13d bytes) 000394 1783137054.907378 trunc from 0x000030 to 0x00257b (0x254c bytes) 000395 1783137054.922720 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x16540): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16540): Mirror 5: 0x3fde9ca (stale) Mirror 6: 0x3fde9ca (stale) CRC-32 checksum value for chunk [0, 0x16540): Mirror 1: 0x2b992ab4 Mirror 7: 0x2b992ab4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16540: skipped 000396 1783137056.289742 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x16540): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16540): Mirror 5: 0x3fde9ca (stale) Mirror 6: 0x3fde9ca (stale) CRC-32 checksum value for chunk [0, 0x16540): Mirror 1: 0x2b992ab4 Mirror 7: 0x2b992ab4 Mirror 8: 0x2b992ab4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16540: skipped 000397 1783137057.508205 trunc from 0x016540 to 0x024377 (0xde38 bytes) 000399 1783137057.584559 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x24378): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24378): Mirror 5: 0x2f5d484 (stale) Mirror 6: 0x2f5d484 (stale) Mirror 7: 0xe8315f3b (stale) Mirror 8: 0xe8315f3b (stale) CRC-32 checksum value for chunk [0, 0x24378): Mirror 1: 0xe8315f3b Mirror 9: 0xe8315f3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24378: skipped 000400 1783137058.824502 mapread 0x01ba65 thru 0x020b48 (0x50e4 bytes) 000401 1783137058.859120 trunc from 0x01094e to 0x01bc50 (0xb303 bytes) 000402 1783137058.924379 punch from 0x023ed0 to 0x024377 (0x4a8 bytes) 000403 1783137058.940089 MIRROR_RESYNC 6 mirrors 000404 1783137060.625151 punch from 0x016ea6 to 0x024377 (0xd4d2 bytes) 000405 1783137060.715036 trunc from 0x01b12f to 0x01bb27 (0x9f9 bytes) 000406 1783137060.739887 write 0x0287dd thru 0x02e3db (0x5bff bytes) 000407 1783137060.741381 mapwrite 0x019fbc thru 0x024ce9 (0xad2e bytes) 000408 1783137060.843354 punch from 0x00c63c to 0x015dcc (0x9791 bytes) 000409 1783137060.855223 mapwrite 0x022c75 thru 0x02f420 (0xc7ac bytes) 000411 1783137060.951444 read 0x02d5cf thru 0x02f420 (0x1e52 bytes) 000412 1783137060.957357 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f421): Mirror 5: 0x78504a6f (stale) Mirror 6: 0x78504a6f (stale) Mirror 7: 0x78504a6f (stale) Mirror 9: 0x78504a6f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f421): Mirror 5: 0x78504a6f (stale) Mirror 6: 0x78504a6f (stale) Mirror 7: 0x78504a6f (stale) Mirror 9: 0x78504a6f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f421: skipped 000413 1783137061.512938 write 0x009db8 thru 0x01805a (0xe2a3 bytes) 000414 1783137061.529733 write 0x01d49c thru 0x0245ec (0x7151 bytes) 000415 1783137061.540523 punch from 0x0257a6 to 0x02922e (0x3a89 bytes) 000416 1783137061.550165 punch from 0x0151dc to 0x024127 (0xef4c bytes) 000417 1783137061.585839 mapread 0x001fd4 thru 0x003a99 (0x1ac6 bytes) 000418 1783137061.606635 trunc from 0x02f421 to 0x00da58 (0x219c8 bytes) 000419 1783137061.626440 fallocate 0x011f46 thru 0x01b306 (0x93c1 bytes) 000420 1783137061.633298 punch from 0x009bd0 to 0x015837 (0xbc68 bytes) 000421 1783137061.640402 mapread 0x0061f2 thru 0x011ade (0xb8ed bytes) 000422 1783137061.652987 mapwrite 0x02ebd4 thru 0x02facb (0xef8 bytes) 000423 1783137061.697284 mapwrite 0x036daf thru 0x039cf4 (0x2f46 bytes) 000424 1783137061.760495 trunc from 0x039cf5 to 0x00d69e (0x2c656 bytes) 000425 1783137061.788641 mapread 0x00a765 thru 0x00d69e (0x2f3a bytes) 000426 1783137061.791724 trunc from 0x003a3c to 0x00d69e (0x9c63 bytes) 000427 1783137061.800287 trunc from 0x006edc to 0x00d69e (0x67c3 bytes) 000428 1783137061.809829 write 0x0383aa thru 0x03ffff (0x7c56 bytes) 000429 1783137061.811443 mapread 0x0350d3 thru 0x0372cd (0x21fb bytes) 000430 1783137061.827409 trunc from 0x02940d to 0x02bdb3 (0x29a7 bytes) 000431 1783137061.832419 trunc from 0x040000 to 0x03c941 (0x36be bytes) 000432 1783137061.864199 punch from 0x01cf0c to 0x023909 (0x69fe bytes) 000433 1783137061.872556 mapwrite 0x0268ff thru 0x0307f3 (0x9ef5 bytes) 000434 1783137061.960243 write 0x0352ef thru 0x03ffff (0xad11 bytes) 000435 1783137061.963191 fallocate 0x037829 thru 0x03e884 (0x705c bytes) 000436 1783137061.976926 MIRROR_RESYNC 5 mirrors 000437 1783137063.718647 mapwrite 0x00a2ab thru 0x0123da (0x8130 bytes) 000438 1783137063.868180 mapread 0x014d1e thru 0x01ed85 (0xa068 bytes) 000439 1783137063.897387 fallocate 0x037687 thru 0x03ffff (0x8979 bytes) 000440 1783137063.904379 punch from 0x0298d5 to 0x02dd0d (0x4439 bytes) 000441 1783137063.911081 write 0x036ff3 thru 0x03f256 (0x8264 bytes) 000442 1783137063.928292 mapread 0x008f4e thru 0x00b0a6 (0x2159 bytes) 000444 1783137063.931368 fallocate 0x01cef6 thru 0x026eb5 (0x9fc0 bytes) 000445 1783137063.940948 fallocate 0x00f422 thru 0x01e740 (0xf31f bytes) 000446 1783137063.959322 fallocate 0x002b4b thru 0x00bbe0 (0x9096 bytes) 000447 1783137063.967215 read 0x003f17 thru 0x005161 (0x124b bytes) 000448 1783137063.978198 trunc from 0x040000 to 0x026338 (0x19cc7 bytes) 000450 1783137064.009507 mapread 0x01dfda thru 0x01f85b (0x1882 bytes) 000451 1783137064.016552 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26339): Mirror 6: 0xc6bb12ae (stale) Mirror 7: 0xc6bb12ae (stale) Mirror 9: 0xc6bb12ae (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26339): Mirror 6: 0xc6bb12ae (stale) Mirror 7: 0xc6bb12ae (stale) Mirror 9: 0xc6bb12ae (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26339: skipped 000452 1783137064.615280 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x26339): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26339): Mirror 6: 0xc6bb12ae (stale) Mirror 7: 0xc6bb12ae (stale) Mirror 9: 0xc6bb12ae (stale) CRC-32 checksum value for chunk [0, 0x26339): Mirror 1: 0xd25312c0 Mirror 10: 0xd25312c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26339: skipped 000453 1783137065.908404 MIRROR_RESYNC 5 mirrors 000454 1783137067.234454 punch from 0x023045 to 0x0242d1 (0x128d bytes) 000455 1783137067.283918 write 0x0211e2 thru 0x02de12 (0xcc31 bytes) 000456 1783137067.292205 read 0x007595 thru 0x011fc1 (0xaa2d bytes) 000457 1783137067.303379 mapwrite 0x03afed thru 0x03fa6f (0x4a83 bytes) 000458 1783137067.390210 punch from 0x02404e to 0x0248fc (0x8af bytes) 000459 1783137067.401732 trunc from 0x03fa70 to 0x00021a (0x3f855 bytes) 000460 1783137067.425761 fallocate 0x02d631 thru 0x02ed3f (0x170f bytes) 000461 1783137067.432981 read 0x003d4f thru 0x011e72 (0xe124 bytes) 000462 1783137067.450024 mapread 0x0098a3 thru 0x0178cc (0xe02a bytes) 000463 1783137067.451449 punch from 0x01e4f6 to 0x022d90 (0x489b bytes) 000464 1783137067.471106 trunc from 0x02ed40 to 0x02316e (0xbbd1 bytes) 000465 1783137067.511928 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2316f): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2316f): Mirror 6: 0x523a83e0 (stale) Mirror 7: 0x523a83e0 (stale) Mirror 9: 0x523a83e0 (stale) Mirror 10: 0x523a83e0 (stale) CRC-32 checksum value for chunk [0, 0x2316f): Mirror 1: 0x3b1fe8ff Mirror 11: 0x3b1fe8ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2316f: skipped 000466 1783137068.535799 fallocate 0x018795 thru 0x01b0a4 (0x2910 bytes) 000467 1783137068.595117 read 0x01b617 thru 0x01e961 (0x334b bytes) 000468 1783137068.615290 trunc from 0x02316f to 0x02ad4d (0x7bdf bytes) 000469 1783137068.745288 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ad4e): Mirror 6: 0xacce17bf (stale) Mirror 7: 0xacce17bf (stale) Mirror 9: 0xacce17bf (stale) Mirror 10: 0xacce17bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ad4e): Mirror 6: 0xacce17bf (stale) Mirror 7: 0xacce17bf (stale) Mirror 9: 0xacce17bf (stale) Mirror 10: 0xacce17bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ad4e: skipped 000470 1783137069.327497 trunc from 0x01beeb to 0x0264a7 (0xa5bd bytes) 000471 1783137069.343678 fallocate 0x023214 thru 0x028691 (0x547e bytes) 000472 1783137069.358439 read 0x01929e thru 0x0243cc (0xb12f bytes) 000473 1783137069.376538 punch from 0x008df4 to 0x00b773 (0x2980 bytes) 000474 1783137069.396205 mapread 0x008a22 thru 0x015ced (0xd2cc bytes) 000475 1783137069.436513 mapwrite 0x015f90 thru 0x01afb9 (0x502a bytes) 000476 1783137069.491404 mapread 0x008014 thru 0x01001a (0x8007 bytes) 000477 1783137069.495329 trunc from 0x02ad4e to 0x00d65d (0x1d6f0 bytes) 000478 1783137069.559153 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd65e): Mirror 6: 0xc399b5ac (stale) Mirror 7: 0xc399b5ac (stale) Mirror 10: 0xc399b5ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd65e): Mirror 6: 0xc399b5ac (stale) Mirror 7: 0xc399b5ac (stale) Mirror 10: 0xc399b5ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd65e: skipped 000479 1783137070.067548 punch from 0x003be5 to 0x00765e (0x3a7a bytes) 000480 1783137070.082336 trunc from 0x00d2e1 to 0x00d65d (0x37d bytes) 000482 1783137070.115295 MIRROR_RESYNC 4 mirrors 000483 1783137071.221321 mapwrite 0x02e5de thru 0x031874 (0x3297 bytes) 000484 1783137071.356740 mapwrite 0x035b75 thru 0x03ffff (0xa48b bytes) 000485 1783137071.421664 write 0x015337 thru 0x022481 (0xd14b bytes) 000486 1783137071.434268 trunc from 0x039a02 to 0x03ffff (0x65fe bytes) 000487 1783137071.447142 trunc from 0x03618f to 0x039ba1 (0x3a13 bytes) 000488 1783137071.467527 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000489 1783137072.002457 fallocate 0x026d85 thru 0x0278db (0xb57 bytes) 000490 1783137072.016232 fallocate 0x0157e2 thru 0x01c709 (0x6f28 bytes) 000491 1783137072.031660 punch from 0x03050a to 0x0376c7 (0x71be bytes) 000492 1783137072.049498 trunc from 0x036b18 to 0x03ffff (0x94e8 bytes) 000493 1783137072.063697 mapread 0x01828f thru 0x025ecb (0xdc3d bytes) 000494 1783137072.111625 trunc from 0x012d3a to 0x017ec3 (0x518a bytes) 000495 1783137072.120990 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe3616257 Mirror 8: 0xe3616257 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000496 1783137073.123512 punch from 0x01bb69 to 0x029cd8 (0xe170 bytes) 000497 1783137073.169908 trunc from 0x040000 to 0x028aa5 (0x1755a bytes) 000498 1783137073.209310 fallocate 0x000249 thru 0x009ca4 (0x9a5c bytes) 000501 1783137073.221675 write 0x02a2fb thru 0x039d11 (0xfa17 bytes) 000502 1783137073.224126 write 0x01a8d0 thru 0x025371 (0xaaa2 bytes) 000503 1783137073.236888 read 0x01ccf9 thru 0x0271e4 (0xa4ec bytes) 000504 1783137073.242360 mapwrite 0x02f473 thru 0x039508 (0xa096 bytes) 000505 1783137073.267418 fallocate 0x011f40 thru 0x01e663 (0xc724 bytes) 000506 1783137073.273646 read 0x02c530 thru 0x039d11 (0xd7e2 bytes) 000507 1783137073.281169 MIRROR_RESYNC 4 mirrors 000508 1783137074.502123 trunc from 0x039d12 to 0x02bceb (0xe026 bytes) 000509 1783137074.558640 trunc from 0x02bcec to 0x030961 (0x4c76 bytes) 000510 1783137074.581053 write 0x033523 thru 0x03c6c6 (0x91a4 bytes) 000511 1783137074.584913 trunc from 0x02b2fa to 0x03488d (0x9594 bytes) 000513 1783137074.611008 fallocate 0x012bd1 thru 0x0171ae (0x45de bytes) 000514 1783137074.622645 trunc from 0x0372a6 to 0x03c6c6 (0x5421 bytes) 000515 1783137074.632735 write 0x00ca32 thru 0x016d95 (0xa364 bytes) 000516 1783137074.647993 trunc from 0x035ab3 to 0x03c6c6 (0x6c14 bytes) 000517 1783137074.671479 trunc from 0x03c6c7 to 0x0110d2 (0x2b5f4 bytes) 000518 1783137074.702560 punch from 0x0055b6 to 0x0110d2 (0xbb1d bytes) 000519 1783137074.727424 trunc from 0x00891a to 0x0110d2 (0x87b9 bytes) 000520 1783137074.735495 mapwrite 0x013731 thru 0x022480 (0xed50 bytes) 000521 1783137074.810190 mapwrite 0x003d2b thru 0x013ac8 (0xfd9e bytes) 000522 1783137074.845368 mapwrite 0x035599 thru 0x03d1cf (0x7c37 bytes) 000523 1783137074.935849 punch from 0x03b366 to 0x03d1cf (0x1e6a bytes) 000524 1783137074.948326 punch from 0x010338 to 0x0133e5 (0x30ae bytes) 000525 1783137074.963198 read 0x01d78d thru 0x01f7cc (0x2040 bytes) 000526 1783137074.964194 mapread 0x013d74 thru 0x01d9cc (0x9c59 bytes) 000527 1783137074.978095 write 0x01ecad thru 0x0214bf (0x2813 bytes) 000528 1783137074.980909 mapread 0x001cb0 thru 0x006c1b (0x4f6c bytes) 000529 1783137074.982188 mapread 0x02d351 thru 0x036670 (0x9320 bytes) 000530 1783137075.015637 punch from 0x038b42 to 0x03d1cf (0x468e bytes) 000531 1783137075.028395 punch from 0x00426c to 0x00a592 (0x6327 bytes) 000532 1783137075.038681 trunc from 0x033b17 to 0x03d1cf (0x96b9 bytes) 000533 1783137075.048653 mapwrite 0x03c634 thru 0x03ffff (0x39cc bytes) 000534 1783137075.110686 punch from 0x03c14e to 0x03ffff (0x3eb2 bytes) 000535 1783137075.116892 mapread 0x0366b5 thru 0x03ffff (0x994b bytes) 000537 1783137075.134300 write 0x02f39f thru 0x03a01e (0xac80 bytes) 000538 1783137075.140108 mapwrite 0x020f0f thru 0x028293 (0x7385 bytes) 000539 1783137075.185401 punch from 0x014bd5 to 0x024618 (0xfa44 bytes) 000540 1783137075.191344 write 0x018d12 thru 0x02180a (0x8af9 bytes) 000541 1783137075.218329 mapwrite 0x034eeb thru 0x0397be (0x48d4 bytes) 000542 1783137075.242785 trunc from 0x024dbd to 0x02d3f4 (0x8638 bytes) 000543 1783137075.247696 fallocate 0x009f51 thru 0x00f195 (0x5245 bytes) 000545 1783137075.254142 trunc from 0x040000 to 0x02f64b (0x109b4 bytes) 000546 1783137075.276277 read 0x011ca6 thru 0x0137fd (0x1b58 bytes) 000547 1783137075.277441 read 0x008dd3 thru 0x01026b (0x7499 bytes) 000548 1783137075.289348 read 0x0172ba thru 0x021a1d (0xa764 bytes) 000549 1783137075.290077 trunc from 0x02f64c to 0x00b6eb (0x23f60 bytes) 000550 1783137075.315179 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xb6ec): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb6ec): Mirror 6: 0xe423c671 (stale) Mirror 7: 0xe423c671 (stale) Mirror 8: 0xe423c671 (stale) CRC-32 checksum value for chunk [0, 0xb6ec): Mirror 1: 0x30578b45 Mirror 9: 0x30578b45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb6ec: skipped 000551 1783137076.255062 write 0x02b3dc thru 0x033d15 (0x893a bytes) 000552 1783137076.323904 read 0x02bd6f thru 0x033d15 (0x7fa7 bytes) 000553 1783137076.324681 mapwrite 0x0067c5 thru 0x012a3c (0xc278 bytes) 000554 1783137076.435977 write 0x02ada5 thru 0x038411 (0xd66d bytes) 000555 1783137076.443904 read 0x00cc2a thru 0x00e458 (0x182f bytes) 000556 1783137076.444622 punch from 0x01ba1c to 0x025c1e (0xa203 bytes) 000557 1783137076.450388 write 0x01c1ad thru 0x024bf5 (0x8a49 bytes) 000558 1783137076.473039 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38412): Mirror 7: 0x842f913e (stale) Mirror 8: 0x842f913e (stale) Mirror 9: 0xb05bfbac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38412): Mirror 7: 0x842f913e (stale) Mirror 8: 0x842f913e (stale) Mirror 9: 0xb05bfbac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38412: skipped 000559 1783137076.953108 write 0x0378bb thru 0x03e9ae (0x70f4 bytes) 000560 1783137076.965172 write 0x028f51 thru 0x02ffa9 (0x7059 bytes) 000561 1783137076.981775 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e9af): Mirror 8: 0x83d7db3b (stale) Mirror 9: 0xfc1784d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e9af): Mirror 8: 0x83d7db3b (stale) Mirror 9: 0xfc1784d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e9af: skipped 000562 1783137077.465260 write 0x0059f6 thru 0x00ea0a (0x9015 bytes) 000563 1783137077.512066 write 0x012443 thru 0x019ebf (0x7a7d bytes) 000564 1783137077.533853 mapread 0x02ec4f thru 0x03aae4 (0xbe96 bytes) 000565 1783137077.594473 punch from 0x008c1f to 0x00ce6f (0x4251 bytes) 000566 1783137077.621053 write 0x032e60 thru 0x03cd91 (0x9f32 bytes) 000567 1783137077.625003 read 0x021324 thru 0x029c6d (0x894a bytes) 000568 1783137077.666434 mapwrite 0x00ebf4 thru 0x013981 (0x4d8e bytes) 000571 1783137077.731497 write 0x00fb43 thru 0x01e284 (0xe742 bytes) 000572 1783137077.749192 trunc from 0x00aa11 to 0x019c74 (0xf264 bytes) 000573 1783137077.775446 mapread 0x030e44 thru 0x03e2a6 (0xd463 bytes) 000574 1783137077.778996 punch from 0x004f42 to 0x00a03e (0x50fd bytes) 000575 1783137077.811528 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e9af): Mirror 8: 0x83d7db3b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e9af): Mirror 8: 0x83d7db3b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e9af: skipped 000577 1783137078.164064 trunc from 0x03e9af to 0x00bd18 (0x32c96 bytes) 000578 1783137078.257750 trunc from 0x00bd19 to 0x00258e (0x978a bytes) 000580 1783137078.292664 mapread 0x002446 thru 0x00258e (0x149 bytes) 000581 1783137078.305130 mapread 0x000842 thru 0x00258e (0x1d4d bytes) 000582 1783137078.319789 read 0x001f1c thru 0x00258e (0x673 bytes) 000583 1783137078.327861 MIRROR_RESYNC 2 mirrors 000585 1783137078.962927 read 0x000438 thru 0x00258e (0x2157 bytes) 000587 1783137078.969350 trunc from 0x00258f to 0x016e5f (0x148d1 bytes) 000589 1783137079.035829 trunc from 0x014cf2 to 0x016e5f (0x216e bytes) 000591 1783137079.042522 trunc from 0x00c4fd to 0x00d188 (0xc8c bytes) 000592 1783137079.047800 trunc from 0x016e60 to 0x02e604 (0x177a5 bytes) 000593 1783137079.072823 punch from 0x028261 to 0x02e604 (0x63a4 bytes) 000594 1783137079.078974 mapread 0x01c9e9 thru 0x02280a (0x5e22 bytes) 000595 1783137079.099280 write 0x004317 thru 0x00e18b (0x9e75 bytes) 000596 1783137079.110103 write 0x00ba13 thru 0x010491 (0x4a7f bytes) 000597 1783137079.115732 punch from 0x02bb54 to 0x02e604 (0x2ab1 bytes) 000599 1783137079.121795 trunc from 0x02e605 to 0x03dc39 (0xf635 bytes) 000600 1783137079.145434 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3dc3a): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc3a): Mirror 8: 0xc62d4078 (stale) CRC-32 checksum value for chunk [0, 0x3dc3a): Mirror 1: 0xc54a4477 Mirror 9: 0xc54a4477 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc3a: skipped 000601 1783137080.371142 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3dc3a): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc3a): Mirror 8: 0xc62d4078 (stale) CRC-32 checksum value for chunk [0, 0x3dc3a): Mirror 1: 0xc54a4477 Mirror 9: 0xc54a4477 Mirror 10: 0xc54a4477 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc3a: skipped 000602 1783137081.631869 write 0x00a9eb thru 0x012b41 (0x8157 bytes) 000604 1783137081.688911 trunc from 0x00f459 to 0x01f134 (0xfcdc bytes) 000605 1783137081.705274 punch from 0x01d024 to 0x023fdf (0x6fbc bytes) 000606 1783137081.715915 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3dc3a): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc3a): Mirror 8: 0xc62d4078 (stale) Mirror 9: 0xc54a4477 (stale) Mirror 10: 0xc54a4477 (stale) CRC-32 checksum value for chunk [0, 0x3dc3a): Mirror 1: 0x4399a10d Mirror 11: 0x4399a10d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc3a: skipped 000608 1783137083.069136 mapread 0x03212a thru 0x033f6a (0x1e41 bytes) 000609 1783137083.108861 write 0x003217 thru 0x00bbba (0x89a4 bytes) 000610 1783137083.164159 fallocate 0x02ce23 thru 0x0321c6 (0x53a4 bytes) 000611 1783137083.171406 punch from 0x0141a7 to 0x01e17a (0x9fd4 bytes) 000612 1783137083.177141 trunc from 0x03dc3a to 0x00e054 (0x2fbe5 bytes) 000613 1783137083.204841 read 0x0013e1 thru 0x00e054 (0xcc74 bytes) 000615 1783137083.216379 mapread 0x00129f thru 0x00e054 (0xcdb6 bytes) 000616 1783137083.218283 mapwrite 0x00496a thru 0x00ae23 (0x64ba bytes) 000617 1783137083.243981 punch from 0x005482 to 0x00971f (0x429e bytes) 000618 1783137083.250511 mapread 0x0038c3 thru 0x003e3c (0x57a bytes) 000619 1783137083.253477 mapread 0x00174b thru 0x007dc5 (0x667b bytes) 000620 1783137083.290294 fallocate 0x00c4d5 thru 0x00eb87 (0x26b3 bytes) 000621 1783137083.312372 read 0x00abbb thru 0x00eb87 (0x3fcd bytes) 000622 1783137083.332724 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xeb88): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xeb88): Mirror 8: 0x4e9c4953 (stale) Mirror 9: 0x5b1eb791 (stale) Mirror 10: 0x5b1eb791 (stale) Mirror 11: 0x4f6f2d77 (stale) CRC-32 checksum value for chunk [0, 0xeb88): Mirror 1: 0xd14af59f Mirror 12: 0xd14af59f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xeb88: skipped 000623 1783137084.572930 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0xeb88): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xeb88): Mirror 8: 0x4e9c4953 (stale) Mirror 9: 0x5b1eb791 (stale) Mirror 10: 0x5b1eb791 (stale) Mirror 11: 0x4f6f2d77 (stale) CRC-32 checksum value for chunk [0, 0xeb88): Mirror 1: 0xd14af59f Mirror 12: 0xd14af59f Mirror 13: 0xd14af59f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xeb88: skipped 000625 1783137086.294960 mapwrite 0x0043a1 thru 0x00aa62 (0x66c2 bytes) 000626 1783137086.488901 punch from 0x007db2 to 0x00eb87 (0x6dd6 bytes) 000627 1783137086.510669 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0xeb88): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xeb88): Mirror 8: 0x4e9c4953 (stale) Mirror 9: 0x5b1eb791 (stale) Mirror 10: 0x5b1eb791 (stale) Mirror 11: 0x4f6f2d77 (stale) Mirror 12: 0xd14af59f (stale) Mirror 13: 0xd14af59f (stale) CRC-32 checksum value for chunk [0, 0xeb88): Mirror 1: 0x5b7e5243 Mirror 14: 0x5b7e5243 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xeb88: skipped 000628 1783137088.430205 write 0x035fee thru 0x03ffff (0xa012 bytes) 000629 1783137088.543244 read 0x00463b thru 0x0113c9 (0xcd8f bytes) 000631 1783137088.563234 read 0x016116 thru 0x024e7a (0xed65 bytes) 000632 1783137088.596714 MIRROR_RESYNC 8 mirrors 000633 1783137092.104115 mapwrite 0x02c170 thru 0x035a23 (0x98b4 bytes) 000634 1783137092.275315 write 0x02174d thru 0x025e88 (0x473c bytes) 000635 1783137092.297371 read 0x02fbdf thru 0x03e3fd (0xe81f bytes) 000636 1783137092.322461 mapwrite 0x017a63 thru 0x01831f (0x8bd bytes) 000637 1783137092.396075 mapwrite 0x013f05 thru 0x01e090 (0xa18c bytes) 000638 1783137092.464322 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x29283a7c (stale) Mirror 9: 0x29283a7c (stale) Mirror 10: 0x29283a7c (stale) Mirror 11: 0x29283a7c (stale) Mirror 12: 0x29283a7c (stale) Mirror 13: 0x29283a7c (stale) Mirror 14: 0x29283a7c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xab0e84b3 Mirror 15: 0xab0e84b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000639 1783137093.824466 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x29283a7c (stale) Mirror 9: 0x29283a7c (stale) Mirror 11: 0x29283a7c (stale) Mirror 12: 0x29283a7c (stale) Mirror 13: 0x29283a7c (stale) Mirror 14: 0x29283a7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x29283a7c (stale) Mirror 9: 0x29283a7c (stale) Mirror 11: 0x29283a7c (stale) Mirror 12: 0x29283a7c (stale) Mirror 13: 0x29283a7c (stale) Mirror 14: 0x29283a7c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xab0e84b3 Mirror 15: 0xab0e84b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000640 1783137095.166692 fallocate 0x01ed8e thru 0x02adc6 (0xc039 bytes) 000641 1783137095.286721 mapread 0x035e94 thru 0x0370f8 (0x1265 bytes) 000642 1783137095.309818 trunc from 0x040000 to 0x010c12 (0x2f3ed bytes) 000643 1783137095.348480 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10c13): Mirror 8: 0x21429378 (stale) Mirror 11: 0x21429378 (stale) Mirror 12: 0x21429378 (stale) Mirror 13: 0x21429378 (stale) Mirror 14: 0x21429378 (stale) Mirror 15: 0x21429378 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10c13): Mirror 8: 0x21429378 (stale) Mirror 11: 0x21429378 (stale) Mirror 12: 0x21429378 (stale) Mirror 13: 0x21429378 (stale) Mirror 14: 0x21429378 (stale) Mirror 15: 0x21429378 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10c13: skipped 000644 1783137095.841398 read 0x00cd44 thru 0x010c12 (0x3ecf bytes) 000645 1783137095.851562 punch from 0x00a2ae to 0x00c5db (0x232e bytes) 000646 1783137095.857772 mapread 0x00a54f thru 0x010c12 (0x66c4 bytes) 000647 1783137095.887998 fallocate 0x021187 thru 0x025df6 (0x4c70 bytes) 000648 1783137095.911702 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25df7): Mirror 8: 0xe57e6e37 (stale) Mirror 12: 0xe57e6e37 (stale) Mirror 13: 0xe57e6e37 (stale) Mirror 14: 0xe57e6e37 (stale) Mirror 15: 0x7d3adee1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25df7): Mirror 8: 0xe57e6e37 (stale) Mirror 12: 0xe57e6e37 (stale) Mirror 13: 0xe57e6e37 (stale) Mirror 14: 0xe57e6e37 (stale) Mirror 15: 0x7d3adee1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25df7: skipped 000650 1783137096.553511 mapread 0x007de6 thru 0x012180 (0xa39b bytes) 000651 1783137096.590224 punch from 0x00aa74 to 0x0152a8 (0xa835 bytes) 000652 1783137096.595496 write 0x034abb thru 0x03ffff (0xb545 bytes) 000653 1783137096.601631 punch from 0x0267eb to 0x03631b (0xfb31 bytes) 000654 1783137096.617104 punch from 0x0024b3 to 0x010399 (0xdee7 bytes) 000655 1783137096.626612 read 0x02daa8 thru 0x03adce (0xd327 bytes) 000656 1783137096.636360 read 0x02ad8c thru 0x02b3c4 (0x639 bytes) 000658 1783137096.654549 mapread 0x01c369 thru 0x01e146 (0x1dde bytes) 000659 1783137096.655724 punch from 0x0334b9 to 0x03ffff (0xcb47 bytes) 000660 1783137096.663728 punch from 0x0049e6 to 0x00bf98 (0x75b3 bytes) 000661 1783137096.670867 fallocate 0x02767b thru 0x02aa84 (0x340a bytes) 000663 1783137096.679248 read 0x000af8 thru 0x00c5ad (0xbab6 bytes) 000664 1783137096.692890 write 0x02caa8 thru 0x03152e (0x4a87 bytes) 000665 1783137096.694985 trunc from 0x00a815 to 0x00fa66 (0x5252 bytes) 000666 1783137096.700218 mapread 0x03158f thru 0x03ffff (0xea71 bytes) 000667 1783137096.704777 read 0x01d35d thru 0x02b526 (0xe1ca bytes) 000668 1783137096.706829 punch from 0x00b1ce to 0x013d7e (0x8bb1 bytes) 000669 1783137096.719458 punch from 0x004bcc to 0x00c297 (0x76cc bytes) 000670 1783137096.732980 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x29283a7c (stale) Mirror 12: 0x29283a7c (stale) Mirror 14: 0x29283a7c (stale) Mirror 15: 0xab0e84b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x29283a7c (stale) Mirror 12: 0x29283a7c (stale) Mirror 14: 0x29283a7c (stale) Mirror 15: 0xab0e84b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000671 1783137097.323513 punch from 0x0191f1 to 0x01a590 (0x13a0 bytes) 000673 1783137097.332323 write 0x019732 thru 0x02825d (0xeb2c bytes) 000674 1783137097.354446 write 0x006928 thru 0x00d15a (0x6833 bytes) 000675 1783137097.391805 mapwrite 0x0044db thru 0x00ecea (0xa810 bytes) 000676 1783137097.473977 punch from 0x01b99c to 0x0237d1 (0x7e36 bytes) 000677 1783137097.481375 write 0x004404 thru 0x01184c (0xd449 bytes) 000678 1783137097.483192 mapwrite 0x0238dc thru 0x02ac5f (0x7384 bytes) 000679 1783137097.536304 trunc from 0x040000 to 0x020807 (0x1f7f8 bytes) 000680 1783137097.593783 write 0x0395df thru 0x039e5a (0x87c bytes) 000681 1783137097.595626 trunc from 0x039e5b to 0x01b0ac (0x1edae bytes) 000683 1783137097.635213 read 0x010ad9 thru 0x015867 (0x4d8f bytes) 000684 1783137097.636080 trunc from 0x00cf39 to 0x018789 (0xb851 bytes) 000685 1783137097.654921 write 0x01fef4 thru 0x029a4e (0x9b5b bytes) 000686 1783137097.665986 mapread 0x014836 thru 0x017a15 (0x31e0 bytes) 000687 1783137097.696145 mapwrite 0x01e3d1 thru 0x026ba5 (0x87d5 bytes) 000688 1783137097.811813 trunc from 0x01b1d3 to 0x028944 (0xd772 bytes) 000689 1783137097.854947 fallocate 0x02b002 thru 0x02e37b (0x337a bytes) 000690 1783137097.882515 mapwrite 0x025607 thru 0x032a04 (0xd3fe bytes) 000691 1783137097.986410 write 0x03dd7d thru 0x03ffff (0x2283 bytes) 000692 1783137097.987949 fallocate 0x03df58 thru 0x03ffff (0x20a8 bytes) 000693 1783137097.995504 punch from 0x00b312 to 0x00db2d (0x281c bytes) 000694 1783137098.007511 fallocate 0x00fd30 thru 0x01492a (0x4bfb bytes) 000695 1783137098.014814 trunc from 0x040000 to 0x02a378 (0x15c87 bytes) 000696 1783137098.076102 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a379): Mirror 12: 0x29f459c3 (stale) Mirror 14: 0x29f459c3 (stale) Mirror 15: 0xa58e7dcd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a379): Mirror 12: 0x29f459c3 (stale) Mirror 14: 0x29f459c3 (stale) Mirror 15: 0xa58e7dcd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a379: skipped 000697 1783137098.778513 write 0x019d23 thru 0x024086 (0xa364 bytes) 000698 1783137098.831557 fallocate 0x000474 thru 0x00ff7c (0xfb09 bytes) 000699 1783137098.873735 fallocate 0x017222 thru 0x02617e (0xef5d bytes) 000700 1783137098.879270 trunc from 0x0214b3 to 0x0249d2 (0x3520 bytes) 000702 1783137098.924186 trunc from 0x016ffd to 0x020594 (0x9598 bytes) 000703 1783137098.942199 mapwrite 0x025a59 thru 0x026968 (0xf10 bytes) 000704 1783137098.993354 fallocate 0x003075 thru 0x00fbc9 (0xcb55 bytes) 000705 1783137099.001300 trunc from 0x027123 to 0x02a378 (0x3256 bytes) 000706 1783137099.012290 trunc from 0x0090ae to 0x00f3b9 (0x630c bytes) 000708 1783137099.034789 punch from 0x012973 to 0x01e7d5 (0xbe63 bytes) 000709 1783137099.044323 write 0x0268ca thru 0x02ea8e (0x81c5 bytes) 000710 1783137099.049145 mapwrite 0x00f71f thru 0x017eb2 (0x8794 bytes) 000711 1783137099.143653 trunc from 0x01290c to 0x01b917 (0x900c bytes) 000712 1783137099.158112 trunc from 0x01ec5b to 0x02b019 (0xc3bf bytes) 000713 1783137099.178256 punch from 0x016204 to 0x017191 (0xf8e bytes) 000714 1783137099.222418 punch from 0x00fce5 to 0x0165d5 (0x68f1 bytes) 000715 1783137099.252681 read 0x005804 thru 0x00d993 (0x8190 bytes) 000716 1783137099.264643 mapread 0x011b3a thru 0x013511 (0x19d8 bytes) 000717 1783137099.299682 mapread 0x0110ca thru 0x020320 (0xf257 bytes) 000718 1783137099.340188 trunc from 0x0170c2 to 0x020bf8 (0x9b37 bytes) 000719 1783137099.355739 write 0x01922d thru 0x0241e4 (0xafb8 bytes) 000720 1783137099.399192 trunc from 0x016eea to 0x02195f (0xaa76 bytes) 000721 1783137099.430623 trunc from 0x02ea8f to 0x01dbfc (0x10e92 bytes) 000722 1783137099.480337 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dbfd): Mirror 12: 0xdcc80be3 (stale) Mirror 15: 0xf9054c06 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dbfd): Mirror 12: 0xdcc80be3 (stale) Mirror 15: 0xf9054c06 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dbfd: skipped 000723 1783137099.998556 write 0x017511 thru 0x01bc2d (0x471d bytes) 000724 1783137100.016526 write 0x0089ee thru 0x00b8cd (0x2ee0 bytes) 000725 1783137100.029120 punch from 0x01c9da to 0x01dbfc (0x1223 bytes) 000727 1783137100.037470 read 0x01658a thru 0x01678d (0x204 bytes) 000729 1783137100.044481 trunc from 0x01dbfd to 0x02af04 (0xd308 bytes) 000730 1783137100.064490 trunc from 0x016b33 to 0x017508 (0x9d6 bytes) 000731 1783137100.084608 read 0x028ec4 thru 0x02af04 (0x2041 bytes) 000732 1783137100.102748 write 0x039f78 thru 0x03ffff (0x6088 bytes) 000734 1783137100.114086 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x29283a7c (stale) Mirror 15: 0xab0e84b3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2be9e3b2 Mirror 16: 0x2be9e3b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000736 1783137101.959132 mapread 0x02e32c thru 0x03019f (0x1e74 bytes) 000738 1783137102.019458 trunc from 0x032f81 to 0x03ef33 (0xbfb3 bytes) 000739 1783137102.096807 read 0x00f482 thru 0x01098e (0x150d bytes) 000741 1783137102.115502 fallocate 0x0191dc thru 0x026e12 (0xdc37 bytes) 000742 1783137102.132719 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x29283a7c (stale) Mirror 15: 0xab0e84b3 (stale) Mirror 16: 0x2be9e3b2 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x64ad035e Mirror 17: 0x64ad035e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000743 1783137103.658860 mapread 0x03de20 thru 0x03ffff (0x21e0 bytes) 000744 1783137103.726049 fallocate 0x000e72 thru 0x0086bd (0x784c bytes) 000745 1783137103.786914 write 0x004f8d thru 0x008d63 (0x3dd7 bytes) 000747 1783137103.803854 trunc from 0x040000 to 0x0041c1 (0x3be3e bytes) 000748 1783137103.826861 mapwrite 0x03f4e6 thru 0x03ffff (0xb1a bytes) 000749 1783137103.963020 trunc from 0x040000 to 0x0390e7 (0x6f18 bytes) 000750 1783137104.011691 write 0x0012dc thru 0x0101cc (0xeef1 bytes) 000751 1783137104.051798 read 0x026299 thru 0x032b7d (0xc8e5 bytes) 000753 1783137104.078647 trunc from 0x0390e8 to 0x02a2fb (0xedec bytes) 000754 1783137104.112189 fallocate 0x01b2d5 thru 0x028c40 (0xd96c bytes) 000755 1783137104.140087 MIRROR_RESYNC 5 mirrors 000756 1783137105.406403 mapread 0x003797 thru 0x0118a0 (0xe10a bytes) 000757 1783137105.469501 mapread 0x011486 thru 0x01bcb7 (0xa832 bytes) 000759 1783137105.494352 mapread 0x00f5df thru 0x0125e9 (0x300b bytes) 000760 1783137105.495036 write 0x031dd6 thru 0x03b581 (0x97ac bytes) 000761 1783137105.567114 write 0x02d201 thru 0x03c00b (0xee0b bytes) 000762 1783137105.581076 mapread 0x017fda thru 0x01c13a (0x4161 bytes) 000763 1783137105.619676 write 0x001db3 thru 0x008da9 (0x6ff7 bytes) 000764 1783137105.634993 punch from 0x01256f to 0x015e99 (0x392b bytes) 000765 1783137105.642875 punch from 0x00fdf4 to 0x011421 (0x162e bytes) 000766 1783137105.647932 read 0x017c9e thru 0x02335e (0xb6c1 bytes) 000767 1783137105.653929 write 0x02a3e9 thru 0x0310a4 (0x6cbc bytes) 000768 1783137105.660219 write 0x012ab3 thru 0x012f5c (0x4aa bytes) 000769 1783137105.664618 trunc from 0x03c00c to 0x029c6c (0x1239f bytes) 000770 1783137105.683772 mapread 0x019dbc thru 0x02028b (0x64d0 bytes) 000771 1783137105.685720 mapwrite 0x03d11a thru 0x03f984 (0x286b bytes) 000772 1783137105.744550 mapread 0x014c34 thru 0x017aa5 (0x2e72 bytes) 000774 1783137105.761519 fallocate 0x004c86 thru 0x0080da (0x3455 bytes) 000775 1783137105.766984 trunc from 0x03f985 to 0x01beb4 (0x23ad0 bytes) 000776 1783137105.789538 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1beb5): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1beb5): Mirror 12: 0xc90131be (stale) Mirror 15: 0xc90131be (stale) Mirror 16: 0xc90131be (stale) Mirror 17: 0xc90131be (stale) CRC-32 checksum value for chunk [0, 0x1beb5): Mirror 1: 0x8691355 Mirror 18: 0x8691355 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1beb5: skipped 000777 1783137106.912483 trunc from 0x01beb5 to 0x01289b (0x9619 bytes) 000778 1783137107.049635 trunc from 0x01289c to 0x00d198 (0x5703 bytes) 000779 1783137107.105704 read 0x00704b thru 0x009e2e (0x2de4 bytes) 000780 1783137107.127992 MIRROR_RESYNC 6 mirrors 000782 1783137108.371247 trunc from 0x00d199 to 0x00c61b (0xb7d bytes) 000783 1783137108.440196 mapwrite 0x03a4d9 thru 0x03c354 (0x1e7c bytes) 000784 1783137108.512644 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3c355): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c355): Mirror 12: 0x1a394aaa (stale) Mirror 15: 0x1a394aaa (stale) Mirror 16: 0x1a394aaa (stale) Mirror 17: 0x1a394aaa (stale) Mirror 18: 0x1a394aaa (stale) CRC-32 checksum value for chunk [0, 0x3c355): Mirror 1: 0xb65a5b42 Mirror 19: 0xb65a5b42 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c355: skipped 000785 1783137109.751686 mapwrite 0x019a74 thru 0x01b041 (0x15ce bytes) 000786 1783137109.914562 trunc from 0x003a6f to 0x008f1a (0x54ac bytes) 000787 1783137109.930376 read 0x00e1eb thru 0x01b022 (0xce38 bytes) 000788 1783137109.952256 read 0x004817 thru 0x008413 (0x3bfd bytes) 000789 1783137109.973149 trunc from 0x03c355 to 0x023b1e (0x18836 bytes) 000790 1783137110.007341 trunc from 0x023b1f to 0x03512c (0x1160e bytes) 000791 1783137110.052524 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3512d): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3512d): Mirror 12: 0xc4b917ef (stale) Mirror 15: 0xc4b917ef (stale) Mirror 16: 0xc4b917ef (stale) Mirror 17: 0xc4b917ef (stale) Mirror 18: 0xc4b917ef (stale) Mirror 19: 0xe01c945a (stale) CRC-32 checksum value for chunk [0, 0x3512d): Mirror 1: 0xc4ea4da Mirror 20: 0xc4ea4da lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3512d: skipped 000792 1783137111.837168 punch from 0x015f0c to 0x01c83e (0x6933 bytes) 000793 1783137111.943256 trunc from 0x0293f3 to 0x03512c (0xbd3a bytes) 000794 1783137111.952705 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3512d): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3512d): Mirror 12: 0xc4b917ef (stale) Mirror 15: 0xc4b917ef (stale) Mirror 16: 0xc4b917ef (stale) Mirror 17: 0xc4b917ef (stale) Mirror 18: 0xc4b917ef (stale) Mirror 19: 0xe01c945a (stale) Mirror 20: 0xc4ea4da (stale) CRC-32 checksum value for chunk [0, 0x3512d): Mirror 1: 0xdf224954 Mirror 21: 0xdf224954 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3512d: skipped 000795 1783137113.691419 fallocate 0x006c9c thru 0x00c123 (0x5488 bytes) 000796 1783137113.766016 fallocate 0x00c946 thru 0x00f26c (0x2927 bytes) 000797 1783137113.779484 mapwrite 0x00d9e7 thru 0x013831 (0x5e4b bytes) 000798 1783137113.860552 mapread 0x02b675 thru 0x03220b (0x6b97 bytes) 000799 1783137113.899121 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3512d): Mirror 15: 0xc4b917ef (stale) Mirror 16: 0xc4b917ef (stale) Mirror 17: 0xc4b917ef (stale) Mirror 18: 0xc4b917ef (stale) Mirror 19: 0xe01c945a (stale) Mirror 20: 0xc4ea4da (stale) Mirror 21: 0xdf224954 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3512d): Mirror 15: 0xc4b917ef (stale) Mirror 16: 0xc4b917ef (stale) Mirror 17: 0xc4b917ef (stale) Mirror 18: 0xc4b917ef (stale) Mirror 19: 0xe01c945a (stale) Mirror 20: 0xc4ea4da (stale) Mirror 21: 0xdf224954 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3512d: skipped 000800 1783137114.713166 fallocate 0x00c964 thru 0x00c970 (0xd bytes) 000801 1783137114.728836 fallocate 0x01be54 thru 0x02ba4a (0xfbf7 bytes) 000802 1783137114.741700 write 0x014629 thru 0x01beeb (0x78c3 bytes) 000804 1783137114.786113 read 0x01fe2e thru 0x022c01 (0x2dd4 bytes) 000805 1783137114.803880 trunc from 0x01714d to 0x01dfbf (0x6e73 bytes) 000806 1783137114.824778 trunc from 0x000515 to 0x003df1 (0x38dd bytes) 000808 1783137114.835539 trunc from 0x03512d to 0x0188f2 (0x1c83a bytes) 000809 1783137114.864898 punch from 0x00a0de to 0x00b59a (0x14bd bytes) 000810 1783137114.873735 mapread 0x001968 thru 0x0074b9 (0x5b52 bytes) 000811 1783137114.907891 trunc from 0x0188f3 to 0x02a681 (0x11d8f bytes) 000812 1783137114.958754 fallocate 0x002efe thru 0x004d07 (0x1e0a bytes) 000813 1783137114.970565 fallocate 0x02317b thru 0x023d15 (0xb9b bytes) 000814 1783137114.990392 trunc from 0x02a682 to 0x02d8fa (0x3279 bytes) 000815 1783137115.026895 write 0x013e93 thru 0x0208c6 (0xca34 bytes) 000817 1783137115.060233 fallocate 0x02d94d thru 0x038460 (0xab14 bytes) 000818 1783137115.072017 punch from 0x002856 to 0x005e6b (0x3616 bytes) 000819 1783137115.089780 write 0x00179a thru 0x0116d2 (0xff39 bytes) 000820 1783137115.111245 mapread 0x01f389 thru 0x02c447 (0xd0bf bytes) 000821 1783137115.164376 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d8fb): Mirror 16: 0x2d31fb45 (stale) Mirror 17: 0x2d31fb45 (stale) Mirror 18: 0x2d31fb45 (stale) Mirror 19: 0xc755500c (stale) Mirror 20: 0x8d80ae6c (stale) Mirror 21: 0xf946f8e3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d8fb): Mirror 16: 0x2d31fb45 (stale) Mirror 17: 0x2d31fb45 (stale) Mirror 18: 0x2d31fb45 (stale) Mirror 19: 0xc755500c (stale) Mirror 20: 0x8d80ae6c (stale) Mirror 21: 0xf946f8e3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d8fb: skipped 000822 1783137116.185029 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d8fb): Mirror 17: 0x2d31fb45 (stale) Mirror 18: 0x2d31fb45 (stale) Mirror 19: 0xc755500c (stale) Mirror 20: 0x8d80ae6c (stale) Mirror 21: 0xf946f8e3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d8fb): Mirror 17: 0x2d31fb45 (stale) Mirror 18: 0x2d31fb45 (stale) Mirror 19: 0xc755500c (stale) Mirror 20: 0x8d80ae6c (stale) Mirror 21: 0xf946f8e3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d8fb: skipped 000823 1783137116.981453 write 0x033272 thru 0x03572f (0x24be bytes) 000824 1783137117.004176 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35730): Mirror 18: 0xf2c52b8f (stale) Mirror 19: 0x5f199970 (stale) Mirror 20: 0x23e1abc8 (stale) Mirror 21: 0xee8dbfb5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35730): Mirror 18: 0xf2c52b8f (stale) Mirror 19: 0x5f199970 (stale) Mirror 20: 0x23e1abc8 (stale) Mirror 21: 0xee8dbfb5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35730: skipped 000825 1783137117.712098 mapwrite 0x0042e8 thru 0x00df96 (0x9caf bytes) 000826 1783137117.805632 fallocate 0x023ce0 thru 0x0327d4 (0xeaf5 bytes) 000827 1783137117.816328 trunc from 0x033b98 to 0x03572f (0x1b98 bytes) 000828 1783137117.826665 fallocate 0x0206fe thru 0x0265e1 (0x5ee4 bytes) 000829 1783137117.841339 MIRROR_RESYNC 5 mirrors 000830 1783137119.734491 write 0x0043d8 thru 0x006504 (0x212d bytes) 000832 1783137119.771957 write 0x01baa0 thru 0x028d40 (0xd2a1 bytes) 000833 1783137119.783563 read 0x023525 thru 0x029c37 (0x6713 bytes) 000834 1783137119.789236 punch from 0x00dd47 to 0x011431 (0x36eb bytes) 000835 1783137119.795469 mapread 0x01807c thru 0x026a52 (0xe9d7 bytes) 000836 1783137119.822060 mapwrite 0x0153f9 thru 0x0194b8 (0x40c0 bytes) 000837 1783137119.864483 mapwrite 0x03fd39 thru 0x03ffff (0x2c7 bytes) 000838 1783137119.984298 punch from 0x03636a to 0x03ffff (0x9c96 bytes) 000839 1783137119.991683 fallocate 0x007691 thru 0x00e101 (0x6a71 bytes) 000840 1783137119.999241 read 0x021644 thru 0x0239de (0x239b bytes) 000841 1783137119.999909 punch from 0x02b7c6 to 0x02dd44 (0x257f bytes) 000842 1783137120.013205 trunc from 0x020122 to 0x02b574 (0xb453 bytes) 000844 1783137120.033172 mapwrite 0x029eeb thru 0x02d7be (0x38d4 bytes) 000845 1783137120.076528 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x421cde7e (stale) Mirror 19: 0x421cde7e (stale) Mirror 20: 0x421cde7e (stale) Mirror 21: 0x421cde7e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xac5d929d Mirror 22: 0xac5d929d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000846 1783137121.606377 punch from 0x00f152 to 0x019781 (0xa630 bytes) 000847 1783137121.700767 write 0x00fac7 thru 0x01f589 (0xfac3 bytes) 000848 1783137121.722320 mapwrite 0x01a5c9 thru 0x02538f (0xadc7 bytes) 000849 1783137121.804205 mapwrite 0x02dbe2 thru 0x0397ed (0xbc0c bytes) 000850 1783137121.867330 MIRROR_RESYNC 6 mirrors 000851 1783137124.307936 trunc from 0x014304 to 0x01e3fd (0xa0fa bytes) 000852 1783137124.407776 punch from 0x016aa1 to 0x018ff9 (0x2559 bytes) 000853 1783137124.422379 read 0x02fcd4 thru 0x038d06 (0x9033 bytes) 000854 1783137124.452148 trunc from 0x040000 to 0x010c98 (0x2f367 bytes) 000856 1783137124.492228 trunc from 0x010c99 to 0x02fe59 (0x1f1c1 bytes) 000857 1783137124.533108 punch from 0x01819e to 0x0262fb (0xe15e bytes) 000858 1783137124.545036 trunc from 0x00ab86 to 0x00d34f (0x27ca bytes) 000859 1783137124.553190 write 0x02081d thru 0x02d920 (0xd104 bytes) 000861 1783137124.577353 trunc from 0x02fe5a to 0x008062 (0x27df7 bytes) 000862 1783137124.607327 mapread 0x003432 thru 0x008062 (0x4c31 bytes) 000863 1783137124.639188 mapwrite 0x02eb69 thru 0x034332 (0x57ca bytes) 000864 1783137124.710874 trunc from 0x034333 to 0x022648 (0x11cea bytes) 000865 1783137124.750556 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x22649): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22649): Mirror 18: 0x5506d14 (stale) Mirror 19: 0x5506d14 (stale) Mirror 20: 0x5506d14 (stale) Mirror 21: 0x5506d14 (stale) Mirror 22: 0x5506d14 (stale) CRC-32 checksum value for chunk [0, 0x22649): Mirror 1: 0xab67fc32 Mirror 23: 0xab67fc32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22649: skipped 000866 1783137126.337315 punch from 0x006687 to 0x00c234 (0x5bae bytes) 000867 1783137126.409106 fallocate 0x01d226 thru 0x023181 (0x5f5c bytes) 000868 1783137126.419546 mapread 0x00d950 thru 0x015506 (0x7bb7 bytes) 000870 1783137126.470339 mapwrite 0x01f0dc thru 0x01f666 (0x58b bytes) 000871 1783137126.548553 punch from 0x01f702 to 0x02010b (0xa0a bytes) 000872 1783137126.562112 fallocate 0x02a14d thru 0x02ea74 (0x4928 bytes) 000873 1783137126.599754 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ea75): Mirror 18: 0xcc8ea788 (stale) Mirror 19: 0xcc8ea788 (stale) Mirror 20: 0xcc8ea788 (stale) Mirror 21: 0xcc8ea788 (stale) Mirror 22: 0xcc8ea788 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ea75): Mirror 18: 0xcc8ea788 (stale) Mirror 19: 0xcc8ea788 (stale) Mirror 20: 0xcc8ea788 (stale) Mirror 21: 0xcc8ea788 (stale) Mirror 22: 0xcc8ea788 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ea75: skipped 000874 1783137127.281541 mapread 0x0151ca thru 0x016eb5 (0x1cec bytes) 000875 1783137127.298240 write 0x00baf5 thru 0x0188e4 (0xcdf0 bytes) 000876 1783137127.316498 read 0x02b54a thru 0x02ea74 (0x352b bytes) 000877 1783137127.330431 punch from 0x012539 to 0x01bd21 (0x97e9 bytes) 000878 1783137127.353620 trunc from 0x02ea75 to 0x017fde (0x16a96 bytes) 000880 1783137127.386951 trunc from 0x0009b3 to 0x000ce2 (0x330 bytes) 000881 1783137127.414749 MIRROR_RESYNC 6 mirrors 000882 1783137129.150865 trunc from 0x017fdf to 0x028c15 (0x10c37 bytes) 000883 1783137129.243157 fallocate 0x008744 thru 0x00e2a4 (0x5b61 bytes) 000884 1783137129.266068 write 0x027048 thru 0x028387 (0x1340 bytes) 000885 1783137129.285621 mapread 0x0098f6 thru 0x013fcf (0xa6da bytes) 000886 1783137129.342653 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28c16): Mirror 19: 0xd8b0bb92 (stale) Mirror 20: 0xd8b0bb92 (stale) Mirror 21: 0xd8b0bb92 (stale) Mirror 22: 0xd8b0bb92 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28c16): Mirror 19: 0xd8b0bb92 (stale) Mirror 20: 0xd8b0bb92 (stale) Mirror 21: 0xd8b0bb92 (stale) Mirror 22: 0xd8b0bb92 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28c16: skipped 000887 1783137129.928913 punch from 0x013ee8 to 0x01ce83 (0x8f9c bytes) 000888 1783137129.963925 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x28c16): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28c16): Mirror 19: 0xd8b0bb92 (stale) Mirror 20: 0xd8b0bb92 (stale) Mirror 21: 0xd8b0bb92 (stale) Mirror 22: 0xd8b0bb92 (stale) CRC-32 checksum value for chunk [0, 0x28c16): Mirror 1: 0x91a6b585 Mirror 23: 0x91a6b585 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28c16: skipped 000889 1783137131.302983 mapwrite 0x03df70 thru 0x03ffff (0x2090 bytes) 000890 1783137131.429706 trunc from 0x040000 to 0x001b74 (0x3e48b bytes) 000891 1783137131.458167 trunc from 0x001b75 to 0x00ecbe (0xd14a bytes) 000892 1783137131.493692 read 0x00816c thru 0x009385 (0x121a bytes) 000893 1783137131.503830 fallocate 0x006631 thru 0x00cc71 (0x6641 bytes) 000894 1783137131.509043 mapread 0x0055ac thru 0x007303 (0x1d58 bytes) 000895 1783137131.526753 write 0x020e75 thru 0x02f84f (0xe9db bytes) 000896 1783137131.531387 trunc from 0x02f850 to 0x024397 (0xb4b8 bytes) 000897 1783137131.552539 trunc from 0x010b10 to 0x01f498 (0xe989 bytes) 000898 1783137131.560028 write 0x03fba4 thru 0x03ffff (0x45c bytes) 000899 1783137131.561324 punch from 0x02c634 to 0x033826 (0x71f3 bytes) 000900 1783137131.569987 punch from 0x035065 to 0x03e5b6 (0x9552 bytes) 000901 1783137131.577636 mapwrite 0x000771 thru 0x00dc36 (0xd4c6 bytes) 000903 1783137131.682803 read 0x007d82 thru 0x013fd5 (0xc254 bytes) 000904 1783137131.695526 punch from 0x01484d to 0x01f161 (0xa915 bytes) 000905 1783137131.702095 read 0x035d76 thru 0x03803a (0x22c5 bytes) 000907 1783137131.703522 trunc from 0x040000 to 0x0203a9 (0x1fc56 bytes) 000908 1783137131.748183 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x203aa): Mirror 19: 0x7f421eaa (stale) Mirror 20: 0x7f421eaa (stale) Mirror 22: 0x7f421eaa (stale) Mirror 23: 0x7f421eaa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x203aa): Mirror 19: 0x7f421eaa (stale) Mirror 20: 0x7f421eaa (stale) Mirror 22: 0x7f421eaa (stale) Mirror 23: 0x7f421eaa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x203aa: skipped 000910 1783137132.500826 punch from 0x00a479 to 0x0155ad (0xb135 bytes) 000911 1783137132.506883 read 0x004b35 thru 0x00579c (0xc68 bytes) 000912 1783137132.516225 trunc from 0x0203aa to 0x013025 (0xd384 bytes) 000913 1783137132.546513 read 0x00ae5a thru 0x013025 (0x81cc bytes) 000914 1783137132.557550 mapwrite 0x020e8b thru 0x02a78e (0x9904 bytes) 000915 1783137132.633884 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2a78f): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a78f): Mirror 19: 0x20fdcc6f (stale) Mirror 20: 0x20fdcc6f (stale) Mirror 22: 0x20fdcc6f (stale) Mirror 23: 0xa53a7a66 (stale) CRC-32 checksum value for chunk [0, 0x2a78f): Mirror 1: 0x169503d0 Mirror 24: 0x169503d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a78f: skipped 000916 1783137134.116372 MIRROR_RESYNC 6 mirrors 000917 1783137135.559819 trunc from 0x0098ba to 0x0113bd (0x7b04 bytes) 000918 1783137135.642657 write 0x0363dc thru 0x0399a9 (0x35ce bytes) 000919 1783137135.644018 trunc from 0x0399aa to 0x0302f2 (0x96b7 bytes) 000921 1783137135.673324 write 0x0144a9 thru 0x019bab (0x5703 bytes) 000922 1783137135.684938 fallocate 0x0366d5 thru 0x03ffff (0x992b bytes) 000923 1783137135.690414 trunc from 0x0302f3 to 0x007326 (0x28fcc bytes) 000924 1783137135.708952 mapwrite 0x00250a thru 0x00995b (0x7452 bytes) 000925 1783137135.775708 read 0x000a1f thru 0x00995b (0x8f3d bytes) 000926 1783137135.782093 punch from 0x004f33 to 0x00995b (0x4a29 bytes) 000927 1783137135.788491 punch from 0x00301f to 0x008403 (0x53e5 bytes) 000928 1783137135.795358 punch from 0x0051c3 to 0x00995b (0x4799 bytes) 000929 1783137135.800770 trunc from 0x00995c to 0x0357ea (0x2be8f bytes) 000930 1783137135.818369 punch from 0x0232c2 to 0x02a18e (0x6ecd bytes) 000931 1783137135.824157 punch from 0x01b5a9 to 0x0281aa (0xcc02 bytes) 000932 1783137135.829670 write 0x026b2f thru 0x02c7b1 (0x5c83 bytes) 000933 1783137135.850374 read 0x024a63 thru 0x033ed7 (0xf475 bytes) 000934 1783137135.874098 trunc from 0x0357eb to 0x03299d (0x2e4d bytes) 000935 1783137135.911039 trunc from 0x03299e to 0x021f73 (0x10a2a bytes) 000936 1783137135.936488 mapwrite 0x00617c thru 0x01503c (0xeec1 bytes) 000937 1783137135.965095 punch from 0x015836 to 0x015eb2 (0x67d bytes) 000938 1783137135.971363 mapwrite 0x0385ea thru 0x03e2e8 (0x5cff bytes) 000939 1783137136.023164 trunc from 0x03e2e9 to 0x01b038 (0x232b0 bytes) 000940 1783137136.043716 mapread 0x002430 thru 0x00b208 (0x8dd9 bytes) 000942 1783137136.052097 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b039): Mirror 19: 0x329e04e2 (stale) Mirror 20: 0x329e04e2 (stale) Mirror 22: 0x329e04e2 (stale) Mirror 24: 0x329e04e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b039): Mirror 19: 0x329e04e2 (stale) Mirror 20: 0x329e04e2 (stale) Mirror 22: 0x329e04e2 (stale) Mirror 24: 0x329e04e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b039: skipped 000943 1783137136.553322 punch from 0x001562 to 0x00df1d (0xc9bc bytes) 000944 1783137136.572221 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b039): Mirror 19: 0x329e04e2 (stale) Mirror 20: 0x329e04e2 (stale) Mirror 22: 0x329e04e2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b039): Mirror 19: 0x329e04e2 (stale) Mirror 20: 0x329e04e2 (stale) Mirror 22: 0x329e04e2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b039: skipped 000945 1783137136.967317 trunc from 0x01b039 to 0x013f28 (0x7110 bytes) 000946 1783137136.994228 fallocate 0x005b11 thru 0x010ad2 (0xafc2 bytes) 000947 1783137137.000026 trunc from 0x00aad4 to 0x00ae31 (0x35e bytes) 000948 1783137137.010640 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x13f29): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13f29): Mirror 19: 0x6bf7d9b9 (stale) Mirror 20: 0x6bf7d9b9 (stale) Mirror 22: 0x6bf7d9b9 (stale) CRC-32 checksum value for chunk [0, 0x13f29): Mirror 1: 0x3f335229 Mirror 23: 0x3f335229 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13f29: skipped 000949 1783137138.270792 mapwrite 0x038b34 thru 0x03ffff (0x74cc bytes) 000950 1783137138.421928 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xfdbca664 (stale) Mirror 20: 0xfdbca664 (stale) Mirror 22: 0xfdbca664 (stale) Mirror 23: 0xdea8bf54 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x63e615c1 Mirror 24: 0x63e615c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000951 1783137139.621546 punch from 0x02c7aa to 0x030b84 (0x43db bytes) 000952 1783137139.759369 mapread 0x01e686 thru 0x02cd37 (0xe6b2 bytes) 000953 1783137139.865329 mapread 0x032882 thru 0x03c9c8 (0xa147 bytes) 000954 1783137139.980445 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xfdbca664 (stale) Mirror 20: 0xfdbca664 (stale) Mirror 22: 0xfdbca664 (stale) Mirror 23: 0xdea8bf54 (stale) Mirror 24: 0x63e615c1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x63e615c1 Mirror 25: 0x63e615c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000955 1783137141.441401 mapread 0x018365 thru 0x024f60 (0xcbfc bytes) 000956 1783137141.545121 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 19: 0xfdbca664 (stale) Mirror 20: 0xfdbca664 (stale) Mirror 22: 0xfdbca664 (stale) Mirror 23: 0xdea8bf54 (stale) Mirror 24: 0x63e615c1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x63e615c1 Mirror 25: 0x63e615c1 Mirror 26: 0x63e615c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000957 1783137142.793361 write 0x006fb9 thru 0x0102cf (0x9317 bytes) 000958 1783137142.902516 trunc from 0x040000 to 0x01e4a3 (0x21b5c bytes) 000959 1783137142.968546 fallocate 0x00c437 thru 0x0129aa (0x6574 bytes) 000960 1783137142.974853 read 0x00a58a thru 0x0199a1 (0xf418 bytes) 000961 1783137143.001650 mapread 0x0119af thru 0x01cb19 (0xb16b bytes) 000962 1783137143.046396 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1e4a4): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e4a4): Mirror 19: 0x35af3fa8 (stale) Mirror 20: 0x35af3fa8 (stale) Mirror 22: 0x35af3fa8 (stale) Mirror 23: 0x1de551a0 (stale) Mirror 24: 0x1de551a0 (stale) Mirror 25: 0x1de551a0 (stale) Mirror 26: 0x1de551a0 (stale) CRC-32 checksum value for chunk [0, 0x1e4a4): Mirror 1: 0x67ffe04 Mirror 27: 0x67ffe04 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e4a4: skipped 000963 1783137144.528543 read 0x0148c6 thru 0x01ca3f (0x817a bytes) 000965 1783137144.564213 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x1e4a4): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e4a4): Mirror 19: 0x35af3fa8 (stale) Mirror 20: 0x35af3fa8 (stale) Mirror 22: 0x35af3fa8 (stale) Mirror 23: 0x1de551a0 (stale) Mirror 24: 0x1de551a0 (stale) Mirror 25: 0x1de551a0 (stale) Mirror 26: 0x1de551a0 (stale) CRC-32 checksum value for chunk [0, 0x1e4a4): Mirror 1: 0x67ffe04 Mirror 27: 0x67ffe04 Mirror 28: 0x67ffe04 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e4a4: skipped 000966 1783137146.566975 read 0x00d709 thru 0x0111a1 (0x3a99 bytes) 000967 1783137146.616668 write 0x00cde0 thru 0x0181e1 (0xb402 bytes) 000968 1783137146.740628 read 0x00a1c0 thru 0x014344 (0xa185 bytes) 000969 1783137146.769368 read 0x00c7c4 thru 0x01c50b (0xfd48 bytes) 000970 1783137146.777779 MIRROR_RESYNC 10 mirrors 000971 1783137149.966324 MIRROR_RESYNC 10 mirrors 000972 1783137151.014066 mapwrite 0x02fbb7 thru 0x03de8d (0xe2d7 bytes) 000973 1783137151.276815 punch from 0x013db4 to 0x019dfb (0x6048 bytes) 000975 1783137151.299659 punch from 0x0129a9 to 0x01f2c6 (0xc91e bytes) 000976 1783137151.320217 mapread 0x036129 thru 0x039b33 (0x3a0b bytes) 000977 1783137151.335017 write 0x028d86 thru 0x03226a (0x94e5 bytes) 000978 1783137151.368319 mapread 0x02f771 thru 0x033d26 (0x45b6 bytes) 000979 1783137151.369650 trunc from 0x0146d6 to 0x021fe4 (0xd90f bytes) 000980 1783137151.396641 fallocate 0x039639 thru 0x03ad44 (0x170c bytes) 000981 1783137151.411052 read 0x012244 thru 0x021c8e (0xfa4b bytes) 000982 1783137151.427185 mapwrite 0x0227dd thru 0x0282dd (0x5b01 bytes) 000984 1783137151.480928 mapread 0x038913 thru 0x03ad70 (0x245e bytes) 000985 1783137151.481994 punch from 0x03b474 to 0x03de8d (0x2a1a bytes) 000986 1783137151.498355 write 0x005ff0 thru 0x010448 (0xa459 bytes) 000988 1783137151.539323 mapwrite 0x0104a9 thru 0x0136c5 (0x321d bytes) 000989 1783137151.604621 mapread 0x023cb1 thru 0x02de21 (0xa171 bytes) 000990 1783137151.609659 read 0x02a01b thru 0x02f72a (0x5710 bytes) 000991 1783137151.610426 write 0x009ae1 thru 0x0131b9 (0x96d9 bytes) 000992 1783137151.634682 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3de8e): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3de8e): Mirror 19: 0x84804263 (stale) Mirror 20: 0x84804263 (stale) Mirror 22: 0x84804263 (stale) Mirror 23: 0x84804263 (stale) Mirror 24: 0x84804263 (stale) Mirror 25: 0x84804263 (stale) Mirror 26: 0x84804263 (stale) Mirror 27: 0x84804263 (stale) Mirror 28: 0x84804263 (stale) CRC-32 checksum value for chunk [0, 0x3de8e): Mirror 1: 0xd8bc7a00 Mirror 29: 0xd8bc7a00 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3de8e: skipped 000993 1783137153.313753 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3de8e): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3de8e): Mirror 19: 0x84804263 (stale) Mirror 20: 0x84804263 (stale) Mirror 22: 0x84804263 (stale) Mirror 23: 0x84804263 (stale) Mirror 24: 0x84804263 (stale) Mirror 25: 0x84804263 (stale) Mirror 26: 0x84804263 (stale) Mirror 27: 0x84804263 (stale) Mirror 28: 0x84804263 (stale) CRC-32 checksum value for chunk [0, 0x3de8e): Mirror 1: 0xd8bc7a00 Mirror 29: 0xd8bc7a00 Mirror 30: 0xd8bc7a00 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3de8e: skipped 000994 1783137155.073888 trunc from 0x03de8e to 0x03452d (0x9960 bytes) 000995 1783137155.199759 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1e:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3452e): Mirror 20: 0x44a92d54 (stale) Mirror 22: 0x44a92d54 (stale) Mirror 23: 0x44a92d54 (stale) Mirror 24: 0x44a92d54 (stale) Mirror 25: 0x44a92d54 (stale) Mirror 26: 0x44a92d54 (stale) Mirror 27: 0x44a92d54 (stale) Mirror 28: 0x44a92d54 (stale) Mirror 29: 0x63605042 (stale) Mirror 30: 0x63605042 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3452e): Mirror 20: 0x44a92d54 (stale) Mirror 22: 0x44a92d54 (stale) Mirror 23: 0x44a92d54 (stale) Mirror 24: 0x44a92d54 (stale) Mirror 25: 0x44a92d54 (stale) Mirror 26: 0x44a92d54 (stale) Mirror 27: 0x44a92d54 (stale) Mirror 28: 0x44a92d54 (stale) Mirror 29: 0x63605042 (stale) Mirror 30: 0x63605042 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3452e: skipped 000996 1783137156.104103 write 0x03067b thru 0x039d5d (0x96e3 bytes) 000997 1783137156.128491 write 0x016da5 thru 0x01cbbe (0x5e1a bytes) 000998 1783137156.146246 trunc from 0x014bc1 to 0x0222de (0xd71e bytes) 000999 1783137156.173033 mapread 0x01d1bc thru 0x029dd4 (0xcc19 bytes) 001000 1783137156.276052 fallocate 0x02d927 thru 0x02f741 (0x1e1b bytes) All operations completed A-OK! PASS 70a (137s) == sanity-flr test 71: check mirror extend parent fid ==== 23:52:43 (1783137163) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x6b:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 23:52:54 (1783137174) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cf:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.241012 s, 8.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cf:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160793 s, 6.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d0:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cf:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d2:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d0:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2cf:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d3:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 23:53:04 (1783137184) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre2 Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4498240 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1733408 .. Extending file size to 5836224 .. Extending file size to 3990688 .. Extending file size to 6258880 .. Extending file size to 5783456 .. Extending file size to 8246176 .. Extending file size to 8784032 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8369952 .. Extending file size to 8372832 .. Extending file size to 9165984 .. Extending file size to 8548928 .. Extending file size to 8644800 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8726624 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8787904 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 61173 61174 61175 61178 61181 192.168.204.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg412-client.virtnet /mnt/lustre2 (opts:) 192.168.204.112@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg412-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:54:25 (1783137265) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre2 Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6414848 .. Extending file size to 8774240 .. Extending file size to 8147840 .. Extending file size to 8900992 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 62740 62741 62742 62745 62747 192.168.204.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg412-client.virtnet /mnt/lustre2 (opts:) 192.168.204.112@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg412-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:55:49 (1783137349) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1268860 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2020 1269284 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18528 3588492 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24200 3582820 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42728 7171312 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1268860 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2020 1269284 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18528 3588492 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24200 3582820 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42728 7171312 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114214 s, 18.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.193848 s, 10.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xa8ecaebc (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg412-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:56:09 (1783137369) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0854678 s, 24.5 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 23:56:16 (1783137376) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1ae:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e6:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e6:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e7:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e8:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:56:24 (1783137384) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139713 s, 15.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x55:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x56:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:56:33 (1783137393) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129912 s, 16.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e8:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2e9:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ea:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:56:43 (1783137403) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13942 s, 15.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2eb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ea:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2eb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ec:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:57:18 (1783137438) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg412-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527299 s, 19.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x56:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x58:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x57:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x56:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x58:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x57:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x57:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:58:14 (1783137494) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg412-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ec:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.148988 s, 14.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ec:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ec:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (51s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:59:05 (1783137545) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg412-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg412-server: Pool lustre.test_204f created Adding targets to pool oleg412-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg412-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.24133 s, 8.7 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ee:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ed:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ee:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg412-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg412-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg412-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 00:00:20 (1783137620) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 00:00:29 (1783137629) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 00:00:36 (1783137636) oleg412-server: Pool lustre.test_206 created Adding targets to pool oleg412-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg412-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg412-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg412-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg412-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 00:01:06 (1783137666) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770991 s, 13.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423189 s, 24.8 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xbaa659fe Mirror 3: 0xbaa659fe PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 00:01:19 (1783137679) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 00:01:23 (1783137683) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 00:01:27 (1783137687) Starting client: oleg412-client.virtnet: -o user_xattr,flock 192.168.204.112@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.095021 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x18b0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1e10000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1e10000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1e10000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2620000 Read this (4 bytes): BBBB 192.168.204.112@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg412-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 00:02:08 (1783137728) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0097198 s, 843 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 00:02:15 (1783137735) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0651259 s, 16.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1cf lmm_fid: [0x200000405:0x1cf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1cf lmm_fid: [0x200000405:0x1cf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f8:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611599 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-04 00:02:17.000000000 -0400 Modify: 2026-07-04 00:02:17.000000000 -0400 Change: 2026-07-04 00:02:19.000000000 -0400 Birth: 2026-07-04 00:02:17.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 00:02:25 (1783137745) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226019 s, 46.4 MB/s oleg412-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg412-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg412-server: O/280000401/d24/2f8: File not found by ext2_lookup PASS 210b (23s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 00:02:47 (1783137767) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.222195 s, 184 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0503082 s, 81.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 4 00:02 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d4:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 4 00:02 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 00:02:56 (1783137776) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 00:03:05 (1783137785) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1845 sec =========== 00:03:15 (1783137795) === sanity-flr: start cleanup 00:03:17 (1783137797) === === sanity-flr: finish cleanup 00:03:26 (1783137806) ===