-----============= acceptance-small: sanity-flr ============----- Wed Aug 19 01:41:06 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg222-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 01:41:32 (1787118092) === oleg222-client.virtnet: executing check_config_client /mnt/lustre oleg222-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg222-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d4ac5f59000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d4ac5f59000.idle_timeout=debug disable quota as required oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 01:42:16 (1787118136) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8228] == sanity-flr test 0a: lfs mirror create with -N option == 01:42:18 (1787118138) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:42:33 (1787118153) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:42:36 (1787118156) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:42:39 (1787118159) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:43:00 (1787118180) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:43:04 (1787118184) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 01:43:07 (1787118187) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8d4ac5f59000.stats=clear osc.lustre-OST0001-osc-ffff8d4ac5f59000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:43:16 (1787118196) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:43:29 (1787118209) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:43:37 (1787118217) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:43:41 (1787118221) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:43:49 (1787118229) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 01:43:57 (1787118237) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:44:04 (1787118244) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:44:13 (1787118253) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:44:20 (1787118260) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:44:29 (1787118269) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:44:35 (1787118275) 14+0 records in 14+0 records out 14680064 bytes (15 MB, 14 MiB) copied, 1.1632 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106707 s, 9.8 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:44:44 (1787118284) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.383097 s, 41.1 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 01:44:53 (1787118293) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:45:05 (1787118305) Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d4ac5f59000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d4ac5f59000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d4ac5f59000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d4ac5f59000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg222-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (80s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:46:26 (1787118386) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.525887 s, 19.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d4ac5f59000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d4ac5f59000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:47:09 (1787118429) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d4ac5f59000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d4ac5f59000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d4ac5f59000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d4ac5f59000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg222-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 01:48:31 (1787118511) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.8442 s, 18.4 MB/s -rw-r--r-- 1 root root 52428800 Aug 19 01:48 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.976s user 0m0.000s sys 0m0.173s Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8d4ac5f59000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8d4ac5f59000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.420s user 0m0.000s sys 0m0.225s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8d4ac5f59000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d4ac5f59000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.124s user 0m0.001s sys 0m0.162s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg222-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg222-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d4ac5f59000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8d4ac5f59000.ost_server_uuid in FULL state after 0 sec PASS 33b (95s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 01:50:06 (1787118606) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 01:50:10 (1787118610) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 01:50:14 (1787118614) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (5s) == sanity-flr test 35: allow to write to mirrored files == 01:50:19 (1787118619) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 01:50:27 (1787118627) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.648497 s, 19.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 01:50:41 (1787118641) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604212 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.024258 s, 43.2 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 01:50:49 (1787118649) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 01:50:58 (1787118658) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00127862 s, 801 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00111646 s, 917 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00281556 s, 364 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00113008 s, 906 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 01:51:09 (1787118669) create test files with size 15759421 .. 4d54cf0eedb7e7e9f0d47893f4cdc339 - 20f442403910d8bb27f238092a433a28 - 2dc0f3f7652cb61a7c0d1c94044e52c1 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre PASS 37 (32s) == sanity-flr test 38: resync ============================ 01:51:41 (1787118701) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 01:51:55 (1787118715) PASS 39 (10s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 01:52:04 (1787118724) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717525 s, 14.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604473 s, 17.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaa:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 01:52:14 (1787118734) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104768 s, 20.0 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.335997 s, 12.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.190596 s, 11.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 01:52:28 (1787118748) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 01:52:31 (1787118751) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 01:52:35 (1787118755) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026734 s, 39.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 01:52:43 (1787118763) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.300566 s, 10.5 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 01:52:52 (1787118772) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb4:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 01:53:02 (1787118782) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.293382 s, 35.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 01:53:13 (1787118793) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8d4ad8888000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.266735 s, 39.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8d4ac648f000.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 01:53:32 (1787118812) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 01:54:11 (1787118851) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (31s) == sanity-flr test 47: Verify mirror obj alloc =========== 01:54:43 (1787118883) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 01:54:46 (1787118886) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.164487 s, 19.1 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.198205 s, 15.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xad02d6dd (nosync,stale) Mirror 2: 0xad02d6dd (nosync,stale) ** verify mirror 2 doesn't change original checksum: 51e0df4447c0c4920b96ce3b1cc0ebeb - mirror 1 checksum: fe845984b57e820fa02330fd527cd21c - mirror 2 checksum: 51e0df4447c0c4920b96ce3b1cc0ebeb - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-19T01:54:47-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbe:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-19T01:54:47-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 51e0df4447c0c4920b96ce3b1cc0ebeb - mirror 1 checksum: fe845984b57e820fa02330fd527cd21c - mirror 2 checksum: fe845984b57e820fa02330fd527cd21c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-19T01:54:47-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbe:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-19T01:54:47-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xbf:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 01:54:55 (1787118895) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240717 s, 43.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 01:55:02 (1787118902) Failing mds1 on oleg222-server Stopping /mnt/lustre-mds1 (opts:) on oleg222-server 01:55:13 (1787118913) shut down facet: mds1 facet_host: oleg222-server facet_failover_host: oleg222-server Failover mds1 to oleg222-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 01:55:36 (1787118936) targets are mounted 01:55:36 (1787118936) facet_failover done oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0278965 s, 37.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 01:55:51 (1787118951) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0370942 s, 552 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0133344 s, 1.5 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (8s) == sanity-flr test 50b: mirror rsync handles sparseness == 01:55:59 (1787118959) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0329078 s, 622 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.008189 s, 2.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 01:56:08 (1787118968) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0051052 s, 3.2 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 01:56:15 (1787118975) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0511237 s, 401 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 01:56:21 (1787118981) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0169665 s, 241 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0309042 s, 133 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 01:56:29 (1787118989) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x18:0x0]: Invalid argument (22) PASS 60b (19s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 01:56:48 (1787119008) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 01:57:14 (1787119034) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x42:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (21s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 01:57:36 (1787119056) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (21s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 01:57:57 (1787119077) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 01:58:03 (1787119083) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57557 mirror split pid 57558 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5091: 57557 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5091: 57558 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 01:59:09 (1787119149) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 9151 skipping zero size read 000003 1787119151.422002 write 0x037b25 thru 0x03c475 (0x4951 bytes) fallocating to largest ever: 0x3c584 000004 1787119151.424396 fallocate 0x0335a1 thru 0x03c583 (0x8fe3 bytes) 000006 1787119151.430918 write 0x01efde thru 0x028406 (0x9429 bytes) 000007 1787119151.441436 fallocate 0x024469 thru 0x027ee0 (0x3a78 bytes) 000008 1787119151.447413 mapwrite 0x009d0b thru 0x01835b (0xe651 bytes) 000009 1787119151.502453 mapwrite 0x01d995 thru 0x020e66 (0x34d2 bytes) 000010 1787119151.536026 mapwrite 0x008a64 thru 0x010bb7 (0x8154 bytes) 000011 1787119151.567941 write 0x01646a thru 0x01808f (0x1c26 bytes) 000013 1787119151.571964 trunc from 0x03c584 to 0x035360 (0x7223 bytes) 000014 1787119151.593136 trunc from 0x035361 to 0x01b6ff (0x19c61 bytes) 000015 1787119151.608711 fallocate 0x0180a7 thru 0x026181 (0xe0db bytes) 000016 1787119151.616868 trunc from 0x0035ef to 0x00bd9f (0x87b1 bytes) 000017 1787119151.624011 trunc from 0x026182 to 0x0238aa (0x28d7 bytes) 000018 1787119151.648723 MIRROR_EXTEND to 2 mirrors 000019 1787119152.184278 read 0x01e7d4 thru 0x0234e2 (0x4d0f bytes) 000021 1787119152.216337 read 0x019489 thru 0x020447 (0x6fbf bytes) 000022 1787119152.237905 trunc from 0x00a860 to 0x01283d (0x7fde bytes) 000025 1787119152.281637 trunc from 0x0238ab to 0x027d6a (0x44c0 bytes) 000026 1787119152.312502 mapread 0x024eb2 thru 0x027d6a (0x2eb9 bytes) 000027 1787119152.319838 mapread 0x01e3ce thru 0x027d40 (0x9973 bytes) 000028 1787119152.350730 write 0x03ab49 thru 0x03cf41 (0x23f9 bytes) 000031 1787119152.354598 fallocate 0x01ca85 thru 0x01ec88 (0x2204 bytes) 000032 1787119152.359109 read 0x03b6c9 thru 0x03cf41 (0x1879 bytes) 000033 1787119152.364571 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3cf42): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf42): Mirror 2: 0x623aff12 (stale) CRC-32 checksum value for chunk [0, 0x3cf42): Mirror 1: 0x131ed379 Mirror 3: 0x131ed379 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf42: skipped 000034 1787119152.887019 mapwrite 0x009af7 thru 0x009e39 (0x343 bytes) 000035 1787119152.958128 read 0x01970e thru 0x019b4f (0x442 bytes) 000036 1787119152.964112 trunc from 0x03cf42 to 0x01476f (0x287d2 bytes) 000037 1787119152.997228 punch from 0x00f14f to 0x0125b6 (0x3468 bytes) 000038 1787119153.002662 read 0x012b12 thru 0x01476f (0x1c5e bytes) 000039 1787119153.012930 mapwrite 0x0188dd thru 0x01f1af (0x68d3 bytes) 000040 1787119153.056895 mapwrite 0x00429d thru 0x00c134 (0x7e98 bytes) 000041 1787119153.077682 write 0x02ccb1 thru 0x0345e1 (0x7931 bytes) 000042 1787119153.078880 punch from 0x02ed60 to 0x0304b1 (0x1752 bytes) 000043 1787119153.091091 trunc from 0x0345e2 to 0x033707 (0xeda bytes) 000044 1787119153.106921 trunc from 0x00bede to 0x00e10d (0x2230 bytes) 000045 1787119153.111679 write 0x03be9c thru 0x03ffff (0x4164 bytes) 000046 1787119153.114169 fallocate 0x034bff thru 0x039eff (0x5301 bytes) 000047 1787119153.123019 write 0x025288 thru 0x0270d6 (0x1e4f bytes) 000048 1787119153.141817 mapwrite 0x02f49d thru 0x034ffe (0x5b62 bytes) 000049 1787119153.164040 read 0x025ed4 thru 0x031adc (0xbc09 bytes) 000050 1787119153.165529 trunc from 0x040000 to 0x016fce (0x29031 bytes) 000051 1787119153.178873 read 0x00d8a4 thru 0x00e37d (0xada bytes) 000052 1787119153.185903 mapwrite 0x031c80 thru 0x03a513 (0x8894 bytes) 000053 1787119153.227771 read 0x02c3ed thru 0x0332f0 (0x6f04 bytes) 000054 1787119153.228903 punch from 0x009e62 to 0x00ccab (0x2e4a bytes) 000055 1787119153.234073 trunc from 0x0014fd to 0x001e17 (0x91b bytes) 000056 1787119153.239806 fallocate 0x00473d thru 0x00de21 (0x96e5 bytes) 000057 1787119153.243145 write 0x021f43 thru 0x0284fc (0x65ba bytes) 000059 1787119153.244424 mapwrite 0x024236 thru 0x02f4c1 (0xb28c bytes) 000060 1787119153.259779 mapwrite 0x026ef2 thru 0x0274d5 (0x5e4 bytes) 000061 1787119153.279418 punch from 0x0200a7 to 0x02b0ea (0xb044 bytes) 000062 1787119153.284763 mapread 0x017544 thru 0x02229c (0xad59 bytes) 000063 1787119153.301907 MIRROR_RESYNC 3 mirrors 000064 1787119154.182092 trunc from 0x03a514 to 0x031957 (0x8bbc bytes) 000065 1787119154.240777 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31958): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31958): Mirror 2: 0x56618146 (stale) Mirror 3: 0x56618146 (stale) CRC-32 checksum value for chunk [0, 0x31958): Mirror 1: 0x56618146 Mirror 4: 0x56618146 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31958: skipped 000066 1787119155.200019 mapwrite 0x03570b thru 0x03cb35 (0x742b bytes) 000067 1787119155.341623 MIRROR_RESYNC 4 mirrors 000068 1787119156.732671 punch from 0x012ec1 to 0x0183ae (0x54ee bytes) 000069 1787119156.785174 MIRROR_RESYNC 4 mirrors 000071 1787119158.102823 write 0x00140e thru 0x00aae2 (0x96d5 bytes) 000072 1787119158.139128 read 0x0020e5 thru 0x0059e7 (0x3903 bytes) 000073 1787119158.139981 trunc from 0x027cdd to 0x032324 (0xa648 bytes) 000074 1787119158.145909 write 0x009080 thru 0x013904 (0xa885 bytes) 000075 1787119158.153497 mapread 0x008d3c thru 0x00c2c1 (0x3586 bytes) 000076 1787119158.155472 read 0x02041d thru 0x02e1c7 (0xddab bytes) 000077 1787119158.179887 write 0x003d5d thru 0x0047a1 (0xa45 bytes) 000078 1787119158.181193 read 0x03bdd0 thru 0x03cb35 (0xd66 bytes) 000079 1787119158.183167 trunc from 0x03cb36 to 0x02cf70 (0xfbc5 bytes) 000081 1787119158.208540 write 0x023910 thru 0x032963 (0xf054 bytes) 000082 1787119158.215013 read 0x02d24f thru 0x030113 (0x2ec5 bytes) 000083 1787119158.216588 mapread 0x009800 thru 0x00b140 (0x1941 bytes) 000084 1787119158.219599 read 0x025023 thru 0x02ce9e (0x7e7c bytes) 000085 1787119158.221012 trunc from 0x030d69 to 0x032963 (0x1bfb bytes) 000086 1787119158.278086 MIRROR_RESYNC 4 mirrors 000087 1787119159.399263 read 0x018a56 thru 0x018b1d (0xc8 bytes) 000088 1787119159.403603 punch from 0x01588d to 0x01a1c8 (0x493c bytes) 000089 1787119159.431470 mapread 0x017f81 thru 0x022f75 (0xaff5 bytes) 000090 1787119159.459655 trunc from 0x032964 to 0x01c58d (0x163d6 bytes) 000091 1787119159.475635 punch from 0x00aca6 to 0x01a234 (0xf58f bytes) 000093 1787119159.482181 write 0x0180bb thru 0x01fc1a (0x7b60 bytes) 000094 1787119159.487707 trunc from 0x01fc1b to 0x02953f (0x9925 bytes) 000095 1787119159.500626 fallocate 0x01314c thru 0x01a032 (0x6ee7 bytes) 000097 1787119159.508830 trunc from 0x029540 to 0x02d5bb (0x407c bytes) 000098 1787119159.520711 mapread 0x007f5f thru 0x00debb (0x5f5d bytes) 000099 1787119159.530537 fallocate 0x0095c8 thru 0x013473 (0x9eac bytes) 000100 1787119159.535074 mapwrite 0x02c181 thru 0x0369d7 (0xa857 bytes) 000101 1787119159.576187 mapwrite 0x00e43d thru 0x0130ba (0x4c7e bytes) 000102 1787119159.603787 mapread 0x02dce2 thru 0x030298 (0x25b7 bytes) 000103 1787119159.604743 mapread 0x02b8f8 thru 0x0369d7 (0xb0e0 bytes) 000104 1787119159.608569 MIRROR_RESYNC 4 mirrors 000105 1787119160.322676 MIRROR_RESYNC 4 mirrors 000106 1787119160.653174 trunc from 0x018be3 to 0x026ae5 (0xdf03 bytes) 000107 1787119160.687768 mapwrite 0x012677 thru 0x017334 (0x4cbe bytes) 000108 1787119160.725536 mapwrite 0x01d4aa thru 0x0283e6 (0xaf3d bytes) 000109 1787119160.759438 write 0x00a4ff thru 0x00c955 (0x2457 bytes) 000110 1787119160.766863 read 0x030355 thru 0x0369d7 (0x6683 bytes) 000111 1787119160.775349 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x369d8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x369d8): Mirror 2: 0xbb7d78e0 (stale) Mirror 3: 0xbb7d78e0 (stale) Mirror 4: 0xbb7d78e0 (stale) CRC-32 checksum value for chunk [0, 0x369d8): Mirror 1: 0x9e212203 Mirror 5: 0x9e212203 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x369d8: skipped 000112 1787119161.910597 punch from 0x009c31 to 0x011431 (0x7801 bytes) 000114 1787119161.966102 write 0x03ae33 thru 0x03ffff (0x51cd bytes) 000115 1787119161.972753 mapwrite 0x039404 thru 0x03fa56 (0x6653 bytes) 000117 1787119162.008965 read 0x018a7a thru 0x01d1e0 (0x4767 bytes) 000118 1787119162.020583 fallocate 0x017150 thru 0x02178a (0xa63b bytes) 000120 1787119162.034521 mapread 0x00976f thru 0x00b619 (0x1eab bytes) 000122 1787119162.052623 fallocate 0x013cc8 thru 0x01f5c2 (0xb8fb bytes) 000124 1787119162.063727 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2eb212de (stale) Mirror 4: 0x2eb212de (stale) Mirror 5: 0x983f56ce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2eb212de (stale) Mirror 4: 0x2eb212de (stale) Mirror 5: 0x983f56ce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000125 1787119162.581580 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x2eb212de (stale) Mirror 4: 0x2eb212de (stale) Mirror 5: 0x983f56ce (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x576d8091 Mirror 6: 0x576d8091 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000126 1787119163.562099 mapwrite 0x0195cd thru 0x019b4d (0x581 bytes) 000127 1787119163.638265 trunc from 0x03ad9b to 0x03ffff (0x5265 bytes) 000128 1787119163.643769 read 0x00ba67 thru 0x0168fe (0xae98 bytes) 000129 1787119163.662326 fallocate 0x0205f5 thru 0x02dfd7 (0xd9e3 bytes) 000130 1787119163.667413 trunc from 0x031cc5 to 0x03e892 (0xcbce bytes) 000131 1787119163.673442 mapwrite 0x0322db thru 0x03f6d1 (0xd3f7 bytes) 000132 1787119163.710380 mapwrite 0x01e2e6 thru 0x01f949 (0x1664 bytes) 000133 1787119163.735922 MIRROR_RESYNC 5 mirrors 000134 1787119165.076537 punch from 0x009696 to 0x0194e2 (0xfe4d bytes) 000136 1787119165.105254 trunc from 0x040000 to 0x02236f (0x1dc90 bytes) 000137 1787119165.128750 read 0x011986 thru 0x020196 (0xe811 bytes) 000138 1787119165.140761 trunc from 0x022370 to 0x0158e7 (0xca88 bytes) 000139 1787119165.160796 punch from 0x000dfa to 0x00a7b9 (0x99c0 bytes) 000140 1787119165.168704 punch from 0x013b42 to 0x0158e7 (0x1da6 bytes) 000141 1787119165.173337 punch from 0x0023bc to 0x0103b9 (0xdffe bytes) 000142 1787119165.176106 read 0x00770d thru 0x00ec04 (0x74f8 bytes) 000143 1787119165.191830 write 0x00144d thru 0x0093ca (0x7f7e bytes) 000144 1787119165.193213 fallocate 0x012d39 thru 0x014d24 (0x1fec bytes) 000145 1787119165.198720 fallocate 0x016ff3 thru 0x01785f (0x86d bytes) 000146 1787119165.203766 write 0x038d08 thru 0x03ffff (0x72f8 bytes) 000147 1787119165.206484 write 0x006b4f thru 0x00a084 (0x3536 bytes) 000148 1787119165.208822 trunc from 0x009213 to 0x013caf (0xaa9d bytes) 000149 1787119165.224077 write 0x020c41 thru 0x02e1b7 (0xd577 bytes) 000150 1787119165.240268 mapwrite 0x00f507 thru 0x00ff6e (0xa68 bytes) 000151 1787119165.274987 write 0x00dd96 thru 0x0102de (0x2549 bytes) 000152 1787119165.287542 punch from 0x023817 to 0x032e01 (0xf5eb bytes) 000153 1787119165.293913 mapwrite 0x006059 thru 0x014adc (0xea84 bytes) 000154 1787119165.368037 trunc from 0x040000 to 0x00e394 (0x31c6b bytes) 000155 1787119165.385924 write 0x01de31 thru 0x01e4c3 (0x693 bytes) 000156 1787119165.387905 punch from 0x01cd6f to 0x01e4c3 (0x1755 bytes) 000157 1787119165.400786 fallocate 0x030589 thru 0x03bbb9 (0xb631 bytes) 000158 1787119165.406832 read 0x038e66 thru 0x03ae61 (0x1ffc bytes) 000159 1787119165.416586 fallocate 0x01fe07 thru 0x025998 (0x5b92 bytes) 000160 1787119165.422322 read 0x01b1b6 thru 0x023110 (0x7f5b bytes) 000161 1787119165.423627 trunc from 0x03bbba to 0x026e24 (0x14d95 bytes) 000162 1787119165.451595 punch from 0x017c51 to 0x024c14 (0xcfc4 bytes) 000163 1787119165.456892 punch from 0x0169d7 to 0x024eca (0xe4f4 bytes) 000164 1787119165.462045 trunc from 0x01f9f9 to 0x026e24 (0x742c bytes) 000165 1787119165.467739 punch from 0x023e91 to 0x026e24 (0x2f94 bytes) 000166 1787119165.472098 fallocate 0x01ae18 thru 0x029e7c (0xf065 bytes) 000167 1787119165.477444 trunc from 0x01e0fe to 0x026e24 (0x8d27 bytes) 000168 1787119165.481801 punch from 0x01ca67 to 0x01dfb2 (0x154c bytes) 000169 1787119165.489639 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x26e25): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26e25): Mirror 3: 0x41fb8413 (stale) Mirror 4: 0x41fb8413 (stale) Mirror 5: 0x41fb8413 (stale) Mirror 6: 0x41fb8413 (stale) CRC-32 checksum value for chunk [0, 0x26e25): Mirror 1: 0x94b9775c Mirror 7: 0x94b9775c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26e25: skipped 000171 1787119166.319904 mapread 0x007f70 thru 0x010026 (0x80b7 bytes) 000172 1787119166.395189 fallocate 0x00b57d thru 0x00c004 (0xa88 bytes) 000173 1787119166.434699 trunc from 0x00e61b to 0x017866 (0x924c bytes) 000174 1787119166.440596 mapwrite 0x03e322 thru 0x03ffff (0x1cde bytes) 000175 1787119166.509617 mapread 0x010d89 thru 0x01e886 (0xdafe bytes) 000176 1787119166.529537 mapread 0x02f217 thru 0x0306ca (0x14b4 bytes) 000177 1787119166.545464 mapread 0x00a2bf thru 0x013f52 (0x9c94 bytes) 000178 1787119166.550909 MIRROR_RESYNC 6 mirrors 000179 1787119167.925890 mapread 0x01b976 thru 0x01d273 (0x18fe bytes) 000180 1787119167.943445 trunc from 0x039674 to 0x039d04 (0x691 bytes) 000181 1787119167.974835 trunc from 0x01cef5 to 0x01e836 (0x1942 bytes) 000182 1787119167.979770 write 0x036a21 thru 0x03ae38 (0x4418 bytes) 000183 1787119167.989294 trunc from 0x040000 to 0x017d39 (0x282c6 bytes) 000184 1787119168.011412 trunc from 0x0108d2 to 0x017d39 (0x7468 bytes) fallocating to largest ever: 0x40000 000185 1787119168.016187 fallocate 0x03c4d8 thru 0x03ffff (0x3b28 bytes) 000186 1787119168.019852 punch from 0x02703d to 0x02ed53 (0x7d17 bytes) 000187 1787119168.023803 trunc from 0x040000 to 0x033c66 (0xc399 bytes) 000188 1787119168.038836 punch from 0x0303df to 0x033c66 (0x3888 bytes) 000189 1787119168.042657 write 0x01f784 thru 0x027ce8 (0x8565 bytes) 000190 1787119168.053042 mapread 0x0113a4 thru 0x01e2d3 (0xcf30 bytes) 000191 1787119168.068265 write 0x034690 thru 0x036cd4 (0x2645 bytes) 000192 1787119168.069534 mapwrite 0x0387b9 thru 0x03ffff (0x7847 bytes) 000193 1787119168.112010 trunc from 0x040000 to 0x012ddf (0x2d220 bytes) 000195 1787119168.131111 mapwrite 0x029a89 thru 0x02d0f6 (0x366e bytes) 000196 1787119168.175503 write 0x0055bf thru 0x014f29 (0xf96b bytes) 000197 1787119168.185561 trunc from 0x0033f4 to 0x00d949 (0xa556 bytes) 000198 1787119168.203922 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d0f7): Mirror 3: 0x11886d8 (stale) Mirror 5: 0x11886d8 (stale) Mirror 6: 0x11886d8 (stale) Mirror 7: 0x11886d8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d0f7): Mirror 3: 0x11886d8 (stale) Mirror 5: 0x11886d8 (stale) Mirror 6: 0x11886d8 (stale) Mirror 7: 0x11886d8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d0f7: skipped 000200 1787119168.525173 trunc from 0x02d0f7 to 0x0123c8 (0x1ad2e bytes) 000201 1787119168.545900 write 0x00697b thru 0x006cc0 (0x346 bytes) 000202 1787119168.552331 mapwrite 0x018894 thru 0x023d2e (0xb49b bytes) 000203 1787119168.617792 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x23d2f): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23d2f): Mirror 3: 0x354181ff (stale) Mirror 5: 0x354181ff (stale) Mirror 6: 0x354181ff (stale) Mirror 7: 0x354181ff (stale) CRC-32 checksum value for chunk [0, 0x23d2f): Mirror 1: 0x997d2636 Mirror 8: 0x997d2636 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23d2f: skipped 000205 1787119169.335623 mapread 0x023982 thru 0x023d2e (0x3ad bytes) 000206 1787119169.363580 punch from 0x01d890 to 0x023d2e (0x649f bytes) 000207 1787119169.408529 trunc from 0x023d2f to 0x036a70 (0x12d42 bytes) 000208 1787119169.469718 trunc from 0x036a71 to 0x016667 (0x20409 bytes) 000209 1787119169.483868 mapread 0x011119 thru 0x015a70 (0x4958 bytes) 000210 1787119169.503527 trunc from 0x001b62 to 0x0114fb (0xf99a bytes) 000212 1787119169.516722 write 0x014141 thru 0x01abb7 (0x6a77 bytes) 000213 1787119169.519446 write 0x0302b8 thru 0x034723 (0x446c bytes) 000214 1787119169.524806 mapread 0x02345a thru 0x02cd5c (0x9903 bytes) 000216 1787119169.542251 write 0x02c0b4 thru 0x02ffa5 (0x3ef2 bytes) 000217 1787119169.545209 read 0x01d59b thru 0x020552 (0x2fb8 bytes) 000218 1787119169.547760 fallocate 0x003cd9 thru 0x00d099 (0x93c1 bytes) 000219 1787119169.551593 fallocate 0x032c0e thru 0x0386a9 (0x5a9c bytes) 000220 1787119169.555056 read 0x00eaef thru 0x015aeb (0x6ffd bytes) 000221 1787119169.567051 write 0x01eca7 thru 0x0290a7 (0xa401 bytes) 000222 1787119169.569893 write 0x027804 thru 0x037495 (0xfc92 bytes) 000223 1787119169.572120 punch from 0x03382e to 0x037495 (0x3c68 bytes) 000224 1787119169.603748 trunc from 0x037496 to 0x01ba2d (0x1ba68 bytes) 000226 1787119169.618874 trunc from 0x01ba2e to 0x0310c7 (0x1569a bytes) 000227 1787119169.632268 mapread 0x02423b thru 0x02ba8d (0x7853 bytes) 000228 1787119169.671888 fallocate 0x02dda6 thru 0x03ca5b (0xecb6 bytes) 000229 1787119169.681999 trunc from 0x0310c8 to 0x017ef1 (0x191d6 bytes) 000230 1787119169.698163 trunc from 0x017ef2 to 0x0000fa (0x17df7 bytes) 000233 1787119169.723747 mapread 0x0000d3 thru 0x0000fa (0x28 bytes) 000234 1787119169.730135 MIRROR_RESYNC 6 mirrors 000235 1787119170.547690 punch from 0x000062 to 0x0000fa (0x99 bytes) 000236 1787119170.611721 trunc from 0x0000fb to 0x038125 (0x3802b bytes) 000237 1787119170.693659 MIRROR_RESYNC 6 mirrors 000238 1787119171.879718 trunc from 0x02716b to 0x02e347 (0x71dd bytes) 000239 1787119171.948090 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38126): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38126): Mirror 3: 0xddcd6cde (stale) Mirror 5: 0xddcd6cde (stale) Mirror 6: 0xddcd6cde (stale) Mirror 7: 0xddcd6cde (stale) Mirror 8: 0xddcd6cde (stale) CRC-32 checksum value for chunk [0, 0x38126): Mirror 1: 0xddcd6cde Mirror 9: 0xddcd6cde lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38126: skipped 000240 1787119172.832845 trunc from 0x037509 to 0x038125 (0xc1d bytes) 000241 1787119172.885963 write 0x00e785 thru 0x015cb7 (0x7533 bytes) 000245 1787119172.911651 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x38126): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38126): Mirror 3: 0xddcd6cde (stale) Mirror 5: 0xddcd6cde (stale) Mirror 6: 0xddcd6cde (stale) Mirror 7: 0xddcd6cde (stale) Mirror 8: 0xddcd6cde (stale) Mirror 9: 0xddcd6cde (stale) CRC-32 checksum value for chunk [0, 0x38126): Mirror 1: 0x2240d21d Mirror 10: 0x2240d21d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38126: skipped 000246 1787119173.937003 write 0x020b95 thru 0x02f43c (0xe8a8 bytes) 000247 1787119173.990936 trunc from 0x031a1f to 0x038125 (0x6707 bytes) 000248 1787119173.995493 trunc from 0x038126 to 0x00b521 (0x2cc04 bytes) 000250 1787119174.014739 mapread 0x00b398 thru 0x00b521 (0x18a bytes) 000251 1787119174.027627 MIRROR_RESYNC 8 mirrors 000252 1787119175.084155 mapread 0x00a0b6 thru 0x00b521 (0x146c bytes) 000253 1787119175.100469 write 0x02dbeb thru 0x02e9b6 (0xdcc bytes) 000254 1787119175.129361 mapread 0x021163 thru 0x02ba14 (0xa8b2 bytes) 000255 1787119175.151442 mapread 0x001ee3 thru 0x0073be (0x54dc bytes) 000256 1787119175.175934 fallocate 0x0133a9 thru 0x0213e6 (0xe03e bytes) 000257 1787119175.180391 write 0x020765 thru 0x02e973 (0xe20f bytes) 000258 1787119175.182817 fallocate 0x03923b thru 0x03ffff (0x6dc5 bytes) 000260 1787119175.188570 mapread 0x01a25b thru 0x028bf7 (0xe99d bytes) 000261 1787119175.189987 mapread 0x024b16 thru 0x028beb (0x40d6 bytes) 000262 1787119175.191594 write 0x00776b thru 0x011d05 (0xa59b bytes) 000263 1787119175.197372 read 0x00a2db thru 0x00cd3f (0x2a65 bytes) 000264 1787119175.198450 fallocate 0x0092c7 thru 0x017beb (0xe925 bytes) 000265 1787119175.203410 mapread 0x01f705 thru 0x02b640 (0xbf3c bytes) 000266 1787119175.209306 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd415d8b4 Mirror 11: 0xd415d8b4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000267 1787119176.011679 fallocate 0x03ddae thru 0x03ffff (0x2252 bytes) 000268 1787119176.069619 fallocate 0x012af9 thru 0x014521 (0x1a29 bytes) 000270 1787119176.079332 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xd415d8b4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xd415d8b4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000271 1787119176.508222 mapwrite 0x007e46 thru 0x008113 (0x2ce bytes) 000272 1787119176.536713 mapwrite 0x013985 thru 0x023899 (0xff15 bytes) 000273 1787119176.578725 mapwrite 0x00415e thru 0x009973 (0x5816 bytes) 000274 1787119176.605410 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 10: 0xe20eea22 (stale) Mirror 11: 0xd415d8b4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf8327049 Mirror 12: 0xf8327049 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000275 1787119177.526475 trunc from 0x040000 to 0x03a48c (0x5b73 bytes) 000276 1787119177.600163 trunc from 0x03a48d to 0x03595e (0x4b2e bytes) 000277 1787119177.620747 fallocate 0x01906f thru 0x023179 (0xa10b bytes) 000279 1787119177.628263 read 0x02612c thru 0x02e93b (0x8810 bytes) 000280 1787119177.636178 trunc from 0x012dc4 to 0x019cad (0x6eea bytes) 000281 1787119177.644318 fallocate 0x01c8c6 thru 0x01d0e9 (0x824 bytes) 000282 1787119177.649427 mapread 0x00426a thru 0x008c50 (0x49e7 bytes) 000283 1787119177.668267 punch from 0x01512c to 0x015187 (0x5c bytes) 000284 1787119177.672420 mapread 0x007985 thru 0x0114ff (0x9b7b bytes) 000285 1787119177.683085 trunc from 0x007ba6 to 0x00b25d (0x36b8 bytes) 000286 1787119177.688836 fallocate 0x0143cc thru 0x014a4c (0x681 bytes) 000287 1787119177.694269 mapread 0x017d71 thru 0x01a49f (0x272f bytes) 000288 1787119177.703707 mapread 0x02796c thru 0x02f884 (0x7f19 bytes) 000289 1787119177.713172 trunc from 0x03595f to 0x003173 (0x327eb bytes) 000290 1787119177.728427 trunc from 0x002993 to 0x003173 (0x7e1 bytes) 000291 1787119177.733779 trunc from 0x000c07 to 0x003173 (0x256d bytes) 000292 1787119177.739786 punch from 0x002063 to 0x003173 (0x1111 bytes) 000293 1787119177.745258 trunc from 0x003174 to 0x0203a2 (0x1d22f bytes) 000294 1787119177.763820 mapwrite 0x01e4e0 thru 0x021402 (0x2f23 bytes) 000295 1787119177.803152 fallocate 0x028644 thru 0x029dcc (0x1789 bytes) 000296 1787119177.808769 trunc from 0x029dcd to 0x03f21e (0x15452 bytes) 000298 1787119177.832002 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f21f): Mirror 5: 0x86d63a6d (stale) Mirror 6: 0x86d63a6d (stale) Mirror 7: 0x86d63a6d (stale) Mirror 8: 0x86d63a6d (stale) Mirror 10: 0x86d63a6d (stale) Mirror 11: 0xb5704d47 (stale) Mirror 12: 0x2d5d3037 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f21f): Mirror 5: 0x86d63a6d (stale) Mirror 6: 0x86d63a6d (stale) Mirror 7: 0x86d63a6d (stale) Mirror 8: 0x86d63a6d (stale) Mirror 10: 0x86d63a6d (stale) Mirror 11: 0xb5704d47 (stale) Mirror 12: 0x2d5d3037 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f21f: skipped 000299 1787119178.346615 trunc from 0x006c20 to 0x00bc83 (0x5064 bytes) 000300 1787119178.352810 trunc from 0x030a1d to 0x03f21e (0xe802 bytes) 000301 1787119178.358657 trunc from 0x0365de to 0x03deeb (0x790e bytes) 000303 1787119178.365176 trunc from 0x00e9f2 to 0x00fcf5 (0x1304 bytes) 000304 1787119178.370575 write 0x0240b2 thru 0x027900 (0x384f bytes) 000305 1787119178.386532 mapwrite 0x01d811 thru 0x022664 (0x4e54 bytes) 000306 1787119178.423590 mapwrite 0x00acc9 thru 0x00d787 (0x2abf bytes) 000307 1787119178.448785 trunc from 0x03f21f to 0x0355b9 (0x9c65 bytes) 000308 1787119178.467022 trunc from 0x0355ba to 0x02495b (0x10c5e bytes) 000309 1787119178.489539 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2495c): Mirror 5: 0x2d4215 (stale) Mirror 6: 0x2d4215 (stale) Mirror 7: 0x2d4215 (stale) Mirror 10: 0x2d4215 (stale) Mirror 11: 0x190438e8 (stale) Mirror 12: 0xb6531bd1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2495c): Mirror 5: 0x2d4215 (stale) Mirror 6: 0x2d4215 (stale) Mirror 7: 0x2d4215 (stale) Mirror 10: 0x2d4215 (stale) Mirror 11: 0x190438e8 (stale) Mirror 12: 0xb6531bd1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2495c: skipped 000310 1787119179.066822 mapwrite 0x00d92c thru 0x017891 (0x9f66 bytes) 000311 1787119179.112818 mapwrite 0x01cef4 thru 0x02862d (0xb73a bytes) 000312 1787119179.249781 mapwrite 0x02894e thru 0x03624a (0xd8fd bytes) 000313 1787119179.324639 trunc from 0x003a14 to 0x013196 (0xf783 bytes) 000314 1787119179.331509 fallocate 0x00db05 thru 0x00f281 (0x177d bytes) 000315 1787119179.337075 write 0x000ded thru 0x0022a6 (0x14ba bytes) 000316 1787119179.355937 read 0x02daa2 thru 0x03624a (0x87a9 bytes) 000317 1787119179.358602 punch from 0x01765f to 0x01e857 (0x71f9 bytes) 000318 1787119179.378642 mapwrite 0x0012ce thru 0x001690 (0x3c3 bytes) 000319 1787119179.406735 trunc from 0x03624b to 0x00a252 (0x2bff8 bytes) 000321 1787119179.464132 mapwrite 0x02a6ad thru 0x0339d9 (0x932d bytes) 000322 1787119179.547985 trunc from 0x03395e to 0x0339d9 (0x7c bytes) 000323 1787119179.556498 trunc from 0x010300 to 0x015b16 (0x5817 bytes) 000324 1787119179.564552 trunc from 0x0339da to 0x02b0ca (0x890f bytes) 000325 1787119179.585772 trunc from 0x005cb4 to 0x00ec91 (0x8fde bytes) 000326 1787119179.598381 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b0cb): Mirror 5: 0xc0fba655 (stale) Mirror 6: 0xc0fba655 (stale) Mirror 7: 0xc0fba655 (stale) Mirror 11: 0xc2968847 (stale) Mirror 12: 0xdffa30d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b0cb): Mirror 5: 0xc0fba655 (stale) Mirror 6: 0xc0fba655 (stale) Mirror 7: 0xc0fba655 (stale) Mirror 11: 0xc2968847 (stale) Mirror 12: 0xdffa30d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b0cb: skipped 000327 1787119179.994964 fallocate 0x030e3f thru 0x033a8b (0x2c4d bytes) 000328 1787119180.001902 fallocate 0x007665 thru 0x00a7e4 (0x3180 bytes) 000329 1787119180.007355 trunc from 0x033a8c to 0x02c5fe (0x748d bytes) 000330 1787119180.027736 trunc from 0x02c5ff to 0x03eea9 (0x128ab bytes) 000331 1787119180.042391 mapwrite 0x00e6d2 thru 0x00f2eb (0xc1a bytes) 000332 1787119180.065361 trunc from 0x00308d to 0x00d4b4 (0xa428 bytes) 000333 1787119180.075674 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3eeaa): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eeaa): Mirror 5: 0xc772f0a3 (stale) Mirror 6: 0xc772f0a3 (stale) Mirror 7: 0xc772f0a3 (stale) Mirror 11: 0xa3b4bc0 (stale) Mirror 12: 0x85fc5174 (stale) CRC-32 checksum value for chunk [0, 0x3eeaa): Mirror 1: 0xdef0d4f1 Mirror 13: 0xdef0d4f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eeaa: skipped 000334 1787119180.832434 mapwrite 0x01f0a9 thru 0x021696 (0x25ee bytes) 000335 1787119180.957801 fallocate 0x02af36 thru 0x0360ca (0xb195 bytes) 000336 1787119180.966096 mapread 0x031a44 thru 0x038822 (0x6ddf bytes) 000337 1787119180.982109 trunc from 0x023336 to 0x030c6d (0xd938 bytes) 000338 1787119180.987195 fallocate 0x03d11e thru 0x03ffff (0x2ee2 bytes) 000339 1787119180.992603 trunc from 0x03eeaa to 0x0241cd (0x1acdc bytes) 000340 1787119181.032613 punch from 0x01b8e2 to 0x0241cd (0x88ec bytes) 000341 1787119181.041763 MIRROR_RESYNC 7 mirrors 000342 1787119182.267079 MIRROR_RESYNC 7 mirrors 000343 1787119182.602617 mapwrite 0x01642d thru 0x01b605 (0x51d9 bytes) 000344 1787119182.681777 fallocate 0x03a092 thru 0x03ffff (0x5f6e bytes) 000345 1787119182.686624 write 0x00e470 thru 0x01c00f (0xdba0 bytes) 000346 1787119182.695592 trunc from 0x00b0f5 to 0x0172be (0xc1ca bytes) 000347 1787119182.717169 trunc from 0x040000 to 0x038105 (0x7efa bytes) 000348 1787119182.743163 trunc from 0x001f70 to 0x002950 (0x9e1 bytes) 000349 1787119182.749080 trunc from 0x038106 to 0x00a992 (0x2d773 bytes) 000350 1787119182.780449 read 0x002d92 thru 0x009032 (0x62a1 bytes) 000351 1787119182.803397 mapwrite 0x037f3d thru 0x03c2b1 (0x4375 bytes) 000352 1787119182.860954 mapwrite 0x02623e thru 0x0347ed (0xe5b0 bytes) 000353 1787119182.901707 write 0x0132d2 thru 0x0211a5 (0xded4 bytes) 000354 1787119182.908627 trunc from 0x03c2b2 to 0x00400f (0x382a2 bytes) 000355 1787119182.929339 trunc from 0x002b38 to 0x00400f (0x14d8 bytes) 000356 1787119182.934697 trunc from 0x004010 to 0x005819 (0x180a bytes) 000357 1787119182.948785 mapread 0x00369e thru 0x005819 (0x217c bytes) 000358 1787119182.957458 trunc from 0x000903 to 0x005819 (0x4f17 bytes) 000359 1787119182.962845 read 0x0050ab thru 0x005819 (0x76f bytes) 000360 1787119182.972899 read 0x004381 thru 0x005819 (0x1499 bytes) 000361 1787119182.974547 punch from 0x000f67 to 0x005819 (0x48b3 bytes) 000362 1787119182.980673 trunc from 0x00581a to 0x00aef7 (0x56de bytes) 000363 1787119183.001980 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xaef8): Mirror 5: 0x917e3ffc (stale) Mirror 7: 0x917e3ffc (stale) Mirror 11: 0x917e3ffc (stale) Mirror 12: 0x917e3ffc (stale) Mirror 13: 0x917e3ffc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xaef8): Mirror 5: 0x917e3ffc (stale) Mirror 7: 0x917e3ffc (stale) Mirror 11: 0x917e3ffc (stale) Mirror 12: 0x917e3ffc (stale) Mirror 13: 0x917e3ffc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xaef8: skipped 000364 1787119183.440753 punch from 0x005f75 to 0x00aef7 (0x4f83 bytes) 000365 1787119183.469946 mapread 0x0020d4 thru 0x002a12 (0x93f bytes) 000366 1787119183.478969 punch from 0x0028e2 to 0x00879d (0x5ebc bytes) 000367 1787119183.487454 punch from 0x002296 to 0x00aef7 (0x8c62 bytes) 000369 1787119183.493185 write 0x01d109 thru 0x022efe (0x5df6 bytes) 000370 1787119183.502416 fallocate 0x01c926 thru 0x027111 (0xa7ec bytes) 000371 1787119183.523090 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x22eff): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22eff): Mirror 5: 0x945c6e2 (stale) Mirror 7: 0x945c6e2 (stale) Mirror 11: 0x945c6e2 (stale) Mirror 12: 0x945c6e2 (stale) Mirror 13: 0x945c6e2 (stale) CRC-32 checksum value for chunk [0, 0x22eff): Mirror 1: 0x7aefb0f0 Mirror 14: 0x7aefb0f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22eff: skipped 000372 1787119184.419986 trunc from 0x022eff to 0x026dce (0x3ed0 bytes) 000373 1787119184.505762 trunc from 0x026dcf to 0x02c818 (0x5a4a bytes) 000374 1787119184.533628 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c819): Mirror 5: 0x46073a58 (stale) Mirror 7: 0x46073a58 (stale) Mirror 12: 0x46073a58 (stale) Mirror 13: 0x46073a58 (stale) Mirror 14: 0x3e6ce7cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c819): Mirror 5: 0x46073a58 (stale) Mirror 7: 0x46073a58 (stale) Mirror 12: 0x46073a58 (stale) Mirror 13: 0x46073a58 (stale) Mirror 14: 0x3e6ce7cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c819: skipped 000375 1787119184.952191 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2c819): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c819): Mirror 5: 0x46073a58 (stale) Mirror 7: 0x46073a58 (stale) Mirror 12: 0x46073a58 (stale) Mirror 13: 0x46073a58 (stale) Mirror 14: 0x3e6ce7cc (stale) CRC-32 checksum value for chunk [0, 0x2c819): Mirror 1: 0x3e6ce7cc Mirror 15: 0x3e6ce7cc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c819: skipped 000376 1787119185.803492 punch from 0x009f83 to 0x015a0c (0xba8a bytes) 000378 1787119185.838474 write 0x00f301 thru 0x015e70 (0x6b70 bytes) 000379 1787119185.851719 mapread 0x00a368 thru 0x00a940 (0x5d9 bytes) 000380 1787119185.857159 fallocate 0x00b197 thru 0x00fc31 (0x4a9b bytes) 000381 1787119185.861504 fallocate 0x03c563 thru 0x03ffff (0x3a9d bytes) 000382 1787119185.866166 mapread 0x025dde thru 0x02e596 (0x87b9 bytes) 000383 1787119185.879189 mapread 0x0398db thru 0x03ffff (0x6725 bytes) 000384 1787119185.888168 mapread 0x028b2c thru 0x028da1 (0x276 bytes) 000386 1787119185.889854 fallocate 0x030737 thru 0x03a6ac (0x9f76 bytes) 000387 1787119185.895872 fallocate 0x018082 thru 0x0200cd (0x804c bytes) 000389 1787119185.900598 punch from 0x020197 to 0x029321 (0x918b bytes) 000390 1787119185.905929 punch from 0x0041b7 to 0x00913e (0x4f88 bytes) 000391 1787119185.909804 fallocate 0x01a062 thru 0x02147a (0x7419 bytes) 000392 1787119185.915278 trunc from 0x02caaa to 0x0351ac (0x8703 bytes) 000393 1787119185.921212 write 0x00e268 thru 0x01784e (0x95e7 bytes) 000394 1787119185.931195 read 0x03a1a8 thru 0x03ffff (0x5e58 bytes) 000395 1787119185.932353 mapread 0x037939 thru 0x03d0b7 (0x577f bytes) 000396 1787119185.933911 write 0x01664e thru 0x025f5d (0xf910 bytes) 000397 1787119185.945763 MIRROR_RESYNC 7 mirrors 000399 1787119187.314092 mapread 0x03d248 thru 0x03ffff (0x2db8 bytes) 000400 1787119187.339405 write 0x029446 thru 0x035aab (0xc666 bytes) 000402 1787119187.394083 mapwrite 0x0180ab thru 0x022094 (0x9fea bytes) 000404 1787119187.448566 fallocate 0x022d89 thru 0x02ed6f (0xbfe7 bytes) 000405 1787119187.459582 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xd917348c (stale) Mirror 7: 0xd917348c (stale) Mirror 12: 0xd917348c (stale) Mirror 13: 0xd917348c (stale) Mirror 14: 0xd917348c (stale) Mirror 15: 0xd917348c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd1fc0364 Mirror 16: 0xd1fc0364 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000406 1787119188.435074 trunc from 0x01b1cb to 0x01fa98 (0x48ce bytes) 000407 1787119188.489413 trunc from 0x01d0f4 to 0x01db3a (0xa47 bytes) 000408 1787119188.494297 mapread 0x017f67 thru 0x01bbd9 (0x3c73 bytes) 000409 1787119188.519883 trunc from 0x033c55 to 0x0343c7 (0x773 bytes) 000410 1787119188.533401 mapwrite 0x000911 thru 0x004a85 (0x4175 bytes) 000411 1787119188.592300 mapread 0x02378d thru 0x030616 (0xce8a bytes) 000412 1787119188.622381 MIRROR_RESYNC 8 mirrors 000413 1787119190.654801 trunc from 0x01a701 to 0x023adf (0x93df bytes) 000414 1787119190.683499 read 0x01d537 thru 0x02ac4a (0xd714 bytes) 000415 1787119190.695825 punch from 0x01e211 to 0x02bbae (0xd99e bytes) 000416 1787119190.706275 mapwrite 0x0089be thru 0x00beaf (0x34f2 bytes) 000417 1787119190.739337 trunc from 0x040000 to 0x034f57 (0xb0a8 bytes) 000418 1787119190.763718 punch from 0x025f2f to 0x031469 (0xb53b bytes) 000419 1787119190.776654 write 0x0316cb thru 0x0350c9 (0x39ff bytes) 000420 1787119190.790667 trunc from 0x00cc58 to 0x016bde (0x9f87 bytes) 000421 1787119190.801333 fallocate 0x006f85 thru 0x007df9 (0xe75 bytes) 000422 1787119190.809495 mapread 0x02bf77 thru 0x02f2bf (0x3349 bytes) 000423 1787119190.826589 write 0x0372d7 thru 0x03ffff (0x8d29 bytes) 000424 1787119190.834924 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x18f5dbbb (stale) Mirror 7: 0x18f5dbbb (stale) Mirror 12: 0x18f5dbbb (stale) Mirror 13: 0x18f5dbbb (stale) Mirror 14: 0x18f5dbbb (stale) Mirror 15: 0x18f5dbbb (stale) Mirror 16: 0x18f5dbbb (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x55d8b5c4 Mirror 17: 0x55d8b5c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000425 1787119191.855013 trunc from 0x00ca38 to 0x013e9a (0x7463 bytes) 000427 1787119191.901407 MIRROR_RESYNC 9 mirrors 000428 1787119194.353907 write 0x03c1a1 thru 0x03ffff (0x3e5f bytes) 000429 1787119194.390397 trunc from 0x01ec6d to 0x02bf77 (0xd30b bytes) 000430 1787119194.398864 read 0x0366f8 thru 0x03ffff (0x9908 bytes) 000431 1787119194.409342 trunc from 0x040000 to 0x017729 (0x288d6 bytes) 000432 1787119194.432266 mapread 0x012648 thru 0x017729 (0x50e2 bytes) 000433 1787119194.453826 read 0x004bec thru 0x0092cc (0x46e1 bytes) 000436 1787119194.466002 trunc from 0x01772a to 0x0229b8 (0xb28f bytes) 000437 1787119194.483023 read 0x020ea0 thru 0x0229b8 (0x1b19 bytes) 000438 1787119194.501768 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x229b9): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x229b9): Mirror 5: 0x8b617192 (stale) Mirror 7: 0x8b617192 (stale) Mirror 12: 0x8b617192 (stale) Mirror 13: 0x8b617192 (stale) Mirror 14: 0x8b617192 (stale) Mirror 15: 0x8b617192 (stale) Mirror 16: 0x8b617192 (stale) Mirror 17: 0x8b617192 (stale) CRC-32 checksum value for chunk [0, 0x229b9): Mirror 1: 0x17e145ce Mirror 18: 0x17e145ce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x229b9: skipped 000440 1787119196.091912 mapwrite 0x01c1f4 thru 0x026e71 (0xac7e bytes) 000441 1787119196.255442 mapread 0x0041e8 thru 0x006640 (0x2459 bytes) 000442 1787119196.271727 punch from 0x005923 to 0x00f0b3 (0x9791 bytes) 000443 1787119196.287061 mapwrite 0x032953 thru 0x03f9d1 (0xd07f bytes) 000444 1787119196.399579 trunc from 0x021e3c to 0x03181b (0xf9e0 bytes) 000445 1787119196.415328 fallocate 0x00500e thru 0x0097a2 (0x4795 bytes) 000446 1787119196.425537 trunc from 0x03f9d2 to 0x02df5b (0x11a76 bytes) 000447 1787119196.487069 write 0x026baf thru 0x03122b (0xa67d bytes) 000448 1787119196.503149 mapread 0x021182 thru 0x030348 (0xf1c7 bytes) 000449 1787119196.534113 fallocate 0x001e63 thru 0x00ca45 (0xabe3 bytes) 000450 1787119196.546656 fallocate 0x00a6dd thru 0x0192ea (0xec0e bytes) 000451 1787119196.554370 mapread 0x013fee thru 0x022515 (0xe528 bytes) 000452 1787119196.579804 read 0x024a95 thru 0x02b128 (0x6694 bytes) 000453 1787119196.581502 read 0x02d566 thru 0x03122b (0x3cc6 bytes) 000454 1787119196.582747 punch from 0x01232b to 0x017657 (0x532d bytes) 000455 1787119196.607059 trunc from 0x03122c to 0x01c5f1 (0x14c3a bytes) 000457 1787119196.641579 write 0x013385 thru 0x01807b (0x4cf7 bytes) 000458 1787119196.668550 punch from 0x00968d to 0x00f32d (0x5ca1 bytes) 000459 1787119196.705397 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x1c5f2): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c5f2): Mirror 5: 0xffefd23 (stale) Mirror 7: 0xffefd23 (stale) Mirror 12: 0xffefd23 (stale) Mirror 13: 0xffefd23 (stale) Mirror 14: 0xffefd23 (stale) Mirror 15: 0xffefd23 (stale) Mirror 16: 0xffefd23 (stale) Mirror 17: 0xffefd23 (stale) Mirror 18: 0xd919474d (stale) CRC-32 checksum value for chunk [0, 0x1c5f2): Mirror 1: 0x505d94bf Mirror 19: 0x505d94bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c5f2: skipped 000460 1787119198.440047 trunc from 0x0168c2 to 0x01c5f1 (0x5d30 bytes) 000461 1787119198.548338 mapread 0x007981 thru 0x007d83 (0x403 bytes) 000462 1787119198.558620 trunc from 0x01c5f2 to 0x036060 (0x19a6f bytes) 000464 1787119198.618339 write 0x01e0e2 thru 0x02191d (0x383c bytes) 000465 1787119198.654127 trunc from 0x036061 to 0x01d54f (0x18b11 bytes) 000466 1787119198.726032 write 0x0222ea thru 0x03122a (0xef41 bytes) 000467 1787119198.734598 trunc from 0x03122b to 0x03ad94 (0x9b6a bytes) 000469 1787119198.829438 trunc from 0x03ad95 to 0x016fc1 (0x23dd3 bytes) 000470 1787119198.894576 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x16fc2): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16fc2): Mirror 5: 0x162b6f7a (stale) Mirror 7: 0x162b6f7a (stale) Mirror 12: 0x162b6f7a (stale) Mirror 13: 0x162b6f7a (stale) Mirror 14: 0x162b6f7a (stale) Mirror 15: 0x162b6f7a (stale) Mirror 16: 0x162b6f7a (stale) Mirror 17: 0x162b6f7a (stale) Mirror 18: 0x162b6f7a (stale) Mirror 19: 0x13b758a2 (stale) CRC-32 checksum value for chunk [0, 0x16fc2): Mirror 1: 0x7304960c Mirror 20: 0x7304960c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16fc2: skipped 000471 1787119200.750823 trunc from 0x011d25 to 0x013c6f (0x1f4b bytes) 000472 1787119200.797648 MIRROR_SPLIT mirror 4 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16fc2): Mirror 5: 0x162b6f7a (stale) Mirror 7: 0x162b6f7a (stale) Mirror 12: 0x162b6f7a (stale) Mirror 14: 0x162b6f7a (stale) Mirror 15: 0x162b6f7a (stale) Mirror 16: 0x162b6f7a (stale) Mirror 17: 0x162b6f7a (stale) Mirror 18: 0x162b6f7a (stale) Mirror 19: 0x13b758a2 (stale) Mirror 20: 0x7304960c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16fc2): Mirror 5: 0x162b6f7a (stale) Mirror 7: 0x162b6f7a (stale) Mirror 12: 0x162b6f7a (stale) Mirror 14: 0x162b6f7a (stale) Mirror 15: 0x162b6f7a (stale) Mirror 16: 0x162b6f7a (stale) Mirror 17: 0x162b6f7a (stale) Mirror 18: 0x162b6f7a (stale) Mirror 19: 0x13b758a2 (stale) Mirror 20: 0x7304960c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16fc2: skipped 000473 1787119201.567172 mapread 0x011169 thru 0x016fc1 (0x5e59 bytes) 000474 1787119201.593976 read 0x00260a thru 0x00f2bb (0xccb2 bytes) 000475 1787119201.602975 mapwrite 0x022b8d thru 0x029e91 (0x7305 bytes) 000476 1787119201.691777 mapwrite 0x002a4b thru 0x0041a5 (0x175b bytes) 000477 1787119201.719469 mapread 0x02591a thru 0x029e91 (0x4578 bytes) 000478 1787119201.721453 read 0x007a5c thru 0x0179a8 (0xff4d bytes) 000479 1787119201.750179 trunc from 0x013b57 to 0x020cce (0xd178 bytes) 000480 1787119201.764729 punch from 0x016d4a to 0x022d5e (0xc015 bytes) 000481 1787119201.774093 write 0x019e25 thru 0x0219da (0x7bb6 bytes) 000482 1787119201.793783 write 0x026735 thru 0x0293d7 (0x2ca3 bytes) 000483 1787119201.795638 fallocate 0x00e5aa thru 0x01d26a (0xecc1 bytes) 000484 1787119201.808470 mapread 0x0297bf thru 0x029e91 (0x6d3 bytes) 000485 1787119201.810824 mapwrite 0x007d04 thru 0x016eec (0xf1e9 bytes) 000486 1787119201.880476 trunc from 0x029e92 to 0x013395 (0x16afc bytes) 000487 1787119201.918566 MIRROR_RESYNC 11 mirrors 000488 1787119204.764486 read 0x008f61 thru 0x00ddd5 (0x4e75 bytes) 000490 1787119204.783101 MIRROR_EXTEND to 12 mirrors 000491 1787119205.727492 punch from 0x01020e to 0x013395 (0x3188 bytes) 000492 1787119205.791880 trunc from 0x013396 to 0x03558e (0x221f9 bytes) 000493 1787119205.816138 fallocate 0x031f31 thru 0x03ffff (0xe0cf bytes) 000494 1787119205.830572 read 0x009ea8 thru 0x00abfd (0xd56 bytes) 000495 1787119205.848575 mapwrite 0x02a9bb thru 0x02db81 (0x31c7 bytes) 000496 1787119205.906814 read 0x02927d thru 0x02ce90 (0x3c14 bytes) 000497 1787119205.907527 read 0x00205f thru 0x00ead1 (0xca73 bytes) 000498 1787119205.929142 read 0x007a61 thru 0x012e07 (0xb3a7 bytes) 000499 1787119205.930159 read 0x00fd90 thru 0x01cd8f (0xd000 bytes) 000500 1787119205.934675 mapwrite 0x01f74f thru 0x0252b1 (0x5b63 bytes) 000501 1787119205.969808 punch from 0x0198c9 to 0x02535f (0xba97 bytes) 000502 1787119205.979615 write 0x01a3ab thru 0x022d48 (0x899e bytes) 000503 1787119205.990925 mapread 0x019184 thru 0x025d2c (0xcba9 bytes) 000504 1787119205.994152 trunc from 0x035865 to 0x03f7f6 (0x9f92 bytes) 000505 1787119206.001080 trunc from 0x040000 to 0x0120ab (0x2df54 bytes) 000506 1787119206.022908 mapwrite 0x022120 thru 0x02b2cb (0x91ac bytes) 000507 1787119206.103689 fallocate 0x00aebd thru 0x0199a3 (0xeae7 bytes) 000508 1787119206.111816 mapwrite 0x03cf47 thru 0x03ffff (0x30b9 bytes) 000509 1787119206.180409 punch from 0x02ad1a to 0x0361d8 (0xb4bf bytes) 000510 1787119206.186929 mapread 0x01bff8 thru 0x01fe47 (0x3e50 bytes) 000511 1787119206.215108 read 0x027416 thru 0x0302ac (0x8e97 bytes) 000512 1787119206.233708 write 0x039e56 thru 0x03d510 (0x36bb bytes) 000513 1787119206.235998 trunc from 0x01d40a to 0x021afa (0x46f1 bytes) 000514 1787119206.248110 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x9c228b59 (stale) Mirror 7: 0x9c228b59 (stale) Mirror 12: 0x9c228b59 (stale) Mirror 14: 0x9c228b59 (stale) Mirror 15: 0x9c228b59 (stale) Mirror 16: 0x9c228b59 (stale) Mirror 17: 0x9c228b59 (stale) Mirror 18: 0x9c228b59 (stale) Mirror 19: 0x9c228b59 (stale) Mirror 20: 0x9c228b59 (stale) Mirror 21: 0x9c228b59 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7c80904a Mirror 22: 0x7c80904a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000515 1787119207.793283 mapread 0x01d21c thru 0x02cfda (0xfdbf bytes) 000516 1787119207.882136 read 0x0390e8 thru 0x03ffff (0x6f18 bytes) 000517 1787119207.923934 mapread 0x020806 thru 0x0262c6 (0x5ac1 bytes) 000518 1787119207.967044 punch from 0x009f9e to 0x011b93 (0x7bf6 bytes) 000519 1787119208.093458 write 0x02f185 thru 0x03b4e4 (0xc360 bytes) 000520 1787119208.143982 trunc from 0x040000 to 0x010e28 (0x2f1d7 bytes) 000521 1787119208.246913 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x10e29): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10e29): Mirror 5: 0x124380 (stale) Mirror 7: 0x124380 (stale) Mirror 12: 0x124380 (stale) Mirror 14: 0x124380 (stale) Mirror 15: 0x124380 (stale) Mirror 16: 0x124380 (stale) Mirror 17: 0x124380 (stale) Mirror 18: 0x124380 (stale) Mirror 19: 0x124380 (stale) Mirror 20: 0x124380 (stale) Mirror 21: 0x124380 (stale) Mirror 22: 0x50ac9c7f (stale) CRC-32 checksum value for chunk [0, 0x10e29): Mirror 1: 0xe776c92e Mirror 23: 0xe776c92e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10e29: skipped 000522 1787119209.572958 trunc from 0x010e29 to 0x035b09 (0x24ce1 bytes) 000523 1787119209.647170 mapwrite 0x025aa8 thru 0x02c121 (0x667a bytes) 000524 1787119209.687174 mapread 0x0325ac thru 0x033417 (0xe6c bytes) 000525 1787119209.699984 write 0x00873e thru 0x01795f (0xf222 bytes) 000526 1787119209.713968 fallocate 0x01e1cf thru 0x02acaa (0xcadc bytes) 000527 1787119209.722539 mapread 0x00b77b thru 0x00fadd (0x4363 bytes) 000528 1787119209.723712 trunc from 0x035b0a to 0x02a490 (0xb679 bytes) 000529 1787119209.750330 write 0x02c977 thru 0x03839e (0xba28 bytes) 000530 1787119209.755399 trunc from 0x03839f to 0x013334 (0x2506a bytes) 000531 1787119209.776293 write 0x00e902 thru 0x012c17 (0x4316 bytes) 000532 1787119209.778432 trunc from 0x006502 to 0x00a795 (0x4294 bytes) 000533 1787119209.794556 mapwrite 0x02bc00 thru 0x02c94f (0xd50 bytes) 000534 1787119209.841907 trunc from 0x023a77 to 0x025b47 (0x20d1 bytes) 000535 1787119209.848698 trunc from 0x02c950 to 0x03ad55 (0xe406 bytes) 000536 1787119209.869669 mapread 0x0280d7 thru 0x0374a9 (0xf3d3 bytes) 000537 1787119209.892263 mapwrite 0x03b3a8 thru 0x03ffff (0x4c58 bytes) 000538 1787119209.933416 trunc from 0x0093e9 to 0x01358c (0xa1a4 bytes) 000539 1787119209.946879 MIRROR_RESYNC 14 mirrors 000540 1787119213.266509 read 0x00657c thru 0x00935d (0x2de2 bytes) 000541 1787119213.276517 mapread 0x02f3a0 thru 0x0314ea (0x214b bytes) 000542 1787119213.292857 mapread 0x02bfc3 thru 0x0310c4 (0x5102 bytes) 000543 1787119213.294424 read 0x0031f9 thru 0x00b4db (0x82e3 bytes) 000544 1787119213.312141 mapwrite 0x03266b thru 0x03c4c4 (0x9e5a bytes) 000545 1787119213.445558 write 0x02c7f2 thru 0x03babd (0xf2cc bytes) 000546 1787119213.451809 fallocate 0x012342 thru 0x012c5b (0x91a bytes) 000547 1787119213.462651 mapwrite 0x0354a4 thru 0x03916a (0x3cc7 bytes) 000548 1787119213.480398 trunc from 0x00725f to 0x00da35 (0x67d7 bytes) 000549 1787119213.488862 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x664dafa4 (stale) Mirror 7: 0x664dafa4 (stale) Mirror 12: 0x664dafa4 (stale) Mirror 14: 0x664dafa4 (stale) Mirror 15: 0x664dafa4 (stale) Mirror 16: 0x664dafa4 (stale) Mirror 17: 0x664dafa4 (stale) Mirror 18: 0x664dafa4 (stale) Mirror 19: 0x664dafa4 (stale) Mirror 20: 0x664dafa4 (stale) Mirror 21: 0x664dafa4 (stale) Mirror 22: 0x664dafa4 (stale) Mirror 23: 0x664dafa4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd31b036b Mirror 24: 0xd31b036b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000550 1787119214.364464 read 0x03876e thru 0x03e1c5 (0x5a58 bytes) 000553 1787119214.392421 trunc from 0x040000 to 0x0083e4 (0x37c1b bytes) 000554 1787119214.497166 punch from 0x004433 to 0x0047bd (0x38b bytes) 000555 1787119214.505891 read 0x0014c4 thru 0x0083e4 (0x6f21 bytes) 000556 1787119214.533632 punch from 0x00285c to 0x0083e4 (0x5b89 bytes) 000558 1787119214.541589 mapwrite 0x02f421 thru 0x03eac7 (0xf6a7 bytes) 000560 1787119214.600418 fallocate 0x013d4f thru 0x01d43e (0x96f0 bytes) 000561 1787119214.605821 read 0x0392d6 thru 0x03eac7 (0x57f2 bytes) 000562 1787119214.606797 fallocate 0x033a9f thru 0x038a53 (0x4fb5 bytes) 000563 1787119214.614632 MIRROR_RESYNC 15 mirrors 000564 1787119216.545602 trunc from 0x03eac8 to 0x00220c (0x3c8bb bytes) 000565 1787119216.602644 punch from 0x0015d6 to 0x00220c (0xc37 bytes) 000566 1787119216.630273 MIRROR_RESYNC 15 mirrors 000567 1787119217.762009 trunc from 0x00220d to 0x037f92 (0x35d86 bytes) 000568 1787119217.794905 trunc from 0x029b5a to 0x037f92 (0xe439 bytes) 000569 1787119217.798960 trunc from 0x037f93 to 0x00e33d (0x29c55 bytes) 000570 1787119217.817809 trunc from 0x00e33e to 0x002f92 (0xb3ab bytes) 000571 1787119217.829569 fallocate 0x00b286 thru 0x00ba38 (0x7b3 bytes) 000572 1787119217.832622 fallocate 0x022e0a thru 0x02b23b (0x8432 bytes) 000573 1787119217.835621 read 0x002e05 thru 0x004d9a (0x1f96 bytes) 000574 1787119217.838851 trunc from 0x004dfc to 0x0094d4 (0x46d9 bytes) 000575 1787119217.842084 punch from 0x005913 to 0x008451 (0x2b3f bytes) 000576 1787119217.844942 mapread 0x00b64a thru 0x00ba38 (0x3ef bytes) 000577 1787119217.848702 write 0x006167 thru 0x0153a9 (0xf243 bytes) 000578 1787119217.852862 write 0x029863 thru 0x02c7be (0x2f5c bytes) 000579 1787119217.853754 write 0x00b9f6 thru 0x01614b (0xa756 bytes) 000580 1787119217.857161 mapwrite 0x0386fd thru 0x03ffff (0x7903 bytes) 000581 1787119217.894845 mapwrite 0x01278a thru 0x0170ab (0x4922 bytes) 000582 1787119217.922919 mapread 0x02185a thru 0x02b886 (0xa02d bytes) 000583 1787119217.934596 trunc from 0x03dd85 to 0x03e223 (0x49f bytes) 000584 1787119217.941019 read 0x011b03 thru 0x01628f (0x478d bytes) 000585 1787119217.942535 mapwrite 0x012fab thru 0x0207a5 (0xd7fb bytes) 000586 1787119217.973213 MIRROR_RESYNC 15 mirrors 000587 1787119221.108022 MIRROR_RESYNC 15 mirrors 000588 1787119222.336704 mapwrite 0x0221dd thru 0x025a4d (0x3871 bytes) 000589 1787119222.534557 mapread 0x0094e7 thru 0x011a01 (0x851b bytes) 000590 1787119222.570988 read 0x030d02 thru 0x032747 (0x1a46 bytes) 000591 1787119222.583384 trunc from 0x040000 to 0x0397c1 (0x683e bytes) 000592 1787119222.605980 punch from 0x027e96 to 0x02c182 (0x42ed bytes) 000593 1787119222.616071 write 0x011221 thru 0x0188d3 (0x76b3 bytes) 000594 1787119222.623636 fallocate 0x00fbee thru 0x012371 (0x2784 bytes) 000595 1787119222.637155 trunc from 0x0397c2 to 0x027143 (0x1267e bytes) 000596 1787119222.668024 fallocate 0x01f6c2 thru 0x0205ff (0xf3e bytes) 000597 1787119222.676024 fallocate 0x00d8f1 thru 0x01bc4d (0xe35d bytes) 000598 1787119222.681797 trunc from 0x01f5c8 to 0x023b32 (0x456b bytes) 000600 1787119222.688501 punch from 0x013842 to 0x01c0fe (0x88bd bytes) 000601 1787119222.708847 MIRROR_RESYNC 15 mirrors 000602 1787119227.609199 write 0x022360 thru 0x02bd6b (0x9a0c bytes) 000603 1787119227.693136 fallocate 0x00552c thru 0x013cc3 (0xe798 bytes) 000604 1787119227.697999 mapread 0x01c1e4 thru 0x01dedf (0x1cfc bytes) 000605 1787119227.706157 mapwrite 0x006679 thru 0x00b39a (0x4d22 bytes) 000606 1787119227.731352 mapwrite 0x01c300 thru 0x0290e1 (0xcde2 bytes) 000607 1787119227.756867 mapread 0x025f06 thru 0x02bd6b (0x5e66 bytes) 000608 1787119227.758210 write 0x01c4b1 thru 0x02245f (0x5faf bytes) 000609 1787119227.760441 mapread 0x01e3d2 thru 0x02bd6b (0xd99a bytes) 000610 1787119227.762824 mapread 0x01b0c6 thru 0x01fd59 (0x4c94 bytes) 000611 1787119227.764029 fallocate 0x031e7a thru 0x039220 (0x73a7 bytes) 000612 1787119227.769172 mapread 0x01929d thru 0x01bf25 (0x2c89 bytes) 000613 1787119227.770782 fallocate 0x034ed9 thru 0x03ffff (0xb127 bytes) 000614 1787119227.779792 MIRROR_RESYNC 15 mirrors 000616 1787119230.282933 write 0x03aee9 thru 0x03ffff (0x5117 bytes) 000617 1787119230.312477 read 0x01938f thru 0x01b546 (0x21b8 bytes) 000618 1787119230.319468 trunc from 0x040000 to 0x022ea4 (0x1d15b bytes) 000619 1787119230.341620 mapread 0x012658 thru 0x0150ae (0x2a57 bytes) 000620 1787119230.359704 mapwrite 0x030bff thru 0x03e806 (0xdc08 bytes) 000621 1787119230.408773 mapwrite 0x02dd45 thru 0x03b071 (0xd32d bytes) 000622 1787119230.429043 mapwrite 0x02d7c8 thru 0x02e069 (0x8a2 bytes) 000623 1787119230.439641 read 0x018e0b thru 0x0233ed (0xa5e3 bytes) 000624 1787119230.452924 MIRROR_RESYNC 15 mirrors 000627 1787119233.932850 trunc from 0x02b1ad to 0x032595 (0x73e9 bytes) 000629 1787119233.981951 trunc from 0x02ec24 to 0x03aed8 (0xc2b5 bytes) 000630 1787119233.991164 fallocate 0x03cb6b thru 0x03ffff (0x3495 bytes) 000631 1787119234.018702 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x4d456943 (stale) Mirror 7: 0x4d456943 (stale) Mirror 12: 0x4d456943 (stale) Mirror 14: 0x4d456943 (stale) Mirror 15: 0x4d456943 (stale) Mirror 16: 0x4d456943 (stale) Mirror 17: 0x4d456943 (stale) Mirror 18: 0x4d456943 (stale) Mirror 19: 0x4d456943 (stale) Mirror 20: 0x4d456943 (stale) Mirror 21: 0x4d456943 (stale) Mirror 22: 0x4d456943 (stale) Mirror 23: 0x4d456943 (stale) Mirror 24: 0x4d456943 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd6cacb1 Mirror 25: 0xd6cacb1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000632 1787119235.377849 write 0x00c6aa thru 0x00daa2 (0x13f9 bytes) 000633 1787119235.431227 trunc from 0x040000 to 0x01909b (0x26f64 bytes) 000634 1787119235.455154 write 0x03ad3e thru 0x03c3f9 (0x16bc bytes) 000635 1787119235.457818 mapwrite 0x0373f0 thru 0x03ffff (0x8c10 bytes) 000636 1787119235.512224 punch from 0x03b59b to 0x03fd65 (0x47cb bytes) 000637 1787119235.520158 fallocate 0x018998 thru 0x01c9ba (0x4023 bytes) 000638 1787119235.526275 trunc from 0x040000 to 0x025cbf (0x1a340 bytes) 000639 1787119235.544375 punch from 0x00361b to 0x01338f (0xfd75 bytes) 000640 1787119235.548864 mapread 0x015fcd thru 0x0196ab (0x36df bytes) 000641 1787119235.565510 MIRROR_RESYNC 16 mirrors 000643 1787119238.565408 mapwrite 0x028396 thru 0x02b712 (0x337d bytes) 000646 1787119238.695163 mapread 0x0084d1 thru 0x016e1f (0xe94f bytes) 000647 1787119238.717833 punch from 0x023b03 to 0x02b712 (0x7c10 bytes) 000648 1787119238.726361 MIRROR_SPLIT mirror 15 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b713): Mirror 5: 0x59bc9847 (stale) Mirror 7: 0x59bc9847 (stale) Mirror 12: 0x59bc9847 (stale) Mirror 14: 0x59bc9847 (stale) Mirror 15: 0x59bc9847 (stale) Mirror 16: 0x59bc9847 (stale) Mirror 17: 0x59bc9847 (stale) Mirror 18: 0x59bc9847 (stale) Mirror 19: 0x59bc9847 (stale) Mirror 20: 0x59bc9847 (stale) Mirror 21: 0x59bc9847 (stale) Mirror 22: 0x59bc9847 (stale) Mirror 23: 0x59bc9847 (stale) Mirror 24: 0x59bc9847 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b713): Mirror 5: 0x59bc9847 (stale) Mirror 7: 0x59bc9847 (stale) Mirror 12: 0x59bc9847 (stale) Mirror 14: 0x59bc9847 (stale) Mirror 15: 0x59bc9847 (stale) Mirror 16: 0x59bc9847 (stale) Mirror 17: 0x59bc9847 (stale) Mirror 18: 0x59bc9847 (stale) Mirror 19: 0x59bc9847 (stale) Mirror 20: 0x59bc9847 (stale) Mirror 21: 0x59bc9847 (stale) Mirror 22: 0x59bc9847 (stale) Mirror 23: 0x59bc9847 (stale) Mirror 24: 0x59bc9847 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b713: skipped 000650 1787119239.507145 read 0x01d6d4 thru 0x026c19 (0x9546 bytes) 000651 1787119239.548421 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x2b713): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b713): Mirror 5: 0x59bc9847 (stale) Mirror 7: 0x59bc9847 (stale) Mirror 12: 0x59bc9847 (stale) Mirror 14: 0x59bc9847 (stale) Mirror 15: 0x59bc9847 (stale) Mirror 16: 0x59bc9847 (stale) Mirror 17: 0x59bc9847 (stale) Mirror 18: 0x59bc9847 (stale) Mirror 19: 0x59bc9847 (stale) Mirror 20: 0x59bc9847 (stale) Mirror 21: 0x59bc9847 (stale) Mirror 22: 0x59bc9847 (stale) Mirror 23: 0x59bc9847 (stale) Mirror 24: 0x59bc9847 (stale) CRC-32 checksum value for chunk [0, 0x2b713): Mirror 1: 0x59bc9847 Mirror 25: 0x59bc9847 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b713: skipped 000652 1787119240.789532 mapwrite 0x03b2dd thru 0x03f1e0 (0x3f04 bytes) 000653 1787119240.932882 write 0x01ec21 thru 0x02d57c (0xe95c bytes) 000654 1787119240.956119 write 0x037f02 thru 0x03bfbe (0x40bd bytes) 000655 1787119240.958120 trunc from 0x03e301 to 0x03f1e0 (0xee0 bytes) 000656 1787119240.964859 trunc from 0x007565 to 0x012034 (0xaad0 bytes) 000657 1787119240.969636 trunc from 0x03f1e1 to 0x03db02 (0x16de bytes) 000658 1787119240.988210 mapread 0x0265af thru 0x035416 (0xee68 bytes) 000659 1787119241.001025 trunc from 0x03db03 to 0x03645d (0x76a5 bytes) 000660 1787119241.022663 mapwrite 0x00e8e7 thru 0x010497 (0x1bb1 bytes) 000661 1787119241.076981 trunc from 0x011c17 to 0x0134ad (0x1897 bytes) 000662 1787119241.094016 read 0x007c6f thru 0x0156d1 (0xda63 bytes) 000663 1787119241.104806 punch from 0x01f5b4 to 0x02f25c (0xfca9 bytes) 000664 1787119241.119846 mapread 0x028fa6 thru 0x02c47d (0x34d8 bytes) 000667 1787119241.136995 read 0x025b22 thru 0x02b2f0 (0x57cf bytes) 000668 1787119241.138668 mapread 0x02cc08 thru 0x03645d (0x9856 bytes) 000669 1787119241.141938 mapwrite 0x0039a9 thru 0x00dff7 (0xa64f bytes) 000670 1787119241.213848 mapwrite 0x01553e thru 0x01fe71 (0xa934 bytes) 000671 1787119241.269405 MIRROR_SPLIT mirror 10 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3645e): Mirror 5: 0x1affb1fc (stale) Mirror 7: 0x1affb1fc (stale) Mirror 12: 0x1affb1fc (stale) Mirror 14: 0x1affb1fc (stale) Mirror 15: 0x1affb1fc (stale) Mirror 16: 0x1affb1fc (stale) Mirror 17: 0x1affb1fc (stale) Mirror 18: 0x1affb1fc (stale) Mirror 19: 0x1affb1fc (stale) Mirror 21: 0x1affb1fc (stale) Mirror 22: 0x1affb1fc (stale) Mirror 23: 0x1affb1fc (stale) Mirror 24: 0x1affb1fc (stale) Mirror 25: 0x1affb1fc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3645e): Mirror 5: 0x1affb1fc (stale) Mirror 7: 0x1affb1fc (stale) Mirror 12: 0x1affb1fc (stale) Mirror 14: 0x1affb1fc (stale) Mirror 15: 0x1affb1fc (stale) Mirror 16: 0x1affb1fc (stale) Mirror 17: 0x1affb1fc (stale) Mirror 18: 0x1affb1fc (stale) Mirror 19: 0x1affb1fc (stale) Mirror 21: 0x1affb1fc (stale) Mirror 22: 0x1affb1fc (stale) Mirror 23: 0x1affb1fc (stale) Mirror 24: 0x1affb1fc (stale) Mirror 25: 0x1affb1fc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3645e: skipped 000672 1787119242.365343 read 0x01d130 thru 0x0238f0 (0x67c1 bytes) 000673 1787119242.397866 fallocate 0x007d2e thru 0x0151f7 (0xd4ca bytes) 000674 1787119242.419809 fallocate 0x0167f4 thru 0x021938 (0xb145 bytes) 000675 1787119242.435044 punch from 0x030e6a to 0x031025 (0x1bc bytes) 000676 1787119242.453760 punch from 0x00f113 to 0x01451a (0x5408 bytes) 000677 1787119242.460371 trunc from 0x0073c8 to 0x01003b (0x8c74 bytes) 000678 1787119242.474149 trunc from 0x03645e to 0x009670 (0x2cded bytes) 000679 1787119242.503810 write 0x028b64 thru 0x02a8bd (0x1d5a bytes) 000680 1787119242.510227 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x2a8be): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a8be): Mirror 5: 0x1605c2cd (stale) Mirror 7: 0x1605c2cd (stale) Mirror 12: 0x1605c2cd (stale) Mirror 14: 0x1605c2cd (stale) Mirror 15: 0x1605c2cd (stale) Mirror 16: 0x1605c2cd (stale) Mirror 17: 0x1605c2cd (stale) Mirror 18: 0x1605c2cd (stale) Mirror 19: 0x1605c2cd (stale) Mirror 21: 0x1605c2cd (stale) Mirror 22: 0x1605c2cd (stale) Mirror 23: 0x1605c2cd (stale) Mirror 24: 0x1605c2cd (stale) Mirror 25: 0x1605c2cd (stale) CRC-32 checksum value for chunk [0, 0x2a8be): Mirror 1: 0x7de34186 Mirror 26: 0x7de34186 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a8be: skipped 000681 1787119243.410782 fallocate 0x00f3e3 thru 0x015310 (0x5f2e bytes) 000683 1787119243.454831 write 0x03f825 thru 0x03ffff (0x7db bytes) 000684 1787119243.461791 trunc from 0x03a962 to 0x03ffff (0x569e bytes) 000685 1787119243.469938 write 0x028fd3 thru 0x035306 (0xc334 bytes) 000686 1787119243.476194 write 0x028574 thru 0x036f4a (0xe9d7 bytes) 000687 1787119243.480056 read 0x00dbe4 thru 0x0145b5 (0x69d2 bytes) 000688 1787119243.487933 punch from 0x00cc94 to 0x0140bf (0x742c bytes) 000689 1787119243.491821 mapread 0x028467 thru 0x032c99 (0xa833 bytes) 000690 1787119243.492974 mapread 0x02b4b0 thru 0x032569 (0x70ba bytes) 000691 1787119243.494053 mapread 0x03fdb0 thru 0x03ffff (0x250 bytes) 000692 1787119243.500109 fallocate 0x012ad5 thru 0x01ea5c (0xbf88 bytes) 000695 1787119243.507567 write 0x0387fe thru 0x03ffff (0x7802 bytes) 000696 1787119243.514274 write 0x03b7cb thru 0x03d89a (0x20d0 bytes) 000697 1787119243.516777 write 0x03355d thru 0x03bc76 (0x871a bytes) 000698 1787119243.520093 mapwrite 0x02df58 thru 0x038d9b (0xae44 bytes) 000699 1787119243.555103 write 0x03f052 thru 0x03ffff (0xfae bytes) 000700 1787119243.556114 read 0x023218 thru 0x032baa (0xf993 bytes) 000701 1787119243.580634 MIRROR_RESYNC 16 mirrors 000702 1787119246.448699 punch from 0x03c4ba to 0x03ffff (0x3b46 bytes) 000704 1787119246.528741 MIRROR_SPLIT mirror 9 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x17e1514f (stale) Mirror 7: 0x17e1514f (stale) Mirror 12: 0x17e1514f (stale) Mirror 14: 0x17e1514f (stale) Mirror 15: 0x17e1514f (stale) Mirror 16: 0x17e1514f (stale) Mirror 17: 0x17e1514f (stale) Mirror 18: 0x17e1514f (stale) Mirror 21: 0x17e1514f (stale) Mirror 22: 0x17e1514f (stale) Mirror 23: 0x17e1514f (stale) Mirror 24: 0x17e1514f (stale) Mirror 25: 0x17e1514f (stale) Mirror 26: 0x17e1514f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x17e1514f (stale) Mirror 7: 0x17e1514f (stale) Mirror 12: 0x17e1514f (stale) Mirror 14: 0x17e1514f (stale) Mirror 15: 0x17e1514f (stale) Mirror 16: 0x17e1514f (stale) Mirror 17: 0x17e1514f (stale) Mirror 18: 0x17e1514f (stale) Mirror 21: 0x17e1514f (stale) Mirror 22: 0x17e1514f (stale) Mirror 23: 0x17e1514f (stale) Mirror 24: 0x17e1514f (stale) Mirror 25: 0x17e1514f (stale) Mirror 26: 0x17e1514f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000705 1787119247.696047 trunc from 0x040000 to 0x012482 (0x2db7d bytes) 000706 1787119247.718158 fallocate 0x02a2a1 thru 0x0384c1 (0xe221 bytes) 000707 1787119247.725514 mapwrite 0x000136 thru 0x00f67f (0xf54a bytes) 000708 1787119247.784553 trunc from 0x02ed5c to 0x03321f (0x44c4 bytes) 000709 1787119247.795843 trunc from 0x01d4f3 to 0x023b3e (0x664c bytes) 000710 1787119247.802936 trunc from 0x00bdcc to 0x015268 (0x949d bytes) 000711 1787119247.815416 fallocate 0x02520f thru 0x02dadf (0x88d1 bytes) 000712 1787119247.841029 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x384c2): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x384c2): Mirror 5: 0x56b969e5 (stale) Mirror 7: 0x56b969e5 (stale) Mirror 12: 0x56b969e5 (stale) Mirror 14: 0x56b969e5 (stale) Mirror 15: 0x56b969e5 (stale) Mirror 16: 0x56b969e5 (stale) Mirror 17: 0x56b969e5 (stale) Mirror 18: 0x56b969e5 (stale) Mirror 21: 0x56b969e5 (stale) Mirror 22: 0x56b969e5 (stale) Mirror 23: 0x56b969e5 (stale) Mirror 24: 0x56b969e5 (stale) Mirror 25: 0x56b969e5 (stale) Mirror 26: 0x56b969e5 (stale) CRC-32 checksum value for chunk [0, 0x384c2): Mirror 1: 0x9585b3d4 Mirror 27: 0x9585b3d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x384c2: skipped 000713 1787119249.158739 read 0x014d2d thru 0x015946 (0xc1a bytes) 000714 1787119249.173829 write 0x00e6b0 thru 0x01c68d (0xdfde bytes) 000716 1787119249.236183 fallocate 0x024e54 thru 0x033dd5 (0xef82 bytes) 000717 1787119249.241464 trunc from 0x00f49b to 0x011455 (0x1fbb bytes) 000718 1787119249.263070 mapwrite 0x029223 thru 0x03136c (0x814a bytes) 000719 1787119249.294932 mapwrite 0x008a22 thru 0x00ca8f (0x406e bytes) 000720 1787119249.324598 fallocate 0x00a348 thru 0x00bdcf (0x1a88 bytes) 000721 1787119249.329381 punch from 0x007555 to 0x00cf11 (0x59bd bytes) 000722 1787119249.334607 fallocate 0x031c92 thru 0x0368e9 (0x4c58 bytes) 000723 1787119249.339386 fallocate 0x03a484 thru 0x03ffff (0x5b7c bytes) 000724 1787119249.344942 fallocate 0x03e13a thru 0x03ffff (0x1ec6 bytes) 000725 1787119249.353731 MIRROR_RESYNC 16 mirrors 000726 1787119254.412391 mapread 0x003285 thru 0x01277d (0xf4f9 bytes) 000727 1787119254.436536 write 0x03fe94 thru 0x03ffff (0x16c bytes) 000728 1787119254.474743 mapread 0x01143b thru 0x020b21 (0xf6e7 bytes) 000729 1787119254.493738 trunc from 0x0243bf to 0x032134 (0xdd76 bytes) 000730 1787119254.498871 trunc from 0x018f67 to 0x020b0d (0x7ba7 bytes) 000731 1787119254.508698 MIRROR_SPLIT mirror 7 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x589fb5ff (stale) Mirror 7: 0x589fb5ff (stale) Mirror 12: 0x589fb5ff (stale) Mirror 14: 0x589fb5ff (stale) Mirror 15: 0x589fb5ff (stale) Mirror 16: 0x589fb5ff (stale) Mirror 18: 0x589fb5ff (stale) Mirror 21: 0x589fb5ff (stale) Mirror 22: 0x589fb5ff (stale) Mirror 23: 0x589fb5ff (stale) Mirror 24: 0x589fb5ff (stale) Mirror 25: 0x589fb5ff (stale) Mirror 26: 0x589fb5ff (stale) Mirror 27: 0x589fb5ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x589fb5ff (stale) Mirror 7: 0x589fb5ff (stale) Mirror 12: 0x589fb5ff (stale) Mirror 14: 0x589fb5ff (stale) Mirror 15: 0x589fb5ff (stale) Mirror 16: 0x589fb5ff (stale) Mirror 18: 0x589fb5ff (stale) Mirror 21: 0x589fb5ff (stale) Mirror 22: 0x589fb5ff (stale) Mirror 23: 0x589fb5ff (stale) Mirror 24: 0x589fb5ff (stale) Mirror 25: 0x589fb5ff (stale) Mirror 26: 0x589fb5ff (stale) Mirror 27: 0x589fb5ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000732 1787119255.274808 read 0x016ade thru 0x01717e (0x6a1 bytes) 000733 1787119255.280122 trunc from 0x021c26 to 0x030890 (0xec6b bytes) 000734 1787119255.286057 read 0x0131d7 thru 0x020f4b (0xdd75 bytes) 000735 1787119255.296951 mapread 0x00a46f thru 0x00c2a2 (0x1e34 bytes) 000736 1787119255.298049 fallocate 0x0281c9 thru 0x03509a (0xced2 bytes) 000738 1787119255.303413 read 0x01301b thru 0x015b55 (0x2b3b bytes) 000740 1787119255.305476 mapread 0x02f581 thru 0x0323e9 (0x2e69 bytes) 000741 1787119255.306983 trunc from 0x03d103 to 0x03ffff (0x2efd bytes) 000742 1787119255.315828 MIRROR_RESYNC 15 mirrors 000743 1787119258.908884 write 0x02d0c4 thru 0x035f02 (0x8e3f bytes) 000744 1787119258.969879 fallocate 0x03fb7a thru 0x03ffff (0x486 bytes) 000745 1787119258.979961 fallocate 0x02f7b6 thru 0x0346b1 (0x4efc bytes) 000747 1787119258.991754 fallocate 0x03af3a thru 0x03ffff (0x50c6 bytes) 000748 1787119259.003098 write 0x03879f thru 0x03ffff (0x7861 bytes) 000749 1787119259.016718 read 0x001c55 thru 0x00e0ba (0xc466 bytes) 000750 1787119259.047148 read 0x006237 thru 0x0154bc (0xf286 bytes) 000751 1787119259.063040 mapread 0x02ebba thru 0x031093 (0x24da bytes) 000752 1787119259.064089 trunc from 0x03bd33 to 0x03beb9 (0x187 bytes) 000753 1787119259.101792 fallocate 0x002df4 thru 0x005a32 (0x2c3f bytes) 000754 1787119259.110898 fallocate 0x01100b thru 0x015bd7 (0x4bcd bytes) 000755 1787119259.115492 mapwrite 0x0108ed thru 0x01af29 (0xa63d bytes) 000756 1787119259.140094 read 0x02c866 thru 0x03b118 (0xe8b3 bytes) 000757 1787119259.155135 mapwrite 0x0063ca thru 0x00d255 (0x6e8c bytes) 000758 1787119259.222960 MIRROR_SPLIT mirror 2 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xc36680e1 (stale) Mirror 12: 0xc36680e1 (stale) Mirror 14: 0xc36680e1 (stale) Mirror 15: 0xc36680e1 (stale) Mirror 16: 0xc36680e1 (stale) Mirror 18: 0xc36680e1 (stale) Mirror 21: 0xc36680e1 (stale) Mirror 22: 0xc36680e1 (stale) Mirror 23: 0xc36680e1 (stale) Mirror 24: 0xc36680e1 (stale) Mirror 25: 0xc36680e1 (stale) Mirror 26: 0xc36680e1 (stale) Mirror 27: 0xc36680e1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xc36680e1 (stale) Mirror 12: 0xc36680e1 (stale) Mirror 14: 0xc36680e1 (stale) Mirror 15: 0xc36680e1 (stale) Mirror 16: 0xc36680e1 (stale) Mirror 18: 0xc36680e1 (stale) Mirror 21: 0xc36680e1 (stale) Mirror 22: 0xc36680e1 (stale) Mirror 23: 0xc36680e1 (stale) Mirror 24: 0xc36680e1 (stale) Mirror 25: 0xc36680e1 (stale) Mirror 26: 0xc36680e1 (stale) Mirror 27: 0xc36680e1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000760 1787119259.812909 mapwrite 0x01cb2a thru 0x0234df (0x69b6 bytes) 000761 1787119259.848340 punch from 0x03291b to 0x03ddb0 (0xb496 bytes) 000762 1787119259.854161 mapwrite 0x008938 thru 0x010b61 (0x822a bytes) 000764 1787119259.881757 punch from 0x03be95 to 0x03ffff (0x416b bytes) 000765 1787119259.885801 write 0x0249fe thru 0x02dd26 (0x9329 bytes) 000766 1787119259.899079 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xc36680e1 (stale) Mirror 12: 0xc36680e1 (stale) Mirror 14: 0xc36680e1 (stale) Mirror 15: 0xc36680e1 (stale) Mirror 16: 0xc36680e1 (stale) Mirror 18: 0xc36680e1 (stale) Mirror 21: 0xc36680e1 (stale) Mirror 22: 0xc36680e1 (stale) Mirror 23: 0xc36680e1 (stale) Mirror 24: 0xc36680e1 (stale) Mirror 25: 0xc36680e1 (stale) Mirror 26: 0xc36680e1 (stale) Mirror 27: 0xc36680e1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x137da893 Mirror 28: 0x137da893 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000767 1787119261.312841 punch from 0x020562 to 0x027dd8 (0x7877 bytes) 000768 1787119261.370812 write 0x0050d7 thru 0x0055a6 (0x4d0 bytes) 000769 1787119261.381850 write 0x0297f8 thru 0x03290e (0x9117 bytes) 000770 1787119261.396620 MIRROR_SPLIT mirror 4 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xc36680e1 (stale) Mirror 12: 0xc36680e1 (stale) Mirror 14: 0xc36680e1 (stale) Mirror 16: 0xc36680e1 (stale) Mirror 18: 0xc36680e1 (stale) Mirror 21: 0xc36680e1 (stale) Mirror 22: 0xc36680e1 (stale) Mirror 23: 0xc36680e1 (stale) Mirror 24: 0xc36680e1 (stale) Mirror 25: 0xc36680e1 (stale) Mirror 26: 0xc36680e1 (stale) Mirror 27: 0xc36680e1 (stale) Mirror 28: 0x137da893 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xc36680e1 (stale) Mirror 12: 0xc36680e1 (stale) Mirror 14: 0xc36680e1 (stale) Mirror 16: 0xc36680e1 (stale) Mirror 18: 0xc36680e1 (stale) Mirror 21: 0xc36680e1 (stale) Mirror 22: 0xc36680e1 (stale) Mirror 23: 0xc36680e1 (stale) Mirror 24: 0xc36680e1 (stale) Mirror 25: 0xc36680e1 (stale) Mirror 26: 0xc36680e1 (stale) Mirror 27: 0xc36680e1 (stale) Mirror 28: 0x137da893 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000771 1787119262.204374 MIRROR_RESYNC 14 mirrors 000772 1787119265.383485 punch from 0x006d78 to 0x00dd94 (0x701d bytes) 000773 1787119265.422052 mapread 0x0358e6 thru 0x036b4a (0x1265 bytes) 000774 1787119265.431204 trunc from 0x040000 to 0x028ab0 (0x1754f bytes) 000775 1787119265.456658 MIRROR_RESYNC 14 mirrors 000777 1787119268.527832 trunc from 0x025618 to 0x028ab0 (0x3499 bytes) 000778 1787119268.553853 punch from 0x00e96d to 0x0107ac (0x1e40 bytes) 000779 1787119268.557861 mapwrite 0x03d542 thru 0x03d95c (0x41b bytes) 000780 1787119268.612215 punch from 0x00bc7c to 0x011158 (0x54dd bytes) 000781 1787119268.635389 MIRROR_SPLIT mirror 1 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d95d): Mirror 12: 0x1d4de381 (stale) Mirror 14: 0x1d4de381 (stale) Mirror 16: 0x1d4de381 (stale) Mirror 18: 0x1d4de381 (stale) Mirror 21: 0x1d4de381 (stale) Mirror 22: 0x1d4de381 (stale) Mirror 23: 0x1d4de381 (stale) Mirror 24: 0x1d4de381 (stale) Mirror 25: 0x1d4de381 (stale) Mirror 26: 0x1d4de381 (stale) Mirror 27: 0x1d4de381 (stale) Mirror 28: 0x1d4de381 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d95d): Mirror 12: 0x1d4de381 (stale) Mirror 14: 0x1d4de381 (stale) Mirror 16: 0x1d4de381 (stale) Mirror 18: 0x1d4de381 (stale) Mirror 21: 0x1d4de381 (stale) Mirror 22: 0x1d4de381 (stale) Mirror 23: 0x1d4de381 (stale) Mirror 24: 0x1d4de381 (stale) Mirror 25: 0x1d4de381 (stale) Mirror 26: 0x1d4de381 (stale) Mirror 27: 0x1d4de381 (stale) Mirror 28: 0x1d4de381 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d95d: skipped 000782 1787119269.397610 punch from 0x02a808 to 0x02ad19 (0x512 bytes) 000784 1787119269.403866 mapwrite 0x01cf08 thru 0x026750 (0x9849 bytes) 000785 1787119269.436782 fallocate 0x02fa86 thru 0x03c219 (0xc794 bytes) 000786 1787119269.442381 punch from 0x018fde to 0x01d577 (0x459a bytes) 000787 1787119269.447807 write 0x0169f7 thru 0x0219a8 (0xafb2 bytes) 000788 1787119269.457934 mapwrite 0x025fb4 thru 0x02fc1b (0x9c68 bytes) 000789 1787119269.483829 mapwrite 0x03b3cd thru 0x03ed70 (0x39a4 bytes) 000791 1787119269.512331 mapread 0x0317b4 thru 0x031f57 (0x7a4 bytes) 000792 1787119269.519399 read 0x0158c4 thru 0x01968e (0x3dcb bytes) 000793 1787119269.523980 mapread 0x01df06 thru 0x01f81c (0x1917 bytes) 000794 1787119269.524999 trunc from 0x0342e8 to 0x037cf7 (0x3a10 bytes) 000795 1787119269.530143 punch from 0x022970 to 0x02b2af (0x8940 bytes) 000796 1787119269.534775 punch from 0x01412c to 0x023d59 (0xfc2e bytes) 000797 1787119269.540291 mapread 0x036d10 thru 0x03730f (0x600 bytes) 000798 1787119269.548741 trunc from 0x03ed71 to 0x022d2b (0x1c045 bytes) 000800 1787119269.568040 mapread 0x021985 thru 0x02257b (0xbf7 bytes) 000801 1787119269.575981 read 0x019228 thru 0x022d2b (0x9b04 bytes) 000802 1787119269.588616 punch from 0x00553c to 0x0057d3 (0x298 bytes) 000804 1787119269.594682 write 0x0241d9 thru 0x02fc05 (0xba2d bytes) 000805 1787119269.596871 trunc from 0x022451 to 0x026c89 (0x4839 bytes) 000806 1787119269.615651 fallocate 0x006a2a thru 0x00f92d (0x8f04 bytes) 000807 1787119269.623873 write 0x0059fb thru 0x00dff7 (0x85fd bytes) 000808 1787119269.632938 punch from 0x007804 to 0x010c68 (0x9465 bytes) 000809 1787119269.645528 read 0x01fc73 thru 0x02d813 (0xdba1 bytes) 000810 1787119269.654823 fallocate 0x03f127 thru 0x03ffff (0xed9 bytes) 000811 1787119269.659440 mapread 0x02daa3 thru 0x03c301 (0xe85f bytes) 000812 1787119269.671530 trunc from 0x040000 to 0x03c7a3 (0x385c bytes) 000813 1787119269.690051 fallocate 0x00a78e thru 0x01750c (0xcd7f bytes) 000814 1787119269.697708 mapwrite 0x0127f7 thru 0x020809 (0xe013 bytes) 000815 1787119269.726837 mapwrite 0x014892 thru 0x024725 (0xfe94 bytes) 000816 1787119269.751517 fallocate 0x02f3d2 thru 0x031f48 (0x2b77 bytes) 000817 1787119269.756869 fallocate 0x003ad1 thru 0x003b3b (0x6b bytes) 000818 1787119269.765345 read 0x021b06 thru 0x02d4dd (0xb9d8 bytes) 000819 1787119269.767045 read 0x01403d thru 0x023905 (0xf8c9 bytes) 000820 1787119269.768703 punch from 0x0051e6 to 0x00e27b (0x9096 bytes) 000821 1787119269.775114 fallocate 0x03710f thru 0x03c118 (0x500a bytes) 000822 1787119269.781208 trunc from 0x03108e to 0x03bc30 (0xaba3 bytes) 000823 1787119269.793185 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3c7a4): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c7a4): Mirror 12: 0xdadc000d (stale) Mirror 14: 0xdadc000d (stale) Mirror 16: 0xdadc000d (stale) Mirror 18: 0xdadc000d (stale) Mirror 21: 0xdadc000d (stale) Mirror 22: 0xdadc000d (stale) Mirror 23: 0xdadc000d (stale) Mirror 24: 0xdadc000d (stale) Mirror 25: 0xdadc000d (stale) Mirror 26: 0xdadc000d (stale) Mirror 27: 0xdadc000d (stale) Mirror 28: 0xdadc000d (stale) CRC-32 checksum value for chunk [0, 0x3c7a4): Mirror 1: 0x13176dfe Mirror 29: 0x13176dfe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c7a4: skipped 000824 1787119270.769932 punch from 0x01eb18 to 0x02ad38 (0xc221 bytes) 000825 1787119270.818738 punch from 0x020147 to 0x025d6e (0x5c28 bytes) 000826 1787119270.823618 read 0x015632 thru 0x0200e9 (0xaab8 bytes) 000827 1787119270.832466 fallocate 0x027a0c thru 0x02b596 (0x3b8b bytes) 000828 1787119270.836846 read 0x00511a thru 0x005cc5 (0xbac bytes) 000829 1787119270.847424 trunc from 0x03c7a4 to 0x00a7b9 (0x31fea bytes) 000830 1787119270.870917 trunc from 0x00a7ba to 0x00ffda (0x5821 bytes) 000831 1787119270.889499 MIRROR_RESYNC 14 mirrors 000832 1787119272.376043 mapread 0x00774f thru 0x00df66 (0x6818 bytes) 000833 1787119272.393464 mapwrite 0x032e98 thru 0x037d77 (0x4ee0 bytes) 000834 1787119272.462710 trunc from 0x037d78 to 0x0389af (0xc38 bytes) 000835 1787119272.481970 write 0x01a768 thru 0x0262c5 (0xbb5e bytes) 000836 1787119272.495602 MIRROR_SPLIT mirror 5 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x389b0): Mirror 12: 0x7984f546 (stale) Mirror 14: 0x7984f546 (stale) Mirror 16: 0x7984f546 (stale) Mirror 18: 0x7984f546 (stale) Mirror 22: 0x7984f546 (stale) Mirror 23: 0x7984f546 (stale) Mirror 24: 0x7984f546 (stale) Mirror 25: 0x7984f546 (stale) Mirror 26: 0x7984f546 (stale) Mirror 27: 0x7984f546 (stale) Mirror 28: 0x7984f546 (stale) Mirror 29: 0x7984f546 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x389b0): Mirror 12: 0x7984f546 (stale) Mirror 14: 0x7984f546 (stale) Mirror 16: 0x7984f546 (stale) Mirror 18: 0x7984f546 (stale) Mirror 22: 0x7984f546 (stale) Mirror 23: 0x7984f546 (stale) Mirror 24: 0x7984f546 (stale) Mirror 25: 0x7984f546 (stale) Mirror 26: 0x7984f546 (stale) Mirror 27: 0x7984f546 (stale) Mirror 28: 0x7984f546 (stale) Mirror 29: 0x7984f546 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x389b0: skipped 000837 1787119273.028706 punch from 0x02217a to 0x0272a3 (0x512a bytes) 000838 1787119273.033919 trunc from 0x0389b0 to 0x01e755 (0x1a25a bytes) 000840 1787119273.052001 punch from 0x00ffaa to 0x01dc60 (0xdcb7 bytes) 000841 1787119273.061028 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x1e756): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e756): Mirror 12: 0x7545c321 (stale) Mirror 14: 0x7545c321 (stale) Mirror 16: 0x7545c321 (stale) Mirror 18: 0x7545c321 (stale) Mirror 22: 0x7545c321 (stale) Mirror 23: 0x7545c321 (stale) Mirror 24: 0x7545c321 (stale) Mirror 25: 0x7545c321 (stale) Mirror 26: 0x7545c321 (stale) Mirror 27: 0x7545c321 (stale) Mirror 28: 0x7545c321 (stale) Mirror 29: 0x7545c321 (stale) CRC-32 checksum value for chunk [0, 0x1e756): Mirror 1: 0x30c72b31 Mirror 30: 0x30c72b31 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e756: skipped 000842 1787119273.971164 read 0x01dff6 thru 0x01e755 (0x760 bytes) 000845 1787119273.999161 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x1e756): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e756): Mirror 12: 0x7545c321 (stale) Mirror 14: 0x7545c321 (stale) Mirror 16: 0x7545c321 (stale) Mirror 18: 0x7545c321 (stale) Mirror 22: 0x7545c321 (stale) Mirror 23: 0x7545c321 (stale) Mirror 24: 0x7545c321 (stale) Mirror 25: 0x7545c321 (stale) Mirror 26: 0x7545c321 (stale) Mirror 27: 0x7545c321 (stale) Mirror 28: 0x7545c321 (stale) Mirror 29: 0x7545c321 (stale) CRC-32 checksum value for chunk [0, 0x1e756): Mirror 1: 0x30c72b31 Mirror 30: 0x30c72b31 Mirror 31: 0x30c72b31 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e756: skipped 000846 1787119274.932071 mapread 0x009070 thru 0x00c326 (0x32b7 bytes) 000847 1787119274.955254 mapread 0x0118b9 thru 0x01c340 (0xaa88 bytes) 000848 1787119274.963707 fallocate 0x0119b3 thru 0x0166d6 (0x4d24 bytes) 000850 1787119275.001985 trunc from 0x015cd0 to 0x01da62 (0x7d93 bytes) 000851 1787119275.006074 punch from 0x005c32 to 0x00e816 (0x8be5 bytes) 000852 1787119275.009158 write 0x03947f thru 0x03ffff (0x6b81 bytes) 000853 1787119275.012577 trunc from 0x02254d to 0x02605a (0x3b0e bytes) 000854 1787119275.019349 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xf138ee3f (stale) Mirror 14: 0xf138ee3f (stale) Mirror 16: 0xf138ee3f (stale) Mirror 18: 0xf138ee3f (stale) Mirror 22: 0xf138ee3f (stale) Mirror 23: 0xf138ee3f (stale) Mirror 24: 0xf138ee3f (stale) Mirror 25: 0xf138ee3f (stale) Mirror 26: 0xf138ee3f (stale) Mirror 27: 0xf138ee3f (stale) Mirror 28: 0xf138ee3f (stale) Mirror 29: 0xf138ee3f (stale) Mirror 30: 0xfa73a0a5 (stale) Mirror 31: 0xfa73a0a5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x377c580d Mirror 32: 0x377c580d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000855 1787119276.264569 trunc from 0x03c21b to 0x03ffff (0x3de5 bytes) 000857 1787119276.335635 mapread 0x03f5aa thru 0x03ffff (0xa56 bytes) 000858 1787119276.343812 read 0x02b7fe thru 0x035cc9 (0xa4cc bytes) 000859 1787119276.355365 punch from 0x032117 to 0x03ffff (0xdee9 bytes) 000860 1787119276.366488 MIRROR_SPLIT mirror 7 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xf138ee3f (stale) Mirror 14: 0xf138ee3f (stale) Mirror 16: 0xf138ee3f (stale) Mirror 18: 0xf138ee3f (stale) Mirror 22: 0xf138ee3f (stale) Mirror 23: 0xf138ee3f (stale) Mirror 25: 0xf138ee3f (stale) Mirror 26: 0xf138ee3f (stale) Mirror 27: 0xf138ee3f (stale) Mirror 28: 0xf138ee3f (stale) Mirror 29: 0xf138ee3f (stale) Mirror 30: 0xfa73a0a5 (stale) Mirror 31: 0xfa73a0a5 (stale) Mirror 32: 0x377c580d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0xf138ee3f (stale) Mirror 14: 0xf138ee3f (stale) Mirror 16: 0xf138ee3f (stale) Mirror 18: 0xf138ee3f (stale) Mirror 22: 0xf138ee3f (stale) Mirror 23: 0xf138ee3f (stale) Mirror 25: 0xf138ee3f (stale) Mirror 26: 0xf138ee3f (stale) Mirror 27: 0xf138ee3f (stale) Mirror 28: 0xf138ee3f (stale) Mirror 29: 0xf138ee3f (stale) Mirror 30: 0xfa73a0a5 (stale) Mirror 31: 0xfa73a0a5 (stale) Mirror 32: 0x377c580d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000861 1787119277.156469 write 0x02f38e thru 0x038664 (0x92d7 bytes) 000862 1787119277.173485 trunc from 0x022886 to 0x02c38f (0x9b0a bytes) 000863 1787119277.180577 trunc from 0x029b4e to 0x02bfb1 (0x2464 bytes) 000864 1787119277.193513 MIRROR_RESYNC 15 mirrors 000865 1787119280.540126 MIRROR_SPLIT mirror 11 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) 000866 1787119281.019998 fallocate 0x004c72 thru 0x00f7bf (0xab4e bytes) 000867 1787119281.059189 fallocate 0x035f42 thru 0x03ffff (0xa0be bytes) 000868 1787119281.066812 mapread 0x019f58 thru 0x025306 (0xb3af bytes) 000869 1787119281.096202 trunc from 0x0350a9 to 0x03ceb9 (0x7e11 bytes) 000870 1787119281.101831 fallocate 0x00dcc6 thru 0x01b7d9 (0xdb14 bytes) 000871 1787119281.108393 read 0x01abc4 thru 0x01ae4d (0x28a bytes) 000872 1787119281.109994 mapwrite 0x0071e6 thru 0x00a26b (0x3086 bytes) 000873 1787119281.136037 fallocate 0x036845 thru 0x039abf (0x327b bytes) 000874 1787119281.142536 write 0x036a71 thru 0x03ffff (0x958f bytes) 000875 1787119281.161018 trunc from 0x0251db to 0x02f737 (0xa55d bytes) 000876 1787119281.166635 fallocate 0x02b35f thru 0x02f53e (0x41e0 bytes) 000877 1787119281.171484 punch from 0x0356e1 to 0x03ffff (0xa91f bytes) 000878 1787119281.186422 fallocate 0x0299e4 thru 0x02ac7b (0x1298 bytes) 000879 1787119281.198400 MIRROR_RESYNC 14 mirrors 000880 1787119284.217829 fallocate 0x03490e thru 0x03e47f (0x9b72 bytes) 000881 1787119284.268165 mapwrite 0x027369 thru 0x02c1f7 (0x4e8f bytes) 000882 1787119284.329766 MIRROR_SPLIT mirror 6 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x2f5296c2 (stale) Mirror 14: 0x2f5296c2 (stale) Mirror 16: 0x2f5296c2 (stale) Mirror 18: 0x2f5296c2 (stale) Mirror 22: 0x2f5296c2 (stale) Mirror 25: 0x2f5296c2 (stale) Mirror 26: 0x2f5296c2 (stale) Mirror 27: 0x2f5296c2 (stale) Mirror 28: 0x2f5296c2 (stale) Mirror 30: 0x2f5296c2 (stale) Mirror 31: 0x2f5296c2 (stale) Mirror 32: 0x2f5296c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x2f5296c2 (stale) Mirror 14: 0x2f5296c2 (stale) Mirror 16: 0x2f5296c2 (stale) Mirror 18: 0x2f5296c2 (stale) Mirror 22: 0x2f5296c2 (stale) Mirror 25: 0x2f5296c2 (stale) Mirror 26: 0x2f5296c2 (stale) Mirror 27: 0x2f5296c2 (stale) Mirror 28: 0x2f5296c2 (stale) Mirror 30: 0x2f5296c2 (stale) Mirror 31: 0x2f5296c2 (stale) Mirror 32: 0x2f5296c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000883 1787119284.844175 write 0x0266b7 thru 0x02a825 (0x416f bytes) 000884 1787119284.859533 mapread 0x03f713 thru 0x03ffff (0x8ed bytes) 000885 1787119284.864337 mapwrite 0x03e9a4 thru 0x03ffff (0x165c bytes) 000886 1787119284.880289 trunc from 0x005878 to 0x00f9ca (0xa153 bytes) 000887 1787119284.889580 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x2f5296c2 (stale) Mirror 16: 0x2f5296c2 (stale) Mirror 18: 0x2f5296c2 (stale) Mirror 22: 0x2f5296c2 (stale) Mirror 25: 0x2f5296c2 (stale) Mirror 26: 0x2f5296c2 (stale) Mirror 27: 0x2f5296c2 (stale) Mirror 28: 0x2f5296c2 (stale) Mirror 30: 0x2f5296c2 (stale) Mirror 31: 0x2f5296c2 (stale) Mirror 32: 0x2f5296c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x2f5296c2 (stale) Mirror 16: 0x2f5296c2 (stale) Mirror 18: 0x2f5296c2 (stale) Mirror 22: 0x2f5296c2 (stale) Mirror 25: 0x2f5296c2 (stale) Mirror 26: 0x2f5296c2 (stale) Mirror 27: 0x2f5296c2 (stale) Mirror 28: 0x2f5296c2 (stale) Mirror 30: 0x2f5296c2 (stale) Mirror 31: 0x2f5296c2 (stale) Mirror 32: 0x2f5296c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000889 1787119285.462490 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x2f5296c2 (stale) Mirror 18: 0x2f5296c2 (stale) Mirror 22: 0x2f5296c2 (stale) Mirror 25: 0x2f5296c2 (stale) Mirror 26: 0x2f5296c2 (stale) Mirror 27: 0x2f5296c2 (stale) Mirror 28: 0x2f5296c2 (stale) Mirror 30: 0x2f5296c2 (stale) Mirror 31: 0x2f5296c2 (stale) Mirror 32: 0x2f5296c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0x2f5296c2 (stale) Mirror 18: 0x2f5296c2 (stale) Mirror 22: 0x2f5296c2 (stale) Mirror 25: 0x2f5296c2 (stale) Mirror 26: 0x2f5296c2 (stale) Mirror 27: 0x2f5296c2 (stale) Mirror 28: 0x2f5296c2 (stale) Mirror 30: 0x2f5296c2 (stale) Mirror 31: 0x2f5296c2 (stale) Mirror 32: 0x2f5296c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000890 1787119285.976920 write 0x01b9a8 thru 0x02341d (0x7a76 bytes) 000891 1787119285.992678 mapread 0x009d2b thru 0x00f77e (0x5a54 bytes) 000892 1787119286.010616 trunc from 0x040000 to 0x02440e (0x1bbf1 bytes) 000894 1787119286.026432 punch from 0x009e4f to 0x013eda (0xa08c bytes) 000895 1787119286.032700 punch from 0x017636 to 0x018a25 (0x13f0 bytes) 000896 1787119286.037596 trunc from 0x02440f to 0x012e87 (0x11587 bytes) 000897 1787119286.050406 write 0x01efce thru 0x027f9d (0x8fd0 bytes) 000898 1787119286.052145 read 0x026b0a thru 0x027f9d (0x1494 bytes) 000899 1787119286.053731 punch from 0x00af55 to 0x012b23 (0x7bcf bytes) 000900 1787119286.057914 mapwrite 0x03882f thru 0x038b34 (0x306 bytes) 000901 1787119286.096324 fallocate 0x024ca3 thru 0x02723a (0x2598 bytes) 000902 1787119286.101579 punch from 0x0389f0 to 0x038b34 (0x145 bytes) 000903 1787119286.109687 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x38b35): Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38b35): Mirror 14: 0x6a5d255 (stale) Mirror 18: 0x6a5d255 (stale) Mirror 22: 0x6a5d255 (stale) Mirror 25: 0x6a5d255 (stale) Mirror 26: 0x6a5d255 (stale) Mirror 27: 0x6a5d255 (stale) Mirror 28: 0x6a5d255 (stale) Mirror 30: 0x6a5d255 (stale) Mirror 31: 0x6a5d255 (stale) Mirror 32: 0x6a5d255 (stale) CRC-32 checksum value for chunk [0, 0x38b35): Mirror 1: 0x9c3027fd Mirror 33: 0x9c3027fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38b35: skipped 000904 1787119287.117871 mapread 0x013e58 thru 0x01e7f0 (0xa999 bytes) 000905 1787119287.193080 write 0x01e819 thru 0x02c3d1 (0xdbb9 bytes) 000906 1787119287.310449 MIRROR_RESYNC 12 mirrors 000907 1787119290.408032 trunc from 0x029bed to 0x038b34 (0xef48 bytes) 000908 1787119290.466728 fallocate 0x0203ec thru 0x02763c (0x7251 bytes) 000909 1787119290.473083 punch from 0x033121 to 0x038b34 (0x5a14 bytes) 000910 1787119290.479103 trunc from 0x0379a5 to 0x038b34 (0x1190 bytes) 000911 1787119290.489837 punch from 0x02e84b to 0x038b34 (0xa2ea bytes) 000912 1787119290.507265 read 0x011bdc thru 0x013e54 (0x2279 bytes) 000913 1787119290.516587 mapwrite 0x013b43 thru 0x0142bd (0x77b bytes) 000915 1787119290.547576 mapwrite 0x021a38 thru 0x0234dc (0x1aa5 bytes) 000916 1787119290.570531 trunc from 0x038b35 to 0x00d061 (0x2bad3 bytes) 000918 1787119290.589298 write 0x02d851 thru 0x033efc (0x66ac bytes) 000919 1787119290.590825 read 0x01614e thru 0x0192c2 (0x3175 bytes) 000920 1787119290.641344 read 0x027f18 thru 0x02ed0d (0x6df6 bytes) 000921 1787119290.642100 trunc from 0x033efd to 0x03c2bd (0x83c1 bytes) 000922 1787119290.679529 fallocate 0x028d00 thru 0x031b6d (0x8e6e bytes) 000923 1787119290.700223 MIRROR_RESYNC 12 mirrors 000924 1787119293.353668 mapread 0x020a37 thru 0x02f7b5 (0xed7f bytes) 000925 1787119293.425961 read 0x0397d0 thru 0x039cc7 (0x4f8 bytes) 000926 1787119293.430756 mapwrite 0x00ed62 thru 0x018902 (0x9ba1 bytes) 000927 1787119293.684275 mapwrite 0x004101 thru 0x013cae (0xfbae bytes) 000928 1787119293.771552 mapread 0x03b53c thru 0x03c2bd (0xd82 bytes) 000929 1787119293.781780 punch from 0x03648f to 0x03bc1c (0x578e bytes) 000930 1787119293.793215 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c2be): Mirror 18: 0xed6532aa (stale) Mirror 22: 0xed6532aa (stale) Mirror 25: 0xed6532aa (stale) Mirror 26: 0xed6532aa (stale) Mirror 27: 0xed6532aa (stale) Mirror 28: 0xed6532aa (stale) Mirror 30: 0xed6532aa (stale) Mirror 31: 0xed6532aa (stale) Mirror 32: 0xed6532aa (stale) Mirror 33: 0xed6532aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c2be): Mirror 18: 0xed6532aa (stale) Mirror 22: 0xed6532aa (stale) Mirror 25: 0xed6532aa (stale) Mirror 26: 0xed6532aa (stale) Mirror 27: 0xed6532aa (stale) Mirror 28: 0xed6532aa (stale) Mirror 30: 0xed6532aa (stale) Mirror 31: 0xed6532aa (stale) Mirror 32: 0xed6532aa (stale) Mirror 33: 0xed6532aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c2be: skipped 000931 1787119294.543576 mapread 0x03625d thru 0x03c2bd (0x6061 bytes) 000932 1787119294.561219 mapread 0x002a77 thru 0x004183 (0x170d bytes) 000933 1787119294.570348 fallocate 0x03c953 thru 0x03ffff (0x36ad bytes) 000934 1787119294.575139 write 0x00ea27 thru 0x0192f3 (0xa8cd bytes) 000935 1787119294.584596 punch from 0x026d14 to 0x032fff (0xc2ec bytes) 000936 1787119294.590577 read 0x0338d3 thru 0x03ffff (0xc72d bytes) 000937 1787119294.600533 fallocate 0x002cb8 thru 0x009747 (0x6a90 bytes) 000938 1787119294.607528 mapread 0x019cbf thru 0x01b3bd (0x16ff bytes) 000939 1787119294.615975 trunc from 0x0343c0 to 0x03660d (0x224e bytes) 000940 1787119294.620314 mapwrite 0x01c667 thru 0x028b98 (0xc532 bytes) 000942 1787119294.654141 MIRROR_RESYNC 11 mirrors 000943 1787119297.340189 punch from 0x00e02b to 0x01b5a0 (0xd576 bytes) 000944 1787119297.395589 fallocate 0x0359d1 thru 0x03a457 (0x4a87 bytes) 000945 1787119297.412008 trunc from 0x00e54c to 0x01ce2a (0xe8df bytes) 000946 1787119297.422276 mapwrite 0x00b266 thru 0x0123a3 (0x713e bytes) 000947 1787119297.462534 read 0x01a888 thru 0x029738 (0xeeb1 bytes) 000948 1787119297.475830 mapread 0x0382eb thru 0x03ffff (0x7d15 bytes) 000949 1787119297.488173 mapwrite 0x014c69 thru 0x01a720 (0x5ab8 bytes) 000950 1787119297.513412 trunc from 0x026f90 to 0x03304c (0xc0bd bytes) 000951 1787119297.523695 MIRROR_RESYNC 11 mirrors 000952 1787119300.009075 write 0x03cb35 thru 0x03ffff (0x34cb bytes) 000953 1787119300.054530 punch from 0x0330bd to 0x03e1d2 (0xb116 bytes) 000954 1787119300.067447 write 0x0394e1 thru 0x03ab76 (0x1696 bytes) 000955 1787119300.080365 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x6c268d11 (stale) Mirror 22: 0x6c268d11 (stale) Mirror 25: 0x6c268d11 (stale) Mirror 26: 0x6c268d11 (stale) Mirror 27: 0x6c268d11 (stale) Mirror 28: 0x6c268d11 (stale) Mirror 30: 0x6c268d11 (stale) Mirror 32: 0x6c268d11 (stale) Mirror 33: 0x6c268d11 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x6c268d11 (stale) Mirror 22: 0x6c268d11 (stale) Mirror 25: 0x6c268d11 (stale) Mirror 26: 0x6c268d11 (stale) Mirror 27: 0x6c268d11 (stale) Mirror 28: 0x6c268d11 (stale) Mirror 30: 0x6c268d11 (stale) Mirror 32: 0x6c268d11 (stale) Mirror 33: 0x6c268d11 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000956 1787119301.081972 mapwrite 0x021bdb thru 0x02c5c9 (0xa9ef bytes) 000957 1787119301.192026 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: (stale - read error) Mirror 22: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0x6c268d11 (stale) Mirror 22: 0x6c268d11 (stale) Mirror 25: 0x6c268d11 (stale) Mirror 26: 0x6c268d11 (stale) Mirror 27: 0x6c268d11 (stale) Mirror 28: 0x6c268d11 (stale) Mirror 30: 0x6c268d11 (stale) Mirror 32: 0x6c268d11 (stale) Mirror 33: 0x6c268d11 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb405c567 Mirror 34: 0xb405c567 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000958 1787119302.588327 punch from 0x006bae to 0x012215 (0xb668 bytes) 000959 1787119302.674370 trunc from 0x040000 to 0x01fcc9 (0x20336 bytes) 000960 1787119302.745084 trunc from 0x0046b0 to 0x00d8eb (0x923c bytes) 000961 1787119302.752861 mapwrite 0x030bb6 thru 0x03319d (0x25e8 bytes) 000962 1787119302.809086 trunc from 0x03319e to 0x00c0fc (0x270a1 bytes) 000963 1787119302.828556 mapread 0x00ad86 thru 0x00adca (0x45 bytes) 000964 1787119302.843174 mapwrite 0x03e23c thru 0x03ffff (0x1dc4 bytes) 000965 1787119302.895766 MIRROR_RESYNC 11 mirrors 000967 1787119304.447613 mapwrite 0x01ada9 thru 0x01c777 (0x19cf bytes) 000968 1787119304.528208 mapwrite 0x0031fd thru 0x00ee9b (0xbc9f bytes) 000969 1787119304.558801 fallocate 0x010a08 thru 0x018688 (0x7c81 bytes) 000970 1787119304.565809 write 0x0338dc thru 0x03e8f7 (0xb01c bytes) 000971 1787119304.575646 trunc from 0x040000 to 0x03e286 (0x1d79 bytes) 000972 1787119304.601167 MIRROR_SPLIT mirror 2 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e287): Mirror 18: 0xdd2f70ed (stale) Mirror 25: 0xdd2f70ed (stale) Mirror 26: 0xdd2f70ed (stale) Mirror 27: 0xdd2f70ed (stale) Mirror 28: 0xdd2f70ed (stale) Mirror 30: 0xdd2f70ed (stale) Mirror 32: 0xdd2f70ed (stale) Mirror 33: 0xdd2f70ed (stale) Mirror 34: 0xdd2f70ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e287): Mirror 18: 0xdd2f70ed (stale) Mirror 25: 0xdd2f70ed (stale) Mirror 26: 0xdd2f70ed (stale) Mirror 27: 0xdd2f70ed (stale) Mirror 28: 0xdd2f70ed (stale) Mirror 30: 0xdd2f70ed (stale) Mirror 32: 0xdd2f70ed (stale) Mirror 33: 0xdd2f70ed (stale) Mirror 34: 0xdd2f70ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e287: skipped 000973 1787119305.270299 mapwrite 0x0139c1 thru 0x017e9d (0x44dd bytes) 000974 1787119305.314575 read 0x020cf5 thru 0x02e207 (0xd513 bytes) 000975 1787119305.323765 fallocate 0x016cda thru 0x024e5d (0xe184 bytes) 000976 1787119305.331428 trunc from 0x028ebf to 0x03792d (0xea6f bytes) 000977 1787119305.337963 mapwrite 0x0212cd thru 0x02fe81 (0xebb5 bytes) 000978 1787119305.400418 MIRROR_SPLIT mirror 7 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e287): Mirror 18: 0xdd2f70ed (stale) Mirror 25: 0xdd2f70ed (stale) Mirror 26: 0xdd2f70ed (stale) Mirror 27: 0xdd2f70ed (stale) Mirror 28: 0xdd2f70ed (stale) Mirror 30: 0xdd2f70ed (stale) Mirror 33: 0xdd2f70ed (stale) Mirror 34: 0xdd2f70ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e287): Mirror 18: 0xdd2f70ed (stale) Mirror 25: 0xdd2f70ed (stale) Mirror 26: 0xdd2f70ed (stale) Mirror 27: 0xdd2f70ed (stale) Mirror 28: 0xdd2f70ed (stale) Mirror 30: 0xdd2f70ed (stale) Mirror 33: 0xdd2f70ed (stale) Mirror 34: 0xdd2f70ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e287: skipped 000979 1787119306.073531 write 0x037bc7 thru 0x03e281 (0x66bb bytes) 000980 1787119306.097335 trunc from 0x01ae4c to 0x0237a8 (0x895d bytes) 000981 1787119306.115045 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e287): Mirror 18: 0xdd2f70ed (stale) Mirror 26: 0xdd2f70ed (stale) Mirror 27: 0xdd2f70ed (stale) Mirror 28: 0xdd2f70ed (stale) Mirror 30: 0xdd2f70ed (stale) Mirror 33: 0xdd2f70ed (stale) Mirror 34: 0xdd2f70ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e287): Mirror 18: 0xdd2f70ed (stale) Mirror 26: 0xdd2f70ed (stale) Mirror 27: 0xdd2f70ed (stale) Mirror 28: 0xdd2f70ed (stale) Mirror 30: 0xdd2f70ed (stale) Mirror 33: 0xdd2f70ed (stale) Mirror 34: 0xdd2f70ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e287: skipped 000982 1787119306.632725 fallocate 0x026238 thru 0x02f6db (0x94a4 bytes) 000983 1787119306.638078 trunc from 0x03e287 to 0x013d84 (0x2a502 bytes) 000985 1787119306.664863 punch from 0x00a161 to 0x0128b4 (0x8754 bytes) 000986 1787119306.670571 fallocate 0x03b07f thru 0x03e00a (0x2f8c bytes) 000987 1787119306.675284 fallocate 0x006693 thru 0x00e6d7 (0x8045 bytes) 000988 1787119306.680338 read 0x011a8b thru 0x01992b (0x7ea1 bytes) 000989 1787119306.692358 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3e00b): Mirror 18: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e00b): Mirror 18: 0x56dffbf5 (stale) Mirror 26: 0x56dffbf5 (stale) Mirror 27: 0x56dffbf5 (stale) Mirror 28: 0x56dffbf5 (stale) Mirror 30: 0x56dffbf5 (stale) Mirror 33: 0x56dffbf5 (stale) Mirror 34: 0x56dffbf5 (stale) CRC-32 checksum value for chunk [0, 0x3e00b): Mirror 1: 0x82f90125 Mirror 35: 0x82f90125 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e00b: skipped 000990 1787119307.503115 mapread 0x0031cd thru 0x0073ca (0x41fe bytes) 000991 1787119307.555407 trunc from 0x03e00b to 0x012048 (0x2bfc2 bytes) 000992 1787119307.632258 mapwrite 0x00a250 thru 0x014ce6 (0xaa97 bytes) 000993 1787119307.736180 trunc from 0x00e9f9 to 0x00fc0e (0x1216 bytes) 000995 1787119307.754432 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x14ce7): Mirror 18: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 30: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14ce7): Mirror 18: 0xb194c1ff (stale) Mirror 26: 0xb194c1ff (stale) Mirror 27: 0xb194c1ff (stale) Mirror 28: 0xb194c1ff (stale) Mirror 30: 0xb194c1ff (stale) Mirror 33: 0xb194c1ff (stale) Mirror 34: 0xb194c1ff (stale) Mirror 35: 0x3a963963 (stale) CRC-32 checksum value for chunk [0, 0x14ce7): Mirror 1: 0x671ce02a Mirror 36: 0x671ce02a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14ce7: skipped 000996 1787119308.519506 mapwrite 0x030d8e thru 0x033850 (0x2ac3 bytes) 000997 1787119308.613144 fallocate 0x02f1bc thru 0x037af9 (0x893e bytes) 000998 1787119308.620314 trunc from 0x001669 to 0x005239 (0x3bd1 bytes) 000999 1787119308.635044 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d6:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33851): Mirror 18: 0xd4b13836 (stale) Mirror 26: 0xd4b13836 (stale) Mirror 28: 0xd4b13836 (stale) Mirror 30: 0xd4b13836 (stale) Mirror 33: 0xd4b13836 (stale) Mirror 34: 0xd4b13836 (stale) Mirror 35: 0x3fff67a4 (stale) Mirror 36: 0x354c7229 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33851): Mirror 18: 0xd4b13836 (stale) Mirror 26: 0xd4b13836 (stale) Mirror 28: 0xd4b13836 (stale) Mirror 30: 0xd4b13836 (stale) Mirror 33: 0xd4b13836 (stale) Mirror 34: 0xd4b13836 (stale) Mirror 35: 0x3fff67a4 (stale) Mirror 36: 0x354c7229 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33851: skipped 001000 1787119309.155852 read 0x00ca02 thru 0x016a4b (0xa04a bytes) All operations completed A-OK! PASS 70a (165s) == sanity-flr test 71: check mirror extend parent fid ==== 02:01:54 (1787119314) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x4d:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 02:02:03 (1787119323) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x37f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x361:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.1414 s, 14.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x37f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x380:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x361:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131129 s, 8.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x37f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x380:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x362:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x361:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x37f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x380:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x362:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x361:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x381:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 02:02:11 (1787119331) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1345984 .. Extending file size to 6485056 .. done Extending file size to 7372992 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8143808 .. Extending file size to 8437056 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8798848 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8774688 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9030560 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 61632 61633 61634 61637 61639 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) 192.168.202.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (78s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:03:29 (1787119409) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7528288 .. Extending file size to 1161632 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7572864 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8389856 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8437024 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8513056 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9228064 .. Extending file size to 8658144 .. Extending file size to 9229568 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 63182 63183 63186 63187 63188 done 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) 192.168.202.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:04:48 (1787119488) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2880 1268424 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1868 1269436 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24516 3582504 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18832 3588188 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43348 7170692 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2880 1268424 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1868 1269436 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24516 3582504 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18832 3588188 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43348 7170692 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0990745 s, 21.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.199606 s, 10.5 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x75e36766 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) PASS 200c (16s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:05:06 (1787119506) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0780197 s, 26.9 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 02:05:13 (1787119513) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x226:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x378:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x378:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x379:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x396:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:05:21 (1787119521) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.101237 s, 20.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x39:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:05:31 (1787119531) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182979 s, 11.5 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3c:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:05:40 (1787119540) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145691 s, 14.4 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3e:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:06:12 (1787119572) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg222-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116717 s, 9.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x40:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x40:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x3f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x3f:0x0] } PASS 204d (48s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:07:00 (1787119620) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg222-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x397:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x398:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.186845 s, 11.2 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x397:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x398:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x37a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x397:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x398:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x37a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (49s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:07:49 (1787119669) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg222-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg222-server: Pool lustre.test_204f created Adding targets to pool oleg222-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg222-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.117276 s, 17.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x399:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x37b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x399:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x37b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg222-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg222-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg222-server: Pool lustre.test_204f destroyed PASS 204f (77s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:09:06 (1787119746) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:09:13 (1787119753) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:09:20 (1787119760) oleg222-server: Pool lustre.test_206 created Adding targets to pool oleg222-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg222-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg222-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg222-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg222-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:09:46 (1787119786) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867084 s, 12.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607249 s, 17.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xdde9f44c Mirror 3: 0xdde9f44c PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:09:58 (1787119798) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:10:01 (1787119801) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:10:06 (1787119806) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0533981 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1560000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1410000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1410000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1410000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x23b0000 Read this (4 bytes): BBBB 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:10:45 (1787119845) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0387822 s, 211 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:10:52 (1787119852) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0667768 s, 15.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x243 lmm_fid: [0x200000405:0x243:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x243 lmm_fid: [0x200000405:0x243:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x386:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611715 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-19 02:10:54.000000000 -0400 Modify: 2026-08-19 02:10:54.000000000 -0400 Change: 2026-08-19 02:10:56.000000000 -0400 Birth: 2026-08-19 02:10:53.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:11:01 (1787119861) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267148 s, 39.3 MB/s oleg222-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg222-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg222-server: O/280000401/d3/3a3: File not found by ext2_lookup PASS 210b (20s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:11:21 (1787119881) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.19114 s, 214 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0195903 s, 209 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 19 02:11 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x248:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 19 02:11 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:11:28 (1787119888) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:11:36 (1787119896) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1839 sec =========== 02:11:47 (1787119907) === sanity-flr: start cleanup 02:11:48 (1787119908) === === sanity-flr: finish cleanup 02:11:55 (1787119915) ===