-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 15:47:09 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg124-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 15:47:30 (1787860050) === oleg124-client.virtnet: executing check_config_client /mnt/lustre oleg124-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg124-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9b11126de000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9b11126de000.idle_timeout=debug disable quota as required oleg124-server: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 15:48:10 (1787860090) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8237] == sanity-flr test 0a: lfs mirror create with -N option == 15:48:13 (1787860093) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:48:27 (1787860107) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:48:31 (1787860111) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:48:35 (1787860115) PASS 0d (23s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:48:58 (1787860138) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:49:02 (1787860142) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 15:49:06 (1787860146) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9b11126de000.stats=clear osc.lustre-OST0001-osc-ffff9b11126de000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:49:15 (1787860155) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:49:28 (1787860168) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:49:36 (1787860176) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 15:49:40 (1787860180) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:49:48 (1787860188) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 15:49:57 (1787860197) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:50:05 (1787860205) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:50:12 (1787860212) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:50:21 (1787860221) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:50:30 (1787860230) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:50:37 (1787860237) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 1.51214 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155408 s, 6.7 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:50:47 (1787860247) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.200866 s, 41.8 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 15:50:58 (1787860258) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:51:11 (1787860271) Stopping /mnt/lustre-ost1 (opts:) on oleg124-server oleg124-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b11126de000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b11126de000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg124-server: oleg124-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg124-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg124-server oleg124-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b11126de000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b11126de000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg124-server: oleg124-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg124-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (76s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:52:27 (1787860347) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.141638 s, 14.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg124-server oleg124-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b11126de000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b11126de000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg124-server: oleg124-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg124-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (54s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:53:21 (1787860401) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg124-server oleg124-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b11126de000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b11126de000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg124-server: oleg124-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg124-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg124-server oleg124-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b11126de000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b11126de000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg124-server: oleg124-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg124-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (88s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:54:49 (1787860489) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.12775 s, 16.8 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 15:54 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.718s user 0m0.001s sys 0m0.147s Stopping /mnt/lustre-ost1 (opts:) on oleg124-server oleg124-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9b11126de000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9b11126de000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.171s user 0m0.001s sys 0m0.185s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg124-server: oleg124-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg124-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg124-server oleg124-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9b11126de000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b11126de000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.237s user 0m0.000s sys 0m0.181s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg124-server: oleg124-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg124-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg124-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b11126de000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9b11126de000.ost_server_uuid in FULL state after 0 sec PASS 33b (104s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:56:34 (1787860594) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:56:37 (1787860597) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:56:41 (1787860601) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (5s) == sanity-flr test 35: allow to write to mirrored files == 15:56:46 (1787860606) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 15:56:54 (1787860614) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.668363 s, 18.8 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:57:07 (1787860627) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0416203 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0434635 s, 24.1 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:57:14 (1787860634) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:57:23 (1787860643) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00179753 s, 570 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000956236 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00203574 s, 503 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00105547 s, 970 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 15:57:32 (1787860652) create test files with size 15734350 .. 8f7731095de090639eeb690772ddaf2a - 8629b2ce7b3a10baf2983b4e9fbecdf6 - 544e0df23c6adc2ad009336e4eab8acb - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg124-client.virtnet /mnt/lustre (opts:) Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 15:57:58 (1787860678) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:58:11 (1787860691) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:58:18 (1787860698) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0770544 s, 13.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc4:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc5:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb3:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220287 s, 47.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb4:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb5:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xc6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 15:58:26 (1787860706) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0577665 s, 36.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.403346 s, 10.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.139938 s, 15.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 15:58:40 (1787860720) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 15:58:44 (1787860724) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:58:48 (1787860728) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037326 s, 28.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 15:58:56 (1787860736) llite.lustre-ffff9b1111be1800.enable_compression=1 striped dir -i1 -c2 -H crush2 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (9s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 15:59:05 (1787860745) llite.lustre-ffff9b1111be1800.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824117 s, 12.7 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0399237 s, 26.3 MB/s **verify components PASS 43d (9s) == sanity-flr test 44a: lfs mirror split check =========== 15:59:14 (1787860754) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.258718 s, 12.2 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:59:23 (1787860763) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc2:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc2:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:59:34 (1787860774) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.29176 s, 35.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x9:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg124-client.virtnet /mnt/lustre (opts:) Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:59:46 (1787860786) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9b11126d9800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.302737 s, 34.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg124-client.virtnet /mnt/lustre (opts:) Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9b11128eb800.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:00:04 (1787860804) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (45s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:00:49 (1787860849) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (37s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:01:26 (1787860886) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 16:01:30 (1787860890) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.295749 s, 10.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.255332 s, 12.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x6fbde840 (nosync,stale) Mirror 2: 0x6fbde840 (nosync,stale) ** verify mirror 2 doesn't change original checksum: bda0a3225e83124c3626b178c6da7d57 - mirror 1 checksum: 7bf774563f028f8df8971aa17f74a752 - mirror 2 checksum: bda0a3225e83124c3626b178c6da7d57 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T16:01:34-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T16:01:34-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xdc:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: bda0a3225e83124c3626b178c6da7d57 - mirror 1 checksum: 7bf774563f028f8df8971aa17f74a752 - mirror 2 checksum: 7bf774563f028f8df8971aa17f74a752 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T16:01:34-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xdb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T16:01:34-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xdc:0x0] } PASS 48 (16s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 16:01:45 (1787860905) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495059 s, 21.2 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (10s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:01:55 (1787860915) Failing mds1 on oleg124-server Stopping /mnt/lustre-mds1 (opts:) on oleg124-server 16:02:07 (1787860927) shut down facet: mds1 facet_host: oleg124-server facet_failover_host: oleg124-server Failover mds1 to oleg124-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg124-server: oleg124-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-MDT0000 16:02:32 (1787860952) targets are mounted 16:02:32 (1787860952) facet_failover done oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538482 s, 19.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xde:0x0] } PASS 50A (53s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:02:48 (1787860968) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0590042 s, 347 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0189832 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:02:59 (1787860979) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.066334 s, 309 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00640604 s, 3.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:03:09 (1787860989) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0126408 s, 1.3 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:03:16 (1787860996) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0781444 s, 262 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:03:24 (1787861004) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00656173 s, 624 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0226389 s, 181 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:03:31 (1787861011) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (18s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:03:49 (1787861029) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:04:16 (1787861056) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:04:38 (1787861078) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:05:02 (1787861102) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 16:05:10 (1787861110) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 58940 mirror split pid 58941 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58940 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58941 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 16:06:17 (1787861177) striped dir -i0 -c2 -H crush /mnt/lustre/d70a.sanity-flr Seed set to 1179 truncating to largest ever: 0x12d15 000002 1787861179.536899 trunc from 00000000 to 0x012d14 (0x12d15 bytes) 000003 1787861179.556401 read 0x00ff7c thru 0x0100d1 (0x156 bytes) 000005 1787861179.562722 mapread 0x004f2a thru 0x006b31 (0x1c08 bytes) 000006 1787861179.582485 trunc from 0x00dbda to 0x012d14 (0x513b bytes) 000007 1787861179.589373 read 0x00dc52 thru 0x012d14 (0x50c3 bytes) 000009 1787861179.599288 write 0x02df4b thru 0x0347e5 (0x689b bytes) 000010 1787861179.601252 punch from 0x01286b to 0x01abd9 (0x836f bytes) 000011 1787861179.605655 mapread 0x0085b4 thru 0x00b589 (0x2fd6 bytes) 000012 1787861179.606543 mapwrite 0x02536f thru 0x029aed (0x477f bytes) 000013 1787861179.645890 punch from 0x0167a1 to 0x01de20 (0x7680 bytes) 000014 1787861179.650366 mapread 0x01a190 thru 0x01cdf4 (0x2c65 bytes) 000015 1787861179.663974 fallocate 0x00094b thru 0x010841 (0xfef7 bytes) zero_range to largest ever: 0x347e6 000016 1787861179.669603 trunc from 0x02555c to 0x0347e5 (0xf28a bytes) 000017 1787861179.674937 read 0x025702 thru 0x02e64f (0x8f4e bytes) 000018 1787861179.684480 mapwrite 0x030cef thru 0x0346d2 (0x39e4 bytes) 000020 1787861179.698876 write 0x015b50 thru 0x0229f0 (0xcea1 bytes) 000021 1787861179.701340 mapwrite 0x033649 thru 0x0369b2 (0x336a bytes) 000022 1787861179.737821 punch from 0x0110e2 to 0x011d79 (0xc98 bytes) 000023 1787861179.743517 trunc from 0x02b022 to 0x03066e (0x564d bytes) 000024 1787861179.753160 MIRROR_EXTEND to 2 mirrors 000025 1787861180.647713 mapwrite 0x0222ad thru 0x023809 (0x155d bytes) 000027 1787861180.762257 fallocate 0x02777d thru 0x0324b3 (0xad37 bytes) 000028 1787861180.772244 trunc from 0x020fa6 to 0x02e79c (0xd7f7 bytes) 000029 1787861180.785522 write 0x03162b thru 0x0372c3 (0x5c99 bytes) 000030 1787861180.816949 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) 000032 1787861181.056560 trunc from 0x0372c4 to 0x0156cc (0x21bf7 bytes) 000033 1787861181.079288 mapread 0x00f769 thru 0x014dc8 (0x5660 bytes) 000034 1787861181.102648 write 0x00c19b thru 0x013863 (0x76c9 bytes) 000035 1787861181.104379 trunc from 0x0156cd to 0x0337d1 (0x1e105 bytes) 000038 1787861181.121132 mapread 0x01cb00 thru 0x0291cc (0xc6cd bytes) 000039 1787861181.141549 mapwrite 0x02d73b thru 0x033b1a (0x63e0 bytes) 000042 1787861181.186649 trunc from 0x033b1b to 0x00aa51 (0x290c9 bytes) 000043 1787861181.203853 fallocate 0x013b74 thru 0x016e5f (0x32ec bytes) 000044 1787861181.208799 write 0x01de28 thru 0x020505 (0x26de bytes) 000045 1787861181.218287 MIRROR_EXTEND to 2 mirrors 000046 1787861181.969261 trunc from 0x017afa to 0x018766 (0xc6d bytes) 000047 1787861182.019287 mapwrite 0x014c68 thru 0x017bf3 (0x2f8c bytes) 000048 1787861182.051634 trunc from 0x020219 to 0x020505 (0x2ed bytes) 000049 1787861182.060417 mapread 0x016f40 thru 0x0202eb (0x93ac bytes) 000050 1787861182.074269 read 0x00abba thru 0x00edd8 (0x421f bytes) 000051 1787861182.093752 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20506): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20506): Mirror 2: 0xc2ffbfcc (stale) CRC-32 checksum value for chunk [0, 0x20506): Mirror 1: 0xbe0e7b3b Mirror 3: 0xbe0e7b3b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20506: skipped 000052 1787861183.384813 read 0x010d37 thru 0x017f49 (0x7213 bytes) 000053 1787861183.403370 punch from 0x003c85 to 0x0093af (0x572b bytes) 000054 1787861183.501489 trunc from 0x019c85 to 0x020505 (0x6881 bytes) 000055 1787861183.514449 mapread 0x005663 thru 0x010168 (0xab06 bytes) fallocating to largest ever: 0x40000 000056 1787861183.577383 fallocate 0x038b17 thru 0x03ffff (0x74e9 bytes) 000057 1787861183.588052 mapwrite 0x03e110 thru 0x03ffff (0x1ef0 bytes) 000058 1787861183.731239 write 0x01fdc4 thru 0x024068 (0x42a5 bytes) 000059 1787861183.745383 write 0x0246dd thru 0x030bf1 (0xc515 bytes) 000060 1787861183.763348 write 0x025819 thru 0x02706c (0x1854 bytes) 000061 1787861183.767400 fallocate 0x006641 thru 0x00997a (0x333a bytes) 000062 1787861183.786756 read 0x0361ab thru 0x03b9fc (0x5852 bytes) 000063 1787861183.810540 trunc from 0x040000 to 0x0226dc (0x1d923 bytes) 000064 1787861183.852565 mapread 0x01f9b1 thru 0x0226dc (0x2d2c bytes) 000065 1787861183.853531 read 0x017326 thru 0x01a258 (0x2f33 bytes) 000066 1787861183.860580 read 0x0175ee thru 0x01b6d1 (0x40e4 bytes) 000068 1787861183.863282 trunc from 0x01f67c to 0x0226dc (0x3061 bytes) 000069 1787861183.900737 mapwrite 0x004320 thru 0x00a542 (0x6223 bytes) 000070 1787861183.926317 write 0x0075fa thru 0x007a80 (0x487 bytes) 000071 1787861183.928418 trunc from 0x0226dd to 0x0194bb (0x9221 bytes) 000072 1787861183.953088 trunc from 0x0194bc to 0x0234cb (0xa010 bytes) 000073 1787861184.012475 fallocate 0x03add5 thru 0x03ffff (0x522b bytes) 000074 1787861184.027614 punch from 0x01f852 to 0x021880 (0x202f bytes) 000075 1787861184.057203 fallocate 0x01f4e9 thru 0x02a811 (0xb329 bytes) 000076 1787861184.087526 MIRROR_RESYNC 3 mirrors 000077 1787861185.041177 punch from 0x03310b to 0x033d56 (0xc4c bytes) 000078 1787861185.071497 read 0x03d70d thru 0x03da9d (0x391 bytes) 000079 1787861185.077499 mapwrite 0x00415c thru 0x00e789 (0xa62e bytes) 000080 1787861185.108826 write 0x00e124 thru 0x018f08 (0xade5 bytes) 000081 1787861185.114575 write 0x03d62d thru 0x03ffff (0x29d3 bytes) 000082 1787861185.115651 mapread 0x03666f thru 0x0381c1 (0x1b53 bytes) 000083 1787861185.128423 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x532407c0 (stale) Mirror 3: 0x532407c0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x37701b0f Mirror 4: 0x37701b0f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000084 1787861185.951948 mapwrite 0x0028b4 thru 0x00ec19 (0xc366 bytes) 000085 1787861186.097702 read 0x00734a thru 0x00f95e (0x8615 bytes) 000086 1787861186.099406 read 0x038cca thru 0x03ffff (0x7336 bytes) 000087 1787861186.121545 mapwrite 0x02b9a2 thru 0x03969c (0xdcfb bytes) 000088 1787861186.158273 read 0x02fde8 thru 0x03590b (0x5b24 bytes) 000089 1787861186.163732 MIRROR_RESYNC 4 mirrors 000090 1787861187.664785 mapread 0x03788e thru 0x03ffff (0x8772 bytes) 000091 1787861187.691384 mapwrite 0x03e3ba thru 0x03ffff (0x1c46 bytes) 000092 1787861187.763356 trunc from 0x040000 to 0x036f10 (0x90ef bytes) 000093 1787861187.802291 fallocate 0x009fa3 thru 0x00ebd2 (0x4c30 bytes) 000094 1787861187.818707 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36f11): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36f11): Mirror 2: 0x8ade12e2 (stale) Mirror 3: 0x8ade12e2 (stale) Mirror 4: 0x8ade12e2 (stale) CRC-32 checksum value for chunk [0, 0x36f11): Mirror 1: 0x8ade12e2 Mirror 5: 0x8ade12e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36f11: skipped 000095 1787861188.723368 mapread 0x017c97 thru 0x022984 (0xacee bytes) 000096 1787861188.765057 read 0x01f746 thru 0x02bc52 (0xc50d bytes) 000097 1787861188.778127 write 0x024c31 thru 0x0257a0 (0xb70 bytes) 000098 1787861188.834246 fallocate 0x00d7d2 thru 0x01ca74 (0xf2a3 bytes) 000099 1787861188.841429 read 0x003cc9 thru 0x010713 (0xca4b bytes) 000100 1787861188.878308 trunc from 0x036f11 to 0x00050d (0x36a03 bytes) 000101 1787861188.907005 fallocate 0x00f8c9 thru 0x0111b1 (0x18e9 bytes) 000103 1787861188.914757 trunc from 0x01006d to 0x0111b1 (0x1145 bytes) 000104 1787861188.926590 mapread 0x0076c9 thru 0x010dc1 (0x96f9 bytes) 000105 1787861189.030679 trunc from 0x0111b2 to 0x016568 (0x53b7 bytes) 000106 1787861189.074008 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16569): Mirror 2: 0x40a93a09 (stale) Mirror 4: 0x40a93a09 (stale) Mirror 5: 0x40a93a09 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16569): Mirror 2: 0x40a93a09 (stale) Mirror 4: 0x40a93a09 (stale) Mirror 5: 0x40a93a09 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16569: skipped 000107 1787861189.669383 write 0x00aa8f thru 0x0174fc (0xca6e bytes) 000108 1787861189.684619 mapread 0x0019f3 thru 0x0087f9 (0x6e07 bytes) 000109 1787861189.702849 punch from 0x0058d5 to 0x00c35a (0x6a86 bytes) 000110 1787861189.721526 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x174fd): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x174fd): Mirror 2: 0x4939573d (stale) Mirror 4: 0x4939573d (stale) Mirror 5: 0x4939573d (stale) CRC-32 checksum value for chunk [0, 0x174fd): Mirror 1: 0x4a9d80b5 Mirror 6: 0x4a9d80b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x174fd: skipped 000111 1787861190.598282 mapread 0x008acc thru 0x00d795 (0x4cca bytes) 000112 1787861190.613259 trunc from 0x014cbc to 0x0174fc (0x2841 bytes) 000113 1787861190.656645 trunc from 0x00d415 to 0x011dbb (0x49a7 bytes) 000114 1787861190.665610 read 0x00cb75 thru 0x0174fc (0xa988 bytes) 000115 1787861190.677172 punch from 0x012e1f to 0x0174fc (0x46de bytes) 000116 1787861190.682890 write 0x00bfa0 thru 0x0125ee (0x664f bytes) 000117 1787861190.701437 fallocate 0x033079 thru 0x03efad (0xbf35 bytes) 000118 1787861190.711883 MIRROR_RESYNC 5 mirrors 000119 1787861192.141389 fallocate 0x000f3a thru 0x009055 (0x811c bytes) 000120 1787861192.215036 punch from 0x0002d9 to 0x00b304 (0xb02c bytes) 000121 1787861192.237440 mapwrite 0x03d931 thru 0x03ffff (0x26cf bytes) 000122 1787861192.339996 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1ec33247 (stale) Mirror 5: 0x1ec33247 (stale) Mirror 6: 0x1ec33247 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1ec33247 (stale) Mirror 5: 0x1ec33247 (stale) Mirror 6: 0x1ec33247 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000123 1787861192.868127 trunc from 0x040000 to 0x029aa5 (0x1655a bytes) 000124 1787861192.888819 mapwrite 0x031c79 thru 0x03facc (0xde54 bytes) 000125 1787861192.941338 trunc from 0x00e605 to 0x010e8d (0x2889 bytes) 000126 1787861192.946176 mapread 0x0385b6 thru 0x03facc (0x7517 bytes) 000127 1787861192.947174 trunc from 0x02e3d0 to 0x03af58 (0xcb89 bytes) 000128 1787861192.953124 trunc from 0x037062 to 0x03facc (0x8a6b bytes) 000129 1787861192.957929 write 0x01359b thru 0x0198a2 (0x6308 bytes) 000130 1787861192.966309 mapread 0x02426c thru 0x026021 (0x1db6 bytes) 000131 1787861192.979167 mapwrite 0x002891 thru 0x004aa6 (0x2216 bytes) 000133 1787861193.008678 mapread 0x01fe7e thru 0x0277bc (0x793f bytes) 000134 1787861193.014121 MIRROR_RESYNC 4 mirrors 000135 1787861194.124103 trunc from 0x03facd to 0x005cb6 (0x39e16 bytes) 000136 1787861194.164834 mapread 0x003779 thru 0x005cb6 (0x253e bytes) 000137 1787861194.183463 mapread 0x000780 thru 0x002afc (0x237d bytes) 000138 1787861194.190892 read 0x00398d thru 0x005cb6 (0x232a bytes) 000140 1787861194.192991 fallocate 0x0238fe thru 0x02fa50 (0xc153 bytes) 000141 1787861194.198002 mapread 0x00f34e thru 0x01a8ff (0xb5b2 bytes) 000142 1787861194.218328 mapwrite 0x0201ec thru 0x028c89 (0x8a9e bytes) 000143 1787861194.246704 write 0x02148c thru 0x023281 (0x1df6 bytes) 000144 1787861194.248491 mapread 0x00a13b thru 0x019092 (0xef58 bytes) 000145 1787861194.250246 mapwrite 0x00dc2b thru 0x00f356 (0x172c bytes) 000146 1787861194.275358 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2fa51): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fa51): Mirror 2: 0x4c5bc3b0 (stale) Mirror 5: 0x4c5bc3b0 (stale) Mirror 6: 0x4c5bc3b0 (stale) CRC-32 checksum value for chunk [0, 0x2fa51): Mirror 1: 0x455dffbf Mirror 7: 0x455dffbf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fa51: skipped 000148 1787861195.005200 trunc from 0x02fa51 to 0x005e69 (0x29be7 bytes) 000149 1787861195.076686 trunc from 0x00565f to 0x005e69 (0x80b bytes) 000150 1787861195.089147 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x5e6a): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5e6a): Mirror 2: 0xd46da13b (stale) Mirror 5: 0xd46da13b (stale) Mirror 6: 0xd46da13b (stale) Mirror 7: 0xd46da13b (stale) CRC-32 checksum value for chunk [0, 0x5e6a): Mirror 1: 0xd46da13b Mirror 8: 0xd46da13b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5e6a: skipped 000151 1787861195.772928 trunc from 0x005e6a to 0x000241 (0x5c28 bytes) 000153 1787861195.823285 read 0x0000ff thru 0x000241 (0x143 bytes) 000154 1787861195.832701 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x242): Mirror 5: 0x6d247b3b (stale) Mirror 6: 0x6d247b3b (stale) Mirror 7: 0x6d247b3b (stale) Mirror 8: 0x6d247b3b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x242): Mirror 5: 0x6d247b3b (stale) Mirror 6: 0x6d247b3b (stale) Mirror 7: 0x6d247b3b (stale) Mirror 8: 0x6d247b3b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x242: skipped 000155 1787861196.119186 punch from 0x00001e to 0x000241 (0x224 bytes) 000157 1787861196.125566 read 0x0000da thru 0x000241 (0x168 bytes) 000158 1787861196.130397 trunc from 0x00014e to 0x000241 (0xf4 bytes) 000159 1787861196.136942 punch from 0x00006f to 0x000241 (0x1d3 bytes) 000162 1787861196.146042 trunc from 0x000039 to 0x000241 (0x209 bytes) 000164 1787861196.150646 fallocate 0x03f37c thru 0x03ffff (0xc84 bytes) 000165 1787861196.155452 read 0x01df4d thru 0x02871b (0xa7cf bytes) 000166 1787861196.164918 trunc from 0x00ca95 to 0x015771 (0x8cdd bytes) 000167 1787861196.169110 mapwrite 0x02766e thru 0x0303cd (0x8d60 bytes) 000169 1787861196.194463 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xfae16209 (stale) Mirror 6: 0xfae16209 (stale) Mirror 7: 0xbf4683cb (stale) Mirror 8: 0xd645476f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x264498cd Mirror 9: 0x264498cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000170 1787861196.977599 fallocate 0x0023ca thru 0x005f90 (0x3bc7 bytes) 000172 1787861197.040661 trunc from 0x040000 to 0x006f59 (0x390a6 bytes) 000175 1787861197.108363 trunc from 0x002a07 to 0x006f59 (0x4553 bytes) 000176 1787861197.117389 trunc from 0x00415f to 0x005ff0 (0x1e92 bytes) 000177 1787861197.121873 read 0x006aed thru 0x006f59 (0x46d bytes) 000178 1787861197.130290 MIRROR_RESYNC 6 mirrors 000179 1787861197.973983 read 0x006686 thru 0x006f59 (0x8d4 bytes) 000180 1787861197.987821 write 0x0038ac thru 0x00395b (0xb0 bytes) 000181 1787861198.071059 read 0x002f89 thru 0x006f59 (0x3fd1 bytes) 000182 1787861198.086307 mapread 0x0030d1 thru 0x006f59 (0x3e89 bytes) 000183 1787861198.087269 mapwrite 0x00a797 thru 0x00ee7f (0x46e9 bytes) 000184 1787861198.187296 punch from 0x00d68e to 0x00ee7f (0x17f2 bytes) 000185 1787861198.192931 punch from 0x008436 to 0x00ee7f (0x6a4a bytes) 000186 1787861198.198469 trunc from 0x007799 to 0x00ee7f (0x76e7 bytes) 000187 1787861198.203362 mapwrite 0x027fd0 thru 0x02e48e (0x64bf bytes) 000188 1787861198.248424 mapwrite 0x00364d thru 0x012fea (0xf99e bytes) 000189 1787861198.298209 write 0x01535f thru 0x02104c (0xbcee bytes) 000190 1787861198.301984 read 0x0088f7 thru 0x00b11f (0x2829 bytes) 000193 1787861198.305116 mapwrite 0x035f2a thru 0x03ffff (0xa0d6 bytes) 000194 1787861198.376887 mapwrite 0x037c5c thru 0x03ea24 (0x6dc9 bytes) 000196 1787861198.401515 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe20eea22 (stale) Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000197 1787861198.759408 punch from 0x00b44c to 0x0178b5 (0xc46a bytes) 000198 1787861198.768967 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000199 1787861199.305406 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3ed4b30f Mirror 10: 0x3ed4b30f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000200 1787861200.626345 punch from 0x02b205 to 0x03489d (0x9699 bytes) 000201 1787861200.685715 mapread 0x00f0f4 thru 0x013872 (0x477f bytes) 000202 1787861200.718975 write 0x01a6b0 thru 0x026cc0 (0xc611 bytes) 000203 1787861200.742730 trunc from 0x040000 to 0x02b825 (0x147da bytes) 000204 1787861200.779474 write 0x023a61 thru 0x029a1f (0x5fbf bytes) 000206 1787861200.787514 fallocate 0x0108d8 thru 0x01800d (0x7736 bytes) 000207 1787861200.794851 mapwrite 0x028557 thru 0x033497 (0xaf41 bytes) 000208 1787861200.839715 read 0x02158a thru 0x021d6e (0x7e5 bytes) 000210 1787861200.840782 trunc from 0x033498 to 0x02b255 (0x8242 bytes) 000211 1787861200.859222 write 0x034ec2 thru 0x03e634 (0x9773 bytes) 000212 1787861200.861525 write 0x03f03d thru 0x03ffff (0xfc3 bytes) 000213 1787861200.863613 mapwrite 0x01f16d thru 0x0279fd (0x8891 bytes) 000214 1787861200.904185 write 0x017371 thru 0x01e3ff (0x708f bytes) 000215 1787861200.912737 write 0x03cf3e thru 0x03ffff (0x30c2 bytes) 000216 1787861200.913729 mapwrite 0x02c963 thru 0x038ba4 (0xc242 bytes) 000217 1787861200.969455 mapwrite 0x022573 thru 0x02cc29 (0xa6b7 bytes) 000218 1787861200.994163 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) Mirror 10: 0x3ed4b30f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x93832a8 Mirror 11: 0x93832a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000219 1787861202.199736 trunc from 0x0008dd to 0x001417 (0xb3b bytes) 000221 1787861202.263586 read 0x00df17 thru 0x0147e7 (0x68d1 bytes) 000222 1787861202.275150 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xe20eea22 (stale) Mirror 8: 0xe20eea22 (stale) Mirror 9: 0xe20eea22 (stale) Mirror 10: 0x3ed4b30f (stale) Mirror 11: 0x93832a8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x93832a8 Mirror 12: 0x93832a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000223 1787861203.562635 mapwrite 0x01c1f4 thru 0x0252f1 (0x90fe bytes) 000225 1787861203.698616 punch from 0x0189d1 to 0x020aeb (0x811b bytes) 000226 1787861203.717915 mapwrite 0x00a88f thru 0x017a91 (0xd203 bytes) 000227 1787861203.803678 MIRROR_RESYNC 7 mirrors 000228 1787861205.650351 trunc from 0x02b2ba to 0x03a507 (0xf24e bytes) 000229 1787861205.690317 trunc from 0x040000 to 0x03b3ff (0x4c00 bytes) 000230 1787861205.712288 write 0x020e12 thru 0x029dc4 (0x8fb3 bytes) 000233 1787861205.725343 mapread 0x03503d thru 0x037f07 (0x2ecb bytes) 000234 1787861205.746933 read 0x03a04a thru 0x03b3ff (0x13b6 bytes) 000236 1787861205.747995 mapwrite 0x02e3eb thru 0x032309 (0x3f1f bytes) 000237 1787861205.781469 trunc from 0x03b400 to 0x032a55 (0x89aa bytes) 000238 1787861205.794476 fallocate 0x00e8fe thru 0x010b35 (0x2238 bytes) 000239 1787861205.805323 MIRROR_RESYNC 7 mirrors 000240 1787861207.561150 write 0x02bd35 thru 0x02dea7 (0x2173 bytes) 000241 1787861207.603496 punch from 0x02e44f to 0x030970 (0x2522 bytes) 000242 1787861207.607857 trunc from 0x032a56 to 0x032287 (0x7ce bytes) 000243 1787861207.630982 write 0x00e863 thru 0x01ae10 (0xc5ae bytes) 000246 1787861207.646432 MIRROR_RESYNC 7 mirrors 000247 1787861209.141659 MIRROR_EXTEND to 8 mirrors 000248 1787861209.749664 fallocate 0x03ff7d thru 0x03ffff (0x83 bytes) 000249 1787861209.793719 mapread 0x013cdd thru 0x021deb (0xe10f bytes) 000250 1787861209.823432 fallocate 0x00d076 thru 0x00dec2 (0xe4d bytes) 000251 1787861209.828043 mapwrite 0x031ba7 thru 0x03c283 (0xa6dd bytes) 000252 1787861209.870292 trunc from 0x040000 to 0x035357 (0xaca8 bytes) 000253 1787861209.893602 mapwrite 0x029c70 thru 0x02ce69 (0x31fa bytes) 000254 1787861209.911190 read 0x01962b thru 0x02596e (0xc344 bytes) 000256 1787861209.912290 fallocate 0x0326e8 thru 0x03cefb (0xa814 bytes) 000257 1787861209.917405 mapread 0x01e977 thru 0x025e82 (0x750c bytes) 000258 1787861209.919297 trunc from 0x03cefc to 0x0194b0 (0x23a4b bytes) 000259 1787861209.935644 mapwrite 0x01fc1c thru 0x023f20 (0x4305 bytes) 000261 1787861209.977672 trunc from 0x0006ae to 0x00c99b (0xc2ee bytes) 000262 1787861209.986812 trunc from 0x001e8f to 0x0040bc (0x222e bytes) 000263 1787861209.991328 read 0x017e32 thru 0x01f7f1 (0x79c0 bytes) 000264 1787861209.992456 trunc from 0x01f12f to 0x023f20 (0x4df2 bytes) 000265 1787861209.996351 read 0x01f1fd thru 0x022435 (0x3239 bytes) 000266 1787861210.005774 trunc from 0x00ca2a to 0x00daed (0x10c4 bytes) 000267 1787861210.010676 fallocate 0x03f8f1 thru 0x03ffff (0x70f bytes) 000268 1787861210.015439 punch from 0x015d2f to 0x02095a (0xac2c bytes) 000269 1787861210.020466 mapwrite 0x0049bc thru 0x009f20 (0x5565 bytes) 000270 1787861210.038345 write 0x033c87 thru 0x033e78 (0x1f2 bytes) 000272 1787861210.040991 trunc from 0x033e79 to 0x01e575 (0x15903 bytes) 000273 1787861210.058736 punch from 0x004a90 to 0x00a39d (0x590e bytes) 000274 1787861210.065416 mapwrite 0x01da4a thru 0x028e73 (0xb42a bytes) 000275 1787861210.134826 fallocate 0x008620 thru 0x00bd9f (0x3780 bytes) 000276 1787861210.142494 mapwrite 0x015abb thru 0x01df18 (0x845e bytes) 000277 1787861210.210454 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x28e74): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28e74): Mirror 7: 0xf964e056 (stale) Mirror 8: 0xf964e056 (stale) Mirror 9: 0xf964e056 (stale) Mirror 10: 0xf964e056 (stale) Mirror 11: 0xf964e056 (stale) Mirror 12: 0xf964e056 (stale) Mirror 13: 0xf964e056 (stale) CRC-32 checksum value for chunk [0, 0x28e74): Mirror 1: 0xab9fdb9 Mirror 14: 0xab9fdb9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28e74: skipped 000278 1787861211.100075 trunc from 0x019c55 to 0x01eb22 (0x4ece bytes) 000279 1787861211.148890 trunc from 0x00bd4b to 0x010a38 (0x4cee bytes) 000280 1787861211.155415 fallocate 0x03d8bf thru 0x03dfcf (0x711 bytes) 000281 1787861211.159859 read 0x036eb1 thru 0x03c8ef (0x5a3f bytes) 000282 1787861211.167301 punch from 0x01f05b to 0x023cdd (0x4c83 bytes) 000283 1787861211.172808 read 0x020988 thru 0x028641 (0x7cba bytes) 000284 1787861211.187983 trunc from 0x03dfd0 to 0x021cf6 (0x1c2d9 bytes) 000286 1787861211.232149 fallocate 0x00fdea thru 0x0189f4 (0x8c0b bytes) 000287 1787861211.237005 fallocate 0x00f301 thru 0x012420 (0x3120 bytes) 000288 1787861211.242212 trunc from 0x021cf7 to 0x001628 (0x206ce bytes) 000290 1787861211.257845 mapread 0x000e34 thru 0x001628 (0x7f5 bytes) 000291 1787861211.258701 fallocate 0x029263 thru 0x0370dd (0xde7b bytes) 000292 1787861211.264325 trunc from 0x0370de to 0x030fad (0x6130 bytes) 000293 1787861211.279160 punch from 0x0147a3 to 0x01f51b (0xad79 bytes) 000294 1787861211.283821 trunc from 0x014f51 to 0x0213a3 (0xc453 bytes) 000295 1787861211.289886 trunc from 0x00d5ff to 0x011be9 (0x45eb bytes) 000296 1787861211.295091 mapread 0x00782c thru 0x0099f4 (0x21c9 bytes) 000297 1787861211.312601 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x30fae): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30fae): Mirror 7: 0xea6faebe (stale) Mirror 8: 0xea6faebe (stale) Mirror 9: 0xea6faebe (stale) Mirror 10: 0xea6faebe (stale) Mirror 11: 0xea6faebe (stale) Mirror 12: 0xea6faebe (stale) Mirror 13: 0xea6faebe (stale) Mirror 14: 0xbe3bf0ad (stale) CRC-32 checksum value for chunk [0, 0x30fae): Mirror 1: 0x220e339f Mirror 15: 0x220e339f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30fae: skipped 000298 1787861212.047003 MIRROR_SPLIT mirror 2 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30fae): Mirror 7: 0xea6faebe (stale) Mirror 9: 0xea6faebe (stale) Mirror 10: 0xea6faebe (stale) Mirror 11: 0xea6faebe (stale) Mirror 12: 0xea6faebe (stale) Mirror 13: 0xea6faebe (stale) Mirror 14: 0xbe3bf0ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30fae): Mirror 7: 0xea6faebe (stale) Mirror 9: 0xea6faebe (stale) Mirror 10: 0xea6faebe (stale) Mirror 11: 0xea6faebe (stale) Mirror 12: 0xea6faebe (stale) Mirror 13: 0xea6faebe (stale) Mirror 14: 0xbe3bf0ad (stale) CRC-32 checksum value for chunk [0, 0x30fae): Mirror 1: 0x220e339f Mirror 15: 0x220e339f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30fae: skipped 000299 1787861212.600949 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x30fae): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30fae): Mirror 7: 0xea6faebe (stale) Mirror 9: 0xea6faebe (stale) Mirror 10: 0xea6faebe (stale) Mirror 11: 0xea6faebe (stale) Mirror 12: 0xea6faebe (stale) Mirror 13: 0xea6faebe (stale) Mirror 14: 0xbe3bf0ad (stale) CRC-32 checksum value for chunk [0, 0x30fae): Mirror 1: 0x220e339f Mirror 15: 0x220e339f Mirror 16: 0x220e339f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30fae: skipped 000301 1787861213.370719 mapwrite 0x032eaa thru 0x03ffff (0xd156 bytes) 000302 1787861213.470884 punch from 0x01390e to 0x01f0b5 (0xb7a8 bytes) 000303 1787861213.476324 trunc from 0x022fb3 to 0x025024 (0x2072 bytes) 000304 1787861213.480792 trunc from 0x040000 to 0x0360ab (0x9f54 bytes) 000305 1787861213.500960 mapread 0x008258 thru 0x010245 (0x7fee bytes) 000306 1787861213.518672 trunc from 0x003a82 to 0x0099a0 (0x5f1f bytes) 000307 1787861213.522528 punch from 0x01028a to 0x017e51 (0x7bc8 bytes) 000308 1787861213.527792 trunc from 0x0360ac to 0x03a36a (0x42bf bytes) 000309 1787861213.541862 trunc from 0x004010 to 0x0123a6 (0xe397 bytes) 000310 1787861213.548356 mapread 0x00e4a0 thru 0x010d09 (0x286a bytes) 000311 1787861213.558326 trunc from 0x03a36b to 0x01f438 (0x1af32 bytes) 000312 1787861213.571321 read 0x01e44c thru 0x01f438 (0xfed bytes) 000313 1787861213.578927 trunc from 0x01f439 to 0x018192 (0x72a6 bytes) 000314 1787861213.592846 write 0x0185b2 thru 0x023a65 (0xb4b4 bytes) 000315 1787861213.601889 fallocate 0x0112a0 thru 0x0145ae (0x330f bytes) 000316 1787861213.608585 write 0x03bbd4 thru 0x03ce81 (0x12ae bytes) 000317 1787861213.609939 punch from 0x01005a to 0x01934c (0x92f3 bytes) 000318 1787861213.624912 mapwrite 0x028a2d thru 0x02b997 (0x2f6b bytes) 000319 1787861213.644684 read 0x0164bf thru 0x01cd79 (0x68bb bytes) 000320 1787861213.654161 mapread 0x00fba4 thru 0x01d001 (0xd45e bytes) 000321 1787861213.655213 fallocate 0x014fbd thru 0x0171a9 (0x21ed bytes) 000322 1787861213.660200 mapread 0x002cfe thru 0x009eb6 (0x71b9 bytes) 000323 1787861213.661278 write 0x03213a thru 0x03ce37 (0xacfe bytes) 000324 1787861213.662962 trunc from 0x03ce82 to 0x01e06d (0x1ee14 bytes) 000325 1787861213.680878 trunc from 0x009a0a to 0x012523 (0x8b1a bytes) 000326 1787861213.685543 trunc from 0x01160f to 0x01837d (0x6d6f bytes) 000327 1787861213.689858 trunc from 0x005934 to 0x010978 (0xb045 bytes) 000328 1787861213.697092 mapread 0x013f46 thru 0x014d0b (0xdc6 bytes) 000329 1787861213.707748 write 0x02aae9 thru 0x038c19 (0xe131 bytes) 000330 1787861213.710151 trunc from 0x02fc38 to 0x032426 (0x27ef bytes) 000331 1787861213.724894 trunc from 0x00c623 to 0x017904 (0xb2e2 bytes) 000332 1787861213.729242 read 0x029b7f thru 0x034f3f (0xb3c1 bytes) 000333 1787861213.748004 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x38c1a): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c1a): Mirror 7: 0x80a83be6 (stale) Mirror 9: 0x80a83be6 (stale) Mirror 10: 0x80a83be6 (stale) Mirror 11: 0x80a83be6 (stale) Mirror 12: 0x80a83be6 (stale) Mirror 13: 0x80a83be6 (stale) Mirror 14: 0x9097ed81 (stale) Mirror 15: 0x6eef38db (stale) Mirror 16: 0x6eef38db (stale) CRC-32 checksum value for chunk [0, 0x38c1a): Mirror 1: 0xe7d00505 Mirror 17: 0xe7d00505 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c1a: skipped 000334 1787861214.630249 punch from 0x0296c9 to 0x038a2f (0xf367 bytes) 000335 1787861214.673598 read 0x00a677 thru 0x0177a0 (0xd12a bytes) 000336 1787861214.683044 read 0x037a03 thru 0x038c19 (0x1217 bytes) 000337 1787861214.697919 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38c1a): Mirror 7: 0x80a83be6 (stale) Mirror 9: 0x80a83be6 (stale) Mirror 10: 0x80a83be6 (stale) Mirror 11: 0x80a83be6 (stale) Mirror 12: 0x80a83be6 (stale) Mirror 13: 0x80a83be6 (stale) Mirror 14: 0x9097ed81 (stale) Mirror 15: 0x6eef38db (stale) Mirror 17: 0xe7d00505 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c1a): Mirror 7: 0x80a83be6 (stale) Mirror 9: 0x80a83be6 (stale) Mirror 10: 0x80a83be6 (stale) Mirror 11: 0x80a83be6 (stale) Mirror 12: 0x80a83be6 (stale) Mirror 13: 0x80a83be6 (stale) Mirror 14: 0x9097ed81 (stale) Mirror 15: 0x6eef38db (stale) Mirror 17: 0xe7d00505 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c1a: skipped 000338 1787861215.247396 punch from 0x029755 to 0x038c19 (0xf4c5 bytes) 000339 1787861215.254773 mapwrite 0x014115 thru 0x020670 (0xc55c bytes) 000340 1787861215.304944 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38c1a): Mirror 7: 0x80a83be6 (stale) Mirror 9: 0x80a83be6 (stale) Mirror 10: 0x80a83be6 (stale) Mirror 11: 0x80a83be6 (stale) Mirror 13: 0x80a83be6 (stale) Mirror 14: 0x9097ed81 (stale) Mirror 15: 0x6eef38db (stale) Mirror 17: 0xe7d00505 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38c1a): Mirror 7: 0x80a83be6 (stale) Mirror 9: 0x80a83be6 (stale) Mirror 10: 0x80a83be6 (stale) Mirror 11: 0x80a83be6 (stale) Mirror 13: 0x80a83be6 (stale) Mirror 14: 0x9097ed81 (stale) Mirror 15: 0x6eef38db (stale) Mirror 17: 0xe7d00505 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38c1a: skipped 000341 1787861215.808880 write 0x02b964 thru 0x02e906 (0x2fa3 bytes) 000343 1787861215.833291 MIRROR_RESYNC 9 mirrors 000344 1787861218.467727 punch from 0x0172d3 to 0x01b37e (0x40ac bytes) 000345 1787861218.510998 trunc from 0x038c1a to 0x0250e3 (0x13b36 bytes) 000347 1787861218.537684 trunc from 0x0250e4 to 0x008451 (0x1cc92 bytes) 000348 1787861218.549574 read 0x002311 thru 0x008451 (0x6141 bytes) 000349 1787861218.557204 mapread 0x007465 thru 0x008451 (0xfed bytes) 000350 1787861218.558590 mapwrite 0x039511 thru 0x03ffff (0x6aef bytes) 000351 1787861218.603634 write 0x03b56c thru 0x03ffff (0x4a94 bytes) 000352 1787861218.605105 fallocate 0x02478f thru 0x02cad2 (0x8344 bytes) 000353 1787861218.629104 trunc from 0x016068 to 0x016107 (0xa0 bytes) 000354 1787861218.633885 trunc from 0x040000 to 0x014f03 (0x2b0fc bytes) 000355 1787861218.658805 mapread 0x006844 thru 0x0073ef (0xbac bytes) 000356 1787861218.659934 write 0x02949a thru 0x034491 (0xaff8 bytes) 000357 1787861218.662964 mapwrite 0x0250f5 thru 0x02db03 (0x8a0f bytes) 000358 1787861218.726191 write 0x022d32 thru 0x032a49 (0xfd18 bytes) 000359 1787861218.727803 trunc from 0x034492 to 0x00c35b (0x28136 bytes) 000360 1787861218.754683 trunc from 0x00c35c to 0x030548 (0x241ed bytes) 000361 1787861218.767890 punch from 0x00090f to 0x007585 (0x6c77 bytes) 000362 1787861218.772208 trunc from 0x030549 to 0x032306 (0x1dbe bytes) 000364 1787861218.788991 fallocate 0x0001ad thru 0x0008b3 (0x707 bytes) 000365 1787861218.809163 MIRROR_RESYNC 9 mirrors 000366 1787861219.940105 mapwrite 0x00b83d thru 0x011dc1 (0x6585 bytes) 000367 1787861220.020133 trunc from 0x01a099 to 0x01d589 (0x34f1 bytes) 000368 1787861220.024766 trunc from 0x032307 to 0x016cc5 (0x1b641 bytes) 000370 1787861220.048285 read 0x010af9 thru 0x016cc5 (0x61cd bytes) 000371 1787861220.058287 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x16cc6): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16cc6): Mirror 7: 0xd221c769 (stale) Mirror 9: 0xd221c769 (stale) Mirror 10: 0xd221c769 (stale) Mirror 11: 0xd221c769 (stale) Mirror 13: 0xd221c769 (stale) Mirror 14: 0xd221c769 (stale) Mirror 15: 0xd221c769 (stale) Mirror 17: 0xd221c769 (stale) CRC-32 checksum value for chunk [0, 0x16cc6): Mirror 1: 0x8cc640b Mirror 18: 0x8cc640b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16cc6: skipped 000372 1787861220.768581 trunc from 0x016cc6 to 0x0329bd (0x1bcf8 bytes) 000373 1787861220.857781 mapwrite 0x039a89 thru 0x03ffff (0x6577 bytes) 000374 1787861220.963930 mapread 0x01dbd4 thru 0x02afdd (0xd40a bytes) 000375 1787861220.988227 trunc from 0x040000 to 0x03ebb6 (0x1449 bytes) 000376 1787861221.005922 trunc from 0x03ebb7 to 0x03cb2a (0x208c bytes) 000377 1787861221.025745 trunc from 0x03cb2b to 0x01360f (0x2951b bytes) 000378 1787861221.041851 mapwrite 0x005f0e thru 0x00f445 (0x9538 bytes) 000379 1787861221.074168 trunc from 0x013610 to 0x028c38 (0x15629 bytes) 000380 1787861221.098766 punch from 0x0136c7 to 0x01904c (0x5986 bytes) 000381 1787861221.106998 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28c39): Mirror 7: 0xfbc1dd6a (stale) Mirror 9: 0xfbc1dd6a (stale) Mirror 10: 0xfbc1dd6a (stale) Mirror 13: 0xfbc1dd6a (stale) Mirror 14: 0xfbc1dd6a (stale) Mirror 15: 0xfbc1dd6a (stale) Mirror 17: 0xfbc1dd6a (stale) Mirror 18: 0x890dbc57 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28c39): Mirror 7: 0xfbc1dd6a (stale) Mirror 9: 0xfbc1dd6a (stale) Mirror 10: 0xfbc1dd6a (stale) Mirror 13: 0xfbc1dd6a (stale) Mirror 14: 0xfbc1dd6a (stale) Mirror 15: 0xfbc1dd6a (stale) Mirror 17: 0xfbc1dd6a (stale) Mirror 18: 0x890dbc57 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28c39: skipped 000382 1787861221.763595 mapwrite 0x002f5c thru 0x0046d3 (0x1778 bytes) 000383 1787861221.810957 fallocate 0x030cda thru 0x03167c (0x9a3 bytes) 000384 1787861221.835529 write 0x0362d0 thru 0x03c800 (0x6531 bytes) 000385 1787861221.853740 trunc from 0x03c801 to 0x016fed (0x25813 bytes) 000386 1787861221.891631 trunc from 0x00b3c9 to 0x00f0b2 (0x3cea bytes) 000387 1787861221.903927 write 0x00c0f5 thru 0x01ae0a (0xed16 bytes) 000388 1787861221.916499 trunc from 0x01ae0b to 0x01f5b9 (0x47af bytes) 000389 1787861221.942840 read 0x001aa2 thru 0x0067d4 (0x4d33 bytes) 000390 1787861221.943607 write 0x03c8ee thru 0x03ffff (0x3712 bytes) 000391 1787861221.946403 read 0x03690c thru 0x03aa83 (0x4178 bytes) 000393 1787861221.954903 trunc from 0x0252e9 to 0x026d6b (0x1a83 bytes) 000394 1787861221.959775 write 0x01106b thru 0x01a1a0 (0x9136 bytes) 000395 1787861221.960962 write 0x0145bc thru 0x014a79 (0x4be bytes) 000396 1787861221.961568 read 0x01e913 thru 0x020f26 (0x2614 bytes) 000397 1787861221.962091 read 0x039164 thru 0x03ffff (0x6e9c bytes) 000398 1787861221.963388 mapread 0x016033 thru 0x0208b0 (0xa87e bytes) 000399 1787861221.968173 MIRROR_RESYNC 9 mirrors 000400 1787861223.850523 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) 000401 1787861224.299658 punch from 0x00b17c to 0x011f6e (0x6df3 bytes) 000402 1787861224.357190 trunc from 0x040000 to 0x030402 (0xfbfd bytes) 000403 1787861224.377508 read 0x00e110 thru 0x016ad3 (0x89c4 bytes) 000404 1787861224.386522 read 0x008f9d thru 0x01301b (0xa07f bytes) 000405 1787861224.397861 trunc from 0x030403 to 0x01fb39 (0x108c9 bytes) 000406 1787861224.409815 fallocate 0x00e3de thru 0x0184b9 (0xa0dc bytes) 000407 1787861224.413702 write 0x024705 thru 0x02a717 (0x6013 bytes) 000408 1787861224.415948 trunc from 0x02a718 to 0x0380ca (0xd9b3 bytes) 000409 1787861224.429613 read 0x02f7a8 thru 0x0380ca (0x8923 bytes) 000410 1787861224.435085 fallocate 0x031268 thru 0x038c3f (0x79d8 bytes) 000411 1787861224.438386 fallocate 0x02dfe6 thru 0x039c88 (0xbca3 bytes) 000412 1787861224.443202 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x39c89): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39c89): Mirror 7: 0xf1fd477d (stale) Mirror 9: 0xf1fd477d (stale) Mirror 10: 0xf1fd477d (stale) Mirror 13: 0xf1fd477d (stale) Mirror 15: 0xf1fd477d (stale) Mirror 17: 0xf1fd477d (stale) Mirror 18: 0xf1fd477d (stale) CRC-32 checksum value for chunk [0, 0x39c89): Mirror 1: 0x8b0bb88c Mirror 19: 0x8b0bb88c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39c89: skipped 000413 1787861225.414274 punch from 0x020510 to 0x02e730 (0xe221 bytes) 000414 1787861225.471601 fallocate 0x0326e9 thru 0x03c9fa (0xa312 bytes) 000416 1787861225.477775 trunc from 0x03c9fb to 0x025c94 (0x16d66 bytes) 000417 1787861225.522676 mapread 0x01cd77 thru 0x022146 (0x53d0 bytes) 000418 1787861225.549188 punch from 0x01692b to 0x02321c (0xc8f2 bytes) 000419 1787861225.554872 trunc from 0x000962 to 0x00c747 (0xbde6 bytes) 000420 1787861225.563177 MIRROR_RESYNC 9 mirrors 000421 1787861226.926190 mapwrite 0x023425 thru 0x02b98d (0x8569 bytes) 000422 1787861227.021960 read 0x00498d thru 0x00f4fe (0xab72 bytes) 000423 1787861227.037584 mapwrite 0x00fd9a thru 0x01c923 (0xcb8a bytes) 000424 1787861227.078885 read 0x01de90 thru 0x022f3a (0x50ab bytes) 000425 1787861227.080818 punch from 0x00b287 to 0x00d737 (0x24b1 bytes) 000426 1787861227.089796 read 0x018d71 thru 0x01bc7f (0x2f0f bytes) 000428 1787861227.092619 trunc from 0x01009c to 0x017f42 (0x7ea7 bytes) 000429 1787861227.101801 fallocate 0x00279e thru 0x00700c (0x486f bytes) 000430 1787861227.106781 fallocate 0x02291c thru 0x030f10 (0xe5f5 bytes) 000431 1787861227.113482 trunc from 0x0137da to 0x01980b (0x6032 bytes) 000432 1787861227.119666 punch from 0x0296b3 to 0x030f10 (0x785e bytes) 000433 1787861227.124167 mapwrite 0x02a55f thru 0x02b42e (0xed0 bytes) 000434 1787861227.146401 write 0x01f86d thru 0x020306 (0xa9a bytes) 000435 1787861227.147311 punch from 0x0220ff to 0x03020a (0xe10c bytes) 000436 1787861227.153132 read 0x015706 thru 0x01cb18 (0x7413 bytes) 000437 1787861227.158284 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x30f11): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30f11): Mirror 7: 0x3d71d884 (stale) Mirror 9: 0x3d71d884 (stale) Mirror 10: 0x3d71d884 (stale) Mirror 13: 0x3d71d884 (stale) Mirror 15: 0x3d71d884 (stale) Mirror 17: 0x3d71d884 (stale) Mirror 18: 0x3d71d884 (stale) Mirror 19: 0x3d71d884 (stale) CRC-32 checksum value for chunk [0, 0x30f11): Mirror 1: 0x62ef1c4e Mirror 20: 0x62ef1c4e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30f11: skipped 000438 1787861228.427547 write 0x03f535 thru 0x03ffff (0xacb bytes) 000439 1787861228.479654 trunc from 0x040000 to 0x027df8 (0x18207 bytes) 000440 1787861228.525325 punch from 0x000e6d to 0x008c22 (0x7db6 bytes) 000441 1787861228.533344 trunc from 0x01febd to 0x026988 (0x6acc bytes) 000442 1787861228.541663 read 0x003f6c thru 0x008d9a (0x4e2f bytes) 000443 1787861228.550319 fallocate 0x01aac7 thru 0x021db3 (0x72ed bytes) 000445 1787861228.558534 mapwrite 0x019bd1 thru 0x01f098 (0x54c8 bytes) 000446 1787861228.600448 punch from 0x020836 to 0x02604c (0x5817 bytes) 000447 1787861228.606080 mapread 0x004f54 thru 0x009dc0 (0x4e6d bytes) 000448 1787861228.617981 fallocate 0x017cbb thru 0x02645f (0xe7a5 bytes) 000449 1787861228.627820 punch from 0x01a9fa to 0x020249 (0x5850 bytes) 000450 1787861228.640754 fallocate 0x03f26e thru 0x03ffff (0xd92 bytes) 000451 1787861228.646895 trunc from 0x003aeb to 0x00d738 (0x9c4e bytes) 000452 1787861228.657258 trunc from 0x0052e5 to 0x00df30 (0x8c4c bytes) 000453 1787861228.662271 punch from 0x00cf6b to 0x010d87 (0x3e1d bytes) 000454 1787861228.669315 mapwrite 0x019fbe thru 0x01da99 (0x3adc bytes) 000455 1787861228.693751 mapwrite 0x0207b3 thru 0x0275b1 (0x6dff bytes) 000456 1787861228.725312 punch from 0x000516 to 0x005213 (0x4cfe bytes) 000457 1787861228.730608 mapwrite 0x01418c thru 0x01d500 (0x9375 bytes) 000458 1787861228.779309 read 0x01317c thru 0x02084f (0xd6d4 bytes) 000459 1787861228.782836 fallocate 0x026d8b thru 0x03319d (0xc413 bytes) 000460 1787861228.789687 fallocate 0x012f3a thru 0x01f940 (0xca07 bytes) 000461 1787861228.807476 trunc from 0x0113f2 to 0x016cb3 (0x58c2 bytes) 000462 1787861228.814402 mapread 0x02138f thru 0x027df8 (0x6a6a bytes) 000463 1787861228.817918 fallocate 0x025de5 thru 0x02f30b (0x9527 bytes) 000464 1787861228.832946 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f30c): Mirror 7: 0x58abf39c (stale) Mirror 9: 0x58abf39c (stale) Mirror 10: 0x58abf39c (stale) Mirror 15: 0x58abf39c (stale) Mirror 17: 0x58abf39c (stale) Mirror 18: 0x58abf39c (stale) Mirror 19: 0x58abf39c (stale) Mirror 20: 0xab3041bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f30c): Mirror 7: 0x58abf39c (stale) Mirror 9: 0x58abf39c (stale) Mirror 10: 0x58abf39c (stale) Mirror 15: 0x58abf39c (stale) Mirror 17: 0x58abf39c (stale) Mirror 18: 0x58abf39c (stale) Mirror 19: 0x58abf39c (stale) Mirror 20: 0xab3041bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f30c: skipped 000465 1787861229.371660 read 0x026013 thru 0x026fc2 (0xfb0 bytes) 000466 1787861229.377361 mapwrite 0x00f1fa thru 0x017538 (0x833f bytes) 000467 1787861229.408616 write 0x01e691 thru 0x027ed7 (0x9847 bytes) 000468 1787861229.419624 punch from 0x02bda6 to 0x02e79b (0x29f6 bytes) 000469 1787861229.423884 write 0x032b2e thru 0x0335a1 (0xa74 bytes) 000470 1787861229.425694 mapwrite 0x01b097 thru 0x01bb7d (0xae7 bytes) 000471 1787861229.444646 punch from 0x008685 to 0x0151e4 (0xcb60 bytes) 000472 1787861229.452859 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x335a2): Mirror 7: 0xa0080649 (stale) Mirror 10: 0xa0080649 (stale) Mirror 15: 0xa0080649 (stale) Mirror 17: 0xa0080649 (stale) Mirror 18: 0xa0080649 (stale) Mirror 19: 0xa0080649 (stale) Mirror 20: 0xbb7bd716 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x335a2): Mirror 7: 0xa0080649 (stale) Mirror 10: 0xa0080649 (stale) Mirror 15: 0xa0080649 (stale) Mirror 17: 0xa0080649 (stale) Mirror 18: 0xa0080649 (stale) Mirror 19: 0xa0080649 (stale) Mirror 20: 0xbb7bd716 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x335a2: skipped 000473 1787861229.881646 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x335a2): Mirror 10: 0xa0080649 (stale) Mirror 15: 0xa0080649 (stale) Mirror 17: 0xa0080649 (stale) Mirror 18: 0xa0080649 (stale) Mirror 19: 0xa0080649 (stale) Mirror 20: 0xbb7bd716 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x335a2): Mirror 10: 0xa0080649 (stale) Mirror 15: 0xa0080649 (stale) Mirror 17: 0xa0080649 (stale) Mirror 18: 0xa0080649 (stale) Mirror 19: 0xa0080649 (stale) Mirror 20: 0xbb7bd716 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x335a2: skipped 000474 1787861230.399633 mapwrite 0x03bc72 thru 0x03f436 (0x37c5 bytes) 000475 1787861230.470572 mapread 0x023989 thru 0x0309c4 (0xd03c bytes) 000476 1787861230.489620 trunc from 0x03f437 to 0x02ad4d (0x146e9 bytes) 000477 1787861230.500930 trunc from 0x02ad4e to 0x0307cb (0x5a7e bytes) 000479 1787861230.514928 fallocate 0x023543 thru 0x026352 (0x2e10 bytes) 000480 1787861230.519888 read 0x02d662 thru 0x0307cb (0x316a bytes) 000481 1787861230.525373 fallocate 0x015d74 thru 0x01dff1 (0x827e bytes) 000482 1787861230.529601 mapwrite 0x03ceb5 thru 0x03ffff (0x314b bytes) 000483 1787861230.561864 mapwrite 0x014f9f thru 0x021776 (0xc7d8 bytes) 000485 1787861230.588962 write 0x037c21 thru 0x0386ad (0xa8d bytes) 000486 1787861230.589960 punch from 0x005aa8 to 0x00e10a (0x8663 bytes) 000487 1787861230.594104 punch from 0x01217f to 0x01f305 (0xd187 bytes) 000488 1787861230.598377 write 0x014ca8 thru 0x0207c1 (0xbb1a bytes) 000489 1787861230.604246 fallocate 0x01be05 thru 0x01e8cf (0x2acb bytes) 000490 1787861230.608713 mapread 0x021783 thru 0x02313c (0x19ba bytes) 000491 1787861230.612891 MIRROR_RESYNC 7 mirrors 000492 1787861231.912271 MIRROR_EXTEND to 8 mirrors 000493 1787861232.607775 punch from 0x0019a8 to 0x003468 (0x1ac1 bytes) 000494 1787861232.643741 trunc from 0x040000 to 0x034d5a (0xb2a5 bytes) 000495 1787861232.665443 mapread 0x019db2 thru 0x027565 (0xd7b4 bytes) 000496 1787861232.689750 mapwrite 0x007263 thru 0x00e457 (0x71f5 bytes) 000497 1787861232.711800 read 0x00c2b3 thru 0x014f57 (0x8ca5 bytes) 000498 1787861232.716616 trunc from 0x006020 to 0x014f5c (0xef3d bytes) 000499 1787861232.721670 read 0x013ad9 thru 0x018969 (0x4e91 bytes) 000500 1787861232.730321 read 0x005f92 thru 0x00751e (0x158d bytes) 000501 1787861232.730861 write 0x03ea3a thru 0x03ffff (0x15c6 bytes) 000502 1787861232.731754 write 0x028488 thru 0x03072b (0x82a4 bytes) 000503 1787861232.732818 mapwrite 0x000f2d thru 0x00157a (0x64e bytes) 000505 1787861232.755854 fallocate 0x013795 thru 0x01b720 (0x7f8c bytes) 000506 1787861232.765779 fallocate 0x016662 thru 0x01ff50 (0x98ef bytes) 000507 1787861232.774679 read 0x01624e thru 0x01db0a (0x78bd bytes) 000509 1787861232.775692 trunc from 0x0018bc to 0x005aa6 (0x41eb bytes) 000510 1787861232.794302 write 0x034491 thru 0x03ffff (0xbb6f bytes) 000511 1787861232.795710 mapwrite 0x02fba0 thru 0x0372d1 (0x7732 bytes) 000512 1787861232.817577 trunc from 0x040000 to 0x0199c1 (0x2663e bytes) 000513 1787861232.849725 write 0x0018a8 thru 0x011555 (0xfcae bytes) 000514 1787861232.857986 trunc from 0x0199c2 to 0x02a345 (0x10984 bytes) 000515 1787861232.876877 trunc from 0x00dc6f to 0x00f150 (0x14e2 bytes) 000516 1787861232.899889 fallocate 0x035885 thru 0x03e10e (0x888a bytes) 000518 1787861232.908711 punch from 0x003f63 to 0x013f5f (0xfffd bytes) 000519 1787861232.917491 fallocate 0x00f115 thru 0x011c29 (0x2b15 bytes) 000520 1787861232.922401 trunc from 0x02a39e to 0x02e9f8 (0x465b bytes) 000521 1787861232.932784 punch from 0x028a6d to 0x02c634 (0x3bc8 bytes) 000522 1787861232.953226 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e10f): Mirror 10: 0xdf2daa22 (stale) Mirror 15: 0xdf2daa22 (stale) Mirror 18: 0xdf2daa22 (stale) Mirror 19: 0xdf2daa22 (stale) Mirror 20: 0xdf2daa22 (stale) Mirror 21: 0xdf2daa22 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e10f): Mirror 10: 0xdf2daa22 (stale) Mirror 15: 0xdf2daa22 (stale) Mirror 18: 0xdf2daa22 (stale) Mirror 19: 0xdf2daa22 (stale) Mirror 20: 0xdf2daa22 (stale) Mirror 21: 0xdf2daa22 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e10f: skipped 000523 1787861233.410851 trunc from 0x03e10f to 0x008ee1 (0x3522d bytes) 000524 1787861233.428799 trunc from 0x002ac2 to 0x008ee1 (0x6420 bytes) 000525 1787861233.433078 trunc from 0x005892 to 0x008ee1 (0x3650 bytes) 000526 1787861233.437284 punch from 0x00476b to 0x008ee1 (0x4777 bytes) 000527 1787861233.443891 read 0x00591b thru 0x008ee1 (0x35c7 bytes) 000528 1787861233.448788 fallocate 0x02fcb9 thru 0x03e6d4 (0xea1c bytes) 000529 1787861233.452795 write 0x01d685 thru 0x0251e1 (0x7b5d bytes) 000530 1787861233.454181 read 0x01f719 thru 0x0251e1 (0x5ac9 bytes) 000531 1787861233.455071 write 0x001e01 thru 0x0021cd (0x3cd bytes) 000532 1787861233.462325 punch from 0x01193b to 0x017c42 (0x6308 bytes) 000533 1787861233.467658 write 0x039e3e thru 0x03c0ee (0x22b1 bytes) 000534 1787861233.468723 trunc from 0x006e51 to 0x0128ea (0xba9a bytes) 000536 1787861233.473484 trunc from 0x03c0ef to 0x038d80 (0x336e bytes) 000538 1787861233.487871 fallocate 0x0342eb thru 0x03f529 (0xb23f bytes) 000539 1787861233.492688 mapread 0x023dcd thru 0x02bdc0 (0x7ff4 bytes) 000540 1787861233.500762 fallocate 0x02c8ac thru 0x031485 (0x4bda bytes) 000541 1787861233.504859 read 0x0085f6 thru 0x009d64 (0x176f bytes) 000542 1787861233.513858 mapwrite 0x0045ae thru 0x009340 (0x4d93 bytes) 000543 1787861233.530559 fallocate 0x03a160 thru 0x03ffff (0x5ea0 bytes) 000544 1787861233.534992 read 0x0392cb thru 0x03ffff (0x6d35 bytes) 000545 1787861233.536280 fallocate 0x028892 thru 0x02f952 (0x70c1 bytes) 000546 1787861233.540736 read 0x03e635 thru 0x03ffff (0x19cb bytes) 000547 1787861233.541674 mapread 0x000125 thru 0x0080db (0x7fb7 bytes) 000548 1787861233.546505 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xa5dd58de (stale) Mirror 18: 0xa5dd58de (stale) Mirror 19: 0xa5dd58de (stale) Mirror 20: 0xa5dd58de (stale) Mirror 21: 0xa5dd58de (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0xa5dd58de (stale) Mirror 18: 0xa5dd58de (stale) Mirror 19: 0xa5dd58de (stale) Mirror 20: 0xa5dd58de (stale) Mirror 21: 0xa5dd58de (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000549 1787861233.923301 trunc from 0x040000 to 0x02b4a6 (0x14b59 bytes) 000550 1787861233.948757 mapwrite 0x032162 thru 0x03bc95 (0x9b34 bytes) 000551 1787861234.012547 mapread 0x03b8f3 thru 0x03bc95 (0x3a3 bytes) 000553 1787861234.013813 mapread 0x01285a thru 0x01c724 (0x9ecb bytes) 000554 1787861234.030708 punch from 0x035870 to 0x03bc95 (0x6426 bytes) 000555 1787861234.035593 punch from 0x00051a to 0x00b7cb (0xb2b2 bytes) 000556 1787861234.040588 trunc from 0x00149d to 0x00d533 (0xc097 bytes) 000557 1787861234.044987 mapread 0x02338e thru 0x031c35 (0xe8a8 bytes) 000558 1787861234.053949 punch from 0x03a0de to 0x03bc95 (0x1bb8 bytes) 000559 1787861234.057989 read 0x023f8f thru 0x02fabe (0xbb30 bytes) 000560 1787861234.058968 mapread 0x00a4fe thru 0x0140da (0x9bdd bytes) 000561 1787861234.073326 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3bc96): Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bc96): Mirror 15: 0x43f8fc53 (stale) Mirror 18: 0x43f8fc53 (stale) Mirror 19: 0x43f8fc53 (stale) Mirror 20: 0x43f8fc53 (stale) Mirror 21: 0x43f8fc53 (stale) CRC-32 checksum value for chunk [0, 0x3bc96): Mirror 1: 0xd24ddc68 Mirror 22: 0xd24ddc68 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bc96: skipped 000563 1787861235.055309 read 0x039623 thru 0x03bc95 (0x2673 bytes) 000564 1787861235.070674 fallocate 0x017cdf thru 0x02402d (0xc34f bytes) 000565 1787861235.142475 mapwrite 0x0110b5 thru 0x0113b0 (0x2fc bytes) 000566 1787861235.159722 trunc from 0x03bc96 to 0x03dd86 (0x20f1 bytes) 000567 1787861235.194985 mapread 0x0275da thru 0x02f5a3 (0x7fca bytes) 000568 1787861235.213509 read 0x028870 thru 0x03151f (0x8cb0 bytes) 000569 1787861235.235358 trunc from 0x02ee37 to 0x03073b (0x1905 bytes) 000570 1787861235.252905 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dd87): Mirror 18: 0x8eb257f4 (stale) Mirror 19: 0x8eb257f4 (stale) Mirror 20: 0x8eb257f4 (stale) Mirror 21: 0x8eb257f4 (stale) Mirror 22: 0x87b67dd6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd87): Mirror 18: 0x8eb257f4 (stale) Mirror 19: 0x8eb257f4 (stale) Mirror 20: 0x8eb257f4 (stale) Mirror 21: 0x8eb257f4 (stale) Mirror 22: 0x87b67dd6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd87: skipped 000571 1787861235.908815 mapwrite 0x03906b thru 0x03d34b (0x42e1 bytes) 000572 1787861235.948482 trunc from 0x00b68e to 0x00ee5e (0x37d1 bytes) 000573 1787861235.952944 trunc from 0x03dd87 to 0x0055c2 (0x387c4 bytes) 000574 1787861235.978824 mapread 0x004af1 thru 0x0055c2 (0xad2 bytes) 000575 1787861235.986354 trunc from 0x0055c3 to 0x005e76 (0x8b4 bytes) 000576 1787861236.000001 fallocate 0x01ebb0 thru 0x0241a9 (0x55fa bytes) 000577 1787861236.005884 mapwrite 0x021831 thru 0x027624 (0x5df4 bytes) 000578 1787861236.084879 fallocate 0x012451 thru 0x01aff9 (0x8ba9 bytes) 000579 1787861236.098377 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27625): Mirror 18: 0xfe49434b (stale) Mirror 20: 0xfe49434b (stale) Mirror 21: 0xfe49434b (stale) Mirror 22: 0x63261a6f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27625): Mirror 18: 0xfe49434b (stale) Mirror 20: 0xfe49434b (stale) Mirror 21: 0xfe49434b (stale) Mirror 22: 0x63261a6f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27625: skipped 000580 1787861236.450002 mapread 0x0259ab thru 0x0259b6 (0xc bytes) 000581 1787861236.456796 trunc from 0x01cbd6 to 0x027624 (0xaa4f bytes) 000582 1787861236.462241 read 0x00370e thru 0x009118 (0x5a0b bytes) 000583 1787861236.471921 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x27625): Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27625): Mirror 18: 0xfe49434b (stale) Mirror 20: 0xfe49434b (stale) Mirror 21: 0xfe49434b (stale) Mirror 22: 0x63261a6f (stale) CRC-32 checksum value for chunk [0, 0x27625): Mirror 1: 0x89417089 Mirror 23: 0x89417089 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27625: skipped 000584 1787861237.268715 write 0x016316 thru 0x018e2f (0x2b1a bytes) 000585 1787861237.348636 mapwrite 0x01cc08 thru 0x026195 (0x958e bytes) 000586 1787861237.430655 mapwrite 0x019159 thru 0x024e3d (0xbce5 bytes) 000587 1787861237.457428 read 0x009abb thru 0x0122bf (0x8805 bytes) 000588 1787861237.493590 mapwrite 0x01ca67 thru 0x02af28 (0xe4c2 bytes) 000589 1787861237.571988 trunc from 0x02af29 to 0x01bc5b (0xf2cd bytes) 000590 1787861237.589258 trunc from 0x01bc5c to 0x01b356 (0x905 bytes) 000591 1787861237.609533 mapread 0x00866a thru 0x017eed (0xf884 bytes) 000594 1787861237.658386 fallocate 0x021bab thru 0x02b9ca (0x9e20 bytes) 000595 1787861237.663684 write 0x020148 thru 0x02e26d (0xe126 bytes) 000596 1787861237.665931 read 0x027057 thru 0x02900b (0x1fb5 bytes) 000597 1787861237.671449 read 0x01ab0e thru 0x029cde (0xf1d1 bytes) 000598 1787861237.689621 punch from 0x027403 to 0x02a08d (0x2c8b bytes) 000599 1787861237.747537 fallocate 0x0049ef thru 0x01117c (0xc78e bytes) 000600 1787861237.753600 read 0x017870 thru 0x022a9a (0xb22b bytes) 000601 1787861237.755364 trunc from 0x00ce88 to 0x01b875 (0xe9ee bytes) 000603 1787861237.771993 fallocate 0x0085b8 thru 0x015b2c (0xd575 bytes) 000604 1787861237.781896 write 0x014237 thru 0x020dce (0xcb98 bytes) 000605 1787861237.799889 trunc from 0x02e26e to 0x01bb21 (0x1274c bytes) 000606 1787861237.820993 punch from 0x01a003 to 0x01bb21 (0x1b1f bytes) 000607 1787861237.836359 write 0x014358 thru 0x01bcd9 (0x7982 bytes) 000608 1787861237.842589 fallocate 0x02842e thru 0x02b260 (0x2e33 bytes) 000609 1787861237.847006 read 0x01859e thru 0x01bcd9 (0x373c bytes) 000610 1787861237.851913 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1bcda): Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1bcda): Mirror 18: 0xddd4f316 (stale) Mirror 20: 0xddd4f316 (stale) Mirror 21: 0xddd4f316 (stale) Mirror 22: 0xaa7ef099 (stale) Mirror 23: 0xaa7ef099 (stale) CRC-32 checksum value for chunk [0, 0x1bcda): Mirror 1: 0x8579f163 Mirror 24: 0x8579f163 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1bcda: skipped 000611 1787861238.965440 mapwrite 0x0106f2 thru 0x014139 (0x3a48 bytes) 000612 1787861239.133266 trunc from 0x00a26b to 0x0130db (0x8e71 bytes) 000613 1787861239.141548 trunc from 0x00688e to 0x009b00 (0x3273 bytes) 000614 1787861239.148518 trunc from 0x01bcda to 0x03ac75 (0x1ef9c bytes) 000615 1787861239.215610 mapread 0x00da51 thru 0x00f6ef (0x1c9f bytes) 000616 1787861239.243411 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ac76): Mirror 18: 0x80357527 (stale) Mirror 20: 0x80357527 (stale) Mirror 21: 0x80357527 (stale) Mirror 22: 0x747f2077 (stale) Mirror 24: 0xcc0a2b82 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac76): Mirror 18: 0x80357527 (stale) Mirror 20: 0x80357527 (stale) Mirror 21: 0x80357527 (stale) Mirror 22: 0x747f2077 (stale) Mirror 24: 0xcc0a2b82 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac76: skipped 000617 1787861239.683704 fallocate 0x026983 thru 0x035fea (0xf668 bytes) 000618 1787861239.690181 punch from 0x013336 to 0x01527b (0x1f46 bytes) 000619 1787861239.696280 mapwrite 0x034e9e thru 0x03f116 (0xa279 bytes) 000620 1787861239.755880 punch from 0x00d62c to 0x012bf5 (0x55ca bytes) 000622 1787861239.761765 punch from 0x01ced8 to 0x025396 (0x84bf bytes) 000623 1787861239.766777 mapwrite 0x03b581 thru 0x03ffff (0x4a7f bytes) 000624 1787861239.804781 mapwrite 0x025d31 thru 0x032559 (0xc829 bytes) 000625 1787861239.835854 fallocate 0x0033da thru 0x0057a6 (0x23cd bytes) 000626 1787861239.840839 read 0x01e434 thru 0x02e0b1 (0xfc7e bytes) 000627 1787861239.842203 punch from 0x03b08f to 0x03fbf3 (0x4b65 bytes) 000628 1787861239.850543 trunc from 0x040000 to 0x029278 (0x16d87 bytes) 000629 1787861239.871176 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29279): Mirror 18: 0xb8c2596 (stale) Mirror 21: 0xb8c2596 (stale) Mirror 22: 0x6573f799 (stale) Mirror 24: 0xec4e217 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29279): Mirror 18: 0xb8c2596 (stale) Mirror 21: 0xb8c2596 (stale) Mirror 22: 0x6573f799 (stale) Mirror 24: 0xec4e217 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29279: skipped 000630 1787861240.186909 punch from 0x027c7c to 0x029278 (0x15fd bytes) 000631 1787861240.191274 punch from 0x000b10 to 0x00f9b0 (0xeea1 bytes) 000632 1787861240.196232 mapread 0x00c4cd thru 0x010844 (0x4378 bytes) 000633 1787861240.211690 fallocate 0x035ed8 thru 0x03ffff (0xa128 bytes) 000634 1787861240.216240 mapread 0x001c2b thru 0x00dd66 (0xc13c bytes) 000635 1787861240.224149 fallocate 0x03d519 thru 0x03ffff (0x2ae7 bytes) 000636 1787861240.227750 mapread 0x005924 thru 0x00c30b (0x69e8 bytes) 000637 1787861240.229119 mapread 0x013fb3 thru 0x01441a (0x468 bytes) 000638 1787861240.229987 mapwrite 0x020d00 thru 0x028a3e (0x7d3f bytes) 000639 1787861240.255694 fallocate 0x0068b8 thru 0x007220 (0x969 bytes) 000640 1787861240.260243 fallocate 0x039d9b thru 0x03d55e (0x37c4 bytes) 000642 1787861240.265602 write 0x022961 thru 0x0254be (0x2b5e bytes) 000643 1787861240.270494 write 0x00ab60 thru 0x012c51 (0x80f2 bytes) 000644 1787861240.271815 mapwrite 0x0256df thru 0x03036e (0xac90 bytes) 000645 1787861240.311750 punch from 0x022458 to 0x02636b (0x3f14 bytes) 000646 1787861240.317676 trunc from 0x02e1ae to 0x03036e (0x21c1 bytes) 000647 1787861240.321626 punch from 0x023b5f to 0x024acf (0xf71 bytes) 000648 1787861240.330048 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3036f): Mirror 21: 0xc47cb4fa (stale) Mirror 22: 0x884ec5ed (stale) Mirror 24: 0xb514b722 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3036f): Mirror 21: 0xc47cb4fa (stale) Mirror 22: 0x884ec5ed (stale) Mirror 24: 0xb514b722 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3036f: skipped 000650 1787861240.651288 trunc from 0x02f6d1 to 0x03036e (0xc9e bytes) 000651 1787861240.665489 trunc from 0x03036f to 0x0024b3 (0x2debb bytes) 000652 1787861240.698854 trunc from 0x0024b4 to 0x02d001 (0x2ab4e bytes) 000653 1787861240.731386 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d002): Mirror 21: 0xddbdd321 (stale) Mirror 22: 0x4bee1832 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d002): Mirror 21: 0xddbdd321 (stale) Mirror 22: 0x4bee1832 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d002: skipped 000654 1787861241.033216 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d002): Mirror 22: 0x4bee1832 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d002): Mirror 22: 0x4bee1832 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d002: skipped 000655 1787861241.278937 mapread 0x0110d6 thru 0x014b7c (0x3aa7 bytes) 000656 1787861241.300469 mapread 0x01c0cb thru 0x029486 (0xd3bc bytes) 000658 1787861241.319363 trunc from 0x02d002 to 0x014fef (0x18012 bytes) 000659 1787861241.335863 fallocate 0x02893a thru 0x030d60 (0x8427 bytes) 000660 1787861241.342448 trunc from 0x016830 to 0x019e4b (0x361c bytes) 000661 1787861241.346832 read 0x02222c thru 0x02e67e (0xc453 bytes) 000662 1787861241.360065 MIRROR_RESYNC 2 mirrors 000663 1787861242.114157 mapread 0x01acf2 thru 0x026da1 (0xc0b0 bytes) 000664 1787861242.142974 write 0x010c6a thru 0x0153b9 (0x4750 bytes) 000665 1787861242.173774 trunc from 0x030d61 to 0x007cc7 (0x29099 bytes) 000666 1787861242.219748 read 0x004a68 thru 0x007cc7 (0x3260 bytes) 000667 1787861242.226298 fallocate 0x00936d thru 0x017d96 (0xea2a bytes) 000668 1787861242.230869 trunc from 0x00a39e to 0x015609 (0xb26c bytes) 000669 1787861242.238506 mapwrite 0x03698b thru 0x03ffff (0x9675 bytes) 000670 1787861242.343672 fallocate 0x020050 thru 0x0298c6 (0x9877 bytes) 000671 1787861242.354971 mapwrite 0x0060c5 thru 0x00c91f (0x685b bytes) 000672 1787861242.407362 trunc from 0x040000 to 0x0137cd (0x2c832 bytes) 000673 1787861242.425231 mapread 0x0095ee thru 0x0137cd (0xa1e0 bytes) 000674 1787861242.433160 fallocate 0x007ac2 thru 0x009673 (0x1bb2 bytes) 000675 1787861242.444551 mapwrite 0x00ed93 thru 0x013ac6 (0x4d34 bytes) 000677 1787861242.479151 trunc from 0x000980 to 0x00b5f2 (0xac73 bytes) 000678 1787861242.484694 punch from 0x006ba9 to 0x00d5f1 (0x6a49 bytes) 000679 1787861242.494428 MIRROR_RESYNC 2 mirrors 000680 1787861243.128532 write 0x007652 thru 0x013e38 (0xc7e7 bytes) 000681 1787861243.164580 read 0x00306a thru 0x008140 (0x50d7 bytes) 000682 1787861243.174355 MIRROR_RESYNC 2 mirrors 000683 1787861243.661435 fallocate 0x00da3a thru 0x00e763 (0xd2a bytes) 000684 1787861243.702242 trunc from 0x011d11 to 0x013e38 (0x2128 bytes) 000685 1787861243.712200 fallocate 0x019bf1 thru 0x02223e (0x864e bytes) 000686 1787861243.716849 fallocate 0x01f5b6 thru 0x02c136 (0xcb81 bytes) 000687 1787861243.721958 write 0x021299 thru 0x022caf (0x1a17 bytes) 000688 1787861243.731741 fallocate 0x03e8a4 thru 0x03ffff (0x175c bytes) 000689 1787861243.736053 mapwrite 0x02e9ef thru 0x030136 (0x1748 bytes) 000690 1787861243.803584 mapread 0x00ec80 thru 0x01addc (0xc15d bytes) 000692 1787861243.829452 read 0x00d8e3 thru 0x00fdaa (0x24c8 bytes) 000693 1787861243.833710 MIRROR_RESYNC 2 mirrors 000694 1787861244.855135 MIRROR_RESYNC 2 mirrors 000695 1787861245.230497 write 0x00680c thru 0x00d68e (0x6e83 bytes) 000696 1787861245.304891 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) 000697 1787861245.510245 write 0x009d9f thru 0x0117a1 (0x7a03 bytes) 000699 1787861245.533816 write 0x0321b4 thru 0x03ed4b (0xcb98 bytes) 000701 1787861245.556773 fallocate 0x0225f1 thru 0x032186 (0xfb96 bytes) 000702 1787861245.565163 mapwrite 0x03c8c8 thru 0x03d613 (0xd4c bytes) 000703 1787861245.622118 punch from 0x001ebd to 0x00abc5 (0x8d09 bytes) 000704 1787861245.630353 trunc from 0x01f04a to 0x02b16f (0xc126 bytes) 000705 1787861245.635480 trunc from 0x040000 to 0x02406d (0x1bf92 bytes) 000706 1787861245.655004 mapread 0x00ad97 thru 0x00d1f0 (0x245a bytes) 000708 1787861245.662111 read 0x012b10 thru 0x01d94d (0xae3e bytes) 000710 1787861245.671441 punch from 0x0178eb to 0x022f05 (0xb61b bytes) 000711 1787861245.676520 fallocate 0x03cff0 thru 0x03ffff (0x3010 bytes) 000712 1787861245.682356 trunc from 0x02406e to 0x010d91 (0x132dc bytes) 000713 1787861245.698620 fallocate 0x00ace5 thru 0x015bbb (0xaed7 bytes) 000714 1787861245.704612 trunc from 0x010d92 to 0x0272c2 (0x16531 bytes) 000715 1787861245.718787 fallocate 0x007580 thru 0x012389 (0xae0a bytes) 000717 1787861245.724629 write 0x03a008 thru 0x03ffff (0x5ff8 bytes) 000719 1787861245.726408 trunc from 0x019f3c to 0x025721 (0xb7e6 bytes) 000720 1787861245.734108 MIRROR_EXTEND to 2 mirrors 000722 1787861246.513635 mapwrite 0x01097f thru 0x013f81 (0x3603 bytes) 000723 1787861246.583626 mapread 0x03586e thru 0x03fd6a (0xa4fd bytes) 000725 1787861246.597399 mapread 0x008566 thru 0x00bf5c (0x39f7 bytes) 000726 1787861246.606127 trunc from 0x032a17 to 0x03431d (0x1907 bytes) 000727 1787861246.610704 punch from 0x011bea to 0x021b98 (0xffaf bytes) 000728 1787861246.616218 trunc from 0x02902a to 0x030b43 (0x7b1a bytes) 000730 1787861246.622405 trunc from 0x040000 to 0x00cfd8 (0x33027 bytes) 000731 1787861246.645322 trunc from 0x008e27 to 0x00cfd8 (0x41b2 bytes) 000732 1787861246.650659 mapwrite 0x03ccb5 thru 0x03ffff (0x334b bytes) 000733 1787861246.687430 punch from 0x019f21 to 0x01ecd7 (0x4db7 bytes) 000734 1787861246.693071 fallocate 0x03df4f thru 0x03ffff (0x20b1 bytes) 000735 1787861246.697686 write 0x03c72e thru 0x03ffff (0x38d2 bytes) 000736 1787861246.699351 punch from 0x01670d to 0x02425b (0xdb4f bytes) 000737 1787861246.704109 punch from 0x014d3e to 0x01984b (0x4b0e bytes) 000738 1787861246.708578 punch from 0x03052d to 0x03ffff (0xfad3 bytes) 000739 1787861246.722516 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) 000740 1787861246.853801 punch from 0x0052e1 to 0x0054f3 (0x213 bytes) 000741 1787861246.861547 write 0x03b0a0 thru 0x03ffff (0x4f60 bytes) 000742 1787861246.869839 mapwrite 0x0220ff thru 0x02f9b8 (0xd8ba bytes) 000743 1787861246.946871 trunc from 0x040000 to 0x024985 (0x1b67a bytes) 000744 1787861246.997795 fallocate 0x038602 thru 0x03ffff (0x79fe bytes) 000745 1787861247.004527 trunc from 0x024986 to 0x02e06b (0x96e6 bytes) 000747 1787861247.026292 trunc from 0x00a0a3 to 0x00f40f (0x536d bytes) 000748 1787861247.031354 trunc from 0x02e06c to 0x01b9dd (0x1268e bytes) 000749 1787861247.064354 trunc from 0x01b9de to 0x02f7e2 (0x13e05 bytes) 000753 1787861247.088160 mapwrite 0x03d7eb thru 0x03ffff (0x2815 bytes) 000754 1787861247.129592 mapwrite 0x000f2f thru 0x0051d4 (0x42a6 bytes) 000755 1787861247.156717 punch from 0x019005 to 0x024bfd (0xbbf9 bytes) 000756 1787861247.161784 mapwrite 0x02f9d3 thru 0x030852 (0xe80 bytes) 000758 1787861247.180162 punch from 0x001ad0 to 0x001be5 (0x116 bytes) 000759 1787861247.184270 mapread 0x004f5e thru 0x00f66c (0xa70f bytes) 000760 1787861247.184974 read 0x017e45 thru 0x01e1f5 (0x63b1 bytes) 000761 1787861247.191321 read 0x01cc1a thru 0x01eecf (0x22b6 bytes) 000762 1787861247.191723 mapread 0x018b08 thru 0x0242bd (0xb7b6 bytes) 000763 1787861247.198234 mapread 0x024841 thru 0x03102a (0xc7ea bytes) 000764 1787861247.198906 write 0x035c8b thru 0x038d0d (0x3083 bytes) 000765 1787861247.200553 punch from 0x00f0f0 to 0x01e60d (0xf51e bytes) 000766 1787861247.203625 read 0x00a73d thru 0x0105ed (0x5eb1 bytes) 000767 1787861247.211368 fallocate 0x03b400 thru 0x03e090 (0x2c91 bytes) 000769 1787861247.217325 punch from 0x000d2c to 0x0100e0 (0xf3b5 bytes) 000770 1787861247.222126 trunc from 0x040000 to 0x017a84 (0x2857b bytes) 000771 1787861247.233138 read 0x003c5b thru 0x0076bb (0x3a61 bytes) 000772 1787861247.241443 read 0x007d4b thru 0x00ab56 (0x2e0c bytes) 000773 1787861247.242457 punch from 0x00803f to 0x00f73a (0x76fc bytes) 000774 1787861247.246453 mapwrite 0x01af86 thru 0x01b0cf (0x14a bytes) 000775 1787861247.281761 trunc from 0x01b0d0 to 0x027609 (0xc53a bytes) 000776 1787861247.293639 read 0x015423 thru 0x018811 (0x33ef bytes) 000777 1787861247.294274 trunc from 0x018b6f to 0x027609 (0xea9b bytes) 000778 1787861247.298737 mapread 0x017a75 thru 0x026b8a (0xf116 bytes) 000779 1787861247.313778 write 0x0134d7 thru 0x01c94d (0x9477 bytes) 000780 1787861247.315113 trunc from 0x003fc9 to 0x00fd06 (0xbd3e bytes) 000782 1787861247.327581 read 0x0071b5 thru 0x011a8b (0xa8d7 bytes) 000783 1787861247.346049 trunc from 0x023ac0 to 0x0274cd (0x3a0e bytes) 000784 1787861247.351787 mapread 0x00ca64 thru 0x019463 (0xca00 bytes) 000785 1787861247.352716 mapwrite 0x00e6d1 thru 0x011b65 (0x3495 bytes) 000786 1787861247.393293 MIRROR_EXTEND to 2 mirrors 000787 1787861248.031123 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) 000788 1787861248.147390 punch from 0x0063ea to 0x00aa33 (0x464a bytes) 000789 1787861248.166281 punch from 0x0117d6 to 0x01f523 (0xdd4e bytes) 000790 1787861248.171655 write 0x02854d thru 0x03722c (0xece0 bytes) 000791 1787861248.177620 write 0x020943 thru 0x02ef17 (0xe5d5 bytes) 000792 1787861248.184045 read 0x0130b2 thru 0x01394f (0x89e bytes) 000793 1787861248.188336 mapread 0x035f2f thru 0x03722c (0x12fe bytes) 000794 1787861248.189385 trunc from 0x03722d to 0x01f70f (0x17b1d bytes) 000795 1787861248.225252 mapwrite 0x0369f9 thru 0x03ffff (0x9607 bytes) 000796 1787861248.275356 mapread 0x004c21 thru 0x011664 (0xca44 bytes) 000797 1787861248.290538 punch from 0x023969 to 0x0283de (0x4a76 bytes) 000798 1787861248.295662 trunc from 0x040000 to 0x028bc9 (0x17436 bytes) 000799 1787861248.312171 write 0x038d9c thru 0x03ffff (0x7264 bytes) 000800 1787861248.313494 read 0x023929 thru 0x030b7f (0xd257 bytes) 000801 1787861248.332315 MIRROR_EXTEND to 2 mirrors 000802 1787861249.021259 write 0x0312b8 thru 0x0324bd (0x1206 bytes) 000803 1787861249.097004 mapread 0x03c81f thru 0x03ffff (0x37e1 bytes) 000804 1787861249.132523 punch from 0x02c3f9 to 0x03ae78 (0xea80 bytes) 000805 1787861249.146896 mapread 0x00ed3e thru 0x01c864 (0xdb27 bytes) 000806 1787861249.180138 write 0x03aebe thru 0x03ffff (0x5142 bytes) 000807 1787861249.189988 fallocate 0x02d5c4 thru 0x0341cd (0x6c0a bytes) 000808 1787861249.195428 trunc from 0x03232c to 0x03e97b (0xc650 bytes) 000809 1787861249.220210 mapread 0x00e2a7 thru 0x01e215 (0xff6f bytes) 000811 1787861249.222334 read 0x006de8 thru 0x013d51 (0xcf6a bytes) 000812 1787861249.231420 read 0x020bec thru 0x02d56e (0xc983 bytes) 000813 1787861249.247233 trunc from 0x040000 to 0x020d60 (0x1f29f bytes) 000814 1787861249.305121 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20d61): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20d61): Mirror 2: 0xbf3bdee6 (stale) CRC-32 checksum value for chunk [0, 0x20d61): Mirror 1: 0xbf3bdee6 Mirror 3: 0xbf3bdee6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20d61: skipped 000815 1787861250.288299 mapread 0x015977 thru 0x01974d (0x3dd7 bytes) 000816 1787861250.306670 write 0x02e457 thru 0x03e035 (0xfbdf bytes) 000817 1787861250.339579 write 0x011326 thru 0x0191d5 (0x7eb0 bytes) 000818 1787861250.351518 fallocate 0x02829e thru 0x038106 (0xfe69 bytes) 000819 1787861250.360652 trunc from 0x03e036 to 0x0250e8 (0x18f4d bytes) 000820 1787861250.403537 trunc from 0x0091c9 to 0x012e4c (0x9c84 bytes) 000823 1787861250.424203 mapwrite 0x02c682 thru 0x03a72a (0xe0a9 bytes) 000824 1787861250.480393 trunc from 0x03a72b to 0x0052dc (0x3544e bytes) 000825 1787861250.498907 write 0x01066c thru 0x01cad0 (0xc465 bytes) 000826 1787861250.500745 fallocate 0x0165b4 thru 0x019308 (0x2d55 bytes) 000827 1787861250.505824 write 0x024297 thru 0x024b1c (0x886 bytes) 000828 1787861250.507320 mapwrite 0x03a54e thru 0x03bb84 (0x1637 bytes) 000829 1787861250.551875 MIRROR_RESYNC 3 mirrors 000830 1787861251.503371 trunc from 0x021324 to 0x02aacd (0x97aa bytes) 000831 1787861251.543775 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bb85): Mirror 2: 0xd6c8f349 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb85): Mirror 2: 0xd6c8f349 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb85: skipped 000832 1787861251.804853 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bb85): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb85): Mirror 2: 0xd6c8f349 (stale) CRC-32 checksum value for chunk [0, 0x3bb85): Mirror 1: 0xe18b559e Mirror 3: 0xe18b559e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb85: skipped 000833 1787861252.680264 mapread 0x0388ee thru 0x03bb84 (0x3297 bytes) 000834 1787861252.702608 mapread 0x00b9fe thru 0x00d380 (0x1983 bytes) 000835 1787861252.714718 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bb85): Mirror 2: 0xd6c8f349 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb85): Mirror 2: 0xd6c8f349 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb85: skipped 000836 1787861253.014637 punch from 0x025aeb to 0x02eda7 (0x92bd bytes) 000837 1787861253.045386 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bb85): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb85): Mirror 2: 0xd6c8f349 (stale) CRC-32 checksum value for chunk [0, 0x3bb85): Mirror 1: 0xe18b559e Mirror 3: 0xe18b559e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb85: skipped 000838 1787861253.895640 write 0x0325c1 thru 0x03ffff (0xda3f bytes) 000839 1787861253.947447 write 0x034584 thru 0x034615 (0x92 bytes) 000840 1787861253.948633 mapwrite 0x038abb thru 0x03ffff (0x7545 bytes) 000841 1787861253.969557 mapwrite 0x03cabf thru 0x03ffff (0x3541 bytes) 000843 1787861253.991636 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb7ba4246 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb7ba4246 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000844 1787861254.232148 mapwrite 0x01e72e thru 0x02aa88 (0xc35b bytes) 000845 1787861254.274358 write 0x03ce35 thru 0x03ffff (0x31cb bytes) 000846 1787861254.283586 punch from 0x02cc3a to 0x039e63 (0xd22a bytes) 000847 1787861254.287595 trunc from 0x040000 to 0x03817b (0x7e84 bytes) 000848 1787861254.303153 write 0x00d947 thru 0x012f19 (0x55d3 bytes) 000849 1787861254.323936 trunc from 0x02b2be to 0x0371d7 (0xbf1a bytes) 000850 1787861254.328986 punch from 0x009565 to 0x0153ea (0xbe86 bytes) 000852 1787861254.366511 MIRROR_RESYNC 2 mirrors 000853 1787861255.155298 mapwrite 0x012a1a thru 0x0229e3 (0xffca bytes) 000854 1787861255.267445 mapread 0x02aece thru 0x030a4d (0x5b80 bytes) 000855 1787861255.281804 write 0x01d435 thru 0x025ad8 (0x86a4 bytes) 000856 1787861255.288138 punch from 0x001656 to 0x00a3a6 (0x8d51 bytes) 000857 1787861255.298126 read 0x029113 thru 0x02fa16 (0x6904 bytes) 000858 1787861255.299065 write 0x020b93 thru 0x02f73d (0xebab bytes) 000859 1787861255.301415 trunc from 0x03817c to 0x02d5a8 (0xabd3 bytes) 000860 1787861255.320247 punch from 0x007a67 to 0x009c3b (0x21d5 bytes) 000861 1787861255.325611 fallocate 0x01fb6c thru 0x02e7f0 (0xec85 bytes) 000862 1787861255.329289 trunc from 0x013929 to 0x018b37 (0x520f bytes) 000863 1787861255.332500 trunc from 0x02d5a9 to 0x03f2f3 (0x11d4b bytes) 000864 1787861255.346636 trunc from 0x027ee3 to 0x02c62e (0x474c bytes) 000866 1787861255.362830 mapread 0x01d90d thru 0x0262ab (0x899f bytes) 000867 1787861255.363794 mapwrite 0x01f59d thru 0x0236e6 (0x414a bytes) 000868 1787861255.382916 mapwrite 0x0012d3 thru 0x0035ae (0x22dc bytes) 000869 1787861255.407411 trunc from 0x03f2f4 to 0x032535 (0xcdbe bytes) 000870 1787861255.425063 write 0x02ecd9 thru 0x039d77 (0xb09f bytes) 000871 1787861255.431566 mapwrite 0x03a39e thru 0x03ffff (0x5c62 bytes) 000872 1787861255.487516 mapwrite 0x01b779 thru 0x026285 (0xab0d bytes) 000873 1787861255.526307 MIRROR_RESYNC 2 mirrors 000875 1787861256.163182 trunc from 0x017d17 to 0x0249e8 (0xccd2 bytes) 000876 1787861256.205760 read 0x025f17 thru 0x034e81 (0xef6b bytes) 000877 1787861256.223377 read 0x032d40 thru 0x03a4fb (0x77bc bytes) 000878 1787861256.234592 write 0x023e9b thru 0x02b0d5 (0x723b bytes) 000879 1787861256.236083 trunc from 0x0249bb to 0x02a7b4 (0x5dfa bytes) 000880 1787861256.270200 read 0x03dec8 thru 0x03ffff (0x2138 bytes) 000881 1787861256.270773 mapwrite 0x03d6ff thru 0x03ffff (0x2901 bytes) 000882 1787861256.287564 mapread 0x036ff0 thru 0x03f982 (0x8993 bytes) 000883 1787861256.292788 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) 000884 1787861256.444726 write 0x0220bc thru 0x02bf3f (0x9e84 bytes) 000887 1787861256.459651 write 0x017403 thru 0x01bcad (0x48ab bytes) 000889 1787861256.469467 write 0x021def thru 0x028d6e (0x6f80 bytes) 000890 1787861256.475158 trunc from 0x027d53 to 0x037351 (0xf5ff bytes) 000892 1787861256.489091 fallocate 0x0370eb thru 0x03ffff (0x8f15 bytes) 000893 1787861256.493345 read 0x00f925 thru 0x0123fb (0x2ad7 bytes) 000894 1787861256.499201 read 0x02c4fc thru 0x033869 (0x736e bytes) 000895 1787861256.508942 trunc from 0x040000 to 0x03e9ff (0x1600 bytes) 000896 1787861256.542931 trunc from 0x03ea00 to 0x004486 (0x3a579 bytes) 000897 1787861256.554781 trunc from 0x00106f to 0x004486 (0x3418 bytes) 000899 1787861256.560318 trunc from 0x004487 to 0x013b14 (0xf68e bytes) 000900 1787861256.571863 mapwrite 0x02cc71 thru 0x03738d (0xa71d bytes) 000902 1787861256.645506 fallocate 0x02589f thru 0x025c55 (0x3b7 bytes) 000905 1787861256.651126 mapwrite 0x02cfd4 thru 0x031b4b (0x4b78 bytes) 000906 1787861256.664782 punch from 0x033d70 to 0x036f98 (0x3229 bytes) 000907 1787861256.668996 trunc from 0x00ad17 to 0x014112 (0x93fc bytes) 000909 1787861256.674440 mapread 0x01f3d5 thru 0x022501 (0x312d bytes) 000910 1787861256.695126 mapwrite 0x035387 thru 0x03fddc (0xaa56 bytes) 000911 1787861256.786119 mapwrite 0x015afe thru 0x019326 (0x3829 bytes) 000913 1787861256.823497 trunc from 0x03fddd to 0x03e5c6 (0x1816 bytes) 000916 1787861256.840509 punch from 0x0004f5 to 0x00a13c (0x9c48 bytes) 000917 1787861256.857917 MIRROR_EXTEND to 2 mirrors 000919 1787861257.579087 write 0x00406d thru 0x01101a (0xcfae bytes) 000920 1787861257.634496 read 0x00e598 thru 0x014044 (0x5aad bytes) 000921 1787861257.640426 trunc from 0x00ec35 to 0x0132f0 (0x46bc bytes) 000922 1787861257.654966 punch from 0x006527 to 0x00f244 (0x8d1e bytes) 000923 1787861257.659576 mapwrite 0x018ad8 thru 0x02724c (0xe775 bytes) 000924 1787861257.702452 trunc from 0x00d97c to 0x019337 (0xb9bc bytes) 000925 1787861257.708041 trunc from 0x03e5c7 to 0x01c5ed (0x21fd9 bytes) 000926 1787861257.722542 trunc from 0x00034f to 0x00f3b2 (0xf064 bytes) 000927 1787861257.727287 trunc from 0x01c5ee to 0x028cdd (0xc6f0 bytes) 000929 1787861257.745272 mapread 0x01780d thru 0x019782 (0x1f76 bytes) 000930 1787861257.764523 trunc from 0x028cde to 0x03d0f2 (0x14415 bytes) 000931 1787861257.812927 mapwrite 0x0231ce thru 0x02537f (0x21b2 bytes) 000932 1787861257.837296 mapwrite 0x0191ee thru 0x01d4f4 (0x4307 bytes) 000933 1787861257.861214 punch from 0x016d7a to 0x01888a (0x1b11 bytes) 000934 1787861257.866531 fallocate 0x016c07 thru 0x022ad2 (0xbecc bytes) 000935 1787861257.872082 fallocate 0x0361ee thru 0x03ffff (0x9e12 bytes) 000936 1787861257.878211 punch from 0x0283ef to 0x02c8fc (0x450e bytes) 000937 1787861257.888171 mapread 0x00cb74 thru 0x01a640 (0xdacd bytes) 000941 1787861257.912109 trunc from 0x03d0f3 to 0x0206f8 (0x1c9fa bytes) 000942 1787861257.926231 trunc from 0x016921 to 0x01d111 (0x67f1 bytes) 000943 1787861257.934124 trunc from 0x0206f9 to 0x0298fc (0x9204 bytes) 000944 1787861257.959721 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) 000945 1787861258.124548 mapwrite 0x00437e thru 0x005f36 (0x1bb9 bytes) 000946 1787861258.155211 MIRROR_EXTEND to 2 mirrors 000947 1787861258.891830 write 0x00b83a thru 0x01336e (0x7b35 bytes) 000948 1787861258.978604 read 0x01f30c thru 0x0298fc (0xa5f1 bytes) 000949 1787861258.989802 mapread 0x015af6 thru 0x019a72 (0x3f7d bytes) 000951 1787861259.016082 punch from 0x0291b3 to 0x0298fc (0x74a bytes) 000952 1787861259.026146 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x298fd): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x298fd): Mirror 2: 0xc6cd1e6d (stale) CRC-32 checksum value for chunk [0, 0x298fd): Mirror 1: 0x838242ad Mirror 3: 0x838242ad lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x298fd: skipped 000953 1787861260.141244 punch from 0x0112de to 0x014968 (0x368b bytes) 000954 1787861260.199283 MIRROR_RESYNC 3 mirrors 000956 1787861261.198434 punch from 0x024925 to 0x0298fc (0x4fd8 bytes) 000957 1787861261.235707 write 0x007e13 thru 0x00a0f0 (0x22de bytes) 000959 1787861261.245307 read 0x007c35 thru 0x012fa8 (0xb374 bytes) 000960 1787861261.257430 trunc from 0x00b9af to 0x018be8 (0xd23a bytes) 000961 1787861261.262700 trunc from 0x0298fd to 0x015fad (0x1394f bytes) 000962 1787861261.279921 trunc from 0x013e0c to 0x0145b8 (0x7ad bytes) 000964 1787861261.285329 mapwrite 0x03a099 thru 0x03ffff (0x5f67 bytes) 000965 1787861261.335955 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1a3f9623 (stale) Mirror 3: 0x1a3f9623 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x51a89e90 Mirror 4: 0x51a89e90 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000966 1787861262.024933 trunc from 0x040000 to 0x0190e3 (0x26f1c bytes) 000967 1787861262.099702 fallocate 0x00d02a thru 0x01a913 (0xd8ea bytes) 000968 1787861262.106330 mapwrite 0x020ead thru 0x0238ec (0x2a40 bytes) 000969 1787861262.157001 mapread 0x01a052 thru 0x0203c5 (0x6374 bytes) 000970 1787861262.165791 mapread 0x009fb2 thru 0x00b537 (0x1586 bytes) 000971 1787861262.173206 mapread 0x01b9b8 thru 0x0238ec (0x7f35 bytes) 000972 1787861262.174431 punch from 0x004863 to 0x005c61 (0x13ff bytes) 000973 1787861262.182335 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x238ed): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x238ed): Mirror 2: 0x224fe5d9 (stale) Mirror 3: 0x224fe5d9 (stale) Mirror 4: 0x164209dc (stale) CRC-32 checksum value for chunk [0, 0x238ed): Mirror 1: 0x8d318f59 Mirror 5: 0x8d318f59 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x238ed: skipped 000974 1787861263.182790 punch from 0x01756b to 0x01d9c9 (0x645f bytes) 000975 1787861263.233240 trunc from 0x0238ed to 0x005668 (0x1e284 bytes) 000976 1787861263.287300 trunc from 0x005669 to 0x036272 (0x30c0a bytes) 000977 1787861263.329186 fallocate 0x03f4c1 thru 0x03ffff (0xb3f bytes) 000978 1787861263.341437 read 0x008127 thru 0x013c4f (0xbb29 bytes) 000979 1787861263.351893 punch from 0x007079 to 0x016eaa (0xfe32 bytes) 000981 1787861263.360261 read 0x03b2ba thru 0x03ffff (0x4d46 bytes) 000984 1787861263.391721 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x1ae:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1a3f9623 (stale) Mirror 4: 0x51a89e90 (stale) Mirror 5: 0x7a47d05c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1a3f9623 (stale) Mirror 4: 0x51a89e90 (stale) Mirror 5: 0x7a47d05c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000985 1787861263.822897 trunc from 0x03f278 to 0x03ffff (0xd88 bytes) 000987 1787861263.834296 trunc from 0x0322ea to 0x03376a (0x1481 bytes) 000988 1787861263.840252 read 0x00e4de thru 0x01c688 (0xe1ab bytes) 000989 1787861263.849187 trunc from 0x02bf2d to 0x02cee7 (0xfbb bytes) 000990 1787861263.853176 fallocate 0x021f2a thru 0x02e461 (0xc538 bytes) 000991 1787861263.858070 mapwrite 0x006ba6 thru 0x00a7f0 (0x3c4b bytes) 000992 1787861263.892346 trunc from 0x002fb7 to 0x010ae8 (0xdb32 bytes) 000993 1787861263.896100 fallocate 0x02d484 thru 0x02fd05 (0x2882 bytes) 000994 1787861263.902565 MIRROR_RESYNC 4 mirrors 000995 1787861264.697944 fallocate 0x02f2a4 thru 0x037bad (0x890a bytes) 000996 1787861264.731870 write 0x025ab9 thru 0x031a37 (0xbf7f bytes) 000997 1787861264.753624 MIRROR_RESYNC 4 mirrors 000998 1787861266.033870 mapread 0x019348 thru 0x019a81 (0x73a bytes) 000999 1787861266.041779 trunc from 0x01cda0 to 0x021bbc (0x4e1d bytes) 001000 1787861266.084733 fallocate 0x03a85a thru 0x03ffff (0x57a6 bytes) All operations completed A-OK! PASS 70a (94s) == sanity-flr test 71: check mirror extend parent fid ==== 16:07:51 (1787861271) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x1f6:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (11s) == sanity-flr test 72: check append on FLR file ========== 16:08:02 (1787861282) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xf5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.216449 s, 9.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xf5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14293 s, 7.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xf6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xf5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xf6:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xf5:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 16:08:13 (1787861293) Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre2 Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 8199776 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1892704 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2204160 .. Extending file size to 2233984 .. Extending file size to 8598464 .. Extending file size to 7629312 .. Extending file size to 8062624 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8766912 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9108544 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9112448 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62843 62844 62845 62848 62850 done 192.168.201.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg124-client.virtnet /mnt/lustre2 (opts:) 192.168.201.124@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg124-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:09:37 (1787861377) Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre2 Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8240896 .. Extending file size to 7513760 .. Extending file size to 8713376 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8958368 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8349024 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8581312 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8846944 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 64573 64574 64575 64576 64580 failed 192.168.201.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg124-client.virtnet /mnt/lustre2 (opts:) 192.168.201.124@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg124-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:10:54 (1787861454) Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2320 1268984 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2240 1269064 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18596 3588424 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24296 3582724 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42892 7171148 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2320 1268984 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2240 1269064 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18596 3588424 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24296 3582724 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42892 7171148 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.102209 s, 20.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.159009 s, 13.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xa574b09c (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.201.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg124-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:11:13 (1787861473) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105266 s, 19.9 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 16:11:22 (1787861482) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x62:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:11:30 (1787861490) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.105477 s, 19.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x263:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x265:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x266:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:11:39 (1787861499) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0990269 s, 21.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x135:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:11:49 (1787861509) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121112 s, 17.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x136:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x136:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x137:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:12:23 (1787861543) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg124-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0952318 s, 11.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x264:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x268:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x267:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x264:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x268:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x267:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x265:0x0] } PASS 204d (47s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:13:10 (1787861590) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg124-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x118:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13073 s, 16.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x118:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x118:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x138:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (52s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:14:02 (1787861642) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg124-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg124-server: Pool lustre.test_204f created Adding targets to pool oleg124-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg124-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.14821 s, 14.1 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x119:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x139:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x119:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x139:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg124-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg124-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg124-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:15:20 (1787861720) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:15:29 (1787861729) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:15:38 (1787861738) oleg124-server: Pool lustre.test_206 created Adding targets to pool oleg124-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg124-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg124-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg124-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg124-server: Pool lustre.test_206 destroyed PASS 206 (29s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:16:06 (1787861766) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110689 s, 9.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031739 s, 33.0 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x2473adf3 Mirror 3: 0x2473adf3 PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:16:20 (1787861780) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:16:25 (1787861785) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:16:29 (1787861789) Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0853475 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2650000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1d70000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1d70000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1d70000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2300000 Read this (4 bytes): BBBB 192.168.201.124@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg124-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:17:08 (1787861828) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0179852 s, 455 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:17:15 (1787861835) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103355 s, 10.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x144:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611267 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 16:17:17.000000000 -0400 Modify: 2026-08-27 16:17:17.000000000 -0400 Change: 2026-08-27 16:17:18.000000000 -0400 Birth: 2026-08-27 16:17:17.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:17:23 (1787861843) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482554 s, 21.7 MB/s oleg124-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg124-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg124-server: O/280000401/d3/123: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:17:45 (1787861865) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.268505 s, 153 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.041195 s, 99.4 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 16:17 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x88:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 16:17 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:17:53 (1787861873) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:18:00 (1787861880) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1859 sec =========== 16:18:09 (1787861889) osc.lustre-OST0000-osc-ffff9b11128eb800.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0001-osc-ffff9b11128eb800.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 obdfilter.lustre-OST0000.stats_compr= snapshot_time 1787861893.602606976 secs.nsecs start_time 1787860519.089770407 secs.nsecs elapsed_time 1374.512836569 secs.nsecs uncompressed_objects 2902 samples [reqs] obdfilter.lustre-OST0001.stats_compr= snapshot_time 1787861893.607405088 secs.nsecs start_time 1787860560.581133518 secs.nsecs elapsed_time 1333.026271570 secs.nsecs uncompressed_objects 2935 samples [reqs] === sanity-flr: start cleanup 16:18:13 (1787861893) === === sanity-flr: finish cleanup 16:18:27 (1787861907) ===