-----============= acceptance-small: sanity-flr ============----- Thu Aug 27 12:14:16 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg222-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 12:14:36 (1787847276) === oleg222-client.virtnet: executing check_config_client /mnt/lustre oleg222-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg222-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff971bd0e8b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff971bd0e8b000.idle_timeout=debug disable quota as required oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 12:15:22 (1787847322) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8256] == sanity-flr test 0a: lfs mirror create with -N option == 12:15:24 (1787847324) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:15:40 (1787847340) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:15:44 (1787847344) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:15:48 (1787847348) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:16:14 (1787847374) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:16:18 (1787847378) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 12:16:21 (1787847381) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff971bd0e8b000.stats=clear osc.lustre-OST0001-osc-ffff971bd0e8b000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:16:31 (1787847391) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:16:42 (1787847402) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:16:50 (1787847410) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 12:16:54 (1787847414) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:17:03 (1787847423) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 12:17:12 (1787847432) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:17:20 (1787847440) PASS 3 (7s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:17:27 (1787847447) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:17:34 (1787847454) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (6s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:17:45 (1787847465) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:17:51 (1787847471) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.216056 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156596 s, 6.7 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:18:00 (1787847480) 19+0 records in 19+0 records out 19922944 bytes (20 MB, 19 MiB) copied, 0.660331 s, 30.2 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 12:18:09 (1787847489) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:18:20 (1787847500) Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff971bd0e8b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff971bd0e8b000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff971bd0e8b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff971bd0e8b000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg222-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (75s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:19:34 (1787847574) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.323887 s, 16.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff971bd0e8b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff971bd0e8b000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (52s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:20:26 (1787847626) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff971bd0e8b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff971bd0e8b000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff971bd0e8b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff971bd0e8b000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg222-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (78s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:21:44 (1787847704) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.10479 s, 16.9 MB/s -rw-r--r-- 1 root root 52428800 Aug 27 12:21 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.934s user 0m0.000s sys 0m0.155s Stopping /mnt/lustre-ost1 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff971bd0e8b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff971bd0e8b000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.190s user 0m0.002s sys 0m0.160s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg222-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg222-server oleg222-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff971bd0e8b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff971bd0e8b000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.360s user 0m0.001s sys 0m0.172s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg222-server: oleg222-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg222-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg222-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff971bd0e8b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff971bd0e8b000.ost_server_uuid in FULL state after 0 sec PASS 33b (101s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:23:25 (1787847805) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:23:29 (1787847809) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:23:34 (1787847814) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 12:23:37 (1787847817) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 12:23:45 (1787847825) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.590322 s, 21.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:23:59 (1787847839) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322298 s, 32.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320869 s, 32.7 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:24:07 (1787847847) PASS 36c (10s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:24:17 (1787847857) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00111682 s, 917 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00213899 s, 479 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000922393 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0025507 s, 401 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 12:24:29 (1787847869) create test files with size 15732535 .. efd62013dda8127feb94ef1a5815a0a6 - 450969a3849e6249d340181fff38f935 - f0791ece4f83d40e83bee0796feed9a9 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 12:24:55 (1787847895) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:25:10 (1787847910) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:25:18 (1787847918) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153871 s, 6.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbf:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491041 s, 21.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 12:25:28 (1787847928) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0596149 s, 35.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.183938 s, 22.8 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.14854 s, 14.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (21s) == sanity-flr test 42: lfs mirror verify ================= 12:25:49 (1787847949) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 12:25:53 (1787847953) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:25:56 (1787847956) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0435824 s, 24.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 43c: read prefer uncompressed mirror == 12:26:06 (1787847966) llite.lustre-ffff971bd2b1b800.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (9s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 12:26:15 (1787847975) llite.lustre-ffff971bd2b1b800.enable_compression=1 striped dir -i1 -c2 -H crush /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242106 s, 43.3 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342899 s, 30.6 MB/s **verify components PASS 43d (9s) == sanity-flr test 44a: lfs mirror split check =========== 12:26:24 (1787847984) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.25946 s, 12.1 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:26:35 (1787847995) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcd:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:26:46 (1787848006) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.455597 s, 23.0 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0xa:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (15s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:27:01 (1787848021) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff971bc5d93000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.420475 s, 24.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg222-client.virtnet /mnt/lustre (opts:) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff971bc9b99000.xattr_cache=1 PASS 44e (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:27:20 (1787848040) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (46s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:28:06 (1787848086) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (50s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:28:56 (1787848136) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 12:29:00 (1787848140) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.306316 s, 10.3 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.144454 s, 21.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x639adafd (nosync,stale) Mirror 2: 0x639adafd (nosync,stale) ** verify mirror 2 doesn't change original checksum: 6654c39666d38bb7518bb858cbebb4c4 - mirror 1 checksum: dbba38bda5fae823157bff8cd5af7dce - mirror 2 checksum: 6654c39666d38bb7518bb858cbebb4c4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T12:29:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-27T12:29:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6654c39666d38bb7518bb858cbebb4c4 - mirror 1 checksum: dbba38bda5fae823157bff8cd5af7dce - mirror 2 checksum: dbba38bda5fae823157bff8cd5af7dce - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T12:29:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-27T12:29:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 48 (14s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 12:29:14 (1787848154) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359873 s, 29.1 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:29:23 (1787848163) Failing mds1 on oleg222-server Stopping /mnt/lustre-mds1 (opts:) on oleg222-server 12:29:37 (1787848177) shut down facet: mds1 facet_host: oleg222-server facet_failover_host: oleg222-server Failover mds1 to oleg222-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Started lustre-MDT0000 12:30:13 (1787848213) targets are mounted 12:30:13 (1787848213) facet_failover done oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037717 s, 27.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } PASS 50A (67s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:30:30 (1787848230) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0530487 s, 386 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0220242 s, 930 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (13s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:30:43 (1787848243) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0389156 s, 526 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00991149 s, 2.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (10s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:30:53 (1787848253) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0153929 s, 1.1 MB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:31:03 (1787848263) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0757023 s, 271 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:31:14 (1787848274) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00810609 s, 505 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0309545 s, 132 kB/s PASS 60a (11s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:31:24 (1787848284) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (27s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:31:51 (1787848311) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (34s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:32:25 (1787848345) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:32:50 (1787848370) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (28s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:33:18 (1787848398) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 12:33:29 (1787848409) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 58893 mirror split pid 58895 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58893 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 58895 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (70s) == sanity-flr test 70a: flr mode fsx test ================ 12:34:39 (1787848479) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 8482 truncating to largest ever: 0x1dcbe 000001 1787848483.072140 trunc from 00000000 to 0x01dcbd (0x1dcbe bytes) 000003 1787848483.093237 trunc from 0x01dcbe to 0x003991 (0x1a32c bytes) 000004 1787848483.108176 punch from 0x002107 to 0x003991 (0x188b bytes) 000005 1787848483.113913 punch from 0x001974 to 0x003991 (0x201e bytes) 000006 1787848483.121515 punch from 0x000a20 to 0x003991 (0x2f72 bytes) 000008 1787848483.136464 trunc from 0x003992 to 0x017f29 (0x14598 bytes) truncating to largest ever: 0x20589 000009 1787848483.160280 trunc from 0x017f2a to 0x020588 (0x865f bytes) 000010 1787848483.189046 write 0x000a7b thru 0x00a5c1 (0x9b47 bytes) 000011 1787848483.204070 punch from 0x0013cc to 0x001c42 (0x877 bytes) 000012 1787848483.222125 write 0x036a9c thru 0x03ffff (0x9564 bytes) 000013 1787848483.225184 mapread 0x000f0a thru 0x00349f (0x2596 bytes) 000014 1787848483.237462 MIRROR_EXTEND to 2 mirrors fallocating to largest ever: 0x40000 000015 1787848484.231843 fallocate 0x034998 thru 0x03ffff (0xb668 bytes) 000016 1787848484.282099 mapread 0x0007af thru 0x0056d8 (0x4f2a bytes) 000017 1787848484.319819 read 0x009481 thru 0x013d6c (0xa8ec bytes) 000018 1787848484.327135 read 0x01b83c thru 0x0260a3 (0xa868 bytes) 000019 1787848484.348064 fallocate 0x001235 thru 0x00bbc3 (0xa98f bytes) 000020 1787848484.353940 mapread 0x006f5e thru 0x012eb2 (0xbf55 bytes) 000021 1787848484.355064 fallocate 0x03570e thru 0x038c4f (0x3542 bytes) 000022 1787848484.363048 mapwrite 0x02a798 thru 0x02f728 (0x4f91 bytes) 000023 1787848484.392187 mapwrite 0x0262a1 thru 0x0335ac (0xd30c bytes) 000024 1787848484.429810 read 0x00358b thru 0x00a236 (0x6cac bytes) 000025 1787848484.430807 read 0x02e854 thru 0x03c15d (0xd90a bytes) 000026 1787848484.431498 punch from 0x02d0e0 to 0x036508 (0x9429 bytes) 000027 1787848484.437880 read 0x035686 thru 0x03ffff (0xa97a bytes) 000028 1787848484.454012 MIRROR_RESYNC 2 mirrors 000029 1787848485.597336 mapread 0x029e4f thru 0x02d662 (0x3814 bytes) 000030 1787848485.622368 read 0x008054 thru 0x0179d2 (0xf97f bytes) 000031 1787848485.645258 read 0x00d00f thru 0x00e1c6 (0x11b8 bytes) 000032 1787848485.646407 trunc from 0x040000 to 0x0054f4 (0x3ab0b bytes) 000033 1787848485.719620 punch from 0x0053bd to 0x0054f4 (0x138 bytes) 000034 1787848485.737449 read 0x005198 thru 0x0054f4 (0x35d bytes) 000035 1787848485.751715 mapwrite 0x027785 thru 0x02c753 (0x4fcf bytes) 000037 1787848485.846958 read 0x00c6d7 thru 0x017116 (0xaa40 bytes) 000038 1787848485.852408 trunc from 0x020b72 to 0x02c753 (0xbbe2 bytes) 000039 1787848485.862640 mapwrite 0x02f188 thru 0x035ff2 (0x6e6b bytes) 000040 1787848485.938835 trunc from 0x027be6 to 0x035ff2 (0xe40d bytes) 000041 1787848485.952408 fallocate 0x016ca0 thru 0x01e276 (0x75d7 bytes) 000042 1787848485.971122 trunc from 0x035ff3 to 0x02c250 (0x9da2 bytes) 000043 1787848485.993857 punch from 0x019ec2 to 0x023771 (0x98b0 bytes) 000044 1787848486.011839 write 0x0041c5 thru 0x00f6eb (0xb527 bytes) 000045 1787848486.023325 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2c251): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c251): Mirror 2: 0x455dffc4 (stale) CRC-32 checksum value for chunk [0, 0x2c251): Mirror 1: 0xc924d651 Mirror 3: 0xc924d651 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c251: skipped 000046 1787848487.298084 mapread 0x002da9 thru 0x00763a (0x4892 bytes) 000047 1787848487.328271 MIRROR_RESYNC 3 mirrors 000048 1787848488.165682 write 0x001d29 thru 0x005bdb (0x3eb3 bytes) 000049 1787848488.220663 trunc from 0x01d8d9 to 0x01e709 (0xe31 bytes) 000050 1787848488.226735 punch from 0x0074b5 to 0x008289 (0xdd5 bytes) 000051 1787848488.233182 read 0x01f6e0 thru 0x027475 (0x7d96 bytes) 000052 1787848488.246149 read 0x0174bd thru 0x01f03b (0x7b7f bytes) 000053 1787848488.258810 trunc from 0x02c251 to 0x033186 (0x6f36 bytes) 000054 1787848488.291376 punch from 0x0179c2 to 0x026895 (0xeed4 bytes) 000055 1787848488.306040 write 0x02b4ff thru 0x02fa9f (0x45a1 bytes) 000056 1787848488.318236 write 0x02b5a9 thru 0x037455 (0xbead bytes) 000057 1787848488.321268 read 0x009e6e thru 0x00a3c4 (0x557 bytes) 000058 1787848488.322882 trunc from 0x037456 to 0x039680 (0x222b bytes) 000059 1787848488.342983 trunc from 0x039681 to 0x033234 (0x644c bytes) 000060 1787848488.367801 MIRROR_RESYNC 3 mirrors 000061 1787848489.232991 write 0x01d681 thru 0x027695 (0xa015 bytes) 000062 1787848489.317325 punch from 0x008d2f to 0x00d1ce (0x44a0 bytes) 000063 1787848489.326520 fallocate 0x03be23 thru 0x03ffff (0x41dd bytes) 000064 1787848489.339752 fallocate 0x01898f thru 0x026aec (0xe15e bytes) 000065 1787848489.347877 read 0x018730 thru 0x01f615 (0x6ee6 bytes) 000066 1787848489.359110 write 0x01e373 thru 0x02e002 (0xfc90 bytes) 000067 1787848489.380018 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6f85bdd2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6f85bdd2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000068 1787848490.015837 fallocate 0x030f8c thru 0x034171 (0x31e6 bytes) 000069 1787848490.038917 write 0x007e48 thru 0x00b379 (0x3532 bytes) 000070 1787848490.082649 read 0x01f83a thru 0x022dbc (0x3583 bytes) 000071 1787848490.099734 read 0x025bf4 thru 0x033187 (0xd594 bytes) 000072 1787848490.131694 fallocate 0x03d675 thru 0x03ffff (0x298b bytes) 000073 1787848490.152101 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000074 1787848490.375234 trunc from 0x013fa9 to 0x01903b (0x5093 bytes) 000075 1787848490.382089 trunc from 0x02ad7d to 0x02b85c (0xae0 bytes) 000076 1787848490.389374 write 0x029cb4 thru 0x02a194 (0x4e1 bytes) 000077 1787848490.406540 fallocate 0x02a15a thru 0x037226 (0xd0cd bytes) 000079 1787848490.412411 mapwrite 0x010a9e thru 0x01e23b (0xd79e bytes) 000080 1787848490.470037 fallocate 0x03f0ee thru 0x03ffff (0xf12 bytes) 000081 1787848490.474951 trunc from 0x040000 to 0x0123be (0x2dc41 bytes) 000082 1787848490.518825 write 0x03c5e6 thru 0x03ffff (0x3a1a bytes) 000083 1787848490.523037 trunc from 0x03f59f to 0x03ffff (0xa61 bytes) 000084 1787848490.536756 mapread 0x027e41 thru 0x0354a0 (0xd660 bytes) 000085 1787848490.570350 trunc from 0x040000 to 0x024afa (0x1b505 bytes) 000087 1787848490.598134 mapread 0x01826c thru 0x024afa (0xc88f bytes) 000092 1787848490.624170 read 0x0064a6 thru 0x01550e (0xf069 bytes) 000093 1787848490.654935 read 0x0011ad thru 0x009485 (0x82d9 bytes) 000094 1787848490.674938 trunc from 0x024afb to 0x014c65 (0xfe95 bytes) 000095 1787848490.694906 mapwrite 0x01432e thru 0x01c693 (0x8366 bytes) 000096 1787848490.818481 write 0x02014b thru 0x02e79d (0xe653 bytes) 000097 1787848490.823705 punch from 0x025273 to 0x02585e (0x5ec bytes) 000098 1787848490.852101 read 0x0145b6 thru 0x01b251 (0x6c9c bytes) 000099 1787848490.852970 trunc from 0x02e79e to 0x02bafc (0x2ca1 bytes) 000100 1787848490.881636 fallocate 0x030d09 thru 0x03d2c5 (0xc5bd bytes) 000101 1787848490.893752 write 0x037653 thru 0x038045 (0x9f3 bytes) 000102 1787848490.896817 read 0x018034 thru 0x02734a (0xf317 bytes) 000104 1787848490.917792 trunc from 0x038046 to 0x00c45a (0x2bbeb bytes) 000105 1787848490.967437 trunc from 0x00a20b to 0x00c45a (0x2250 bytes) 000106 1787848490.975110 write 0x022fbe thru 0x023105 (0x148 bytes) 000107 1787848490.976892 trunc from 0x0144e5 to 0x021c1a (0xd736 bytes) 000109 1787848490.986115 read 0x01b0af thru 0x022be5 (0x7b37 bytes) 000110 1787848491.050624 mapwrite 0x0048c2 thru 0x0120a8 (0xd7e7 bytes) 000112 1787848491.085225 read 0x01d32b thru 0x023105 (0x5ddb bytes) 000113 1787848491.088624 write 0x037e07 thru 0x03f04f (0x7249 bytes) 000114 1787848491.096088 read 0x00f41b thru 0x01246b (0x3051 bytes) 000115 1787848491.098023 mapwrite 0x01b71c thru 0x027af8 (0xc3dd bytes) 000116 1787848491.158804 mapwrite 0x01a018 thru 0x022197 (0x8180 bytes) 000118 1787848491.238420 trunc from 0x00bc7e to 0x015b85 (0x9f08 bytes) 000119 1787848491.249106 trunc from 0x03f050 to 0x01c11c (0x22f33 bytes) 000120 1787848491.295788 trunc from 0x01c11d to 0x00176d (0x1a9af bytes) 000121 1787848491.350037 MIRROR_EXTEND to 2 mirrors 000122 1787848491.942202 punch from 0x000a0b to 0x00176d (0xd63 bytes) 000123 1787848492.034928 mapwrite 0x00b093 thru 0x010913 (0x5881 bytes) 000125 1787848492.161949 trunc from 0x00613a to 0x006320 (0x1e7 bytes) 000126 1787848492.168033 trunc from 0x001771 to 0x001867 (0xf7 bytes) 000127 1787848492.177727 write 0x03b642 thru 0x03ffff (0x49be bytes) 000128 1787848492.179894 trunc from 0x040000 to 0x02ca5b (0x135a4 bytes) 000130 1787848492.202235 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000131 1787848492.440163 punch from 0x01f211 to 0x027a7a (0x886a bytes) 000132 1787848492.449385 punch from 0x00e300 to 0x014d26 (0x6a27 bytes) 000133 1787848492.464870 mapwrite 0x01633e thru 0x018b2e (0x27f1 bytes) 000134 1787848492.530597 trunc from 0x02ca5c to 0x02454e (0x850d bytes) 000135 1787848492.577625 trunc from 0x018634 to 0x01a560 (0x1f2d bytes) 000136 1787848492.583852 trunc from 0x02454f to 0x02aa12 (0x64c4 bytes) 000137 1787848492.622220 punch from 0x01ad8c to 0x01d696 (0x290b bytes) 000138 1787848492.635489 mapwrite 0x00baad thru 0x0123b7 (0x690b bytes) 000139 1787848492.717489 trunc from 0x02aa13 to 0x0175a9 (0x13469 bytes) 000140 1787848492.756951 mapread 0x0064a4 thru 0x0085ae (0x210b bytes) 000141 1787848492.770971 write 0x004dae thru 0x0132c0 (0xe513 bytes) 000142 1787848492.780279 trunc from 0x0175aa to 0x023d22 (0xc779 bytes) 000143 1787848492.848390 trunc from 0x000099 to 0x002981 (0x28e9 bytes) 000144 1787848492.861154 read 0x021659 thru 0x023d22 (0x26ca bytes) 000146 1787848492.869747 trunc from 0x023d23 to 0x003259 (0x20ac9 bytes) 000148 1787848492.888415 read 0x000a93 thru 0x003259 (0x27c7 bytes) 000149 1787848492.896269 trunc from 0x00325a to 0x0390aa (0x35e51 bytes) 000151 1787848492.920500 trunc from 0x0330bd to 0x0390aa (0x5fee bytes) 000152 1787848492.926508 punch from 0x02e6d7 to 0x038b60 (0xa48a bytes) 000153 1787848492.934873 trunc from 0x0390ab to 0x019830 (0x1f87a bytes) 000154 1787848492.982847 trunc from 0x019831 to 0x006b8b (0x12ca5 bytes) 000155 1787848493.016054 punch from 0x0007fc to 0x005586 (0x4d8b bytes) 000156 1787848493.024681 read 0x001007 thru 0x006b8b (0x5b85 bytes) 000157 1787848493.047099 punch from 0x001196 to 0x00510d (0x3f78 bytes) 000158 1787848493.053617 write 0x007453 thru 0x00f04c (0x7bfa bytes) 000159 1787848493.056699 write 0x03d2eb thru 0x03ffff (0x2d15 bytes) 000160 1787848493.058178 trunc from 0x010a91 to 0x012a25 (0x1f95 bytes) 000161 1787848493.064938 trunc from 0x040000 to 0x002d05 (0x3d2fa bytes) 000162 1787848493.082040 fallocate 0x0305b9 thru 0x03e8f3 (0xe33b bytes) 000163 1787848493.087599 trunc from 0x03e8f4 to 0x01594e (0x28fa5 bytes) 000164 1787848493.107494 mapwrite 0x017d33 thru 0x01d347 (0x5615 bytes) 000165 1787848493.168888 write 0x03168a thru 0x032032 (0x9a9 bytes) 000166 1787848493.170214 mapwrite 0x036f41 thru 0x03fd49 (0x8e09 bytes) 000167 1787848493.262288 write 0x02baea thru 0x036c12 (0xb129 bytes) 000168 1787848493.263808 punch from 0x03e2de to 0x03f8ba (0x15dd bytes) 000171 1787848493.274691 fallocate 0x0314d5 thru 0x03985f (0x838b bytes) 000172 1787848493.283061 punch from 0x0039bc to 0x00bd41 (0x8386 bytes) 000173 1787848493.311190 fallocate 0x006251 thru 0x00bccd (0x5a7d bytes) 000174 1787848493.330303 trunc from 0x013c29 to 0x023a54 (0xfe2c bytes) 000175 1787848493.344017 write 0x02583f thru 0x02ec83 (0x9445 bytes) 000176 1787848493.348686 trunc from 0x03fd4a to 0x03e195 (0x1bb4 bytes) 000177 1787848493.386725 mapwrite 0x008f04 thru 0x00d32a (0x4427 bytes) 000178 1787848493.479118 read 0x02751f thru 0x0356f4 (0xe1d6 bytes) 000179 1787848493.483309 fallocate 0x024cef thru 0x02f356 (0xa668 bytes) 000180 1787848493.502837 punch from 0x02bf35 to 0x030810 (0x48dc bytes) 000181 1787848493.512640 write 0x035365 thru 0x03ffff (0xac9b bytes) 000182 1787848493.513883 write 0x00b667 thru 0x0140e0 (0x8a7a bytes) 000183 1787848493.526720 trunc from 0x040000 to 0x00758c (0x38a73 bytes) 000184 1787848493.589350 mapwrite 0x0381d1 thru 0x03f6d4 (0x7504 bytes) 000185 1787848493.720457 fallocate 0x03a4c5 thru 0x03d269 (0x2da5 bytes) 000186 1787848493.726836 trunc from 0x0356e5 to 0x03f6d4 (0x9ff0 bytes) 000189 1787848493.739696 punch from 0x01a90a to 0x02767b (0xcd72 bytes) 000191 1787848493.746664 write 0x01b580 thru 0x02769a (0xc11b bytes) 000192 1787848493.771970 MIRROR_EXTEND to 2 mirrors 000193 1787848494.724685 trunc from 0x023865 to 0x02a877 (0x7013 bytes) 000194 1787848494.761368 trunc from 0x01ace8 to 0x026d59 (0xc072 bytes) 000195 1787848494.766819 mapread 0x01a663 thru 0x023373 (0x8d11 bytes) 000196 1787848494.785497 mapwrite 0x020b77 thru 0x021d3f (0x11c9 bytes) 000197 1787848494.800454 read 0x02b275 thru 0x037f5c (0xcce8 bytes) 000198 1787848494.813905 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000199 1787848495.045365 mapwrite 0x02b7e2 thru 0x030537 (0x4d56 bytes) 000200 1787848495.102318 trunc from 0x03f6d5 to 0x03acb0 (0x4a24 bytes) 000201 1787848495.143053 mapwrite 0x01a492 thru 0x028e8c (0xe9fb bytes) 000202 1787848495.225543 mapread 0x016d84 thru 0x01f516 (0x8793 bytes) 000203 1787848495.226811 mapwrite 0x038d4c thru 0x03ffff (0x72b4 bytes) 000204 1787848495.282500 read 0x0042ed thru 0x00b112 (0x6e26 bytes) 000205 1787848495.291874 punch from 0x014593 to 0x018aed (0x455b bytes) 000206 1787848495.298463 punch from 0x03cb03 to 0x03f724 (0x2c22 bytes) 000207 1787848495.313252 write 0x00fc38 thru 0x0189b1 (0x8d7a bytes) 000209 1787848495.337553 mapread 0x00dabd thru 0x01279f (0x4ce3 bytes) 000210 1787848495.350095 mapwrite 0x03a45b thru 0x03ffff (0x5ba5 bytes) 000214 1787848495.393248 trunc from 0x01f3a2 to 0x0218fc (0x255b bytes) 000215 1787848495.400717 trunc from 0x040000 to 0x039632 (0x69cd bytes) 000216 1787848495.412236 trunc from 0x0232d0 to 0x0256e3 (0x2414 bytes) 000217 1787848495.419085 write 0x031577 thru 0x03960d (0x8097 bytes) 000218 1787848495.420364 punch from 0x02ee72 to 0x03152e (0x26bd bytes) 000219 1787848495.435803 write 0x03c57f thru 0x03e885 (0x2307 bytes) 000220 1787848495.437071 fallocate 0x002450 thru 0x011ccf (0xf880 bytes) 000221 1787848495.442993 fallocate 0x0328fb thru 0x03e7b5 (0xbebb bytes) 000222 1787848495.450279 mapwrite 0x009fd5 thru 0x0104ce (0x64fa bytes) 000223 1787848495.470189 trunc from 0x03e886 to 0x00e494 (0x303f1 bytes) 000224 1787848495.514742 punch from 0x00bcdd to 0x00e494 (0x27b8 bytes) 000225 1787848495.533305 mapwrite 0x03e401 thru 0x03ffff (0x1bff bytes) 000226 1787848495.642976 trunc from 0x040000 to 0x02fc79 (0x10386 bytes) 000227 1787848495.672266 punch from 0x01d289 to 0x029a08 (0xc780 bytes) 000228 1787848495.706318 mapwrite 0x02cab1 thru 0x02d354 (0x8a4 bytes) 000229 1787848495.761771 mapwrite 0x00ab67 thru 0x014a5e (0x9ef8 bytes) 000230 1787848495.880191 trunc from 0x02fc7a to 0x02acb6 (0x4fc3 bytes) 000231 1787848495.907527 mapread 0x0137ed thru 0x018155 (0x4969 bytes) 000232 1787848495.910697 punch from 0x01758a to 0x01cca9 (0x5720 bytes) 000233 1787848495.920287 mapread 0x019d44 thru 0x020450 (0x670d bytes) 000234 1787848495.939889 mapwrite 0x020d99 thru 0x026f1b (0x6183 bytes) 000235 1787848496.057431 mapwrite 0x009355 thru 0x019081 (0xfd2d bytes) 000236 1787848496.138834 trunc from 0x02acb7 to 0x00d515 (0x1d7a1 bytes) 000237 1787848496.188040 trunc from 0x00d516 to 0x002a95 (0xaa80 bytes) 000238 1787848496.220698 write 0x021a06 thru 0x024d38 (0x3333 bytes) 000240 1787848496.227783 read 0x003d72 thru 0x004144 (0x3d3 bytes) 000242 1787848496.256714 fallocate 0x012cde thru 0x013b0a (0xe2d bytes) 000244 1787848496.283846 write 0x00f1c5 thru 0x019a70 (0xa8ac bytes) 000245 1787848496.305534 trunc from 0x000083 to 0x00f2ed (0xf26b bytes) 000246 1787848496.350605 mapread 0x00c013 thru 0x00f92f (0x391d bytes) 000247 1787848496.441475 punch from 0x020bd1 to 0x021ff3 (0x1423 bytes) 000248 1787848496.452366 read 0x017e43 thru 0x0210d4 (0x9292 bytes) 000249 1787848496.477060 read 0x01a2ad thru 0x01df47 (0x3c9b bytes) 000250 1787848496.477811 read 0x019a20 thru 0x024d38 (0xb319 bytes) 000251 1787848496.484099 fallocate 0x035694 thru 0x03da5d (0x83ca bytes) 000252 1787848496.501755 read 0x009d70 thru 0x01832a (0xe5bb bytes) 000254 1787848496.511273 trunc from 0x007d1b to 0x008709 (0x9ef bytes) 000256 1787848496.536777 write 0x00aeb1 thru 0x012d76 (0x7ec6 bytes) 000257 1787848496.543944 punch from 0x016b3f to 0x021909 (0xadcb bytes) 000260 1787848496.574100 punch from 0x037051 to 0x0370a3 (0x53 bytes) 000261 1787848496.584436 fallocate 0x020faf thru 0x02d145 (0xc197 bytes) 000262 1787848496.591916 mapwrite 0x00b942 thru 0x0168fe (0xafbd bytes) 000264 1787848496.669040 write 0x00e820 thru 0x00f68c (0xe6d bytes) 000266 1787848496.671975 fallocate 0x036f22 thru 0x03ffff (0x90de bytes) 000267 1787848496.678446 write 0x024e51 thru 0x031123 (0xc2d3 bytes) 000268 1787848496.680902 trunc from 0x03da5e to 0x012cfd (0x2ad60 bytes) 000269 1787848496.723686 mapwrite 0x007b8f thru 0x01095e (0x8dd0 bytes) 000270 1787848496.798993 write 0x002f15 thru 0x00ec72 (0xbd5e bytes) 000274 1787848496.801617 mapwrite 0x012fcb thru 0x01ab1b (0x7b51 bytes) 000275 1787848496.937510 write 0x03015d thru 0x039c04 (0x9aa8 bytes) 000277 1787848496.945898 trunc from 0x00ffdd to 0x01881d (0x8841 bytes) 000279 1787848496.972117 punch from 0x027dc4 to 0x029f06 (0x2143 bytes) 000280 1787848496.979907 punch from 0x0121bd to 0x01ab7c (0x89c0 bytes) 000281 1787848497.003908 fallocate 0x0086fd thru 0x014a72 (0xc376 bytes) 000282 1787848497.012686 trunc from 0x028684 to 0x02d499 (0x4e16 bytes) 000283 1787848497.020114 write 0x01f85f thru 0x027267 (0x7a09 bytes) 000284 1787848497.039441 read 0x01197e thru 0x01b4e1 (0x9b64 bytes) 000285 1787848497.058319 mapwrite 0x024df4 thru 0x02ef73 (0xa180 bytes) 000286 1787848497.097304 trunc from 0x039c05 to 0x01972e (0x204d6 bytes) 000287 1787848497.120680 trunc from 0x010d99 to 0x017da9 (0x7011 bytes) 000288 1787848497.126967 write 0x02770e thru 0x02f98b (0x827e bytes) 000289 1787848497.129865 write 0x0200ff thru 0x024667 (0x4569 bytes) 000290 1787848497.141762 read 0x010a3a thru 0x01e215 (0xd7dc bytes) 000291 1787848497.154791 fallocate 0x0042b0 thru 0x00c1b8 (0x7f09 bytes) 000292 1787848497.159820 punch from 0x020fff to 0x029653 (0x8655 bytes) 000293 1787848497.186388 punch from 0x00d239 to 0x01bf35 (0xecfd bytes) 000294 1787848497.194058 trunc from 0x02f98c to 0x02cf02 (0x2a89 bytes) 000295 1787848497.230930 fallocate 0x02b76e thru 0x032601 (0x6e94 bytes) 000296 1787848497.236659 mapread 0x01a04d thru 0x01c2b9 (0x226d bytes) 000297 1787848497.250674 trunc from 0x0011db to 0x00b6f3 (0xa519 bytes) 000298 1787848497.256651 mapwrite 0x0343a1 thru 0x03ffff (0xbc5f bytes) 000299 1787848497.326844 MIRROR_EXTEND to 2 mirrors 000300 1787848498.413428 fallocate 0x0306c4 thru 0x03d779 (0xd0b6 bytes) 000301 1787848498.465862 mapread 0x01c8cd thru 0x02469a (0x7dce bytes) 000302 1787848498.535642 punch from 0x03c3bb to 0x03ffff (0x3c45 bytes) 000303 1787848498.548236 mapwrite 0x01e31f thru 0x02a652 (0xc334 bytes) 000304 1787848498.596831 trunc from 0x01c563 to 0x021115 (0x4bb3 bytes) 000305 1787848498.604007 mapwrite 0x02cf8c thru 0x032259 (0x52ce bytes) 000306 1787848498.640636 trunc from 0x040000 to 0x03ad24 (0x52db bytes) 000307 1787848498.752352 MIRROR_RESYNC 2 mirrors 000308 1787848500.066350 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000309 1787848500.218549 trunc from 0x03ad25 to 0x03a0cb (0xc59 bytes) 000311 1787848500.255034 write 0x03d994 thru 0x03ffff (0x266c bytes) 000312 1787848500.258498 write 0x034389 thru 0x03ffff (0xbc77 bytes) 000314 1787848500.266304 trunc from 0x040000 to 0x00d76a (0x32895 bytes) 000315 1787848500.283951 trunc from 0x00d76b to 0x036ece (0x29764 bytes) 000316 1787848500.299977 write 0x01b0da thru 0x020476 (0x539d bytes) 000317 1787848500.308931 mapread 0x006cd0 thru 0x014e62 (0xe193 bytes) 000318 1787848500.331330 read 0x01467f thru 0x01dc2f (0x95b1 bytes) 000319 1787848500.343035 mapwrite 0x03aaf4 thru 0x03ffff (0x550c bytes) 000320 1787848500.497753 mapread 0x037ccf thru 0x03ffff (0x8331 bytes) 000321 1787848500.501598 write 0x00e2d4 thru 0x01952f (0xb25c bytes) 000322 1787848500.503743 mapwrite 0x02ad5e thru 0x02c6e8 (0x198b bytes) 000323 1787848500.569732 punch from 0x01fe23 to 0x029fda (0xa1b8 bytes) 000324 1787848500.584992 trunc from 0x01d815 to 0x02a38f (0xcb7b bytes) 000325 1787848500.594666 trunc from 0x040000 to 0x00647e (0x39b81 bytes) 000326 1787848500.644124 trunc from 0x00647f to 0x022b32 (0x1c6b4 bytes) 000327 1787848500.693174 write 0x018487 thru 0x020e36 (0x89b0 bytes) 000328 1787848500.708646 read 0x016a68 thru 0x021e55 (0xb3ee bytes) 000329 1787848500.718471 fallocate 0x019c73 thru 0x020f4a (0x72d8 bytes) 000331 1787848500.726334 trunc from 0x000b0d to 0x00fbfd (0xf0f1 bytes) 000333 1787848500.732435 trunc from 0x022b33 to 0x0183c0 (0xa772 bytes) 000334 1787848500.748017 fallocate 0x011dbf thru 0x019809 (0x7a4b bytes) 000336 1787848500.751910 mapread 0x004c68 thru 0x008ccc (0x4065 bytes) 000337 1787848500.765694 punch from 0x0080f9 to 0x00f734 (0x763c bytes) 000338 1787848500.768075 fallocate 0x01df83 thru 0x02d3c4 (0xf442 bytes) 000339 1787848500.770954 mapread 0x02ba79 thru 0x02d3c4 (0x194c bytes) 000340 1787848500.780900 mapwrite 0x02acc2 thru 0x039688 (0xe9c7 bytes) 000343 1787848500.924017 trunc from 0x02007b to 0x0242e6 (0x426c bytes) 000345 1787848500.930692 mapwrite 0x010d9a thru 0x019716 (0x897d bytes) 000346 1787848500.985038 mapread 0x0208fb thru 0x02437c (0x3a82 bytes) 000347 1787848501.020037 mapwrite 0x002096 thru 0x008907 (0x6872 bytes) 000348 1787848501.078331 read 0x011bda thru 0x015690 (0x3ab7 bytes) 000349 1787848501.078969 read 0x020e1b thru 0x024ed6 (0x40bc bytes) 000350 1787848501.081525 trunc from 0x0060f9 to 0x00eb59 (0x8a61 bytes) 000351 1787848501.098265 trunc from 0x039689 to 0x033299 (0x63ef bytes) 000353 1787848501.124298 read 0x01211e thru 0x01ac88 (0x8b6b bytes) 000354 1787848501.137939 write 0x0390a4 thru 0x03c3cf (0x332c bytes) 000355 1787848501.139320 mapread 0x00fff2 thru 0x016054 (0x6063 bytes) 000356 1787848501.141594 mapread 0x0207a2 thru 0x024163 (0x39c2 bytes) 000357 1787848501.143697 write 0x02a873 thru 0x036fb0 (0xc73e bytes) 000358 1787848501.153501 trunc from 0x03c3d0 to 0x006150 (0x3627f bytes) 000359 1787848501.182974 read 0x00235b thru 0x002f83 (0xc29 bytes) 000360 1787848501.183697 mapread 0x005930 thru 0x006150 (0x821 bytes) 000361 1787848501.186529 trunc from 0x006151 to 0x013192 (0xd042 bytes) 000362 1787848501.212293 trunc from 0x012c25 to 0x013192 (0x56e bytes) 000363 1787848501.219739 trunc from 0x013193 to 0x03b7f7 (0x28665 bytes) 000365 1787848501.244947 read 0x0032a9 thru 0x009ee5 (0x6c3d bytes) 000366 1787848501.262918 punch from 0x037bca to 0x03b7f7 (0x3c2e bytes) 000368 1787848501.271003 mapread 0x038c55 thru 0x03b7f7 (0x2ba3 bytes) 000369 1787848501.306475 mapwrite 0x014c73 thru 0x01afd7 (0x6365 bytes) 000370 1787848501.328501 trunc from 0x0108bf to 0x015e98 (0x55da bytes) 000372 1787848501.338297 punch from 0x02ef9d to 0x02fb5d (0xbc1 bytes) 000373 1787848501.345301 mapwrite 0x01916f thru 0x0246d6 (0xb568 bytes) 000375 1787848501.369712 mapread 0x007322 thru 0x00a69d (0x337c bytes) 000376 1787848501.370917 read 0x00ee16 thru 0x0175e5 (0x87d0 bytes) 000378 1787848501.384752 mapwrite 0x015ce8 thru 0x01be97 (0x61b0 bytes) 000379 1787848501.413765 fallocate 0x018d60 thru 0x0286ba (0xf95b bytes) 000380 1787848501.422662 mapwrite 0x031a50 thru 0x0372b5 (0x5866 bytes) 000381 1787848501.444706 fallocate 0x030ab2 thru 0x031220 (0x76f bytes) 000382 1787848501.449693 write 0x001d57 thru 0x005bba (0x3e64 bytes) 000383 1787848501.451673 write 0x01f49d thru 0x0219e9 (0x254d bytes) 000384 1787848501.452893 mapread 0x036474 thru 0x03a50c (0x4099 bytes) 000386 1787848501.455046 fallocate 0x0263a4 thru 0x02a0dc (0x3d39 bytes) 000387 1787848501.464080 trunc from 0x03ad54 to 0x03b7f7 (0xaa4 bytes) 000388 1787848501.468987 mapwrite 0x001192 thru 0x004cee (0x3b5d bytes) 000389 1787848501.489959 mapread 0x01c35b thru 0x024cf9 (0x899f bytes) 000390 1787848501.492455 trunc from 0x03b7f8 to 0x02a16f (0x11688 bytes) 000391 1787848501.508064 trunc from 0x00137f to 0x00ff1b (0xeb9d bytes) 000393 1787848501.515201 trunc from 0x000443 to 0x00d68a (0xd248 bytes) 000394 1787848501.520300 fallocate 0x006a8b thru 0x014f32 (0xe4a8 bytes) 000395 1787848501.526758 mapwrite 0x009b55 thru 0x00e6f8 (0x4ba4 bytes) 000396 1787848501.582802 trunc from 0x02a170 to 0x0394ee (0xf37f bytes) 000397 1787848501.609510 write 0x02940a thru 0x038ab7 (0xf6ae bytes) 000399 1787848501.620573 read 0x00ee2f thru 0x013a04 (0x4bd6 bytes) 000400 1787848501.632535 trunc from 0x01ec8e to 0x0202ac (0x161f bytes) 000401 1787848501.638154 read 0x02346e thru 0x02544a (0x1fdd bytes) 000402 1787848501.642994 MIRROR_EXTEND to 2 mirrors 000403 1787848502.723701 punch from 0x02f05d to 0x032a04 (0x39a8 bytes) 000404 1787848502.809860 fallocate 0x0130e8 thru 0x01e2f7 (0xb210 bytes) 000405 1787848502.823425 write 0x01f3cb thru 0x028095 (0x8ccb bytes) 000406 1787848502.873896 trunc from 0x0394ef to 0x015858 (0x23c96 bytes) 000407 1787848502.920290 trunc from 0x015859 to 0x02b060 (0x15808 bytes) 000408 1787848502.962333 mapread 0x022f4c thru 0x02b060 (0x8115 bytes) 000409 1787848503.025038 write 0x00a695 thru 0x019440 (0xedac bytes) 000410 1787848503.089736 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000411 1787848503.419179 punch from 0x006cc0 to 0x016816 (0xfb57 bytes) 000412 1787848503.426496 write 0x02f9bb thru 0x030774 (0xdba bytes) 000413 1787848503.436790 mapwrite 0x02a4da thru 0x02a635 (0x15c bytes) 000414 1787848503.478756 trunc from 0x0055c2 to 0x005f17 (0x956 bytes) 000415 1787848503.486303 punch from 0x00995c to 0x017de8 (0xe48d bytes) 000416 1787848503.509960 write 0x03cf10 thru 0x03ffff (0x30f0 bytes) 000417 1787848503.515752 trunc from 0x01582e to 0x024950 (0xf123 bytes) 000418 1787848503.543365 punch from 0x03ecc7 to 0x03ffff (0x1339 bytes) 000419 1787848503.558526 mapwrite 0x02801d thru 0x031dfc (0x9de0 bytes) 000421 1787848503.598676 mapwrite 0x01da51 thru 0x029442 (0xb9f2 bytes) 000422 1787848503.695163 punch from 0x02db73 to 0x0330bd (0x554b bytes) 000423 1787848503.701367 mapread 0x01d9f2 thru 0x02d002 (0xf611 bytes) 000426 1787848503.718418 mapwrite 0x020c3f thru 0x02b0f6 (0xa4b8 bytes) 000427 1787848503.756987 read 0x01e129 thru 0x02ba75 (0xd94d bytes) 000428 1787848503.762009 mapread 0x006581 thru 0x009a25 (0x34a5 bytes) 000429 1787848503.772664 trunc from 0x040000 to 0x03a640 (0x59bf bytes) 000430 1787848503.802824 fallocate 0x0358f7 thru 0x03ee95 (0x959f bytes) 000431 1787848503.811183 punch from 0x00e6cd to 0x019a7c (0xb3b0 bytes) 000432 1787848503.816459 trunc from 0x03a641 to 0x00b9ae (0x2ec92 bytes) 000433 1787848503.840786 mapread 0x00a43d thru 0x00b9ae (0x1572 bytes) 000434 1787848503.841997 trunc from 0x00370e to 0x00535e (0x1c51 bytes) 000435 1787848503.849513 read 0x00050d thru 0x00a5cf (0xa0c3 bytes) 000436 1787848503.858923 fallocate 0x01bbb7 thru 0x0241ab (0x85f5 bytes) 000437 1787848503.864238 read 0x000436 thru 0x001479 (0x1044 bytes) 000438 1787848503.864872 punch from 0x0020ab to 0x00b9ae (0x9904 bytes) 000439 1787848503.870731 punch from 0x0032fc to 0x005331 (0x2036 bytes) 000440 1787848503.876365 fallocate 0x03f52f thru 0x03ffff (0xad1 bytes) 000441 1787848503.881961 read 0x03ce73 thru 0x03ffff (0x318d bytes) 000444 1787848503.916421 MIRROR_EXTEND to 2 mirrors 000445 1787848504.979759 read 0x02e3d7 thru 0x03a0b7 (0xbce1 bytes) 000447 1787848505.049422 MIRROR_RESYNC 2 mirrors 000448 1787848505.593387 mapwrite 0x022bbb thru 0x030b1a (0xdf60 bytes) 000449 1787848505.770904 read 0x02065f thru 0x0285c9 (0x7f6b bytes) 000450 1787848505.771513 read 0x004308 thru 0x00487d (0x576 bytes) 000451 1787848505.801460 write 0x01cfc4 thru 0x01d18b (0x1c8 bytes) 000452 1787848505.809685 write 0x02bacb thru 0x02d267 (0x179d bytes) 000453 1787848505.810740 trunc from 0x040000 to 0x01dece (0x22131 bytes) 000454 1787848505.835474 mapwrite 0x005049 thru 0x00e11c (0x90d4 bytes) 000455 1787848505.860264 trunc from 0x0019ee to 0x00ab08 (0x911b bytes) 000456 1787848505.871280 fallocate 0x011d99 thru 0x01aeeb (0x9153 bytes) 000457 1787848505.882385 mapread 0x01bb34 thru 0x01dece (0x239b bytes) 000458 1787848505.888928 write 0x010a27 thru 0x01ecd8 (0xe2b2 bytes) 000459 1787848505.892005 fallocate 0x009580 thru 0x00f0fc (0x5b7d bytes) 000460 1787848505.899456 mapwrite 0x01fe62 thru 0x02b091 (0xb230 bytes) 000461 1787848506.066206 punch from 0x0183c6 to 0x01c20c (0x3e47 bytes) 000462 1787848506.076381 fallocate 0x02d8c4 thru 0x034e80 (0x75bd bytes) 000463 1787848506.081523 trunc from 0x033b2e to 0x034e80 (0x1353 bytes) 000464 1787848506.087701 write 0x00a723 thru 0x0191da (0xeab8 bytes) 000465 1787848506.097881 trunc from 0x034e81 to 0x034e7c (0x4 bytes) 000467 1787848506.113761 punch from 0x02ace6 to 0x0300f9 (0x5414 bytes) 000468 1787848506.123850 mapwrite 0x027c79 thru 0x02dc85 (0x600d bytes) 000469 1787848506.165386 mapread 0x012297 thru 0x021463 (0xf1cd bytes) 000470 1787848506.184543 fallocate 0x030bee thru 0x030c67 (0x7a bytes) 000471 1787848506.190022 fallocate 0x02459b thru 0x02e67e (0xa0e4 bytes) 000472 1787848506.199056 write 0x011e60 thru 0x014ef3 (0x3094 bytes) 000473 1787848506.203425 trunc from 0x034e7d to 0x0321e7 (0x2c95 bytes) 000474 1787848506.239702 fallocate 0x000bef thru 0x00143d (0x84f bytes) 000475 1787848506.251449 mapwrite 0x013d43 thru 0x023c63 (0xff21 bytes) 000476 1787848506.358295 punch from 0x024a63 to 0x02f87f (0xae1d bytes) 000477 1787848506.370490 trunc from 0x0321e8 to 0x0081c5 (0x2a022 bytes) 000478 1787848506.402508 read 0x00690f thru 0x0081c5 (0x18b7 bytes) 000479 1787848506.417117 trunc from 0x0045d7 to 0x0081c5 (0x3bef bytes) 000480 1787848506.429851 trunc from 0x00102f to 0x0081c5 (0x7197 bytes) 000481 1787848506.456955 read 0x0070ff thru 0x0081c5 (0x10c7 bytes) 000482 1787848506.500156 fallocate 0x03a634 thru 0x03ffff (0x59cc bytes) 000483 1787848506.505646 mapread 0x002d13 thru 0x00f38b (0xc679 bytes) 000484 1787848506.539458 read 0x00722e thru 0x01427f (0xd052 bytes) 000485 1787848506.540952 trunc from 0x040000 to 0x01aab3 (0x2554c bytes) 000486 1787848506.588072 fallocate 0x00baee thru 0x00bc9c (0x1af bytes) 000487 1787848506.599528 write 0x030416 thru 0x03cbc0 (0xc7ab bytes) 000488 1787848506.609498 trunc from 0x03cbc1 to 0x010802 (0x2c3be bytes) 000489 1787848506.631848 mapread 0x00dbce thru 0x010802 (0x2c35 bytes) 000490 1787848506.632736 trunc from 0x0005f3 to 0x00d0c6 (0xcad4 bytes) 000491 1787848506.640183 write 0x036746 thru 0x0395d2 (0x2e8d bytes) 000493 1787848506.641976 mapwrite 0x0088d6 thru 0x00e3ad (0x5ad8 bytes) 000494 1787848506.736371 trunc from 0x0395d3 to 0x021315 (0x182bd bytes) 000495 1787848506.783067 trunc from 0x021316 to 0x031078 (0xfd63 bytes) 000496 1787848506.821653 mapwrite 0x03a9ca thru 0x03cea2 (0x24d9 bytes) 000497 1787848506.864495 read 0x0014c9 thru 0x008183 (0x6cbb bytes) 000498 1787848506.871017 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000499 1787848507.041822 fallocate 0x01e572 thru 0x026bea (0x8679 bytes) 000501 1787848507.052333 trunc from 0x03309b to 0x03cea2 (0x9e08 bytes) 000502 1787848507.060832 fallocate 0x00a31e thru 0x0161a5 (0xbe88 bytes) 000503 1787848507.073594 trunc from 0x03cea3 to 0x0248a9 (0x185f9 bytes) 000504 1787848507.157010 read 0x0193c0 thru 0x01e111 (0x4d52 bytes) 000505 1787848507.187946 read 0x0098a6 thru 0x00bebe (0x2619 bytes) 000506 1787848507.222946 mapwrite 0x030ee1 thru 0x03b849 (0xa969 bytes) 000507 1787848507.339253 write 0x0163bc thru 0x016f14 (0xb59 bytes) 000508 1787848507.340023 mapread 0x037bf6 thru 0x038f04 (0x130f bytes) 000511 1787848507.345231 read 0x03351a thru 0x033ed7 (0x9be bytes) 000512 1787848507.346942 write 0x030be1 thru 0x03d192 (0xc5b2 bytes) 000513 1787848507.353331 read 0x01d43a thru 0x026963 (0x952a bytes) 000514 1787848507.355834 punch from 0x0275ce to 0x02814d (0xb80 bytes) 000515 1787848507.370763 read 0x02cef1 thru 0x02fb29 (0x2c39 bytes) 000516 1787848507.373818 mapread 0x00c4c6 thru 0x017575 (0xb0b0 bytes) 000517 1787848507.377992 punch from 0x035071 to 0x03d192 (0x8122 bytes) 000519 1787848507.434888 MIRROR_EXTEND to 2 mirrors 000520 1787848509.036480 fallocate 0x023060 thru 0x03194d (0xe8ee bytes) 000521 1787848509.141450 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000522 1787848509.391049 read 0x0186ee thru 0x0215d1 (0x8ee4 bytes) 000523 1787848509.415334 read 0x018140 thru 0x0200c1 (0x7f82 bytes) 000524 1787848509.415993 trunc from 0x02a091 to 0x03a040 (0xffb0 bytes) 000525 1787848509.427227 mapread 0x0153f5 thru 0x0180da (0x2ce6 bytes) 000526 1787848509.445380 punch from 0x015de1 to 0x020c37 (0xae57 bytes) 000527 1787848509.459062 trunc from 0x03d193 to 0x00382a (0x39968 bytes) 000528 1787848509.489548 fallocate 0x00763f thru 0x00fec7 (0x8889 bytes) 000529 1787848509.505737 MIRROR_EXTEND to 2 mirrors 000530 1787848510.886497 MIRROR_EXTEND to 3 mirrors 000531 1787848512.104173 punch from 0x000c3a to 0x00fec7 (0xf28e bytes) 000532 1787848512.163480 trunc from 0x00064f to 0x00a359 (0x9d0b bytes) 000533 1787848512.187305 mapwrite 0x0059e7 thru 0x009352 (0x396c bytes) 000534 1787848512.352356 trunc from 0x008f23 to 0x00fec7 (0x6fa5 bytes) 000535 1787848512.376329 MIRROR_RESYNC 3 mirrors 000536 1787848513.958829 mapwrite 0x0154f1 thru 0x019660 (0x4170 bytes) 000537 1787848514.110332 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19661): Mirror 3: 0xc1b69e02 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19661): Mirror 3: 0xc1b69e02 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19661: skipped 000538 1787848514.706149 write 0x010388 thru 0x017f6e (0x7be7 bytes) 000540 1787848514.753821 trunc from 0x0153cc to 0x01882f (0x3464 bytes) 000542 1787848514.810965 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000543 1787848515.264304 mapwrite 0x039588 thru 0x03ffff (0x6a78 bytes) 000544 1787848515.425388 trunc from 0x03cdbf to 0x03eac6 (0x1d08 bytes) 000545 1787848515.437381 punch from 0x0053c5 to 0x013276 (0xdeb2 bytes) 000546 1787848515.463346 MIRROR_EXTEND to 2 mirrors 000547 1787848517.054268 punch from 0x00dd8d to 0x0137d5 (0x5a49 bytes) 000548 1787848517.141404 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1e95cdff (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf6af1353 Mirror 3: 0xf6af1353 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000549 1787848519.195530 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1e95cdff (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf6af1353 Mirror 3: 0xf6af1353 Mirror 4: 0xf6af1353 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000552 1787848521.437777 mapwrite 0x004fd7 thru 0x0098e8 (0x4912 bytes) 000555 1787848521.652628 mapread 0x01b082 thru 0x020e11 (0x5d90 bytes) 000556 1787848521.685699 mapread 0x029666 thru 0x0318d8 (0x8273 bytes) 000557 1787848521.785221 write 0x027a8e thru 0x033e8e (0xc401 bytes) 000558 1787848521.822410 mapwrite 0x00f9ae thru 0x01cc92 (0xd2e5 bytes) 000559 1787848521.888393 read 0x014707 thru 0x01ba50 (0x734a bytes) 000560 1787848521.904675 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1e95cdff (stale) Mirror 3: 0xf6af1353 (stale) Mirror 4: 0xf6af1353 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb173c182 Mirror 5: 0xb173c182 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000561 1787848524.206374 mapwrite 0x025306 thru 0x02f13e (0x9e39 bytes) 000562 1787848524.386341 write 0x006f62 thru 0x016898 (0xf937 bytes) 000563 1787848524.458735 trunc from 0x0148a0 to 0x021452 (0xcbb3 bytes) 000564 1787848524.509037 mapread 0x02518c thru 0x0282af (0x3124 bytes) 000565 1787848524.511783 read 0x01e559 thru 0x022d52 (0x47fa bytes) 000567 1787848524.541046 trunc from 0x03c471 to 0x03ffff (0x3b8f bytes) 000568 1787848524.579491 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1e95cdff (stale) Mirror 3: 0xf6af1353 (stale) Mirror 4: 0xf6af1353 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1e95cdff (stale) Mirror 3: 0xf6af1353 (stale) Mirror 4: 0xf6af1353 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000569 1787848525.467347 punch from 0x002b0a to 0x00a022 (0x7519 bytes) 000571 1787848525.477717 mapwrite 0x000e50 thru 0x00492a (0x3adb bytes) 000572 1787848525.543296 write 0x03c4a0 thru 0x03ffff (0x3b60 bytes) 000574 1787848525.558383 fallocate 0x0084e9 thru 0x0162ee (0xde06 bytes) 000575 1787848525.572035 trunc from 0x03a358 to 0x03cf16 (0x2bbf bytes) 000576 1787848525.599424 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1e95cdff (stale) Mirror 4: 0xf6af1353 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1e95cdff (stale) Mirror 4: 0xf6af1353 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000578 1787848526.238504 trunc from 0x0147ea to 0x020671 (0xbe88 bytes) 000579 1787848526.244333 mapread 0x006690 thru 0x00b303 (0x4c74 bytes) 000580 1787848526.266390 mapread 0x01a133 thru 0x0208f2 (0x67c0 bytes) 000581 1787848526.281835 mapread 0x02987a thru 0x03109f (0x7826 bytes) 000582 1787848526.291924 read 0x01cbfc thru 0x027d41 (0xb146 bytes) 000583 1787848526.292801 mapwrite 0x024e7b thru 0x0334bc (0xe642 bytes) 000584 1787848526.330231 fallocate 0x00dae3 thru 0x01c7fa (0xed18 bytes) 000585 1787848526.337022 trunc from 0x02a6b4 to 0x02fc98 (0x55e5 bytes) 000586 1787848526.345845 mapread 0x0353e6 thru 0x03c01f (0x6c3a bytes) 000587 1787848526.359874 trunc from 0x016864 to 0x020a53 (0xa1f0 bytes) 000588 1787848526.365613 trunc from 0x040000 to 0x01956c (0x26a93 bytes) 000589 1787848526.396301 punch from 0x00c3a4 to 0x01956c (0xd1c9 bytes) 000590 1787848526.406139 trunc from 0x00dfca to 0x01956c (0xb5a3 bytes) 000593 1787848526.413170 mapread 0x004328 thru 0x010b81 (0xc85a bytes) 000594 1787848526.449439 read 0x004bbd thru 0x007a6b (0x2eaf bytes) 000595 1787848526.450134 trunc from 0x01956d to 0x00b29c (0xe2d0 bytes) 000596 1787848526.491388 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xb29d): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb29d): Mirror 2: 0xede16be4 (stale) Mirror 4: 0xede16be4 (stale) CRC-32 checksum value for chunk [0, 0xb29d): Mirror 1: 0xac1a53af Mirror 5: 0xac1a53af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb29d: skipped 000597 1787848528.061273 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0xb29d): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb29d): Mirror 2: 0xede16be4 (stale) Mirror 4: 0xede16be4 (stale) CRC-32 checksum value for chunk [0, 0xb29d): Mirror 1: 0xac1a53af Mirror 5: 0xac1a53af Mirror 6: 0xac1a53af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb29d: skipped 000598 1787848529.521406 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb29d): Mirror 4: 0xede16be4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb29d): Mirror 4: 0xede16be4 (stale) CRC-32 checksum value for chunk [0, 0xb29d): Mirror 1: 0xac1a53af Mirror 5: 0xac1a53af Mirror 6: 0xac1a53af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb29d: skipped 000599 1787848530.206185 trunc from 0x0088f5 to 0x00b29c (0x29a8 bytes) 000600 1787848530.336396 trunc from 0x00b29d to 0x0179db (0xc73f bytes) 000601 1787848530.367114 write 0x030a12 thru 0x0390ec (0x86db bytes) 000602 1787848530.369235 mapread 0x028f4a thru 0x038ea9 (0xff60 bytes) 000603 1787848530.389210 write 0x01355a thru 0x0139c5 (0x46c bytes) 000605 1787848530.396513 trunc from 0x0390ed to 0x004f99 (0x34153 bytes) 000606 1787848530.413166 trunc from 0x004f9a to 0x037a32 (0x32a99 bytes) 000607 1787848530.429300 fallocate 0x02ec67 thru 0x039e35 (0xb1cf bytes) 000608 1787848530.441640 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x37a33): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a33): Mirror 4: 0x8f99d088 (stale) Mirror 5: 0x4c51e7a0 (stale) Mirror 6: 0x4c51e7a0 (stale) CRC-32 checksum value for chunk [0, 0x37a33): Mirror 1: 0xe222116 Mirror 7: 0xe222116 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a33: skipped 000609 1787848531.626804 write 0x007f34 thru 0x00cd0c (0x4dd9 bytes) 000610 1787848531.680411 punch from 0x033279 to 0x037a32 (0x47ba bytes) 000611 1787848531.684476 fallocate 0x0189b8 thru 0x02114d (0x8796 bytes) 000612 1787848531.687221 fallocate 0x031cb6 thru 0x03642e (0x4779 bytes) 000613 1787848531.691881 punch from 0x0288e0 to 0x036fb7 (0xe6d8 bytes) 000614 1787848531.696809 mapwrite 0x035f4a thru 0x03865b (0x2712 bytes) 000615 1787848531.753931 mapwrite 0x011f0f thru 0x01b617 (0x9709 bytes) 000616 1787848531.787995 trunc from 0x01096e to 0x01cd0d (0xc3a0 bytes) 000617 1787848531.792340 write 0x000c08 thru 0x0107cd (0xfbc6 bytes) 000618 1787848531.807169 punch from 0x02882b to 0x02c563 (0x3d39 bytes) 000619 1787848531.827897 trunc from 0x01c2cd to 0x020902 (0x4636 bytes) 000620 1787848531.838971 trunc from 0x00e7b3 to 0x01017e (0x19cc bytes) 000622 1787848531.862776 punch from 0x017913 to 0x02632e (0xea1c bytes) 000623 1787848531.870704 write 0x035089 thru 0x03d72a (0x86a2 bytes) 000624 1787848531.873494 trunc from 0x036789 to 0x03d72a (0x6fa2 bytes) 000625 1787848531.902993 mapread 0x01465e thru 0x01bbce (0x7571 bytes) 000627 1787848531.924688 trunc from 0x01bd67 to 0x02a6d1 (0xe96b bytes) 000628 1787848531.929358 read 0x02e4cf thru 0x02fec2 (0x19f4 bytes) 000629 1787848531.933909 mapread 0x026fb2 thru 0x033211 (0xc260 bytes) 000630 1787848531.942726 trunc from 0x027f82 to 0x030731 (0x87b0 bytes) 000631 1787848531.954232 MIRROR_RESYNC 5 mirrors 000633 1787848533.795404 read 0x00dd88 thru 0x014829 (0x6aa2 bytes) 000634 1787848533.806873 mapread 0x030713 thru 0x03d72a (0xd018 bytes) 000636 1787848533.839350 trunc from 0x03d72b to 0x01c7ee (0x20f3c bytes) 000637 1787848533.940369 mapread 0x006ec6 thru 0x014d7f (0xdeba bytes) 000638 1787848533.969342 read 0x00c845 thru 0x0124d2 (0x5c8e bytes) 000640 1787848533.970349 write 0x0129a1 thru 0x013328 (0x988 bytes) 000641 1787848533.985927 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c7ef): Mirror 4: 0x26e7fdf5 (stale) Mirror 5: 0x26e7fdf5 (stale) Mirror 6: 0x26e7fdf5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c7ef): Mirror 4: 0x26e7fdf5 (stale) Mirror 5: 0x26e7fdf5 (stale) Mirror 6: 0x26e7fdf5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c7ef: skipped 000642 1787848534.671670 punch from 0x0172ca to 0x01c7ee (0x5525 bytes) 000643 1787848534.682751 write 0x01b6e6 thru 0x024fa9 (0x98c4 bytes) 000644 1787848534.708452 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24faa): Mirror 4: 0x3c2236c0 (stale) Mirror 5: 0x3c2236c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24faa): Mirror 4: 0x3c2236c0 (stale) Mirror 5: 0x3c2236c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24faa: skipped 000645 1787848535.226940 write 0x02ef03 thru 0x039116 (0xa214 bytes) 000646 1787848535.239114 mapwrite 0x023607 thru 0x0268cc (0x32c6 bytes) 000647 1787848535.316179 punch from 0x025d51 to 0x02d13f (0x73ef bytes) 000648 1787848535.325078 punch from 0x02e8b0 to 0x039116 (0xa867 bytes) 000649 1787848535.333450 mapread 0x02ca1b thru 0x036550 (0x9b36 bytes) 000651 1787848535.378623 fallocate 0x0183fc thru 0x01ad1f (0x2924 bytes) 000652 1787848535.390099 fallocate 0x02cce1 thru 0x035371 (0x8691 bytes) 000656 1787848535.413756 MIRROR_RESYNC 3 mirrors 000657 1787848536.784521 trunc from 0x00061d to 0x004182 (0x3b66 bytes) 000658 1787848536.828209 trunc from 0x039117 to 0x00ab22 (0x2e5f4 bytes) 000660 1787848536.864248 trunc from 0x00ab23 to 0x01eb96 (0x14074 bytes) 000661 1787848536.893822 trunc from 0x000444 to 0x00d380 (0xcf3d bytes) 000662 1787848536.898275 fallocate 0x00359b thru 0x008e51 (0x58b7 bytes) 000663 1787848536.914356 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1eb97): Mirror 4: 0xcc545097 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1eb97): Mirror 4: 0xcc545097 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1eb97: skipped 000666 1787848537.235919 write 0x02ca8e thru 0x03b054 (0xe5c7 bytes) 000667 1787848537.246008 mapwrite 0x0065d1 thru 0x009bce (0x35fe bytes) 000668 1787848537.317938 punch from 0x03623f to 0x03b054 (0x4e16 bytes) 000669 1787848537.330715 mapread 0x01c589 thru 0x02bd5c (0xf7d4 bytes) 000670 1787848537.360652 write 0x0071ff thru 0x007c5d (0xa5f bytes) 000671 1787848537.367237 mapwrite 0x036111 thru 0x03ffff (0x9eef bytes) 000672 1787848537.517273 trunc from 0x004b85 to 0x00aa2a (0x5ea6 bytes) 000673 1787848537.535409 mapwrite 0x027f0e thru 0x036d13 (0xee06 bytes) 000674 1787848537.609182 fallocate 0x001eef thru 0x00c1a2 (0xa2b4 bytes) 000675 1787848537.622478 mapwrite 0x00899f thru 0x011a18 (0x907a bytes) 000676 1787848537.736361 mapread 0x0030ac thru 0x0051cb (0x2120 bytes) 000677 1787848537.737298 read 0x024a5c thru 0x02a26f (0x5814 bytes) 000679 1787848537.738300 trunc from 0x00e01c to 0x019f00 (0xbee5 bytes) 000680 1787848537.768594 punch from 0x0198d7 to 0x01e44f (0x4b79 bytes) 000681 1787848537.783405 mapread 0x013b6d thru 0x017e84 (0x4318 bytes) 000683 1787848537.802461 mapwrite 0x022a3c thru 0x0329d0 (0xff95 bytes) 000684 1787848537.863533 MIRROR_RESYNC 2 mirrors 000685 1787848538.743786 MIRROR_EXTEND to 3 mirrors 000686 1787848539.465155 trunc from 0x040000 to 0x00e835 (0x317ca bytes) 000688 1787848539.568043 read 0x001600 thru 0x007ea6 (0x68a7 bytes) 000689 1787848539.578905 mapread 0x004803 thru 0x00e835 (0xa033 bytes) 000690 1787848539.614112 read 0x006b8c thru 0x00e835 (0x7caa bytes) 000691 1787848539.622001 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe836): Mirror 5: 0xff1b91c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe836): Mirror 5: 0xff1b91c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe836: skipped 000692 1787848539.998351 punch from 0x008c90 to 0x00adba (0x212b bytes) 000693 1787848540.004517 mapread 0x006e47 thru 0x00e835 (0x79ef bytes) 000694 1787848540.026753 fallocate 0x02999b thru 0x029bd8 (0x23e bytes) 000695 1787848540.031671 mapwrite 0x002bf7 thru 0x007c25 (0x502f bytes) 000696 1787848540.049764 mapread 0x006e96 thru 0x00e835 (0x79a0 bytes) 000697 1787848540.052491 trunc from 0x00e836 to 0x01e11f (0xf8ea bytes) 000698 1787848540.074239 mapread 0x00d787 thru 0x017cd6 (0xa550 bytes) 000701 1787848540.091521 write 0x039b3b thru 0x03ffff (0x64c5 bytes) 000702 1787848540.094409 read 0x027cfd thru 0x0331a2 (0xb4a6 bytes) 000703 1787848540.102863 mapwrite 0x02018c thru 0x025670 (0x54e5 bytes) 000704 1787848540.130285 trunc from 0x040000 to 0x00aa1c (0x355e3 bytes) 000705 1787848540.148140 trunc from 0x0000f6 to 0x00aa1c (0xa927 bytes) 000706 1787848540.160838 MIRROR_RESYNC 2 mirrors 000707 1787848540.689877 read 0x00998e thru 0x00aa1c (0x108f bytes) 000708 1787848540.697925 trunc from 0x00662a to 0x00aa1c (0x43f3 bytes) 000709 1787848540.740645 read 0x0000ce thru 0x00aa1c (0xa94f bytes) 000710 1787848540.749089 punch from 0x00058b to 0x002d2a (0x27a0 bytes) 000711 1787848540.754207 punch from 0x00134c to 0x004f9c (0x3c51 bytes) 000712 1787848540.759228 mapwrite 0x02de02 thru 0x03335d (0x555c bytes) 000713 1787848540.837899 read 0x00c205 thru 0x012813 (0x660f bytes) 000714 1787848540.839237 fallocate 0x0355dd thru 0x03bd5d (0x6781 bytes) 000715 1787848540.846440 trunc from 0x03bd5e to 0x01c1e3 (0x1fb7a bytes) 000716 1787848540.863486 write 0x00dd0f thru 0x0178d0 (0x9bc2 bytes) 000717 1787848540.866049 punch from 0x016e5f to 0x018190 (0x1332 bytes) 000719 1787848540.882847 punch from 0x00cf97 to 0x016d1e (0x9d88 bytes) 000720 1787848540.889436 fallocate 0x00e8ac thru 0x01de4c (0xf5a1 bytes) 000722 1787848540.896266 mapread 0x00579c thru 0x00fa1a (0xa27f bytes) 000723 1787848540.910047 trunc from 0x01de4d to 0x02e6a7 (0x1085b bytes) 000724 1787848540.930141 mapread 0x01ad5f thru 0x01d38a (0x262c bytes) 000725 1787848540.932913 write 0x00bf1b thru 0x01b7f9 (0xf8df bytes) 000726 1787848540.937537 trunc from 0x02e6a8 to 0x02ad22 (0x3985 bytes) 000727 1787848540.956993 trunc from 0x0212dd to 0x023632 (0x2356 bytes) 000728 1787848540.963882 read 0x0170da thru 0x019f55 (0x2e7c bytes) 000729 1787848540.964580 read 0x00df3f thru 0x01505e (0x7120 bytes) 000730 1787848540.966295 trunc from 0x01a141 to 0x029437 (0xf2f7 bytes) 000731 1787848540.982688 trunc from 0x02ad23 to 0x02389e (0x7484 bytes) 000732 1787848541.013262 write 0x038a85 thru 0x03ffff (0x757b bytes) 000735 1787848541.018061 trunc from 0x006029 to 0x01169b (0xb673 bytes) 000736 1787848541.026302 trunc from 0x040000 to 0x017028 (0x28fd7 bytes) 000737 1787848541.050282 trunc from 0x0004af to 0x00eb90 (0xe6e2 bytes) 000738 1787848541.056404 trunc from 0x00830e to 0x0150e2 (0xcdd5 bytes) 000739 1787848541.062527 mapwrite 0x035d88 thru 0x03ffff (0xa278 bytes) 000740 1787848541.131597 trunc from 0x040000 to 0x03ff20 (0xdf bytes) 000741 1787848541.160087 fallocate 0x033746 thru 0x03e42e (0xace9 bytes) 000742 1787848541.172678 read 0x008ca4 thru 0x00eca4 (0x6001 bytes) 000743 1787848541.174691 trunc from 0x03707a to 0x03ff20 (0x8ea7 bytes) 000744 1787848541.181641 trunc from 0x03ff21 to 0x02f8e3 (0x1063d bytes) 000745 1787848541.212386 mapwrite 0x03f52c thru 0x03ffff (0xad4 bytes) 000747 1787848541.313371 fallocate 0x03afcc thru 0x03b19f (0x1d4 bytes) 000748 1787848541.323250 mapread 0x004323 thru 0x009989 (0x5667 bytes) 000751 1787848541.325010 punch from 0x03315d to 0x03ffff (0xcea3 bytes) 000752 1787848541.362897 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8d9cee1b Mirror 6: 0x8d9cee1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000755 1787848542.629431 trunc from 0x02f97a to 0x0302d5 (0x95c bytes) 000756 1787848542.668859 read 0x023b3d thru 0x032044 (0xe508 bytes) 000757 1787848542.689517 mapwrite 0x032f3d thru 0x03af6f (0x8033 bytes) 000758 1787848542.779752 read 0x021186 thru 0x024276 (0x30f1 bytes) 000759 1787848542.837356 MIRROR_RESYNC 3 mirrors 000761 1787848544.040063 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000762 1787848544.526593 trunc from 0x040000 to 0x013ed5 (0x2c12a bytes) 000763 1787848544.684302 trunc from 0x013c78 to 0x013ed5 (0x25e bytes) 000764 1787848544.710966 punch from 0x00ed54 to 0x013ed5 (0x5182 bytes) 000765 1787848544.759146 MIRROR_RESYNC 2 mirrors 000766 1787848545.555761 write 0x03557f thru 0x03814d (0x2bcf bytes) 000767 1787848545.603645 read 0x032102 thru 0x03232a (0x229 bytes) 000768 1787848545.615299 fallocate 0x02905c thru 0x031874 (0x8819 bytes) 000769 1787848545.621955 fallocate 0x00a350 thru 0x011aab (0x775c bytes) 000770 1787848545.635343 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3814e): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3814e): Mirror 5: 0x1fc2de6f (stale) CRC-32 checksum value for chunk [0, 0x3814e): Mirror 1: 0x2f19c977 Mirror 6: 0x2f19c977 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3814e: skipped 000772 1787848546.786373 punch from 0x00f36a to 0x015ec8 (0x6b5f bytes) 000773 1787848546.845071 read 0x01ca86 thru 0x0293c1 (0xc93c bytes) 000774 1787848546.858138 mapwrite 0x004af5 thru 0x007ab0 (0x2fbc bytes) 000775 1787848546.906421 write 0x035180 thru 0x035ee6 (0xd67 bytes) 000776 1787848546.916683 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3814e): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3814e): Mirror 5: 0x1fc2de6f (stale) Mirror 6: 0x2f19c977 (stale) CRC-32 checksum value for chunk [0, 0x3814e): Mirror 1: 0xf23b751b Mirror 7: 0xf23b751b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3814e: skipped 000777 1787848548.134722 read 0x018cba thru 0x0231e7 (0xa52e bytes) 000778 1787848548.145095 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3814e): Mirror 6: 0x2f19c977 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3814e): Mirror 6: 0x2f19c977 (stale) CRC-32 checksum value for chunk [0, 0x3814e): Mirror 1: 0xf23b751b Mirror 7: 0xf23b751b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3814e: skipped 000779 1787848548.649666 punch from 0x024ad9 to 0x03402e (0xf556 bytes) 000780 1787848548.705338 MIRROR_RESYNC 3 mirrors 000782 1787848549.591205 trunc from 0x03814e to 0x02bbbf (0xc58e bytes) 000783 1787848549.644220 fallocate 0x00e673 thru 0x015be8 (0x7576 bytes) 000784 1787848549.650113 mapwrite 0x03784c thru 0x03ffff (0x87b4 bytes) 000786 1787848549.703678 trunc from 0x039f7f to 0x03a1a2 (0x224 bytes) 000787 1787848549.707883 read 0x007478 thru 0x016748 (0xf2d1 bytes) 000788 1787848549.717735 read 0x012d45 thru 0x01ae81 (0x813d bytes) 000789 1787848549.729043 trunc from 0x0243de to 0x02dcc2 (0x98e5 bytes) 000790 1787848549.739215 trunc from 0x038e3f to 0x03ffff (0x71c1 bytes) 000792 1787848549.745392 write 0x0036ea thru 0x00ce33 (0x974a bytes) 000793 1787848549.747312 mapread 0x034814 thru 0x035c90 (0x147d bytes) 000794 1787848549.749036 punch from 0x031211 to 0x03d0b3 (0xbea3 bytes) 000795 1787848549.754797 punch from 0x00dc0a to 0x01c027 (0xe41e bytes) 000796 1787848549.760144 write 0x0361ac thru 0x039072 (0x2ec7 bytes) 000797 1787848549.771163 trunc from 0x040000 to 0x03d450 (0x2baf bytes) 000799 1787848549.785156 read 0x014f53 thru 0x023ff8 (0xf0a6 bytes) 000800 1787848549.797245 fallocate 0x02c247 thru 0x03374b (0x7505 bytes) 000801 1787848549.807029 MIRROR_RESYNC 3 mirrors 000802 1787848550.864039 trunc from 0x00ac0a to 0x00ecc9 (0x40c0 bytes) 000803 1787848550.928418 trunc from 0x033119 to 0x03d450 (0xa338 bytes) 000804 1787848550.939415 trunc from 0x034a5e to 0x0357a8 (0xd4b bytes) 000805 1787848550.957012 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d451): Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d451): Mirror 6: 0x3079042b (stale) Mirror 7: 0x3079042b (stale) CRC-32 checksum value for chunk [0, 0x3d451): Mirror 1: 0x5cd35ab0 Mirror 8: 0x5cd35ab0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d451: skipped 000806 1787848552.332858 fallocate 0x035d1b thru 0x03ffff (0xa2e5 bytes) 000808 1787848552.409967 mapwrite 0x01419e thru 0x01f592 (0xb3f5 bytes) 000809 1787848552.489198 fallocate 0x010f09 thru 0x018296 (0x738e bytes) 000811 1787848552.495369 read 0x03fa5b thru 0x03fe02 (0x3a8 bytes) 000812 1787848552.502275 mapread 0x0228ef thru 0x024145 (0x1857 bytes) 000813 1787848552.506462 write 0x02ae7a thru 0x038bce (0xdd55 bytes) 000814 1787848552.532176 fallocate 0x01d905 thru 0x02ac4c (0xd348 bytes) 000815 1787848552.541746 mapread 0x006bb5 thru 0x0161f7 (0xf643 bytes) 000816 1787848552.555211 mapread 0x0289aa thru 0x028b63 (0x1ba bytes) 000817 1787848552.562369 trunc from 0x00683c to 0x0102d8 (0x9a9d bytes) 000818 1787848552.568910 fallocate 0x03d54d thru 0x03ffff (0x2ab3 bytes) 000819 1787848552.579351 mapread 0x02ac81 thru 0x034144 (0x94c4 bytes) 000820 1787848552.588171 MIRROR_RESYNC 4 mirrors 000821 1787848554.325831 mapwrite 0x003051 thru 0x007e88 (0x4e38 bytes) 000822 1787848554.547145 mapwrite 0x001a60 thru 0x004d9f (0x3340 bytes) 000823 1787848554.583089 fallocate 0x0347cd thru 0x03ffff (0xb833 bytes) 000824 1787848554.596599 trunc from 0x00b644 to 0x011ea7 (0x6864 bytes) 000825 1787848554.602142 trunc from 0x040000 to 0x026413 (0x19bec bytes) 000826 1787848554.633665 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26414): Mirror 6: 0x5320c72a (stale) Mirror 8: 0x5320c72a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26414): Mirror 6: 0x5320c72a (stale) Mirror 8: 0x5320c72a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26414: skipped 000827 1787848555.079418 trunc from 0x026414 to 0x025a17 (0x9fc bytes) 000828 1787848555.102401 punch from 0x0228ca to 0x025a17 (0x314e bytes) 000829 1787848555.108898 trunc from 0x025a18 to 0x024c1c (0xdfb bytes) 000830 1787848555.132171 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24c1d): Mirror 8: 0xb14396c8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24c1d): Mirror 8: 0xb14396c8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24c1d: skipped 000831 1787848555.445687 read 0x015297 thru 0x0171c1 (0x1f2b bytes) 000832 1787848555.452315 trunc from 0x024c1d to 0x00cb19 (0x18103 bytes) 000833 1787848555.478261 mapread 0x0075a9 thru 0x007729 (0x181 bytes) 000834 1787848555.484300 mapread 0x006fe5 thru 0x00cb19 (0x5b35 bytes) 000836 1787848555.500543 trunc from 0x0043b7 to 0x00cb19 (0x8763 bytes) 000837 1787848555.506339 read 0x001942 thru 0x00cb19 (0xb1d8 bytes) 000838 1787848555.521304 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000839 1787848555.730345 mapread 0x00b960 thru 0x00cb19 (0x11ba bytes) 000840 1787848555.747834 write 0x010b21 thru 0x01aa83 (0x9f63 bytes) 000843 1787848555.763457 fallocate 0x035304 thru 0x03ffff (0xacfc bytes) 000844 1787848555.780414 read 0x0070c3 thru 0x016ee6 (0xfe24 bytes) 000846 1787848555.803172 mapwrite 0x00e7b8 thru 0x014946 (0x618f bytes) 000847 1787848555.848469 trunc from 0x01aa84 to 0x037254 (0x1c7d1 bytes) 000848 1787848555.872127 fallocate 0x029018 thru 0x02e52d (0x5516 bytes) 000849 1787848555.882020 trunc from 0x00c826 to 0x01b611 (0xedec bytes) 000850 1787848555.895002 punch from 0x011ddf to 0x01c678 (0xa89a bytes) 000851 1787848555.900118 trunc from 0x037255 to 0x01a927 (0x1c92d bytes) 000852 1787848555.916141 fallocate 0x03cbdd thru 0x03ffff (0x3423 bytes) 000853 1787848555.921338 punch from 0x0056ef to 0x00ae95 (0x57a7 bytes) 000854 1787848555.926538 trunc from 0x01a928 to 0x01bc43 (0x131c bytes) 000855 1787848555.939419 mapwrite 0x03d2da thru 0x03ffff (0x2d26 bytes) 000856 1787848555.979030 punch from 0x012b1c to 0x02085b (0xdd40 bytes) 000857 1787848555.984098 punch from 0x025a2c to 0x032454 (0xca29 bytes) 000858 1787848555.989295 trunc from 0x01c89a to 0x025319 (0x8a80 bytes) 000859 1787848555.995126 trunc from 0x009192 to 0x00c670 (0x34df bytes) 000860 1787848555.999983 read 0x00d127 thru 0x019271 (0xc14b bytes) 000861 1787848556.013257 mapread 0x007d92 thru 0x01018b (0x83fa bytes) 000862 1787848556.014390 read 0x03fd1d thru 0x03ffff (0x2e3 bytes) 000863 1787848556.014998 trunc from 0x03a360 to 0x03ffff (0x5ca0 bytes) 000864 1787848556.018327 write 0x01f821 thru 0x0235bf (0x3d9f bytes) 000865 1787848556.020169 mapread 0x00e87d thru 0x01daf8 (0xf27c bytes) 000866 1787848556.021987 punch from 0x0111eb to 0x0141ca (0x2fe0 bytes) 000867 1787848556.027101 punch from 0x01fd78 to 0x02ae7d (0xb106 bytes) 000868 1787848556.040179 read 0x029c04 thru 0x02e3d5 (0x47d2 bytes) 000869 1787848556.053461 read 0x00cd5c thru 0x00fdf4 (0x3099 bytes) 000870 1787848556.054524 punch from 0x005d6b to 0x0073e2 (0x1678 bytes) 000872 1787848556.060954 read 0x01ac0f thru 0x0200c5 (0x54b7 bytes) 000873 1787848556.062609 trunc from 0x01c088 to 0x01d268 (0x11e1 bytes) 000874 1787848556.070074 mapread 0x03d118 thru 0x03ffff (0x2ee8 bytes) 000875 1787848556.074467 write 0x027846 thru 0x035168 (0xd923 bytes) 000876 1787848556.078801 write 0x00a371 thru 0x00b44f (0x10df bytes) 000877 1787848556.080458 write 0x006b32 thru 0x012a84 (0xbf53 bytes) 000878 1787848556.093969 read 0x02b985 thru 0x033142 (0x77be bytes) 000879 1787848556.094948 write 0x000889 thru 0x002d9d (0x2515 bytes) 000880 1787848556.097832 punch from 0x008e09 to 0x009a43 (0xc3b bytes) 000881 1787848556.111178 read 0x00b06c thru 0x0181a9 (0xd13e bytes) 000882 1787848556.112153 punch from 0x03da3f to 0x03ffff (0x25c1 bytes) 000883 1787848556.118248 trunc from 0x00e918 to 0x016ce7 (0x83d0 bytes) 000884 1787848556.124421 punch from 0x031207 to 0x03ffff (0xedf9 bytes) 000885 1787848556.130263 trunc from 0x02bb49 to 0x032693 (0x6b4b bytes) 000887 1787848556.138104 fallocate 0x02ea04 thru 0x033677 (0x4c74 bytes) 000888 1787848556.142851 read 0x038f42 thru 0x03c263 (0x3322 bytes) 000889 1787848556.167510 read 0x036730 thru 0x03ad53 (0x4624 bytes) 000890 1787848556.168118 mapread 0x023d6c thru 0x025b7f (0x1e14 bytes) 000891 1787848556.168776 write 0x03d722 thru 0x03f899 (0x2178 bytes) 000892 1787848556.169448 write 0x0128e0 thru 0x01ed6d (0xc48e bytes) 000894 1787848556.170920 punch from 0x03ea13 to 0x03ffff (0x15ed bytes) 000895 1787848556.184321 write 0x005c0f thru 0x01088b (0xac7d bytes) 000897 1787848556.186326 write 0x018941 thru 0x026baf (0xe26f bytes) 000898 1787848556.192222 mapread 0x026312 thru 0x029626 (0x3315 bytes) 000899 1787848556.195944 trunc from 0x040000 to 0x0180d5 (0x27f2a bytes) 000900 1787848556.207920 mapread 0x0023ce thru 0x00de13 (0xba46 bytes) 000901 1787848556.214470 MIRROR_EXTEND to 2 mirrors 000902 1787848556.824253 trunc from 0x0180d6 to 0x00e629 (0x9aac bytes) 000903 1787848556.918690 MIRROR_RESYNC 2 mirrors 000904 1787848557.623922 write 0x018e9d thru 0x01ee4a (0x5fae bytes) 000905 1787848557.684414 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1ee4b): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ee4b): Mirror 2: 0x39a2ea0e (stale) CRC-32 checksum value for chunk [0, 0x1ee4b): Mirror 1: 0xd0dde85a Mirror 3: 0xd0dde85a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ee4b: skipped 000908 1787848558.602461 trunc from 0x01ee4b to 0x02433f (0x54f5 bytes) 000909 1787848558.655613 mapread 0x00b97f thru 0x0113ba (0x5a3c bytes) 000910 1787848558.677420 read 0x011ce8 thru 0x01e49d (0xc7b6 bytes) 000911 1787848558.690845 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x24340): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24340): Mirror 2: 0xb50eff9b (stale) Mirror 3: 0x303fb2cb (stale) CRC-32 checksum value for chunk [0, 0x24340): Mirror 1: 0x303fb2cb Mirror 4: 0x303fb2cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24340: skipped 000912 1787848559.500536 mapwrite 0x0276e9 thru 0x0295c7 (0x1edf bytes) 000913 1787848559.616012 trunc from 0x0295c8 to 0x0256ee (0x3ed9 bytes) 000914 1787848559.641966 trunc from 0x0256ef to 0x03a2f4 (0x14c06 bytes) 000915 1787848559.668304 read 0x0304c5 thru 0x03a2f4 (0x9e30 bytes) 000916 1787848559.679201 trunc from 0x016f9f to 0x022da3 (0xbe05 bytes) 000918 1787848559.684907 mapread 0x01d7f9 thru 0x01e55d (0xd65 bytes) 000919 1787848559.695500 write 0x006869 thru 0x010f56 (0xa6ee bytes) 000920 1787848559.709255 trunc from 0x0208f6 to 0x02a9b2 (0xa0bd bytes) 000921 1787848559.717029 fallocate 0x03baab thru 0x03ffff (0x4555 bytes) 000922 1787848559.723473 punch from 0x01aae7 to 0x0276c7 (0xcbe1 bytes) 000923 1787848559.730137 mapwrite 0x015b9e thru 0x01bf5a (0x63bd bytes) 000924 1787848559.759294 write 0x00dc89 thru 0x016c45 (0x8fbd bytes) 000925 1787848559.760555 mapwrite 0x039af5 thru 0x03e753 (0x4c5f bytes) 000926 1787848559.805915 read 0x00b6e4 thru 0x00c962 (0x127f bytes) 000927 1787848559.806994 mapread 0x003938 thru 0x00d05c (0x9725 bytes) 000928 1787848559.817314 fallocate 0x035670 thru 0x038303 (0x2c94 bytes) 000929 1787848559.823981 fallocate 0x0352d8 thru 0x03c1f4 (0x6f1d bytes) 000930 1787848559.830105 write 0x01d199 thru 0x02c0ff (0xef67 bytes) 000932 1787848559.834210 punch from 0x00b0c4 to 0x01150c (0x6449 bytes) 000933 1787848559.847119 trunc from 0x03e754 to 0x023ca2 (0x1aab1 bytes) 000934 1787848559.868838 write 0x003bc5 thru 0x01156b (0xd9a7 bytes) 000935 1787848559.888109 MIRROR_RESYNC 4 mirrors 000936 1787848560.993648 fallocate 0x020267 thru 0x028be6 (0x8980 bytes) 000937 1787848561.027418 write 0x028f46 thru 0x038043 (0xf0fe bytes) 000938 1787848561.030687 trunc from 0x00ec47 to 0x015bc0 (0x6f7a bytes) 000939 1787848561.036930 mapread 0x033977 thru 0x038043 (0x46cd bytes) 000940 1787848561.038112 read 0x02d719 thru 0x02e1ec (0xad4 bytes) 000941 1787848561.039280 mapread 0x030ce2 thru 0x03491e (0x3c3d bytes) 000942 1787848561.041076 write 0x03c720 thru 0x03ffff (0x38e0 bytes) 000943 1787848561.043083 trunc from 0x040000 to 0x001ed3 (0x3e12c bytes) 000944 1787848561.066339 write 0x007635 thru 0x00e244 (0x6c10 bytes) 000945 1787848561.076342 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe245): Mirror 2: 0x1eeeb612 (stale) Mirror 3: 0x1eeeb612 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe245): Mirror 2: 0x1eeeb612 (stale) Mirror 3: 0x1eeeb612 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe245: skipped 000946 1787848561.404009 mapread 0x00ce1f thru 0x00e197 (0x1379 bytes) 000947 1787848561.436687 fallocate 0x036696 thru 0x03fab5 (0x9420 bytes) 000948 1787848561.443177 mapwrite 0x03ec90 thru 0x03ffff (0x1370 bytes) 000950 1787848561.518091 read 0x02eb6a thru 0x0387c9 (0x9c60 bytes) 000951 1787848561.541224 trunc from 0x040000 to 0x03da5a (0x25a5 bytes) 000952 1787848561.568485 mapread 0x00cca8 thru 0x014b84 (0x7edd bytes) 000953 1787848561.581027 mapwrite 0x0242e6 thru 0x02e9c3 (0xa6de bytes) 000954 1787848561.618796 punch from 0x00853f to 0x00d49e (0x4f60 bytes) 000955 1787848561.628232 trunc from 0x03319d to 0x03648c (0x32f0 bytes) 000956 1787848561.636525 write 0x030851 thru 0x03c021 (0xb7d1 bytes) 000957 1787848561.639050 trunc from 0x000d35 to 0x00ba2b (0xacf7 bytes) 000958 1787848561.646638 trunc from 0x02f701 to 0x032ee8 (0x37e8 bytes) 000960 1787848561.668346 write 0x035596 thru 0x03ffff (0xaa6a bytes) 000961 1787848561.672327 write 0x02f30b thru 0x031b8e (0x2884 bytes) 000962 1787848561.683929 mapread 0x01445e thru 0x01fc99 (0xb83c bytes) 000963 1787848561.693846 trunc from 0x0357b6 to 0x03b0a2 (0x58ed bytes) 000964 1787848561.715459 fallocate 0x023d7a thru 0x026fa5 (0x322c bytes) 000965 1787848561.721019 mapwrite 0x003e7d thru 0x006fad (0x3131 bytes) 000966 1787848561.745477 mapwrite 0x02b1f6 thru 0x0398c5 (0xe6d0 bytes) 000967 1787848561.780825 MIRROR_RESYNC 3 mirrors 000968 1787848562.647287 write 0x0164fa thru 0x01b3ad (0x4eb4 bytes) 000969 1787848562.690289 trunc from 0x040000 to 0x01489d (0x2b762 bytes) 000970 1787848562.715291 fallocate 0x0263a6 thru 0x028ce5 (0x2940 bytes) 000971 1787848562.720929 write 0x00e0a2 thru 0x00ee92 (0xdf1 bytes) 000972 1787848562.727251 mapread 0x00c15a thru 0x019a0f (0xd8b6 bytes) 000973 1787848562.754243 trunc from 0x028ce6 to 0x0125ee (0x166f7 bytes) 000974 1787848562.778091 MIRROR_RESYNC 3 mirrors 000976 1787848563.611358 trunc from 0x00d658 to 0x0125ee (0x4f97 bytes) 000977 1787848563.648462 trunc from 0x00161a to 0x00496a (0x3351 bytes) 000978 1787848563.657962 trunc from 0x00b549 to 0x00b6fa (0x1b2 bytes) 000979 1787848563.663044 read 0x008fea thru 0x0125ee (0x9605 bytes) 000980 1787848563.673352 fallocate 0x016435 thru 0x023264 (0xce30 bytes) 000981 1787848563.682668 trunc from 0x005749 to 0x0125ee (0xcea6 bytes) 000982 1787848563.687368 write 0x03af7a thru 0x03f060 (0x40e7 bytes) 000984 1787848563.698025 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f061): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f061): Mirror 2: 0x5d2c9169 (stale) Mirror 3: 0x5d2c9169 (stale) CRC-32 checksum value for chunk [0, 0x3f061): Mirror 1: 0x78716eb3 Mirror 4: 0x78716eb3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f061: skipped 000985 1787848564.782380 trunc from 0x03f061 to 0x02cd2f (0x12331 bytes) 000986 1787848564.844335 mapread 0x00c1da thru 0x00c825 (0x64c bytes) 000987 1787848564.855032 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cd30): Mirror 2: 0x64eb6d20 (stale) Mirror 4: 0x2b0b0d19 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cd30): Mirror 2: 0x64eb6d20 (stale) Mirror 4: 0x2b0b0d19 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cd30: skipped 000988 1787848565.181643 mapread 0x012fa5 thru 0x01a7f1 (0x784d bytes) 000989 1787848565.214643 fallocate 0x03f833 thru 0x03f992 (0x160 bytes) 000991 1787848565.223172 read 0x02752d thru 0x02a43b (0x2f0f bytes) 000992 1787848565.231176 trunc from 0x01c106 to 0x02ba95 (0xf990 bytes) 000993 1787848565.237990 trunc from 0x03f993 to 0x012b08 (0x2ce8a bytes) 000994 1787848565.262926 mapwrite 0x0151c3 thru 0x018759 (0x3597 bytes) 000995 1787848565.309974 read 0x010e27 thru 0x018759 (0x7933 bytes) 000996 1787848565.310943 trunc from 0x01875a to 0x03e9f4 (0x2629b bytes) 000997 1787848565.338101 mapread 0x010f20 thru 0x020371 (0xf452 bytes) 000998 1787848565.353255 punch from 0x01ce4a to 0x029939 (0xcaf0 bytes) 001000 1787848565.359199 write 0x022b1a thru 0x02f48c (0xc973 bytes) All operations completed A-OK! PASS 70a (92s) == sanity-flr test 71: check mirror extend parent fid ==== 12:36:11 (1787848571) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x165:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (13s) == sanity-flr test 72: check append on FLR file ========== 12:36:25 (1787848585) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.291379 s, 7.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141508 s, 7.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (13s) == sanity-flr test 200a: stress test ===================== 12:36:38 (1787848598) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1140352 .. Extending file size to 2474624 .. Extending file size to 8907616 .. Extending file size to 3553536 .. Extending file size to 4227840 .. Extending file size to 7538976 .. Extending file size to 9212416 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9104864 .. Extending file size to 9233632 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9176768 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 62501 62502 62505 62508 62509 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) 192.168.202.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:38:00 (1787848680) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2357984 .. Extending file size to 1382240 .. done Extending file size to 3967008 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6021120 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3256544 .. Extending file size to 7281312 .. Extending file size to 7697536 .. Extending file size to 4741344 .. Extending file size to 7280160 .. Extending file size to 7953280 .. Extending file size to 8872416 .. done Extending file size to 9324064 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9406240 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 64137 64138 64139 64142 64143 done 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) 192.168.202.122@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:39:23 (1787848763) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1268920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2032 1269272 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26164 3580856 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18032 3588988 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 44196 7169844 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2384 1268920 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2032 1269272 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 26164 3580856 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18032 3588988 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 44196 7169844 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.169596 s, 12.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.285291 s, 7.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x47f168b8 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) PASS 200c (22s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:39:47 (1787848787) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108621 s, 19.3 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 12:39:58 (1787848798) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x94:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x14f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } PASS 203 (11s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:40:09 (1787848809) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123379 s, 17.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (12s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:40:21 (1787848821) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109237 s, 19.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x151:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x151:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:40:32 (1787848832) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128323 s, 16.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:41:06 (1787848866) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg222-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108178 s, 9.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x153:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x153:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x154:0x0] } PASS 204d (50s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:41:56 (1787848916) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg222-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.199154 s, 10.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x155:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x153:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x155:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:42:51 (1787848971) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg222-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg222-server: Pool lustre.test_204f created Adding targets to pool oleg222-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg222-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.186136 s, 11.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1c6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg222-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg222-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg222-server: Pool lustre.test_204f destroyed PASS 204f (83s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:44:14 (1787849054) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:44:23 (1787849063) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:44:33 (1787849073) oleg222-server: Pool lustre.test_206 created Adding targets to pool oleg222-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg222-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg222-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg222-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg222-server: Pool lustre.test_206 destroyed PASS 206 (35s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:45:08 (1787849108) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131068 s, 8.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076799 s, 13.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xc64a86e5 Mirror 3: 0xc64a86e5 PASS 207 (15s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:45:23 (1787849123) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:45:27 (1787849127) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:45:32 (1787849132) Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0387106 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x20d0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1ef0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1ef0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1ef0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1940000 Read this (4 bytes): BBBB 192.168.202.122@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg222-client.virtnet /mnt/lustre2 (opts:) PASS 209a (46s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:46:18 (1787849178) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0275904 s, 297 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:46:27 (1787849187) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595513 s, 17.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb5 lmm_fid: [0x200000405:0xb5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb5 lmm_fid: [0x200000405:0xb5:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15f:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611317 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-27 12:46:29.000000000 -0400 Modify: 2026-08-27 12:46:29.000000000 -0400 Change: 2026-08-27 12:46:31.000000000 -0400 Birth: 2026-08-27 12:46:29.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:46:37 (1787849197) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362513 s, 28.9 MB/s oleg222-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg222-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg222-server: O/280000401/d0/160: File not found by ext2_lookup PASS 210b (29s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:47:06 (1787849226) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.315962 s, 130 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0317928 s, 129 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 27 12:47 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xba:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 27 12:47 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 12:47:17 (1787849237) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 12:47:26 (1787849246) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1999 sec =========== 12:47:37 (1787849257) === sanity-flr: start cleanup 12:47:40 (1787849260) === === sanity-flr: finish cleanup 12:47:51 (1787849271) ===