-----============= acceptance-small: sanity-flr ============----- Wed Sep 2 10:19:11 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg136-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 10:19:24 (1788358764) === oleg136-client.virtnet: executing check_config_client /mnt/lustre oleg136-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg136-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8e0cd0c04000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8e0cd0c04000.idle_timeout=debug disable quota as required oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 10:19:50 (1788358790) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8219] == sanity-flr test 0a: lfs mirror create with -N option == 10:19:52 (1788358792) PASS 0a (11s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 10:20:03 (1788358803) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 10:20:05 (1788358805) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 10:20:07 (1788358807) PASS 0d (16s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 10:20:23 (1788358823) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 10:20:25 (1788358825) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 10:20:27 (1788358827) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8e0cd0c04000.stats=clear osc.lustre-OST0001-osc-ffff8e0cd0c04000.stats=clear PASS 0g (6s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 10:20:33 (1788358833) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (7s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 10:20:40 (1788358840) PASS 0j (5s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 10:20:45 (1788358845) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 10:20:47 (1788358847) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (5s) == sanity-flr test 1: create components with setstripe options ========================================================== 10:20:53 (1788358853) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (6s) == sanity-flr test 2: create components from existing files ========================================================== 10:20:59 (1788358859) PASS 2 (5s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 10:21:04 (1788358864) PASS 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 10:21:09 (1788358869) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (4s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 10:21:14 (1788358874) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (3s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 10:21:19 (1788358879) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 10:21:22 (1788358882) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.170314 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0368764 s, 28.4 MB/s PASS 21 (4s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 10:21:26 (1788358886) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.124134 s, 42.2 MB/s PASS 22 (5s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 10:21:31 (1788358891) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (6s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 10:21:37 (1788358897) Stopping /mnt/lustre-ost1 (opts:) on oleg136-server oleg136-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e0cd0c04000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e0cd0c04000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg136-server: oleg136-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg136-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg136-server oleg136-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e0cd0c04000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e0cd0c04000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg136-server: oleg136-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg136-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (43s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 10:22:20 (1788358940) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.530888 s, 21.7 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg136-server oleg136-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e0cd0c04000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e0cd0c04000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg136-server: oleg136-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg136-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (41s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 10:23:01 (1788358981) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg136-server oleg136-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e0cd0c04000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e0cd0c04000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg136-server: oleg136-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg136-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg136-server oleg136-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e0cd0c04000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e0cd0c04000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg136-server: oleg136-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg136-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (117s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 10:24:58 (1788359098) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.99358 s, 17.5 MB/s -rw-r--r-- 1 root root 52428800 Sep 2 10:25 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.175s user 0m0.002s sys 0m0.193s Stopping /mnt/lustre-ost1 (opts:) on oleg136-server oleg136-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8e0cd0c04000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8e0cd0c04000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.917s user 0m0.000s sys 0m0.175s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg136-server: oleg136-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg136-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg136-server oleg136-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8e0cd0c04000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e0cd0c04000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.374s user 0m0.000s sys 0m0.206s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg136-server: oleg136-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg136-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg136-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e0cd0c04000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8e0cd0c04000.ost_server_uuid in FULL state after 0 sec PASS 33b (99s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 10:26:38 (1788359198) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 10:26:42 (1788359202) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 10:26:45 (1788359205) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (5s) == sanity-flr test 35: allow to write to mirrored files == 10:26:50 (1788359210) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 10:26:57 (1788359217) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.584591 s, 21.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 10:27:09 (1788359229) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402542 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302899 s, 34.6 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 10:27:16 (1788359236) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 10:27:23 (1788359243) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00260288 s, 393 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00668364 s, 153 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00118417 s, 865 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00137832 s, 743 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 10:27:33 (1788359253) create test files with size 15755504 .. f4394be8121377125b910ca6b0a4b08f - 14ce1e72041d0a53ae209bb9879cfca0 - 83af195aa2bced22535bdeeb0ebdec20 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg136-client.virtnet /mnt/lustre (opts:) Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 10:27:54 (1788359274) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 10:28:08 (1788359288) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 10:28:15 (1788359295) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501025 s, 20.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222315 s, 47.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 10:28:24 (1788359304) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0792843 s, 26.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.280864 s, 14.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.146324 s, 14.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (12s) == sanity-flr test 42: lfs mirror verify ================= 10:28:36 (1788359316) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 10:28:40 (1788359320) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 10:28:43 (1788359323) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0311291 s, 33.7 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 10:28:50 (1788359330) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.256625 s, 12.3 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 10:29:00 (1788359340) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 10:29:08 (1788359348) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.234334 s, 44.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg136-client.virtnet /mnt/lustre (opts:) Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (8s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 10:29:16 (1788359356) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8e0ccb5e0000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.335379 s, 31.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg136-client.virtnet /mnt/lustre (opts:) Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8e0cd0c06000.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 10:29:29 (1788359369) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (30s) == sanity-flr test 46: Verify setstripe --copy option ==== 10:29:59 (1788359399) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (32s) == sanity-flr test 47: Verify mirror obj alloc =========== 10:30:31 (1788359431) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 10:30:34 (1788359434) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.229101 s, 13.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.140806 s, 22.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xfd5f5c30 (nosync,stale) Mirror 2: 0xfd5f5c30 (nosync,stale) ** verify mirror 2 doesn't change original checksum: e228077fb1e4e809e012b972a16c9dc5 - mirror 1 checksum: 7a3eda38ea163eb8a4d3655234731508 - mirror 2 checksum: e228077fb1e4e809e012b972a16c9dc5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-02T10:30:36-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-02T10:30:36-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: e228077fb1e4e809e012b972a16c9dc5 - mirror 1 checksum: 7a3eda38ea163eb8a4d3655234731508 - mirror 2 checksum: 7a3eda38ea163eb8a4d3655234731508 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-02T10:30:36-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-02T10:30:36-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 10:30:44 (1788359444) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510814 s, 20.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 10:30:52 (1788359452) Failing mds1 on oleg136-server Stopping /mnt/lustre-mds1 (opts:) on oleg136-server 10:31:02 (1788359462) shut down facet: mds1 facet_host: oleg136-server facet_failover_host: oleg136-server Failover mds1 to oleg136-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg136-server: oleg136-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg136-client: oleg136-server: ssh exited with exit code 1 Started lustre-MDT0000 10:31:24 (1788359484) targets are mounted 10:31:24 (1788359484) facet_failover done oleg136-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623505 s, 16.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd3:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 10:31:39 (1788359499) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0471455 s, 434 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0237443 s, 863 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (8s) == sanity-flr test 50b: mirror rsync handles sparseness == 10:31:48 (1788359508) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0386539 s, 530 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0110018 s, 1.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (8s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 10:31:56 (1788359516) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00663248 s, 2.5 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 10:32:02 (1788359522) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0411931 s, 497 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (6s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 10:32:09 (1788359529) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00379073 s, 1.1 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0339651 s, 121 kB/s PASS 60a (5s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 10:32:14 (1788359534) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1b:0x0]: Invalid argument (22) PASS 60b (17s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 10:32:31 (1788359551) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 10:32:57 (1788359577) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x16:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 10:33:20 (1788359600) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 10:33:42 (1788359622) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 10:33:48 (1788359628) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57533 mirror split pid 57534 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 57533 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5070: 57534 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 10:34:54 (1788359694) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 9695 skipping zero length zero range 000002 1788359695.874498 mapwrite 0x0377d5 thru 0x03ffff (0x882b bytes) truncating to largest ever: 0x7dc4 000003 1788359695.941178 trunc from 0x040000 to 0x007dc3 (0x3823c bytes) 000004 1788359696.004132 write 0x004108 thru 0x00ebc5 (0xaabe bytes) 000005 1788359696.015883 punch from 0x00e7b7 to 0x00ebc5 (0x40f bytes) 000006 1788359696.035867 punch from 0x0084b3 to 0x00ebc5 (0x6713 bytes) 000007 1788359696.043187 read 0x00c86a thru 0x00ebc5 (0x235c bytes) 000009 1788359696.050246 write 0x0147a2 thru 0x01cd68 (0x85c7 bytes) fallocating to largest ever: 0x1a04f 000010 1788359696.052863 fallocate 0x00de16 thru 0x01a04e (0xc239 bytes) 000011 1788359696.059353 punch from 0x011a9a to 0x01cd68 (0xb2cf bytes) 000012 1788359696.073806 punch from 0x000b1a to 0x00446f (0x3956 bytes) 000013 1788359696.080193 write 0x001164 thru 0x0057f9 (0x4696 bytes) truncating to largest ever: 0x3b83f 000016 1788359696.089295 trunc from 0x01cd69 to 0x03b83e (0x1ead6 bytes) 000018 1788359696.111567 read 0x02af3f thru 0x037708 (0xc7ca bytes) 000019 1788359696.134807 fallocate 0x02a551 thru 0x035c3d (0xb6ed bytes) 000020 1788359696.140652 write 0x012237 thru 0x018a3f (0x6809 bytes) 000021 1788359696.142677 mapread 0x01b003 thru 0x01f526 (0x4524 bytes) 000022 1788359696.144660 mapread 0x0302cc thru 0x03b83e (0xb573 bytes) 000023 1788359696.153691 MIRROR_EXTEND to 2 mirrors 000024 1788359696.713816 mapread 0x02eeca thru 0x031425 (0x255c bytes) 000025 1788359696.750248 trunc from 0x03b83f to 0x001620 (0x3a21e bytes) 000026 1788359696.822002 fallocate 0x015e0b thru 0x020bbe (0xadb4 bytes) 000027 1788359696.828940 write 0x017f4e thru 0x01c853 (0x4906 bytes) 000028 1788359696.839360 trunc from 0x020bbf to 0x039be9 (0x1902b bytes) 000030 1788359696.860769 fallocate 0x01bd39 thru 0x026ed1 (0xb199 bytes) 000031 1788359696.866218 mapread 0x0154e4 thru 0x0243af (0xeecc bytes) 000032 1788359696.878942 mapread 0x01db8f thru 0x023ce4 (0x6156 bytes) 000034 1788359696.880161 mapread 0x0313d8 thru 0x0394cb (0x80f4 bytes) 000035 1788359696.889840 MIRROR_RESYNC 2 mirrors 000036 1788359697.414344 write 0x029daf thru 0x0305f9 (0x684b bytes) 000037 1788359697.443808 fallocate 0x00e64b thru 0x0116d4 (0x308a bytes) 000038 1788359697.447861 write 0x0249ae thru 0x02bf84 (0x75d7 bytes) 000039 1788359697.452371 trunc from 0x00a619 to 0x00b12f (0xb17 bytes) 000040 1788359697.456370 write 0x036f21 thru 0x03c9c3 (0x5aa3 bytes) 000041 1788359697.469619 MIRROR_RESYNC 2 mirrors 000042 1788359698.166882 mapread 0x00e2e4 thru 0x0192ef (0xb00c bytes) 000043 1788359698.217545 write 0x01c7d9 thru 0x01e3f7 (0x1c1f bytes) 000044 1788359698.253843 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c9c4): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c9c4): Mirror 2: 0x6e05a69f (stale) CRC-32 checksum value for chunk [0, 0x3c9c4): Mirror 1: 0xf9cbdf2f Mirror 3: 0xf9cbdf2f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c9c4: skipped 000045 1788359699.113769 punch from 0x025400 to 0x02c05b (0x6c5c bytes) 000046 1788359699.137821 write 0x020f7f thru 0x0260c0 (0x5142 bytes) 000047 1788359699.151235 read 0x0355b6 thru 0x03c9c3 (0x740e bytes) 000048 1788359699.163250 punch from 0x021a06 to 0x02c035 (0xa630 bytes) 000049 1788359699.183093 trunc from 0x00dcd2 to 0x010f94 (0x32c3 bytes) 000050 1788359699.195617 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c9c4): Mirror 3: 0xf9cbdf2f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c9c4): Mirror 3: 0xf9cbdf2f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c9c4: skipped 000051 1788359699.448574 read 0x02814a thru 0x02e7f8 (0x66af bytes) 000052 1788359699.458511 mapwrite 0x0032f0 thru 0x00f4d1 (0xc1e2 bytes) 000053 1788359699.517847 mapread 0x013239 thru 0x0166bf (0x3487 bytes) 000054 1788359699.529256 trunc from 0x005d70 to 0x0076b9 (0x194a bytes) 000055 1788359699.537195 trunc from 0x03c9c4 to 0x00995e (0x33065 bytes) 000056 1788359699.561602 punch from 0x007714 to 0x00995e (0x224b bytes) fallocating to largest ever: 0x3c0b0 000057 1788359699.567784 fallocate 0x033426 thru 0x03c0af (0x8c8a bytes) 000058 1788359699.573012 punch from 0x003689 to 0x009436 (0x5dae bytes) 000059 1788359699.582213 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) 000062 1788359699.721580 write 0x00ca45 thru 0x018f64 (0xc520 bytes) 000063 1788359699.727329 read 0x010511 thru 0x01197f (0x146f bytes) 000065 1788359699.728872 fallocate 0x0257e1 thru 0x02e2d5 (0x8af5 bytes) 000066 1788359699.736581 trunc from 0x01138e to 0x0180f3 (0x6d66 bytes) 000067 1788359699.756228 mapwrite 0x030d56 thru 0x03781f (0x6aca bytes) 000068 1788359699.827054 mapread 0x011c3c thru 0x0155b4 (0x3979 bytes) 000069 1788359699.845650 punch from 0x004310 to 0x0048e3 (0x5d4 bytes) 000071 1788359699.854130 fallocate 0x01c50a thru 0x028d7e (0xc875 bytes) 000072 1788359699.858909 trunc from 0x01ce44 to 0x02a4a0 (0xd65d bytes) 000073 1788359699.865655 trunc from 0x037820 to 0x0135e5 (0x2423a bytes) 000076 1788359699.888604 read 0x00607a thru 0x00ebe0 (0x8b67 bytes) 000077 1788359699.898519 mapwrite 0x030e1e thru 0x03997a (0x8b5d bytes) 000078 1788359699.938339 mapread 0x017a32 thru 0x019675 (0x1c44 bytes) 000079 1788359699.939457 trunc from 0x028fdf to 0x02cbf7 (0x3c19 bytes) 000080 1788359699.944917 read 0x014275 thru 0x016460 (0x21ec bytes) 000081 1788359699.945622 fallocate 0x0236d1 thru 0x02d5cd (0x9efd bytes) 000082 1788359699.950259 punch from 0x026891 to 0x036242 (0xf9b2 bytes) 000083 1788359699.955857 mapread 0x0302fb thru 0x03997a (0x9680 bytes) 000084 1788359699.963461 write 0x00976a thru 0x010657 (0x6eee bytes) fallocating to largest ever: 0x40000 000085 1788359699.965594 fallocate 0x03d0e3 thru 0x03ffff (0x2f1d bytes) 000088 1788359699.971540 trunc from 0x00612a to 0x00b447 (0x531e bytes) 000089 1788359699.991109 mapread 0x003bad thru 0x00c617 (0x8a6b bytes) 000091 1788359700.021634 mapread 0x02b89b thru 0x0375a1 (0xbd07 bytes) 000092 1788359700.023996 mapwrite 0x01ec50 thru 0x021cd8 (0x3089 bytes) 000093 1788359700.037636 mapread 0x00bcc7 thru 0x0125b4 (0x68ee bytes) 000094 1788359700.038473 mapwrite 0x021e7a thru 0x0257af (0x3936 bytes) 000095 1788359700.056993 mapread 0x03e77d thru 0x03ffff (0x1883 bytes) 000096 1788359700.068252 MIRROR_EXTEND to 2 mirrors 000097 1788359700.737919 mapwrite 0x017d9b thru 0x01c884 (0x4aea bytes) 000098 1788359700.875378 mapwrite 0x0235e7 thru 0x028fd4 (0x59ee bytes) 000099 1788359700.903516 trunc from 0x00d082 to 0x00ea53 (0x19d2 bytes) 000101 1788359700.908253 trunc from 0x018ff7 to 0x01bcb1 (0x2cbb bytes) 000102 1788359700.912230 mapwrite 0x0236b9 thru 0x025376 (0x1cbe bytes) 000103 1788359700.929380 trunc from 0x040000 to 0x02e1c7 (0x11e38 bytes) 000104 1788359701.000656 fallocate 0x00a244 thru 0x018a2d (0xe7ea bytes) 000105 1788359701.005314 trunc from 0x0147e8 to 0x015c48 (0x1461 bytes) 000106 1788359701.009194 mapread 0x00355c thru 0x00d58d (0xa032 bytes) 000109 1788359701.018327 write 0x006278 thru 0x00a733 (0x44bc bytes) 000110 1788359701.019892 write 0x030920 thru 0x03b888 (0xaf69 bytes) 000111 1788359701.021904 punch from 0x016587 to 0x01d4e3 (0x6f5d bytes) 000112 1788359701.025824 read 0x0187c6 thru 0x023de4 (0xb61f bytes) 000113 1788359701.033129 read 0x0358c1 thru 0x03b888 (0x5fc8 bytes) 000114 1788359701.033907 fallocate 0x02023a thru 0x02c3d0 (0xc197 bytes) 000115 1788359701.041247 mapread 0x029600 thru 0x03681b (0xd21c bytes) 000116 1788359701.053721 trunc from 0x03b889 to 0x01590e (0x25f7a bytes) 000119 1788359701.071803 punch from 0x005019 to 0x014f56 (0xff3e bytes) 000120 1788359701.084773 mapread 0x00f2e6 thru 0x014a19 (0x5734 bytes) 000121 1788359701.114638 punch from 0x00b230 to 0x01590e (0xa6df bytes) 000122 1788359701.120820 trunc from 0x00f88f to 0x01590e (0x6080 bytes) 000123 1788359701.127819 mapread 0x005334 thru 0x00b079 (0x5d46 bytes) 000124 1788359701.153610 trunc from 0x004418 to 0x00d5a6 (0x918f bytes) 000125 1788359701.161979 read 0x00606c thru 0x013b54 (0xdae9 bytes) 000126 1788359701.174708 punch from 0x010654 to 0x01590e (0x52bb bytes) 000127 1788359701.182678 MIRROR_RESYNC 2 mirrors 000129 1788359701.619262 mapwrite 0x00dd50 thru 0x019414 (0xb6c5 bytes) 000130 1788359701.756006 mapwrite 0x031b09 thru 0x03c153 (0xa64b bytes) 000131 1788359701.824543 trunc from 0x00be9e to 0x01652f (0xa692 bytes) 000132 1788359701.828913 trunc from 0x026f37 to 0x02f690 (0x875a bytes) 000133 1788359701.833891 trunc from 0x03c154 to 0x03aa96 (0x16bd bytes) 000134 1788359701.845451 read 0x00362b thru 0x00b194 (0x7b6a bytes) 000135 1788359701.850946 fallocate 0x005005 thru 0x008d54 (0x3d50 bytes) 000136 1788359701.854835 mapread 0x017b10 thru 0x025a62 (0xdf53 bytes) 000137 1788359701.866776 trunc from 0x01bc67 to 0x023d57 (0x80f1 bytes) 000138 1788359701.870812 mapwrite 0x037c2d thru 0x03ffff (0x83d3 bytes) 000139 1788359701.912670 trunc from 0x040000 to 0x01471f (0x2b8e0 bytes) 000140 1788359701.928385 trunc from 0x001234 to 0x010f71 (0xfd3e bytes) 000141 1788359701.933964 punch from 0x0105b4 to 0x01471f (0x416c bytes) 000142 1788359701.938139 write 0x035141 thru 0x0359c1 (0x881 bytes) 000144 1788359701.939554 fallocate 0x031e21 thru 0x03a056 (0x8236 bytes) 000145 1788359701.944187 fallocate 0x033190 thru 0x038ad3 (0x5944 bytes) 000146 1788359701.947394 mapwrite 0x028ea5 thru 0x02b846 (0x29a2 bytes) 000147 1788359701.966091 fallocate 0x03d09f thru 0x03ffff (0x2f61 bytes) 000148 1788359701.970089 punch from 0x02e028 to 0x032287 (0x4260 bytes) 000149 1788359701.974256 mapwrite 0x02609d thru 0x02ff9f (0x9f03 bytes) 000150 1788359702.009057 mapread 0x004a78 thru 0x00d103 (0x868c bytes) 000151 1788359702.021109 read 0x014dd1 thru 0x01c7cb (0x79fb bytes) 000152 1788359702.033190 trunc from 0x0359c2 to 0x0182d9 (0x1d6e8 bytes) 000153 1788359702.047037 trunc from 0x0182da to 0x01881d (0x544 bytes) 000154 1788359702.064493 trunc from 0x01881e to 0x02c731 (0x13f14 bytes) 000155 1788359702.082991 trunc from 0x018a88 to 0x021cd3 (0x924c bytes) 000156 1788359702.088940 trunc from 0x02c732 to 0x007c60 (0x24ad1 bytes) 000157 1788359702.119432 fallocate 0x0102c5 thru 0x01d049 (0xcd85 bytes) 000158 1788359702.126039 read 0x0060fd thru 0x007c60 (0x1b64 bytes) 000159 1788359702.127367 mapwrite 0x00aafb thru 0x014b76 (0xa07c bytes) 000160 1788359702.193803 MIRROR_RESYNC 2 mirrors 000161 1788359702.701344 trunc from 0x006b66 to 0x012152 (0xb5ed bytes) 000162 1788359702.736977 trunc from 0x014b77 to 0x036146 (0x215d0 bytes) 000163 1788359702.778105 MIRROR_RESYNC 2 mirrors 000164 1788359703.317623 mapwrite 0x02a3a8 thru 0x0304a8 (0x6101 bytes) 000165 1788359703.416817 trunc from 0x036147 to 0x01c6b1 (0x19a95 bytes) 000166 1788359703.432694 fallocate 0x01c94b thru 0x01d5aa (0xc60 bytes) 000167 1788359703.436787 punch from 0x008378 to 0x014f03 (0xcb8c bytes) 000168 1788359703.440551 write 0x001522 thru 0x009eaa (0x8989 bytes) 000169 1788359703.458902 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) 000170 1788359703.589734 punch from 0x000151 to 0x00f351 (0xf201 bytes) 000172 1788359703.596096 punch from 0x0193ab to 0x01c6b1 (0x3307 bytes) 000173 1788359703.600096 read 0x00ed81 thru 0x014234 (0x54b4 bytes) 000174 1788359703.609534 read 0x00f2b9 thru 0x013113 (0x3e5b bytes) 000175 1788359703.617020 MIRROR_EXTEND to 2 mirrors 000176 1788359704.139535 mapread 0x0050bc thru 0x0139cf (0xe914 bytes) 000177 1788359704.206943 punch from 0x014f99 to 0x01c6b1 (0x7719 bytes) 000178 1788359704.252492 write 0x039ece thru 0x03ffff (0x6132 bytes) 000179 1788359704.271103 trunc from 0x030026 to 0x030f68 (0xf43 bytes) 000180 1788359704.277685 punch from 0x039052 to 0x03ffff (0x6fae bytes) 000181 1788359704.308154 mapread 0x039eb1 thru 0x03ffff (0x614f bytes) 000182 1788359704.332245 punch from 0x03d6ed to 0x03ffff (0x2913 bytes) 000183 1788359704.346604 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe20eea22 Mirror 3: 0xe20eea22 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000184 1788359705.120473 punch from 0x03d7d8 to 0x03ffff (0x2828 bytes) 000185 1788359705.163247 mapread 0x02bd45 thru 0x0347fb (0x8ab7 bytes) 000186 1788359705.180648 trunc from 0x040000 to 0x019f82 (0x2607d bytes) 000187 1788359705.223951 mapread 0x009ffc thru 0x0100ef (0x60f4 bytes) 000188 1788359705.243577 punch from 0x0169f2 to 0x019f82 (0x3591 bytes) 000189 1788359705.248269 read 0x017ec9 thru 0x019f82 (0x20ba bytes) 000190 1788359705.253751 read 0x0086be thru 0x00dbc2 (0x5505 bytes) 000191 1788359705.254389 read 0x00009e thru 0x006020 (0x5f83 bytes) 000192 1788359705.266490 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x19f83): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19f83): Mirror 2: 0x54a3b085 (stale) Mirror 3: 0x54a3b085 (stale) CRC-32 checksum value for chunk [0, 0x19f83): Mirror 1: 0x54a3b085 Mirror 4: 0x54a3b085 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19f83: skipped 000193 1788359706.096099 read 0x00dd70 thru 0x019f82 (0xc213 bytes) 000194 1788359706.123075 fallocate 0x021692 thru 0x02641e (0x4d8d bytes) 000195 1788359706.202491 write 0x0350a7 thru 0x03d6a0 (0x85fa bytes) 000196 1788359706.208721 punch from 0x01a06b to 0x022e8c (0x8e22 bytes) 000197 1788359706.214377 punch from 0x0293f7 to 0x0348f6 (0xb500 bytes) 000198 1788359706.219999 mapwrite 0x02a144 thru 0x02c3b1 (0x226e bytes) 000199 1788359706.257942 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3d6a1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d6a1): Mirror 2: 0x5b171eef (stale) Mirror 3: 0x5b171eef (stale) Mirror 4: 0x5b171eef (stale) CRC-32 checksum value for chunk [0, 0x3d6a1): Mirror 1: 0xaa8bbaf5 Mirror 5: 0xaa8bbaf5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d6a1: skipped 000200 1788359707.127987 punch from 0x00c53d to 0x0110d3 (0x4b97 bytes) 000201 1788359707.156953 fallocate 0x010375 thru 0x018307 (0x7f93 bytes) 000202 1788359707.164513 write 0x024104 thru 0x025f7e (0x1e7b bytes) 000203 1788359707.183140 trunc from 0x03d6a1 to 0x02ba74 (0x11c2c bytes) 000204 1788359707.261614 MIRROR_RESYNC 5 mirrors 000205 1788359708.422533 punch from 0x026a4d to 0x02ba74 (0x5028 bytes) 000206 1788359708.451320 read 0x0176f8 thru 0x026f5e (0xf867 bytes) 000207 1788359708.463830 fallocate 0x035681 thru 0x039473 (0x3df3 bytes) 000208 1788359708.469563 fallocate 0x023e97 thru 0x02dce5 (0x9e4f bytes) 000209 1788359708.479161 MIRROR_RESYNC 5 mirrors 000210 1788359709.631873 trunc from 0x039474 to 0x00a194 (0x2f2df bytes) 000211 1788359709.669042 fallocate 0x03d881 thru 0x03ffff (0x277f bytes) 000213 1788359709.675842 trunc from 0x006be3 to 0x00827a (0x1698 bytes) 000214 1788359709.688155 mapwrite 0x0267ca thru 0x03384f (0xd086 bytes) 000215 1788359709.748132 punch from 0x02291f to 0x03272a (0xfe0c bytes) 000216 1788359709.756130 mapwrite 0x035a22 thru 0x03ffff (0xa5de bytes) 000217 1788359709.842169 punch from 0x025173 to 0x02fde8 (0xac76 bytes) 000218 1788359709.854496 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb2748b03 (stale) Mirror 4: 0xb2748b03 (stale) Mirror 5: 0xb2748b03 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb2748b03 (stale) Mirror 4: 0xb2748b03 (stale) Mirror 5: 0xb2748b03 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000219 1788359710.207586 MIRROR_RESYNC 4 mirrors 000220 1788359710.871164 fallocate 0x01daf9 thru 0x02d96d (0xfe75 bytes) 000221 1788359710.899035 trunc from 0x040000 to 0x026270 (0x19d8f bytes) 000222 1788359710.926546 trunc from 0x01a322 to 0x022ec9 (0x8ba8 bytes) 000223 1788359710.932894 write 0x0206fd thru 0x02daae (0xd3b2 bytes) 000224 1788359710.939238 write 0x031d98 thru 0x034d4f (0x2fb8 bytes) 000225 1788359710.941885 mapread 0x00eaeb thru 0x017635 (0x8b4b bytes) 000226 1788359710.962854 punch from 0x0042fe to 0x00f41a (0xb11d bytes) 000227 1788359710.968979 write 0x025bd2 thru 0x029f06 (0x4335 bytes) 000228 1788359710.970961 read 0x015ebd thru 0x017c92 (0x1dd6 bytes) 000229 1788359710.971962 mapwrite 0x002288 thru 0x00fcee (0xda67 bytes) 000230 1788359711.010615 trunc from 0x034d50 to 0x0218a8 (0x134a7 bytes) 000231 1788359711.031593 write 0x02f55e thru 0x03673f (0x71e2 bytes) 000232 1788359711.034152 fallocate 0x02610f thru 0x0277dd (0x16cf bytes) 000233 1788359711.046046 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x36740): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36740): Mirror 2: 0x3cda3454 (stale) Mirror 4: 0x3cda3454 (stale) Mirror 5: 0x3cda3454 (stale) CRC-32 checksum value for chunk [0, 0x36740): Mirror 1: 0xe19a5f30 Mirror 6: 0xe19a5f30 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36740: skipped 000234 1788359711.950597 punch from 0x0301cc to 0x034b5c (0x4991 bytes) 000236 1788359711.994561 trunc from 0x01ffdd to 0x0231c2 (0x31e6 bytes) 000237 1788359712.000023 trunc from 0x036740 to 0x029376 (0xd3c9 bytes) 000238 1788359712.022439 trunc from 0x017697 to 0x01d81e (0x6188 bytes) 000239 1788359712.032582 MIRROR_RESYNC 5 mirrors 000240 1788359712.813832 mapread 0x00b0f2 thru 0x00bc2a (0xb39 bytes) 000241 1788359712.820809 write 0x0361e4 thru 0x03ffff (0x9e1c bytes) 000242 1788359712.854121 read 0x014692 thru 0x023e22 (0xf791 bytes) 000243 1788359712.873059 MIRROR_RESYNC 5 mirrors 000244 1788359713.758027 read 0x023750 thru 0x02c552 (0x8e03 bytes) 000245 1788359713.766694 read 0x01eeff thru 0x022587 (0x3689 bytes) 000246 1788359713.776085 write 0x024b0a thru 0x0334c4 (0xe9bb bytes) 000247 1788359713.823030 mapread 0x00e0ef thru 0x017fee (0x9f00 bytes) 000248 1788359713.904642 mapread 0x03d6f2 thru 0x03ffff (0x290e bytes) 000249 1788359713.923522 punch from 0x023fb4 to 0x02be3b (0x7e88 bytes) 000250 1788359713.935505 punch from 0x027077 to 0x02ca9c (0x5a26 bytes) 000251 1788359713.941258 write 0x020744 thru 0x0233f9 (0x2cb6 bytes) 000252 1788359713.950014 read 0x000dca thru 0x003889 (0x2ac0 bytes) 000253 1788359713.958259 trunc from 0x040000 to 0x029872 (0x1678d bytes) 000254 1788359713.975873 punch from 0x0024d8 to 0x0031ca (0xcf3 bytes) 000255 1788359713.982789 read 0x00c5a1 thru 0x00d574 (0xfd4 bytes) 000256 1788359713.983659 fallocate 0x02b3f4 thru 0x02fcbf (0x48cc bytes) 000257 1788359713.995122 MIRROR_RESYNC 5 mirrors 000258 1788359715.258615 mapwrite 0x02400f thru 0x02fecf (0xbec1 bytes) 000259 1788359715.375012 trunc from 0x01aa5a to 0x0271ed (0xc794 bytes) 000260 1788359715.388047 mapwrite 0x027c54 thru 0x02ebfa (0x6fa7 bytes) 000261 1788359715.414204 fallocate 0x03dbbf thru 0x03f146 (0x1588 bytes) 000262 1788359715.419284 write 0x0301f4 thru 0x0331c6 (0x2fd3 bytes) 000263 1788359715.432197 read 0x002049 thru 0x00531f (0x32d7 bytes) 000264 1788359715.442639 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f147): Mirror 2: 0x88d2d2e6 (stale) Mirror 4: 0x88d2d2e6 (stale) Mirror 6: 0x88d2d2e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f147): Mirror 2: 0x88d2d2e6 (stale) Mirror 4: 0x88d2d2e6 (stale) Mirror 6: 0x88d2d2e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f147: skipped 000265 1788359715.783675 trunc from 0x00325a to 0x00b1cf (0x7f76 bytes) 000266 1788359715.788959 punch from 0x019dfa to 0x01d180 (0x3387 bytes) 000267 1788359715.794322 fallocate 0x00bb32 thru 0x0193cd (0xd89c bytes) 000268 1788359715.799777 write 0x03e9a2 thru 0x03ffff (0x165e bytes) 000269 1788359715.809938 punch from 0x016416 to 0x01dae0 (0x76cb bytes) 000270 1788359715.816281 trunc from 0x011dfb to 0x015844 (0x3a4a bytes) 000271 1788359715.822798 punch from 0x005ca3 to 0x011acb (0xbe29 bytes) 000272 1788359715.828340 write 0x019fab thru 0x025547 (0xb59d bytes) 000273 1788359715.841214 write 0x0054fe thru 0x006e87 (0x198a bytes) 000274 1788359715.853172 write 0x00484b thru 0x00d229 (0x89df bytes) 000275 1788359715.865766 read 0x02b20b thru 0x02e1b3 (0x2fa9 bytes) 000276 1788359715.873066 read 0x0078f9 thru 0x00fd53 (0x845b bytes) 000277 1788359715.892468 MIRROR_RESYNC 4 mirrors 000278 1788359716.999216 trunc from 0x022bac to 0x02556f (0x29c4 bytes) 000279 1788359717.025471 mapwrite 0x036dd6 thru 0x03b267 (0x4492 bytes) 000281 1788359717.072616 trunc from 0x0038f2 to 0x011bf7 (0xe306 bytes) 000283 1788359717.080319 punch from 0x025c6e to 0x02c9fd (0x6d90 bytes) 000284 1788359717.090862 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x49ed2ce6 (stale) Mirror 6: 0x49ed2ce6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x49ed2ce6 (stale) Mirror 6: 0x49ed2ce6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000285 1788359717.519301 punch from 0x0136e1 to 0x01e187 (0xaaa7 bytes) 000286 1788359717.526157 trunc from 0x040000 to 0x01b9a4 (0x2465b bytes) 000287 1788359717.542912 punch from 0x01332b to 0x01b9a4 (0x867a bytes) 000288 1788359717.552389 mapread 0x008ed1 thru 0x00f10d (0x623d bytes) 000289 1788359717.577583 punch from 0x011c77 to 0x01b9a4 (0x9d2e bytes) 000290 1788359717.583759 punch from 0x018309 to 0x01b9a4 (0x369c bytes) 000291 1788359717.590241 mapwrite 0x033b6a thru 0x03ffff (0xc496 bytes) 000292 1788359717.638595 write 0x0016ee thru 0x002fdd (0x18f0 bytes) 000293 1788359717.649648 read 0x0166b1 thru 0x01ae4f (0x479f bytes) 000294 1788359717.656658 read 0x011d99 thru 0x021a32 (0xfc9a bytes) 000295 1788359717.664932 write 0x03a1a8 thru 0x03ffff (0x5e58 bytes) 000297 1788359717.666548 fallocate 0x01b4da thru 0x021c08 (0x672f bytes) 000299 1788359717.671060 punch from 0x01881e to 0x024894 (0xc077 bytes) 000300 1788359717.675289 trunc from 0x011428 to 0x013c3d (0x2816 bytes) 000301 1788359717.679091 fallocate 0x014214 thru 0x01d3c7 (0x91b4 bytes) 000302 1788359717.682789 write 0x0321ab thru 0x03dae7 (0xb93d bytes) 000303 1788359717.684379 read 0x031f3b thru 0x033ae9 (0x1baf bytes) 000304 1788359717.689205 MIRROR_RESYNC 3 mirrors 000305 1788359718.404332 read 0x00b8b2 thru 0x01986b (0xdfba bytes) 000306 1788359718.415612 MIRROR_EXTEND to 4 mirrors 000307 1788359719.103786 trunc from 0x0236e4 to 0x030315 (0xcc32 bytes) 000308 1788359719.173079 fallocate 0x031e24 thru 0x03366c (0x1849 bytes) 000309 1788359719.188550 write 0x0214dc thru 0x0251cb (0x3cf0 bytes) 000310 1788359719.212307 read 0x03003e thru 0x039110 (0x90d3 bytes) 000311 1788359719.228794 trunc from 0x02632b to 0x028900 (0x25d6 bytes) 000312 1788359719.237532 fallocate 0x02b38f thru 0x033057 (0x7cc9 bytes) 000313 1788359719.264829 MIRROR_RESYNC 4 mirrors 000314 1788359720.229131 mapread 0x019281 thru 0x01f975 (0x66f5 bytes) 000315 1788359720.250849 fallocate 0x0251ca thru 0x02ac5f (0x5a96 bytes) 000316 1788359720.290186 write 0x007403 thru 0x009614 (0x2212 bytes) 000317 1788359720.306330 punch from 0x033c58 to 0x03ffff (0xc3a8 bytes) 000319 1788359720.316734 mapwrite 0x0388cf thru 0x03dbdd (0x530f bytes) 000320 1788359720.343014 read 0x016424 thru 0x01a41a (0x3ff7 bytes) 000322 1788359720.352231 trunc from 0x040000 to 0x0220f1 (0x1df0e bytes) 000323 1788359720.398280 trunc from 0x01fe5f to 0x0220f1 (0x2293 bytes) 000324 1788359720.406304 punch from 0x00cc4e to 0x00da02 (0xdb5 bytes) 000325 1788359720.410270 trunc from 0x0220f2 to 0x03b502 (0x19411 bytes) 000326 1788359720.420752 write 0x035534 thru 0x035991 (0x45e bytes) 000327 1788359720.430747 read 0x00d3b2 thru 0x013d34 (0x6983 bytes) 000328 1788359720.448583 punch from 0x013932 to 0x01822c (0x48fb bytes) 000329 1788359720.454280 trunc from 0x0222c9 to 0x0227f1 (0x529 bytes) 000330 1788359720.462727 trunc from 0x03361e to 0x03b502 (0x7ee5 bytes) 000331 1788359720.472232 trunc from 0x03b503 to 0x02a9b4 (0x10b4e bytes) 000332 1788359720.486752 write 0x0383a5 thru 0x03f4d3 (0x712f bytes) 000333 1788359720.488350 write 0x00e564 thru 0x01c440 (0xdedd bytes) 000334 1788359720.498114 MIRROR_RESYNC 4 mirrors 000335 1788359721.632831 fallocate 0x01f6bf thru 0x028cb3 (0x95f5 bytes) 000336 1788359721.661862 trunc from 0x01498e to 0x01e760 (0x9dd3 bytes) 000337 1788359721.670813 read 0x00dfa3 thru 0x011825 (0x3883 bytes) 000340 1788359721.678843 trunc from 0x005ab8 to 0x00828a (0x27d3 bytes) 000341 1788359721.685015 mapwrite 0x01fa18 thru 0x0265b4 (0x6b9d bytes) 000342 1788359721.724894 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f4d4): Mirror 6: 0x647bb5bb (stale) Mirror 7: 0x647bb5bb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f4d4): Mirror 6: 0x647bb5bb (stale) Mirror 7: 0x647bb5bb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f4d4: skipped 000344 1788359722.011703 read 0x031427 thru 0x03e1cf (0xcda9 bytes) 000345 1788359722.022260 mapread 0x02866e thru 0x0323c0 (0x9d53 bytes) 000347 1788359722.044729 write 0x03f984 thru 0x03ffff (0x67c bytes) 000348 1788359722.054931 trunc from 0x040000 to 0x0031b3 (0x3ce4c bytes) 000349 1788359722.072185 read 0x0015f0 thru 0x0031b3 (0x1bc4 bytes) 000350 1788359722.082649 trunc from 0x0031b4 to 0x0320d6 (0x2ef23 bytes) 000351 1788359722.091262 read 0x0191fa thru 0x0213aa (0x81b1 bytes) 000352 1788359722.098072 write 0x03c4c8 thru 0x03ffff (0x3b38 bytes) 000353 1788359722.098938 trunc from 0x040000 to 0x007a3a (0x385c5 bytes) 000355 1788359722.108520 write 0x014d5b thru 0x02278d (0xda33 bytes) 000357 1788359722.110338 mapwrite 0x02d188 thru 0x03292b (0x57a4 bytes) 000359 1788359722.179832 trunc from 0x02ef19 to 0x03292b (0x3a13 bytes) 000360 1788359722.190141 write 0x03173e thru 0x032d61 (0x1624 bytes) 000361 1788359722.205007 read 0x00c11d thru 0x011bfd (0x5ae1 bytes) 000362 1788359722.222215 punch from 0x003899 to 0x007803 (0x3f6b bytes) 000363 1788359722.227538 read 0x01c5d3 thru 0x029f8f (0xd9bd bytes) 000364 1788359722.228096 read 0x000c50 thru 0x00e74e (0xdaff bytes) 000365 1788359722.234773 trunc from 0x032d62 to 0x02b844 (0x751d bytes) 000366 1788359722.258780 trunc from 0x005aad to 0x013699 (0xdbed bytes) 000367 1788359722.266648 mapread 0x01693b thru 0x020c7f (0xa345 bytes) 000368 1788359722.270260 punch from 0x00592f to 0x006e5b (0x152d bytes) 000369 1788359722.274860 write 0x0240f1 thru 0x02ccde (0x8bee bytes) 000370 1788359722.277141 write 0x0057ec thru 0x0132a2 (0xdab7 bytes) 000371 1788359722.289143 read 0x02966a thru 0x02ccde (0x3675 bytes) 000372 1788359722.295010 MIRROR_RESYNC 3 mirrors 000373 1788359723.093574 mapread 0x01fe42 thru 0x02c074 (0xc233 bytes) 000374 1788359723.128288 mapread 0x005161 thru 0x00597c (0x81c bytes) 000375 1788359723.138532 mapread 0x009a05 thru 0x009ebe (0x4ba bytes) 000376 1788359723.139018 trunc from 0x024a43 to 0x02c6db (0x7c99 bytes) 000377 1788359723.173228 trunc from 0x02ccdf to 0x024239 (0x8aa5 bytes) 000378 1788359723.214260 read 0x0059d3 thru 0x00e77a (0x8da8 bytes) 000379 1788359723.224835 trunc from 0x01b4c4 to 0x024239 (0x8d76 bytes) 000380 1788359723.235453 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2423a): Mirror 7: 0xe0845899 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2423a): Mirror 7: 0xe0845899 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2423a: skipped 000381 1788359723.466824 trunc from 0x02423a to 0x0282a3 (0x406a bytes) 000382 1788359723.484691 fallocate 0x016fdd thru 0x01d702 (0x6726 bytes) 000383 1788359723.490763 punch from 0x001ba8 to 0x009628 (0x7a81 bytes) 000385 1788359723.497557 write 0x02236b thru 0x030e3c (0xead2 bytes) 000386 1788359723.507866 mapwrite 0x02a6bc thru 0x034ae4 (0xa429 bytes) 000387 1788359723.557342 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x34ae5): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34ae5): Mirror 7: 0x63f69c40 (stale) CRC-32 checksum value for chunk [0, 0x34ae5): Mirror 1: 0x7c33f304 Mirror 8: 0x7c33f304 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34ae5: skipped 000388 1788359724.341160 punch from 0x0207e7 to 0x027c82 (0x749c bytes) 000390 1788359724.377062 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34ae5): Mirror 8: 0x7c33f304 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34ae5): Mirror 8: 0x7c33f304 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34ae5: skipped 000391 1788359724.612213 fallocate 0x029664 thru 0x03229b (0x8c38 bytes) 000392 1788359724.616453 trunc from 0x034ae5 to 0x038070 (0x358c bytes) 000393 1788359724.671133 fallocate 0x023f73 thru 0x02f6c8 (0xb756 bytes) 000394 1788359724.676623 fallocate 0x00c36a thru 0x00ec78 (0x290f bytes) 000395 1788359724.680805 mapread 0x020300 thru 0x02f8a1 (0xf5a2 bytes) 000396 1788359724.723939 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38071): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38071): Mirror 8: 0xf7a5c5d4 (stale) CRC-32 checksum value for chunk [0, 0x38071): Mirror 1: 0x4b02ecfa Mirror 9: 0x4b02ecfa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38071: skipped 000398 1788359725.667068 trunc from 0x01fc0e to 0x022fc0 (0x33b3 bytes) 000399 1788359725.723481 mapread 0x02c0cd thru 0x02fd4a (0x3c7e bytes) 000400 1788359725.745492 write 0x0380ea thru 0x03ffff (0x7f16 bytes) 000401 1788359725.761128 mapread 0x020cd8 thru 0x02f586 (0xe8af bytes) 000403 1788359725.770184 trunc from 0x040000 to 0x03d1eb (0x2e14 bytes) 000405 1788359725.803554 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d1ec): Mirror 8: 0x4d104722 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1ec): Mirror 8: 0x4d104722 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1ec: skipped 000406 1788359726.134643 read 0x008fb5 thru 0x014457 (0xb4a3 bytes) 000407 1788359726.150319 fallocate 0x038585 thru 0x03ffff (0x7a7b bytes) 000408 1788359726.156698 mapwrite 0x025ac6 thru 0x029119 (0x3654 bytes) 000409 1788359726.211240 read 0x03f2b8 thru 0x03ffff (0xd48 bytes) 000410 1788359726.233134 mapwrite 0x020e35 thru 0x030020 (0xf1ec bytes) 000412 1788359726.265958 read 0x039e49 thru 0x03ffff (0x61b7 bytes) 000413 1788359726.266519 fallocate 0x013603 thru 0x01d20f (0x9c0d bytes) 000415 1788359726.272714 punch from 0x039c32 to 0x03ffff (0x63ce bytes) 000416 1788359726.277716 punch from 0x00e3c3 to 0x017d9e (0x99dc bytes) 000417 1788359726.285964 read 0x038405 thru 0x03ffff (0x7bfb bytes) 000418 1788359726.294452 read 0x023077 thru 0x026388 (0x3312 bytes) 000419 1788359726.295021 trunc from 0x040000 to 0x015ab1 (0x2a54e bytes) 000420 1788359726.318601 punch from 0x0010e6 to 0x003d53 (0x2c6e bytes) 000421 1788359726.323368 trunc from 0x015ab2 to 0x03da2c (0x27f7b bytes) 000422 1788359726.344936 write 0x0069ef thru 0x016416 (0xfa28 bytes) 000423 1788359726.357119 mapwrite 0x0348ae thru 0x03594f (0x10a2 bytes) 000424 1788359726.396363 write 0x015d84 thru 0x01f641 (0x98be bytes) 000426 1788359726.399304 mapwrite 0x029f6e thru 0x03125e (0x72f1 bytes) 000427 1788359726.423059 mapwrite 0x029fac thru 0x03884e (0xe8a3 bytes) 000428 1788359726.451941 MIRROR_RESYNC 2 mirrors 000429 1788359727.061549 trunc from 0x03da2d to 0x038fd6 (0x4a56 bytes) 000431 1788359727.127325 fallocate 0x00f359 thru 0x010a9f (0x1747 bytes) 000432 1788359727.135530 mapwrite 0x011aba thru 0x0190c3 (0x760a bytes) 000433 1788359727.175763 mapwrite 0x029317 thru 0x02f09d (0x5d87 bytes) 000434 1788359727.199260 read 0x020024 thru 0x0221a9 (0x2186 bytes) 000435 1788359727.204804 read 0x01b362 thru 0x028299 (0xcf38 bytes) 000436 1788359727.233218 fallocate 0x036739 thru 0x03fb58 (0x9420 bytes) 000438 1788359727.240511 trunc from 0x01dd3e to 0x020f54 (0x3217 bytes) 000440 1788359727.248290 punch from 0x012e2f to 0x013c3a (0xe0c bytes) 000441 1788359727.253390 read 0x0364cc thru 0x036e4c (0x981 bytes) 000442 1788359727.254854 trunc from 0x038fd7 to 0x03079c (0x883a bytes) 000443 1788359727.277014 mapread 0x00526d thru 0x005c70 (0xa04 bytes) 000444 1788359727.277747 mapread 0x013e4a thru 0x017821 (0x39d8 bytes) 000445 1788359727.285776 mapread 0x019570 thru 0x01991d (0x3ae bytes) 000446 1788359727.287068 write 0x0019d9 thru 0x00da50 (0xc078 bytes) 000447 1788359727.288221 trunc from 0x03079d to 0x025eb6 (0xa8e6 bytes) 000448 1788359727.323863 write 0x01648e thru 0x022d63 (0xc8d6 bytes) 000449 1788359727.330554 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) 000450 1788359727.512987 trunc from 0x00e745 to 0x017a02 (0x92be bytes) 000451 1788359727.518158 trunc from 0x0152a3 to 0x01ada6 (0x5b04 bytes) 000452 1788359727.528273 mapread 0x01df84 thru 0x025eb6 (0x7f33 bytes) 000453 1788359727.544017 trunc from 0x025eb7 to 0x022337 (0x3b7f bytes) 000454 1788359727.561120 fallocate 0x014d1c thru 0x0150ae (0x393 bytes) 000455 1788359727.566159 punch from 0x01eeb5 to 0x022337 (0x3483 bytes) 000456 1788359727.575090 MIRROR_EXTEND to 2 mirrors 000457 1788359727.982070 punch from 0x00d59a to 0x017db8 (0xa81f bytes) 000458 1788359728.034760 trunc from 0x00a3b6 to 0x017c9f (0xd8ea bytes) 000459 1788359728.051166 mapread 0x00e01e thru 0x016208 (0x81eb bytes) 000460 1788359728.091366 punch from 0x006bc1 to 0x00e4f6 (0x7936 bytes) 000461 1788359728.097203 mapread 0x00ffd2 thru 0x01b90e (0xb93d bytes) 000462 1788359728.108378 punch from 0x01062a to 0x015e43 (0x581a bytes) 000463 1788359728.113614 write 0x00d695 thru 0x010e13 (0x377f bytes) 000464 1788359728.126492 mapwrite 0x02e21f thru 0x03998a (0xb76c bytes) 000465 1788359728.208213 write 0x014687 thru 0x01ee96 (0xa810 bytes) 000466 1788359728.219908 MIRROR_RESYNC 2 mirrors 000467 1788359728.979175 read 0x01d555 thru 0x020be0 (0x368c bytes) 000468 1788359728.989642 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) 000472 1788359729.095128 punch from 0x02857e to 0x028f78 (0x9fb bytes) 000474 1788359729.108795 mapread 0x000e5d thru 0x008118 (0x72bc bytes) 000475 1788359729.128790 trunc from 0x03998b to 0x006839 (0x33151 bytes) 000476 1788359729.174333 read 0x0001ac thru 0x006839 (0x668e bytes) 000477 1788359729.174950 mapwrite 0x01f8db thru 0x02b00d (0xb733 bytes) 000478 1788359729.219058 write 0x029839 thru 0x03667e (0xce46 bytes) 000479 1788359729.221634 punch from 0x02804e to 0x030ec9 (0x8e7c bytes) 000480 1788359729.235484 read 0x004c88 thru 0x00c163 (0x74dc bytes) 000481 1788359729.248505 read 0x007793 thru 0x013b23 (0xc391 bytes) 000483 1788359729.249514 punch from 0x00d309 to 0x019328 (0xc020 bytes) 000484 1788359729.253117 write 0x02a756 thru 0x037c93 (0xd53e bytes) 000485 1788359729.254391 punch from 0x017a25 to 0x02624b (0xe827 bytes) 000488 1788359729.259140 punch from 0x01bb59 to 0x01bbed (0x95 bytes) 000489 1788359729.263163 trunc from 0x02f8e2 to 0x037c93 (0x83b2 bytes) 000492 1788359729.276442 write 0x019acd thru 0x01f827 (0x5d5b bytes) 000493 1788359729.287774 mapread 0x01f9e7 thru 0x02c200 (0xc81a bytes) 000494 1788359729.289198 read 0x035a2e thru 0x036a76 (0x1049 bytes) 000495 1788359729.289811 mapread 0x027254 thru 0x033b38 (0xc8e5 bytes) 000496 1788359729.291504 read 0x01cf36 thru 0x024c92 (0x7d5d bytes) 000498 1788359729.303466 MIRROR_EXTEND to 2 mirrors 000499 1788359729.887891 punch from 0x02a2a7 to 0x02e119 (0x3e73 bytes) 000500 1788359729.923373 fallocate 0x03f96e thru 0x03ffff (0x692 bytes) 000501 1788359729.928312 trunc from 0x040000 to 0x008f5b (0x370a4 bytes) 000502 1788359729.976980 write 0x003645 thru 0x008e08 (0x57c4 bytes) 000503 1788359729.988298 punch from 0x0063ae to 0x008f5b (0x2bae bytes) 000504 1788359730.017979 trunc from 0x008f5c to 0x03689e (0x2d943 bytes) 000505 1788359730.035198 punch from 0x00df52 to 0x016edd (0x8f8c bytes) 000506 1788359730.041047 punch from 0x010c1e to 0x01b6fc (0xaadf bytes) 000507 1788359730.046982 fallocate 0x022bcb thru 0x031e31 (0xf267 bytes) 000508 1788359730.057615 mapwrite 0x02341b thru 0x02bbd1 (0x87b7 bytes) 000509 1788359730.099175 punch from 0x0229c1 to 0x02b12f (0x876f bytes) 000510 1788359730.108894 MIRROR_RESYNC 2 mirrors 000511 1788359730.695978 mapread 0x02affb thru 0x03566a (0xa670 bytes) 000512 1788359730.716622 mapwrite 0x0339e9 thru 0x035303 (0x191b bytes) 000513 1788359730.779230 punch from 0x004129 to 0x01190a (0xd7e2 bytes) 000514 1788359730.788748 mapwrite 0x008bfd thru 0x00cfa5 (0x43a9 bytes) 000515 1788359730.822113 mapwrite 0x03e4f1 thru 0x03ffff (0x1b0f bytes) 000516 1788359730.887110 read 0x0082a5 thru 0x015684 (0xd3e0 bytes) 000517 1788359730.897587 mapread 0x00ae59 thru 0x00b6e3 (0x88b bytes) 000518 1788359730.899071 fallocate 0x004e05 thru 0x00688c (0x1a88 bytes) 000519 1788359730.916203 MIRROR_RESYNC 2 mirrors 000520 1788359731.582975 mapread 0x03edb5 thru 0x03ffff (0x124b bytes) 000521 1788359731.593506 write 0x027260 thru 0x02d0af (0x5e50 bytes) 000522 1788359731.617228 fallocate 0x03bfa2 thru 0x03ffff (0x405e bytes) 000523 1788359731.622481 punch from 0x02dc16 to 0x030b77 (0x2f62 bytes) 000524 1788359731.634342 mapwrite 0x02cfe7 thru 0x02e042 (0x105c bytes) 000525 1788359731.652799 mapread 0x039455 thru 0x03ffff (0x6bab bytes) 000526 1788359731.662098 read 0x013833 thru 0x0188dd (0x50ab bytes) 000527 1788359731.672981 read 0x001b1f thru 0x00bcfb (0xa1dd bytes) 000528 1788359731.682760 trunc from 0x040000 to 0x0370e7 (0x8f18 bytes) 000529 1788359731.700572 trunc from 0x0370e8 to 0x02afce (0xc119 bytes) 000530 1788359731.718908 fallocate 0x0144ad thru 0x014a05 (0x559 bytes) 000533 1788359731.728880 fallocate 0x02438f thru 0x02b184 (0x6df6 bytes) 000534 1788359731.734720 mapwrite 0x0350d7 thru 0x035b75 (0xa9f bytes) 000535 1788359731.767907 mapread 0x00cf6a thru 0x010cd9 (0x3d70 bytes) 000536 1788359731.770046 mapwrite 0x0374e6 thru 0x039a3a (0x2555 bytes) 000537 1788359731.812083 mapwrite 0x033295 thru 0x0335fd (0x369 bytes) 000538 1788359731.830065 mapread 0x015544 thru 0x024e36 (0xf8f3 bytes) 000539 1788359731.831816 read 0x006d7d thru 0x00b4e2 (0x4766 bytes) 000540 1788359731.833744 write 0x037c46 thru 0x03f7b3 (0x7b6e bytes) 000541 1788359731.836471 mapwrite 0x02b8fe thru 0x0316c3 (0x5dc6 bytes) 000542 1788359731.861976 punch from 0x019871 to 0x0227ea (0x8f7a bytes) 000544 1788359731.871691 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f7b4): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f7b4): Mirror 2: 0x921b6a38 (stale) CRC-32 checksum value for chunk [0, 0x3f7b4): Mirror 1: 0x7fd8d584 Mirror 3: 0x7fd8d584 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f7b4: skipped 000545 1788359732.770617 punch from 0x02711a to 0x02b4ab (0x4392 bytes) 000546 1788359732.804651 trunc from 0x03f7b4 to 0x03ca38 (0x2d7b bytes) 000547 1788359732.832969 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ca39): Mirror 3: 0xc68da0f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ca39): Mirror 3: 0xc68da0f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ca39: skipped 000549 1788359733.052698 fallocate 0x010571 thru 0x013e71 (0x3901 bytes) 000550 1788359733.058793 trunc from 0x03ca39 to 0x039287 (0x37b1 bytes) 000551 1788359733.089128 write 0x029317 thru 0x037c13 (0xe8fd bytes) 000552 1788359733.112456 trunc from 0x01a6d2 to 0x02779d (0xd0cc bytes) 000553 1788359733.117365 mapwrite 0x029c26 thru 0x02f435 (0x5810 bytes) 000554 1788359733.148032 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39288): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39288): Mirror 3: 0xb35431ef (stale) CRC-32 checksum value for chunk [0, 0x39288): Mirror 1: 0x5e93a3e3 Mirror 4: 0x5e93a3e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39288: skipped 000555 1788359733.851390 fallocate 0x00446a thru 0x00b2f5 (0x6e8c bytes) 000557 1788359733.891616 trunc from 0x02631a to 0x02da0b (0x76f2 bytes) 000558 1788359733.900561 write 0x02beae thru 0x02f70d (0x3860 bytes) 000559 1788359733.914689 punch from 0x008872 to 0x00e991 (0x6120 bytes) 000560 1788359733.921613 read 0x029f03 thru 0x036d34 (0xce32 bytes) 000561 1788359733.934371 mapwrite 0x03aed5 thru 0x03ffff (0x512b bytes) 000562 1788359734.034168 write 0x013203 thru 0x01642b (0x3229 bytes) 000563 1788359734.053767 mapread 0x035374 thru 0x03e33f (0x8fcc bytes) 000564 1788359734.059986 read 0x004a7e thru 0x012429 (0xd9ac bytes) 000565 1788359734.081374 mapwrite 0x01dd44 thru 0x02b4b4 (0xd771 bytes) 000566 1788359734.114040 read 0x0124d3 thru 0x01ad42 (0x8870 bytes) 000567 1788359734.126596 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x75170f42 (stale) Mirror 4: 0x87a44ddf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x39928418 Mirror 5: 0x39928418 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000568 1788359734.877066 write 0x03b806 thru 0x03ffff (0x47fa bytes) 000569 1788359734.904631 mapwrite 0x037899 thru 0x03e16e (0x68d6 bytes) 000571 1788359734.956251 MIRROR_RESYNC 4 mirrors 000572 1788359736.170110 mapread 0x02d185 thru 0x03ce33 (0xfcaf bytes) 000573 1788359736.194185 read 0x02f473 thru 0x03eb1f (0xf6ad bytes) 000575 1788359736.194952 read 0x011a14 thru 0x01737b (0x5968 bytes) 000576 1788359736.212536 trunc from 0x0193cb to 0x01be6c (0x2aa2 bytes) 000577 1788359736.239445 fallocate 0x020c81 thru 0x02432b (0x36ab bytes) 000579 1788359736.244946 trunc from 0x040000 to 0x0219be (0x1e641 bytes) 000580 1788359736.260582 trunc from 0x0219bf to 0x003a02 (0x1dfbc bytes) 000581 1788359736.273647 read 0x00144e thru 0x003a02 (0x25b5 bytes) 000582 1788359736.279768 fallocate 0x03caa9 thru 0x03ffff (0x3557 bytes) 000583 1788359736.284268 punch from 0x00120f to 0x003a02 (0x27f4 bytes) 000584 1788359736.288358 punch from 0x0019b6 to 0x001f6a (0x5b5 bytes) 000585 1788359736.292400 trunc from 0x003a03 to 0x034a5e (0x3105c bytes) 000586 1788359736.309341 trunc from 0x034a5f to 0x005f69 (0x2eaf5 bytes) 000587 1788359736.324596 punch from 0x005a5d to 0x005f69 (0x50d bytes) 000588 1788359736.330646 fallocate 0x023e21 thru 0x0279e6 (0x3bc6 bytes) 000589 1788359736.337189 punch from 0x0046b7 to 0x005f69 (0x18b3 bytes) 000590 1788359736.341592 punch from 0x0003aa to 0x005f69 (0x5bc0 bytes) 000591 1788359736.346850 read 0x0056ab thru 0x005ea4 (0x7fa bytes) 000592 1788359736.358994 MIRROR_RESYNC 4 mirrors 000593 1788359736.917384 trunc from 0x005f6a to 0x034af2 (0x2eb89 bytes) 000594 1788359736.992364 mapread 0x015c64 thru 0x01fcd1 (0xa06e bytes) 000595 1788359737.020617 write 0x00483b thru 0x00a6b8 (0x5e7e bytes) 000596 1788359737.037340 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34af3): Mirror 3: 0x4386fe59 (stale) Mirror 5: 0x4386fe59 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34af3): Mirror 3: 0x4386fe59 (stale) Mirror 5: 0x4386fe59 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34af3: skipped 000597 1788359737.319008 read 0x02f33c thru 0x034af2 (0x57b7 bytes) 000598 1788359737.327361 trunc from 0x02da6b to 0x034af2 (0x7088 bytes) 000599 1788359737.331320 write 0x0326c1 thru 0x03a81e (0x815e bytes) 000600 1788359737.344357 trunc from 0x01a801 to 0x01bf29 (0x1729 bytes) 000601 1788359737.348878 fallocate 0x01c026 thru 0x0268ca (0xa8a5 bytes) 000602 1788359737.360132 MIRROR_RESYNC 3 mirrors 000603 1788359738.095821 punch from 0x027477 to 0x0361de (0xed68 bytes) 000604 1788359738.134389 trunc from 0x00417b to 0x006526 (0x23ac bytes) 000605 1788359738.141589 punch from 0x025297 to 0x02ce47 (0x7bb1 bytes) 000606 1788359738.146657 mapwrite 0x0049ea thru 0x00b25f (0x6876 bytes) 000607 1788359738.201505 read 0x00f751 thru 0x0154e1 (0x5d91 bytes) 000608 1788359738.211251 trunc from 0x03a81f to 0x009de5 (0x30a39 bytes) 000609 1788359738.242315 write 0x01d487 thru 0x02bb03 (0xe67d bytes) 000610 1788359738.244190 punch from 0x0269c7 to 0x02bb03 (0x513d bytes) 000611 1788359738.260211 trunc from 0x026693 to 0x02bb03 (0x5471 bytes) 000612 1788359738.270151 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2bb04): Mirror 5: 0x7268af2c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bb04): Mirror 5: 0x7268af2c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bb04: skipped 000613 1788359738.628457 punch from 0x00f3ed to 0x01d040 (0xdc54 bytes) 000614 1788359738.633132 fallocate 0x00939b thru 0x00e686 (0x52ec bytes) 000615 1788359738.638369 write 0x012fb6 thru 0x017c96 (0x4ce1 bytes) 000617 1788359738.651914 trunc from 0x02bb04 to 0x0350de (0x95db bytes) 000618 1788359738.674380 MIRROR_RESYNC 2 mirrors 000619 1788359739.441007 punch from 0x024b38 to 0x02d2a5 (0x876e bytes) 000621 1788359739.489970 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x350df): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x350df): Mirror 5: 0xd7d70e9f (stale) CRC-32 checksum value for chunk [0, 0x350df): Mirror 1: 0xafab3e5a Mirror 6: 0xafab3e5a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x350df: skipped 000622 1788359740.349387 mapread 0x008cee thru 0x00a03e (0x1351 bytes) 000623 1788359740.395950 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) 000624 1788359740.807081 mapread 0x012c5c thru 0x019b75 (0x6f1a bytes) 000625 1788359740.847025 punch from 0x00d125 to 0x017e72 (0xad4e bytes) 000626 1788359740.891367 punch from 0x02ff09 to 0x0340db (0x41d3 bytes) 000627 1788359740.901239 trunc from 0x0350df to 0x011d0c (0x233d2 bytes) 000628 1788359740.930371 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x11d0d): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11d0d): Mirror 6: 0x33272213 (stale) CRC-32 checksum value for chunk [0, 0x11d0d): Mirror 1: 0x33272213 Mirror 7: 0x33272213 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11d0d: skipped 000629 1788359741.686917 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x11d0d): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11d0d): Mirror 6: 0x33272213 (stale) CRC-32 checksum value for chunk [0, 0x11d0d): Mirror 1: 0x33272213 Mirror 7: 0x33272213 Mirror 8: 0x33272213 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11d0d: skipped 000630 1788359742.465921 read 0x01015a thru 0x01108a (0xf31 bytes) 000631 1788359742.470471 punch from 0x0096c6 to 0x00e0c6 (0x4a01 bytes) 000633 1788359742.504279 punch from 0x004aa7 to 0x00e102 (0x965c bytes) 000634 1788359742.509923 mapread 0x010d7b thru 0x011d0c (0xf92 bytes) 000635 1788359742.524768 trunc from 0x010108 to 0x011d0c (0x1c05 bytes) 000636 1788359742.538144 MIRROR_RESYNC 4 mirrors 000637 1788359743.470062 trunc from 0x00fafa to 0x011d0c (0x2213 bytes) 000638 1788359743.509197 mapwrite 0x03c179 thru 0x03ffff (0x3e87 bytes) 000639 1788359743.561548 write 0x001b15 thru 0x002962 (0xe4e bytes) 000640 1788359743.569778 write 0x00a9d1 thru 0x01684a (0xbe7a bytes) 000641 1788359743.578647 punch from 0x00d27b to 0x017d11 (0xaa97 bytes) 000642 1788359743.595363 fallocate 0x015924 thru 0x023534 (0xdc11 bytes) 000643 1788359743.599258 trunc from 0x040000 to 0x021f9a (0x1e065 bytes) 000644 1788359743.635504 mapread 0x0032d1 thru 0x003d55 (0xa85 bytes) 000645 1788359743.655532 trunc from 0x01c0bb to 0x021f9a (0x5ee0 bytes) 000646 1788359743.660189 trunc from 0x021f9b to 0x0389bf (0x16a25 bytes) 000648 1788359743.675047 trunc from 0x0389c0 to 0x00d667 (0x2b358 bytes) 000649 1788359743.693460 trunc from 0x00d668 to 0x01d3c9 (0xfd62 bytes) 000650 1788359743.721042 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1d3ca): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d3ca): Mirror 6: 0x5b5fd697 (stale) Mirror 7: 0x5b5fd697 (stale) Mirror 8: 0x5b5fd697 (stale) CRC-32 checksum value for chunk [0, 0x1d3ca): Mirror 1: 0xf9d528cd Mirror 9: 0xf9d528cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d3ca: skipped 000651 1788359744.439649 mapread 0x000138 thru 0x007918 (0x77e1 bytes) 000652 1788359744.465944 write 0x001eab thru 0x009a9f (0x7bf5 bytes) 000653 1788359744.521382 trunc from 0x003c95 to 0x00e68c (0xa9f8 bytes) 000654 1788359744.542760 read 0x019870 thru 0x01a8d8 (0x1069 bytes) 000655 1788359744.553744 write 0x0054d7 thru 0x014755 (0xf27f bytes) 000656 1788359744.567079 write 0x026930 thru 0x028b4f (0x2220 bytes) 000657 1788359744.569468 trunc from 0x021084 to 0x022836 (0x17b3 bytes) 000658 1788359744.576232 read 0x027d13 thru 0x028b4c (0xe3a bytes) 000659 1788359744.577291 fallocate 0x038796 thru 0x03ffff (0x786a bytes) 000660 1788359744.582528 read 0x00d86a thru 0x014073 (0x680a bytes) 000661 1788359744.583735 trunc from 0x016756 to 0x020ea3 (0xa74e bytes) 000663 1788359744.590919 mapread 0x017ab2 thru 0x02471b (0xcc6a bytes) 000664 1788359744.612959 mapwrite 0x001381 thru 0x0032b0 (0x1f30 bytes) 000665 1788359744.630341 trunc from 0x028b50 to 0x023bfa (0x4f55 bytes) 000667 1788359744.649749 write 0x01606f thru 0x01efdc (0x8f6e bytes) 000668 1788359744.659855 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x23bfb): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23bfb): Mirror 6: 0x443b9ce5 (stale) Mirror 7: 0x443b9ce5 (stale) Mirror 8: 0x443b9ce5 (stale) Mirror 9: 0x1443583a (stale) CRC-32 checksum value for chunk [0, 0x23bfb): Mirror 1: 0x1bcf3863 Mirror 10: 0x1bcf3863 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23bfb: skipped 000669 1788359745.406628 mapwrite 0x03b8c0 thru 0x03ffff (0x4740 bytes) 000670 1788359745.497325 read 0x01933e thru 0x0235e3 (0xa2a6 bytes) 000671 1788359745.503913 fallocate 0x026ddf thru 0x036c2c (0xfe4e bytes) 000672 1788359745.508455 trunc from 0x040000 to 0x03dbec (0x2413 bytes) 000673 1788359745.521845 punch from 0x014316 to 0x01a542 (0x622d bytes) 000674 1788359745.533993 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3dbed): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dbed): Mirror 6: 0xb772bdb3 (stale) Mirror 7: 0xb772bdb3 (stale) Mirror 8: 0xb772bdb3 (stale) Mirror 9: 0x9daf95cd (stale) Mirror 10: 0x1975d08e (stale) CRC-32 checksum value for chunk [0, 0x3dbed): Mirror 1: 0xbdf87691 Mirror 11: 0xbdf87691 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dbed: skipped 000675 1788359746.339253 trunc from 0x03dbed to 0x027b73 (0x16079 bytes) 000676 1788359746.402882 mapread 0x0200a8 thru 0x027b73 (0x7acc bytes) 000677 1788359746.422323 write 0x01e83a thru 0x023a1c (0x51e3 bytes) 000678 1788359746.423742 punch from 0x00fec7 to 0x01a78b (0xa8c5 bytes) 000680 1788359746.432083 mapread 0x023939 thru 0x027265 (0x392d bytes) 000681 1788359746.433224 mapread 0x023799 thru 0x027b73 (0x43db bytes) 000682 1788359746.438907 write 0x037cd3 thru 0x03d107 (0x5435 bytes) 000683 1788359746.440298 trunc from 0x03d108 to 0x027446 (0x15cc1 bytes) 000684 1788359746.463343 trunc from 0x027447 to 0x0169e2 (0x10a64 bytes) 000685 1788359746.480330 trunc from 0x0169e3 to 0x006401 (0x105e1 bytes) 000686 1788359746.503352 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6402): Mirror 7: 0x81e77f9e (stale) Mirror 8: 0x81e77f9e (stale) Mirror 9: 0xbec5c583 (stale) Mirror 10: 0x9459c37a (stale) Mirror 11: 0x9459c37a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6402): Mirror 7: 0x81e77f9e (stale) Mirror 8: 0x81e77f9e (stale) Mirror 9: 0xbec5c583 (stale) Mirror 10: 0x9459c37a (stale) Mirror 11: 0x9459c37a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6402: skipped 000687 1788359746.793693 MIRROR_RESYNC 6 mirrors 000688 1788359747.778916 fallocate 0x02b39e thru 0x0361e6 (0xae49 bytes) 000689 1788359747.802870 mapread 0x02fa59 thru 0x034ae1 (0x5089 bytes) 000690 1788359747.817818 mapread 0x01d724 thru 0x01dcf2 (0x5cf bytes) 000691 1788359747.828505 trunc from 0x02ee73 to 0x0330ea (0x4278 bytes) 000692 1788359747.836230 trunc from 0x0130c2 to 0x01c360 (0x929f bytes) 000693 1788359747.842938 trunc from 0x0361e7 to 0x02374c (0x12a9a bytes) 000694 1788359747.859972 mapread 0x0004cf thru 0x00e892 (0xe3c4 bytes) 000695 1788359747.867801 write 0x013fa3 thru 0x01c3a4 (0x8402 bytes) 000696 1788359747.877744 read 0x0081dd thru 0x00dc3b (0x5a5f bytes) 000697 1788359747.878287 fallocate 0x02319f thru 0x0312c4 (0xe126 bytes) 000698 1788359747.884247 write 0x014daf thru 0x020326 (0xb578 bytes) 000700 1788359747.886324 mapread 0x02cd5c thru 0x0312c4 (0x4569 bytes) 000701 1788359747.898987 punch from 0x00ce1e to 0x0136e2 (0x68c5 bytes) 000702 1788359747.916726 mapread 0x00c5d0 thru 0x0134f3 (0x6f24 bytes) 000703 1788359747.929171 trunc from 0x028201 to 0x02b59c (0x339c bytes) 000704 1788359747.936686 punch from 0x019274 to 0x01da52 (0x47df bytes) 000705 1788359747.943369 mapread 0x017c18 thru 0x01e9f3 (0x6ddc bytes) 000706 1788359747.953140 read 0x019440 thru 0x029332 (0xfef3 bytes) 000707 1788359747.962951 read 0x02bf91 thru 0x0312c4 (0x5334 bytes) 000708 1788359747.963770 trunc from 0x0312c5 to 0x00133d (0x2ff87 bytes) 000709 1788359747.993978 mapread 0x0007d2 thru 0x00133d (0xb6c bytes) 000710 1788359747.995165 mapread 0x000330 thru 0x00133d (0x100e bytes) 000711 1788359747.996900 trunc from 0x00133e to 0x029a66 (0x28729 bytes) 000712 1788359748.014160 trunc from 0x01fb00 to 0x022e4d (0x334e bytes) 000713 1788359748.023316 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29a67): Mirror 7: 0x7259fb6d (stale) Mirror 8: 0x7259fb6d (stale) Mirror 9: 0x7259fb6d (stale) Mirror 10: 0x7259fb6d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29a67): Mirror 7: 0x7259fb6d (stale) Mirror 8: 0x7259fb6d (stale) Mirror 9: 0x7259fb6d (stale) Mirror 10: 0x7259fb6d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29a67: skipped 000714 1788359748.384670 punch from 0x0283e5 to 0x029a66 (0x1682 bytes) 000715 1788359748.389277 trunc from 0x029a67 to 0x035913 (0xbead bytes) 000716 1788359748.401857 write 0x01f604 thru 0x02e4ab (0xeea8 bytes) 000717 1788359748.410795 read 0x01c5f9 thru 0x026f86 (0xa98e bytes) 000718 1788359748.416141 read 0x00bebb thru 0x016984 (0xaaca bytes) 000719 1788359748.425146 mapwrite 0x039630 thru 0x03e971 (0x5342 bytes) 000720 1788359748.465747 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e972): Mirror 7: 0x7c5abea8 (stale) Mirror 8: 0x7c5abea8 (stale) Mirror 10: 0x7c5abea8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e972): Mirror 7: 0x7c5abea8 (stale) Mirror 8: 0x7c5abea8 (stale) Mirror 10: 0x7c5abea8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e972: skipped 000721 1788359748.743432 trunc from 0x01975f to 0x024c7b (0xb51d bytes) 000722 1788359748.747006 punch from 0x00e2ed to 0x01c479 (0xe18d bytes) 000723 1788359748.753271 trunc from 0x03e972 to 0x02ecd2 (0xfc9f bytes) 000724 1788359748.767977 punch from 0x029610 to 0x029e89 (0x87a bytes) 000725 1788359748.773355 punch from 0x024393 to 0x0268f7 (0x2565 bytes) 000726 1788359748.777844 mapread 0x01bdf0 thru 0x024819 (0x8a2a bytes) 000727 1788359748.811043 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2ecd3): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ecd3): Mirror 7: 0xea34e96d (stale) Mirror 8: 0xea34e96d (stale) Mirror 10: 0xea34e96d (stale) CRC-32 checksum value for chunk [0, 0x2ecd3): Mirror 1: 0xec7a1d80 Mirror 11: 0xec7a1d80 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ecd3: skipped 000728 1788359749.680745 punch from 0x02991f to 0x02c392 (0x2a74 bytes) 000729 1788359749.723218 MIRROR_RESYNC 5 mirrors 000730 1788359750.872329 mapwrite 0x021616 thru 0x026278 (0x4c63 bytes) 000731 1788359750.956147 write 0x0041a7 thru 0x00c70b (0x8565 bytes) 000732 1788359750.966529 fallocate 0x01efdc thru 0x020778 (0x179d bytes) 000733 1788359750.978970 punch from 0x0250ee to 0x02ecd2 (0x9be5 bytes) 000734 1788359750.983910 trunc from 0x01eb69 to 0x0247b9 (0x5c51 bytes) 000735 1788359750.990729 read 0x021f51 thru 0x02ecd2 (0xcd82 bytes) 000736 1788359750.998047 mapread 0x0006d6 thru 0x001a96 (0x13c1 bytes) 000737 1788359751.006974 mapread 0x0017c8 thru 0x006042 (0x487b bytes) 000738 1788359751.007851 mapwrite 0x039f93 thru 0x03ffff (0x606d bytes) 000739 1788359751.060367 trunc from 0x040000 to 0x02e6c1 (0x1193e bytes) 000740 1788359751.076196 fallocate 0x01ee1e thru 0x022a30 (0x3c13 bytes) 000741 1788359751.079840 trunc from 0x009f47 to 0x00bae7 (0x1ba1 bytes) 000742 1788359751.084949 trunc from 0x02e6c2 to 0x03f4e6 (0x10e25 bytes) 000743 1788359751.102968 mapwrite 0x004ffd thru 0x00efb6 (0x9fba bytes) 000744 1788359751.134866 mapwrite 0x027407 thru 0x032aab (0xb6a5 bytes) 000746 1788359751.162662 trunc from 0x03f4e7 to 0x0398bf (0x5c27 bytes) 000747 1788359751.178044 mapread 0x030f9c thru 0x0398bf (0x8924 bytes) 000748 1788359751.182264 punch from 0x02324b to 0x02ac24 (0x79da bytes) 000749 1788359751.186166 read 0x033f08 thru 0x03738e (0x3487 bytes) 000750 1788359751.186696 read 0x015977 thru 0x0201f0 (0xa87a bytes) 000753 1788359751.196391 mapwrite 0x0332f4 thru 0x03ac65 (0x7972 bytes) 000756 1788359751.242835 punch from 0x010a23 to 0x016874 (0x5e52 bytes) 000758 1788359751.251893 write 0x0187bf thru 0x01dfdb (0x581d bytes) 000759 1788359751.253053 trunc from 0x000de3 to 0x006615 (0x5833 bytes) 000760 1788359751.260308 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ac66): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ac66): Mirror 7: 0xcdc909ce (stale) Mirror 8: 0xcdc909ce (stale) Mirror 10: 0xcdc909ce (stale) Mirror 11: 0xcdc909ce (stale) CRC-32 checksum value for chunk [0, 0x3ac66): Mirror 1: 0x79a7d16e Mirror 12: 0x79a7d16e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ac66: skipped 000761 1788359751.900788 mapread 0x033459 thru 0x03ac65 (0x780d bytes) 000764 1788359751.943957 read 0x0032e9 thru 0x00b200 (0x7f18 bytes) 000765 1788359751.962627 write 0x0209c0 thru 0x02dadc (0xd11d bytes) 000767 1788359752.017130 fallocate 0x01222b thru 0x01a0d8 (0x7eae bytes) 000768 1788359752.026157 punch from 0x00dded to 0x00f9ef (0x1c03 bytes) 000769 1788359752.031331 fallocate 0x002f63 thru 0x005af0 (0x2b8e bytes) 000770 1788359752.035888 read 0x0348dc thru 0x03781a (0x2f3f bytes) 000771 1788359752.046718 punch from 0x0031c4 to 0x01056e (0xd3ab bytes) 000772 1788359752.052372 write 0x0159dd thru 0x01768b (0x1caf bytes) 000774 1788359752.054451 mapwrite 0x033a2a thru 0x03ffff (0xc5d6 bytes) 000775 1788359752.114653 mapwrite 0x03ca55 thru 0x03ffff (0x35ab bytes) 000776 1788359752.127538 mapwrite 0x037a3b thru 0x03db8f (0x6155 bytes) 000777 1788359752.154793 trunc from 0x030ba5 to 0x03771a (0x6b76 bytes) 000778 1788359752.172394 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xb314052a (stale) Mirror 8: 0xb314052a (stale) Mirror 10: 0xb314052a (stale) Mirror 12: 0xf25e1db5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xb314052a (stale) Mirror 8: 0xb314052a (stale) Mirror 10: 0xb314052a (stale) Mirror 12: 0xf25e1db5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000779 1788359752.465746 fallocate 0x034074 thru 0x03d9dd (0x996a bytes) 000780 1788359752.469843 punch from 0x0173c4 to 0x018036 (0xc73 bytes) 000782 1788359752.474774 write 0x023520 thru 0x0317ff (0xe2e0 bytes) 000783 1788359752.485227 mapread 0x025cc6 thru 0x027cb4 (0x1fef bytes) 000784 1788359752.486049 punch from 0x00489a to 0x008863 (0x3fca bytes) 000785 1788359752.493001 read 0x021a40 thru 0x0296c2 (0x7c83 bytes) 000786 1788359752.498985 mapwrite 0x01a389 thru 0x01b745 (0x13bd bytes) 000787 1788359752.530199 punch from 0x0180b4 to 0x022a70 (0xa9bd bytes) 000788 1788359752.534247 read 0x032009 thru 0x03ffff (0xdff7 bytes) 000789 1788359752.545963 mapwrite 0x00e164 thru 0x01a80a (0xc6a7 bytes) 000790 1788359752.559613 trunc from 0x040000 to 0x008477 (0x37b88 bytes) 000791 1788359752.569147 fallocate 0x03c880 thru 0x03ffff (0x3780 bytes) 000792 1788359752.572184 read 0x0034d2 thru 0x008477 (0x4fa6 bytes) 000793 1788359752.572684 trunc from 0x003961 to 0x008477 (0x4b17 bytes) 000794 1788359752.576366 fallocate 0x01853d thru 0x01a23e (0x1d02 bytes) 000795 1788359752.579554 trunc from 0x0001ea to 0x00226c (0x2083 bytes) 000796 1788359752.582278 fallocate 0x01a109 thru 0x01bec2 (0x1dba bytes) 000797 1788359752.585225 mapread 0x004986 thru 0x00560c (0xc87 bytes) 000799 1788359752.591359 read 0x00734c thru 0x008477 (0x112c bytes) 000800 1788359752.592155 write 0x038b8f thru 0x03ffff (0x7471 bytes) 000801 1788359752.593863 mapwrite 0x00285e thru 0x007662 (0x4e05 bytes) 000802 1788359752.606919 read 0x0339d0 thru 0x03a0ff (0x6730 bytes) 000803 1788359752.615912 trunc from 0x00183d to 0x008db7 (0x757b bytes) 000804 1788359752.621156 fallocate 0x038e2c thru 0x03ffff (0x71d4 bytes) 000805 1788359752.624635 mapwrite 0x02d5bd thru 0x03c90f (0xf353 bytes) 000806 1788359752.648615 mapwrite 0x011e3a thru 0x01ea19 (0xcbe0 bytes) 000807 1788359752.672802 mapread 0x035d03 thru 0x039601 (0x38ff bytes) 000808 1788359752.674581 write 0x02ce49 thru 0x02e5f7 (0x17af bytes) 000809 1788359752.675859 punch from 0x034696 to 0x03ed47 (0xa6b2 bytes) 000810 1788359752.685788 punch from 0x02af49 to 0x037853 (0xc90b bytes) 000811 1788359752.694893 fallocate 0x034785 thru 0x03ffff (0xb87b bytes) 000812 1788359752.699925 mapread 0x03b05c thru 0x03ffff (0x4fa4 bytes) 000813 1788359752.717259 trunc from 0x0204d2 to 0x02fd0c (0xf83b bytes) 000814 1788359752.721370 trunc from 0x040000 to 0x022f4b (0x1d0b4 bytes) 000815 1788359752.731770 write 0x0385e8 thru 0x03f93f (0x7358 bytes) 000816 1788359752.733126 mapwrite 0x008717 thru 0x01080c (0x80f6 bytes) 000817 1788359752.766367 mapwrite 0x00a751 thru 0x00c51f (0x1dcf bytes) 000818 1788359752.782034 trunc from 0x03f940 to 0x03a72d (0x5212 bytes) 000819 1788359752.804247 MIRROR_RESYNC 5 mirrors 000820 1788359753.525333 write 0x032a69 thru 0x03ffff (0xd597 bytes) 000821 1788359753.550295 punch from 0x00d20d to 0x012705 (0x54f9 bytes) 000823 1788359753.559151 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xed1e5c25 (stale) Mirror 8: 0xed1e5c25 (stale) Mirror 10: 0xed1e5c25 (stale) Mirror 12: 0xed1e5c25 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1a0263aa Mirror 13: 0x1a0263aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000824 1788359754.250006 punch from 0x00612d to 0x01182b (0xb6ff bytes) 000825 1788359754.293139 write 0x034ea5 thru 0x03d15d (0x82b9 bytes) 000826 1788359754.303642 punch from 0x02779b to 0x0316ea (0x9f50 bytes) 000827 1788359754.307721 fallocate 0x03e45e thru 0x03ffff (0x1ba2 bytes) 000828 1788359754.313536 punch from 0x0036a6 to 0x00e492 (0xaded bytes) 000829 1788359754.317276 trunc from 0x0392e8 to 0x03d5c1 (0x42da bytes) 000830 1788359754.335809 trunc from 0x040000 to 0x03200d (0xdff2 bytes) 000831 1788359754.361534 punch from 0x00c33f to 0x01665e (0xa320 bytes) 000832 1788359754.368803 mapwrite 0x028bf6 thru 0x037fb4 (0xf3bf bytes) 000833 1788359754.449553 trunc from 0x027395 to 0x02e46b (0x70d7 bytes) 000834 1788359754.455269 write 0x00abfd thru 0x01a4ad (0xf8b1 bytes) 000836 1788359754.466749 fallocate 0x034ba8 thru 0x03ffff (0xb458 bytes) 000837 1788359754.471254 trunc from 0x022703 to 0x02c867 (0xa165 bytes) 000838 1788359754.482400 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37fb5): Mirror 7: 0x2290cf (stale) Mirror 8: 0x2290cf (stale) Mirror 10: 0x2290cf (stale) Mirror 13: 0xe24e6e9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37fb5): Mirror 7: 0x2290cf (stale) Mirror 8: 0x2290cf (stale) Mirror 10: 0x2290cf (stale) Mirror 13: 0xe24e6e9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37fb5: skipped 000840 1788359754.814343 trunc from 0x02ed42 to 0x037fb4 (0x9273 bytes) 000842 1788359754.819149 trunc from 0x037fb5 to 0x03f913 (0x795f bytes) 000843 1788359754.833718 mapwrite 0x02d4a1 thru 0x037165 (0x9cc5 bytes) 000844 1788359754.871489 write 0x031cd8 thru 0x03de31 (0xc15a bytes) 000845 1788359754.878613 read 0x039cd6 thru 0x03f913 (0x5c3e bytes) 000847 1788359754.886069 trunc from 0x03f914 to 0x02b057 (0x148bc bytes) 000848 1788359754.905143 trunc from 0x02b058 to 0x0394c8 (0xe471 bytes) 000849 1788359754.918146 fallocate 0x02ba16 thru 0x033974 (0x7f5f bytes) 000850 1788359754.923536 trunc from 0x0394c9 to 0x02b583 (0xdf45 bytes) 000851 1788359754.940983 trunc from 0x02b584 to 0x008107 (0x2347c bytes) 000852 1788359754.957270 trunc from 0x008108 to 0x01f2ad (0x171a6 bytes) 000853 1788359754.975141 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1f2ae): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f2ae): Mirror 7: 0x1276f48 (stale) Mirror 8: 0x1276f48 (stale) Mirror 10: 0x1276f48 (stale) Mirror 13: 0x160fc78e (stale) CRC-32 checksum value for chunk [0, 0x1f2ae): Mirror 1: 0xc8b5a061 Mirror 14: 0xc8b5a061 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f2ae: skipped 000854 1788359755.607816 trunc from 0x01f2ae to 0x029a3b (0xa78e bytes) 000855 1788359755.659335 mapread 0x00e0aa thru 0x015ca1 (0x7bf8 bytes) 000856 1788359755.675978 punch from 0x005046 to 0x006954 (0x190f bytes) 000857 1788359755.681317 mapread 0x003fb2 thru 0x01015e (0xc1ad bytes) 000858 1788359755.691241 trunc from 0x029a3c to 0x023b4c (0x5eef bytes) 000859 1788359755.716516 mapread 0x00c0bb thru 0x0137b6 (0x76fc bytes) 000860 1788359755.720882 MIRROR_RESYNC 6 mirrors 000861 1788359756.439953 write 0x03654f thru 0x03ffff (0x9ab1 bytes) 000862 1788359756.460649 read 0x031bea thru 0x0342a8 (0x26bf bytes) 000863 1788359756.467374 trunc from 0x040000 to 0x03d484 (0x2b7b bytes) 000864 1788359756.485630 trunc from 0x013b1b to 0x0196cc (0x5bb2 bytes) 000865 1788359756.490021 mapread 0x00b86d thru 0x00fa54 (0x41e8 bytes) 000866 1788359756.510718 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3d485): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d485): Mirror 7: 0x95e8d78d (stale) Mirror 8: 0x95e8d78d (stale) Mirror 10: 0x95e8d78d (stale) Mirror 13: 0x95e8d78d (stale) Mirror 14: 0x95e8d78d (stale) CRC-32 checksum value for chunk [0, 0x3d485): Mirror 1: 0x9131a8a5 Mirror 15: 0x9131a8a5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d485: skipped 000868 1788359757.259975 mapread 0x03315f thru 0x03d484 (0xa326 bytes) 000869 1788359757.291063 mapread 0x0216b3 thru 0x029d8e (0x86dc bytes) 000870 1788359757.317314 mapwrite 0x03393e thru 0x034fbe (0x1681 bytes) 000871 1788359757.389722 mapwrite 0x007c49 thru 0x00f21f (0x75d7 bytes) 000872 1788359757.424148 mapwrite 0x018c60 thru 0x01aa99 (0x1e3a bytes) 000873 1788359757.455174 MIRROR_RESYNC 7 mirrors 000874 1788359759.028561 trunc from 0x03607e to 0x03d484 (0x7407 bytes) 000875 1788359759.058000 mapwrite 0x03b71d thru 0x03d7c3 (0x20a7 bytes) 000876 1788359759.104209 punch from 0x01afff to 0x0295d0 (0xe5d2 bytes) 000877 1788359759.110089 write 0x01d737 thru 0x01d75a (0x24 bytes) 000878 1788359759.116547 punch from 0x03d268 to 0x03d7c3 (0x55c bytes) 000879 1788359759.122262 punch from 0x02c6bc to 0x032921 (0x6266 bytes) 000880 1788359759.126990 mapread 0x03406b thru 0x03d7c3 (0x9759 bytes) 000881 1788359759.141935 trunc from 0x0103e3 to 0x015385 (0x4fa3 bytes) 000882 1788359759.146675 fallocate 0x010b39 thru 0x0194c6 (0x898e bytes) 000883 1788359759.152551 read 0x02b2ec thru 0x02b8a6 (0x5bb bytes) 000884 1788359759.158542 read 0x00cb12 thru 0x0120bc (0x55ab bytes) 000886 1788359759.169610 fallocate 0x0317d3 thru 0x03e7a0 (0xcfce bytes) 000887 1788359759.179366 MIRROR_RESYNC 7 mirrors 000888 1788359761.296048 fallocate 0x01d29a thru 0x020a01 (0x3768 bytes) 000889 1788359761.326970 trunc from 0x03489f to 0x03e7a0 (0x9f02 bytes) 000890 1788359761.331473 write 0x02e70e thru 0x034574 (0x5e67 bytes) 000892 1788359761.344921 trunc from 0x03e7a1 to 0x01ee86 (0x1f91a bytes) 000893 1788359761.364213 mapwrite 0x01f78e thru 0x02f471 (0xfce4 bytes) 000894 1788359761.430743 mapread 0x00f182 thru 0x00f6a8 (0x527 bytes) 000895 1788359761.440551 write 0x0220c5 thru 0x022f83 (0xebf bytes) 000896 1788359761.442169 trunc from 0x02f472 to 0x008548 (0x26f29 bytes) 000897 1788359761.460159 read 0x0020f6 thru 0x004637 (0x2542 bytes) 000898 1788359761.468865 fallocate 0x000c4a thru 0x002905 (0x1cbc bytes) 000899 1788359761.478458 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x8549): Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8549): Mirror 7: 0x36d1cff (stale) Mirror 8: 0x36d1cff (stale) Mirror 10: 0x36d1cff (stale) Mirror 13: 0x36d1cff (stale) Mirror 14: 0x36d1cff (stale) Mirror 15: 0x36d1cff (stale) CRC-32 checksum value for chunk [0, 0x8549): Mirror 1: 0x36d1cff Mirror 16: 0x36d1cff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8549: skipped 000900 1788359762.191747 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8549): Mirror 8: 0x36d1cff (stale) Mirror 10: 0x36d1cff (stale) Mirror 13: 0x36d1cff (stale) Mirror 14: 0x36d1cff (stale) Mirror 15: 0x36d1cff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8549): Mirror 8: 0x36d1cff (stale) Mirror 10: 0x36d1cff (stale) Mirror 13: 0x36d1cff (stale) Mirror 14: 0x36d1cff (stale) Mirror 15: 0x36d1cff (stale) CRC-32 checksum value for chunk [0, 0x8549): Mirror 1: 0x36d1cff Mirror 16: 0x36d1cff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8549: skipped 000901 1788359762.881148 read 0x005020 thru 0x008548 (0x3529 bytes) 000902 1788359762.906303 mapread 0x002da8 thru 0x008548 (0x57a1 bytes) 000903 1788359762.949912 mapwrite 0x033c5b thru 0x03d7e4 (0x9b8a bytes) 000904 1788359763.039227 trunc from 0x03d7e5 to 0x03d441 (0x3a3 bytes) 000906 1788359763.063978 fallocate 0x00ea05 thru 0x014384 (0x5980 bytes) 000907 1788359763.068224 mapread 0x02ab1a thru 0x02b9d1 (0xeb8 bytes) 000908 1788359763.077775 trunc from 0x03d442 to 0x020a82 (0x1c9bf bytes) 000909 1788359763.109121 MIRROR_RESYNC 7 mirrors 000911 1788359764.222110 mapread 0x003c6b thru 0x00c29d (0x8633 bytes) 000912 1788359764.249079 punch from 0x008fba to 0x012204 (0x924b bytes) 000913 1788359764.291121 write 0x035fcb thru 0x03d701 (0x7737 bytes) 000914 1788359764.298839 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3d702): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d702): Mirror 8: 0x95e4dff5 (stale) Mirror 10: 0x95e4dff5 (stale) Mirror 13: 0x95e4dff5 (stale) Mirror 14: 0x95e4dff5 (stale) Mirror 15: 0x95e4dff5 (stale) Mirror 16: 0x95e4dff5 (stale) CRC-32 checksum value for chunk [0, 0x3d702): Mirror 1: 0xa1ce2608 Mirror 17: 0xa1ce2608 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d702: skipped 000915 1788359765.173301 trunc from 0x03d702 to 0x03be67 (0x189a bytes) 000917 1788359765.219832 mapwrite 0x022c81 thru 0x02f0b6 (0xc436 bytes) 000918 1788359765.268025 write 0x00e53c thru 0x00ee18 (0x8dd bytes) 000919 1788359765.274514 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3be68): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3be68): Mirror 8: 0xe2988c2a (stale) Mirror 10: 0xe2988c2a (stale) Mirror 13: 0xe2988c2a (stale) Mirror 14: 0xe2988c2a (stale) Mirror 15: 0xe2988c2a (stale) Mirror 16: 0xe2988c2a (stale) Mirror 17: 0xbce129ed (stale) CRC-32 checksum value for chunk [0, 0x3be68): Mirror 1: 0xc140440b Mirror 18: 0xc140440b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3be68: skipped 000921 1788359766.141917 trunc from 0x038531 to 0x03be67 (0x3937 bytes) 000922 1788359766.184943 write 0x03c877 thru 0x03ffff (0x3789 bytes) 000923 1788359766.190682 trunc from 0x040000 to 0x03aeb6 (0x5149 bytes) 000924 1788359766.233803 fallocate 0x03ff40 thru 0x03ffff (0xc0 bytes) 000925 1788359766.238512 fallocate 0x01675a thru 0x0235bf (0xce66 bytes) 000926 1788359766.242870 trunc from 0x040000 to 0x01e12c (0x21ed3 bytes) 000928 1788359766.265798 mapwrite 0x0071cb thru 0x0113c2 (0xa1f8 bytes) 000929 1788359766.313695 mapread 0x00e3fd thru 0x014a59 (0x665d bytes) 000930 1788359766.323856 read 0x000123 thru 0x00b42c (0xb30a bytes) 000931 1788359766.329361 trunc from 0x01e12d to 0x00016d (0x1dfbf bytes) 000932 1788359766.342676 punch from 0x000082 to 0x00016d (0xec bytes) 000933 1788359766.353732 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x16e): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16e): Mirror 8: 0xa2e31206 (stale) Mirror 10: 0xa2e31206 (stale) Mirror 13: 0xa2e31206 (stale) Mirror 14: 0xa2e31206 (stale) Mirror 15: 0xa2e31206 (stale) Mirror 16: 0xa2e31206 (stale) Mirror 17: 0xa2e31206 (stale) Mirror 18: 0xa2e31206 (stale) CRC-32 checksum value for chunk [0, 0x16e): Mirror 1: 0xa2e31206 Mirror 19: 0xa2e31206 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16e: skipped 000934 1788359767.002921 read 0x00000e thru 0x00016d (0x160 bytes) 000936 1788359767.020786 read 0x00010d thru 0x00016d (0x61 bytes) 000937 1788359767.027732 read 0x000037 thru 0x00016d (0x137 bytes) 000938 1788359767.035702 trunc from 0x00016e to 0x025d90 (0x25c23 bytes) 000939 1788359767.127660 trunc from 0x025d91 to 0x01b1ae (0xabe2 bytes) 000940 1788359767.144828 mapwrite 0x0336c4 thru 0x03c114 (0x8a51 bytes) 000941 1788359767.188866 fallocate 0x039d7b thru 0x03ffff (0x6285 bytes) 000942 1788359767.193833 mapread 0x03f823 thru 0x03ffff (0x7dd bytes) 000943 1788359767.198721 mapwrite 0x004fe5 thru 0x012fba (0xdfd6 bytes) 000944 1788359767.217500 write 0x02c28d thru 0x032d0b (0x6a7f bytes) 000945 1788359767.218803 read 0x025022 thru 0x030c60 (0xbc3f bytes) 000946 1788359767.226683 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xcf4a1937 (stale) Mirror 10: 0xcf4a1937 (stale) Mirror 13: 0xcf4a1937 (stale) Mirror 14: 0xcf4a1937 (stale) Mirror 15: 0xcf4a1937 (stale) Mirror 16: 0xcf4a1937 (stale) Mirror 17: 0x5b83c91d (stale) Mirror 18: 0x9eaeb9f9 (stale) Mirror 19: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x83e7699a Mirror 20: 0x83e7699a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000947 1788359767.895811 punch from 0x0101db to 0x0119c3 (0x17e9 bytes) 000948 1788359767.950092 trunc from 0x040000 to 0x032413 (0xdbec bytes) 000949 1788359768.003074 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32414): Mirror 8: 0x33efff5d (stale) Mirror 10: 0x33efff5d (stale) Mirror 13: 0x33efff5d (stale) Mirror 15: 0x33efff5d (stale) Mirror 16: 0x33efff5d (stale) Mirror 17: 0x33efff5d (stale) Mirror 18: 0x44e379d (stale) Mirror 19: 0xb9547b87 (stale) Mirror 20: 0x1b4157a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32414): Mirror 8: 0x33efff5d (stale) Mirror 10: 0x33efff5d (stale) Mirror 13: 0x33efff5d (stale) Mirror 15: 0x33efff5d (stale) Mirror 16: 0x33efff5d (stale) Mirror 17: 0x33efff5d (stale) Mirror 18: 0x44e379d (stale) Mirror 19: 0xb9547b87 (stale) Mirror 20: 0x1b4157a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32414: skipped 000950 1788359768.368132 trunc from 0x015b1d to 0x02115a (0xb63e bytes) 000952 1788359768.378870 punch from 0x0273c8 to 0x032413 (0xb04c bytes) 000953 1788359768.384106 write 0x00e5e7 thru 0x01de9d (0xf8b7 bytes) 000954 1788359768.402093 read 0x01e998 thru 0x026399 (0x7a02 bytes) 000955 1788359768.413765 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32414): Mirror 8: 0x33efff5d (stale) Mirror 10: 0x33efff5d (stale) Mirror 13: 0x33efff5d (stale) Mirror 15: 0x33efff5d (stale) Mirror 16: 0x33efff5d (stale) Mirror 18: 0x44e379d (stale) Mirror 19: 0xb9547b87 (stale) Mirror 20: 0x1b4157a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32414): Mirror 8: 0x33efff5d (stale) Mirror 10: 0x33efff5d (stale) Mirror 13: 0x33efff5d (stale) Mirror 15: 0x33efff5d (stale) Mirror 16: 0x33efff5d (stale) Mirror 18: 0x44e379d (stale) Mirror 19: 0xb9547b87 (stale) Mirror 20: 0x1b4157a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32414: skipped 000956 1788359768.840935 mapread 0x02233f thru 0x02352b (0x11ed bytes) 000957 1788359768.852310 trunc from 0x02b761 to 0x030cd3 (0x5573 bytes) 000958 1788359768.862110 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x32414): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32414): Mirror 8: 0x33efff5d (stale) Mirror 10: 0x33efff5d (stale) Mirror 13: 0x33efff5d (stale) Mirror 15: 0x33efff5d (stale) Mirror 16: 0x33efff5d (stale) Mirror 18: 0x44e379d (stale) Mirror 19: 0xb9547b87 (stale) Mirror 20: 0x1b4157a2 (stale) CRC-32 checksum value for chunk [0, 0x32414): Mirror 1: 0x11d67791 Mirror 21: 0x11d67791 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32414: skipped 000959 1788359769.736562 mapwrite 0x0375f5 thru 0x03e1c2 (0x6bce bytes) 000960 1788359769.810613 punch from 0x013d51 to 0x0207d4 (0xca84 bytes) 000961 1788359769.824277 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x20c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e1c3): Mirror 8: 0xf0ffd2b0 (stale) Mirror 10: 0xf0ffd2b0 (stale) Mirror 13: 0xf0ffd2b0 (stale) Mirror 15: 0xf0ffd2b0 (stale) Mirror 18: 0x2bf65e84 (stale) Mirror 19: 0x7ed07251 (stale) Mirror 20: 0xb05bcc19 (stale) Mirror 21: 0xc1fd50cc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e1c3): Mirror 8: 0xf0ffd2b0 (stale) Mirror 10: 0xf0ffd2b0 (stale) Mirror 13: 0xf0ffd2b0 (stale) Mirror 15: 0xf0ffd2b0 (stale) Mirror 18: 0x2bf65e84 (stale) Mirror 19: 0x7ed07251 (stale) Mirror 20: 0xb05bcc19 (stale) Mirror 21: 0xc1fd50cc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e1c3: skipped 000962 1788359770.223303 write 0x001a31 thru 0x0036d5 (0x1ca5 bytes) 000963 1788359770.240683 trunc from 0x00a29a to 0x01110e (0x6e75 bytes) 000964 1788359770.244824 mapread 0x0094b3 thru 0x012317 (0x8e65 bytes) 000965 1788359770.260083 read 0x00e9b6 thru 0x0183bb (0x9a06 bytes) 000966 1788359770.269458 punch from 0x00e64c to 0x01ab90 (0xc545 bytes) 000967 1788359770.274160 mapwrite 0x013a6d thru 0x01d6de (0x9c72 bytes) 000968 1788359770.317739 read 0x005940 thru 0x008853 (0x2f14 bytes) 000969 1788359770.318899 punch from 0x01d364 to 0x0275d0 (0xa26d bytes) 000970 1788359770.329879 write 0x03c259 thru 0x03ffff (0x3da7 bytes) 000971 1788359770.337751 write 0x008fb4 thru 0x00907e (0xcb bytes) 000972 1788359770.338587 trunc from 0x040000 to 0x033495 (0xcb6a bytes) 000973 1788359770.348208 punch from 0x025e1d to 0x031c36 (0xbe1a bytes) 000974 1788359770.353514 fallocate 0x00e613 thru 0x0113ff (0x2ded bytes) 000975 1788359770.362067 MIRROR_RESYNC 9 mirrors 000977 1788359772.594587 mapread 0x00acba thru 0x017fd9 (0xd320 bytes) 000978 1788359772.616215 punch from 0x00cc27 to 0x00e0de (0x14b8 bytes) 000979 1788359772.643705 mapwrite 0x0343eb thru 0x03e632 (0xa248 bytes) 000980 1788359772.728230 fallocate 0x02db48 thru 0x03716f (0x9628 bytes) 000981 1788359772.733938 mapwrite 0x02670c thru 0x028d92 (0x2687 bytes) 000982 1788359772.759484 trunc from 0x03e633 to 0x009745 (0x34eed bytes) 000983 1788359772.781536 trunc from 0x009746 to 0x004a36 (0x4d0f bytes) 000984 1788359772.801911 read 0x000332 thru 0x004a36 (0x4705 bytes) 000985 1788359772.824268 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x4a37): Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4a37): Mirror 8: 0xead871e7 (stale) Mirror 10: 0xead871e7 (stale) Mirror 13: 0xead871e7 (stale) Mirror 15: 0xead871e7 (stale) Mirror 18: 0xead871e7 (stale) Mirror 19: 0xead871e7 (stale) Mirror 20: 0xead871e7 (stale) Mirror 21: 0xead871e7 (stale) CRC-32 checksum value for chunk [0, 0x4a37): Mirror 1: 0xead871e7 Mirror 22: 0xead871e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4a37: skipped 000986 1788359773.471833 write 0x0340d8 thru 0x03ffff (0xbf28 bytes) 000987 1788359773.534484 MIRROR_RESYNC 10 mirrors 000988 1788359774.985850 fallocate 0x01f8a8 thru 0x0286b7 (0x8e10 bytes) 000989 1788359775.019173 mapread 0x028280 thru 0x0314d8 (0x9259 bytes) 000990 1788359775.048046 mapread 0x02424b thru 0x030bff (0xc9b5 bytes) 000991 1788359775.051970 fallocate 0x03fdef thru 0x03ffff (0x211 bytes) 000992 1788359775.068792 mapwrite 0x00c4fc thru 0x00f410 (0x2f15 bytes) 000993 1788359775.098899 read 0x009b81 thru 0x00beaf (0x232f bytes) 000994 1788359775.101061 read 0x00739d thru 0x00c284 (0x4ee8 bytes) 000995 1788359775.103177 read 0x0048bb thru 0x00cb10 (0x8256 bytes) 000996 1788359775.122623 mapwrite 0x02f41e thru 0x0310ed (0x1cd0 bytes) 000997 1788359775.146118 MIRROR_RESYNC 10 mirrors 000998 1788359777.375305 read 0x01d9c7 thru 0x02119d (0x37d7 bytes) 000999 1788359777.386066 trunc from 0x01986f to 0x0269ed (0xd17f bytes) All operations completed A-OK! PASS 70a (87s) == sanity-flr test 71: check mirror extend parent fid ==== 10:36:21 (1788359781) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x24c:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (7s) == sanity-flr test 72: check append on FLR file ========== 10:36:28 (1788359788) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.202248 s, 10.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864273 s, 12.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x117:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 10:36:36 (1788359796) Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre2 Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 186112 .. Extending file size to 1167424 .. failed Extending file size to 6039008 .. Extending file size to 5436544 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7068928 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7855392 .. Extending file size to 9347968 .. Extending file size to 6790752 .. Extending file size to 7192064 .. Extending file size to 7900448 .. Extending file size to 8232992 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8719552 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61609 61610 61611 61613 61617 done 192.168.201.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg136-client.virtnet /mnt/lustre2 (opts:) 192.168.201.136@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg136-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 10:37:55 (1788359875) Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre2 Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5790624 .. Extending file size to 7543392 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7834272 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7719488 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8616608 .. Extending file size to 8433120 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8587104 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 9264384 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8818912 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 63316 63317 63318 63322 63324 done 192.168.201.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg136-client.virtnet /mnt/lustre2 (opts:) 192.168.201.136@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg136-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (74s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 10:39:09 (1788359949) Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2312 1268992 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2292 1269012 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 24928 3582092 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18936 3588084 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43864 7170176 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2312 1268992 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2292 1269012 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 24928 3582092 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18936 3588084 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43864 7170176 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0788033 s, 26.6 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.151592 s, 13.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x5661834b (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg136-client.virtnet /mnt/lustre2 (opts:) PASS 200c (13s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 10:39:23 (1788359963) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.128324 s, 16.3 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 10:39:28 (1788359968) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5e:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 10:39:34 (1788359974) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0743811 s, 28.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2d7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2d8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (5s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 10:39:39 (1788359979) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0766758 s, 27.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2d9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2d9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2da:0x0] } PASS 204b (5s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 10:39:44 (1788359984) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.060491 s, 34.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } PASS 204c (29s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 10:40:13 (1788360013) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg136-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054601 s, 19.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2db:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d9:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2dc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2db:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2da:0x0] } PASS 204d (43s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 10:40:57 (1788360057) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg136-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0731757 s, 28.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 10:41:41 (1788360101) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg136-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg136-server: Pool lustre.test_204f created Adding targets to pool oleg136-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg136-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0570107 s, 36.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg136-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg136-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg136-server: Pool lustre.test_204f destroyed PASS 204f (62s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 10:42:43 (1788360163) PASS 205a (5s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 10:42:48 (1788360168) PASS 205b (5s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 10:42:54 (1788360174) oleg136-server: Pool lustre.test_206 created Adding targets to pool oleg136-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg136-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg136-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg136-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg136-server: Pool lustre.test_206 destroyed PASS 206 (21s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 10:43:15 (1788360195) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0829609 s, 12.6 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266877 s, 39.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xb4caef1 Mirror 3: 0xb4caef1 PASS 207 (7s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 10:43:22 (1788360202) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 10:43:25 (1788360205) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 10:43:27 (1788360207) Starting client: oleg136-client.virtnet: -o user_xattr,flock 192.168.201.136@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0403469 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x8f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x19c0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x19c0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x19c0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x10d0000 Read this (4 bytes): BBBB 192.168.201.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg136-client.virtnet /mnt/lustre2 (opts:) PASS 209a (29s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 10:43:56 (1788360236) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0165539 s, 495 kB/s the very first read read with warm cache PASS 209b (5s) == sanity-flr test 210a: handle broken mirrored lovea ==== 10:44:02 (1788360242) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291698 s, 35.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7d lmm_fid: [0x200000405:0x7d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7d lmm_fid: [0x200000405:0x7d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611261 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-02 10:44:03.000000000 -0400 Modify: 2026-09-02 10:44:03.000000000 -0400 Change: 2026-09-02 10:44:04.000000000 -0400 Birth: 2026-09-02 10:44:02.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 10:44:08 (1788360248) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345734 s, 30.3 MB/s oleg136-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg136-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg136-server: O/280000401/d28/13c: File not found by ext2_lookup PASS 210b (21s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 10:44:29 (1788360269) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.134904 s, 304 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0161172 s, 254 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 2 10:44 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x82:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 2 10:44 /mnt/lustre/f211.sanity-flr PASS 211 (5s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 10:44:34 (1788360274) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (5s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 10:44:40 (1788360280) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (6s) == sanity-flr test complete, duration 1533 sec =========== 10:44:46 (1788360286) === sanity-flr: start cleanup 10:44:47 (1788360287) === === sanity-flr: finish cleanup 10:44:52 (1788360292) ===