-----============= acceptance-small: sanity-flr ============----- Sun Sep 6 22:32:35 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg240-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 22:32:55 (1788748375) === oleg240-client.virtnet: executing check_config_client /mnt/lustre oleg240-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg240-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93686062a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff93686062a800.idle_timeout=debug disable quota as required oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 22:33:28 (1788748408) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7614] == sanity-flr test 0a: lfs mirror create with -N option == 22:33:31 (1788748411) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:33:46 (1788748426) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:33:49 (1788748429) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:33:52 (1788748432) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:34:16 (1788748456) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:34:21 (1788748461) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (5s) == sanity-flr test 0g: lfs mirror create flags support === 22:34:25 (1788748465) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff93686062a800.stats=clear osc.lustre-OST0001-osc-ffff93686062a800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:34:35 (1788748475) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:34:48 (1788748488) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:34:56 (1788748496) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:35:00 (1788748500) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:35:07 (1788748507) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 22:35:18 (1788748518) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:35:27 (1788748527) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:35:31 (1788748531) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:35:37 (1788748537) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:35:47 (1788748547) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:35:55 (1788748555) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.521756 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121407 s, 8.6 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:36:05 (1788748565) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.231987 s, 36.2 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:36:13 (1788748573) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (9s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:36:23 (1788748583) Stopping /mnt/lustre-ost1 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93686062a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93686062a800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg240-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93686062a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93686062a800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg240-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (71s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:37:34 (1788748654) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.384779 s, 19.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93686062a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93686062a800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg240-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:38:16 (1788748696) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93686062a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93686062a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg240-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93686062a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93686062a800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg240-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (69s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:39:25 (1788748765) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 4.89245 s, 10.7 MB/s -rw-r--r-- 1 root root 52428800 Sep 6 22:39 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.024s user 0m0.001s sys 0m0.166s Stopping /mnt/lustre-ost1 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93686062a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93686062a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.956s user 0m0.001s sys 0m0.176s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg240-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93686062a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93686062a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.978s user 0m0.002s sys 0m0.179s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg240-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg240-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93686062a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93686062a800.ost_server_uuid in FULL state after 0 sec PASS 33b (79s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:40:44 (1788748844) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:40:47 (1788748847) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:40:50 (1788748850) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 22:40:52 (1788748852) PASS 35 (6s) == sanity-flr test 36a: write to mirrored files ========== 22:40:58 (1788748858) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.516781 s, 24.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (9s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:41:08 (1788748868) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417776 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363077 s, 28.9 MB/s PASS 36b (5s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:41:13 (1788748873) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:41:20 (1788748880) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00118315 s, 865 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00108444 s, 944 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00060673 s, 1.7 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000946764 s, 1.1 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 22:41:29 (1788748889) create test files with size 15729393 .. c62b04e438002e5d7aa1446a0ca2b1f1 - 0e5768af78fe4b0464bd3f5325986221 - 7621884ea3f408762a7835a4c2e2b5f0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg240-client.virtnet /mnt/lustre (opts:) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre PASS 37 (20s) == sanity-flr test 38: resync ============================ 22:41:49 (1788748909) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:42:01 (1788748921) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:42:07 (1788748927) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613428 s, 17.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0407893 s, 25.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xe0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 22:42:15 (1788748935) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114091 s, 18.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.390357 s, 10.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.15171 s, 13.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 22:42:28 (1788748948) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 22:42:31 (1788748951) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:42:34 (1788748954) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359756 s, 29.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 22:42:42 (1788748962) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:42:45 (1788748965) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc6:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:42:55 (1788748975) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.230735 s, 45.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg240-client.virtnet /mnt/lustre (opts:) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:43:05 (1788748985) llite.lustre-ffff936847b66000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.354437 s, 29.6 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg240-client.virtnet /mnt/lustre (opts:) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff93684597e800.xattr_cache=1 PASS 44e (12s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:43:17 (1788748997) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (32s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:43:49 (1788749029) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (30s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:44:20 (1788749060) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 22:44:23 (1788749063) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.197578 s, 15.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.19242 s, 16.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x39ba753b (nosync,stale) Mirror 2: 0x39ba753b (nosync,stale) ** verify mirror 2 doesn't change original checksum: 3b8c3d4e881d550a0724d404e04e00a1 - mirror 1 checksum: 0617f55cb22a93987d473ba2c53a5946 - mirror 2 checksum: 3b8c3d4e881d550a0724d404e04e00a1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-06T22:44:26-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-06T22:44:26-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 3b8c3d4e881d550a0724d404e04e00a1 - mirror 1 checksum: 0617f55cb22a93987d473ba2c53a5946 - mirror 2 checksum: 0617f55cb22a93987d473ba2c53a5946 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-06T22:44:26-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-06T22:44:26-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } PASS 48 (10s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:44:35 (1788749075) Failing mds1 on oleg240-server Stopping /mnt/lustre-mds1 (opts:) on oleg240-server 22:44:40 (1788749080) shut down facet: mds1 facet_host: oleg240-server facet_failover_host: oleg240-server Failover mds1 to oleg240-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-MDT0000 22:45:04 (1788749104) targets are mounted 22:45:04 (1788749104) facet_failover done oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0695022 s, 15.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 50A (45s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 22:45:20 (1788749120) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (2s) == sanity-flr test 50b: mirror rsync handles sparseness == 22:45:23 (1788749123) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 22:45:26 (1788749126) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00817338 s, 2.0 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 22:45:33 (1788749133) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 22:45:36 (1788749136) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00523509 s, 782 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0290649 s, 141 kB/s PASS 60a (6s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 22:45:43 (1788749143) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 22:46:02 (1788749162) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 22:46:29 (1788749189) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 22:46:51 (1788749211) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (23s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 22:47:14 (1788749234) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 22:47:21 (1788749241) mirror create pid 54288 mirror split pid 54289 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54288 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 54289 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 22:48:27 (1788749307) Seed set to 9308 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x1a8ef 000002 1788749309.083900 trunc from 00000000 to 0x01a8ee (0x1a8ef bytes) 000003 1788749309.100612 mapread 0x007234 thru 0x00bf10 (0x4cdd bytes) 000004 1788749309.118986 mapread 0x010e5b thru 0x01a8ee (0x9a94 bytes) 000006 1788749309.128544 mapread 0x0046eb thru 0x00e946 (0xa25c bytes) 000007 1788749309.133107 MIRROR_EXTEND to 2 mirrors 000008 1788749309.612874 read 0x0030d8 thru 0x007492 (0x43bb bytes) 000009 1788749309.629041 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) 000013 1788749309.756621 read 0x012340 thru 0x01a8ee (0x85af bytes) 000016 1788749309.769835 read 0x01398b thru 0x0182d4 (0x494a bytes) 000018 1788749309.771168 mapread 0x00e97e thru 0x016f3a (0x85bd bytes) 000019 1788749309.780187 punch from 0x012b62 to 0x01a8ee (0x7d8d bytes) 000021 1788749309.794139 read 0x00cbd7 thru 0x012d61 (0x618b bytes) 000023 1788749309.802251 read 0x0131e6 thru 0x01a8ee (0x7709 bytes) 000025 1788749309.803912 read 0x001e3f thru 0x00883b (0x69fd bytes) 000027 1788749309.805629 mapread 0x00081d thru 0x01075b (0xff3f bytes) 000028 1788749309.807099 write 0x0265fa thru 0x030562 (0x9f69 bytes) 000029 1788749309.809048 read 0x02e027 thru 0x030562 (0x253c bytes) 000030 1788749309.810065 trunc from 0x030563 to 0x003f60 (0x2c602 bytes) 000031 1788749309.853787 mapread 0x001bc6 thru 0x003f60 (0x239b bytes) 000032 1788749309.855161 write 0x019657 thru 0x01e331 (0x4cdb bytes) 000033 1788749309.856775 mapread 0x003cf8 thru 0x00c0e5 (0x83ee bytes) truncating to largest ever: 0x3b7cf 000035 1788749309.867204 trunc from 0x01e332 to 0x03b7ce (0x1d49d bytes) 000037 1788749309.888400 mapwrite 0x00ccb8 thru 0x019c42 (0xcf8b bytes) 000039 1788749340.004985 MIRROR_EXTEND to 2 mirrors 000041 1788749340.874853 trunc from 0x03b7cf to 0x01eb78 (0x1cc56 bytes) 000044 1788749340.964571 read 0x01c74c thru 0x01eb78 (0x242d bytes) 000045 1788749340.971440 punch from 0x019e4c to 0x01eb78 (0x4d2d bytes) 000046 1788749340.976642 mapread 0x005557 thru 0x015047 (0xfaf1 bytes) 000048 1788749341.010849 trunc from 0x01eb79 to 0x01e452 (0x726 bytes) 000049 1788749341.032756 punch from 0x008ec1 to 0x00cb67 (0x3ca7 bytes) 000050 1788749341.037416 punch from 0x009101 to 0x00ba46 (0x2946 bytes) 000051 1788749341.041130 trunc from 0x01e453 to 0x033815 (0x153c3 bytes) 000052 1788749341.061753 punch from 0x01ab2c to 0x0234f8 (0x89cd bytes) 000053 1788749341.066535 mapread 0x0246fe thru 0x02866c (0x3f6f bytes) 000054 1788749341.094478 mapwrite 0x02bf69 thru 0x038e06 (0xce9e bytes) 000056 1788749341.255698 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x38e07): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38e07): Mirror 2: 0x30bf81d3 (stale) CRC-32 checksum value for chunk [0, 0x38e07): Mirror 1: 0xb49ccf40 Mirror 3: 0xb49ccf40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38e07: skipped 000057 1788749342.204000 write 0x016827 thru 0x022b40 (0xc31a bytes) 000058 1788749342.370353 mapwrite 0x01c6a8 thru 0x01c861 (0x1ba bytes) 000059 1788749342.471469 mapwrite 0x03069c thru 0x032d9a (0x26ff bytes) 000060 1788749342.533287 write 0x03dc79 thru 0x03ffff (0x2387 bytes) 000063 1788749342.538239 read 0x032975 thru 0x03c57c (0x9c08 bytes) 000064 1788749342.542973 mapwrite 0x03d704 thru 0x03ffff (0x28fc bytes) 000065 1788749342.612911 write 0x03d26b thru 0x03d8ee (0x684 bytes) 000066 1788749342.613887 punch from 0x0306c8 to 0x03459c (0x3ed5 bytes) 000067 1788749342.622501 write 0x034868 thru 0x03c51c (0x7cb5 bytes) 000069 1788749342.628850 punch from 0x010984 to 0x01937f (0x89fc bytes) 000070 1788749342.640429 write 0x00c399 thru 0x0124ff (0x6167 bytes) 000071 1788749342.649616 mapwrite 0x020017 thru 0x02cdd7 (0xcdc1 bytes) 000073 1788749342.701616 read 0x02b8f7 thru 0x02ccb0 (0x13ba bytes) 000074 1788749342.702462 write 0x03bb6d thru 0x03ffff (0x4493 bytes) 000075 1788749342.705052 write 0x016945 thru 0x023d04 (0xd3c0 bytes) 000076 1788749342.722659 punch from 0x00ed36 to 0x018695 (0x9960 bytes) 000077 1788749342.737372 punch from 0x027d1d to 0x02d5fc (0x58e0 bytes) 000079 1788749342.742554 punch from 0x018465 to 0x01deaa (0x5a46 bytes) 000080 1788749342.746901 read 0x027c92 thru 0x032ae9 (0xae58 bytes) 000081 1788749342.755066 write 0x02abfa thru 0x02e989 (0x3d90 bytes) 000082 1788749342.756197 punch from 0x0196d2 to 0x01e009 (0x4938 bytes) 000083 1788749342.768549 mapwrite 0x03f9da thru 0x03ffff (0x626 bytes) 000086 1788749342.803849 write 0x039acf thru 0x03ffff (0x6531 bytes) 000087 1788749342.805605 read 0x025096 thru 0x02c56a (0x74d5 bytes) 000089 1788749342.806901 mapread 0x0054a5 thru 0x010b0a (0xb666 bytes) 000090 1788749342.813052 MIRROR_RESYNC 3 mirrors 000091 1788749343.641227 trunc from 0x040000 to 0x0384f7 (0x7b08 bytes) 000092 1788749343.706666 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x384f8): Mirror 3: 0x9cf8711c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x384f8): Mirror 3: 0x9cf8711c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x384f8: skipped 000093 1788749344.026241 mapread 0x03788d thru 0x0384f7 (0xc6b bytes) 000094 1788749344.051386 trunc from 0x0384f8 to 0x00cfca (0x2b52d bytes) 000095 1788749344.094182 mapwrite 0x0297be thru 0x030d62 (0x75a5 bytes) 000097 1788749344.176456 mapread 0x003b7d thru 0x005258 (0x16dc bytes) 000098 1788749344.203694 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30d63): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30d63): Mirror 3: 0x728e94ef (stale) CRC-32 checksum value for chunk [0, 0x30d63): Mirror 1: 0x8b192f06 Mirror 4: 0x8b192f06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30d63: skipped 000099 1788749345.142734 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30d63): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30d63): Mirror 3: 0x728e94ef (stale) CRC-32 checksum value for chunk [0, 0x30d63): Mirror 1: 0x8b192f06 Mirror 4: 0x8b192f06 Mirror 5: 0x8b192f06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30d63: skipped 000100 1788749346.181533 mapread 0x0169ad thru 0x01fe4f (0x94a3 bytes) 000101 1788749346.230792 write 0x02037d thru 0x020491 (0x115 bytes) 000102 1788749346.377557 mapread 0x00c685 thru 0x0116bc (0x5038 bytes) 000103 1788749346.392627 mapread 0x022447 thru 0x026757 (0x4311 bytes) 000105 1788749346.403760 punch from 0x02cd34 to 0x030d62 (0x402f bytes) 000107 1788749346.433220 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x30d63): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30d63): Mirror 3: 0x728e94ef (stale) Mirror 4: 0x8b192f06 (stale) Mirror 5: 0x8b192f06 (stale) CRC-32 checksum value for chunk [0, 0x30d63): Mirror 1: 0xd89f7f7e Mirror 6: 0xd89f7f7e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30d63: skipped 000108 1788749347.807921 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30d63): Mirror 4: 0x8b192f06 (stale) Mirror 5: 0x8b192f06 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30d63): Mirror 4: 0x8b192f06 (stale) Mirror 5: 0x8b192f06 (stale) CRC-32 checksum value for chunk [0, 0x30d63): Mirror 1: 0xd89f7f7e Mirror 6: 0xd89f7f7e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30d63: skipped 000109 1788749348.327825 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30d63): Mirror 4: 0x8b192f06 (stale) Mirror 5: 0x8b192f06 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30d63): Mirror 4: 0x8b192f06 (stale) Mirror 5: 0x8b192f06 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30d63: skipped 000110 1788749348.604707 mapread 0x012254 thru 0x0157d9 (0x3586 bytes) 000111 1788749348.621860 read 0x011005 thru 0x015c4e (0x4c4a bytes) 000112 1788749348.622591 mapwrite 0x035eb5 thru 0x03811b (0x2267 bytes) 000113 1788749348.693260 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3811c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3811c): Mirror 4: 0x9d76636c (stale) Mirror 5: 0x9d76636c (stale) CRC-32 checksum value for chunk [0, 0x3811c): Mirror 1: 0x5dcaa9dd Mirror 6: 0x5dcaa9dd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3811c: skipped 000115 1788749349.504381 write 0x008e95 thru 0x0098e2 (0xa4e bytes) 000116 1788749349.650189 read 0x030909 thru 0x033ea4 (0x359c bytes) 000117 1788749349.663578 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3811c): Mirror 4: 0x9d76636c (stale) Mirror 6: 0x5dcaa9dd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3811c): Mirror 4: 0x9d76636c (stale) Mirror 6: 0x5dcaa9dd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3811c: skipped 000118 1788749349.905822 mapread 0x019e50 thru 0x0274af (0xd660 bytes) 000119 1788749349.942150 MIRROR_RESYNC 3 mirrors 000120 1788749351.105684 mapwrite 0x009724 thru 0x00995e (0x23b bytes) 000123 1788749351.239683 punch from 0x0073e2 to 0x0104ea (0x9109 bytes) 000125 1788749351.251060 trunc from 0x03811c to 0x02bcd5 (0xc446 bytes) 000127 1788749351.286099 trunc from 0x02bcd6 to 0x02fa68 (0x3d93 bytes) 000129 1788749351.305949 read 0x023035 thru 0x02d014 (0x9fe0 bytes) 000130 1788749351.324922 mapwrite 0x0172d1 thru 0x02470e (0xd43e bytes) 000133 1788749351.392616 read 0x02aa2c thru 0x02c43b (0x1a10 bytes) 000134 1788749351.393623 mapwrite 0x03f360 thru 0x03ffff (0xca0 bytes) 000136 1788749351.452051 read 0x007e53 thru 0x009880 (0x1a2e bytes) 000137 1788749351.455269 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x50e87179 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x50e87179 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000138 1788749351.758496 mapwrite 0x0233f2 thru 0x023f47 (0xb56 bytes) 000139 1788749351.817590 punch from 0x0122d8 to 0x015ded (0x3b16 bytes) 000142 1788749351.823683 mapread 0x025c77 thru 0x0339f9 (0xdd83 bytes) 000144 1788749351.844724 read 0x00104f thru 0x00f58a (0xe53c bytes) 000145 1788749351.858583 MIRROR_RESYNC 2 mirrors 000146 1788749352.747629 read 0x023a14 thru 0x032e29 (0xf416 bytes) 000147 1788749352.754023 write 0x020a8f thru 0x02fb12 (0xf084 bytes) 000148 1788749352.858490 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) 000150 1788749353.064792 trunc from 0x040000 to 0x013d52 (0x2c2ad bytes) 000152 1788749353.083470 read 0x00694e thru 0x01246a (0xbb1d bytes) 000154 1788749353.100860 mapread 0x0036bd thru 0x00eef6 (0xb83a bytes) 000155 1788749353.118717 mapread 0x00bbc4 thru 0x013d52 (0x818f bytes) truncating to largest ever: 0x3be29 000160 1788749353.128538 trunc from 0x013d53 to 0x03be28 (0x280d6 bytes) 000161 1788749353.148686 trunc from 0x03be29 to 0x02c90e (0xf51a bytes) 000162 1788749353.171485 write 0x013ae3 thru 0x01463b (0xb59 bytes) 000163 1788749353.182803 punch from 0x014197 to 0x0153da (0x1244 bytes) 000164 1788749353.195639 mapwrite 0x009e4b thru 0x011607 (0x77bd bytes) 000165 1788749353.247689 write 0x01987c thru 0x020f1b (0x76a0 bytes) 000167 1788749353.262169 read 0x01284c thru 0x015c51 (0x3406 bytes) 000174 1788749353.271713 write 0x03a0a9 thru 0x03ffff (0x5f57 bytes) 000176 1788749353.273872 punch from 0x01b631 to 0x0239b7 (0x8387 bytes) 000177 1788749353.284286 mapwrite 0x0051da thru 0x009768 (0x458f bytes) 000178 1788749353.310263 punch from 0x031211 to 0x03dbe2 (0xc9d2 bytes) 000179 1788749353.313285 punch from 0x00a0bd to 0x018e70 (0xedb4 bytes) 000180 1788749353.316632 punch from 0x025f03 to 0x0294c3 (0x35c1 bytes) 000182 1788749353.320297 mapwrite 0x01bfc0 thru 0x02a50f (0xe550 bytes) 000183 1788749353.400801 trunc from 0x040000 to 0x03180f (0xe7f0 bytes) 000185 1788749353.417968 mapwrite 0x03d580 thru 0x03ffff (0x2a80 bytes) 000186 1788749353.507379 read 0x0054f1 thru 0x00970c (0x421c bytes) 000187 1788749353.507985 mapread 0x02f991 thru 0x035295 (0x5905 bytes) 000192 1788749353.520183 mapwrite 0x02d298 thru 0x033f8c (0x6cf5 bytes) 000193 1788749353.565566 MIRROR_EXTEND to 2 mirrors 000194 1788749354.131069 trunc from 0x040000 to 0x01dd9a (0x22265 bytes) 000195 1788749354.233941 mapwrite 0x011f0a thru 0x020b72 (0xec69 bytes) 000197 1788749354.339843 mapread 0x01ac2c thru 0x020b72 (0x5f47 bytes) 000199 1788749354.349142 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20b73): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20b73): Mirror 2: 0x1e63aec1 (stale) CRC-32 checksum value for chunk [0, 0x20b73): Mirror 1: 0x2e2359b0 Mirror 3: 0x2e2359b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20b73: skipped 000200 1788749354.985917 trunc from 0x020b73 to 0x01da50 (0x3122 bytes) 000202 1788749355.100139 trunc from 0x01da51 to 0x01eb97 (0x1147 bytes) 000204 1788749355.129957 read 0x00fe0c thru 0x0107c4 (0x9b9 bytes) 000205 1788749355.135173 write 0x00cbf4 thru 0x01bd37 (0xf144 bytes) 000206 1788749355.144808 read 0x00f893 thru 0x010fd3 (0x1741 bytes) 000207 1788749355.145454 trunc from 0x01eb98 to 0x00d564 (0x11633 bytes) 000208 1788749355.159766 mapread 0x00b26b thru 0x00d564 (0x22fa bytes) 000209 1788749355.170255 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd565): Mirror 2: 0x705ce369 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd565): Mirror 2: 0x705ce369 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd565: skipped 000211 1788749355.364936 read 0x00c4a2 thru 0x00d564 (0x10c3 bytes) 000212 1788749355.369577 mapwrite 0x02df68 thru 0x036b34 (0x8bcd bytes) 000216 1788749355.438276 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36b35): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36b35): Mirror 2: 0x2e3be665 (stale) CRC-32 checksum value for chunk [0, 0x36b35): Mirror 1: 0xe27d3f0c Mirror 3: 0xe27d3f0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36b35: skipped 000218 1788749356.193061 mapwrite 0x0360ed thru 0x03ffff (0x9f13 bytes) 000219 1788749356.351670 write 0x022b3d thru 0x02c478 (0x993c bytes) 000222 1788749356.361587 mapwrite 0x039b5d thru 0x03ffff (0x64a3 bytes) 000223 1788749356.410611 write 0x0038a6 thru 0x005500 (0x1c5b bytes) 000224 1788749356.420112 write 0x01587f thru 0x01cbd3 (0x7355 bytes) 000228 1788749356.433483 punch from 0x015d3e to 0x021818 (0xbadb bytes) 000229 1788749356.449822 read 0x01c54b thru 0x0250f2 (0x8ba8 bytes) 000230 1788749356.459136 read 0x02971e thru 0x032f98 (0x987b bytes) 000232 1788749356.479664 mapwrite 0x02807a thru 0x0323b5 (0xa33c bytes) 000235 1788749356.525957 read 0x013abc thru 0x01e4d9 (0xaa1e bytes) 000236 1788749356.531088 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd70f09d2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd70f09d2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000237 1788749356.767607 write 0x005268 thru 0x014ae3 (0xf87c bytes) 000239 1788749356.776550 write 0x0313dc thru 0x03dbc5 (0xc7ea bytes) 000240 1788749356.791897 trunc from 0x040000 to 0x02586a (0x1a795 bytes) 000242 1788749356.815800 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) 000243 1788749356.922839 mapread 0x00354e thru 0x012281 (0xed34 bytes) 000244 1788749356.943998 mapwrite 0x02c9ba thru 0x03607b (0x96c2 bytes) 000247 1788749356.993529 mapread 0x01ffe9 thru 0x025998 (0x59b0 bytes) 000248 1788749356.999586 write 0x004de5 thru 0x00c1bc (0x73d8 bytes) 000251 1788749357.005510 MIRROR_EXTEND to 2 mirrors 000252 1788749357.464610 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) truncating to largest ever: 0x3cac3 000254 1788749357.599950 trunc from 0x03607c to 0x03cac2 (0x6a47 bytes) 000255 1788749357.633834 read 0x012777 thru 0x017c25 (0x54af bytes) 000256 1788749357.644756 read 0x03456e thru 0x03cac2 (0x8555 bytes) 000258 1788749357.654746 mapwrite 0x028983 thru 0x03121a (0x8898 bytes) 000259 1788749357.749025 mapread 0x015e53 thru 0x01d767 (0x7915 bytes) 000260 1788749357.750597 write 0x036a6d thru 0x03b056 (0x45ea bytes) 000262 1788749357.752212 punch from 0x00d7a4 to 0x019c76 (0xc4d3 bytes) truncating to largest ever: 0x3f4b9 000263 1788749357.765083 trunc from 0x03cac3 to 0x03f4b8 (0x29f6 bytes) 000264 1788749357.783111 mapread 0x03c683 thru 0x03f4b8 (0x2e36 bytes) 000267 1788749357.790780 read 0x00c2b2 thru 0x00ec93 (0x29e2 bytes) 000268 1788749357.810191 punch from 0x01b677 to 0x0284ff (0xce89 bytes) 000269 1788749357.814862 mapread 0x02798c thru 0x02f560 (0x7bd5 bytes) 000271 1788749357.822695 mapread 0x0060bc thru 0x00968a (0x35cf bytes) 000273 1788749357.825696 write 0x01460b thru 0x01d148 (0x8b3e bytes) 000274 1788749357.830372 write 0x00a59f thru 0x011d19 (0x777b bytes) 000276 1788749357.831754 write 0x010a83 thru 0x01197c (0xefa bytes) 000277 1788749357.832333 mapread 0x03ebf4 thru 0x03f4b8 (0x8c5 bytes) 000278 1788749357.836354 MIRROR_EXTEND to 2 mirrors 000279 1788749358.424890 trunc from 0x03f4b9 to 0x028641 (0x16e77 bytes) 000281 1788749358.483699 trunc from 0x028642 to 0x0307e0 (0x819f bytes) 000282 1788749358.495449 punch from 0x01a4ff to 0x01efa2 (0x4aa4 bytes) 000283 1788749358.498814 mapread 0x00703b thru 0x012eb0 (0xbe76 bytes) 000284 1788749358.516933 mapread 0x00756a thru 0x009b30 (0x25c7 bytes) 000285 1788749358.517731 mapread 0x00448e thru 0x011273 (0xcde6 bytes) 000286 1788749358.519081 mapwrite 0x00f4d0 thru 0x014afa (0x562b bytes) 000287 1788749358.551602 trunc from 0x0307e1 to 0x004a2d (0x2bdb3 bytes) 000288 1788749358.580499 trunc from 0x004a2e to 0x02448a (0x1fa5d bytes) 000289 1788749358.589986 mapwrite 0x025c15 thru 0x0276fb (0x1ae7 bytes) 000290 1788749358.648821 punch from 0x0034cc to 0x004cdc (0x1811 bytes) 000292 1788749358.653813 mapread 0x01fee8 thru 0x02241c (0x2535 bytes) 000294 1788749358.655189 punch from 0x0074e2 to 0x009aa5 (0x25c4 bytes) 000296 1788749358.662002 punch from 0x0071a5 to 0x01057b (0x93d7 bytes) 000297 1788749358.666378 trunc from 0x0276fc to 0x03ac12 (0x13517 bytes) 000298 1788749358.683731 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) 000300 1788749358.801031 punch from 0x032856 to 0x03ac12 (0x83bd bytes) 000302 1788749358.808763 mapread 0x0193d8 thru 0x01f0fc (0x5d25 bytes) 000304 1788749358.830206 write 0x02059b thru 0x0220a4 (0x1b0a bytes) 000306 1788749358.832145 mapread 0x01be03 thru 0x01d756 (0x1954 bytes) 000308 1788749358.833719 trunc from 0x03ac13 to 0x007a4f (0x331c3 bytes) 000309 1788749358.847902 write 0x03674f thru 0x038653 (0x1f05 bytes) 000312 1788749358.850284 mapread 0x0236cf thru 0x02cfb7 (0x98e9 bytes) 000313 1788749358.864224 punch from 0x012264 to 0x0219a9 (0xf746 bytes) 000314 1788749358.870195 mapwrite 0x00725e thru 0x0154a9 (0xe24c bytes) 000315 1788749358.924147 mapread 0x02677d thru 0x029eb5 (0x3739 bytes) 000316 1788749358.925493 mapwrite 0x01d288 thru 0x01db03 (0x87c bytes) 000320 1788749358.970033 mapwrite 0x025231 thru 0x02e160 (0x8f30 bytes) 000321 1788749359.010192 mapwrite 0x026cd8 thru 0x02f38c (0x86b5 bytes) 000323 1788749359.066247 trunc from 0x038654 to 0x0256c2 (0x12f91 bytes) 000325 1788749359.087470 read 0x013b2d thru 0x019c40 (0x6114 bytes) 000327 1788749359.088480 trunc from 0x0256c3 to 0x005619 (0x200a9 bytes) 000328 1788749359.099235 write 0x02eea6 thru 0x0357cc (0x6927 bytes) 000329 1788749359.101431 read 0x001307 thru 0x00981f (0x8519 bytes) 000330 1788749359.110391 trunc from 0x0357cd to 0x032686 (0x3146 bytes) 000332 1788749359.122795 punch from 0x02dd5d to 0x032686 (0x492a bytes) 000333 1788749359.134201 trunc from 0x032687 to 0x01cf97 (0x156ef bytes) 000335 1788749359.154134 write 0x02ac0a thru 0x03029b (0x5692 bytes) 000336 1788749359.155741 read 0x01ba4d thru 0x026d50 (0xb304 bytes) 000338 1788749359.166741 trunc from 0x03029c to 0x027daa (0x84f1 bytes) 000341 1788749359.178032 read 0x00203a thru 0x010268 (0xe22f bytes) 000342 1788749359.178663 write 0x011905 thru 0x01250a (0xc06 bytes) 000343 1788749359.179411 read 0x00ac25 thru 0x01225e (0x763a bytes) 000344 1788749359.180053 read 0x01a80b thru 0x01ad61 (0x557 bytes) 000346 1788749359.181033 read 0x005575 thru 0x00a140 (0x4bcc bytes) 000349 1788749359.182621 write 0x000ac9 thru 0x00b77b (0xacb3 bytes) 000352 1788749359.184806 mapread 0x010d13 thru 0x0187fb (0x7ae9 bytes) 000355 1788749359.189695 MIRROR_EXTEND to 2 mirrors 000357 1788749359.796145 write 0x00274d thru 0x00be46 (0x96fa bytes) 000361 1788749359.850738 punch from 0x0220f2 to 0x0268a2 (0x47b1 bytes) 000362 1788749359.866341 mapwrite 0x009664 thru 0x01920d (0xfbaa bytes) 000363 1788749359.975972 mapwrite 0x03ee91 thru 0x03ffff (0x116f bytes) 000366 1788749360.068872 mapread 0x01ac2d thru 0x01c3d3 (0x17a7 bytes) 000367 1788749360.070381 mapread 0x02568c thru 0x02b7d2 (0x6147 bytes) 000368 1788749360.080814 write 0x014b63 thru 0x01a159 (0x55f7 bytes) 000369 1788749360.082532 trunc from 0x040000 to 0x03e713 (0x18ec bytes) 000370 1788749360.096705 punch from 0x034d52 to 0x03e713 (0x99c2 bytes) 000371 1788749360.106067 mapread 0x032fa6 thru 0x036179 (0x31d4 bytes) 000372 1788749360.113482 mapwrite 0x00a18c thru 0x011fef (0x7e64 bytes) 000373 1788749360.142092 read 0x02de8c thru 0x038e13 (0xaf88 bytes) 000374 1788749360.144340 read 0x034585 thru 0x038abc (0x4538 bytes) 000375 1788749360.144913 punch from 0x02de0f to 0x039f8b (0xc17d bytes) 000376 1788749360.151523 trunc from 0x03e714 to 0x00871d (0x35ff6 bytes) 000377 1788749360.175702 mapread 0x000569 thru 0x0073c7 (0x6e5f bytes) 000378 1788749360.180129 mapread 0x0077c4 thru 0x00871d (0xf5a bytes) 000379 1788749360.183668 write 0x00ba5f thru 0x010503 (0x4aa5 bytes) 000380 1788749360.191105 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x10504): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10504): Mirror 2: 0xe2f6fdc1 (stale) CRC-32 checksum value for chunk [0, 0x10504): Mirror 1: 0x6a6762b3 Mirror 3: 0x6a6762b3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10504: skipped 000381 1788749360.792256 trunc from 0x010504 to 0x010c84 (0x781 bytes) 000382 1788749360.874105 trunc from 0x010c85 to 0x00b95b (0x5329 bytes) 000384 1788749360.894076 read 0x00aacc thru 0x00b95b (0xe90 bytes) 000385 1788749360.899034 write 0x01381f thru 0x01ea11 (0xb1f3 bytes) 000386 1788749360.900906 write 0x030407 thru 0x03e545 (0xe13f bytes) 000388 1788749360.903583 write 0x003ceb thru 0x004b5e (0xe74 bytes) 000389 1788749360.911645 punch from 0x00ac11 to 0x00c718 (0x1b08 bytes) 000391 1788749360.923139 trunc from 0x03e546 to 0x01efef (0x1f556 bytes) 000392 1788749360.939558 MIRROR_RESYNC 3 mirrors 000393 1788749361.497076 punch from 0x0114b5 to 0x012803 (0x134f bytes) 000395 1788749361.551657 trunc from 0x01eff0 to 0x02e029 (0xf03a bytes) 000396 1788749361.564241 read 0x022c7e thru 0x029c15 (0x6f98 bytes) 000398 1788749361.570741 write 0x03dbc7 thru 0x03ffff (0x2439 bytes) 000399 1788749361.571648 read 0x018955 thru 0x021167 (0x8813 bytes) 000400 1788749361.577505 read 0x012b81 thru 0x018484 (0x5904 bytes) 000401 1788749361.577969 mapread 0x03ff91 thru 0x03ffff (0x6f bytes) 000403 1788749361.578827 trunc from 0x040000 to 0x01cf4a (0x230b5 bytes) 000407 1788749361.593573 read 0x00c3bd thru 0x013f4a (0x7b8e bytes) 000409 1788749361.594560 mapread 0x003c42 thru 0x011d3f (0xe0fe bytes) 000410 1788749361.596497 read 0x013483 thru 0x014300 (0xe7e bytes) 000412 1788749361.597288 mapwrite 0x019bf1 thru 0x0263a3 (0xc7b3 bytes) 000414 1788749361.659733 write 0x021a02 thru 0x02246d (0xa6c bytes) 000415 1788749361.668250 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x263a4): Mirror 3: 0x2bab582a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x263a4): Mirror 3: 0x2bab582a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x263a4: skipped 000416 1788749361.899639 trunc from 0x0263a4 to 0x03d5c7 (0x17224 bytes) 000424 1788749361.924062 read 0x002877 thru 0x004cdb (0x2465 bytes) 000425 1788749361.930442 write 0x02c1fb thru 0x033223 (0x7029 bytes) 000426 1788749361.944129 MIRROR_RESYNC 2 mirrors 000428 1788749362.490786 read 0x03824d thru 0x03d5c7 (0x537b bytes) 000431 1788749362.497617 write 0x01b479 thru 0x0253a8 (0x9f30 bytes) 000432 1788749362.551940 trunc from 0x03d5c8 to 0x00b9b7 (0x31c10 bytes) 000434 1788749362.574521 write 0x0098a8 thru 0x01028f (0x69e8 bytes) 000435 1788749362.580433 trunc from 0x010290 to 0x02a86e (0x1a5df bytes) 000436 1788749362.594426 mapwrite 0x00701c thru 0x008de3 (0x1dc8 bytes) 000437 1788749362.648440 read 0x000a1b thru 0x001d25 (0x130b bytes) 000439 1788749362.658497 read 0x009fe7 thru 0x0173a7 (0xd3c1 bytes) 000440 1788749362.659104 trunc from 0x02a86f to 0x01304a (0x17824 bytes) 000442 1788749362.676202 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) 000445 1788749362.779835 write 0x00947f thru 0x00df3d (0x4abf bytes) 000446 1788749362.786343 mapwrite 0x001acf thru 0x00b103 (0x9635 bytes) 000447 1788749362.837754 mapread 0x0116e1 thru 0x01304a (0x196a bytes) 000449 1788749362.851444 mapread 0x01295c thru 0x01304a (0x6ef bytes) 000450 1788749362.852194 mapread 0x00f788 thru 0x01304a (0x38c3 bytes) 000451 1788749362.852955 write 0x01d07b thru 0x0298d2 (0xc858 bytes) 000452 1788749362.855655 punch from 0x01f23f to 0x01fbd3 (0x995 bytes) 000453 1788749362.866960 write 0x000cb0 thru 0x0040eb (0x343c bytes) 000454 1788749362.872757 mapwrite 0x011659 thru 0x0160ee (0x4a96 bytes) 000455 1788749362.935849 write 0x02b5b4 thru 0x02d313 (0x1d60 bytes) 000456 1788749362.937234 mapwrite 0x00220c thru 0x00e137 (0xbf2c bytes) 000458 1788749362.974290 trunc from 0x02d314 to 0x015241 (0x180d2 bytes) 000459 1788749362.989688 punch from 0x00fada to 0x014837 (0x4d5e bytes) 000461 1788749362.998535 mapwrite 0x0193a5 thru 0x01fa47 (0x66a3 bytes) 000464 1788749363.086770 mapread 0x003ecb thru 0x00dbc9 (0x9cff bytes) 000468 1788749363.089269 trunc from 0x01fa48 to 0x023ca0 (0x4259 bytes) 000470 1788749363.126189 MIRROR_EXTEND to 2 mirrors 000472 1788749363.859162 mapwrite 0x005e1c thru 0x011c6c (0xbe51 bytes) 000474 1788749364.069574 trunc from 0x023ca1 to 0x012401 (0x1189f bytes) 000475 1788749364.093267 mapread 0x0049a9 thru 0x007045 (0x269d bytes) 000476 1788749364.095612 punch from 0x00a4db to 0x012401 (0x7f27 bytes) 000477 1788749364.101828 trunc from 0x012402 to 0x02c19e (0x19d9d bytes) 000480 1788749364.120948 read 0x01625f thru 0x024adc (0xe87e bytes) 000481 1788749364.132977 trunc from 0x02c19f to 0x014eaf (0x172ef bytes) 000482 1788749364.166888 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x14eb0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14eb0): Mirror 2: 0xe0c00784 (stale) CRC-32 checksum value for chunk [0, 0x14eb0): Mirror 1: 0xd42d3bfa Mirror 3: 0xd42d3bfa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14eb0: skipped 000484 1788749364.930847 read 0x012816 thru 0x012c3d (0x428 bytes) 000485 1788749364.935094 write 0x025685 thru 0x02edf7 (0x9773 bytes) 000486 1788749365.063473 read 0x01bd3b thru 0x0218e4 (0x5baa bytes) 000487 1788749365.072897 punch from 0x02ed6e to 0x02edf7 (0x8a bytes) 000490 1788749365.088423 read 0x011738 thru 0x01759b (0x5e64 bytes) 000491 1788749365.088878 trunc from 0x02edf8 to 0x01ff1d (0xeeda bytes) 000493 1788749365.131125 read 0x01d3dc thru 0x01ff1d (0x2b42 bytes) 000494 1788749365.131902 trunc from 0x01ff1e to 0x035a2d (0x15b10 bytes) 000495 1788749365.148495 mapread 0x005604 thru 0x0063e8 (0xde5 bytes) 000497 1788749365.151115 mapwrite 0x00dde4 thru 0x017f14 (0xa131 bytes) 000499 1788749365.211722 mapwrite 0x03cda8 thru 0x03ffff (0x3258 bytes) 000500 1788749365.307263 punch from 0x014238 to 0x01aa2f (0x67f8 bytes) 000501 1788749365.331554 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x5e68e3c4 (stale) Mirror 3: 0x8ff3a66e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7f853f5d Mirror 4: 0x7f853f5d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000503 1788749366.155174 trunc from 0x040000 to 0x000589 (0x3fa76 bytes) truncating to largest ever: 0x3fa6a 000504 1788749366.260561 trunc from 0x00058a to 0x03fa69 (0x3f4e0 bytes) 000505 1788749366.280061 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fa6a): Mirror 3: 0x5766a61b (stale) Mirror 4: 0x851d9852 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fa6a): Mirror 3: 0x5766a61b (stale) Mirror 4: 0x851d9852 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fa6a: skipped 000506 1788749366.582786 mapread 0x01e29c thru 0x026c98 (0x89fd bytes) 000507 1788749366.606616 MIRROR_RESYNC 3 mirrors 000508 1788749367.295787 mapread 0x0037eb thru 0x008dca (0x55e0 bytes) 000509 1788749367.321631 write 0x038c63 thru 0x03ffff (0x739d bytes) 000510 1788749367.417121 MIRROR_RESYNC 3 mirrors 000511 1788749368.127467 write 0x015b3b thru 0x01b062 (0x5528 bytes) 000512 1788749368.205830 MIRROR_RESYNC 3 mirrors 000515 1788749369.551494 trunc from 0x040000 to 0x0146dc (0x2b923 bytes) 000516 1788749369.630762 trunc from 0x0146dd to 0x03e8f3 (0x2a217 bytes) 000517 1788749369.661627 mapread 0x010801 thru 0x01c42f (0xbc2f bytes) 000518 1788749369.705514 write 0x03c924 thru 0x03ffff (0x36dc bytes) 000519 1788749369.709575 write 0x010b23 thru 0x01b426 (0xa904 bytes) 000520 1788749369.710414 write 0x0344de thru 0x03ffff (0xbb22 bytes) 000522 1788749369.714040 punch from 0x00c8ca to 0x01a9e6 (0xe11d bytes) 000523 1788749369.723131 punch from 0x00e45f to 0x00fb8a (0x172c bytes) 000524 1788749369.728768 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa7ea149a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xa7ea149a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000525 1788749369.957170 mapread 0x00007d thru 0x00d56a (0xd4ee bytes) 000526 1788749369.982493 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) 000530 1788749370.149447 write 0x02ef48 thru 0x030190 (0x1249 bytes) 000531 1788749370.165392 read 0x01753b thru 0x01c821 (0x52e7 bytes) 000532 1788749370.182580 MIRROR_EXTEND to 2 mirrors 000533 1788749370.859531 mapread 0x029c05 thru 0x0391e5 (0xf5e1 bytes) 000534 1788749370.919169 MIRROR_RESYNC 2 mirrors 000535 1788749371.228832 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) 000538 1788749371.337018 write 0x0185cb thru 0x02093f (0x8375 bytes) 000539 1788749371.361215 read 0x035c51 thru 0x03ffff (0xa3af bytes) 000541 1788749371.371790 write 0x018f66 thru 0x024b3a (0xbbd5 bytes) 000542 1788749371.377487 mapread 0x0034b2 thru 0x0099a2 (0x64f1 bytes) 000543 1788749371.397822 trunc from 0x040000 to 0x00786e (0x38791 bytes) 000545 1788749371.441540 mapread 0x004213 thru 0x00786e (0x365c bytes) 000546 1788749371.443384 read 0x0017be thru 0x00786e (0x60b1 bytes) 000547 1788749371.444044 trunc from 0x00786f to 0x026d22 (0x1f4b4 bytes) 000548 1788749371.465449 read 0x02198e thru 0x026d22 (0x5395 bytes) 000550 1788749371.479851 mapread 0x01de21 thru 0x02221a (0x43fa bytes) 000551 1788749371.481844 mapread 0x0217c1 thru 0x026d22 (0x5562 bytes) 000553 1788749371.484675 read 0x019a9c thru 0x022c49 (0x91ae bytes) 000555 1788749371.486957 write 0x010774 thru 0x0153fd (0x4c8a bytes) 000556 1788749371.489698 mapwrite 0x00e3aa thru 0x012831 (0x4488 bytes) 000557 1788749371.550188 MIRROR_EXTEND to 2 mirrors 000558 1788749372.311872 MIRROR_EXTEND to 3 mirrors 000559 1788749372.885579 write 0x014317 thru 0x02368c (0xf376 bytes) 000560 1788749372.990842 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26d23): Mirror 3: 0x6ef256e7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26d23): Mirror 3: 0x6ef256e7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26d23: skipped 000563 1788749373.257764 punch from 0x01c1c4 to 0x0224d2 (0x630f bytes) 000564 1788749373.261706 read 0x01d5b2 thru 0x0215da (0x4029 bytes) 000566 1788749373.272377 read 0x01b05f thru 0x02483b (0x97dd bytes) 000567 1788749373.285372 mapread 0x00677f thru 0x013d8e (0xd610 bytes) 000568 1788749373.287262 trunc from 0x026d23 to 0x02fab4 (0x8d92 bytes) 000569 1788749373.304211 write 0x003d8a thru 0x00b78e (0x7a05 bytes) 000570 1788749373.305938 trunc from 0x02fab5 to 0x03978e (0x9cda bytes) 000571 1788749373.324273 mapwrite 0x008fe6 thru 0x00ddbc (0x4dd7 bytes) 000574 1788749373.392849 mapread 0x007d33 thru 0x008e46 (0x1114 bytes) 000577 1788749373.400445 mapwrite 0x00800e thru 0x008210 (0x203 bytes) 000579 1788749373.470954 punch from 0x0090ab to 0x016249 (0xd19f bytes) 000580 1788749373.476158 read 0x0091e1 thru 0x011587 (0x83a7 bytes) 000582 1788749373.497074 MIRROR_RESYNC 2 mirrors 000583 1788749374.049896 punch from 0x032324 to 0x03978e (0x746b bytes) 000584 1788749374.118860 mapread 0x031652 thru 0x03978e (0x813d bytes) 000585 1788749374.138685 mapread 0x029a0b thru 0x02d8ee (0x3ee4 bytes) 000586 1788749374.146718 read 0x012f1b thru 0x0196ae (0x6794 bytes) 000587 1788749374.157696 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) 000588 1788749374.293828 MIRROR_EXTEND to 2 mirrors 000589 1788749375.070959 trunc from 0x03978f to 0x02bebe (0xd8d0 bytes) 000593 1788749375.158068 write 0x00bd77 thru 0x017502 (0xb78c bytes) 000596 1788749375.171361 mapread 0x004d8c thru 0x01020c (0xb481 bytes) 000597 1788749375.194586 write 0x002610 thru 0x00520c (0x2bfd bytes) 000598 1788749375.198159 mapread 0x000498 thru 0x002717 (0x2280 bytes) 000599 1788749375.206297 mapwrite 0x02760f thru 0x031f87 (0xa979 bytes) 000600 1788749375.323659 read 0x02536c thru 0x029c7f (0x4914 bytes) 000601 1788749375.324239 mapread 0x0176e5 thru 0x01e2b0 (0x6bcc bytes) 000602 1788749375.335007 trunc from 0x031f88 to 0x024d70 (0xd217 bytes) 000603 1788749375.360177 trunc from 0x024d71 to 0x00170d (0x23663 bytes) 000605 1788749375.377730 mapread 0x001634 thru 0x00170d (0xda bytes) 000606 1788749375.384848 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) 000607 1788749375.584755 mapwrite 0x023e9d thru 0x0243da (0x53e bytes) 000610 1788749375.677262 write 0x03c919 thru 0x03ffff (0x36e7 bytes) 000612 1788749375.678636 trunc from 0x040000 to 0x015f1f (0x2a0e0 bytes) 000613 1788749375.702672 punch from 0x005e7d to 0x014578 (0xe6fc bytes) 000614 1788749375.710281 read 0x010495 thru 0x0123a3 (0x1f0f bytes) 000618 1788749375.737017 punch from 0x0108dd to 0x015f1f (0x5643 bytes) 000619 1788749375.743572 trunc from 0x015f20 to 0x00cbb3 (0x936c bytes) 000621 1788749375.765946 write 0x02c48b thru 0x03b5d5 (0xf14b bytes) 000622 1788749375.768468 write 0x034992 thru 0x0366e5 (0x1d54 bytes) 000623 1788749375.770289 write 0x0393fb thru 0x03ffff (0x6c05 bytes) 000627 1788749375.773221 write 0x0076de thru 0x011e73 (0xa796 bytes) 000628 1788749375.789499 MIRROR_EXTEND to 2 mirrors 000631 1788749376.645971 punch from 0x020b5d to 0x026d80 (0x6224 bytes) 000632 1788749376.714119 mapwrite 0x02d27f thru 0x033000 (0x5d82 bytes) 000633 1788749376.784468 mapwrite 0x03eb27 thru 0x03ffff (0x14d9 bytes) 000634 1788749376.842809 write 0x02e9cd thru 0x037096 (0x86ca bytes) 000635 1788749376.844298 punch from 0x018754 to 0x022764 (0xa011 bytes) 000637 1788749376.857174 punch from 0x0181bb to 0x0257b5 (0xd5fb bytes) 000638 1788749376.861100 write 0x020b0c thru 0x027c2d (0x7122 bytes) 000639 1788749376.869984 punch from 0x0141bc to 0x01804d (0x3e92 bytes) 000640 1788749376.877261 write 0x03886d thru 0x03ffff (0x7793 bytes) 000642 1788749376.878783 write 0x01c491 thru 0x01d765 (0x12d5 bytes) 000643 1788749376.886300 write 0x004917 thru 0x00def7 (0x95e1 bytes) 000645 1788749376.894113 punch from 0x0066c8 to 0x012102 (0xba3b bytes) 000648 1788749376.914491 punch from 0x038e54 to 0x03f607 (0x67b4 bytes) 000649 1788749376.921815 write 0x01c3ac thru 0x02bfbe (0xfc13 bytes) 000650 1788749376.931756 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) 000651 1788749377.157697 write 0x037e67 thru 0x03ab82 (0x2d1c bytes) 000652 1788749377.172774 mapwrite 0x0337d2 thru 0x0338d9 (0x108 bytes) 000653 1788749377.270651 write 0x0251c4 thru 0x034962 (0xf79f bytes) 000655 1788749377.291829 mapwrite 0x03d48b thru 0x03ffff (0x2b75 bytes) 000656 1788749377.395760 mapwrite 0x030dd9 thru 0x034142 (0x336a bytes) 000657 1788749377.447116 mapwrite 0x038665 thru 0x03ffff (0x799b bytes) 000658 1788749377.503282 mapread 0x0098e2 thru 0x011b4d (0x826c bytes) 000659 1788749377.515171 mapwrite 0x02712d thru 0x02bdd6 (0x4caa bytes) 000660 1788749377.563941 read 0x00d3d7 thru 0x01b877 (0xe4a1 bytes) 000662 1788749377.572067 read 0x018477 thru 0x026625 (0xe1af bytes) 000663 1788749377.579072 trunc from 0x040000 to 0x038a9c (0x7563 bytes) 000666 1788749377.595756 mapread 0x01b51a thru 0x01ce19 (0x1900 bytes) 000667 1788749377.596775 mapwrite 0x0276bb thru 0x034e78 (0xd7be bytes) 000669 1788749377.651011 read 0x0104f0 thru 0x01d741 (0xd252 bytes) 000670 1788749377.651475 trunc from 0x038a9d to 0x02fb04 (0x8f98 bytes) 000671 1788749377.684789 mapread 0x00c855 thru 0x01b6f7 (0xeea3 bytes) 000673 1788749377.693974 write 0x028940 thru 0x031a0c (0x90cd bytes) 000675 1788749377.705055 trunc from 0x031a0d to 0x0153e3 (0x1c629 bytes) 000677 1788749377.734716 punch from 0x012d59 to 0x0153e3 (0x268b bytes) 000678 1788749377.751476 MIRROR_EXTEND to 2 mirrors 000679 1788749378.544350 trunc from 0x0153e4 to 0x009262 (0xc181 bytes) 000680 1788749378.630553 mapread 0x0008e7 thru 0x009262 (0x897c bytes) 000681 1788749378.658038 mapwrite 0x00ebe5 thru 0x01cb3f (0xdf5b bytes) 000682 1788749378.740760 trunc from 0x01cb40 to 0x00bd1f (0x10e20 bytes) 000683 1788749378.765903 MIRROR_RESYNC 2 mirrors truncating to largest ever: 0x3fbb0 000684 1788749379.306594 trunc from 0x00bd20 to 0x03fbaf (0x33e90 bytes) 000685 1788749379.394934 write 0x00b2f2 thru 0x00d8b3 (0x25c2 bytes) 000686 1788749379.417867 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) 000690 1788749379.586021 punch from 0x03e88b to 0x03fbaf (0x1325 bytes) 000691 1788749379.594446 mapwrite 0x03bf42 thru 0x03ffff (0x40be bytes) 000692 1788749379.694236 punch from 0x03b8e9 to 0x03cb55 (0x126d bytes) 000694 1788749379.700181 mapwrite 0x0193ad thru 0x01efd3 (0x5c27 bytes) 000695 1788749379.747577 read 0x01151d thru 0x017baf (0x6693 bytes) 000696 1788749379.752903 punch from 0x015b92 to 0x01be82 (0x62f1 bytes) 000697 1788749379.759008 write 0x03660f thru 0x03cedf (0x68d1 bytes) 000698 1788749379.765599 read 0x0390da thru 0x03de87 (0x4dae bytes) 000699 1788749379.770269 MIRROR_EXTEND to 2 mirrors 000700 1788749380.726242 read 0x0200c8 thru 0x02f9dc (0xf915 bytes) 000701 1788749380.746004 mapwrite 0x015e57 thru 0x0239c7 (0xdb71 bytes) 000702 1788749380.918702 write 0x032843 thru 0x03e550 (0xbd0e bytes) 000703 1788749380.930945 MIRROR_RESYNC 2 mirrors 000704 1788749381.622568 read 0x0170ac thru 0x01a10c (0x3061 bytes) 000705 1788749381.628931 read 0x0050a8 thru 0x007581 (0x24da bytes) 000706 1788749381.653405 MIRROR_EXTEND to 3 mirrors 000708 1788749382.225814 MIRROR_RESYNC 3 mirrors 000709 1788749382.533617 read 0x002929 thru 0x0048e3 (0x1fbb bytes) 000710 1788749382.543901 mapread 0x0170c6 thru 0x025cb7 (0xebf2 bytes) 000711 1788749382.577240 write 0x020f5d thru 0x0253a9 (0x444d bytes) 000712 1788749382.662619 trunc from 0x040000 to 0x033510 (0xcaef bytes) 000714 1788749382.686173 read 0x01589f thru 0x0176cf (0x1e31 bytes) 000715 1788749382.697867 mapwrite 0x008731 thru 0x015f7e (0xd84e bytes) 000716 1788749382.748297 trunc from 0x033511 to 0x02aeb4 (0x865c bytes) 000717 1788749382.770019 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2aeb5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aeb5): Mirror 2: 0x6960d71 (stale) Mirror 3: 0x6960d71 (stale) CRC-32 checksum value for chunk [0, 0x2aeb5): Mirror 1: 0xb01983fe Mirror 4: 0xb01983fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aeb5: skipped 000718 1788749383.853813 mapwrite 0x007366 thru 0x009830 (0x24cb bytes) 000720 1788749384.052878 mapread 0x014548 thru 0x01b510 (0x6fc9 bytes) 000722 1788749384.065953 read 0x006578 thru 0x00a1b7 (0x3c40 bytes) 000723 1788749384.066799 mapread 0x0034c4 thru 0x00e661 (0xb19e bytes) 000724 1788749384.068280 write 0x001b42 thru 0x004187 (0x2646 bytes) 000725 1788749384.073073 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2aeb5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aeb5): Mirror 2: 0x6960d71 (stale) Mirror 3: 0x6960d71 (stale) Mirror 4: 0xb01983fe (stale) CRC-32 checksum value for chunk [0, 0x2aeb5): Mirror 1: 0x7bd42969 Mirror 5: 0x7bd42969 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aeb5: skipped 000726 1788749384.986711 mapread 0x0238d8 thru 0x02aeb4 (0x75dd bytes) 000729 1788749385.054675 write 0x03d390 thru 0x03ffff (0x2c70 bytes) 000731 1788749385.129280 read 0x03899f thru 0x03ffff (0x7661 bytes) 000735 1788749385.142413 mapwrite 0x01f07e thru 0x024a24 (0x59a7 bytes) 000736 1788749385.194611 trunc from 0x040000 to 0x014e0a (0x2b1f5 bytes) 000739 1788749385.281479 mapwrite 0x0098fd thru 0x01243d (0x8b41 bytes) 000740 1788749385.409009 read 0x000e47 thru 0x00a631 (0x97eb bytes) 000741 1788749385.414139 trunc from 0x014e0b to 0x01559a (0x790 bytes) 000742 1788749385.433804 write 0x030082 thru 0x034e68 (0x4de7 bytes) 000743 1788749385.434747 mapread 0x007710 thru 0x009b01 (0x23f2 bytes) 000746 1788749385.436055 write 0x023028 thru 0x027dd7 (0x4db0 bytes) 000747 1788749385.446893 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x34e69): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34e69): Mirror 2: 0xd6759e63 (stale) Mirror 3: 0xd6759e63 (stale) Mirror 4: 0xeb3c7361 (stale) Mirror 5: 0x9129e4ed (stale) CRC-32 checksum value for chunk [0, 0x34e69): Mirror 1: 0x3e2dc58f Mirror 6: 0x3e2dc58f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34e69: skipped 000749 1788749386.333836 read 0x013380 thru 0x01c400 (0x9081 bytes) 000750 1788749386.354508 write 0x00ec9c thru 0x01a2ee (0xb653 bytes) 000751 1788749386.431728 punch from 0x006a43 to 0x01454f (0xdb0d bytes) 000752 1788749386.449839 MIRROR_RESYNC 6 mirrors 000754 1788749388.320172 MIRROR_RESYNC 6 mirrors 000758 1788749388.709080 trunc from 0x034e69 to 0x015056 (0x1fe12 bytes) 000759 1788749388.775829 mapread 0x002522 thru 0x00525c (0x2d3b bytes) 000760 1788749388.797056 punch from 0x00aeef to 0x015056 (0xa168 bytes) 000765 1788749388.805457 trunc from 0x015057 to 0x01302f (0x2027 bytes) 000766 1788749388.832823 punch from 0x0031c5 to 0x00b01c (0x7e58 bytes) 000768 1788749388.840946 mapread 0x00aad0 thru 0x00ce37 (0x2368 bytes) 000770 1788749388.856608 write 0x03cc3c thru 0x03fa81 (0x2e46 bytes) 000771 1788749388.858092 trunc from 0x03fa82 to 0x015f08 (0x29b79 bytes) 000773 1788749388.878911 mapwrite 0x00a186 thru 0x013666 (0x94e1 bytes) 000774 1788749388.941820 read 0x00c2e6 thru 0x00e8aa (0x25c5 bytes) 000777 1788749388.947794 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x15f09): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15f09): Mirror 2: 0x67da4e39 (stale) Mirror 3: 0x67da4e39 (stale) Mirror 4: 0x67da4e39 (stale) Mirror 5: 0x67da4e39 (stale) Mirror 6: 0x67da4e39 (stale) CRC-32 checksum value for chunk [0, 0x15f09): Mirror 1: 0x99c90403 Mirror 7: 0x99c90403 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15f09: skipped 000778 1788749389.564587 mapwrite 0x01e287 thru 0x029a02 (0xb77c bytes) 000780 1788749389.855842 trunc from 0x029a03 to 0x00ebcc (0x1ae36 bytes) 000781 1788749389.881932 write 0x006f60 thru 0x00a04b (0x30ec bytes) 000782 1788749389.892566 write 0x01057a thru 0x0181fd (0x7c84 bytes) 000785 1788749389.901738 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x181fe): Mirror 2: 0x741caf67 (stale) Mirror 3: 0x741caf67 (stale) Mirror 4: 0x741caf67 (stale) Mirror 5: 0x741caf67 (stale) Mirror 7: 0xf9eadc7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x181fe): Mirror 2: 0x741caf67 (stale) Mirror 3: 0x741caf67 (stale) Mirror 4: 0x741caf67 (stale) Mirror 5: 0x741caf67 (stale) Mirror 7: 0xf9eadc7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x181fe: skipped 000786 1788749390.250682 punch from 0x00abbd to 0x00f542 (0x4986 bytes) 000787 1788749390.254036 write 0x03a955 thru 0x03ffff (0x56ab bytes) 000788 1788749390.255227 read 0x0360e9 thru 0x03bc95 (0x5bad bytes) 000791 1788749390.262574 punch from 0x02793d to 0x0290c3 (0x1787 bytes) 000793 1788749390.273880 trunc from 0x040000 to 0x01753e (0x28ac1 bytes) 000795 1788749390.290111 read 0x010ddd thru 0x01753e (0x6762 bytes) 000796 1788749390.315461 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1753f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1753f): Mirror 2: 0xe4e68061 (stale) Mirror 3: 0xe4e68061 (stale) Mirror 4: 0xe4e68061 (stale) Mirror 5: 0xe4e68061 (stale) Mirror 7: 0x751b494d (stale) CRC-32 checksum value for chunk [0, 0x1753f): Mirror 1: 0x355a7e12 Mirror 8: 0x355a7e12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1753f: skipped 000797 1788749391.238877 read 0x016fb5 thru 0x01753e (0x58a bytes) 000798 1788749391.243293 trunc from 0x01753f to 0x023a3f (0xc501 bytes) 000799 1788749391.366477 mapwrite 0x00c928 thru 0x013775 (0x6e4e bytes) 000800 1788749391.419046 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23a40): Mirror 3: 0xa98118cf (stale) Mirror 4: 0xa98118cf (stale) Mirror 5: 0xa98118cf (stale) Mirror 7: 0x78d6b8b5 (stale) Mirror 8: 0xc3cf0c6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23a40): Mirror 3: 0xa98118cf (stale) Mirror 4: 0xa98118cf (stale) Mirror 5: 0xa98118cf (stale) Mirror 7: 0x78d6b8b5 (stale) Mirror 8: 0xc3cf0c6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23a40: skipped 000802 1788749391.772399 trunc from 0x023a40 to 0x027bbc (0x417d bytes) 000803 1788749391.787653 mapwrite 0x004028 thru 0x005ee9 (0x1ec2 bytes) 000804 1788749391.835911 trunc from 0x027bbd to 0x00bc53 (0x1bf69 bytes) 000805 1788749391.851119 write 0x026f70 thru 0x0366aa (0xf73b bytes) 000806 1788749391.852830 mapread 0x032e8b thru 0x0366aa (0x3820 bytes) 000807 1788749391.858412 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x366ab): Mirror 3: 0x833d804 (stale) Mirror 5: 0x833d804 (stale) Mirror 7: 0xe9a86296 (stale) Mirror 8: 0xe6880327 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x366ab): Mirror 3: 0x833d804 (stale) Mirror 5: 0x833d804 (stale) Mirror 7: 0xe9a86296 (stale) Mirror 8: 0xe6880327 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x366ab: skipped 000809 1788749392.152103 write 0x03511a thru 0x03ef17 (0x9dfe bytes) 000810 1788749392.161903 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ef18): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ef18): Mirror 3: 0xe615f494 (stale) Mirror 5: 0xe615f494 (stale) Mirror 7: 0xa23c7e95 (stale) Mirror 8: 0xe7ea4806 (stale) CRC-32 checksum value for chunk [0, 0x3ef18): Mirror 1: 0x61186e1a Mirror 9: 0x61186e1a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ef18: skipped 000812 1788749393.032923 write 0x01444d thru 0x01b167 (0x6d1b bytes) 000813 1788749393.126138 punch from 0x016583 to 0x020547 (0x9fc5 bytes) 000814 1788749393.141068 mapread 0x01d5c3 thru 0x025dcf (0x880d bytes) 000815 1788749393.184819 mapwrite 0x000177 thru 0x0041bc (0x4046 bytes) 000817 1788749393.258095 MIRROR_RESYNC 6 mirrors 000819 1788749394.955366 mapread 0x017664 thru 0x01e5c6 (0x6f63 bytes) 000822 1788749394.984345 write 0x00ec1c thru 0x01c004 (0xd3e9 bytes) 000823 1788749395.110763 MIRROR_RESYNC 6 mirrors 000824 1788749396.279682 trunc from 0x03ef18 to 0x023083 (0x1be94 bytes) 000825 1788749396.349827 read 0x01c8fa thru 0x023083 (0x678a bytes) 000826 1788749396.358250 write 0x010000 thru 0x013205 (0x3206 bytes) 000827 1788749396.363466 write 0x01653c thru 0x0256d6 (0xf19b bytes) 000828 1788749396.370653 punch from 0x001747 to 0x00dc41 (0xc4fb bytes) 000832 1788749396.383847 read 0x005c46 thru 0x00c8b4 (0x6c6f bytes) 000834 1788749396.394551 trunc from 0x0256d7 to 0x027896 (0x21c0 bytes) 000836 1788749396.417005 mapread 0x00f518 thru 0x018b42 (0x962b bytes) 000837 1788749396.418403 write 0x03c343 thru 0x03ffff (0x3cbd bytes) 000838 1788749396.419761 punch from 0x012ff8 to 0x01480c (0x1815 bytes) 000839 1788749396.430109 write 0x019dfa thru 0x01aba4 (0xdab bytes) 000840 1788749396.430810 mapwrite 0x003a62 thru 0x00b360 (0x78ff bytes) 000843 1788749396.477359 trunc from 0x040000 to 0x01cefd (0x23102 bytes) 000844 1788749396.497430 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1cefe): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cefe): Mirror 3: 0x3282bc58 (stale) Mirror 5: 0x3282bc58 (stale) Mirror 7: 0x3282bc58 (stale) Mirror 8: 0x3282bc58 (stale) Mirror 9: 0x3282bc58 (stale) CRC-32 checksum value for chunk [0, 0x1cefe): Mirror 1: 0xc8ee3ff6 Mirror 10: 0xc8ee3ff6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cefe: skipped 000846 1788749397.354141 punch from 0x006c1c to 0x014883 (0xdc68 bytes) 000848 1788749397.495905 mapwrite 0x03b94d thru 0x03f698 (0x3d4c bytes) 000849 1788749397.631640 mapread 0x025863 thru 0x0265ce (0xd6c bytes) 000850 1788749397.650009 mapwrite 0x022b6c thru 0x03290e (0xfda3 bytes) 000851 1788749397.703091 punch from 0x01d633 to 0x01f02f (0x19fd bytes) 000852 1788749397.714335 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f699): Mirror 5: 0x5ed12264 (stale) Mirror 7: 0x5ed12264 (stale) Mirror 8: 0x5ed12264 (stale) Mirror 9: 0x5ed12264 (stale) Mirror 10: 0x91cb6c6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f699): Mirror 5: 0x5ed12264 (stale) Mirror 7: 0x5ed12264 (stale) Mirror 8: 0x5ed12264 (stale) Mirror 9: 0x5ed12264 (stale) Mirror 10: 0x91cb6c6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f699: skipped 000854 1788749398.187379 write 0x03a9a6 thru 0x03ffff (0x565a bytes) 000855 1788749398.192244 trunc from 0x040000 to 0x015a23 (0x2a5dc bytes) 000856 1788749398.218423 MIRROR_RESYNC 6 mirrors 000857 1788749399.711972 read 0x012e00 thru 0x015a23 (0x2c24 bytes) 000858 1788749399.722337 write 0x03a441 thru 0x03ffff (0x5bbf bytes) 000859 1788749399.856374 write 0x02007d thru 0x026c26 (0x6baa bytes) 000860 1788749399.865380 write 0x00b992 thru 0x0102db (0x494a bytes) 000864 1788749399.881614 write 0x01d412 thru 0x02ca85 (0xf674 bytes) 000865 1788749399.894962 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xc5bebccc (stale) Mirror 7: 0xc5bebccc (stale) Mirror 8: 0xc5bebccc (stale) Mirror 9: 0xc5bebccc (stale) Mirror 10: 0xc5bebccc (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2bb61c15 Mirror 11: 0x2bb61c15 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000866 1788749401.176945 write 0x028c78 thru 0x02ed71 (0x60fa bytes) 000867 1788749401.286723 mapwrite 0x00a666 thru 0x00e18c (0x3b27 bytes) 000868 1788749401.407903 MIRROR_RESYNC 7 mirrors 000869 1788749403.177220 write 0x037ed9 thru 0x03ffff (0x8127 bytes) 000871 1788749403.271933 mapwrite 0x002e4f thru 0x00861e (0x57d0 bytes) 000872 1788749403.351967 mapwrite 0x02d945 thru 0x02fcd9 (0x2395 bytes) 000874 1788749403.422717 MIRROR_RESYNC 7 mirrors 000875 1788749404.388020 MIRROR_RESYNC 7 mirrors 000876 1788749404.784635 write 0x014bdb thru 0x01a556 (0x597c bytes) 000877 1788749404.849504 trunc from 0x040000 to 0x038c9e (0x7361 bytes) 000879 1788749404.869653 mapwrite 0x0040f5 thru 0x012987 (0xe893 bytes) 000880 1788749404.946528 read 0x0201b1 thru 0x026d4f (0x6b9f bytes) 000882 1788749404.961794 mapread 0x003a35 thru 0x012b53 (0xf11f bytes) 000883 1788749404.964112 write 0x031160 thru 0x03e19a (0xd03b bytes) 000884 1788749404.977468 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3e19b): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e19b): Mirror 5: 0xd41031d (stale) Mirror 7: 0xd41031d (stale) Mirror 8: 0xd41031d (stale) Mirror 9: 0xd41031d (stale) Mirror 10: 0xd41031d (stale) Mirror 11: 0xd41031d (stale) CRC-32 checksum value for chunk [0, 0x3e19b): Mirror 1: 0x844046c4 Mirror 12: 0x844046c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e19b: skipped 000887 1788749406.158666 punch from 0x021b44 to 0x0269fc (0x4eb9 bytes) 000888 1788749406.244803 read 0x0175b8 thru 0x026b00 (0xf549 bytes) 000889 1788749406.253776 read 0x01dd34 thru 0x0260a5 (0x8372 bytes) 000890 1788749406.254285 punch from 0x0064e1 to 0x0129b4 (0xc4d4 bytes) 000891 1788749406.258216 mapwrite 0x01c8c0 thru 0x02a2a2 (0xd9e3 bytes) 000892 1788749406.340770 trunc from 0x03e19b to 0x016f87 (0x27213 bytes) 000893 1788749406.380494 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16f88): Mirror 5: 0xe09d8d72 (stale) Mirror 7: 0xe09d8d72 (stale) Mirror 8: 0xe09d8d72 (stale) Mirror 10: 0xe09d8d72 (stale) Mirror 11: 0xe09d8d72 (stale) Mirror 12: 0xa53e741b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16f88): Mirror 5: 0xe09d8d72 (stale) Mirror 7: 0xe09d8d72 (stale) Mirror 8: 0xe09d8d72 (stale) Mirror 10: 0xe09d8d72 (stale) Mirror 11: 0xe09d8d72 (stale) Mirror 12: 0xa53e741b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16f88: skipped 000894 1788749406.769381 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16f88): Mirror 5: 0xe09d8d72 (stale) Mirror 7: 0xe09d8d72 (stale) Mirror 10: 0xe09d8d72 (stale) Mirror 11: 0xe09d8d72 (stale) Mirror 12: 0xa53e741b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16f88): Mirror 5: 0xe09d8d72 (stale) Mirror 7: 0xe09d8d72 (stale) Mirror 10: 0xe09d8d72 (stale) Mirror 11: 0xe09d8d72 (stale) Mirror 12: 0xa53e741b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16f88: skipped 000895 1788749407.092258 read 0x003a2a thru 0x00e3e9 (0xa9c0 bytes) 000899 1788749407.100750 write 0x03a5a6 thru 0x03ffff (0x5a5a bytes) 000900 1788749407.101843 mapwrite 0x01c3e1 thru 0x029d52 (0xd972 bytes) 000901 1788749407.184731 punch from 0x01b8ac to 0x026bb0 (0xb305 bytes) 000903 1788749407.189657 read 0x01b87f thru 0x01cec5 (0x1647 bytes) 000904 1788749407.198890 read 0x004504 thru 0x005cba (0x17b7 bytes) 000905 1788749407.199664 mapwrite 0x001e2a thru 0x01195e (0xfb35 bytes) 000906 1788749407.243287 trunc from 0x040000 to 0x018c53 (0x273ac bytes) 000907 1788749407.258617 punch from 0x0052f2 to 0x00d3a5 (0x80b4 bytes) 000909 1788749407.263762 trunc from 0x018c54 to 0x036124 (0x1d4d1 bytes) 000911 1788749407.276151 write 0x014ec6 thru 0x019799 (0x48d4 bytes) 000912 1788749407.291085 trunc from 0x036125 to 0x020b81 (0x155a3 bytes) 000913 1788749407.306906 punch from 0x01fc34 to 0x020b81 (0xf4e bytes) 000915 1788749407.318259 punch from 0x019abc to 0x020b81 (0x70c6 bytes) 000916 1788749407.322698 mapwrite 0x03e98c thru 0x03ffff (0x1674 bytes) 000917 1788749407.382533 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x1e4df838 (stale) Mirror 7: 0x1e4df838 (stale) Mirror 10: 0x1e4df838 (stale) Mirror 11: 0x1e4df838 (stale) Mirror 12: 0x1714ff57 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe5f27b76 Mirror 13: 0xe5f27b76 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000918 1788749408.190864 punch from 0x0027cf to 0x00c2d1 (0x9b03 bytes) 000919 1788749408.287220 MIRROR_RESYNC 7 mirrors 000920 1788749409.898057 read 0x025133 thru 0x0319ba (0xc888 bytes) 000921 1788749409.907215 write 0x00fde6 thru 0x01ddb7 (0xdfd2 bytes) 000922 1788749409.989071 write 0x03f546 thru 0x03ffff (0xaba bytes) 000923 1788749409.996543 mapread 0x020aeb thru 0x022893 (0x1da9 bytes) 000926 1788749410.010402 mapread 0x01e910 thru 0x027152 (0x8843 bytes) 000927 1788749410.011251 write 0x035ddd thru 0x03b2f3 (0x5517 bytes) 000929 1788749410.018077 read 0x036930 thru 0x03ffff (0x96d0 bytes) 000930 1788749410.025080 write 0x027414 thru 0x028d37 (0x1924 bytes) 000931 1788749410.025841 trunc from 0x040000 to 0x00dadc (0x32523 bytes) 000933 1788749410.045193 mapwrite 0x03402f thru 0x0378fd (0x38cf bytes) 000934 1788749410.125536 mapwrite 0x00766c thru 0x00a460 (0x2df5 bytes) 000936 1788749410.165610 mapread 0x018da7 thru 0x01c86e (0x3ac8 bytes) 000937 1788749410.191753 mapread 0x01c8c9 thru 0x02413a (0x7872 bytes) 000938 1788749410.225427 mapwrite 0x02e885 thru 0x03b883 (0xcfff bytes) 000939 1788749410.351294 punch from 0x02debd to 0x033a34 (0x5b78 bytes) 000942 1788749410.359784 mapwrite 0x036752 thru 0x03ffff (0x98ae bytes) 000943 1788749410.490516 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x202:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x3a5e82f6 (stale) Mirror 10: 0x3a5e82f6 (stale) Mirror 11: 0x3a5e82f6 (stale) Mirror 12: 0x3a5e82f6 (stale) Mirror 13: 0x3a5e82f6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x3a5e82f6 (stale) Mirror 10: 0x3a5e82f6 (stale) Mirror 11: 0x3a5e82f6 (stale) Mirror 12: 0x3a5e82f6 (stale) Mirror 13: 0x3a5e82f6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000944 1788749410.859166 read 0x01e122 thru 0x02c8ba (0xe799 bytes) 000945 1788749410.870850 read 0x00bc44 thru 0x013ea9 (0x8266 bytes) 000946 1788749410.886130 mapread 0x032f96 thru 0x033326 (0x391 bytes) 000947 1788749410.890414 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x3a5e82f6 (stale) Mirror 10: 0x3a5e82f6 (stale) Mirror 11: 0x3a5e82f6 (stale) Mirror 12: 0x3a5e82f6 (stale) Mirror 13: 0x3a5e82f6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x631a20d Mirror 14: 0x631a20d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000949 1788749411.936868 punch from 0x0134eb to 0x01f278 (0xbd8e bytes) 000950 1788749412.002812 punch from 0x00c060 to 0x018407 (0xc3a8 bytes) 000952 1788749412.008978 mapwrite 0x02ea09 thru 0x03ae7a (0xc472 bytes) 000953 1788749412.075191 read 0x03fc99 thru 0x03ffff (0x367 bytes) 000955 1788749412.076122 read 0x008eca thru 0x00f89a (0x69d1 bytes) 000956 1788749412.082140 punch from 0x021389 to 0x028b94 (0x780c bytes) 000957 1788749412.085361 trunc from 0x040000 to 0x00ceb3 (0x3314c bytes) 000958 1788749412.104239 trunc from 0x00ceb4 to 0x036e73 (0x29fc0 bytes) 000959 1788749412.116858 write 0x007de7 thru 0x013f36 (0xc150 bytes) 000960 1788749412.121837 punch from 0x01aa53 to 0x01fb11 (0x50bf bytes) 000961 1788749412.136969 read 0x00589a thru 0x007dec (0x2553 bytes) 000966 1788749412.139436 trunc from 0x036e74 to 0x039aac (0x2c39 bytes) 000968 1788749412.150874 read 0x000324 thru 0x00671b (0x63f8 bytes) 000972 1788749412.152751 mapread 0x001e61 thru 0x007748 (0x58e8 bytes) 000973 1788749412.153728 read 0x004866 thru 0x009de1 (0x557c bytes) 000975 1788749412.155078 punch from 0x025f09 to 0x02e407 (0x84ff bytes) 000976 1788749412.158460 write 0x01ae09 thru 0x01d117 (0x230f bytes) 000978 1788749412.168836 mapread 0x030f69 thru 0x037b2f (0x6bc7 bytes) 000980 1788749412.177188 write 0x01da74 thru 0x02a676 (0xcc03 bytes) 000981 1788749412.178675 trunc from 0x039aad to 0x03e515 (0x4a69 bytes) 000982 1788749412.201719 mapread 0x00f5c5 thru 0x0198a1 (0xa2dd bytes) 000984 1788749412.203177 mapwrite 0x0221f1 thru 0x02ea0f (0xc81f bytes) 000985 1788749412.235347 mapread 0x002add thru 0x010702 (0xdc26 bytes) 000987 1788749412.238557 punch from 0x01d91f to 0x02bef4 (0xe5d6 bytes) 000989 1788749412.245256 write 0x0233ef thru 0x030310 (0xcf22 bytes) 000990 1788749412.262142 mapread 0x003bc0 thru 0x00df3b (0xa37c bytes) 000991 1788749412.263181 mapread 0x02eefe thru 0x02f2a8 (0x3ab bytes) 000993 1788749412.269602 MIRROR_RESYNC 7 mirrors 000994 1788749413.603322 punch from 0x02e0ed to 0x033488 (0x539c bytes) 000997 1788749413.743690 read 0x0308bf thru 0x03e515 (0xdc57 bytes) 000998 1788749413.758976 mapread 0x032374 thru 0x032eff (0xb8c bytes) 000999 1788749413.760614 trunc from 0x03e516 to 0x030667 (0xdeae bytes) 001000 1788749413.787106 punch from 0x005458 to 0x00ab00 (0x56a9 bytes) All operations completed A-OK! PASS 70a (111s) == sanity-flr test 71: check mirror extend parent fid ==== 22:50:18 (1788749418) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 22:50:21 (1788749421) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x348:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x364:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.166879 s, 12.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x348:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x349:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x364:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116849 s, 9.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x348:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x349:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x365:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x364:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x348:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x349:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x365:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x364:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34a:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 22:50:28 (1788749428) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7159328 .. Extending file size to 8787168 .. Extending file size to 8732928 .. Extending file size to 8971136 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 57555 57556 57557 57558 57562 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) 192.168.202.140@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (78s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 22:51:46 (1788749506) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4706208 .. Extending file size to 4575008 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5876800 .. Extending file size to 6098496 .. Extending file size to 6452640 .. Extending file size to 6518592 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8411104 .. failed Extending file size to 7205120 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7270336 .. Extending file size to 8079360 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8482496 .. Extending file size to 8447136 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8677696 .. Waiting 59204 59205 59207 59208 59211 done 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) 192.168.202.140@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 22:53:07 (1788749587) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 15360 3753984 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 22528 3746816 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 37888 7500800 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.158258 s, 13.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.339551 s, 6.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x2b0f1276 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) PASS 200c (17s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 22:53:25 (1788749605) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0898749 s, 23.3 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 22:53:32 (1788749612) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x255:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x37b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x37b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x37c:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x35f:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 22:53:39 (1788749619) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.13126 s, 16.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x37d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x360:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x361:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 22:53:46 (1788749626) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.269758 s, 7.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x37e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x37f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x362:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x37e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x37f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x362:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x363:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 22:53:54 (1788749634) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.239205 s, 8.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x380:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x364:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x380:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x364:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x365:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 22:54:27 (1788749667) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg240-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151021 s, 6.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x381:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x367:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x366:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x381:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x367:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x366:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x382:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 22:55:26 (1788749726) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg240-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x383:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x384:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.276834 s, 7.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x383:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x384:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x368:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x383:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x384:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x368:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (59s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 22:56:25 (1788749785) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg240-server mds-ost sync done. Creating new pool test_204f oleg240-server: Pool lustre.test_204f created Adding targets to pool oleg240-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg240-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.340677 s, 6.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x385:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x369:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x385:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x369:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg240-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg240-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg240-server: Pool lustre.test_204f destroyed PASS 204f (76s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 22:57:41 (1788749861) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 22:57:49 (1788749869) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 22:57:57 (1788749877) oleg240-server: Pool lustre.test_206 created Adding targets to pool oleg240-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg240-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Updated after 2s: want 'lustre-OST0000_UUID lustre-OST0001_UUID ' got 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg240-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg240-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg240-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 22:58:20 (1788749900) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104459 s, 10.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471341 s, 22.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x83caed83 Mirror 3: 0x83caed83 PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 22:58:32 (1788749912) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 22:58:35 (1788749915) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 22:58:39 (1788749919) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.139031 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x22f0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1fb0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1fb0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1fb0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1c00000 Read this (4 bytes): BBBB 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 22:59:15 (1788749955) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0351551 s, 233 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 22:59:22 (1788749962) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0529424 s, 19.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x27a lmm_fid: [0x200000404:0x27a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x27a lmm_fid: [0x200000404:0x27a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x374:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834554 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-06 22:59:23.000000000 -0400 Modify: 2026-09-06 22:59:23.000000000 -0400 Change: 2026-09-06 22:59:25.000000000 -0400 Birth: 2026-09-06 22:59:23.000000000 -0400 PASS 210a (6s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 22:59:29 (1788749969) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 22:59:32 (1788749972) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.772365 s, 53.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0919838 s, 44.5 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 6 22:59 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x27c:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 6 22:59 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 22:59:42 (1788749982) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 22:59:51 (1788749991) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1645 sec =========== 23:00:01 (1788750001) === sanity-flr: start cleanup 23:00:02 (1788750002) === === sanity-flr: finish cleanup 23:00:08 (1788750008) ===