-----============= acceptance-small: sanity-flr ============----- Mon Mar 30 08:45:03 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg637-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 08:45:19 (1774874719) === oleg637-client.virtnet: executing check_config_client /mnt/lustre oleg637-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg637-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff998f8639c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff998f8639c000.idle_timeout=debug disable quota as required oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 08:45:43 (1774874743) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7554] == sanity-flr test 0a: lfs mirror create with -N option == 08:45:44 (1774874744) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:45:58 (1774874758) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:46:02 (1774874762) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:46:05 (1774874765) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:46:27 (1774874787) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:46:29 (1774874789) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 08:46:33 (1774874793) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff998f8639c000.stats=clear osc.lustre-OST0001-osc-ffff998f8639c000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:46:40 (1774874800) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:46:49 (1774874809) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:46:56 (1774874816) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 08:46:59 (1774874819) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x23:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 08:47:07 (1774874827) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:47:14 (1774874834) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:47:17 (1774874837) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (5s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:47:22 (1774874842) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 08:47:29 (1774874849) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:47:35 (1774874855) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 1.95891 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0864836 s, 12.1 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:47:45 (1774874865) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.165646 s, 25.3 MB/s PASS 22 (7s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:47:52 (1774874872) Stopping /mnt/lustre-ost1 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff998f8639c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff998f8639c000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg637-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff998f8639c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff998f8639c000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg637-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (47s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:48:39 (1774874919) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.178997 s, 29.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff998f8639c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff998f8639c000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg637-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (25s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:49:04 (1774874944) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff998f8639c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff998f8639c000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg637-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff998f8639c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff998f8639c000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg637-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (45s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:49:49 (1774874989) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.37816 s, 15.5 MB/s -rw-r--r-- 1 root root 52428800 Mar 30 08:49 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.981s user 0m0.001s sys 0m0.190s Stopping /mnt/lustre-ost1 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff998f8639c000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff998f8639c000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.977s user 0m0.000s sys 0m0.194s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg637-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff998f8639c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff998f8639c000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost1)... real 0m0.829s user 0m0.002s sys 0m0.204s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg637-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg637-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff998f8639c000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff998f8639c000.ost_server_uuid in FULL state after 0 sec PASS 33b (58s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:50:47 (1774875047) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:50:50 (1774875050) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:50:52 (1774875052) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (2s) == sanity-flr test 35: allow to write to mirrored files == 08:50:54 (1774875054) PASS 35 (5s) == sanity-flr test 36a: write to mirrored files ========== 08:50:59 (1774875059) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.369559 s, 34.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (9s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:51:08 (1774875068) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412169 s, 25.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0340093 s, 30.8 MB/s PASS 36b (5s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:51:13 (1774875073) PASS 36c (6s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:51:19 (1774875079) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000878568 s, 1.2 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00162217 s, 631 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000830815 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00149503 s, 685 kB/s flv=10 olv=10 PASS 36d (8s) == sanity-flr test 37: mirror I/O API verification ======= 08:51:27 (1774875087) create test files with size 15747961 .. 0dbcdbdf341eaa924cd481956f5e9fd6 - 9a6411ba14278eadba2b67e1808a3438 - a32d70698a3ede5b879d0b1a265ae5eb - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg637-client.virtnet /mnt/lustre (opts:) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre PASS 37 (18s) == sanity-flr test 38: resync ============================ 08:51:45 (1774875105) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:51:57 (1774875117) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:52:04 (1774875124) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144325 s, 7.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4d:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4e:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4a:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322186 s, 32.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4b:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0x4c:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0x4f:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (7s) == sanity-flr test 41: lfs mirror resync check =========== 08:52:12 (1774875132) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.178405 s, 11.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.299806 s, 14.0 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170014 s, 12.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 08:52:25 (1774875145) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 08:52:28 (1774875148) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (2s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:52:30 (1774875150) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200587 s, 52.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 08:52:36 (1774875156) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:52:39 (1774875159) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x56:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:52:47 (1774875167) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.216036 s, 48.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg637-client.virtnet /mnt/lustre (opts:) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:52:57 (1774875177) llite.lustre-ffff998f992cc800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.202096 s, 51.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg637-client.virtnet /mnt/lustre (opts:) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff998f99386800.xattr_cache=1 PASS 44e (10s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:53:07 (1774875187) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (30s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:53:37 (1774875217) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (29s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:54:06 (1774875246) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 08:54:09 (1774875249) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.230319 s, 13.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.196099 s, 16.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xfaa412c3 (stale) Mirror 2: 0xfaa412c3 (stale) lfs: mirrors are inconsistent but no stale components marked in '/mnt/lustre/f48.sanity-flr'. Use 'lfs mirror resync --only N --force-resync' to force resync specific mirrors. ** verify mirror 2 doesn't change original checksum: f590748c7d79e4b9ce6f34deb54af359 - mirror 1 checksum: 2821af829956b03c84f749427762677a - mirror 2 checksum: f590748c7d79e4b9ce6f34deb54af359 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 30 08:54:12 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Mon Mar 30 08:54:12 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: f590748c7d79e4b9ce6f34deb54af359 - mirror 1 checksum: 2821af829956b03c84f749427762677a - mirror 2 checksum: 2821af829956b03c84f749427762677a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 30 08:54:12 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x69:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Mon Mar 30 08:54:12 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6a:0x0] } PASS 48 (9s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:54:19 (1774875259) Failing mds1 on oleg637-server Stopping /mnt/lustre-mds1 (opts:) on oleg637-server 08:54:25 (1774875265) shut down facet: mds1 facet_host: oleg637-server facet_failover_host: oleg637-server Failover mds1 to oleg637-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-MDT0000 08:54:43 (1774875283) targets are mounted 08:54:43 (1774875283) facet_failover done oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0194577 s, 53.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x6b:0x0] } PASS 50A (35s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:54:54 (1774875294) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:54:57 (1774875297) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (2s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:54:59 (1774875299) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0112925 s, 1.5 MB/s PASS 50c (5s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:55:04 (1774875304) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (2s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:55:07 (1774875307) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:55:09 (1774875309) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (19s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:55:28 (1774875328) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (25s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:55:53 (1774875353) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:56:15 (1774875375) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (22s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:56:38 (1774875398) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 08:56:44 (1774875404) mirror create pid 52264 mirror split pid 52265 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52264 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4991: 52265 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 08:57:50 (1774875470) Seed set to 5471 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001 1774875471.943930 write 0x0060f8 thru 0x01441d (0xe326 bytes) 000002 1774875471.959626 read 0x0092e9 thru 0x00f22e (0x5f46 bytes) truncating to largest ever: 0x118d0 000008 1774875471.968805 trunc from 0x01441e to 0x0118cf (0x2b4e bytes) 000012 1774875471.998719 read 0x00cbde thru 0x00f6bc (0x2adf bytes) 000013 1774875471.999670 read 0x007ab2 thru 0x00e9ce (0x6f1d bytes) 000019 1774875472.003190 write 0x030d6a thru 0x03ffff (0xf296 bytes) 000020 1774875472.006053 read 0x017b8e thru 0x02098e (0x8e01 bytes) 000022 1774875472.028945 write 0x03f4c0 thru 0x03ffff (0xb40 bytes) 000025 1774875472.051114 MIRROR_EXTEND to 2 mirrors 000026 1774875472.767018 write 0x026963 thru 0x028a0a (0x20a8 bytes) 000027 1774875472.884932 write 0x01bb91 thru 0x02b8ef (0xfd5f bytes) 000029 1774875472.926429 read 0x00f959 thru 0x016334 (0x69dc bytes) 000030 1774875472.953044 mapread 0x02f367 thru 0x036c06 (0x78a0 bytes) 000031 1774875472.982630 mapwrite 0x0197a6 thru 0x024f8f (0xb7ea bytes) 000032 1774875473.066138 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) 000033 1774875473.268490 write 0x025029 thru 0x033d14 (0xecec bytes) 000034 1774875473.284153 mapread 0x022405 thru 0x031ae7 (0xf6e3 bytes) 000036 1774875473.310983 MIRROR_EXTEND to 2 mirrors 000037 1774875474.001368 trunc from 0x040000 to 0x0098f2 (0x3670d bytes) 000038 1774875474.099178 mapwrite 0x0136b2 thru 0x022526 (0xee75 bytes) 000039 1774875474.215440 mapread 0x0201f1 thru 0x022526 (0x2336 bytes) 000040 1774875474.218495 mapwrite 0x00134f thru 0x00a284 (0x8f36 bytes) 000041 1774875474.298061 write 0x0039bb thru 0x005985 (0x1fcb bytes) 000043 1774875474.299721 write 0x029fb9 thru 0x02bbd7 (0x1c1f bytes) 000044 1774875474.301894 mapread 0x009b35 thru 0x017c41 (0xe10d bytes) 000045 1774875474.305481 write 0x00230c thru 0x00543b (0x3130 bytes) 000046 1774875474.308095 read 0x020d5e thru 0x020ec1 (0x164 bytes) 000047 1774875474.315287 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2bbd8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bbd8): Mirror 2: 0xc80e81a6 (stale) CRC-32 checksum value for chunk [0, 0x2bbd8): Mirror 1: 0xcb6d14ff Mirror 3: 0xcb6d14ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bbd8: skipped 000048 1774875475.028008 mapwrite 0x033185 thru 0x03ffff (0xce7b bytes) 000051 1774875475.242431 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8e411e7e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x8e411e7e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000054 1774875475.588500 mapwrite 0x01e97b thru 0x02cca1 (0xe327 bytes) 000055 1774875475.690851 read 0x00d881 thru 0x00eacb (0x124b bytes) truncating to largest ever: 0x2d169 000057 1774875475.698453 trunc from 0x040000 to 0x02d168 (0x12e97 bytes) 000059 1774875475.724707 read 0x016d8f thru 0x023de8 (0xd05a bytes) 000062 1774875475.747549 mapread 0x011dd2 thru 0x018329 (0x6558 bytes) 000063 1774875475.754730 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) 000064 1774875475.920114 mapwrite 0x009905 thru 0x01363a (0x9d36 bytes) 000069 1774875476.033114 read 0x0011f2 thru 0x00fbf3 (0xea02 bytes) 000070 1774875476.041039 mapwrite 0x03b15e thru 0x03ffff (0x4ea2 bytes) 000071 1774875476.151403 trunc from 0x040000 to 0x00289f (0x3d760 bytes) 000072 1774875476.176301 MIRROR_EXTEND to 2 mirrors 000073 1774875476.736660 read 0x00161f thru 0x00289f (0x1281 bytes) 000075 1774875476.756432 write 0x023e6b thru 0x027cc5 (0x3e5b bytes) 000076 1774875476.818981 mapread 0x01b178 thru 0x01fab1 (0x493a bytes) 000077 1774875476.840426 read 0x024275 thru 0x027cc5 (0x3a51 bytes) 000078 1774875476.841572 write 0x02c4f1 thru 0x037bdb (0xb6eb bytes) 000079 1774875476.843117 read 0x034dfb thru 0x037bdb (0x2de1 bytes) 000081 1774875476.844720 read 0x033425 thru 0x037bdb (0x47b7 bytes) 000082 1774875476.851426 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37bdc): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37bdc): Mirror 2: 0x19438d74 (stale) CRC-32 checksum value for chunk [0, 0x37bdc): Mirror 1: 0xc7cbf934 Mirror 3: 0xc7cbf934 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37bdc: skipped 000083 1774875477.719202 mapread 0x02a958 thru 0x02c25a (0x1903 bytes) 000085 1774875477.741625 trunc from 0x037bdc to 0x0017e3 (0x363f8 bytes) 000087 1774875477.841633 write 0x030513 thru 0x037ea9 (0x7997 bytes) 000088 1774875477.844528 mapwrite 0x00cc25 thru 0x0104f1 (0x38cd bytes) 000089 1774875477.919660 mapread 0x01c42a thru 0x026dc7 (0xa99e bytes) truncating to largest ever: 0x359db 000090 1774875477.940532 trunc from 0x037eaa to 0x0359da (0x24cf bytes) 000094 1774875477.968647 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x359db): Mirror 3: 0x4ebdb86a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x359db): Mirror 3: 0x4ebdb86a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x359db: skipped 000095 1774875478.219656 mapwrite 0x036371 thru 0x03fa59 (0x96e9 bytes) 000096 1774875478.311333 write 0x0107f0 thru 0x01c18d (0xb99e bytes) 000097 1774875478.347655 mapread 0x01a7d1 thru 0x01e968 (0x4198 bytes) 000099 1774875478.369374 read 0x02643e thru 0x02f512 (0x90d5 bytes) 000103 1774875478.394032 trunc from 0x03fa5a to 0x010beb (0x2ee6e bytes) truncating to largest ever: 0x3c68e 000105 1774875478.418708 trunc from 0x010bec to 0x03c68d (0x2baa2 bytes) 000107 1774875478.458506 mapwrite 0x016b88 thru 0x019b4a (0x2fc3 bytes) 000108 1774875478.524803 write 0x018de1 thru 0x01be6a (0x308a bytes) 000109 1774875478.525868 trunc from 0x03c68e to 0x036f0e (0x577f bytes) 000110 1774875478.551029 mapwrite 0x01a0b8 thru 0x0230b0 (0x8ff9 bytes) 000111 1774875478.617646 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36f0f): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36f0f): Mirror 3: 0x13653b06 (stale) CRC-32 checksum value for chunk [0, 0x36f0f): Mirror 1: 0xb2bea014 Mirror 4: 0xb2bea014 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36f0f: skipped 000112 1774875479.520465 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36f0f): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36f0f): Mirror 3: 0x13653b06 (stale) CRC-32 checksum value for chunk [0, 0x36f0f): Mirror 1: 0xb2bea014 Mirror 4: 0xb2bea014 Mirror 5: 0xb2bea014 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36f0f: skipped 000117 1774875480.482924 read 0x02d566 thru 0x036011 (0x8aac bytes) 000118 1774875480.503741 read 0x00043b thru 0x00ce54 (0xca1a bytes) 000121 1774875480.570747 trunc from 0x036f0f to 0x01e149 (0x18dc5 bytes) 000122 1774875480.730331 trunc from 0x01e14a to 0x01f7ec (0x16a3 bytes) 000124 1774875480.761833 trunc from 0x01f7ed to 0x00ac74 (0x14b78 bytes) 000128 1774875480.779368 write 0x03ae36 thru 0x03ba65 (0xc30 bytes) 000129 1774875480.781716 write 0x006ec8 thru 0x01049e (0x95d7 bytes) truncating to largest ever: 0x3f5d6 000131 1774875480.794545 trunc from 0x03ba66 to 0x03f5d5 (0x3b70 bytes) 000133 1774875480.811590 write 0x028081 thru 0x028d7c (0xcfc bytes) 000135 1774875480.828057 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f5d6): Mirror 3: 0x5e7c551c (stale) Mirror 5: 0xa81e8a9e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f5d6): Mirror 3: 0x5e7c551c (stale) Mirror 5: 0xa81e8a9e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f5d6: skipped 000136 1774875481.134384 MIRROR_RESYNC 3 mirrors 000137 1774875482.072040 MIRROR_RESYNC 3 mirrors 000139 1774875482.475809 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) 000142 1774875482.729942 read 0x02b0f7 thru 0x038a7b (0xd985 bytes) 000143 1774875482.740118 mapwrite 0x01d798 thru 0x02882a (0xb093 bytes) 000145 1774875482.950844 mapwrite 0x0072af thru 0x0157fa (0xe54c bytes) 000146 1774875483.007278 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) 000147 1774875483.165116 mapwrite 0x005d2c thru 0x008859 (0x2b2e bytes) 000148 1774875483.251019 mapread 0x02231b thru 0x02eadb (0xc7c1 bytes) 000149 1774875483.272073 trunc from 0x03f5d6 to 0x03a18a (0x544b bytes) 000150 1774875483.297713 write 0x03ec19 thru 0x03ffff (0x13e7 bytes) 000151 1774875483.298942 mapread 0x02875f thru 0x030747 (0x7fe9 bytes) 000152 1774875483.305562 trunc from 0x040000 to 0x03ab8d (0x5472 bytes) 000154 1774875483.326004 mapwrite 0x032740 thru 0x03ee77 (0xc738 bytes) 000155 1774875483.397768 mapread 0x001e4f thru 0x009ff9 (0x81ab bytes) 000156 1774875483.402195 MIRROR_EXTEND to 2 mirrors 000158 1774875483.915348 trunc from 0x03ee78 to 0x01e8a5 (0x205d2 bytes) 000159 1774875484.011120 mapread 0x00b088 thru 0x0148d3 (0x984c bytes) 000163 1774875484.039376 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) 000164 1774875484.191760 mapread 0x00736d thru 0x010409 (0x909d bytes) 000168 1774875484.218961 write 0x015a6c thru 0x020694 (0xac29 bytes) 000169 1774875484.231631 mapread 0x005aa9 thru 0x014907 (0xee5f bytes) 000170 1774875484.238454 mapwrite 0x023c67 thru 0x02f138 (0xb4d2 bytes) 000171 1774875484.329163 read 0x001db6 thru 0x002c8d (0xed8 bytes) 000173 1774875484.335234 mapread 0x0117f1 thru 0x01c69a (0xaeaa bytes) 000174 1774875484.336363 mapwrite 0x004d69 thru 0x005c61 (0xef9 bytes) 000176 1774875484.375858 mapread 0x028bef thru 0x02f138 (0x654a bytes) 000178 1774875484.378095 write 0x0156e5 thru 0x017634 (0x1f50 bytes) 000179 1774875484.392009 MIRROR_EXTEND to 2 mirrors 000180 1774875485.004656 write 0x0090a9 thru 0x016fe6 (0xdf3e bytes) 000181 1774875485.104051 write 0x031def thru 0x03bc20 (0x9e32 bytes) 000182 1774875485.105466 read 0x011b25 thru 0x01ed19 (0xd1f5 bytes) 000184 1774875485.112848 write 0x006245 thru 0x00bd55 (0x5b11 bytes) 000185 1774875485.120556 mapread 0x033deb thru 0x03bc20 (0x7e36 bytes) 000186 1774875485.123133 mapwrite 0x00ae40 thru 0x01951d (0xe6de bytes) 000187 1774875485.165647 mapread 0x0334fd thru 0x03bc20 (0x8724 bytes) 000188 1774875485.167023 trunc from 0x03bc21 to 0x002ab2 (0x3916e bytes) 000189 1774875485.201920 read 0x001f29 thru 0x002ab2 (0xb8a bytes) 000190 1774875485.202694 mapread 0x00248f thru 0x002ab2 (0x624 bytes) 000191 1774875485.203748 read 0x001459 thru 0x002ab2 (0x165a bytes) 000192 1774875485.204353 read 0x0008aa thru 0x002ab2 (0x2209 bytes) 000196 1774875485.211008 read 0x000b72 thru 0x002ab2 (0x1f41 bytes) 000197 1774875485.220629 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) 000202 1774875485.384961 write 0x02770e thru 0x02f171 (0x7a64 bytes) 000203 1774875485.386978 mapread 0x021708 thru 0x02b829 (0xa122 bytes) 000204 1774875485.399777 trunc from 0x02f172 to 0x029a85 (0x56ec bytes) 000206 1774875485.414668 write 0x0072a5 thru 0x008f2f (0x1c8b bytes) 000207 1774875485.431873 write 0x019c97 thru 0x0262c6 (0xc630 bytes) 000209 1774875485.441485 write 0x0166c7 thru 0x01e84a (0x8184 bytes) 000210 1774875485.442688 trunc from 0x029a86 to 0x03e83d (0x14db8 bytes) 000211 1774875485.466554 mapread 0x01fb88 thru 0x022dfe (0x3277 bytes) 000214 1774875485.470991 mapwrite 0x01983b thru 0x026fcd (0xd793 bytes) 000215 1774875485.520716 MIRROR_EXTEND to 2 mirrors 000216 1774875485.971161 read 0x023998 thru 0x025371 (0x19da bytes) 000218 1774875485.999935 mapread 0x008d8b thru 0x01883d (0xfab3 bytes) 000220 1774875486.050059 mapwrite 0x01db85 thru 0x024235 (0x66b1 bytes) 000221 1774875486.253529 trunc from 0x03e83e to 0x009d14 (0x34b29 bytes) 000223 1774875486.295582 trunc from 0x009d15 to 0x0390a4 (0x2f390 bytes) 000227 1774875486.315123 read 0x02a0fa thru 0x02a5b8 (0x4bf bytes) 000229 1774875486.361855 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x390a5): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x390a5): Mirror 2: 0x3014c070 (stale) CRC-32 checksum value for chunk [0, 0x390a5): Mirror 1: 0x1e70730b Mirror 3: 0x1e70730b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x390a5: skipped 000230 1774875487.109064 write 0x03d1a1 thru 0x03ffff (0x2e5f bytes) 000232 1774875487.182068 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x217c981a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x217c981a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000233 1774875487.531663 mapwrite 0x03a556 thru 0x03ffff (0x5aaa bytes) 000234 1774875487.612050 mapwrite 0x02966a thru 0x02ff9e (0x6935 bytes) 000235 1774875487.684451 write 0x033ad1 thru 0x03ffff (0xc52f bytes) 000236 1774875487.695026 mapread 0x03e1a6 thru 0x03e724 (0x57f bytes) 000237 1774875487.696105 mapwrite 0x00ed7f thru 0x015086 (0x6308 bytes) 000238 1774875487.759103 mapread 0x01cc9e thru 0x021be2 (0x4f45 bytes) 000241 1774875487.775666 trunc from 0x040000 to 0x01d3d7 (0x22c28 bytes) 000244 1774875487.845099 read 0x00c307 thru 0x010d3b (0x4a35 bytes) 000246 1774875487.847528 read 0x01627e thru 0x01d3d7 (0x715a bytes) 000250 1774875487.851063 mapread 0x010dd9 thru 0x01393a (0x2b62 bytes) 000251 1774875487.859610 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1d3d8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d3d8): Mirror 2: 0xb041c426 (stale) CRC-32 checksum value for chunk [0, 0x1d3d8): Mirror 1: 0xab3c49f5 Mirror 3: 0xab3c49f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d3d8: skipped 000252 1774875488.719391 write 0x031576 thru 0x03318a (0x1c15 bytes) 000256 1774875488.788944 trunc from 0x03318b to 0x018d9c (0x1a3ee bytes) 000258 1774875488.807558 write 0x004869 thru 0x00eb33 (0xa2cb bytes) 000259 1774875488.816531 mapwrite 0x03f9c4 thru 0x03ffff (0x63c bytes) 000260 1774875488.885974 write 0x035ba7 thru 0x03f1e2 (0x963c bytes) 000263 1774875488.903955 trunc from 0x040000 to 0x02fd48 (0x102b7 bytes) 000266 1774875488.934381 MIRROR_RESYNC 3 mirrors 000267 1774875489.689046 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) 000268 1774875489.991615 MIRROR_EXTEND to 3 mirrors 000269 1774875490.804793 trunc from 0x02fd49 to 0x03dbfb (0xdeb3 bytes) 000270 1774875490.905708 trunc from 0x03dbfc to 0x032607 (0xb5f4 bytes) 000274 1774875490.958304 mapwrite 0x03c2e1 thru 0x03ffff (0x3d1f bytes) 000277 1774875491.084031 write 0x00a272 thru 0x00e579 (0x4308 bytes) 000279 1774875491.096005 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xf87a80e9 (stale) Mirror 3: 0xf87a80e9 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd47af321 Mirror 4: 0xd47af321 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000280 1774875491.957383 write 0x014e99 thru 0x0186a9 (0x3811 bytes) 000283 1774875492.067947 write 0x031ec9 thru 0x03801a (0x6152 bytes) 000284 1774875492.078076 mapread 0x026083 thru 0x02cf3a (0x6eb8 bytes) 000285 1774875492.090027 write 0x036dda thru 0x03ffff (0x9226 bytes) 000287 1774875492.092017 trunc from 0x040000 to 0x02bdf7 (0x14208 bytes) 000288 1774875492.121788 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2bdf8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bdf8): Mirror 2: 0x646a5006 (stale) Mirror 3: 0x646a5006 (stale) Mirror 4: 0xe3698338 (stale) CRC-32 checksum value for chunk [0, 0x2bdf8): Mirror 1: 0xd5a8d39b Mirror 5: 0xd5a8d39b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bdf8: skipped 000289 1774875492.973763 write 0x018002 thru 0x01cc7b (0x4c7a bytes) 000291 1774875493.113709 mapread 0x003ff2 thru 0x005d10 (0x1d1f bytes) 000295 1774875493.130951 write 0x0309d5 thru 0x036a32 (0x605e bytes) 000297 1774875493.133408 mapwrite 0x03a995 thru 0x03ffff (0x566b bytes) 000299 1774875493.236941 mapread 0x037826 thru 0x03dbed (0x63c8 bytes) 000300 1774875493.238299 write 0x00b324 thru 0x016458 (0xb135 bytes) 000302 1774875493.251046 mapwrite 0x018864 thru 0x027a54 (0xf1f1 bytes) 000303 1774875493.345613 read 0x017cd4 thru 0x01cf78 (0x52a5 bytes) 000304 1774875493.346613 read 0x037534 thru 0x03ffff (0x8acc bytes) 000308 1774875493.356498 mapwrite 0x02abf8 thru 0x02fd05 (0x510e bytes) 000309 1774875493.451320 trunc from 0x040000 to 0x0351d0 (0xae2f bytes) 000310 1774875493.470300 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x351d1): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x351d1): Mirror 2: 0xb5abaa6d (stale) Mirror 3: 0xb5abaa6d (stale) Mirror 4: 0x6b9be13 (stale) Mirror 5: 0xd03e7948 (stale) CRC-32 checksum value for chunk [0, 0x351d1): Mirror 1: 0xf72fdca1 Mirror 6: 0xf72fdca1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x351d1: skipped 000311 1774875494.122495 write 0x00671b thru 0x0115e2 (0xaec8 bytes) 000312 1774875494.188022 trunc from 0x0351d1 to 0x00cbfb (0x285d5 bytes) 000314 1774875494.211747 mapwrite 0x002523 thru 0x004fa7 (0x2a85 bytes) 000315 1774875494.264588 mapread 0x001ca5 thru 0x00cbfb (0xaf57 bytes) 000319 1774875494.269466 trunc from 0x00cbfc to 0x0106e7 (0x3aec bytes) 000321 1774875494.300674 mapread 0x001856 thru 0x004e53 (0x35fe bytes) 000324 1774875494.309384 write 0x01c3f9 thru 0x02a115 (0xdd1d bytes) 000325 1774875494.314828 write 0x025054 thru 0x032f31 (0xdede bytes) 000326 1774875494.324875 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x32f32): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32f32): Mirror 2: 0xe06b4440 (stale) Mirror 3: 0xe06b4440 (stale) Mirror 4: 0xd05dc5c (stale) Mirror 5: 0x923f59f5 (stale) Mirror 6: 0xcdec6e1a (stale) CRC-32 checksum value for chunk [0, 0x32f32): Mirror 1: 0x2869751b Mirror 7: 0x2869751b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32f32: skipped 000333 1774875495.076596 MIRROR_RESYNC 7 mirrors 000334 1774875496.045911 mapread 0x032ecb thru 0x032f31 (0x67 bytes) 000335 1774875496.054814 read 0x00d749 thru 0x015837 (0x80ef bytes) 000336 1774875496.071149 read 0x022466 thru 0x02ea45 (0xc5e0 bytes) 000337 1774875496.092795 mapwrite 0x020bc4 thru 0x0289c0 (0x7dfd bytes) 000339 1774875496.311800 write 0x0074f8 thru 0x00acba (0x37c3 bytes) 000341 1774875496.336080 mapread 0x01ca76 thru 0x01ee9c (0x2427 bytes) 000342 1774875496.360686 mapread 0x009824 thru 0x017c4a (0xe427 bytes) 000345 1774875496.452527 MIRROR_RESYNC 7 mirrors 000347 1774875497.436589 mapwrite 0x013981 thru 0x016e27 (0x34a7 bytes) 000349 1774875497.634986 mapread 0x03212b thru 0x032f31 (0xe07 bytes) 000352 1774875497.656147 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x32f32): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32f32): Mirror 2: 0x138a85e1 (stale) Mirror 3: 0x138a85e1 (stale) Mirror 4: 0x138a85e1 (stale) Mirror 5: 0x138a85e1 (stale) Mirror 6: 0x138a85e1 (stale) Mirror 7: 0x138a85e1 (stale) CRC-32 checksum value for chunk [0, 0x32f32): Mirror 1: 0x356974fd Mirror 8: 0x356974fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32f32: skipped 000354 1774875498.357162 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x32f32): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32f32): Mirror 2: 0x138a85e1 (stale) Mirror 3: 0x138a85e1 (stale) Mirror 4: 0x138a85e1 (stale) Mirror 5: 0x138a85e1 (stale) Mirror 6: 0x138a85e1 (stale) Mirror 7: 0x138a85e1 (stale) CRC-32 checksum value for chunk [0, 0x32f32): Mirror 1: 0x356974fd Mirror 8: 0x356974fd Mirror 9: 0x356974fd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32f32: skipped 000356 1774875499.306894 MIRROR_RESYNC 9 mirrors 000357 1774875500.491976 mapread 0x0251b1 thru 0x030e04 (0xbc54 bytes) 000358 1774875500.544675 write 0x0221c6 thru 0x03172b (0xf566 bytes) 000360 1774875500.695498 write 0x004d66 thru 0x00cf24 (0x81bf bytes) 000361 1774875500.711909 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32f32): Mirror 3: 0x356974fd (stale) Mirror 4: 0x356974fd (stale) Mirror 5: 0x356974fd (stale) Mirror 6: 0x356974fd (stale) Mirror 7: 0x356974fd (stale) Mirror 8: 0x356974fd (stale) Mirror 9: 0x356974fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32f32): Mirror 3: 0x356974fd (stale) Mirror 4: 0x356974fd (stale) Mirror 5: 0x356974fd (stale) Mirror 6: 0x356974fd (stale) Mirror 7: 0x356974fd (stale) Mirror 8: 0x356974fd (stale) Mirror 9: 0x356974fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32f32: skipped 000362 1774875501.393678 trunc from 0x032f32 to 0x021d36 (0x111fb bytes) 000367 1774875501.471652 write 0x016468 thru 0x01bd17 (0x58b0 bytes) 000368 1774875501.489465 trunc from 0x021d37 to 0x03ed1c (0x1cfe6 bytes) 000369 1774875501.525715 mapwrite 0x014465 thru 0x022373 (0xdf0f bytes) 000371 1774875501.618782 read 0x0318d2 thru 0x03ed1c (0xd44b bytes) 000373 1774875501.640481 write 0x035d42 thru 0x03dc01 (0x7ec0 bytes) 000374 1774875501.644905 trunc from 0x03ed1d to 0x03c4c3 (0x2859 bytes) 000377 1774875501.663677 read 0x002731 thru 0x00c1f1 (0x9ac1 bytes) 000378 1774875501.677128 mapwrite 0x018673 thru 0x01be6b (0x37f9 bytes) 000379 1774875501.726103 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c4c4): Mirror 3: 0x48f26c85 (stale) Mirror 4: 0x48f26c85 (stale) Mirror 5: 0x48f26c85 (stale) Mirror 6: 0x48f26c85 (stale) Mirror 8: 0x48f26c85 (stale) Mirror 9: 0x48f26c85 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c4c4): Mirror 3: 0x48f26c85 (stale) Mirror 4: 0x48f26c85 (stale) Mirror 5: 0x48f26c85 (stale) Mirror 6: 0x48f26c85 (stale) Mirror 8: 0x48f26c85 (stale) Mirror 9: 0x48f26c85 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c4c4: skipped 000380 1774875502.129886 write 0x025c1e thru 0x029c32 (0x4015 bytes) 000382 1774875502.146294 write 0x012634 thru 0x0177d7 (0x51a4 bytes) 000384 1774875502.163025 read 0x01277b thru 0x01c801 (0xa087 bytes) 000385 1774875502.173472 mapread 0x004d5d thru 0x00bd72 (0x7016 bytes) 000387 1774875502.187646 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3c4c4): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c4c4): Mirror 3: 0x48f26c85 (stale) Mirror 4: 0x48f26c85 (stale) Mirror 5: 0x48f26c85 (stale) Mirror 6: 0x48f26c85 (stale) Mirror 8: 0x48f26c85 (stale) Mirror 9: 0x48f26c85 (stale) CRC-32 checksum value for chunk [0, 0x3c4c4): Mirror 1: 0xb4ea8c40 Mirror 10: 0xb4ea8c40 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c4c4: skipped 000388 1774875503.249570 trunc from 0x03c4c4 to 0x015690 (0x26e33 bytes) 000389 1774875503.420140 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x15691): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15691): Mirror 3: 0x51563eea (stale) Mirror 4: 0x51563eea (stale) Mirror 5: 0x51563eea (stale) Mirror 6: 0x51563eea (stale) Mirror 8: 0x51563eea (stale) Mirror 9: 0x51563eea (stale) Mirror 10: 0x7b4fbe33 (stale) CRC-32 checksum value for chunk [0, 0x15691): Mirror 1: 0x7b4fbe33 Mirror 11: 0x7b4fbe33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15691: skipped 000390 1774875504.512483 trunc from 0x015691 to 0x025984 (0x102f4 bytes) 000393 1774875504.667718 mapread 0x001cb9 thru 0x00d448 (0xb790 bytes) 000394 1774875504.706012 mapread 0x00aaee thru 0x00eb48 (0x405b bytes) 000395 1774875504.707190 trunc from 0x025985 to 0x01d54b (0x8439 bytes) 000397 1774875504.754118 mapwrite 0x02e10a thru 0x02f14e (0x1045 bytes) 000398 1774875504.848483 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2f14f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f14f): Mirror 3: 0xee7aa058 (stale) Mirror 4: 0xee7aa058 (stale) Mirror 5: 0xee7aa058 (stale) Mirror 6: 0xee7aa058 (stale) Mirror 8: 0xee7aa058 (stale) Mirror 9: 0xee7aa058 (stale) Mirror 10: 0x34e998f3 (stale) Mirror 11: 0x4feacd8c (stale) CRC-32 checksum value for chunk [0, 0x2f14f): Mirror 1: 0xa797c721 Mirror 12: 0xa797c721 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f14f: skipped 000399 1774875505.953759 MIRROR_RESYNC 10 mirrors 000401 1774875507.696046 write 0x0237aa thru 0x02c504 (0x8d5b bytes) 000402 1774875507.780656 write 0x032fc1 thru 0x03bc47 (0x8c87 bytes) 000403 1774875507.783642 mapwrite 0x0059e6 thru 0x0086e1 (0x2cfc bytes) 000406 1774875507.868010 read 0x00519f thru 0x013208 (0xe06a bytes) 000408 1774875507.882058 mapwrite 0x0149cb thru 0x01a9b3 (0x5fe9 bytes) 000409 1774875507.974805 read 0x014bab thru 0x021082 (0xc4d8 bytes) 000411 1774875507.983406 read 0x00616d thru 0x00e092 (0x7f26 bytes) 000413 1774875507.984714 mapwrite 0x02f540 thru 0x03dde4 (0xe8a5 bytes) 000415 1774875508.105002 mapwrite 0x0216b9 thru 0x024cf1 (0x3639 bytes) 000416 1774875508.201334 mapwrite 0x00e33a thru 0x01226d (0x3f34 bytes) 000417 1774875508.276390 read 0x038a6f thru 0x03d78c (0x4d1e bytes) 000418 1774875508.277074 write 0x03a5ba thru 0x03d451 (0x2e98 bytes) 000420 1774875508.280914 trunc from 0x03dde5 to 0x0035ea (0x3a7fa bytes) 000423 1774875508.315334 mapread 0x00355f thru 0x0035ea (0x8c bytes) 000424 1774875508.318900 read 0x000cd7 thru 0x0035ea (0x2914 bytes) 000425 1774875508.320039 trunc from 0x0035eb to 0x00d201 (0x9c17 bytes) 000426 1774875508.355757 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd202): Mirror 3: 0x54433b2c (stale) Mirror 4: 0x54433b2c (stale) Mirror 5: 0x54433b2c (stale) Mirror 6: 0x54433b2c (stale) Mirror 8: 0x54433b2c (stale) Mirror 9: 0x54433b2c (stale) Mirror 10: 0x54433b2c (stale) Mirror 11: 0x54433b2c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd202): Mirror 3: 0x54433b2c (stale) Mirror 4: 0x54433b2c (stale) Mirror 5: 0x54433b2c (stale) Mirror 6: 0x54433b2c (stale) Mirror 8: 0x54433b2c (stale) Mirror 9: 0x54433b2c (stale) Mirror 10: 0x54433b2c (stale) Mirror 11: 0x54433b2c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd202: skipped 000428 1774875509.040399 mapread 0x006532 thru 0x00b345 (0x4e14 bytes) 000430 1774875509.132160 mapread 0x00b66d thru 0x00d201 (0x1b95 bytes) 000433 1774875509.135013 read 0x0062c0 thru 0x00d201 (0x6f42 bytes) 000435 1774875509.136958 write 0x0382a0 thru 0x03ffff (0x7d60 bytes) 000437 1774875509.140691 mapread 0x013cda thru 0x01b251 (0x7578 bytes) 000438 1774875509.169844 write 0x011d6f thru 0x01fcaa (0xdf3c bytes) 000439 1774875509.185344 trunc from 0x040000 to 0x022fcb (0x1d034 bytes) 000440 1774875509.227083 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x22fcc): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22fcc): Mirror 3: 0xc045dee (stale) Mirror 4: 0xc045dee (stale) Mirror 5: 0xc045dee (stale) Mirror 6: 0xc045dee (stale) Mirror 8: 0xc045dee (stale) Mirror 9: 0xc045dee (stale) Mirror 10: 0xc045dee (stale) Mirror 11: 0xc045dee (stale) CRC-32 checksum value for chunk [0, 0x22fcc): Mirror 1: 0x3900bd1c Mirror 12: 0x3900bd1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22fcc: skipped 000441 1774875510.514040 write 0x036912 thru 0x03c96b (0x605a bytes) 000443 1774875510.701417 read 0x0219be thru 0x02d0b8 (0xb6fb bytes) 000444 1774875510.712175 read 0x037f44 thru 0x03c96b (0x4a28 bytes) 000446 1774875510.717442 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3c96c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c96c): Mirror 3: 0x45af85e8 (stale) Mirror 4: 0x45af85e8 (stale) Mirror 5: 0x45af85e8 (stale) Mirror 6: 0x45af85e8 (stale) Mirror 8: 0x45af85e8 (stale) Mirror 9: 0x45af85e8 (stale) Mirror 10: 0x45af85e8 (stale) Mirror 11: 0x45af85e8 (stale) Mirror 12: 0xff468f07 (stale) CRC-32 checksum value for chunk [0, 0x3c96c): Mirror 1: 0xa65eb3a3 Mirror 13: 0xa65eb3a3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c96c: skipped 000447 1774875511.827509 mapwrite 0x008bef thru 0x00e1bc (0x55ce bytes) 000448 1774875512.093647 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3c96c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c96c): Mirror 3: 0x45af85e8 (stale) Mirror 4: 0x45af85e8 (stale) Mirror 5: 0x45af85e8 (stale) Mirror 6: 0x45af85e8 (stale) Mirror 8: 0x45af85e8 (stale) Mirror 9: 0x45af85e8 (stale) Mirror 10: 0x45af85e8 (stale) Mirror 11: 0x45af85e8 (stale) Mirror 12: 0xff468f07 (stale) Mirror 13: 0xa65eb3a3 (stale) CRC-32 checksum value for chunk [0, 0x3c96c): Mirror 1: 0xa9678576 Mirror 14: 0xa9678576 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c96c: skipped 000451 1774875513.233667 mapread 0x022e4a thru 0x031d1a (0xeed1 bytes) 000452 1774875513.286163 read 0x00dcdc thru 0x0170dc (0x9401 bytes) 000455 1774875513.316001 mapread 0x00ee5c thru 0x01ae33 (0xbfd8 bytes) 000456 1774875513.337164 write 0x006996 thru 0x00730e (0x979 bytes) 000459 1774875513.430162 mapread 0x02a938 thru 0x02b4f9 (0xbc2 bytes) 000460 1774875513.439813 write 0x0162e5 thru 0x01e255 (0x7f71 bytes) 000461 1774875513.449921 write 0x007a1f thru 0x010ddc (0x93be bytes) 000462 1774875513.455333 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3c96c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c96c): Mirror 3: 0x45af85e8 (stale) Mirror 4: 0x45af85e8 (stale) Mirror 5: 0x45af85e8 (stale) Mirror 6: 0x45af85e8 (stale) Mirror 8: 0x45af85e8 (stale) Mirror 9: 0x45af85e8 (stale) Mirror 10: 0x45af85e8 (stale) Mirror 11: 0x45af85e8 (stale) Mirror 12: 0xff468f07 (stale) Mirror 13: 0xa65eb3a3 (stale) Mirror 14: 0xa9678576 (stale) CRC-32 checksum value for chunk [0, 0x3c96c): Mirror 1: 0xbb7457d9 Mirror 15: 0xbb7457d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c96c: skipped 000463 1774875514.449076 write 0x00560d thru 0x008b54 (0x3548 bytes) 000465 1774875514.548992 read 0x014e19 thru 0x016aa0 (0x1c88 bytes) 000467 1774875514.558118 trunc from 0x03c96c to 0x0112b7 (0x2b6b4 bytes) 000470 1774875514.681286 MIRROR_SPLIT mirror 3 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x112b8): Mirror 3: 0xf891cac8 (stale) Mirror 4: 0xf891cac8 (stale) Mirror 6: 0xf891cac8 (stale) Mirror 8: 0xf891cac8 (stale) Mirror 9: 0xf891cac8 (stale) Mirror 10: 0xf891cac8 (stale) Mirror 11: 0xf891cac8 (stale) Mirror 12: 0xf7ea33c1 (stale) Mirror 13: 0xf7ea33c1 (stale) Mirror 14: 0x1c42d6c1 (stale) Mirror 15: 0x748d1e62 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x112b8): Mirror 3: 0xf891cac8 (stale) Mirror 4: 0xf891cac8 (stale) Mirror 6: 0xf891cac8 (stale) Mirror 8: 0xf891cac8 (stale) Mirror 9: 0xf891cac8 (stale) Mirror 10: 0xf891cac8 (stale) Mirror 11: 0xf891cac8 (stale) Mirror 12: 0xf7ea33c1 (stale) Mirror 13: 0xf7ea33c1 (stale) Mirror 14: 0x1c42d6c1 (stale) Mirror 15: 0x748d1e62 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x112b8: skipped 000472 1774875515.394911 mapwrite 0x00759b thru 0x011ff5 (0xaa5b bytes) 000474 1774875515.484652 read 0x00cfd8 thru 0x011ff5 (0x501e bytes) 000475 1774875515.485573 mapwrite 0x03ff26 thru 0x03ffff (0xda bytes) 000476 1774875515.594088 mapread 0x00b54a thru 0x00cbd1 (0x1688 bytes) 000477 1774875515.597307 mapread 0x03b96a thru 0x03ffff (0x4696 bytes) 000479 1774875515.601059 write 0x013111 thru 0x01cee3 (0x9dd3 bytes) 000481 1774875515.616749 mapwrite 0x0212c4 thru 0x0292cc (0x8009 bytes) 000483 1774875515.670113 write 0x016335 thru 0x01ff1c (0x9be8 bytes) 000484 1774875515.671379 mapwrite 0x018cf0 thru 0x021070 (0x8381 bytes) 000488 1774875515.740299 MIRROR_RESYNC 12 mirrors 000489 1774875517.476196 read 0x0144a5 thru 0x0234d8 (0xf034 bytes) 000490 1774875517.492155 mapread 0x02ede4 thru 0x033708 (0x4925 bytes) 000491 1774875517.524859 write 0x035b55 thru 0x03ffff (0xa4ab bytes) 000492 1774875517.613252 mapread 0x0206f1 thru 0x0241ae (0x3abe bytes) 000493 1774875517.628767 mapwrite 0x01f665 thru 0x0270bc (0x7a58 bytes) 000495 1774875517.679788 mapwrite 0x016b78 thru 0x019e0d (0x3296 bytes) 000496 1774875517.727672 mapwrite 0x02912d thru 0x034a9c (0xb970 bytes) 000497 1774875517.783496 read 0x01ede7 thru 0x026066 (0x7280 bytes) 000499 1774875517.784786 read 0x001da4 thru 0x00ac7a (0x8ed7 bytes) 000501 1774875517.797975 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xd6f7c52f (stale) Mirror 4: 0xd6f7c52f (stale) Mirror 6: 0xd6f7c52f (stale) Mirror 8: 0xd6f7c52f (stale) Mirror 9: 0xd6f7c52f (stale) Mirror 10: 0xd6f7c52f (stale) Mirror 11: 0xd6f7c52f (stale) Mirror 12: 0xd6f7c52f (stale) Mirror 13: 0xd6f7c52f (stale) Mirror 14: 0xd6f7c52f (stale) Mirror 15: 0xd6f7c52f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7d8e25d3 Mirror 16: 0x7d8e25d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000502 1774875518.801787 mapwrite 0x019222 thru 0x0249ff (0xb7de bytes) 000504 1774875519.112633 read 0x0321e7 thru 0x03b2d6 (0x90f0 bytes) 000505 1774875519.123548 write 0x02192c thru 0x025afa (0x41cf bytes) 000508 1774875519.125818 mapread 0x00343f thru 0x0045d3 (0x1195 bytes) 000509 1774875519.141697 mapread 0x01ebb1 thru 0x02797d (0x8dcd bytes) 000510 1774875519.143051 mapread 0x01f43e thru 0x02ee0e (0xf9d1 bytes) 000511 1774875519.167201 read 0x037462 thru 0x03fce8 (0x8887 bytes) 000512 1774875519.179365 write 0x0297e5 thru 0x034350 (0xab6c bytes) 000513 1774875519.180753 write 0x0033c9 thru 0x01236f (0xefa7 bytes) 000514 1774875519.183787 trunc from 0x040000 to 0x02cf0d (0x130f2 bytes) 000515 1774875519.233517 mapread 0x00c013 thru 0x012840 (0x682e bytes) 000519 1774875519.246697 mapread 0x025ad0 thru 0x026d22 (0x1253 bytes) 000522 1774875519.262095 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cf0e): Mirror 3: 0xff0d0168 (stale) Mirror 4: 0xff0d0168 (stale) Mirror 6: 0xff0d0168 (stale) Mirror 8: 0xff0d0168 (stale) Mirror 9: 0xff0d0168 (stale) Mirror 10: 0xff0d0168 (stale) Mirror 11: 0xff0d0168 (stale) Mirror 12: 0xff0d0168 (stale) Mirror 13: 0xff0d0168 (stale) Mirror 14: 0xff0d0168 (stale) Mirror 15: 0xff0d0168 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cf0e): Mirror 3: 0xff0d0168 (stale) Mirror 4: 0xff0d0168 (stale) Mirror 6: 0xff0d0168 (stale) Mirror 8: 0xff0d0168 (stale) Mirror 9: 0xff0d0168 (stale) Mirror 10: 0xff0d0168 (stale) Mirror 11: 0xff0d0168 (stale) Mirror 12: 0xff0d0168 (stale) Mirror 13: 0xff0d0168 (stale) Mirror 14: 0xff0d0168 (stale) Mirror 15: 0xff0d0168 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cf0e: skipped 000523 1774875520.172927 write 0x01517b thru 0x01a62d (0x54b3 bytes) 000525 1774875520.194645 write 0x0097a1 thru 0x012910 (0x9170 bytes) 000527 1774875520.208840 trunc from 0x02cf0e to 0x023856 (0x96b7 bytes) 000528 1774875520.223949 read 0x019a7c thru 0x01cadb (0x3060 bytes) 000530 1774875520.232624 mapread 0x00f7e1 thru 0x011bba (0x23da bytes) 000533 1774875520.236086 trunc from 0x023857 to 0x026c8a (0x3434 bytes) 000535 1774875520.251812 write 0x008acb thru 0x011694 (0x8bca bytes) 000539 1774875520.260137 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26c8b): Mirror 3: 0x1e127f2f (stale) Mirror 4: 0x1e127f2f (stale) Mirror 8: 0x1e127f2f (stale) Mirror 9: 0x1e127f2f (stale) Mirror 10: 0x1e127f2f (stale) Mirror 11: 0x1e127f2f (stale) Mirror 12: 0x1e127f2f (stale) Mirror 13: 0x1e127f2f (stale) Mirror 14: 0x1e127f2f (stale) Mirror 15: 0x1e127f2f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26c8b): Mirror 3: 0x1e127f2f (stale) Mirror 4: 0x1e127f2f (stale) Mirror 8: 0x1e127f2f (stale) Mirror 9: 0x1e127f2f (stale) Mirror 10: 0x1e127f2f (stale) Mirror 11: 0x1e127f2f (stale) Mirror 12: 0x1e127f2f (stale) Mirror 13: 0x1e127f2f (stale) Mirror 14: 0x1e127f2f (stale) Mirror 15: 0x1e127f2f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26c8b: skipped 000541 1774875520.777591 mapread 0x012ab3 thru 0x01b15a (0x86a8 bytes) 000546 1774875520.798715 mapwrite 0x024eb7 thru 0x029385 (0x44cf bytes) 000550 1774875520.961915 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x29386): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29386): Mirror 3: 0x7cf3a6e6 (stale) Mirror 4: 0x7cf3a6e6 (stale) Mirror 8: 0x7cf3a6e6 (stale) Mirror 9: 0x7cf3a6e6 (stale) Mirror 10: 0x7cf3a6e6 (stale) Mirror 11: 0x7cf3a6e6 (stale) Mirror 12: 0x7cf3a6e6 (stale) Mirror 13: 0x7cf3a6e6 (stale) Mirror 14: 0x7cf3a6e6 (stale) Mirror 15: 0x7cf3a6e6 (stale) CRC-32 checksum value for chunk [0, 0x29386): Mirror 1: 0x15d6e214 Mirror 16: 0x15d6e214 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29386: skipped 000551 1774875522.450094 read 0x01aa6d thru 0x020b6e (0x6102 bytes) 000555 1774875522.557899 trunc from 0x029386 to 0x034b97 (0xb812 bytes) 000559 1774875522.768611 write 0x0230e6 thru 0x028914 (0x582f bytes) 000564 1774875522.798385 read 0x018ab6 thru 0x022c80 (0xa1cb bytes) 000565 1774875522.821413 write 0x027e9f thru 0x028557 (0x6b9 bytes) 000566 1774875522.823772 mapread 0x03305c thru 0x034b97 (0x1b3c bytes) 000567 1774875522.840359 read 0x0088fc thru 0x00ff3d (0x7642 bytes) 000568 1774875522.847253 write 0x02d776 thru 0x033d56 (0x65e1 bytes) 000569 1774875522.851248 trunc from 0x034b98 to 0x021981 (0x13216 bytes) 000570 1774875522.918663 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x21982): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21982): Mirror 3: 0x77452aca (stale) Mirror 4: 0x77452aca (stale) Mirror 8: 0x77452aca (stale) Mirror 9: 0x77452aca (stale) Mirror 10: 0x77452aca (stale) Mirror 11: 0x77452aca (stale) Mirror 12: 0x77452aca (stale) Mirror 13: 0x77452aca (stale) Mirror 14: 0x77452aca (stale) Mirror 15: 0x77452aca (stale) Mirror 16: 0x7c33b9c7 (stale) CRC-32 checksum value for chunk [0, 0x21982): Mirror 1: 0x7c33b9c7 Mirror 17: 0x7c33b9c7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21982: skipped 000571 1774875524.263768 trunc from 0x021982 to 0x00b7f7 (0x1618a bytes) 000573 1774875524.389803 mapread 0x00a876 thru 0x00b7f7 (0xf82 bytes) 000574 1774875524.401473 read 0x00027d thru 0x00202d (0x1db1 bytes) 000575 1774875524.406745 write 0x03ba50 thru 0x03ffff (0x45b0 bytes) 000576 1774875524.408634 trunc from 0x040000 to 0x0285a1 (0x17a5e bytes) 000579 1774875524.439543 mapwrite 0x03f4f6 thru 0x03ffff (0xb0a bytes) 000580 1774875524.549384 mapread 0x00184d thru 0x010049 (0xe7fd bytes) 000584 1774875524.565704 mapwrite 0x0247db thru 0x03169f (0xcec5 bytes) 000585 1774875524.714422 write 0x03cf7e thru 0x03d138 (0x1bb bytes) 000587 1774875524.719233 read 0x039f5e thru 0x03ffff (0x60a2 bytes) 000589 1774875524.722625 write 0x033a5a thru 0x03ffff (0xc5a6 bytes) 000592 1774875524.728699 trunc from 0x040000 to 0x01f59a (0x20a65 bytes) 000593 1774875524.759600 write 0x007bed thru 0x01542f (0xd843 bytes) 000594 1774875524.769107 read 0x0022d6 thru 0x00690d (0x4638 bytes) 000597 1774875524.773541 mapread 0x009050 thru 0x013231 (0xa1e2 bytes) 000599 1774875524.776373 mapread 0x005e48 thru 0x011c07 (0xbdc0 bytes) 000601 1774875524.779391 read 0x00f5be thru 0x010739 (0x117c bytes) 000604 1774875524.784197 read 0x002485 thru 0x00e10a (0xbc86 bytes) 000605 1774875524.785745 write 0x02883d thru 0x033cc5 (0xb489 bytes) 000606 1774875524.789124 write 0x01ee27 thru 0x0201e8 (0x13c2 bytes) 000607 1774875524.806253 write 0x038bad thru 0x03e7af (0x5c03 bytes) 000608 1774875524.807650 trunc from 0x03e7b0 to 0x034a3d (0x9d72 bytes) 000611 1774875524.838430 read 0x027634 thru 0x02eb8b (0x7558 bytes) 000613 1774875524.839837 write 0x02ad5c thru 0x02da33 (0x2cd8 bytes) 000614 1774875524.845189 trunc from 0x034a3e to 0x031f4c (0x2af1 bytes) 000616 1774875524.877662 trunc from 0x031f4d to 0x036d87 (0x4e3b bytes) 000617 1774875524.904488 mapread 0x00e9e3 thru 0x018d8f (0xa3ad bytes) 000619 1774875524.914347 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36d88): Mirror 3: 0x52550356 (stale) Mirror 4: 0x52550356 (stale) Mirror 8: 0x52550356 (stale) Mirror 9: 0x52550356 (stale) Mirror 10: 0x52550356 (stale) Mirror 11: 0x52550356 (stale) Mirror 12: 0x52550356 (stale) Mirror 13: 0x52550356 (stale) Mirror 14: 0x52550356 (stale) Mirror 15: 0x52550356 (stale) Mirror 16: 0xbc673ff0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36d88): Mirror 3: 0x52550356 (stale) Mirror 4: 0x52550356 (stale) Mirror 8: 0x52550356 (stale) Mirror 9: 0x52550356 (stale) Mirror 10: 0x52550356 (stale) Mirror 11: 0x52550356 (stale) Mirror 12: 0x52550356 (stale) Mirror 13: 0x52550356 (stale) Mirror 14: 0x52550356 (stale) Mirror 15: 0x52550356 (stale) Mirror 16: 0xbc673ff0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36d88: skipped 000620 1774875525.672835 read 0x02eaa6 thru 0x032f5e (0x44b9 bytes) 000624 1774875525.704370 MIRROR_RESYNC 12 mirrors 000625 1774875527.475634 mapwrite 0x013fc1 thru 0x015b02 (0x1b42 bytes) 000626 1774875527.643628 mapread 0x00c34c thru 0x00f54d (0x3202 bytes) 000628 1774875527.660397 trunc from 0x036d88 to 0x027213 (0xfb74 bytes) 000629 1774875527.685863 mapread 0x0049c6 thru 0x004d89 (0x3c4 bytes) 000632 1774875527.698575 MIRROR_RESYNC 12 mirrors 000633 1774875529.664906 MIRROR_EXTEND to 13 mirrors 000636 1774875530.478532 read 0x005202 thru 0x00a2e0 (0x50df bytes) 000637 1774875530.486878 trunc from 0x027214 to 0x03ca2a (0x15817 bytes) 000638 1774875530.568595 write 0x015f00 thru 0x01cbb8 (0x6cb9 bytes) 000640 1774875530.578762 write 0x03efa5 thru 0x03ffff (0x105b bytes) 000641 1774875530.580755 write 0x037c8f thru 0x03a506 (0x2878 bytes) 000642 1774875530.589296 mapread 0x03fe25 thru 0x03ffff (0x1db bytes) 000643 1774875530.590311 mapread 0x010d81 thru 0x01c9ee (0xbc6e bytes) 000645 1774875530.598741 write 0x03a9e4 thru 0x03ffff (0x561c bytes) 000647 1774875530.601128 trunc from 0x040000 to 0x01a6aa (0x25955 bytes) 000648 1774875530.621134 read 0x009c8b thru 0x018c32 (0xefa8 bytes) 000649 1774875530.626043 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x1a6ab): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a6ab): Mirror 3: 0xf9c32cd2 (stale) Mirror 4: 0xf9c32cd2 (stale) Mirror 8: 0xf9c32cd2 (stale) Mirror 9: 0xf9c32cd2 (stale) Mirror 10: 0xf9c32cd2 (stale) Mirror 11: 0xf9c32cd2 (stale) Mirror 12: 0xf9c32cd2 (stale) Mirror 13: 0xf9c32cd2 (stale) Mirror 14: 0xf9c32cd2 (stale) Mirror 15: 0xf9c32cd2 (stale) Mirror 16: 0xf9c32cd2 (stale) Mirror 17: 0xf9c32cd2 (stale) CRC-32 checksum value for chunk [0, 0x1a6ab): Mirror 1: 0x70808d59 Mirror 18: 0x70808d59 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a6ab: skipped 000650 1774875531.897337 mapwrite 0x030d5f thru 0x031f1f (0x11c1 bytes) 000651 1774875532.148319 write 0x017d18 thru 0x0207fb (0x8ae4 bytes) 000653 1774875532.190806 read 0x02b8f0 thru 0x02fd13 (0x4424 bytes) 000655 1774875532.222275 read 0x029ae4 thru 0x02a834 (0xd51 bytes) 000656 1774875532.236757 write 0x00851b thru 0x00bbe3 (0x36c9 bytes) 000657 1774875532.238154 mapread 0x029ad4 thru 0x031f1f (0x844c bytes) 000658 1774875532.240666 read 0x020304 thru 0x020eaf (0xbac bytes) 000659 1774875532.250487 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x31f20): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31f20): Mirror 3: 0x5e568c13 (stale) Mirror 4: 0x5e568c13 (stale) Mirror 8: 0x5e568c13 (stale) Mirror 9: 0x5e568c13 (stale) Mirror 10: 0x5e568c13 (stale) Mirror 11: 0x5e568c13 (stale) Mirror 12: 0x5e568c13 (stale) Mirror 13: 0x5e568c13 (stale) Mirror 14: 0x5e568c13 (stale) Mirror 15: 0x5e568c13 (stale) Mirror 16: 0x5e568c13 (stale) Mirror 17: 0x5e568c13 (stale) Mirror 18: 0xa8793073 (stale) CRC-32 checksum value for chunk [0, 0x31f20): Mirror 1: 0x280f1c54 Mirror 19: 0x280f1c54 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31f20: skipped 000662 1774875533.478345 read 0x00b877 thru 0x0139a6 (0x8130 bytes) 000663 1774875533.494349 MIRROR_RESYNC 15 mirrors 000670 1774875535.533122 MIRROR_EXTEND to 16 mirrors 000672 1774875536.610037 mapwrite 0x02874d thru 0x034031 (0xb8e5 bytes) 000673 1774875536.913763 trunc from 0x034032 to 0x002032 (0x31fff bytes) 000674 1774875536.944662 mapwrite 0x03bea4 thru 0x03ffff (0x415c bytes) 000679 1774875537.044932 read 0x000a18 thru 0x008ed3 (0x84bc bytes) 000681 1774875537.063577 mapread 0x026cf6 thru 0x02ad0a (0x4015 bytes) 000682 1774875537.082449 read 0x0061c7 thru 0x009d34 (0x3b6e bytes) 000684 1774875537.105431 read 0x01df43 thru 0x02c5e0 (0xe69e bytes) 000688 1774875537.108530 trunc from 0x040000 to 0x026162 (0x19e9d bytes) 000689 1774875537.123688 read 0x00c0ec thru 0x01b84f (0xf764 bytes) 000690 1774875537.124511 mapwrite 0x03ec2a thru 0x03ffff (0x13d6 bytes) 000691 1774875537.233820 mapread 0x030085 thru 0x03fed5 (0xfe51 bytes) 000692 1774875537.235304 read 0x006d81 thru 0x0139fd (0xcc7d bytes) 000698 1774875537.254689 mapwrite 0x033faa thru 0x038aca (0x4b21 bytes) 000700 1774875537.331427 read 0x03464a thru 0x036e33 (0x27ea bytes) 000701 1774875537.332609 write 0x00606a thru 0x01401e (0xdfb5 bytes) 000703 1774875537.335361 write 0x03ebc0 thru 0x03ffff (0x1440 bytes) 000704 1774875537.338080 read 0x0256ba thru 0x02c4ae (0x6df5 bytes) 000708 1774875537.342340 read 0x011b53 thru 0x012dd1 (0x127f bytes) 000709 1774875537.343393 trunc from 0x040000 to 0x02e5ef (0x11a10 bytes) 000711 1774875537.368115 write 0x0198f3 thru 0x01c1cc (0x28da bytes) 000712 1774875537.369196 mapread 0x00f0db thru 0x019f86 (0xaeac bytes) 000713 1774875537.372442 mapwrite 0x036bdc thru 0x036cfc (0x121 bytes) 000716 1774875537.463717 write 0x014b16 thru 0x01eff3 (0xa4de bytes) 000717 1774875537.465017 trunc from 0x036cfd to 0x002342 (0x349ba bytes) 000719 1774875537.491501 read 0x001f90 thru 0x002342 (0x3b3 bytes) 000721 1774875537.498379 MIRROR_SPLIT mirror 9 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2343): Mirror 3: 0xf052c2ca (stale) Mirror 4: 0xf052c2ca (stale) Mirror 8: 0xf052c2ca (stale) Mirror 9: 0xf052c2ca (stale) Mirror 10: 0xf052c2ca (stale) Mirror 11: 0xf052c2ca (stale) Mirror 12: 0xf052c2ca (stale) Mirror 13: 0xf052c2ca (stale) Mirror 15: 0xf052c2ca (stale) Mirror 16: 0xf052c2ca (stale) Mirror 17: 0xf052c2ca (stale) Mirror 18: 0xf052c2ca (stale) Mirror 19: 0xf052c2ca (stale) Mirror 20: 0xf052c2ca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2343): Mirror 3: 0xf052c2ca (stale) Mirror 4: 0xf052c2ca (stale) Mirror 8: 0xf052c2ca (stale) Mirror 9: 0xf052c2ca (stale) Mirror 10: 0xf052c2ca (stale) Mirror 11: 0xf052c2ca (stale) Mirror 12: 0xf052c2ca (stale) Mirror 13: 0xf052c2ca (stale) Mirror 15: 0xf052c2ca (stale) Mirror 16: 0xf052c2ca (stale) Mirror 17: 0xf052c2ca (stale) Mirror 18: 0xf052c2ca (stale) Mirror 19: 0xf052c2ca (stale) Mirror 20: 0xf052c2ca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2343: skipped 000723 1774875537.893631 read 0x000b34 thru 0x0017fd (0xcca bytes) 000724 1774875537.900059 write 0x022cc5 thru 0x02893e (0x5c7a bytes) 000726 1774875537.903755 mapwrite 0x03a75a thru 0x03ffff (0x58a6 bytes) 000727 1774875538.016081 trunc from 0x040000 to 0x03cb4b (0x34b4 bytes) 000729 1774875538.041087 mapwrite 0x024b58 thru 0x025879 (0xd22 bytes) 000730 1774875538.087079 read 0x001075 thru 0x009a25 (0x89b1 bytes) 000737 1774875538.105121 read 0x026a1c thru 0x027c85 (0x126a bytes) 000739 1774875538.106595 trunc from 0x03cb4c to 0x0178e5 (0x25266 bytes) 000741 1774875538.132680 MIRROR_SPLIT mirror 11 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x178e6): Mirror 3: 0x401382ed (stale) Mirror 4: 0x401382ed (stale) Mirror 8: 0x401382ed (stale) Mirror 9: 0x401382ed (stale) Mirror 10: 0x401382ed (stale) Mirror 11: 0x401382ed (stale) Mirror 12: 0x401382ed (stale) Mirror 13: 0x401382ed (stale) Mirror 15: 0x401382ed (stale) Mirror 16: 0x401382ed (stale) Mirror 18: 0x401382ed (stale) Mirror 19: 0x401382ed (stale) Mirror 20: 0x401382ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x178e6): Mirror 3: 0x401382ed (stale) Mirror 4: 0x401382ed (stale) Mirror 8: 0x401382ed (stale) Mirror 9: 0x401382ed (stale) Mirror 10: 0x401382ed (stale) Mirror 11: 0x401382ed (stale) Mirror 12: 0x401382ed (stale) Mirror 13: 0x401382ed (stale) Mirror 15: 0x401382ed (stale) Mirror 16: 0x401382ed (stale) Mirror 18: 0x401382ed (stale) Mirror 19: 0x401382ed (stale) Mirror 20: 0x401382ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x178e6: skipped 000743 1774875538.695819 MIRROR_RESYNC 14 mirrors 000744 1774875540.569983 write 0x03cd06 thru 0x03d74b (0xa46 bytes) 000746 1774875540.695716 MIRROR_SPLIT mirror 10 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d74c): Mirror 3: 0xb5fc2264 (stale) Mirror 4: 0xb5fc2264 (stale) Mirror 8: 0xb5fc2264 (stale) Mirror 9: 0xb5fc2264 (stale) Mirror 10: 0xb5fc2264 (stale) Mirror 11: 0xb5fc2264 (stale) Mirror 12: 0xb5fc2264 (stale) Mirror 13: 0xb5fc2264 (stale) Mirror 15: 0xb5fc2264 (stale) Mirror 18: 0xb5fc2264 (stale) Mirror 19: 0xb5fc2264 (stale) Mirror 20: 0xb5fc2264 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d74c): Mirror 3: 0xb5fc2264 (stale) Mirror 4: 0xb5fc2264 (stale) Mirror 8: 0xb5fc2264 (stale) Mirror 9: 0xb5fc2264 (stale) Mirror 10: 0xb5fc2264 (stale) Mirror 11: 0xb5fc2264 (stale) Mirror 12: 0xb5fc2264 (stale) Mirror 13: 0xb5fc2264 (stale) Mirror 15: 0xb5fc2264 (stale) Mirror 18: 0xb5fc2264 (stale) Mirror 19: 0xb5fc2264 (stale) Mirror 20: 0xb5fc2264 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d74c: skipped 000747 1774875541.641648 mapread 0x012d94 thru 0x01cce5 (0x9f52 bytes) 000749 1774875541.681547 trunc from 0x03d74c to 0x019518 (0x24233 bytes) 000750 1774875541.715804 mapread 0x00f45f thru 0x019518 (0xa0ba bytes) 000751 1774875541.717369 write 0x008c24 thru 0x015ef3 (0xd2d0 bytes) 000754 1774875541.742469 trunc from 0x019519 to 0x0241cd (0xacb5 bytes) 000755 1774875541.776179 mapread 0x00ca25 thru 0x015b95 (0x9171 bytes) 000756 1774875541.777496 read 0x01cc1e thru 0x01f504 (0x28e7 bytes) 000758 1774875541.789361 mapwrite 0x004683 thru 0x007d20 (0x369e bytes) 000761 1774875541.888742 trunc from 0x0241ce to 0x035566 (0x11399 bytes) 000764 1774875541.920754 read 0x00941c thru 0x016c3d (0xd822 bytes) 000766 1774875541.926431 mapread 0x019c18 thru 0x022249 (0x8632 bytes) 000768 1774875541.936820 trunc from 0x035567 to 0x03de61 (0x88fb bytes) 000772 1774875541.961071 trunc from 0x03de62 to 0x032286 (0xbbdb bytes) 000773 1774875541.979539 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32287): Mirror 3: 0x76aeecaa (stale) Mirror 4: 0x76aeecaa (stale) Mirror 8: 0x76aeecaa (stale) Mirror 9: 0x76aeecaa (stale) Mirror 10: 0x76aeecaa (stale) Mirror 11: 0x76aeecaa (stale) Mirror 12: 0x76aeecaa (stale) Mirror 13: 0x76aeecaa (stale) Mirror 15: 0x76aeecaa (stale) Mirror 18: 0x76aeecaa (stale) Mirror 19: 0x76aeecaa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32287): Mirror 3: 0x76aeecaa (stale) Mirror 4: 0x76aeecaa (stale) Mirror 8: 0x76aeecaa (stale) Mirror 9: 0x76aeecaa (stale) Mirror 10: 0x76aeecaa (stale) Mirror 11: 0x76aeecaa (stale) Mirror 12: 0x76aeecaa (stale) Mirror 13: 0x76aeecaa (stale) Mirror 15: 0x76aeecaa (stale) Mirror 18: 0x76aeecaa (stale) Mirror 19: 0x76aeecaa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32287: skipped 000775 1774875542.583363 trunc from 0x032287 to 0x01e4bf (0x13dc7 bytes) 000777 1774875542.604434 trunc from 0x01e4c0 to 0x025643 (0x7184 bytes) 000778 1774875542.623328 mapwrite 0x029db1 thru 0x02b9d7 (0x1c27 bytes) 000779 1774875542.723865 write 0x03556b thru 0x038f87 (0x3a1d bytes) 000781 1774875542.725419 read 0x01273e thru 0x01a2cd (0x7b90 bytes) 000784 1774875542.738823 trunc from 0x038f88 to 0x00f03d (0x29f4a bytes) 000786 1774875542.757950 mapwrite 0x007d5b thru 0x01192b (0x9bd1 bytes) 000788 1774875542.846756 trunc from 0x01192c to 0x005806 (0xc125 bytes) 000789 1774875542.874068 write 0x00b869 thru 0x011c9b (0x6433 bytes) 000792 1774875542.889558 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x11c9c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x11c9c): Mirror 3: 0x137e599d (stale) Mirror 4: 0x137e599d (stale) Mirror 8: 0x137e599d (stale) Mirror 9: 0x137e599d (stale) Mirror 10: 0x137e599d (stale) Mirror 11: 0x137e599d (stale) Mirror 12: 0x137e599d (stale) Mirror 13: 0x137e599d (stale) Mirror 15: 0x137e599d (stale) Mirror 18: 0x137e599d (stale) Mirror 19: 0x137e599d (stale) CRC-32 checksum value for chunk [0, 0x11c9c): Mirror 1: 0x721223ea Mirror 20: 0x721223ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x11c9c: skipped 000793 1774875543.859927 mapwrite 0x03380f thru 0x03cae3 (0x92d5 bytes) 000794 1774875544.043007 write 0x01866c thru 0x02834a (0xfcdf bytes) 000795 1774875544.069710 read 0x009d16 thru 0x00c40d (0x26f8 bytes) 000796 1774875544.075710 read 0x036605 thru 0x03cae3 (0x64df bytes) 000797 1774875544.076524 mapwrite 0x017f96 thru 0x021fa0 (0xa00b bytes) 000798 1774875544.146169 MIRROR_SPLIT mirror 12 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cae4): Mirror 3: 0x3c3ee789 (stale) Mirror 4: 0x3c3ee789 (stale) Mirror 8: 0x3c3ee789 (stale) Mirror 9: 0x3c3ee789 (stale) Mirror 10: 0x3c3ee789 (stale) Mirror 11: 0x3c3ee789 (stale) Mirror 12: 0x3c3ee789 (stale) Mirror 13: 0x3c3ee789 (stale) Mirror 15: 0x3c3ee789 (stale) Mirror 18: 0x3c3ee789 (stale) Mirror 19: 0x3c3ee789 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cae4): Mirror 3: 0x3c3ee789 (stale) Mirror 4: 0x3c3ee789 (stale) Mirror 8: 0x3c3ee789 (stale) Mirror 9: 0x3c3ee789 (stale) Mirror 10: 0x3c3ee789 (stale) Mirror 11: 0x3c3ee789 (stale) Mirror 12: 0x3c3ee789 (stale) Mirror 13: 0x3c3ee789 (stale) Mirror 15: 0x3c3ee789 (stale) Mirror 18: 0x3c3ee789 (stale) Mirror 19: 0x3c3ee789 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cae4: skipped 000799 1774875544.859953 mapwrite 0x002adb thru 0x00f3e5 (0xc90b bytes) 000800 1774875544.986908 mapread 0x03c1f4 thru 0x03cae3 (0x8f0 bytes) 000802 1774875545.001263 trunc from 0x03cae4 to 0x00bac4 (0x3101f bytes) 000803 1774875545.015503 mapwrite 0x00636d thru 0x008434 (0x20c8 bytes) 000805 1774875545.125176 MIRROR_RESYNC 12 mirrors 000806 1774875546.952917 MIRROR_EXTEND to 13 mirrors 000807 1774875547.672653 trunc from 0x00bac5 to 0x01c687 (0x10bc3 bytes) 000809 1774875547.802821 mapread 0x014b8f thru 0x01c687 (0x7af9 bytes) 000811 1774875547.854649 write 0x036d82 thru 0x03ffff (0x927e bytes) 000812 1774875547.859239 read 0x039b59 thru 0x03ffff (0x64a7 bytes) 000813 1774875547.860014 mapwrite 0x00c8db thru 0x017b7f (0xb2a5 bytes) 000814 1774875547.937409 read 0x02c807 thru 0x037377 (0xab71 bytes) 000816 1774875547.950343 mapwrite 0x028e43 thru 0x02c28d (0x344b bytes) 000817 1774875547.992312 mapread 0x0245c9 thru 0x02ecf5 (0xa72d bytes) 000818 1774875548.000718 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xeb850f3b (stale) Mirror 4: 0xeb850f3b (stale) Mirror 8: 0xeb850f3b (stale) Mirror 9: 0xeb850f3b (stale) Mirror 10: 0xeb850f3b (stale) Mirror 11: 0xeb850f3b (stale) Mirror 12: 0xeb850f3b (stale) Mirror 13: 0xeb850f3b (stale) Mirror 15: 0xeb850f3b (stale) Mirror 18: 0xeb850f3b (stale) Mirror 19: 0xeb850f3b (stale) Mirror 20: 0xeb850f3b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8c2d8a78 Mirror 21: 0x8c2d8a78 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000820 1774875549.071369 trunc from 0x040000 to 0x0002b0 (0x3fd4f bytes) 000821 1774875549.189311 trunc from 0x0002b1 to 0x0259d1 (0x25721 bytes) 000822 1774875549.208848 mapwrite 0x013208 thru 0x01f3ee (0xc1e7 bytes) 000824 1774875549.292808 mapread 0x00d023 thru 0x01c942 (0xf920 bytes) 000825 1774875549.304252 read 0x011124 thru 0x01bcf5 (0xabd2 bytes) 000826 1774875549.306230 mapread 0x015199 thru 0x016101 (0xf69 bytes) 000827 1774875549.307517 trunc from 0x0259d2 to 0x008f78 (0x1ca59 bytes) 000828 1774875549.324811 write 0x000d18 thru 0x00b81b (0xab04 bytes) 000829 1774875549.335656 mapwrite 0x03b1d8 thru 0x03ffff (0x4e28 bytes) 000832 1774875549.440413 read 0x01fb35 thru 0x02ef9c (0xf468 bytes) 000834 1774875549.466394 write 0x003113 thru 0x00cb99 (0x9a87 bytes) 000835 1774875549.467751 read 0x000cc6 thru 0x0105b0 (0xf8eb bytes) 000836 1774875549.471986 MIRROR_RESYNC 14 mirrors 000837 1774875551.253102 write 0x026b23 thru 0x035886 (0xed64 bytes) 000839 1774875551.337065 mapwrite 0x02b6f9 thru 0x02bc6c (0x574 bytes) 000841 1774875551.383539 trunc from 0x040000 to 0x025b50 (0x1a4af bytes) 000842 1774875551.413411 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x25b51): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25b51): Mirror 3: 0xffde66b2 (stale) Mirror 4: 0xffde66b2 (stale) Mirror 8: 0xffde66b2 (stale) Mirror 9: 0xffde66b2 (stale) Mirror 10: 0xffde66b2 (stale) Mirror 11: 0xffde66b2 (stale) Mirror 12: 0xffde66b2 (stale) Mirror 13: 0xffde66b2 (stale) Mirror 15: 0xffde66b2 (stale) Mirror 18: 0xffde66b2 (stale) Mirror 19: 0xffde66b2 (stale) Mirror 20: 0xffde66b2 (stale) Mirror 21: 0xffde66b2 (stale) CRC-32 checksum value for chunk [0, 0x25b51): Mirror 1: 0xffde66b2 Mirror 22: 0xffde66b2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25b51: skipped 000843 1774875552.495835 mapwrite 0x01c97c thru 0x02af27 (0xe5ac bytes) 000844 1774875552.701619 mapread 0x00db3e thru 0x01c816 (0xecd9 bytes) 000845 1774875552.721279 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x2af28): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2af28): Mirror 3: 0x47e55fac (stale) Mirror 4: 0x47e55fac (stale) Mirror 8: 0x47e55fac (stale) Mirror 9: 0x47e55fac (stale) Mirror 10: 0x47e55fac (stale) Mirror 11: 0x47e55fac (stale) Mirror 12: 0x47e55fac (stale) Mirror 13: 0x47e55fac (stale) Mirror 15: 0x47e55fac (stale) Mirror 18: 0x47e55fac (stale) Mirror 19: 0x47e55fac (stale) Mirror 20: 0x47e55fac (stale) Mirror 21: 0x47e55fac (stale) Mirror 22: 0x47e55fac (stale) CRC-32 checksum value for chunk [0, 0x2af28): Mirror 1: 0x3ba30822 Mirror 23: 0x3ba30822 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2af28: skipped 000847 1774875553.575980 trunc from 0x02af28 to 0x016fb4 (0x13f73 bytes) 000848 1774875553.698572 MIRROR_RESYNC 16 mirrors 000852 1774875555.766655 mapread 0x00e737 thru 0x0150a4 (0x696e bytes) 000853 1774875555.787446 write 0x02f372 thru 0x0363e3 (0x7072 bytes) 000854 1774875555.863799 read 0x01011c thru 0x0123cb (0x22b0 bytes) 000855 1774875555.870893 mapwrite 0x0218c7 thru 0x02c9b6 (0xb0f0 bytes) 000856 1774875556.045815 read 0x01d01c thru 0x01f601 (0x25e6 bytes) 000857 1774875556.058671 MIRROR_RESYNC 16 mirrors 000858 1774875558.224671 mapwrite 0x005b0e thru 0x008ef7 (0x33ea bytes) 000859 1774875558.433922 mapwrite 0x039c4c thru 0x03c797 (0x2b4c bytes) 000861 1774875558.527329 mapwrite 0x000d80 thru 0x010aa6 (0xfd27 bytes) 000862 1774875558.612803 MIRROR_RESYNC 16 mirrors 000865 1774875560.525364 write 0x039986 thru 0x03ffff (0x667a bytes) 000866 1774875560.604615 trunc from 0x040000 to 0x003440 (0x3cbbf bytes) 000867 1774875560.626799 mapread 0x0028c3 thru 0x003440 (0xb7e bytes) 000868 1774875560.643524 read 0x002f8c thru 0x003440 (0x4b5 bytes) 000871 1774875560.646469 trunc from 0x003441 to 0x007adf (0x469f bytes) 000872 1774875560.664960 mapread 0x003efa thru 0x007adf (0x3be6 bytes) 000874 1774875560.680667 MIRROR_RESYNC 16 mirrors 000875 1774875562.846286 trunc from 0x007ae0 to 0x016f15 (0xf436 bytes) 000877 1774875562.919579 read 0x002255 thru 0x00abff (0x89ab bytes) 000880 1774875562.931857 read 0x00dbac thru 0x00e291 (0x6e6 bytes) 000882 1774875562.951627 trunc from 0x016f16 to 0x01fd79 (0x8e64 bytes) 000889 1774875562.994611 mapread 0x0150e8 thru 0x01d792 (0x86ab bytes) 000893 1774875563.017758 write 0x003786 thru 0x00a6ce (0x6f49 bytes) 000894 1774875563.020926 mapwrite 0x00fd78 thru 0x0162a1 (0x652a bytes) 000895 1774875563.101669 trunc from 0x01fd7a to 0x031b45 (0x11dcc bytes) 000897 1774875563.128999 trunc from 0x031b46 to 0x021694 (0x104b1 bytes) 000898 1774875563.154940 mapread 0x00808d thru 0x00bca0 (0x3c14 bytes) 000899 1774875563.161651 trunc from 0x021695 to 0x032636 (0x10fa2 bytes) 000900 1774875563.184288 mapwrite 0x002db6 thru 0x002ef4 (0x13f bytes) 000901 1774875563.231521 write 0x034da7 thru 0x03ffff (0xb259 bytes) 000902 1774875563.237724 mapread 0x004eec thru 0x005b4b (0xc60 bytes) 000905 1774875563.243719 mapwrite 0x02ac65 thru 0x038683 (0xda1f bytes) 000906 1774875563.373819 write 0x01b4e4 thru 0x0276a6 (0xc1c3 bytes) 000908 1774875563.379857 trunc from 0x040000 to 0x0125a2 (0x2da5d bytes) 000909 1774875563.412720 mapread 0x00be0c thru 0x0125a2 (0x6797 bytes) 000910 1774875563.413981 mapread 0x0025f9 thru 0x0054d1 (0x2ed9 bytes) 000911 1774875563.431580 MIRROR_RESYNC 16 mirrors 000916 1774875565.830008 mapwrite 0x03e465 thru 0x03ffff (0x1b9b bytes) 000917 1774875566.060519 trunc from 0x040000 to 0x03612b (0x9ed4 bytes) 000918 1774875566.085291 mapread 0x00ebc8 thru 0x01c367 (0xd7a0 bytes) 000919 1774875566.108529 read 0x00c0c5 thru 0x00cfd3 (0xf0f bytes) 000920 1774875566.109504 read 0x008db6 thru 0x0143ae (0xb5f9 bytes) 000929 1774875566.120325 read 0x02ab8b thru 0x02df9e (0x3414 bytes) 000931 1774875566.173164 trunc from 0x03612c to 0x019e50 (0x1c2db bytes) 000932 1774875566.198046 read 0x008d23 thru 0x00f33d (0x661b bytes) 000934 1774875566.199637 trunc from 0x019e51 to 0x01bba5 (0x1d55 bytes) 000935 1774875566.238539 MIRROR_RESYNC 16 mirrors 000937 1774875568.117833 read 0x00a9d4 thru 0x00c737 (0x1d64 bytes) 000940 1774875568.123477 mapwrite 0x026fac thru 0x02c29e (0x52f3 bytes) 000942 1774875568.361697 write 0x0253aa thru 0x031c36 (0xc88d bytes) 000944 1774875568.369567 read 0x00fe04 thru 0x015b48 (0x5d45 bytes) 000947 1774875568.392639 mapread 0x01b267 thru 0x01c7a7 (0x1541 bytes) 000949 1774875568.397459 write 0x0108e6 thru 0x010b2b (0x246 bytes) 000951 1774875568.408882 MIRROR_SPLIT mirror 8 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31c37): Mirror 3: 0xaf978216 (stale) Mirror 4: 0xaf978216 (stale) Mirror 8: 0xaf978216 (stale) Mirror 9: 0xaf978216 (stale) Mirror 10: 0xaf978216 (stale) Mirror 11: 0xaf978216 (stale) Mirror 12: 0xaf978216 (stale) Mirror 15: 0xaf978216 (stale) Mirror 18: 0xaf978216 (stale) Mirror 19: 0xaf978216 (stale) Mirror 20: 0xaf978216 (stale) Mirror 21: 0xaf978216 (stale) Mirror 22: 0xaf978216 (stale) Mirror 23: 0xaf978216 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31c37): Mirror 3: 0xaf978216 (stale) Mirror 4: 0xaf978216 (stale) Mirror 8: 0xaf978216 (stale) Mirror 9: 0xaf978216 (stale) Mirror 10: 0xaf978216 (stale) Mirror 11: 0xaf978216 (stale) Mirror 12: 0xaf978216 (stale) Mirror 15: 0xaf978216 (stale) Mirror 18: 0xaf978216 (stale) Mirror 19: 0xaf978216 (stale) Mirror 20: 0xaf978216 (stale) Mirror 21: 0xaf978216 (stale) Mirror 22: 0xaf978216 (stale) Mirror 23: 0xaf978216 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31c37: skipped 000952 1774875569.188202 read 0x00c470 thru 0x0114c4 (0x5055 bytes) 000954 1774875569.206233 mapread 0x00a881 thru 0x011b2b (0x72ab bytes) 000956 1774875569.224577 MIRROR_SPLIT mirror 14 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1de:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31c37): Mirror 3: 0xaf978216 (stale) Mirror 4: 0xaf978216 (stale) Mirror 8: 0xaf978216 (stale) Mirror 9: 0xaf978216 (stale) Mirror 10: 0xaf978216 (stale) Mirror 11: 0xaf978216 (stale) Mirror 12: 0xaf978216 (stale) Mirror 15: 0xaf978216 (stale) Mirror 18: 0xaf978216 (stale) Mirror 19: 0xaf978216 (stale) Mirror 20: 0xaf978216 (stale) Mirror 21: 0xaf978216 (stale) Mirror 22: 0xaf978216 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31c37): Mirror 3: 0xaf978216 (stale) Mirror 4: 0xaf978216 (stale) Mirror 8: 0xaf978216 (stale) Mirror 9: 0xaf978216 (stale) Mirror 10: 0xaf978216 (stale) Mirror 11: 0xaf978216 (stale) Mirror 12: 0xaf978216 (stale) Mirror 15: 0xaf978216 (stale) Mirror 18: 0xaf978216 (stale) Mirror 19: 0xaf978216 (stale) Mirror 20: 0xaf978216 (stale) Mirror 21: 0xaf978216 (stale) Mirror 22: 0xaf978216 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31c37: skipped 000958 1774875570.091067 mapread 0x0009bf thru 0x00c73d (0xbd7f bytes) 000961 1774875570.114092 trunc from 0x031c37 to 0x0242de (0xd958 bytes) 000963 1774875570.136976 write 0x0131cd thru 0x0173d1 (0x4205 bytes) 000964 1774875570.146487 read 0x010f31 thru 0x019a5b (0x8b2b bytes) 000967 1774875570.148317 trunc from 0x0242df to 0x003151 (0x2118d bytes) 000968 1774875570.204403 write 0x0344af thru 0x03b02c (0x6b7e bytes) 000969 1774875570.206795 mapwrite 0x00e258 thru 0x013ceb (0x5a94 bytes) 000970 1774875570.295062 MIRROR_RESYNC 14 mirrors 000971 1774875572.217036 trunc from 0x03b02d to 0x038fd3 (0x2059 bytes) 000973 1774875572.302179 read 0x006ece thru 0x00bb66 (0x4c99 bytes) 000974 1774875572.310787 write 0x00ad42 thru 0x01aa14 (0xfcd3 bytes) 000975 1774875572.316074 write 0x009a35 thru 0x00f6bd (0x5c89 bytes) 000976 1774875572.317355 trunc from 0x038fd4 to 0x022499 (0x16b3a bytes) 000977 1774875572.337574 mapwrite 0x0241f5 thru 0x02b9be (0x77ca bytes) 000979 1774875572.437698 trunc from 0x02b9bf to 0x008d07 (0x22cb7 bytes) 000980 1774875572.456108 trunc from 0x008d08 to 0x0106ec (0x79e5 bytes) 000981 1774875572.482982 trunc from 0x0106ed to 0x01b3a7 (0xacbb bytes) 000982 1774875572.521654 MIRROR_RESYNC 14 mirrors 000983 1774875574.291538 MIRROR_RESYNC 14 mirrors 000984 1774875574.905813 MIRROR_RESYNC 14 mirrors 000985 1774875575.486409 trunc from 0x01b3a8 to 0x00b03b (0x1036c bytes) 000986 1774875575.577264 trunc from 0x00b03c to 0x009793 (0x18a8 bytes) 000988 1774875575.594844 write 0x02e17e thru 0x036e62 (0x8ce5 bytes) 000989 1774875575.600177 MIRROR_RESYNC 14 mirrors 000991 1774875577.499972 read 0x02cc4f thru 0x02e26a (0x161c bytes) 000992 1774875577.505459 read 0x03311f thru 0x036e62 (0x3d44 bytes) 000993 1774875577.518015 mapwrite 0x00548c thru 0x01085e (0xb3d3 bytes) 000995 1774875577.790976 mapread 0x03449c thru 0x036e62 (0x29c7 bytes) 000996 1774875577.812871 MIRROR_RESYNC 14 mirrors 000998 1774875579.882756 mapwrite 0x03b8f8 thru 0x03ffff (0x4708 bytes) 000999 1774875580.114150 write 0x006e5d thru 0x008b2a (0x1cce bytes) 001000 1774875580.129713 mapread 0x025eac thru 0x0288c0 (0x2a15 bytes) All operations completed A-OK! PASS 70a (114s) == sanity-flr test 71: check mirror extend parent fid ==== 08:59:45 (1774875585) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 08:59:48 (1774875588) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.207456 s, 10.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200854 s, 5.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 08:59:55 (1774875595) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre2 Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1205280 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3698304 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 6291456 .. Extending file size to 5459168 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7960320 .. Extending file size to 8877088 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7850848 .. Extending file size to 8732192 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8889792 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9063424 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 55538 55539 55540 55543 55544 done 192.168.206.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg637-client.virtnet /mnt/lustre2 (opts:) 192.168.206.137@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg637-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (77s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:01:13 (1774875673) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre2 Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3465024 .. Extending file size to 3675744 .. done Extending file size to 7539296 .. Extending file size to 8783456 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7544384 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7602688 .. Extending file size to 9106624 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 7714848 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9134432 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8199840 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 57077 57078 57081 57083 57085 192.168.206.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg637-client.virtnet /mnt/lustre2 (opts:) 192.168.206.137@tcp:/lustre /mnt/lustre3 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg637-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:02:29 (1774875749) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4224 2204288 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 25600 3743744 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 44032 7494656 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0987522 s, 21.2 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.331043 s, 6.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xf108e8ff (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg637-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:02:51 (1774875771) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.145358 s, 14.4 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 09:02:58 (1774875778) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22e:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c2:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c3:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2be:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:03:06 (1774875786) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.485305 s, 4.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2bf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (12s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:03:18 (1774875798) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.344838 s, 6.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c6:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c1:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c2:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:03:28 (1774875808) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.304595 s, 6.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } PASS 204c (30s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:03:58 (1774875838) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg637-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150369 s, 7.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:04:54 (1774875894) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg637-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.235205 s, 8.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:05:51 (1774875951) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg637-server mds-ost sync done. Creating new pool test_204f oleg637-server: Pool lustre.test_204f created Adding targets to pool oleg637-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg637-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.190787 s, 11.0 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg637-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg637-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg637-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:07:06 (1774876026) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:07:12 (1774876032) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:07:19 (1774876039) oleg637-server: Pool lustre.test_206 created Adding targets to pool oleg637-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg637-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg637-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg637-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg637-server: Pool lustre.test_206 destroyed PASS 206 (22s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:07:41 (1774876061) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794948 s, 13.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640728 s, 16.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xae749f67 Mirror 3: 0xae749f67 PASS 207 (9s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:07:51 (1774876071) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:07:54 (1774876074) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:07:57 (1774876077) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0623557 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2030000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2450000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2450000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2450000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x9a0000 Read this (4 bytes): BBBB 192.168.206.137@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg637-client.virtnet /mnt/lustre2 (opts:) PASS 209a (32s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:08:30 (1774876110) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00995145 s, 823 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:08:36 (1774876116) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423475 s, 24.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x253 lmm_fid: [0x200000404:0x253:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x253 lmm_fid: [0x200000404:0x253:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d6:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834515 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-03-30 09:08:37.000000000 -0400 Modify: 2026-03-30 09:08:37.000000000 -0400 Change: 2026-03-30 09:08:38.000000000 -0400 Birth: 2026-03-30 09:08:37.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:08:43 (1774876123) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:08:47 (1774876127) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.554977 s, 73.8 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0499963 s, 81.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Mar 30 09:08 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x255:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Mar 30 09:08 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:08:54 (1774876134) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:09:01 (1774876141) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1445 sec =========== 09:09:09 (1774876149) === sanity-flr: start cleanup 09:09:11 (1774876151) === === sanity-flr: finish cleanup 09:09:16 (1774876156) ===