-----============= acceptance-small: sanity-flr ============----- Wed Sep 30 08:42:07 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITY_FLR_ONLY='' SANITY_FLR_EXCEPT='' oleg637-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 08:42:25 (1790772145) === oleg637-client.virtnet: executing check_config_client /mnt/lustre oleg637-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg637-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff920043d01000.idle_timeout=debug osc.lustre-OST0001-osc-ffff920043d01000.idle_timeout=debug disable quota as required oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 08:42:57 (1790772177) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7617] == sanity-flr test 0a: lfs mirror create with -N option == 08:43:00 (1790772180) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 08:43:17 (1790772197) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 08:43:20 (1790772200) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 08:43:24 (1790772204) PASS 0d (32s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 08:43:56 (1790772236) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 08:44:00 (1790772240) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 08:44:03 (1790772243) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff920043d01000.stats=clear osc.lustre-OST0001-osc-ffff920043d01000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 08:44:11 (1790772251) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--ec-map] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 08:44:23 (1790772263) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 08:44:33 (1790772273) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 08:44:36 (1790772276) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 08:44:45 (1790772285) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 08:44:55 (1790772295) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 08:45:02 (1790772302) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 08:45:06 (1790772306) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 08:45:13 (1790772313) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 08:45:24 (1790772324) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 08:45:31 (1790772331) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.695913 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129652 s, 8.1 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 08:45:42 (1790772342) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206282 s, 50.8 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 08:45:51 (1790772351) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 08:46:02 (1790772362) Stopping /mnt/lustre-ost1 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff920043d01000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff920043d01000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg637-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff920043d01000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff920043d01000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg637-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (75s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 08:47:18 (1790772438) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.485462 s, 17.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff920043d01000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff920043d01000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 8388608 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg637-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (51s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 08:48:09 (1790772489) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff920043d01000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff920043d01000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg637-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff920043d01000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff920043d01000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg637-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (86s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 08:49:35 (1790772575) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 7.17997 s, 7.3 MB/s -rw-r--r-- 1 root root 52428800 Sep 30 08:49 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.039s user 0m0.000s sys 0m0.188s Stopping /mnt/lustre-ost1 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff920043d01000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff920043d01000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.323s user 0m0.000s sys 0m0.203s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg637-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg637-server oleg637-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff920043d01000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff920043d01000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.415s user 0m0.001s sys 0m0.196s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg637-server: oleg637-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg637-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg637-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff920043d01000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff920043d01000.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 08:51:13 (1790772673) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 08:51:17 (1790772677) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 08:51:20 (1790772680) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 08:51:23 (1790772683) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 08:51:31 (1790772691) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.813857 s, 15.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 08:51:44 (1790772704) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540247 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481697 s, 21.8 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 08:51:52 (1790772712) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 08:52:00 (1790772720) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00146052 s, 701 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00114131 s, 897 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00102776 s, 996 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000730857 s, 1.4 MB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 08:52:11 (1790772731) create test files with size 15760364 .. 1107b369f929538ce63f20b690ce564d - 60b60962a4159621bad8f2fdfd19046b - 3d6dd61b390f0e0bb0052f8104167dd7 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg637-client.virtnet /mnt/lustre (opts:) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 08:52:37 (1790772757) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 08:52:52 (1790772772) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 08:52:59 (1790772779) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130205 s, 8.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411422 s, 25.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 08:53:07 (1790772787) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164337 s, 12.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.398199 s, 10.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.174661 s, 12.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 08:53:22 (1790772802) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 08:53:26 (1790772806) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 08:53:29 (1790772809) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0284475 s, 36.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 08:53:37 (1790772817) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 08:53:41 (1790772821) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } split mirror_id 2 id 131073 error lfs split: file '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' has 1 component, cannot split make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 08:53:51 (1790772831) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.412721 s, 25.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg637-client.virtnet /mnt/lustre (opts:) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (12s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 08:54:03 (1790772843) llite.lustre-ffff920046b17000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.287666 s, 36.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg637-client.virtnet /mnt/lustre (opts:) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff920051653800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 08:54:18 (1790772858) error lfs split: create victim file failed: File exists PASS 44f (21s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 08:54:39 (1790772879) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 08:55:18 (1790772918) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 08:55:58 (1790772958) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 08:56:02 (1790772962) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.417153 s, 7.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.214045 s, 14.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x807fe12e (nosync,stale) Mirror 2: 0x807fe12e (nosync,stale) ** verify mirror 2 doesn't change original checksum: c4b1d9ec14d95abc0b53ae571b828a78 - mirror 1 checksum: 089d101b394843c80a9dbc72c90d11e2 - mirror 2 checksum: c4b1d9ec14d95abc0b53ae571b828a78 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-30T08:56:06-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x112:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-30T08:56:06-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x113:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: c4b1d9ec14d95abc0b53ae571b828a78 - mirror 1 checksum: 089d101b394843c80a9dbc72c90d11e2 - mirror 2 checksum: 089d101b394843c80a9dbc72c90d11e2 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-30T08:56:06-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x112:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-30T08:56:06-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x113:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 08:56:17 (1790772977) Failing mds1 on oleg637-server Stopping /mnt/lustre-mds1 (opts:) on oleg637-server 08:56:24 (1790772984) shut down facet: mds1 facet_host: oleg637-server facet_failover_host: oleg637-server Failover mds1 to oleg637-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg637-server: oleg637-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg637-client: oleg637-server: ssh exited with exit code 1 Started lustre-MDT0000 08:56:57 (1790773017) targets are mounted 08:56:57 (1790773017) facet_failover done oleg637-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268411 s, 39.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x69 lmm_fid: [0x200000404:0x69:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x114:0x0] } PASS 50A (63s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 08:57:21 (1790773041) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 08:57:25 (1790773045) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 08:57:28 (1790773048) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0203835 s, 804 kB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 08:57:37 (1790773057) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 08:57:40 (1790773060) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00948427 s, 432 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0782807 s, 52.3 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 08:57:48 (1790773068) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 08:58:13 (1790773093) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 08:58:42 (1790773122) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa7:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 08:59:06 (1790773146) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 08:59:30 (1790773170) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 08:59:38 (1790773178) mirror create pid 54999 mirror split pid 55000 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 54999 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 55000 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 09:00:45 (1790773245) Seed set to 3247 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000006 1790773247.730045 mapwrite 0x00642c thru 0x00adff (0x49d4 bytes) truncating to largest ever: 0x15fd6 000007 1790773247.835065 trunc from 0x00ae00 to 0x015fd5 (0xb1d6 bytes) 000008 1790773247.884346 write 0x031e92 thru 0x03779b (0x590a bytes) 000009 1790773247.891332 mapread 0x0033b4 thru 0x0115ac (0xe1f9 bytes) 000011 1790773247.910718 mapwrite 0x039e77 thru 0x03ffff (0x6189 bytes) 000012 1790773248.032646 mapwrite 0x02eb4f thru 0x0373dc (0x888e bytes) 000013 1790773248.102001 read 0x00b5f4 thru 0x00ee02 (0x380f bytes) 000014 1790773248.108361 MIRROR_EXTEND to 2 mirrors truncating to largest ever: 0x2dc60 000016 1790773249.115288 trunc from 0x040000 to 0x02dc5f (0x123a0 bytes) 000017 1790773249.235744 punch from 0x010467 to 0x017432 (0x6fcc bytes) 000019 1790773249.242772 mapwrite 0x0326b7 thru 0x03ffff (0xd949 bytes) 000023 1790773249.420705 punch from 0x016565 to 0x0226b2 (0xc14e bytes) 000024 1790773249.433127 trunc from 0x040000 to 0x00840f (0x37bf0 bytes) 000025 1790773249.460627 write 0x03e219 thru 0x03ffff (0x1de7 bytes) 000026 1790773249.470957 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x57d9771c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa677cb9e Mirror 3: 0xa677cb9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000027 1790773250.415997 mapread 0x028f87 thru 0x03185c (0x88d6 bytes) 000028 1790773250.458748 MIRROR_RESYNC 3 mirrors 000029 1790773251.607656 mapwrite 0x033ab3 thru 0x03c702 (0x8c50 bytes) 000030 1790773251.992612 read 0x02eed7 thru 0x03687e (0x79a8 bytes) 000033 1790773251.994968 mapread 0x01cd0d thru 0x02c9fb (0xfcef bytes) 000034 1790773252.041394 mapread 0x0187f4 thru 0x026e93 (0xe6a0 bytes) 000035 1790773252.043491 write 0x0055a4 thru 0x012876 (0xd2d3 bytes) 000036 1790773252.062961 read 0x007200 thru 0x0074b0 (0x2b1 bytes) 000037 1790773252.063841 read 0x02ce0a thru 0x0327a8 (0x599f bytes) 000039 1790773252.085643 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa677cb9e (stale) Mirror 3: 0xa677cb9e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1a233c24 Mirror 4: 0x1a233c24 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000040 1790773253.652292 punch from 0x0186d9 to 0x0257cb (0xd0f3 bytes) 000041 1790773253.898345 mapread 0x0028ad thru 0x00a7f9 (0x7f4d bytes) 000042 1790773253.924137 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa677cb9e (stale) Mirror 4: 0x1a233c24 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa677cb9e (stale) Mirror 4: 0x1a233c24 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000044 1790773254.502586 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa677cb9e (stale) Mirror 4: 0x1a233c24 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1a233c24 Mirror 5: 0x1a233c24 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000046 1790773256.234054 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa677cb9e (stale) Mirror 4: 0x1a233c24 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1a233c24 Mirror 5: 0x1a233c24 Mirror 6: 0x1a233c24 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000047 1790773257.747623 punch from 0x001d81 to 0x00ef61 (0xd1e1 bytes) 000048 1790773257.852610 read 0x01621b thru 0x01ba39 (0x581f bytes) 000049 1790773257.863497 trunc from 0x040000 to 0x023845 (0x1c7ba bytes) 000050 1790773257.884693 punch from 0x011b45 to 0x0205d2 (0xea8e bytes) 000051 1790773257.889704 trunc from 0x023846 to 0x01cca1 (0x6ba4 bytes) 000052 1790773257.914411 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1cca2): Mirror 3: 0x1fd844a7 (stale) Mirror 5: 0x97b586e4 (stale) Mirror 6: 0x97b586e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cca2): Mirror 3: 0x1fd844a7 (stale) Mirror 5: 0x97b586e4 (stale) Mirror 6: 0x97b586e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cca2: skipped 000053 1790773258.406739 read 0x015082 thru 0x01cca1 (0x7c20 bytes) 000056 1790773258.426965 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1cca2): Mirror 3: 0x1fd844a7 (stale) Mirror 5: 0x97b586e4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cca2): Mirror 3: 0x1fd844a7 (stale) Mirror 5: 0x97b586e4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cca2: skipped 000057 1790773258.845768 mapread 0x007ec3 thru 0x016291 (0xe3cf bytes) 000058 1790773258.882856 mapread 0x013a81 thru 0x01cca1 (0x9221 bytes) 000059 1790773258.888756 punch from 0x018337 to 0x01cca1 (0x496b bytes) 000060 1790773258.895499 mapread 0x005215 thru 0x0093cc (0x41b8 bytes) 000061 1790773258.896973 read 0x00cbcf thru 0x01b0dd (0xe50f bytes) 000062 1790773258.906821 MIRROR_RESYNC 3 mirrors 000063 1790773259.817953 MIRROR_RESYNC 3 mirrors 000064 1790773260.491906 write 0x0380a1 thru 0x03d9d6 (0x5936 bytes) 000065 1790773260.614266 mapwrite 0x01ecac thru 0x01eff6 (0x34b bytes) 000066 1790773260.701147 mapread 0x0017f7 thru 0x002e22 (0x162c bytes) 000067 1790773260.720709 MIRROR_RESYNC 3 mirrors 000068 1790773263.122612 write 0x026549 thru 0x026faf (0xa67 bytes) 000071 1790773263.215702 read 0x028acc thru 0x0386fa (0xfc2f bytes) 000072 1790773263.226878 mapwrite 0x0388ad thru 0x03fe81 (0x75d5 bytes) 000073 1790773263.429498 write 0x0024fc thru 0x00521f (0x2d24 bytes) 000075 1790773263.455258 read 0x033bc9 thru 0x03e3e4 (0xa81c bytes) 000078 1790773263.459309 read 0x025819 thru 0x031e6b (0xc653 bytes) 000081 1790773263.489468 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fe82): Mirror 5: 0x6094335b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe82): Mirror 5: 0x6094335b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe82: skipped 000083 1790773263.870502 write 0x02a16b thru 0x02af20 (0xdb6 bytes) 000084 1790773263.887743 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3fe82): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe82): Mirror 5: 0x6094335b (stale) CRC-32 checksum value for chunk [0, 0x3fe82): Mirror 1: 0xb1d72f7 Mirror 6: 0xb1d72f7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe82: skipped 000087 1790773266.286385 mapread 0x00f528 thru 0x01d6d7 (0xe1b0 bytes) 000088 1790773266.339642 mapread 0x030849 thru 0x03c194 (0xb94c bytes) 000089 1790773266.386934 read 0x029bf2 thru 0x033300 (0x970f bytes) 000090 1790773266.397404 mapread 0x009806 thru 0x00a5fd (0xdf8 bytes) 000092 1790773266.438300 mapwrite 0x03912a thru 0x03ffff (0x6ed6 bytes) 000095 1790773266.551161 trunc from 0x040000 to 0x011158 (0x2eea7 bytes) 000096 1790773266.580485 read 0x003275 thru 0x007a5d (0x47e9 bytes) 000097 1790773266.590977 mapwrite 0x007ad2 thru 0x00d2e9 (0x5818 bytes) 000098 1790773266.658148 mapwrite 0x01786a thru 0x0267ce (0xef65 bytes) 000099 1790773266.777251 write 0x038fb1 thru 0x03ffff (0x704f bytes) 000100 1790773266.785968 read 0x01fa51 thru 0x02e722 (0xecd2 bytes) 000104 1790773266.810288 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x2dfda4a2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x2dfda4a2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000105 1790773267.281827 punch from 0x01ab51 to 0x01e6fb (0x3bab bytes) 000106 1790773267.289049 read 0x0304d0 thru 0x0346e0 (0x4211 bytes) 000107 1790773267.303361 write 0x01b669 thru 0x025987 (0xa31f bytes) 000108 1790773267.321208 mapread 0x00cba9 thru 0x01394c (0x6da4 bytes) 000112 1790773267.354020 punch from 0x02236e to 0x025aac (0x373f bytes) 000113 1790773267.376895 mapread 0x005ad7 thru 0x009bdd (0x4107 bytes) 000114 1790773267.412203 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) 000117 1790773267.592680 MIRROR_EXTEND to 2 mirrors 000118 1790773269.038664 punch from 0x01ad35 to 0x01b63e (0x90a bytes) 000123 1790773269.142483 write 0x032d2a thru 0x039769 (0x6a40 bytes) 000124 1790773269.152773 write 0x033b77 thru 0x03b666 (0x7af0 bytes) 000127 1790773269.162811 mapwrite 0x02adee thru 0x032814 (0x7a27 bytes) 000131 1790773269.253836 mapwrite 0x02636a thru 0x03191b (0xb5b2 bytes) 000133 1790773269.306260 read 0x03f7e3 thru 0x03ffff (0x81d bytes) 000134 1790773269.316577 mapread 0x03aa8d thru 0x03ffff (0x5573 bytes) 000135 1790773269.324481 mapwrite 0x025c11 thru 0x03121d (0xb60d bytes) 000136 1790773269.414610 write 0x02d5b0 thru 0x0309e8 (0x3439 bytes) 000138 1790773269.422810 write 0x020077 thru 0x02a366 (0xa2f0 bytes) 000140 1790773269.440818 mapwrite 0x004c0d thru 0x00ec44 (0xa038 bytes) 000142 1790773269.588425 write 0x01eb86 thru 0x028518 (0x9993 bytes) 000143 1790773269.594879 read 0x01cfa8 thru 0x024296 (0x72ef bytes) 000144 1790773269.616802 write 0x00df41 thru 0x010b11 (0x2bd1 bytes) 000146 1790773269.618414 write 0x007e0d thru 0x0159e8 (0xdbdc bytes) 000147 1790773269.621226 mapread 0x004910 thru 0x0065b7 (0x1ca8 bytes) 000148 1790773269.636848 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x67a6bc21 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2a374abc Mirror 3: 0x2a374abc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000150 1790773271.249287 mapwrite 0x03fe34 thru 0x03ffff (0x1cc bytes) 000151 1790773271.508080 punch from 0x027884 to 0x0340eb (0xc868 bytes) 000153 1790773271.528250 MIRROR_RESYNC 3 mirrors 000157 1790773273.395196 mapwrite 0x03d198 thru 0x03ffff (0x2e68 bytes) 000161 1790773273.736500 trunc from 0x040000 to 0x02023b (0x1fdc4 bytes) 000163 1790773273.771179 read 0x01edc0 thru 0x02023b (0x147c bytes) 000164 1790773273.781398 mapwrite 0x01cce5 thru 0x02b56d (0xe889 bytes) 000165 1790773273.928429 punch from 0x01acc7 to 0x01bdda (0x1114 bytes) 000166 1790773273.963516 mapread 0x0074cb thru 0x007dd4 (0x90a bytes) 000168 1790773273.990839 punch from 0x020394 to 0x02b56d (0xb1da bytes) 000169 1790773274.000758 punch from 0x00b8d9 to 0x01a231 (0xe959 bytes) 000170 1790773274.013952 mapwrite 0x013108 thru 0x022793 (0xf68c bytes) 000171 1790773274.116432 mapread 0x025370 thru 0x02aac0 (0x5751 bytes) 000173 1790773274.129982 trunc from 0x02b56e to 0x0020fc (0x29471 bytes) 000174 1790773274.180947 punch from 0x001b92 to 0x0020fc (0x56b bytes) 000176 1790773274.200407 mapwrite 0x03b773 thru 0x03ffff (0x488d bytes) 000178 1790773274.347595 trunc from 0x040000 to 0x0215c4 (0x1ea3b bytes) truncating to largest ever: 0x370d0 000180 1790773274.375230 trunc from 0x0215c5 to 0x0370cf (0x15b0b bytes) 000181 1790773274.401629 write 0x0283eb thru 0x028ef1 (0xb07 bytes) 000182 1790773274.411701 read 0x02888c thru 0x035380 (0xcaf5 bytes) 000185 1790773274.432494 read 0x00d869 thru 0x010577 (0x2d0f bytes) 000186 1790773274.434587 mapread 0x0303e4 thru 0x0370cf (0x6cec bytes) 000187 1790773274.435712 read 0x02779b thru 0x036879 (0xf0df bytes) 000189 1790773274.437918 punch from 0x02c6ce to 0x0370cf (0xaa02 bytes) 000190 1790773274.463162 mapwrite 0x0342ab thru 0x03a349 (0x609f bytes) 000192 1790773274.639541 mapread 0x026a0f thru 0x027b77 (0x1169 bytes) 000195 1790773274.645689 mapread 0x009ff4 thru 0x01202a (0x8037 bytes) 000197 1790773274.647685 read 0x02e82b thru 0x036497 (0x7c6d bytes) 000198 1790773274.649032 punch from 0x00d50f to 0x013a21 (0x6513 bytes) 000199 1790773274.658018 trunc from 0x03a34a to 0x01710a (0x2323f bytes) 000200 1790773274.704602 mapread 0x000c46 thru 0x00bd8a (0xb145 bytes) 000201 1790773274.706774 punch from 0x00fd61 to 0x01710a (0x73aa bytes) 000205 1790773274.715458 mapwrite 0x026750 thru 0x02720e (0xabf bytes) 000208 1790773274.863666 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2720f): Mirror 3: 0x4d3eb46b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2720f): Mirror 3: 0x4d3eb46b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2720f: skipped truncating to largest ever: 0x3b2cb 000209 1790773275.204044 trunc from 0x02720f to 0x03b2ca (0x140bc bytes) 000210 1790773275.232578 trunc from 0x03b2cb to 0x0168a4 (0x24a26 bytes) 000211 1790773275.267875 write 0x0276e9 thru 0x031735 (0xa04d bytes) 000212 1790773275.269275 mapread 0x02cdf0 thru 0x031735 (0x4946 bytes) 000216 1790773275.272380 mapwrite 0x01521b thru 0x0229f0 (0xd7d6 bytes) 000217 1790773275.448343 punch from 0x00a73f to 0x01a71f (0xffe1 bytes) 000222 1790773275.459251 trunc from 0x031736 to 0x0118cb (0x1fe6a bytes) 000224 1790773275.502197 read 0x00299f thru 0x0118cb (0xef2d bytes) 000225 1790773275.516312 punch from 0x002e22 to 0x002fe7 (0x1c6 bytes) 000227 1790773275.521437 read 0x00f07c thru 0x0118cb (0x2850 bytes) 000228 1790773275.522085 punch from 0x00e82c to 0x0118cb (0x30a0 bytes) 000229 1790773275.533619 mapwrite 0x012727 thru 0x01ad17 (0x85f1 bytes) 000230 1790773275.646994 mapread 0x00ec78 thru 0x01ad17 (0xc0a0 bytes) 000231 1790773275.648172 write 0x022ea2 thru 0x025226 (0x2385 bytes) 000232 1790773275.650607 write 0x030aef thru 0x036640 (0x5b52 bytes) 000234 1790773275.652563 read 0x022af6 thru 0x026e2e (0x4339 bytes) 000235 1790773275.661325 read 0x0010f6 thru 0x007aec (0x69f7 bytes) 000238 1790773275.665246 mapwrite 0x00b21d thru 0x0173d7 (0xc1bb bytes) 000239 1790773275.720055 mapwrite 0x02fcbb thru 0x031bf6 (0x1f3c bytes) 000240 1790773275.762242 trunc from 0x036641 to 0x0307a6 (0x5e9a bytes) 000241 1790773275.787265 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) 000242 1790773275.954976 punch from 0x0004b1 to 0x00ad4f (0xa89f bytes) 000245 1790773275.967076 trunc from 0x0307a7 to 0x03264f (0x1ea9 bytes) 000247 1790773275.999168 trunc from 0x032650 to 0x0152ce (0x1d381 bytes) 000248 1790773276.035652 punch from 0x00a7e1 to 0x0152ce (0xaaee bytes) 000250 1790773276.042258 read 0x002303 thru 0x002a21 (0x71f bytes) 000252 1790773276.050120 punch from 0x00f98f to 0x0152ce (0x5940 bytes) 000253 1790773276.058117 read 0x011611 thru 0x0152ce (0x3cbe bytes) 000254 1790773276.077804 MIRROR_EXTEND to 2 mirrors 000255 1790773277.285814 mapread 0x0082e1 thru 0x00936f (0x108f bytes) 000256 1790773277.312758 read 0x000d7a thru 0x00498f (0x3c16 bytes) 000257 1790773277.331276 mapread 0x0029e0 thru 0x006162 (0x3783 bytes) 000258 1790773277.354041 write 0x01d08d thru 0x02c7a4 (0xf718 bytes) 000260 1790773277.439856 write 0x021dc0 thru 0x023f3c (0x217d bytes) 000261 1790773277.440918 trunc from 0x02c7a5 to 0x018e8e (0x13916 bytes) 000262 1790773277.473638 punch from 0x014e67 to 0x016b16 (0x1cb0 bytes) 000263 1790773277.493671 mapwrite 0x0303aa thru 0x0344a0 (0x40f7 bytes) 000264 1790773277.655456 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) 000266 1790773277.921879 trunc from 0x0344a1 to 0x02711d (0xd383 bytes) 000269 1790773277.949590 mapwrite 0x037b2a thru 0x03ffff (0x84d6 bytes) 000271 1790773278.044704 write 0x01cf87 thru 0x01da03 (0xa7d bytes) 000272 1790773278.055762 mapread 0x008ad6 thru 0x00fd47 (0x7272 bytes) 000273 1790773278.066979 punch from 0x014f38 to 0x017486 (0x254f bytes) 000274 1790773278.076004 read 0x006b64 thru 0x015b4d (0xefea bytes) 000277 1790773278.088355 read 0x03c71b thru 0x03ffff (0x38e5 bytes) 000278 1790773278.089333 mapwrite 0x0079e9 thru 0x014653 (0xcc6b bytes) 000280 1790773278.166910 mapwrite 0x031a63 thru 0x03c4a6 (0xaa44 bytes) 000281 1790773278.272023 punch from 0x02f791 to 0x03aba6 (0xb416 bytes) 000283 1790773278.279334 trunc from 0x040000 to 0x002bca (0x3d435 bytes) 000284 1790773278.331555 write 0x006fb7 thru 0x00d893 (0x68dd bytes) 000285 1790773278.332853 read 0x007856 thru 0x007f5f (0x70a bytes) 000290 1790773278.344856 mapread 0x00430d thru 0x00b53e (0x7232 bytes) 000292 1790773278.357475 trunc from 0x00d894 to 0x028c2f (0x1b39c bytes) 000293 1790773278.382873 read 0x010aef thru 0x01cef8 (0xc40a bytes) 000294 1790773278.413140 MIRROR_EXTEND to 2 mirrors 000295 1790773279.421781 write 0x00302e thru 0x010fbc (0xdf8f bytes) 000296 1790773279.514849 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) 000297 1790773279.691768 trunc from 0x028c30 to 0x005ab2 (0x2317d bytes) 000299 1790773279.726689 read 0x002117 thru 0x005ab2 (0x399c bytes) 000301 1790773279.735169 write 0x002906 thru 0x00747b (0x4b76 bytes) 000302 1790773279.737207 read 0x00094f thru 0x0013c4 (0xa76 bytes) 000303 1790773279.746467 mapread 0x007019 thru 0x00747b (0x463 bytes) 000304 1790773279.748425 mapwrite 0x014a42 thru 0x023663 (0xec22 bytes) 000305 1790773279.872438 trunc from 0x023664 to 0x0151da (0xe489 bytes) 000306 1790773279.888312 read 0x005d36 thru 0x011596 (0xb861 bytes) 000307 1790773279.889031 punch from 0x011b53 to 0x0151da (0x3688 bytes) 000311 1790773279.898622 punch from 0x007d9c to 0x00f8f0 (0x7b55 bytes) 000312 1790773279.906956 punch from 0x006681 to 0x0151da (0xeb5a bytes) 000315 1790773279.916776 mapread 0x006761 thru 0x0147eb (0xe08b bytes) 000316 1790773279.933884 write 0x003ab5 thru 0x00d628 (0x9b74 bytes) 000320 1790773279.940039 mapwrite 0x00ed19 thru 0x013eb6 (0x519e bytes) 000321 1790773280.010807 trunc from 0x0151db to 0x01ed89 (0x9baf bytes) 000322 1790773280.039823 read 0x004b40 thru 0x00e81f (0x9ce0 bytes) 000323 1790773280.041826 mapwrite 0x003d9d thru 0x006bf6 (0x2e5a bytes) 000324 1790773280.113522 mapwrite 0x01095a thru 0x01f525 (0xebcc bytes) 000325 1790773280.217747 write 0x030041 thru 0x032ad7 (0x2a97 bytes) 000326 1790773280.220426 mapwrite 0x01621a thru 0x01ff92 (0x9d79 bytes) 000327 1790773280.293087 read 0x017cb5 thru 0x025c33 (0xdf7f bytes) 000329 1790773280.309212 read 0x0255d9 thru 0x02b152 (0x5b7a bytes) 000330 1790773280.309865 mapread 0x003937 thru 0x008508 (0x4bd2 bytes) 000333 1790773280.315446 mapread 0x00bb7c thru 0x01b07f (0xf504 bytes) 000335 1790773280.318797 write 0x019459 thru 0x0222a1 (0x8e49 bytes) 000336 1790773280.327214 trunc from 0x032ad8 to 0x0265a5 (0xc532 bytes) 000339 1790773280.359001 read 0x009882 thru 0x009e2d (0x5ac bytes) 000341 1790773280.362637 punch from 0x022536 to 0x0265a5 (0x4070 bytes) 000344 1790773280.384807 punch from 0x0237d1 to 0x023ca0 (0x4d0 bytes) truncating to largest ever: 0x3c304 000346 1790773280.390893 trunc from 0x0265a6 to 0x03c303 (0x15d5e bytes) 000347 1790773280.416842 mapread 0x024101 thru 0x02b345 (0x7245 bytes) 000349 1790773280.452221 mapwrite 0x006b77 thru 0x0147af (0xdc39 bytes) 000350 1790773280.595031 read 0x0280ff thru 0x028c10 (0xb12 bytes) 000351 1790773280.595713 trunc from 0x03c304 to 0x019f97 (0x2236c bytes) truncating to largest ever: 0x3ee5b 000353 1790773280.618250 trunc from 0x019f98 to 0x03ee5a (0x24ec3 bytes) 000354 1790773280.641685 read 0x01f543 thru 0x02d9ae (0xe46c bytes) 000355 1790773280.655910 trunc from 0x03ee5b to 0x03619d (0x8cbd bytes) 000356 1790773280.674334 punch from 0x00f500 to 0x01217a (0x2c7b bytes) 000357 1790773280.679045 mapwrite 0x012873 thru 0x01d8e1 (0xb06f bytes) 000358 1790773280.755195 read 0x023128 thru 0x028cf7 (0x5bd0 bytes) 000359 1790773280.755885 read 0x017935 thru 0x0222c2 (0xa98e bytes) 000361 1790773280.757038 punch from 0x008e86 to 0x014a46 (0xbbc1 bytes) 000363 1790773280.772229 mapwrite 0x03dad9 thru 0x03e0ac (0x5d4 bytes) 000364 1790773280.889873 punch from 0x0206f1 to 0x0278fa (0x720a bytes) 000365 1790773280.896465 read 0x02ac6a thru 0x02b9de (0xd75 bytes) 000366 1790773280.897431 mapread 0x0017cb thru 0x00dd34 (0xc56a bytes) 000367 1790773280.904279 mapread 0x039965 thru 0x03e0ac (0x4748 bytes) 000369 1790773280.905939 punch from 0x030edc to 0x033e31 (0x2f56 bytes) 000370 1790773280.914871 mapread 0x0026e1 thru 0x00e13a (0xba5a bytes) 000371 1790773280.917193 punch from 0x02a45f to 0x02c3a2 (0x1f44 bytes) 000372 1790773280.927010 write 0x0334ef thru 0x035916 (0x2428 bytes) 000373 1790773280.945416 mapread 0x01467d thru 0x01d5b1 (0x8f35 bytes) 000374 1790773280.946474 trunc from 0x03e0ad to 0x039228 (0x4e84 bytes) 000375 1790773280.965333 write 0x00e846 thru 0x00e8ed (0xa8 bytes) 000376 1790773280.966991 mapwrite 0x00cc7c thru 0x00fa96 (0x2e1b bytes) 000377 1790773281.018792 trunc from 0x039229 to 0x00cc0e (0x2c61a bytes) 000378 1790773281.034868 write 0x013929 thru 0x01b02d (0x7705 bytes) 000379 1790773281.038070 mapread 0x0044a6 thru 0x006293 (0x1dee bytes) 000383 1790773281.042846 trunc from 0x01b02e to 0x002648 (0x189e5 bytes) 000384 1790773281.058183 mapwrite 0x038a47 thru 0x03ffff (0x75b9 bytes) 000387 1790773281.228118 trunc from 0x040000 to 0x028fa6 (0x17059 bytes) 000388 1790773281.263738 punch from 0x01a97c to 0x01acdd (0x362 bytes) 000389 1790773281.269406 punch from 0x00fbab to 0x012d6e (0x31c4 bytes) 000390 1790773281.274375 write 0x01beef thru 0x01cceb (0xdfd bytes) 000392 1790773281.283364 read 0x015df9 thru 0x0240c3 (0xe2cb bytes) 000394 1790773281.296062 read 0x019c73 thru 0x0243a1 (0xa72f bytes) 000397 1790773281.298703 write 0x0001ca thru 0x003b52 (0x3989 bytes) 000398 1790773281.299816 write 0x0184eb thru 0x027df3 (0xf909 bytes) 000399 1790773281.302673 write 0x01f9e5 thru 0x026cc8 (0x72e4 bytes) 000400 1790773281.308966 MIRROR_EXTEND to 2 mirrors 000402 1790773282.204084 write 0x03cc59 thru 0x03ffff (0x33a7 bytes) 000403 1790773282.310965 punch from 0x00e912 to 0x01a08a (0xb779 bytes) 000404 1790773282.321407 write 0x019b2e thru 0x0295ff (0xfad2 bytes) 000407 1790773282.334833 punch from 0x00cf59 to 0x0168ae (0x9956 bytes) 000408 1790773282.382369 read 0x015e22 thru 0x017a4e (0x1c2d bytes) 000409 1790773282.388832 mapwrite 0x0368ea thru 0x039a4e (0x3165 bytes) 000411 1790773282.448433 mapread 0x029a2c thru 0x038826 (0xedfb bytes) 000413 1790773282.460170 punch from 0x03a8a7 to 0x03c913 (0x206d bytes) 000414 1790773282.468184 punch from 0x024522 to 0x030c81 (0xc760 bytes) 000415 1790773282.474184 punch from 0x01afd9 to 0x022f5b (0x7f83 bytes) 000416 1790773282.480084 write 0x02ead5 thru 0x03b714 (0xcc40 bytes) 000417 1790773282.491452 punch from 0x02a612 to 0x036385 (0xbd74 bytes) 000420 1790773282.508470 write 0x0201a3 thru 0x02b70e (0xb56c bytes) 000421 1790773282.523361 mapwrite 0x00d888 thru 0x017215 (0x998e bytes) 000423 1790773282.617072 read 0x027421 thru 0x0301a1 (0x8d81 bytes) 000425 1790773282.651933 mapwrite 0x032cc3 thru 0x036b83 (0x3ec1 bytes) 000426 1790773282.747440 write 0x032c60 thru 0x03ffff (0xd3a0 bytes) 000427 1790773282.759770 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc01a6a35 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb4ec0b9c Mirror 3: 0xb4ec0b9c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000428 1790773284.186872 punch from 0x012a92 to 0x01d8a7 (0xae16 bytes) 000430 1790773284.278270 trunc from 0x040000 to 0x021ba3 (0x1e45c bytes) 000432 1790773284.304343 read 0x00487f thru 0x0091b6 (0x4938 bytes) 000433 1790773284.320390 MIRROR_RESYNC 3 mirrors 000434 1790773285.561622 MIRROR_RESYNC 3 mirrors 000435 1790773286.062829 MIRROR_RESYNC 3 mirrors 000436 1790773286.564964 mapread 0x00b45c thru 0x01a1af (0xed54 bytes) 000437 1790773286.601014 punch from 0x01adf4 to 0x021ba3 (0x6db0 bytes) 000439 1790773286.649253 punch from 0x017dbf to 0x021ba3 (0x9de5 bytes) 000440 1790773286.659156 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x21ba4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21ba4): Mirror 2: 0x7bdd6f16 (stale) Mirror 3: 0x7bdd6f16 (stale) CRC-32 checksum value for chunk [0, 0x21ba4): Mirror 1: 0x7abc9c30 Mirror 4: 0x7abc9c30 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21ba4: skipped 000441 1790773287.951463 trunc from 0x021ba4 to 0x01dd34 (0x3e6f bytes) 000442 1790773288.059844 write 0x00ed18 thru 0x01b4dc (0xc7c5 bytes) 000443 1790773288.073497 write 0x01701d thru 0x026d98 (0xfd7c bytes) 000445 1790773288.078141 mapread 0x00843e thru 0x00e9bb (0x657e bytes) 000446 1790773288.098610 mapread 0x01d988 thru 0x0249da (0x7053 bytes) 000447 1790773288.099621 mapwrite 0x026912 thru 0x02c58e (0x5c7d bytes) 000448 1790773288.279337 punch from 0x0136a6 to 0x0173b1 (0x3d0c bytes) 000450 1790773288.298252 mapread 0x012493 thru 0x01f896 (0xd404 bytes) 000451 1790773288.330781 trunc from 0x02c58f to 0x0046fa (0x27e94 bytes) 000452 1790773288.351461 mapread 0x000fb5 thru 0x0046fa (0x3746 bytes) 000453 1790773288.365252 trunc from 0x0046fb to 0x024d86 (0x2068c bytes) 000455 1790773288.404944 trunc from 0x024d87 to 0x00e19f (0x16be7 bytes) 000456 1790773288.433020 mapwrite 0x02e214 thru 0x034695 (0x6482 bytes) 000457 1790773288.588933 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x34696): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34696): Mirror 2: 0x852bcb00 (stale) Mirror 3: 0x852bcb00 (stale) Mirror 4: 0x82450d6c (stale) CRC-32 checksum value for chunk [0, 0x34696): Mirror 1: 0xfb694b51 Mirror 5: 0xfb694b51 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34696: skipped 000458 1790773290.006431 mapwrite 0x00c3c2 thru 0x00f423 (0x3062 bytes) 000459 1790773290.256931 mapread 0x025b72 thru 0x031e8e (0xc31d bytes) 000461 1790773290.285919 mapread 0x025a56 thru 0x02a273 (0x481e bytes) 000462 1790773290.293289 MIRROR_RESYNC 5 mirrors 000463 1790773292.187863 punch from 0x02c38d to 0x034695 (0x8309 bytes) 000464 1790773292.331493 trunc from 0x034696 to 0x03648e (0x1df9 bytes) 000466 1790773292.365419 read 0x024b83 thru 0x02566c (0xaea bytes) 000471 1790773292.373483 trunc from 0x03648f to 0x004f61 (0x3152d bytes) 000472 1790773292.397804 punch from 0x0009f7 to 0x004f61 (0x456b bytes) 000474 1790773292.404887 mapwrite 0x03e707 thru 0x03ffff (0x18f9 bytes) 000475 1790773292.481952 mapwrite 0x027420 thru 0x02745f (0x40 bytes) 000476 1790773292.543329 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7e1deda1 (stale) Mirror 3: 0x7e1deda1 (stale) Mirror 4: 0x7e1deda1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7e1deda1 (stale) Mirror 3: 0x7e1deda1 (stale) Mirror 4: 0x7e1deda1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000477 1790773293.013232 write 0x0315ce thru 0x03d212 (0xbc45 bytes) 000478 1790773293.027545 mapwrite 0x00bc07 thru 0x00e5aa (0x29a4 bytes) 000479 1790773293.141023 mapwrite 0x02218a thru 0x023d1d (0x1b94 bytes) 000480 1790773293.217963 punch from 0x0177a6 to 0x019a8d (0x22e8 bytes) 000481 1790773293.238713 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7e1deda1 (stale) Mirror 3: 0x7e1deda1 (stale) Mirror 4: 0x7e1deda1 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7d188c9e Mirror 5: 0x7d188c9e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000482 1790773294.983962 write 0x01f3a9 thru 0x027701 (0x8359 bytes) 000485 1790773295.138330 write 0x00aa90 thru 0x015cd1 (0xb242 bytes) 000486 1790773295.164377 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7e1deda1 (stale) Mirror 4: 0x7e1deda1 (stale) Mirror 5: 0x7d188c9e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7e1deda1 (stale) Mirror 4: 0x7e1deda1 (stale) Mirror 5: 0x7d188c9e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000487 1790773295.600618 mapwrite 0x015bf4 thru 0x02455f (0xe96c bytes) 000488 1790773295.724187 write 0x03f859 thru 0x03ffff (0x7a7 bytes) 000489 1790773295.744963 write 0x001e7c thru 0x004d26 (0x2eab bytes) 000490 1790773295.773867 mapwrite 0x00314b thru 0x00c689 (0x953f bytes) 000491 1790773295.874003 mapwrite 0x03d821 thru 0x03ffff (0x27df bytes) 000492 1790773295.958646 punch from 0x029869 to 0x036b95 (0xd32d bytes) 000494 1790773295.966185 read 0x00c86f thru 0x019bdf (0xd371 bytes) 000495 1790773295.972530 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7e1deda1 (stale) Mirror 5: 0x7d188c9e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7e1deda1 (stale) Mirror 5: 0x7d188c9e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000498 1790773296.348243 read 0x00c87f thru 0x0137c1 (0x6f43 bytes) 000502 1790773296.360271 mapread 0x01e459 thru 0x0244c6 (0x606e bytes) 000503 1790773296.396824 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7d188c9e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7d188c9e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000504 1790773296.727884 punch from 0x01fd6e to 0x02d6e3 (0xd976 bytes) 000505 1790773296.749610 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) 000508 1790773296.859953 trunc from 0x040000 to 0x01960f (0x269f0 bytes) 000510 1790773296.877214 mapread 0x0020af thru 0x00a994 (0x88e6 bytes) 000511 1790773296.894042 write 0x02729f thru 0x036151 (0xeeb3 bytes) 000512 1790773296.897248 mapwrite 0x0046d6 thru 0x004b2c (0x457 bytes) 000514 1790773296.984081 punch from 0x019b48 to 0x02182e (0x7ce7 bytes) 000516 1790773296.994726 mapread 0x0273a1 thru 0x031ae8 (0xa748 bytes) 000517 1790773296.996520 read 0x023e5c thru 0x0326c3 (0xe868 bytes) 000518 1790773297.007582 read 0x00e772 thru 0x011341 (0x2bd0 bytes) 000519 1790773297.037553 write 0x00cda8 thru 0x00e069 (0x12c2 bytes) 000520 1790773297.040373 read 0x02270a thru 0x03067a (0xdf71 bytes) 000521 1790773297.041945 mapread 0x016633 thru 0x01ee11 (0x87df bytes) 000522 1790773297.043411 mapwrite 0x0087e0 thru 0x013222 (0xaa43 bytes) 000524 1790773297.118732 write 0x018bfb thru 0x020b66 (0x7f6c bytes) 000526 1790773297.121263 read 0x02c9e4 thru 0x036151 (0x976e bytes) 000527 1790773297.122548 read 0x00b38e thru 0x011fbc (0x6c2f bytes) 000528 1790773297.123522 mapread 0x03153f thru 0x031c17 (0x6d9 bytes) 000531 1790773297.128919 mapread 0x021a98 thru 0x029948 (0x7eb1 bytes) 000532 1790773297.141557 MIRROR_EXTEND to 2 mirrors 000535 1790773298.082853 mapwrite 0x024c41 thru 0x0338aa (0xec6a bytes) 000538 1790773298.410352 punch from 0x031286 to 0x036151 (0x4ecc bytes) 000541 1790773298.417736 punch from 0x035a89 to 0x036151 (0x6c9 bytes) 000542 1790773298.424334 punch from 0x0151fe to 0x0236b6 (0xe4b9 bytes) 000543 1790773298.430456 mapread 0x0019e7 thru 0x002426 (0xa40 bytes) 000544 1790773298.440167 mapwrite 0x003997 thru 0x00f257 (0xb8c1 bytes) 000545 1790773298.489459 mapread 0x00f902 thru 0x014a0c (0x510b bytes) 000546 1790773298.500061 mapwrite 0x001581 thru 0x0114e1 (0xff61 bytes) 000547 1790773298.559648 mapwrite 0x00a0cd thru 0x00fee5 (0x5e19 bytes) 000548 1790773298.613671 trunc from 0x036152 to 0x03da5d (0x790c bytes) 000549 1790773298.664859 write 0x02fe17 thru 0x03a8e3 (0xaacd bytes) 000551 1790773298.672371 mapread 0x0038b4 thru 0x00d1fa (0x9947 bytes) 000552 1790773298.673842 mapwrite 0x029ade thru 0x02bbf0 (0x2113 bytes) 000553 1790773298.720594 mapread 0x01d50f thru 0x02d0c9 (0xfbbb bytes) 000554 1790773298.736415 MIRROR_RESYNC 2 mirrors 000555 1790773299.713344 punch from 0x03d19e to 0x03da5d (0x8c0 bytes) 000556 1790773299.787669 mapread 0x02b69b thru 0x03a638 (0xef9e bytes) 000557 1790773299.833321 mapread 0x02f755 thru 0x039e8c (0xa738 bytes) 000558 1790773299.838792 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) 000559 1790773299.988577 trunc from 0x03da5e to 0x03b5a0 (0x24bd bytes) 000560 1790773300.004440 punch from 0x01ba39 to 0x024d01 (0x92c9 bytes) 000562 1790773300.017948 mapread 0x026b30 thru 0x0364fd (0xf9ce bytes) 000563 1790773300.049135 punch from 0x028e06 to 0x02b8e6 (0x2ae1 bytes) 000564 1790773300.056273 write 0x02bdb0 thru 0x02cb39 (0xd8a bytes) 000565 1790773300.063133 mapwrite 0x00281e thru 0x00f20d (0xc9f0 bytes) 000567 1790773300.169841 trunc from 0x03b5a1 to 0x0167a3 (0x24dfd bytes) 000568 1790773300.198411 mapread 0x00e0f5 thru 0x0167a3 (0x86af bytes) 000569 1790773300.216669 read 0x00d450 thru 0x0167a3 (0x9354 bytes) 000571 1790773300.219754 mapread 0x012c99 thru 0x0167a3 (0x3b0b bytes) 000572 1790773300.220851 write 0x023560 thru 0x027d27 (0x47c8 bytes) 000575 1790773300.227198 punch from 0x01b9c3 to 0x027d27 (0xc365 bytes) 000577 1790773300.249079 trunc from 0x027d28 to 0x02e2ae (0x6587 bytes) 000578 1790773300.276968 mapwrite 0x0327fe thru 0x036383 (0x3b86 bytes) 000579 1790773300.368903 mapwrite 0x0160c6 thru 0x02342e (0xd369 bytes) 000580 1790773300.454827 trunc from 0x036384 to 0x027158 (0xf22b bytes) 000583 1790773300.483359 mapwrite 0x030f2a thru 0x03a336 (0x940d bytes) 000584 1790773300.595993 trunc from 0x03a337 to 0x019f09 (0x2042d bytes) 000587 1790773300.640158 trunc from 0x019f0a to 0x010032 (0x9ed7 bytes) 000589 1790773300.655976 mapread 0x00104f thru 0x00a2b4 (0x9266 bytes) 000591 1790773300.657875 write 0x032d98 thru 0x037892 (0x4afb bytes) 000592 1790773300.671631 MIRROR_EXTEND to 2 mirrors 000593 1790773301.985760 punch from 0x00d3c6 to 0x013ead (0x6ae8 bytes) 000596 1790773302.080924 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x37893): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37893): Mirror 2: 0xc95ac980 (stale) CRC-32 checksum value for chunk [0, 0x37893): Mirror 1: 0xe0696b0e Mirror 3: 0xe0696b0e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37893: skipped 000601 1790773303.477859 MIRROR_RESYNC 3 mirrors 000602 1790773304.374789 write 0x01ea0a thru 0x0298d6 (0xaecd bytes) 000603 1790773304.465527 write 0x011fcb thru 0x018ad9 (0x6b0f bytes) 000605 1790773304.483714 mapwrite 0x01bc7d thru 0x01c874 (0xbf8 bytes) 000606 1790773304.576485 mapread 0x028e27 thru 0x03735e (0xe538 bytes) 000607 1790773304.599823 write 0x030ddd thru 0x03b35d (0xa581 bytes) 000609 1790773304.602994 punch from 0x039597 to 0x03b35d (0x1dc7 bytes) 000610 1790773304.616688 punch from 0x0318a1 to 0x03b35d (0x9abd bytes) 000611 1790773304.622746 write 0x0343dc thru 0x03e71f (0xa344 bytes) 000612 1790773304.630558 read 0x01a0e1 thru 0x028164 (0xe084 bytes) 000613 1790773304.639523 mapwrite 0x00968d thru 0x00c3e8 (0x2d5c bytes) 000615 1790773304.709271 mapwrite 0x01dd03 thru 0x0227cc (0x4aca bytes) 000616 1790773304.791047 mapwrite 0x00b753 thru 0x00f2b9 (0x3b67 bytes) 000617 1790773304.875728 mapread 0x029be5 thru 0x02de72 (0x428e bytes) 000618 1790773304.878809 write 0x00814d thru 0x010c3f (0x8af3 bytes) 000619 1790773304.883227 write 0x013264 thru 0x01fb70 (0xc90d bytes) 000621 1790773304.888277 mapwrite 0x008019 thru 0x00ca51 (0x4a39 bytes) 000622 1790773304.993070 read 0x009605 thru 0x00e4bf (0x4ebb bytes) 000624 1790773304.994283 read 0x01766d thru 0x01bd35 (0x46c9 bytes) 000625 1790773305.000390 punch from 0x00652b to 0x008fb8 (0x2a8e bytes) 000626 1790773305.005504 trunc from 0x03e720 to 0x01396d (0x2adb2 bytes) 000627 1790773305.049805 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1396e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1396e): Mirror 2: 0xd38cab53 (stale) Mirror 3: 0xd38cab53 (stale) CRC-32 checksum value for chunk [0, 0x1396e): Mirror 1: 0xc180b57c Mirror 4: 0xc180b57c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1396e: skipped 000628 1790773306.353738 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1396e): Mirror 3: 0xd38cab53 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1396e): Mirror 3: 0xd38cab53 (stale) CRC-32 checksum value for chunk [0, 0x1396e): Mirror 1: 0xc180b57c Mirror 4: 0xc180b57c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1396e: skipped 000630 1790773307.052060 write 0x01fda0 thru 0x02af61 (0xb1c2 bytes) 000631 1790773307.136008 write 0x01fe94 thru 0x0261e8 (0x6355 bytes) 000632 1790773307.142582 write 0x012008 thru 0x013101 (0x10fa bytes) 000633 1790773307.151426 mapwrite 0x0261a9 thru 0x02e847 (0x869f bytes) 000635 1790773307.322899 mapread 0x00f0d8 thru 0x01c445 (0xd36e bytes) 000636 1790773307.349806 write 0x032ef9 thru 0x033f85 (0x108d bytes) 000638 1790773307.354532 read 0x009af6 thru 0x009b7b (0x86 bytes) 000639 1790773307.355901 punch from 0x02702a to 0x030e4e (0x9e25 bytes) 000641 1790773307.371309 write 0x0067c9 thru 0x009baa (0x33e2 bytes) 000642 1790773307.377919 mapwrite 0x010a7e thru 0x013f2e (0x34b1 bytes) 000643 1790773307.464520 punch from 0x0007c2 to 0x000e62 (0x6a1 bytes) 000644 1790773307.475178 read 0x000ba3 thru 0x00cb0e (0xbf6c bytes) 000645 1790773307.488307 punch from 0x008575 to 0x011427 (0x8eb3 bytes) 000646 1790773307.497696 mapwrite 0x024bf7 thru 0x0294c1 (0x48cb bytes) 000647 1790773307.608199 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33f86): Mirror 3: 0x2c6063c0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33f86): Mirror 3: 0x2c6063c0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33f86: skipped 000648 1790773307.992407 trunc from 0x033f86 to 0x036785 (0x2800 bytes) 000653 1790773308.015999 write 0x036b24 thru 0x0376a2 (0xb7f bytes) 000654 1790773308.021400 write 0x03c373 thru 0x03ffff (0x3c8d bytes) 000655 1790773308.023155 punch from 0x028a84 to 0x0311fa (0x8777 bytes) 000657 1790773308.043236 mapread 0x02eb6b thru 0x02fee3 (0x1379 bytes) 000658 1790773308.053711 write 0x017a7e thru 0x01d44c (0x59cf bytes) 000659 1790773308.067490 punch from 0x028c87 to 0x02c00f (0x3389 bytes) 000660 1790773308.085873 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) 000661 1790773308.247054 mapread 0x016fe7 thru 0x01e42a (0x7444 bytes) 000663 1790773308.274220 mapread 0x013460 thru 0x01ab96 (0x7737 bytes) 000664 1790773308.276004 mapwrite 0x02f5eb thru 0x02fcff (0x715 bytes) 000665 1790773308.327041 write 0x0226d0 thru 0x027823 (0x5154 bytes) 000667 1790773308.338611 read 0x019a18 thru 0x0248f8 (0xaee1 bytes) 000670 1790773308.351328 MIRROR_EXTEND to 2 mirrors 000671 1790773309.702367 trunc from 0x040000 to 0x038ba3 (0x745c bytes) 000672 1790773309.828566 read 0x02bc9e thru 0x0324f7 (0x685a bytes) 000673 1790773309.844125 punch from 0x01220b to 0x01520a (0x3000 bytes) 000675 1790773309.859936 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) 000678 1790773310.071659 read 0x009831 thru 0x013ed8 (0xa6a8 bytes) 000680 1790773310.099990 mapwrite 0x02d4e9 thru 0x031a7f (0x4597 bytes) 000683 1790773310.260009 punch from 0x00eeb5 to 0x0138ae (0x49fa bytes) 000684 1790773310.273323 punch from 0x0362bf to 0x038ba3 (0x28e5 bytes) 000685 1790773310.278997 mapread 0x038884 thru 0x038ba3 (0x320 bytes) 000686 1790773310.293863 write 0x03df7b thru 0x03ffff (0x2085 bytes) 000688 1790773310.301605 mapwrite 0x03a266 thru 0x03ffff (0x5d9a bytes) 000691 1790773310.407223 write 0x03e26f thru 0x03ffff (0x1d91 bytes) 000693 1790773310.414627 mapread 0x0082e4 thru 0x015b89 (0xd8a6 bytes) 000694 1790773310.441400 MIRROR_EXTEND to 2 mirrors 000696 1790773311.889414 mapwrite 0x0169dd thru 0x02643b (0xfa5f bytes) 000698 1790773312.185069 trunc from 0x040000 to 0x029f9d (0x16062 bytes) 000700 1790773312.210812 trunc from 0x029f9e to 0x0367f8 (0xc85b bytes) 000702 1790773312.262304 punch from 0x02dce4 to 0x0367f8 (0x8b15 bytes) 000703 1790773312.281056 trunc from 0x0367f9 to 0x00678d (0x3006b bytes) 000704 1790773312.299531 mapwrite 0x029919 thru 0x031f34 (0x861c bytes) 000706 1790773312.393766 punch from 0x01a1d6 to 0x025568 (0xb393 bytes) 000707 1790773312.404270 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) 000708 1790773312.562164 trunc from 0x031f35 to 0x02777d (0xa7b7 bytes) 000711 1790773312.583563 read 0x0077f2 thru 0x01551c (0xdd2b bytes) 000712 1790773312.593342 mapwrite 0x02c402 thru 0x03a78c (0xe38b bytes) 000714 1790773312.726005 write 0x002499 thru 0x00ae99 (0x8a01 bytes) 000715 1790773312.742391 trunc from 0x03a78d to 0x01c982 (0x1de0a bytes) 000716 1790773312.769337 MIRROR_EXTEND to 2 mirrors 000717 1790773313.914607 mapwrite 0x002b57 thru 0x010646 (0xdaf0 bytes) 000718 1790773314.154380 trunc from 0x01c983 to 0x02e3ba (0x11a38 bytes) 000719 1790773314.185193 punch from 0x00862d to 0x014870 (0xc244 bytes) 000722 1790773314.198176 trunc from 0x02e3bb to 0x0026ba (0x2bd00 bytes) 000725 1790773314.226760 trunc from 0x0026bb to 0x011300 (0xec46 bytes) 000726 1790773314.251574 read 0x00739d thru 0x011300 (0x9f64 bytes) 000728 1790773314.261045 mapwrite 0x035ccc thru 0x03ffff (0xa334 bytes) 000729 1790773314.363201 MIRROR_RESYNC 2 mirrors 000733 1790773315.612038 read 0x0215e1 thru 0x027d88 (0x67a8 bytes) 000736 1790773315.638146 trunc from 0x040000 to 0x01b432 (0x24bcd bytes) 000737 1790773315.718679 read 0x00fc2d thru 0x016928 (0x6cfc bytes) 000738 1790773315.735078 read 0x00df7f thru 0x00ee14 (0xe96 bytes) 000739 1790773315.735847 mapread 0x00ef7e thru 0x01421b (0x529e bytes) 000740 1790773315.737644 mapread 0x003153 thru 0x00d76d (0xa61b bytes) 000742 1790773315.742760 write 0x0184bc thru 0x01e58d (0x60d2 bytes) 000744 1790773315.749474 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) 000745 1790773315.891586 mapwrite 0x00002a thru 0x002296 (0x226d bytes) 000746 1790773315.949924 punch from 0x019c14 to 0x01e1b4 (0x45a1 bytes) 000747 1790773315.954720 trunc from 0x01e58e to 0x00a795 (0x13df8 bytes) 000749 1790773315.972547 mapwrite 0x001715 thru 0x005a64 (0x4350 bytes) 000750 1790773316.021929 read 0x001966 thru 0x00a795 (0x8e30 bytes) 000751 1790773316.022690 mapwrite 0x0048e4 thru 0x00ffb5 (0xb6d2 bytes) 000752 1790773316.134026 punch from 0x006d65 to 0x00ffb5 (0x9251 bytes) 000753 1790773316.140934 write 0x03ba7f thru 0x03ffff (0x4581 bytes) 000754 1790773316.143447 read 0x01f45c thru 0x02913a (0x9cdf bytes) 000758 1790773316.166718 read 0x029670 thru 0x02bbd3 (0x2564 bytes) 000759 1790773316.171922 write 0x02d691 thru 0x03a383 (0xccf3 bytes) 000762 1790773316.188504 mapread 0x00d521 thru 0x013d78 (0x6858 bytes) 000763 1790773316.194046 MIRROR_EXTEND to 2 mirrors 000764 1790773317.076689 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) 000766 1790773317.196465 punch from 0x02735f to 0x027a2c (0x6ce bytes) 000767 1790773317.210956 mapwrite 0x01870d thru 0x01ea9a (0x638e bytes) 000768 1790773317.286166 mapwrite 0x0143d6 thru 0x01c402 (0x802d bytes) 000770 1790773317.360396 mapwrite 0x01299b thru 0x01318e (0x7f4 bytes) 000771 1790773317.423079 mapwrite 0x00740b thru 0x016c0a (0xf800 bytes) 000772 1790773317.530941 punch from 0x0244f7 to 0x024be5 (0x6ef bytes) 000774 1790773317.548855 read 0x009368 thru 0x00ae79 (0x1b12 bytes) 000777 1790773317.554612 read 0x02045c thru 0x0255dd (0x5182 bytes) 000782 1790773317.566548 mapwrite 0x0018cb thru 0x009c55 (0x838b bytes) 000785 1790773317.621195 trunc from 0x040000 to 0x02ab71 (0x1548e bytes) 000786 1790773317.674901 mapread 0x026232 thru 0x02ab71 (0x4940 bytes) 000787 1790773317.675977 write 0x039ebe thru 0x03ffff (0x6142 bytes) 000789 1790773317.682692 mapread 0x031381 thru 0x034a01 (0x3681 bytes) 000790 1790773317.713693 punch from 0x008467 to 0x0126c8 (0xa262 bytes) 000791 1790773317.730899 punch from 0x0143bb to 0x0189ff (0x4645 bytes) 000792 1790773317.736213 mapwrite 0x02b690 thru 0x03a012 (0xe983 bytes) 000795 1790773317.880527 mapwrite 0x01951a thru 0x01fb00 (0x65e7 bytes) 000799 1790773317.975979 mapwrite 0x030d63 thru 0x031b55 (0xdf3 bytes) 000800 1790773318.028982 punch from 0x0264d3 to 0x02bdfa (0x5928 bytes) 000803 1790773318.040664 mapread 0x0096c5 thru 0x014488 (0xadc4 bytes) 000805 1790773318.114255 mapwrite 0x015f0f thru 0x01727e (0x1370 bytes) 000806 1790773318.206697 mapwrite 0x02ca38 thru 0x0333d1 (0x699a bytes) 000808 1790773318.279226 write 0x025b9b thru 0x02797b (0x1de1 bytes) 000809 1790773318.287439 punch from 0x0088ca to 0x0098bc (0xff3 bytes) 000810 1790773318.312248 punch from 0x02da11 to 0x03506f (0x765f bytes) 000812 1790773318.330690 write 0x008a5c thru 0x01050a (0x7aaf bytes) 000814 1790773318.354123 read 0x03c9a8 thru 0x03ffff (0x3658 bytes) 000816 1790773318.358074 punch from 0x00ddde to 0x01a336 (0xc559 bytes) 000818 1790773318.400392 write 0x012c16 thru 0x01f9d4 (0xcdbf bytes) 000820 1790773318.444070 write 0x01d4b5 thru 0x0209c3 (0x350f bytes) 000822 1790773318.447954 trunc from 0x040000 to 0x027980 (0x1867f bytes) 000823 1790773318.477011 punch from 0x018aa9 to 0x01e2e0 (0x5838 bytes) 000824 1790773318.505400 trunc from 0x027981 to 0x014831 (0x1314f bytes) 000826 1790773318.533564 punch from 0x0022f7 to 0x00e747 (0xc451 bytes) 000827 1790773318.539732 mapwrite 0x0258dc thru 0x0281a4 (0x28c9 bytes) 000828 1790773318.725926 mapwrite 0x01207f thru 0x020d1a (0xec9c bytes) 000830 1790773318.910487 mapread 0x009580 thru 0x014c1b (0xb69c bytes) 000834 1790773318.940406 trunc from 0x0281a5 to 0x03db57 (0x159b3 bytes) 000835 1790773318.956575 mapread 0x031e70 thru 0x03a681 (0x8812 bytes) 000839 1790773319.007541 mapread 0x037d51 thru 0x039a1b (0x1ccb bytes) 000840 1790773319.009261 punch from 0x00b963 to 0x019291 (0xd92f bytes) 000841 1790773319.016563 mapwrite 0x02f943 thru 0x036a90 (0x714e bytes) 000844 1790773319.077662 punch from 0x02e818 to 0x0355ce (0x6db7 bytes) 000845 1790773319.083185 trunc from 0x03db58 to 0x02e646 (0xf511 bytes) 000846 1790773319.108395 mapread 0x014ec5 thru 0x01ff2a (0xb066 bytes) 000847 1790773319.156732 trunc from 0x02e647 to 0x01bf67 (0x126df bytes) 000848 1790773319.209870 MIRROR_EXTEND to 2 mirrors 000849 1790773320.170564 punch from 0x012fd6 to 0x01bf67 (0x8f92 bytes) 000850 1790773320.248554 read 0x01b2cb thru 0x01bf67 (0xc9d bytes) 000851 1790773320.260690 punch from 0x00ec2c to 0x0120ad (0x3482 bytes) 000852 1790773320.265698 read 0x0168d1 thru 0x01bdc1 (0x54f1 bytes) 000853 1790773320.290681 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) 000854 1790773320.470938 mapread 0x009041 thru 0x014b3e (0xbafe bytes) 000857 1790773320.514782 trunc from 0x01bf68 to 0x00a745 (0x11822 bytes) 000858 1790773320.530645 punch from 0x005441 to 0x006557 (0x1117 bytes) 000859 1790773320.535942 punch from 0x00753c to 0x00a745 (0x320a bytes) 000860 1790773320.540169 mapwrite 0x0021ae thru 0x002ed6 (0xd29 bytes) 000861 1790773320.605365 write 0x02f7bf thru 0x036b1b (0x735d bytes) 000863 1790773320.607064 mapwrite 0x03ce7d thru 0x03ffff (0x3183 bytes) 000864 1790773320.735475 mapread 0x012025 thru 0x019005 (0x6fe1 bytes) 000865 1790773320.751259 trunc from 0x040000 to 0x024a89 (0x1b576 bytes) 000867 1790773320.789152 mapwrite 0x02a0f2 thru 0x02d3e3 (0x32f2 bytes) 000869 1790773320.946902 read 0x01155b thru 0x011643 (0xe9 bytes) 000870 1790773320.948939 trunc from 0x02d3e4 to 0x02c96d (0xa76 bytes) 000872 1790773320.983665 mapread 0x023e07 thru 0x02c96d (0x8b67 bytes) 000873 1790773320.984696 write 0x0186ba thru 0x024fc2 (0xc909 bytes) 000874 1790773321.005620 MIRROR_EXTEND to 2 mirrors 000875 1790773322.009139 punch from 0x02bab8 to 0x02c96d (0xeb6 bytes) 000876 1790773322.094982 punch from 0x008cd5 to 0x01015a (0x7486 bytes) 000877 1790773322.107512 write 0x01bfc1 thru 0x020426 (0x4466 bytes) 000879 1790773322.132241 mapread 0x017caf thru 0x01c365 (0x46b7 bytes) 000881 1790773322.177362 read 0x013c62 thru 0x020acd (0xce6c bytes) 000882 1790773322.183941 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) 000883 1790773322.433292 write 0x03054f thru 0x03286a (0x231c bytes) 000885 1790773322.436921 mapwrite 0x015d2d thru 0x01fae8 (0x9dbc bytes) 000888 1790773322.515641 write 0x015734 thru 0x024780 (0xf04d bytes) 000889 1790773322.523557 mapread 0x02694a thru 0x032741 (0xbdf8 bytes) 000891 1790773322.538929 mapwrite 0x001d41 thru 0x00f1a5 (0xd465 bytes) 000893 1790773322.609531 read 0x03080f thru 0x03286a (0x205c bytes) 000894 1790773322.610152 trunc from 0x03286b to 0x00ac92 (0x27bd8 bytes) 000897 1790773322.625605 mapwrite 0x028f3e thru 0x031f13 (0x8fd6 bytes) 000898 1790773322.704865 read 0x02c299 thru 0x02d44d (0x11b5 bytes) 000902 1790773322.707688 trunc from 0x031f14 to 0x012f65 (0x1efae bytes) 000903 1790773322.723535 read 0x0105d9 thru 0x012a96 (0x24be bytes) 000904 1790773322.739454 MIRROR_EXTEND to 2 mirrors 000905 1790773323.532935 MIRROR_RESYNC 2 mirrors 000906 1790773324.023280 mapwrite 0x03ee27 thru 0x03ffff (0x11d9 bytes) 000907 1790773324.212818 read 0x00c1f5 thru 0x01ba25 (0xf831 bytes) 000908 1790773324.221723 mapwrite 0x00a3af thru 0x00ce9d (0x2aef bytes) 000909 1790773324.270938 mapwrite 0x013975 thru 0x0142c4 (0x950 bytes) 000911 1790773324.312818 write 0x013923 thru 0x01ec7c (0xb35a bytes) 000912 1790773324.320250 punch from 0x003a69 to 0x008d25 (0x52bd bytes) 000913 1790773324.339993 trunc from 0x040000 to 0x0063d8 (0x39c27 bytes) 000915 1790773324.362131 read 0x0013f0 thru 0x002c98 (0x18a9 bytes) 000917 1790773324.371284 trunc from 0x0063d9 to 0x0300ce (0x29cf6 bytes) 000918 1790773324.384693 trunc from 0x0300cf to 0x02b0d2 (0x4ffc bytes) 000919 1790773324.406699 trunc from 0x02b0d3 to 0x036619 (0xb547 bytes) 000920 1790773324.425517 write 0x00b53b thru 0x013122 (0x7be8 bytes) 000921 1790773324.438866 mapwrite 0x037764 thru 0x03b6e5 (0x3f82 bytes) 000922 1790773324.518082 read 0x03a1ac thru 0x03b6e5 (0x153a bytes) 000923 1790773324.519022 punch from 0x016728 to 0x01f859 (0x9132 bytes) 000924 1790773324.525325 mapwrite 0x036500 thru 0x03c559 (0x605a bytes) 000925 1790773324.633133 write 0x00a897 thru 0x00d7f8 (0x2f62 bytes) 000927 1790773324.634842 read 0x00f991 thru 0x01c862 (0xced2 bytes) 000928 1790773324.636936 mapwrite 0x03fe73 thru 0x03ffff (0x18d bytes) 000929 1790773324.758588 mapwrite 0x00b9ac thru 0x0150e9 (0x973e bytes) 000930 1790773324.839040 mapread 0x00db12 thru 0x0169fa (0x8ee9 bytes) 000934 1790773324.842172 mapread 0x018509 thru 0x023203 (0xacfb bytes) 000935 1790773324.843617 mapwrite 0x0054b1 thru 0x00d81e (0x836e bytes) 000936 1790773324.904566 mapread 0x03b707 thru 0x03ffff (0x48f9 bytes) 000937 1790773324.905888 mapwrite 0x02d290 thru 0x03330b (0x607c bytes) 000938 1790773324.985029 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x210:0x0]: Invalid argument (22) 000939 1790773325.251569 mapread 0x00a6e5 thru 0x0155fa (0xaf16 bytes) 000940 1790773325.313063 write 0x02299f thru 0x0302ba (0xd91c bytes) 000941 1790773325.345639 write 0x00a459 thru 0x015562 (0xb10a bytes) 000943 1790773325.361326 mapwrite 0x01998d thru 0x019f4f (0x5c3 bytes) 000948 1790773325.446962 trunc from 0x040000 to 0x00e388 (0x31c77 bytes) 000949 1790773325.494962 mapread 0x008b0d thru 0x00e388 (0x587c bytes) 000951 1790773325.498611 mapwrite 0x019fbc thru 0x01ac5b (0xca0 bytes) 000952 1790773325.587348 mapwrite 0x0303f8 thru 0x032800 (0x2409 bytes) 000955 1790773325.740819 mapread 0x012e5b thru 0x01e7d5 (0xb97b bytes) 000957 1790773325.758960 trunc from 0x032801 to 0x03c887 (0xa087 bytes) 000958 1790773325.789525 mapread 0x00b4ee thru 0x01aac6 (0xf5d9 bytes) 000959 1790773325.799982 read 0x035f49 thru 0x0379ba (0x1a72 bytes) 000960 1790773325.808571 mapread 0x029518 thru 0x035764 (0xc24d bytes) 000961 1790773325.820011 read 0x002983 thru 0x0055f7 (0x2c75 bytes) 000963 1790773325.839799 MIRROR_EXTEND to 2 mirrors 000964 1790773327.219597 mapwrite 0x03183f thru 0x032267 (0xa29 bytes) 000965 1790773327.394048 trunc from 0x03c888 to 0x03b519 (0x136e bytes) 000967 1790773327.427039 mapread 0x016372 thru 0x01879d (0x242c bytes) 000969 1790773327.446508 read 0x015af1 thru 0x01e63c (0x8b4c bytes) 000970 1790773327.455308 MIRROR_RESYNC 2 mirrors 000971 1790773328.747159 read 0x0062ce thru 0x0162a8 (0xffdb bytes) 000972 1790773328.761469 mapread 0x01da3f thru 0x02ae87 (0xd449 bytes) 000973 1790773328.797083 punch from 0x01673a to 0x017df2 (0x16b9 bytes) 000974 1790773328.861063 write 0x0018f2 thru 0x00bfbe (0xa6cd bytes) 000975 1790773328.876020 read 0x01dc49 thru 0x02d333 (0xf6eb bytes) 000976 1790773328.894085 write 0x0000a7 thru 0x00caf1 (0xca4b bytes) 000977 1790773328.898502 punch from 0x01ab52 to 0x02058e (0x5a3d bytes) 000980 1790773328.929640 mapwrite 0x028c31 thru 0x02a1c8 (0x1598 bytes) 000981 1790773328.994367 MIRROR_RESYNC 2 mirrors 000982 1790773330.259424 mapread 0x00aff0 thru 0x00dac4 (0x2ad5 bytes) 000983 1790773330.285517 mapread 0x035f77 thru 0x03b519 (0x55a3 bytes) 000985 1790773330.300491 mapwrite 0x00be66 thru 0x0173c3 (0xb55e bytes) 000987 1790773330.486565 write 0x02a560 thru 0x03553f (0xafe0 bytes) 000991 1790773330.508402 MIRROR_RESYNC 2 mirrors 000993 1790773331.654169 mapwrite 0x010f28 thru 0x01811b (0x71f4 bytes) 000995 1790773331.812063 punch from 0x017027 to 0x01e91d (0x78f7 bytes) 000996 1790773331.820534 punch from 0x000df1 to 0x004cd8 (0x3ee8 bytes) 000997 1790773331.830817 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b51a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b51a): Mirror 2: 0x755f7747 (stale) CRC-32 checksum value for chunk [0, 0x3b51a): Mirror 1: 0xe5a5e23c Mirror 3: 0xe5a5e23c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b51a: skipped 000998 1790773333.487250 mapwrite 0x00a48b thru 0x00fa74 (0x55ea bytes) 000999 1790773333.817697 trunc from 0x03b51a to 0x002453 (0x390c6 bytes) 001000 1790773333.837953 write 0x022b4c thru 0x02f1eb (0xc6a0 bytes) All operations completed A-OK! PASS 70a (94s) == sanity-flr test 71: check mirror extend parent fid ==== 09:02:19 (1790773339) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 09:02:22 (1790773342) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.328264 s, 6.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284124 s, 3.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32b:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x32a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 09:02:32 (1790773352) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre2 Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3010752 .. Extending file size to 5752800 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8803744 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8510176 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8656992 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8797696 .. failed Extending file size to 9026688 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8899840 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58102 58103 58106 58107 58108 done 192.168.206.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg637-client.virtnet /mnt/lustre2 (opts:) 192.168.206.137@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg637-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 09:03:51 (1790773431) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre2 Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2054336 .. Extending file size to 2066432 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2332576 .. failed Extending file size to 5986944 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3356160 .. Extending file size to 3666848 .. Extending file size to 7751264 .. Extending file size to 8017504 .. Extending file size to 8083104 .. Extending file size to 6197536 .. Extending file size to 7588832 .. Extending file size to 8585696 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8221504 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8359840 .. Extending file size to 9284064 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9212384 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59712 59713 59714 59716 59717 done 192.168.206.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg637-client.virtnet /mnt/lustre2 (opts:) 192.168.206.137@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg637-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (77s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 09:05:08 (1790773508) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0966371 s, 21.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.290362 s, 7.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x6de03935 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg637-client.virtnet /mnt/lustre2 (opts:) PASS 200c (14s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 09:05:24 (1790773524) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0827643 s, 25.3 MB/s PASS 202 (5s) == sanity-flr test 203: mirror file preserve mirror ID === 09:05:29 (1790773529) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x257:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x341:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x341:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x342:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x33d:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 09:05:35 (1790773535) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132949 s, 15.8 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x343:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x33f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 09:05:43 (1790773543) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.168541 s, 12.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x344:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x345:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x340:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x344:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x345:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x340:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x341:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 09:05:50 (1790773550) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.257893 s, 8.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x346:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x342:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x346:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x342:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x343:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 09:06:22 (1790773582) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg637-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146639 s, 7.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x347:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x345:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x344:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x347:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x345:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x344:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x348:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 09:07:17 (1790773637) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg637-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x346:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x347:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.313686 s, 6.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x346:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x347:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x349:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x346:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x347:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x349:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (55s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 09:08:12 (1790773692) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg637-server mds-ost sync done. Creating new pool test_204f oleg637-server: Pool lustre.test_204f created Adding targets to pool oleg637-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg637-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.403023 s, 5.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x348:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x348:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg637-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg637-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg637-server: Pool lustre.test_204f destroyed PASS 204f (81s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 09:09:34 (1790773774) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 09:09:42 (1790773782) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 09:09:49 (1790773789) oleg637-server: Pool lustre.test_206 created Adding targets to pool oleg637-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg637-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg637-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg637-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg637-server: Pool lustre.test_206 destroyed PASS 206 (24s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 09:10:13 (1790773813) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745194 s, 14.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249087 s, 42.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xb96736a5 Mirror 3: 0xb96736a5 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 09:10:23 (1790773823) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 09:10:27 (1790773827) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 09:10:30 (1790773830) Starting client: oleg637-client.virtnet: -o user_xattr,flock 192.168.206.137@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0934837 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1dc0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1520000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1520000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1520000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2320000 Read this (4 bytes): BBBB 192.168.206.137@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg637-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 09:11:04 (1790773864) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0123771 s, 662 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 09:11:11 (1790773871) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491167 s, 21.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x27c lmm_fid: [0x200000404:0x27c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x352:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x27c lmm_fid: [0x200000404:0x27c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x354:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834556 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 09:11:12.000000000 -0400 Modify: 2026-09-30 09:11:12.000000000 -0400 Change: 2026-09-30 09:11:14.000000000 -0400 Birth: 2026-09-30 09:11:13.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 09:11:18 (1790773878) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 09:11:21 (1790773881) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.585078 s, 70.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.084808 s, 48.3 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 30 09:11 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x27e:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 30 09:11 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 09:11:29 (1790773889) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 09:11:37 (1790773897) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1778 sec =========== 09:11:46 (1790773906) === sanity-flr: start cleanup 09:11:48 (1790773908) === === sanity-flr: finish cleanup 09:11:53 (1790773913) ===