-----============= acceptance-small: sanity-flr ============----- Tue Sep 22 13:07:47 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITY_FLR_ONLY='' SANITY_FLR_EXCEPT='' oleg201-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 13:08:06 (1790096886) === oleg201-client.virtnet: executing check_config_client /mnt/lustre oleg201-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg201-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88ca03664800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88ca03664800.idle_timeout=debug disable quota as required oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 13:08:41 (1790096921) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7629] == sanity-flr test 0a: lfs mirror create with -N option == 13:08:43 (1790096923) PASS 0a (22s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:09:05 (1790096945) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:09:09 (1790096949) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:09:12 (1790096952) PASS 0d (25s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:09:38 (1790096978) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:09:42 (1790096982) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 13:09:45 (1790096985) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff88ca03664800.stats=clear osc.lustre-OST0001-osc-ffff88ca03664800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:09:54 (1790096994) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--ec-map] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:10:06 (1790097006) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:10:13 (1790097013) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 13:10:17 (1790097017) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:10:24 (1790097024) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 13:10:33 (1790097033) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:10:41 (1790097041) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:10:44 (1790097044) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:10:51 (1790097051) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:11:05 (1790097065) PASS 7 (11s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:11:16 (1790097076) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 1.73623 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197333 s, 5.3 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:11:29 (1790097089) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.190608 s, 27.5 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 13:11:39 (1790097099) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:11:51 (1790097111) Stopping /mnt/lustre-ost1 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88ca03664800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88ca03664800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg201-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88ca03664800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88ca03664800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg201-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (77s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:13:08 (1790097188) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.428726 s, 12.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88ca03664800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88ca03664800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg201-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (51s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:13:59 (1790097239) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88ca03664800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88ca03664800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg201-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88ca03664800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88ca03664800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg201-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (81s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:15:20 (1790097320) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.93158 s, 13.3 MB/s -rw-r--r-- 1 root root 52428800 Sep 22 13:15 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.980s user 0m0.001s sys 0m0.166s Stopping /mnt/lustre-ost1 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff88ca03664800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88ca03664800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.189s user 0m0.000s sys 0m0.191s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg201-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg201-server oleg201-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff88ca03664800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88ca03664800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.298s user 0m0.001s sys 0m0.195s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg201-server: oleg201-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg201-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg201-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88ca03664800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88ca03664800.ost_server_uuid in FULL state after 0 sec PASS 33b (103s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:17:03 (1790097423) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:17:07 (1790097427) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:17:10 (1790097430) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 13:17:14 (1790097434) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 13:17:22 (1790097442) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.655854 s, 19.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:17:34 (1790097454) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297878 s, 35.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379923 s, 27.6 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:17:42 (1790097462) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:17:50 (1790097470) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00102374 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00147603 s, 694 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00124139 s, 825 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000937494 s, 1.1 MB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 13:18:01 (1790097481) create test files with size 15743684 .. cfde92210a3709c718370793c37b84d4 - 92d1e3c380318ce05420a93e9545d730 - f20c8bf0e49d5239b97d1dd3b755a4c5 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg201-client.virtnet /mnt/lustre (opts:) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre PASS 37 (30s) == sanity-flr test 38: resync ============================ 13:18:31 (1790097511) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:18:47 (1790097527) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:18:55 (1790097535) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102392 s, 10.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0566471 s, 18.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xd8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 13:19:03 (1790097543) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.119606 s, 17.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.330224 s, 12.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.220729 s, 9.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (15s) == sanity-flr test 42: lfs mirror verify ================= 13:19:18 (1790097558) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 13:19:21 (1790097561) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:19:24 (1790097564) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635669 s, 16.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 13:19:33 (1790097573) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:19:37 (1790097577) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xdf:0x0] } split mirror_id 2 id 131073 error lfs split: file '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' has 1 component, cannot split make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:19:48 (1790097588) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.153074 s, 68.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg201-client.virtnet /mnt/lustre (opts:) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:19:58 (1790097598) llite.lustre-ffff88ca0622f800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.237373 s, 44.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg201-client.virtnet /mnt/lustre (opts:) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff88ca114cc800.xattr_cache=1 PASS 44e (12s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 13:20:11 (1790097611) error lfs split: create victim file failed: File exists PASS 44f (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:20:27 (1790097627) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:21:11 (1790097671) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:21:52 (1790097712) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 13:21:57 (1790097717) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.294529 s, 10.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.247872 s, 12.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x1915ffc (nosync,stale) Mirror 2: 0x1915ffc (nosync,stale) ** verify mirror 2 doesn't change original checksum: 2c58221e7cb30672afec0099a3b8da06 - mirror 1 checksum: b2bfa9ab118f253d83e2065341bce077 - mirror 2 checksum: 2c58221e7cb30672afec0099a3b8da06 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-22T13:22:01-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x104:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-22T13:22:01-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x105:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 2c58221e7cb30672afec0099a3b8da06 - mirror 1 checksum: b2bfa9ab118f253d83e2065341bce077 - mirror 2 checksum: b2bfa9ab118f253d83e2065341bce077 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-22T13:22:01-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x104:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-22T13:22:01-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x65 lmm_fid: [0x200000404:0x65:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x105:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:22:12 (1790097732) Failing mds1 on oleg201-server Stopping /mnt/lustre-mds1 (opts:) on oleg201-server 13:22:20 (1790097740) shut down facet: mds1 facet_host: oleg201-server facet_failover_host: oleg201-server Failover mds1 to oleg201-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 13:22:53 (1790097773) targets are mounted 13:22:53 (1790097773) facet_failover done oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x69:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0322241 s, 32.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x69 lmm_fid: [0x200000404:0x69:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x106:0x0] } PASS 50A (59s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:23:10 (1790097790) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:23:13 (1790097793) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:23:17 (1790097797) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00518343 s, 3.2 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:23:24 (1790097804) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:23:27 (1790097807) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0166922 s, 245 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0905547 s, 45.2 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:23:35 (1790097815) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x76:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:23:58 (1790097838) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:24:27 (1790097867) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0xa7:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:24:51 (1790097891) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (26s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:25:16 (1790097916) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 13:25:24 (1790097924) mirror create pid 54994 mirror split pid 54995 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 7881: 54995 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5127: 54994 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 13:26:31 (1790097991) Seed set to 7993 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x35e62 000001 1790097993.571903 trunc from 00000000 to 0x035e61 (0x35e62 bytes) 000002 1790097993.586569 mapread 0x007f44 thru 0x00c535 (0x45f2 bytes) 000003 1790097993.607373 mapread 0x02277f thru 0x02c4ec (0x9d6e bytes) 000004 1790097993.625685 write 0x011d6d thru 0x01d0b7 (0xb34b bytes) 000005 1790097993.628763 punch from 0x002740 to 0x00948e (0x6d4f bytes) 000006 1790097993.642883 trunc from 0x035e62 to 0x018a86 (0x1d3db bytes) 000012 1790097993.664472 trunc from 0x018a87 to 0x00433b (0x1474b bytes) 000013 1790097993.686328 punch from 0x002f23 to 0x00433b (0x1419 bytes) 000015 1790097993.694534 write 0x036a1f thru 0x03ffff (0x95e1 bytes) 000016 1790097993.696690 mapread 0x02e97f thru 0x03565f (0x6ce1 bytes) 000017 1790097993.704606 trunc from 0x040000 to 0x0249ee (0x1b611 bytes) 000018 1790097993.725510 punch from 0x006c7d to 0x015827 (0xebab bytes) 000019 1790097993.733300 mapwrite 0x024d7b thru 0x030bfb (0xbe81 bytes) 000021 1790097993.828259 mapread 0x001ef9 thru 0x010e66 (0xef6e bytes) 000022 1790097993.855086 write 0x02d6fc thru 0x02daca (0x3cf bytes) 000023 1790097993.857666 trunc from 0x030bfc to 0x012371 (0x1e88a bytes) 000027 1790097993.890539 read 0x00b2f9 thru 0x012371 (0x7079 bytes) 000028 1790097993.891346 mapread 0x001d4b thru 0x0102d7 (0xe58d bytes) 000030 1790097993.895300 punch from 0x00e368 to 0x012371 (0x400a bytes) 000031 1790097993.900811 read 0x00f656 thru 0x012371 (0x2d1c bytes) 000032 1790097993.911072 write 0x03b012 thru 0x03ffff (0x4fee bytes) 000034 1790097993.917875 MIRROR_EXTEND to 2 mirrors 000035 1790097994.889637 read 0x017ea9 thru 0x0238d3 (0xba2b bytes) 000036 1790097994.910414 punch from 0x00cdfc to 0x00f9e0 (0x2be5 bytes) 000037 1790097995.006706 punch from 0x011035 to 0x0152ed (0x42b9 bytes) 000038 1790097995.013628 mapread 0x0075a7 thru 0x00b82c (0x4286 bytes) 000039 1790097995.035827 mapwrite 0x03c01c thru 0x03ffff (0x3fe4 bytes) 000040 1790097995.114947 write 0x01cd56 thru 0x02224e (0x54f9 bytes) 000042 1790097995.127800 read 0x01a92f thru 0x0298af (0xef81 bytes) 000043 1790097995.141585 write 0x032536 thru 0x03ffff (0xdaca bytes) truncating to largest ever: 0x3ef7e 000044 1790097995.142872 trunc from 0x040000 to 0x03ef7d (0x1082 bytes) 000045 1790097995.166669 read 0x0310e9 thru 0x033161 (0x2079 bytes) 000047 1790097995.169418 punch from 0x02078b to 0x02db11 (0xd387 bytes) 000048 1790097995.192707 mapread 0x01c93a thru 0x027365 (0xaa2c bytes) 000049 1790097995.207685 mapread 0x00885f thru 0x010376 (0x7b18 bytes) 000050 1790097995.209421 mapwrite 0x02ff05 thru 0x039fe9 (0xa0e5 bytes) 000051 1790097995.265643 write 0x01d103 thru 0x02cf3a (0xfe38 bytes) 000052 1790097995.267030 trunc from 0x03ef7e to 0x016d9b (0x281e2 bytes) 000053 1790097995.287096 mapwrite 0x01bf0d thru 0x027e39 (0xbf2d bytes) 000055 1790097995.487338 write 0x00b8d5 thru 0x00e10d (0x2839 bytes) 000056 1790097995.492584 mapread 0x021b01 thru 0x027e39 (0x6339 bytes) 000058 1790097995.497045 mapread 0x010619 thru 0x01a86c (0xa254 bytes) 000060 1790097995.521663 mapwrite 0x03bf3d thru 0x03ffff (0x40c3 bytes) 000061 1790097995.631070 write 0x01a281 thru 0x01ab9a (0x91a bytes) 000064 1790097995.637211 trunc from 0x040000 to 0x028ac6 (0x17539 bytes) 000065 1790097995.663592 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x28ac7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28ac7): Mirror 2: 0x5e9929b (stale) CRC-32 checksum value for chunk [0, 0x28ac7): Mirror 1: 0x55732151 Mirror 3: 0x55732151 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28ac7: skipped 000066 1790097996.653099 mapwrite 0x03cd5c thru 0x03ffff (0x32a4 bytes) 000068 1790097996.892879 trunc from 0x040000 to 0x02a500 (0x15aff bytes) 000069 1790097996.934312 write 0x00f4dc thru 0x01a15e (0xac83 bytes) 000070 1790097996.949671 read 0x01b2a6 thru 0x01f1a7 (0x3f02 bytes) 000072 1790097996.963219 punch from 0x018122 to 0x025ca7 (0xdb86 bytes) 000073 1790097996.978295 write 0x00fad0 thru 0x015d9c (0x62cd bytes) 000075 1790097996.980566 trunc from 0x02a501 to 0x02ef4a (0x4a4a bytes) 000076 1790097997.009554 read 0x00c2d5 thru 0x00ce61 (0xb8d bytes) 000077 1790097997.028448 write 0x014fdf thru 0x0217cf (0xc7f1 bytes) 000079 1790097997.033827 read 0x01d104 thru 0x02cfe5 (0xfee2 bytes) 000081 1790097997.035622 trunc from 0x02ef4b to 0x026a27 (0x8523 bytes) 000082 1790097997.052267 mapwrite 0x01d160 thru 0x025cec (0x8b8d bytes) 000084 1790097997.158044 mapread 0x013bf0 thru 0x02297c (0xed8d bytes) 000087 1790097997.174036 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x26a28): Mirror 3: 0xc6f4af7d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x26a28): Mirror 3: 0xc6f4af7d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x26a28: skipped 000088 1790097997.505002 mapread 0x0229c3 thru 0x0269ad (0x3feb bytes) 000089 1790097997.522521 mapread 0x00807b thru 0x009f1b (0x1ea1 bytes) 000090 1790097997.531419 punch from 0x00d3d5 to 0x01630f (0x8f3b bytes) 000091 1790097997.535357 write 0x0187c3 thru 0x01eb56 (0x6394 bytes) 000092 1790097997.543774 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) 000095 1790097997.677433 mapread 0x01acbc thru 0x026a27 (0xbd6c bytes) 000096 1790097997.707896 punch from 0x02506e to 0x026a27 (0x19ba bytes) 000098 1790097997.712253 trunc from 0x026a28 to 0x029f0f (0x34e8 bytes) 000101 1790097997.726066 trunc from 0x029f10 to 0x0209a8 (0x9567 bytes) 000102 1790097997.744420 punch from 0x002966 to 0x00e437 (0xbad2 bytes) 000103 1790097997.749282 mapread 0x017017 thru 0x01ad6e (0x3d58 bytes) 000104 1790097997.752340 write 0x00d18d thru 0x01027a (0x30ee bytes) 000105 1790097997.765698 trunc from 0x0209a9 to 0x037400 (0x16a58 bytes) 000106 1790097997.779164 trunc from 0x037401 to 0x021f4d (0x154b3 bytes) 000107 1790097997.795653 read 0x005b13 thru 0x00df8d (0x847b bytes) 000109 1790097997.806301 trunc from 0x021f4e to 0x009ae6 (0x18467 bytes) 000110 1790097997.822057 mapread 0x004c47 thru 0x009ae6 (0x4ea0 bytes) 000111 1790097997.828413 mapwrite 0x02c64f thru 0x034abe (0x8470 bytes) 000113 1790097997.896820 mapread 0x0117f6 thru 0x01cfba (0xb7c5 bytes) 000114 1790097997.930503 read 0x010bc1 thru 0x01db5b (0xcf9b bytes) 000115 1790097997.931486 trunc from 0x034abf to 0x015cff (0x1edbf bytes) 000120 1790097997.952352 punch from 0x007978 to 0x015cff (0xe388 bytes) 000121 1790097997.956868 read 0x00b19d thru 0x00b999 (0x7fd bytes) 000123 1790097997.967177 trunc from 0x015d00 to 0x03325d (0x1d55e bytes) 000125 1790097997.983519 mapread 0x002024 thru 0x00d797 (0xb774 bytes) 000126 1790097997.984599 mapwrite 0x025db5 thru 0x0339e0 (0xdc2c bytes) 000127 1790097998.050614 read 0x0337da thru 0x0339e0 (0x207 bytes) truncating to largest ever: 0x3f714 000129 1790097998.052862 trunc from 0x0339e1 to 0x03f713 (0xbd33 bytes) 000130 1790097998.066489 trunc from 0x03f714 to 0x010b53 (0x2ebc0 bytes) 000132 1790097998.084703 write 0x004ac7 thru 0x0140be (0xf5f8 bytes) 000133 1790097998.086700 mapread 0x011606 thru 0x0140be (0x2ab9 bytes) 000134 1790097998.088469 write 0x03c38e thru 0x03ffff (0x3c72 bytes) 000135 1790097998.089693 punch from 0x001400 to 0x007456 (0x6057 bytes) 000137 1790097998.108303 MIRROR_EXTEND to 2 mirrors 000140 1790097998.908745 read 0x034143 thru 0x03ffff (0xbebd bytes) 000141 1790097998.925631 punch from 0x00492d to 0x0068ac (0x1f80 bytes) 000142 1790097999.017856 write 0x0252c8 thru 0x0330be (0xddf7 bytes) 000147 1790097999.039514 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x96dbeaac (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4475e41b Mirror 3: 0x4475e41b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000148 1790097999.980520 mapwrite 0x021cc2 thru 0x03103f (0xf37e bytes) 000149 1790098000.184243 punch from 0x0121c2 to 0x01dcf7 (0xbb36 bytes) 000151 1790098000.196191 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x96dbeaac (stale) Mirror 3: 0x4475e41b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5292297c Mirror 4: 0x5292297c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000154 1790098001.143280 read 0x00abb8 thru 0x00ae13 (0x25c bytes) 000155 1790098001.155021 mapread 0x014b9c thru 0x01e751 (0x9bb6 bytes) 000157 1790098001.207285 trunc from 0x040000 to 0x02588b (0x1a774 bytes) 000158 1790098001.320411 write 0x0165f8 thru 0x0214e2 (0xaeeb bytes) 000161 1790098001.334493 punch from 0x011aab to 0x01c6d4 (0xac2a bytes) 000163 1790098001.354240 MIRROR_RESYNC 4 mirrors 000164 1790098002.206096 write 0x02512d thru 0x032ac7 (0xd99b bytes) 000165 1790098002.287878 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x32ac8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32ac8): Mirror 2: 0x4b11875b (stale) Mirror 3: 0x4b11875b (stale) Mirror 4: 0x4b11875b (stale) CRC-32 checksum value for chunk [0, 0x32ac8): Mirror 1: 0x88b895ef Mirror 5: 0x88b895ef lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32ac8: skipped 000166 1790098003.144268 mapread 0x03110c thru 0x032ac7 (0x19bc bytes) 000167 1790098003.177382 mapread 0x008c9a thru 0x012953 (0x9cba bytes) 000168 1790098003.225326 read 0x023a1c thru 0x03091b (0xcf00 bytes) 000169 1790098003.249041 trunc from 0x032ac8 to 0x03518b (0x26c4 bytes) 000171 1790098003.358503 mapread 0x02c76a thru 0x03518b (0x8a22 bytes) 000173 1790098003.372701 punch from 0x00dba9 to 0x011083 (0x34db bytes) 000175 1790098003.378404 write 0x006ad2 thru 0x007c55 (0x1184 bytes) 000176 1790098003.386397 mapread 0x02b665 thru 0x02da2f (0x23cb bytes) 000177 1790098003.387505 trunc from 0x03518c to 0x00eb20 (0x2666b bytes) 000179 1790098003.411112 mapread 0x00652c thru 0x00db65 (0x763a bytes) 000183 1790098003.414587 write 0x019b27 thru 0x02039f (0x6879 bytes) 000184 1790098003.416390 read 0x00930c thru 0x01891a (0xf60f bytes) 000185 1790098003.424885 trunc from 0x0203a0 to 0x00ba29 (0x14976 bytes) 000186 1790098003.475091 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xba2a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xba2a): Mirror 2: 0x66df70bd (stale) Mirror 3: 0x66df70bd (stale) Mirror 4: 0x66df70bd (stale) Mirror 5: 0x66df70bd (stale) CRC-32 checksum value for chunk [0, 0xba2a): Mirror 1: 0xe32ab504 Mirror 6: 0xe32ab504 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xba2a: skipped 000187 1790098004.795812 mapread 0x00829a thru 0x00ba29 (0x3790 bytes) 000188 1790098004.889522 mapread 0x009e81 thru 0x00ba29 (0x1ba9 bytes) 000189 1790098004.922469 punch from 0x0057fa to 0x00b674 (0x5e7b bytes) 000190 1790098005.016076 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xba2a): Mirror 3: 0x66df70bd (stale) Mirror 4: 0x66df70bd (stale) Mirror 5: 0x66df70bd (stale) Mirror 6: 0xe32ab504 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xba2a): Mirror 3: 0x66df70bd (stale) Mirror 4: 0x66df70bd (stale) Mirror 5: 0x66df70bd (stale) Mirror 6: 0xe32ab504 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xba2a: skipped 000191 1790098005.348370 mapwrite 0x025667 thru 0x02c231 (0x6bcb bytes) 000192 1790098005.426797 punch from 0x01a76a to 0x029ecc (0xf763 bytes) 000193 1790098005.431362 mapread 0x0267dc thru 0x02c231 (0x5a56 bytes) 000194 1790098005.443600 mapwrite 0x03f0a2 thru 0x03ffff (0xf5e bytes) 000196 1790098005.545588 mapwrite 0x01aef5 thru 0x022896 (0x79a2 bytes) 000197 1790098005.624135 write 0x009f26 thru 0x016b7a (0xcc55 bytes) 000198 1790098005.630190 read 0x0098ca thru 0x00c6d5 (0x2e0c bytes) 000199 1790098005.630723 read 0x0235c4 thru 0x032fd1 (0xfa0e bytes) 000200 1790098005.642081 mapwrite 0x005b11 thru 0x00ca90 (0x6f80 bytes) 000201 1790098005.692296 write 0x01d723 thru 0x020300 (0x2bde bytes) 000202 1790098005.693767 trunc from 0x040000 to 0x03f44c (0xbb3 bytes) 000203 1790098005.724773 read 0x028b44 thru 0x02ea77 (0x5f34 bytes) 000204 1790098005.725863 mapread 0x03d68f thru 0x03f44c (0x1dbe bytes) 000205 1790098005.735842 trunc from 0x03f44d to 0x002bcb (0x3c881 bytes) 000206 1790098005.770148 read 0x0022ed thru 0x002bcb (0x8df bytes) 000207 1790098005.785260 MIRROR_RESYNC 5 mirrors 000208 1790098006.799247 mapread 0x001df4 thru 0x002bcb (0xdd8 bytes) 000210 1790098006.813239 MIRROR_RESYNC 5 mirrors 000211 1790098007.073313 MIRROR_RESYNC 5 mirrors 000213 1790098007.389752 read 0x00022f thru 0x002bcb (0x299d bytes) 000214 1790098007.395853 read 0x00212e thru 0x002bcb (0xa9e bytes) 000216 1790098007.396482 write 0x01c871 thru 0x021745 (0x4ed5 bytes) 000219 1790098007.473140 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21746): Mirror 3: 0x853a9fd (stale) Mirror 4: 0x853a9fd (stale) Mirror 6: 0x853a9fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21746): Mirror 3: 0x853a9fd (stale) Mirror 4: 0x853a9fd (stale) Mirror 6: 0x853a9fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21746: skipped 000220 1790098007.921451 punch from 0x003306 to 0x00e64c (0xb347 bytes) 000221 1790098007.928038 mapwrite 0x0182ed thru 0x01d31a (0x502e bytes) 000223 1790098007.996991 trunc from 0x021746 to 0x025bb5 (0x4470 bytes) 000224 1790098008.013392 punch from 0x00fc3a to 0x018830 (0x8bf7 bytes) 000229 1790098008.033456 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25bb6): Mirror 3: 0x51da1444 (stale) Mirror 4: 0x51da1444 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25bb6): Mirror 3: 0x51da1444 (stale) Mirror 4: 0x51da1444 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25bb6: skipped 000230 1790098008.322378 mapread 0x023c71 thru 0x025bb5 (0x1f45 bytes) 000232 1790098008.345463 write 0x030409 thru 0x03c6dd (0xc2d5 bytes) 000233 1790098008.347198 write 0x01a1e1 thru 0x0220c9 (0x7ee9 bytes) 000234 1790098008.358405 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c6de): Mirror 3: 0x8defdab8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c6de): Mirror 3: 0x8defdab8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c6de: skipped 000235 1790098008.602435 mapwrite 0x00d40a thru 0x00f08c (0x1c83 bytes) 000236 1790098008.661070 mapread 0x0041b3 thru 0x00b47a (0x72c8 bytes) 000238 1790098008.670421 punch from 0x00f920 to 0x01b240 (0xb921 bytes) 000240 1790098008.677236 trunc from 0x03c6de to 0x0118b7 (0x2ae26 bytes) 000241 1790098008.698361 mapwrite 0x02e654 thru 0x0361aa (0x7b57 bytes) 000242 1790098008.804030 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x361ab): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x361ab): Mirror 3: 0x98f4bff9 (stale) CRC-32 checksum value for chunk [0, 0x361ab): Mirror 1: 0x958c6939 Mirror 4: 0x958c6939 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x361ab: skipped 000243 1790098009.657302 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x361ab): Mirror 3: 0x98f4bff9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x361ab): Mirror 3: 0x98f4bff9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x361ab: skipped 000244 1790098010.026082 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) 000247 1790098010.237648 trunc from 0x0361ab to 0x0134db (0x22ccf bytes) 000248 1790098010.297058 MIRROR_EXTEND to 2 mirrors 000249 1790098011.149735 trunc from 0x0134dc to 0x004fbe (0xe51d bytes) 000250 1790098011.228414 punch from 0x00028f to 0x004fbe (0x4d30 bytes) 000252 1790098011.236870 mapread 0x003ac5 thru 0x004fbe (0x14fa bytes) 000255 1790098011.252052 write 0x02e6b4 thru 0x03df81 (0xf8ce bytes) 000256 1790098011.256018 mapread 0x0026c4 thru 0x0087bf (0x60fc bytes) 000257 1790098011.266338 punch from 0x00a7b6 to 0x011a77 (0x72c2 bytes) 000259 1790098011.284024 trunc from 0x03df82 to 0x01219c (0x2bde5 bytes) 000260 1790098011.316434 MIRROR_RESYNC 2 mirrors 000261 1790098011.945311 trunc from 0x01219d to 0x01ef89 (0xcded bytes) 000262 1790098012.014311 write 0x01c5e8 thru 0x02b2b9 (0xecd2 bytes) 000264 1790098012.021149 read 0x026550 thru 0x0283e3 (0x1e94 bytes) 000265 1790098012.021739 trunc from 0x02b2ba to 0x00811d (0x2319c bytes) 000266 1790098012.047499 write 0x0067b6 thru 0x00fb52 (0x939d bytes) 000267 1790098012.055617 write 0x02b826 thru 0x034d0a (0x94e5 bytes) 000269 1790098012.061076 punch from 0x0236f8 to 0x032259 (0xeb62 bytes) 000270 1790098012.077477 trunc from 0x034d0b to 0x03a6c7 (0x59bd bytes) 000271 1790098012.096400 trunc from 0x03a6c8 to 0x024808 (0x15ebf bytes) 000272 1790098012.118573 mapread 0x01bc0c thru 0x01cc39 (0x102e bytes) 000273 1790098012.134233 trunc from 0x024809 to 0x01f92d (0x4edb bytes) 000276 1790098012.163678 write 0x00e85b thru 0x0143c8 (0x5b6e bytes) 000278 1790098012.190669 mapread 0x005adb thru 0x0095bc (0x3ae2 bytes) 000279 1790098012.206302 trunc from 0x01f92e to 0x020955 (0x1028 bytes) 000281 1790098012.254303 read 0x010f04 thru 0x01fd7d (0xee7a bytes) 000283 1790098012.273729 write 0x00afd9 thru 0x012287 (0x72af bytes) 000284 1790098012.284482 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20956): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20956): Mirror 2: 0x6be0b8b5 (stale) CRC-32 checksum value for chunk [0, 0x20956): Mirror 1: 0x2ff9cc8f Mirror 3: 0x2ff9cc8f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20956: skipped 000285 1790098013.521504 punch from 0x016ec7 to 0x018c1d (0x1d57 bytes) 000286 1790098013.661136 trunc from 0x020956 to 0x03e6b9 (0x1dd64 bytes) 000287 1790098013.701835 write 0x02ee26 thru 0x03eb89 (0xfd64 bytes) 000288 1790098013.742044 write 0x002025 thru 0x00f147 (0xd123 bytes) 000289 1790098013.761442 punch from 0x001f28 to 0x00493b (0x2a14 bytes) 000290 1790098013.804899 read 0x01627e thru 0x01e1b6 (0x7f39 bytes) 000291 1790098013.823246 trunc from 0x03eb8a to 0x023ec8 (0x1acc1 bytes) 000293 1790098013.876057 write 0x01454d thru 0x0149d5 (0x489 bytes) 000295 1790098013.890034 mapread 0x014a4a thru 0x01c266 (0x781d bytes) 000296 1790098013.901606 read 0x01d744 thru 0x0217d1 (0x408e bytes) 000299 1790098013.915658 write 0x02caab thru 0x033c17 (0x716d bytes) 000303 1790098013.919248 read 0x01b24a thru 0x023bda (0x8991 bytes) 000305 1790098013.922188 mapread 0x010490 thru 0x0173b2 (0x6f23 bytes) 000306 1790098013.926587 write 0x029d6d thru 0x035e3e (0xc0d2 bytes) 000307 1790098013.954637 trunc from 0x035e3f to 0x015abf (0x2037f bytes) 000309 1790098014.021189 punch from 0x00f644 to 0x015abf (0x647c bytes) 000312 1790098014.053930 punch from 0x00c428 to 0x015abf (0x9698 bytes) 000313 1790098014.063760 mapwrite 0x00b28b thru 0x01975d (0xe4d3 bytes) 000314 1790098014.238494 write 0x036926 thru 0x03d45f (0x6b3a bytes) 000315 1790098014.241332 read 0x036ecb thru 0x03d45f (0x6595 bytes) 000316 1790098014.242077 trunc from 0x03d460 to 0x0105f4 (0x2ce6b bytes) 000317 1790098014.273991 punch from 0x000078 to 0x0088a0 (0x8829 bytes) 000318 1790098014.280405 mapwrite 0x012802 thru 0x01f2b4 (0xcab3 bytes) 000322 1790098014.400341 mapread 0x004b51 thru 0x004e24 (0x2d4 bytes) 000323 1790098014.402122 mapwrite 0x03b666 thru 0x03ffff (0x499a bytes) 000325 1790098014.516972 punch from 0x017310 to 0x019f58 (0x2c49 bytes) 000326 1790098014.521549 punch from 0x03a805 to 0x03ffff (0x57fb bytes) 000328 1790098014.529585 MIRROR_RESYNC 3 mirrors 000329 1790098015.548192 trunc from 0x040000 to 0x019501 (0x26afe bytes) 000332 1790098015.631131 mapwrite 0x027b69 thru 0x02c18e (0x4626 bytes) 000333 1790098015.739641 punch from 0x01dc51 to 0x02b83e (0xdbee bytes) 000334 1790098015.749217 trunc from 0x02c18f to 0x000d06 (0x2b488 bytes) 000335 1790098015.769520 trunc from 0x000d07 to 0x01e85c (0x1db56 bytes) 000336 1790098015.791854 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e85d): Mirror 3: 0x21efab66 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e85d): Mirror 3: 0x21efab66 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e85d: skipped 000337 1790098016.060492 write 0x00c24f thru 0x01759f (0xb351 bytes) 000338 1790098016.074592 read 0x01a350 thru 0x01e85c (0x450d bytes) 000345 1790098016.088293 mapwrite 0x0016cb thru 0x00f686 (0xdfbc bytes) 000346 1790098016.147514 punch from 0x00827c to 0x0179cd (0xf752 bytes) 000348 1790098016.154740 punch from 0x008444 to 0x012a9b (0xa658 bytes) 000349 1790098016.159481 read 0x01495e thru 0x01e7aa (0x9e4d bytes) 000351 1790098016.169666 write 0x03b16f thru 0x03cbda (0x1a6c bytes) 000352 1790098016.171694 write 0x008ef9 thru 0x015ca6 (0xcdae bytes) 000353 1790098016.173517 write 0x02b160 thru 0x03a261 (0xf102 bytes) 000354 1790098016.181294 mapwrite 0x03e1b9 thru 0x03ffff (0x1e47 bytes) 000356 1790098016.266565 punch from 0x00deb7 to 0x011709 (0x3853 bytes) 000357 1790098016.271842 punch from 0x0099c5 to 0x009d9c (0x3d8 bytes) 000359 1790098016.291161 trunc from 0x040000 to 0x01971d (0x268e2 bytes) 000361 1790098016.323559 mapread 0x00647d thru 0x012b4b (0xc6cf bytes) 000362 1790098016.364479 MIRROR_RESYNC 2 mirrors 000363 1790098016.894177 punch from 0x01148f to 0x01971d (0x828f bytes) 000364 1790098016.987848 mapwrite 0x0070f5 thru 0x010a7e (0x998a bytes) 000365 1790098017.148251 punch from 0x005ccb to 0x00fa51 (0x9d87 bytes) 000366 1790098017.154286 mapread 0x00aad4 thru 0x010e42 (0x636f bytes) 000367 1790098017.171744 write 0x02adbb thru 0x02db61 (0x2da7 bytes) 000368 1790098017.180820 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) 000370 1790098017.372305 MIRROR_EXTEND to 2 mirrors 000371 1790098017.976523 MIRROR_EXTEND to 3 mirrors 000373 1790098018.629574 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) 000374 1790098019.021089 write 0x039dbb thru 0x03c154 (0x239a bytes) 000375 1790098019.153026 mapwrite 0x0256a2 thru 0x030a92 (0xb3f1 bytes) 000376 1790098019.267244 mapread 0x013fb4 thru 0x021b99 (0xdbe6 bytes) 000378 1790098019.293086 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) 000380 1790098019.478561 punch from 0x009f1e to 0x01009f (0x6182 bytes) 000382 1790098019.493205 mapwrite 0x004a12 thru 0x005664 (0xc53 bytes) 000383 1790098019.561887 mapwrite 0x032d0c thru 0x03caa4 (0x9d99 bytes) 000385 1790098019.646756 trunc from 0x03caa5 to 0x024a34 (0x18070 bytes) 000386 1790098019.685775 read 0x016ec6 thru 0x01b740 (0x487b bytes) 000387 1790098019.702512 mapread 0x020caa thru 0x022f6d (0x22c4 bytes) 000388 1790098019.718406 write 0x01a364 thru 0x0276f9 (0xd396 bytes) 000389 1790098019.731700 MIRROR_EXTEND to 2 mirrors 000391 1790098020.447654 read 0x0213b8 thru 0x0276f9 (0x6342 bytes) 000393 1790098020.477491 trunc from 0x0276fa to 0x03b4f1 (0x13df8 bytes) 000395 1790098020.553627 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b4f2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b4f2): Mirror 2: 0xc0a35d2e (stale) CRC-32 checksum value for chunk [0, 0x3b4f2): Mirror 1: 0xc0a35d2e Mirror 3: 0xc0a35d2e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b4f2: skipped 000396 1790098021.486612 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b4f2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b4f2): Mirror 2: 0xc0a35d2e (stale) CRC-32 checksum value for chunk [0, 0x3b4f2): Mirror 1: 0xc0a35d2e Mirror 3: 0xc0a35d2e Mirror 4: 0xc0a35d2e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b4f2: skipped 000399 1790098022.348509 mapread 0x011cb4 thru 0x0167e2 (0x4b2f bytes) 000400 1790098022.367172 punch from 0x0320bd to 0x03b4f1 (0x9435 bytes) 000401 1790098022.446906 trunc from 0x03b4f2 to 0x02b935 (0xfbbc bytes) 000402 1790098022.467476 trunc from 0x02b936 to 0x0117d3 (0x1a162 bytes) 000404 1790098022.487311 read 0x00d25f thru 0x0117d3 (0x4575 bytes) 000405 1790098022.495513 write 0x0395d6 thru 0x03ffff (0x6a2a bytes) 000406 1790098022.496905 read 0x029c7f thru 0x034cc3 (0xb045 bytes) 000408 1790098022.507533 mapread 0x016a28 thru 0x021705 (0xacde bytes) 000409 1790098022.509047 punch from 0x03947c to 0x03c552 (0x30d7 bytes) 000410 1790098022.523501 punch from 0x02e6af to 0x02e730 (0x82 bytes) 000411 1790098022.528382 write 0x013a84 thru 0x01a569 (0x6ae6 bytes) 000412 1790098022.529930 trunc from 0x040000 to 0x03cda3 (0x325c bytes) 000413 1790098022.543803 read 0x01b99d thru 0x02237b (0x69df bytes) 000414 1790098022.545326 read 0x01936a thru 0x01c02c (0x2cc3 bytes) 000415 1790098022.546668 punch from 0x00b6b4 to 0x0150a9 (0x99f6 bytes) 000416 1790098022.558033 trunc from 0x03cda4 to 0x02e7c7 (0xe5dc bytes) 000417 1790098022.576605 MIRROR_RESYNC 4 mirrors 000419 1790098023.483895 trunc from 0x02e7c8 to 0x038d46 (0xa57f bytes) 000420 1790098023.575509 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x38d47): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38d47): Mirror 2: 0x1ddfaf14 (stale) Mirror 3: 0x1ddfaf14 (stale) Mirror 4: 0x1ddfaf14 (stale) CRC-32 checksum value for chunk [0, 0x38d47): Mirror 1: 0x1ddfaf14 Mirror 5: 0x1ddfaf14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38d47: skipped 000422 1790098024.666963 trunc from 0x038d47 to 0x00cc1f (0x2c127 bytes) 000423 1790098024.776126 read 0x003629 thru 0x00cc1f (0x95f7 bytes) 000424 1790098024.809756 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xcc20): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xcc20): Mirror 2: 0x924ca901 (stale) Mirror 3: 0x924ca901 (stale) Mirror 4: 0x924ca901 (stale) Mirror 5: 0x924ca901 (stale) CRC-32 checksum value for chunk [0, 0xcc20): Mirror 1: 0x924ca901 Mirror 6: 0x924ca901 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xcc20: skipped 000425 1790098025.642982 mapread 0x008e7e thru 0x00cc1f (0x3da2 bytes) 000426 1790098025.672033 mapread 0x00346f thru 0x00cc1f (0x97b1 bytes) 000428 1790098025.697405 write 0x016e78 thru 0x01a95c (0x3ae5 bytes) 000430 1790098025.775933 punch from 0x01a749 to 0x01a95c (0x214 bytes) 000431 1790098025.803257 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1a95d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a95d): Mirror 2: 0x57f31f17 (stale) Mirror 3: 0x57f31f17 (stale) Mirror 4: 0x57f31f17 (stale) Mirror 5: 0x57f31f17 (stale) Mirror 6: 0x187cff4 (stale) CRC-32 checksum value for chunk [0, 0x1a95d): Mirror 1: 0x287e0771 Mirror 7: 0x287e0771 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a95d: skipped 000432 1790098026.677996 trunc from 0x01a95d to 0x02e4d3 (0x13b77 bytes) 000434 1790098026.750635 mapread 0x02e016 thru 0x02e4d3 (0x4be bytes) 000435 1790098026.755798 mapread 0x010a90 thru 0x01d360 (0xc8d1 bytes) 000436 1790098026.775867 write 0x028b34 thru 0x0358be (0xcd8b bytes) 000439 1790098026.787677 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x358bf): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x358bf): Mirror 2: 0xc88df7f8 (stale) Mirror 3: 0xc88df7f8 (stale) Mirror 4: 0xc88df7f8 (stale) Mirror 5: 0xc88df7f8 (stale) Mirror 6: 0xc72204d8 (stale) Mirror 7: 0xde02ebc4 (stale) CRC-32 checksum value for chunk [0, 0x358bf): Mirror 1: 0xcd9c146c Mirror 8: 0xcd9c146c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x358bf: skipped 000440 1790098027.601616 mapwrite 0x013637 thru 0x01c182 (0x8b4c bytes) 000443 1790098027.789373 trunc from 0x0358bf to 0x002497 (0x33427 bytes) 000445 1790098027.820375 mapread 0x0018d1 thru 0x002497 (0xbc7 bytes) 000446 1790098027.826843 mapwrite 0x02cfa7 thru 0x039e38 (0xce92 bytes) 000447 1790098027.918932 read 0x0108ab thru 0x0143a0 (0x3af6 bytes) 000449 1790098027.927378 mapread 0x019026 thru 0x028918 (0xf8f3 bytes) 000450 1790098027.943114 punch from 0x02a711 to 0x030f5a (0x684a bytes) 000451 1790098027.948326 trunc from 0x039e39 to 0x019497 (0x209a1 bytes) 000452 1790098027.961451 mapread 0x017235 thru 0x019497 (0x2263 bytes) 000453 1790098027.962320 punch from 0x0172e8 to 0x019497 (0x21b0 bytes) 000454 1790098027.967428 read 0x0033a8 thru 0x005bc0 (0x2819 bytes) 000455 1790098027.980077 write 0x003039 thru 0x006c49 (0x3c11 bytes) 000457 1790098027.981698 write 0x009fca thru 0x00ef9d (0x4fd4 bytes) 000458 1790098027.983242 punch from 0x006378 to 0x00da6a (0x76f3 bytes) 000461 1790098028.000751 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x19498): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19498): Mirror 2: 0x438a5313 (stale) Mirror 3: 0x438a5313 (stale) Mirror 4: 0x438a5313 (stale) Mirror 5: 0x438a5313 (stale) Mirror 6: 0xb5cee663 (stale) Mirror 7: 0x328f16d0 (stale) Mirror 8: 0x328f16d0 (stale) CRC-32 checksum value for chunk [0, 0x19498): Mirror 1: 0xa6e20412 Mirror 9: 0xa6e20412 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19498: skipped 000463 1790098028.871459 read 0x015fde thru 0x019497 (0x34ba bytes) 000464 1790098028.895144 punch from 0x009f0c to 0x01594a (0xba3f bytes) 000465 1790098028.977584 read 0x000504 thru 0x003d57 (0x3854 bytes) 000466 1790098029.000568 read 0x0047d1 thru 0x008e31 (0x4661 bytes) 000468 1790098029.002536 read 0x0048b4 thru 0x009224 (0x4971 bytes) 000469 1790098029.003813 write 0x00892e thru 0x00d035 (0x4708 bytes) 000470 1790098029.007521 write 0x014a2b thru 0x017b5c (0x3132 bytes) 000471 1790098029.012586 write 0x0229d7 thru 0x030cc7 (0xe2f1 bytes) 000472 1790098029.014759 trunc from 0x030cc8 to 0x007f55 (0x28d72 bytes) 000473 1790098029.046373 mapread 0x00671e thru 0x007f55 (0x1838 bytes) 000474 1790098029.047543 read 0x0067a2 thru 0x006ee9 (0x748 bytes) 000475 1790098029.056116 MIRROR_RESYNC 9 mirrors 000478 1790098030.576527 write 0x02d17c thru 0x0331ff (0x6084 bytes) 000479 1790098030.639618 mapread 0x026bb9 thru 0x0313e3 (0xa82b bytes) 000480 1790098030.658378 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33200): Mirror 2: 0x97ddc280 (stale) Mirror 3: 0x97ddc280 (stale) Mirror 4: 0x97ddc280 (stale) Mirror 6: 0x97ddc280 (stale) Mirror 7: 0x97ddc280 (stale) Mirror 8: 0x97ddc280 (stale) Mirror 9: 0x97ddc280 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33200): Mirror 2: 0x97ddc280 (stale) Mirror 3: 0x97ddc280 (stale) Mirror 4: 0x97ddc280 (stale) Mirror 6: 0x97ddc280 (stale) Mirror 7: 0x97ddc280 (stale) Mirror 8: 0x97ddc280 (stale) Mirror 9: 0x97ddc280 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33200: skipped 000481 1790098031.174122 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33200): Mirror 2: 0x97ddc280 (stale) Mirror 3: 0x97ddc280 (stale) Mirror 4: 0x97ddc280 (stale) Mirror 7: 0x97ddc280 (stale) Mirror 8: 0x97ddc280 (stale) Mirror 9: 0x97ddc280 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33200): Mirror 2: 0x97ddc280 (stale) Mirror 3: 0x97ddc280 (stale) Mirror 4: 0x97ddc280 (stale) Mirror 7: 0x97ddc280 (stale) Mirror 8: 0x97ddc280 (stale) Mirror 9: 0x97ddc280 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33200: skipped 000482 1790098031.565728 mapread 0x011ca2 thru 0x01bcc1 (0xa020 bytes) 000484 1790098031.586374 write 0x004ba5 thru 0x01193d (0xcd99 bytes) 000486 1790098031.593448 mapread 0x014e9f thru 0x015e2d (0xf8f bytes) 000488 1790098031.595873 read 0x032d12 thru 0x0331ff (0x4ee bytes) 000489 1790098031.601474 read 0x02c157 thru 0x0331ff (0x70a9 bytes) 000491 1790098031.617873 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x33200): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33200): Mirror 2: 0x97ddc280 (stale) Mirror 3: 0x97ddc280 (stale) Mirror 4: 0x97ddc280 (stale) Mirror 7: 0x97ddc280 (stale) Mirror 8: 0x97ddc280 (stale) Mirror 9: 0x97ddc280 (stale) CRC-32 checksum value for chunk [0, 0x33200): Mirror 1: 0x3022ab0b Mirror 10: 0x3022ab0b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33200: skipped 000492 1790098032.540856 mapread 0x0189d3 thru 0x023683 (0xacb1 bytes) 000494 1790098032.572633 read 0x02a8bb thru 0x0331ff (0x8945 bytes) 000495 1790098032.581701 mapread 0x01ab0c thru 0x01d6ba (0x2baf bytes) 000497 1790098032.590329 trunc from 0x033200 to 0x021871 (0x1198e bytes) 000499 1790098032.693142 write 0x0324cf thru 0x033e76 (0x19a8 bytes) 000500 1790098032.694218 write 0x0253a8 thru 0x028c50 (0x38a9 bytes) 000501 1790098032.701380 punch from 0x0331c5 to 0x033e6d (0xca9 bytes) 000502 1790098032.711191 mapwrite 0x0207f9 thru 0x02f7ec (0xeff4 bytes) 000503 1790098032.767053 read 0x018d2c thru 0x0276fc (0xe9d1 bytes) 000505 1790098032.782023 trunc from 0x033e77 to 0x01821d (0x1bc59 bytes) 000506 1790098032.800181 MIRROR_RESYNC 8 mirrors 000507 1790098034.297996 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) 000508 1790098034.743848 read 0x00b075 thru 0x0123da (0x7366 bytes) 000510 1790098034.753732 trunc from 0x01821e to 0x02e39e (0x16181 bytes) 000511 1790098034.837774 read 0x01aaf3 thru 0x0222bb (0x77c9 bytes) 000512 1790098034.851395 read 0x00b36b thru 0x00ff4f (0x4be5 bytes) 000513 1790098034.852109 trunc from 0x02e39f to 0x002ffb (0x2b3a3 bytes) 000514 1790098034.883032 mapread 0x00084b thru 0x002ffb (0x27b1 bytes) 000516 1790098034.886273 mapread 0x001848 thru 0x001866 (0x1f bytes) 000517 1790098034.891397 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2ffc): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ffc): Mirror 2: 0x88a26a76 (stale) Mirror 4: 0x88a26a76 (stale) Mirror 7: 0x88a26a76 (stale) Mirror 8: 0x88a26a76 (stale) Mirror 9: 0x88a26a76 (stale) Mirror 10: 0x88a26a76 (stale) CRC-32 checksum value for chunk [0, 0x2ffc): Mirror 1: 0x88a26a76 Mirror 11: 0x88a26a76 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ffc: skipped 000518 1790098035.760771 mapread 0x0017f4 thru 0x002ffb (0x1808 bytes) 000520 1790098035.813763 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2ffc): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ffc): Mirror 2: 0x88a26a76 (stale) Mirror 4: 0x88a26a76 (stale) Mirror 7: 0x88a26a76 (stale) Mirror 8: 0x88a26a76 (stale) Mirror 9: 0x88a26a76 (stale) Mirror 10: 0x88a26a76 (stale) CRC-32 checksum value for chunk [0, 0x2ffc): Mirror 1: 0x88a26a76 Mirror 11: 0x88a26a76 Mirror 12: 0x88a26a76 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ffc: skipped 000521 1790098036.567353 trunc from 0x002ffc to 0x0007fb (0x2800 bytes) 000522 1790098036.640252 mapread 0x0002f5 thru 0x0007fb (0x507 bytes) 000524 1790098036.646181 read 0x000373 thru 0x0007fb (0x489 bytes) 000525 1790098036.646772 punch from 0x0003f8 to 0x0007fb (0x404 bytes) 000526 1790098036.650176 mapwrite 0x038e82 thru 0x03f7bb (0x693a bytes) 000528 1790098036.778479 read 0x00d256 thru 0x013624 (0x63cf bytes) 000529 1790098036.798195 mapwrite 0x01b2b1 thru 0x024970 (0x96c0 bytes) 000530 1790098036.892015 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f7bc): Mirror 2: 0xf27c9587 (stale) Mirror 4: 0xf27c9587 (stale) Mirror 7: 0xf27c9587 (stale) Mirror 8: 0xf27c9587 (stale) Mirror 10: 0xf27c9587 (stale) Mirror 11: 0x6c9065ce (stale) Mirror 12: 0x6c9065ce (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f7bc): Mirror 2: 0xf27c9587 (stale) Mirror 4: 0xf27c9587 (stale) Mirror 7: 0xf27c9587 (stale) Mirror 8: 0xf27c9587 (stale) Mirror 10: 0xf27c9587 (stale) Mirror 11: 0x6c9065ce (stale) Mirror 12: 0x6c9065ce (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f7bc: skipped 000533 1790098037.501025 mapread 0x01d571 thru 0x02cdbf (0xf84f bytes) 000535 1790098037.585607 mapread 0x0100e5 thru 0x0171da (0x70f6 bytes) 000536 1790098037.602899 read 0x03874e thru 0x03b26f (0x2b22 bytes) 000538 1790098037.613225 trunc from 0x03f7bc to 0x02e92f (0x10e8c bytes) 000540 1790098037.634469 trunc from 0x02e930 to 0x0041a4 (0x2a78b bytes) 000541 1790098037.653788 trunc from 0x0041a5 to 0x03ddc4 (0x39c20 bytes) 000542 1790098037.681307 mapread 0x02c883 thru 0x0397a5 (0xcf23 bytes) 000543 1790098037.714876 read 0x033b62 thru 0x03ddc4 (0xa263 bytes) 000544 1790098037.715516 write 0x01500f thru 0x018f4a (0x3f3c bytes) 000546 1790098037.733134 punch from 0x009fed to 0x0185c8 (0xe5dc bytes) 000548 1790098037.751144 mapread 0x02d447 thru 0x035bf5 (0x87af bytes) 000549 1790098037.758019 MIRROR_RESYNC 8 mirrors 000551 1790098039.486864 read 0x02c13f thru 0x03b93b (0xf7fd bytes) 000552 1790098039.500599 mapwrite 0x03193b thru 0x03ba08 (0xa0ce bytes) 000553 1790098039.754550 punch from 0x02f438 to 0x039224 (0x9ded bytes) 000554 1790098039.759199 punch from 0x038ca4 to 0x03d0c8 (0x4425 bytes) 000555 1790098039.762842 punch from 0x01b775 to 0x01e00e (0x289a bytes) 000556 1790098039.768895 read 0x015bfb thru 0x01db3d (0x7f43 bytes) 000561 1790098039.792937 read 0x0161fa thru 0x02593d (0xf744 bytes) 000563 1790098039.801904 trunc from 0x03ddc5 to 0x035795 (0x862f bytes) 000564 1790098039.836366 write 0x0247fa thru 0x02a05c (0x5863 bytes) 000565 1790098039.839220 trunc from 0x035796 to 0x02b35e (0xa437 bytes) 000568 1790098039.870531 read 0x0263fd thru 0x02b35e (0x4f62 bytes) 000569 1790098039.871846 mapread 0x025b38 thru 0x027598 (0x1a61 bytes) 000570 1790098039.877358 trunc from 0x02b35f to 0x03d2a4 (0x11f46 bytes) 000571 1790098039.909011 mapwrite 0x0053d2 thru 0x00d39b (0x7fca bytes) 000575 1790098039.977777 punch from 0x00647d to 0x00886f (0x23f3 bytes) 000576 1790098039.982643 trunc from 0x03d2a5 to 0x00b53b (0x31d69 bytes) 000578 1790098040.014563 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xb53c): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb53c): Mirror 2: 0x9e67914 (stale) Mirror 4: 0x9e67914 (stale) Mirror 7: 0x9e67914 (stale) Mirror 8: 0x9e67914 (stale) Mirror 10: 0x9e67914 (stale) Mirror 11: 0x9e67914 (stale) Mirror 12: 0x9e67914 (stale) CRC-32 checksum value for chunk [0, 0xb53c): Mirror 1: 0xb0467582 Mirror 13: 0xb0467582 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb53c: skipped 000579 1790098041.035638 read 0x0025a9 thru 0x009a9f (0x74f7 bytes) 000580 1790098041.059106 punch from 0x0077ff to 0x00b53b (0x3d3d bytes) 000581 1790098041.196856 read 0x00ac62 thru 0x00b53b (0x8da bytes) 000582 1790098041.206996 trunc from 0x00b53c to 0x00a9cb (0xb70 bytes) 000583 1790098041.232084 read 0x003e82 thru 0x00a9cb (0x6b4a bytes) 000585 1790098041.233271 punch from 0x002baa to 0x009250 (0x66a7 bytes) 000586 1790098041.237094 punch from 0x00a222 to 0x00a9cb (0x7aa bytes) 000587 1790098041.240602 punch from 0x005dcb to 0x00a9cb (0x4c01 bytes) 000588 1790098041.243839 mapwrite 0x01a969 thru 0x01fb96 (0x522e bytes) 000589 1790098041.350867 trunc from 0x01fb97 to 0x029c29 (0xa093 bytes) 000591 1790098041.377805 MIRROR_RESYNC 9 mirrors 000592 1790098043.478873 trunc from 0x029c2a to 0x01aad0 (0xf159 bytes) 000593 1790098043.625152 mapread 0x010a1e thru 0x01aad0 (0xa0b3 bytes) 000595 1790098043.646497 write 0x009532 thru 0x017960 (0xe42f bytes) 000596 1790098043.668047 MIRROR_RESYNC 9 mirrors 000597 1790098044.996692 write 0x030c9a thru 0x03cd7b (0xc0e2 bytes) 000599 1790098045.061119 mapread 0x00d0ff thru 0x01252e (0x5430 bytes) 000600 1790098045.078493 mapwrite 0x010c8d thru 0x016efc (0x6270 bytes) 000602 1790098045.139335 mapwrite 0x025a71 thru 0x025d9a (0x32a bytes) 000603 1790098045.198491 read 0x03701e thru 0x03c102 (0x50e5 bytes) 000604 1790098045.199729 mapwrite 0x002da5 thru 0x010ccc (0xdf28 bytes) 000605 1790098045.254325 trunc from 0x03cd7c to 0x029f09 (0x12e72 bytes) 000606 1790098045.281197 write 0x0250d1 thru 0x02c599 (0x74c9 bytes) 000608 1790098045.284234 read 0x023648 thru 0x02c599 (0x8f52 bytes) 000609 1790098045.287679 write 0x031592 thru 0x037ea5 (0x6914 bytes) 000610 1790098045.294034 MIRROR_RESYNC 9 mirrors 000612 1790098046.961835 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) 000613 1790098047.402312 write 0x00228c thru 0x00e6fa (0xc46f bytes) 000614 1790098047.485366 trunc from 0x037ea6 to 0x009fe7 (0x2debe bytes) 000617 1790098047.503327 MIRROR_RESYNC 8 mirrors 000618 1790098048.440359 mapwrite 0x006968 thru 0x00dd7b (0x7414 bytes) 000620 1790098048.528582 write 0x03d277 thru 0x03ffff (0x2d89 bytes) 000621 1790098048.529477 mapread 0x00a7e5 thru 0x015fcf (0xb7eb bytes) 000623 1790098048.534056 punch from 0x00a45a to 0x01531e (0xaec5 bytes) 000626 1790098048.542684 MIRROR_RESYNC 8 mirrors 000627 1790098049.975406 write 0x01c7a5 thru 0x023f9d (0x77f9 bytes) 000629 1790098050.046244 MIRROR_SPLIT mirror 3 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xce1537c5 (stale) Mirror 7: 0xce1537c5 (stale) Mirror 10: 0xce1537c5 (stale) Mirror 11: 0xce1537c5 (stale) Mirror 12: 0xce1537c5 (stale) Mirror 13: 0xce1537c5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xce1537c5 (stale) Mirror 7: 0xce1537c5 (stale) Mirror 10: 0xce1537c5 (stale) Mirror 11: 0xce1537c5 (stale) Mirror 12: 0xce1537c5 (stale) Mirror 13: 0xce1537c5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000631 1790098050.345407 read 0x0079f6 thru 0x00cddf (0x53ea bytes) 000632 1790098050.358942 mapwrite 0x03a2da thru 0x03ffff (0x5d26 bytes) 000633 1790098050.438743 mapwrite 0x039c1f thru 0x03d639 (0x3a1b bytes) 000634 1790098050.492684 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xce1537c5 (stale) Mirror 7: 0xce1537c5 (stale) Mirror 10: 0xce1537c5 (stale) Mirror 12: 0xce1537c5 (stale) Mirror 13: 0xce1537c5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xce1537c5 (stale) Mirror 7: 0xce1537c5 (stale) Mirror 10: 0xce1537c5 (stale) Mirror 12: 0xce1537c5 (stale) Mirror 13: 0xce1537c5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000635 1790098050.992440 mapwrite 0x038e43 thru 0x03d3cb (0x4589 bytes) 000638 1790098051.108250 mapwrite 0x00dc1e thru 0x01ba3f (0xde22 bytes) 000639 1790098051.206215 trunc from 0x040000 to 0x01089c (0x2f763 bytes) 000640 1790098051.228189 mapwrite 0x03fdf7 thru 0x03ffff (0x209 bytes) 000642 1790098051.315238 mapwrite 0x0345c3 thru 0x03ffff (0xba3d bytes) 000644 1790098051.394291 mapwrite 0x02bd37 thru 0x0305b5 (0x487f bytes) 000645 1790098051.467370 MIRROR_RESYNC 6 mirrors 000648 1790098052.337111 mapwrite 0x0008f7 thru 0x00b9f7 (0xb101 bytes) 000650 1790098052.553822 MIRROR_RESYNC 6 mirrors 000651 1790098053.650360 MIRROR_RESYNC 6 mirrors 000652 1790098054.245373 MIRROR_EXTEND to 7 mirrors 000653 1790098055.039200 mapwrite 0x02606f thru 0x026784 (0x716 bytes) 000656 1790098055.187988 write 0x02db83 thru 0x03d727 (0xfba5 bytes) 000657 1790098055.209713 mapwrite 0x00757a thru 0x0171b6 (0xfc3d bytes) 000658 1790098055.368021 mapread 0x013e23 thru 0x02062f (0xc80d bytes) 000659 1790098055.406321 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe336a564 (stale) Mirror 7: 0xe336a564 (stale) Mirror 10: 0xe336a564 (stale) Mirror 12: 0xe336a564 (stale) Mirror 13: 0xe336a564 (stale) Mirror 14: 0xe336a564 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x21030ce7 Mirror 15: 0x21030ce7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000660 1790098056.249015 mapwrite 0x02bc5b thru 0x035884 (0x9c2a bytes) 000661 1790098056.546657 trunc from 0x040000 to 0x01327c (0x2cd83 bytes) 000662 1790098056.612073 mapread 0x00dd87 thru 0x01327c (0x54f6 bytes) 000665 1790098056.628154 mapread 0x002205 thru 0x00f756 (0xd552 bytes) 000666 1790098056.644575 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1327d): Mirror 7: 0xdf52d02c (stale) Mirror 10: 0xdf52d02c (stale) Mirror 12: 0xdf52d02c (stale) Mirror 13: 0xdf52d02c (stale) Mirror 14: 0xdf52d02c (stale) Mirror 15: 0x8a90796 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1327d): Mirror 7: 0xdf52d02c (stale) Mirror 10: 0xdf52d02c (stale) Mirror 12: 0xdf52d02c (stale) Mirror 13: 0xdf52d02c (stale) Mirror 14: 0xdf52d02c (stale) Mirror 15: 0x8a90796 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1327d: skipped 000667 1790098057.095223 punch from 0x005d0a to 0x0098a1 (0x3b98 bytes) 000668 1790098057.104270 punch from 0x00b40a to 0x013093 (0x7c8a bytes) 000669 1790098057.111671 MIRROR_RESYNC 7 mirrors 000670 1790098058.252029 MIRROR_EXTEND to 8 mirrors 000671 1790098059.303440 punch from 0x00d79b to 0x01327c (0x5ae2 bytes) 000672 1790098059.376804 mapread 0x00bcb1 thru 0x01327c (0x75cc bytes) 000673 1790098059.399412 trunc from 0x01327d to 0x039c76 (0x269fa bytes) 000674 1790098059.421672 mapwrite 0x030c55 thru 0x032571 (0x191d bytes) 000676 1790098059.498579 trunc from 0x039c77 to 0x00e1ce (0x2baa8 bytes) 000677 1790098059.534584 write 0x00fc92 thru 0x01c7d7 (0xcb46 bytes) 000681 1790098059.538688 mapread 0x00318f thru 0x00f6f3 (0xc565 bytes) 000685 1790098059.550729 trunc from 0x01c7d8 to 0x00b536 (0x112a1 bytes) 000686 1790098059.564622 punch from 0x001f75 to 0x00b27f (0x930b bytes) 000688 1790098059.569460 trunc from 0x00b537 to 0x0166cd (0xb197 bytes) 000690 1790098059.592271 punch from 0x00c0cc to 0x0166cd (0xa602 bytes) 000691 1790098059.597987 read 0x0037a4 thru 0x006b39 (0x3396 bytes) 000692 1790098059.605359 punch from 0x00835e to 0x00decc (0x5b6f bytes) 000693 1790098059.614869 punch from 0x00f335 to 0x0146fd (0x53c9 bytes) 000695 1790098059.620805 read 0x0083c9 thru 0x010282 (0x7eba bytes) 000696 1790098059.657667 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x166ce): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x166ce): Mirror 7: 0x29d6887d (stale) Mirror 10: 0x29d6887d (stale) Mirror 12: 0x29d6887d (stale) Mirror 13: 0x29d6887d (stale) Mirror 14: 0x29d6887d (stale) Mirror 15: 0x29d6887d (stale) Mirror 16: 0x29d6887d (stale) CRC-32 checksum value for chunk [0, 0x166ce): Mirror 1: 0x910b1f50 Mirror 17: 0x910b1f50 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x166ce: skipped 000697 1790098061.542648 write 0x034132 thru 0x038f81 (0x4e50 bytes) 000700 1790098061.731726 write 0x0123c7 thru 0x018522 (0x615c bytes) 000701 1790098061.770226 mapwrite 0x02f3bc thru 0x03cbc1 (0xd806 bytes) 000702 1790098062.025265 MIRROR_SPLIT mirror 6 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cbc2): Mirror 7: 0x7df179e7 (stale) Mirror 10: 0x7df179e7 (stale) Mirror 12: 0x7df179e7 (stale) Mirror 13: 0x7df179e7 (stale) Mirror 14: 0x7df179e7 (stale) Mirror 16: 0x7df179e7 (stale) Mirror 17: 0xb90fccc2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cbc2): Mirror 7: 0x7df179e7 (stale) Mirror 10: 0x7df179e7 (stale) Mirror 12: 0x7df179e7 (stale) Mirror 13: 0x7df179e7 (stale) Mirror 14: 0x7df179e7 (stale) Mirror 16: 0x7df179e7 (stale) Mirror 17: 0xb90fccc2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cbc2: skipped 000705 1790098063.068905 write 0x000f68 thru 0x009038 (0x80d1 bytes) 000706 1790098063.081892 punch from 0x0051a8 to 0x0067da (0x1633 bytes) 000709 1790098063.097687 write 0x03808f thru 0x03ffff (0x7f71 bytes) 000711 1790098063.103667 mapwrite 0x036aab thru 0x03ffff (0x9555 bytes) 000712 1790098063.169351 read 0x016378 thru 0x01c4f5 (0x617e bytes) 000714 1790098063.180830 mapread 0x000fe8 thru 0x006c3b (0x5c54 bytes) 000716 1790098063.211423 read 0x00ee7f thru 0x01319d (0x431f bytes) 000718 1790098063.225218 mapwrite 0x02a6f5 thru 0x03058b (0x5e97 bytes) 000719 1790098063.287194 trunc from 0x040000 to 0x0107eb (0x2f814 bytes) 000720 1790098063.305112 mapwrite 0x0242b4 thru 0x02c221 (0x7f6e bytes) 000721 1790098063.429523 read 0x00f6db thru 0x0126c9 (0x2fef bytes) 000723 1790098063.454498 mapread 0x012c4d thru 0x017346 (0x46fa bytes) 000726 1790098063.458507 read 0x02aa7a thru 0x02c221 (0x17a8 bytes) 000727 1790098063.459238 punch from 0x016fb0 to 0x020b0b (0x9b5c bytes) 000729 1790098063.464901 write 0x02cb3c thru 0x03573f (0x8c04 bytes) 000730 1790098063.470652 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35740): Mirror 7: 0xaf97b8b5 (stale) Mirror 10: 0xaf97b8b5 (stale) Mirror 12: 0xaf97b8b5 (stale) Mirror 13: 0xaf97b8b5 (stale) Mirror 14: 0xaf97b8b5 (stale) Mirror 17: 0x7267bb08 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35740): Mirror 7: 0xaf97b8b5 (stale) Mirror 10: 0xaf97b8b5 (stale) Mirror 12: 0xaf97b8b5 (stale) Mirror 13: 0xaf97b8b5 (stale) Mirror 14: 0xaf97b8b5 (stale) Mirror 17: 0x7267bb08 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35740: skipped 000733 1790098064.067672 write 0x0217d6 thru 0x0231b1 (0x19dc bytes) 000734 1790098064.076610 mapwrite 0x008ec9 thru 0x00a011 (0x1149 bytes) 000736 1790098064.143037 read 0x03156a thru 0x03573f (0x41d6 bytes) 000738 1790098064.150997 trunc from 0x035740 to 0x03d121 (0x79e2 bytes) 000740 1790098064.180222 read 0x0070d5 thru 0x0104da (0x9406 bytes) 000741 1790098064.181037 mapwrite 0x0235ca thru 0x0296f0 (0x6127 bytes) 000743 1790098064.256076 MIRROR_RESYNC 7 mirrors 000744 1790098066.074433 mapread 0x01e85c thru 0x0296fd (0xaea2 bytes) 000746 1790098066.132313 MIRROR_EXTEND to 8 mirrors 000748 1790098067.829568 trunc from 0x03d122 to 0x02bd44 (0x113dd bytes) 000750 1790098067.994892 mapwrite 0x001297 thru 0x008541 (0x72ab bytes) 000751 1790098068.121764 mapread 0x0071e7 thru 0x00809c (0xeb6 bytes) 000752 1790098068.129682 read 0x00d18f thru 0x01572b (0x859d bytes) 000753 1790098068.146058 read 0x029534 thru 0x02bd44 (0x2811 bytes) 000755 1790098068.165697 punch from 0x018dbd to 0x01d1d3 (0x4417 bytes) 000757 1790098068.174575 write 0x022a06 thru 0x029cd9 (0x72d4 bytes) 000759 1790098068.187200 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2bd45): Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2bd45): Mirror 7: 0xb55f154b (stale) Mirror 10: 0xb55f154b (stale) Mirror 12: 0xb55f154b (stale) Mirror 13: 0xb55f154b (stale) Mirror 14: 0xb55f154b (stale) Mirror 17: 0xb55f154b (stale) Mirror 18: 0xb55f154b (stale) CRC-32 checksum value for chunk [0, 0x2bd45): Mirror 1: 0xe259915d Mirror 19: 0xe259915d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2bd45: skipped 000760 1790098069.225372 trunc from 0x02bd45 to 0x008f38 (0x22e0c bytes) 000762 1790098069.379705 write 0x01712a thru 0x02290f (0xb7e6 bytes) 000763 1790098069.381110 read 0x005ed1 thru 0x00e5a0 (0x86d0 bytes) 000765 1790098069.398537 write 0x01aada thru 0x01b4ad (0x9d4 bytes) 000767 1790098069.401070 punch from 0x012eb3 to 0x0166e6 (0x3834 bytes) 000768 1790098069.414792 write 0x0200e0 thru 0x02c96c (0xc88d bytes) 000769 1790098069.416692 punch from 0x005791 to 0x00f621 (0x9e91 bytes) 000770 1790098069.432062 read 0x000819 thru 0x001b33 (0x131b bytes) 000771 1790098069.441632 mapread 0x023408 thru 0x029170 (0x5d69 bytes) 000772 1790098069.443288 punch from 0x024495 to 0x026f2b (0x2a97 bytes) 000773 1790098069.448635 punch from 0x00879e to 0x00b8b0 (0x3113 bytes) 000774 1790098069.453874 mapwrite 0x032e13 thru 0x0382ff (0x54ed bytes) 000776 1790098069.546684 write 0x010ba4 thru 0x014ed1 (0x432e bytes) 000777 1790098069.549131 punch from 0x026074 to 0x0314ec (0xb479 bytes) 000778 1790098069.563239 write 0x0324ec thru 0x035c16 (0x372b bytes) 000779 1790098069.565181 trunc from 0x038300 to 0x028280 (0x1007f bytes) 000780 1790098069.583224 punch from 0x020eb1 to 0x024420 (0x3570 bytes) 000781 1790098069.589601 punch from 0x011a87 to 0x0177e0 (0x5d5a bytes) 000783 1790098069.594795 trunc from 0x028281 to 0x00f75a (0x18b26 bytes) 000784 1790098069.615963 mapread 0x00659d thru 0x00f75a (0x91be bytes) 000785 1790098069.641226 read 0x008179 thru 0x00f4b2 (0x733a bytes) 000787 1790098069.643163 punch from 0x002a55 to 0x006285 (0x3831 bytes) 000791 1790098069.662411 MIRROR_RESYNC 9 mirrors 000792 1790098071.434465 mapwrite 0x019b90 thru 0x01fc9d (0x610e bytes) 000793 1790098071.634042 read 0x00e474 thru 0x018a7c (0xa609 bytes) 000797 1790098071.672687 MIRROR_RESYNC 9 mirrors 000799 1790098074.158308 write 0x019725 thru 0x0276c3 (0xdf9f bytes) 000801 1790098074.338412 mapwrite 0x01a850 thru 0x01c1a0 (0x1951 bytes) 000805 1790098074.494291 mapwrite 0x03e23b thru 0x03ffff (0x1dc5 bytes) 000806 1790098074.697718 trunc from 0x040000 to 0x029fb1 (0x1604e bytes) 000807 1790098074.738084 punch from 0x01348a to 0x01a905 (0x747c bytes) 000808 1790098074.747624 mapread 0x0228b8 thru 0x029fb1 (0x76fa bytes) 000809 1790098074.755046 punch from 0x00cc0e to 0x01be79 (0xf26c bytes) 000810 1790098074.763406 mapread 0x00883f thru 0x0125b8 (0x9d7a bytes) 000812 1790098074.794992 read 0x005adb thru 0x00ba38 (0x5f5e bytes) 000813 1790098074.799701 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29fb2): Mirror 10: 0xf2a9f359 (stale) Mirror 12: 0xf2a9f359 (stale) Mirror 13: 0xf2a9f359 (stale) Mirror 14: 0xf2a9f359 (stale) Mirror 17: 0xf2a9f359 (stale) Mirror 18: 0xf2a9f359 (stale) Mirror 19: 0xf2a9f359 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29fb2): Mirror 10: 0xf2a9f359 (stale) Mirror 12: 0xf2a9f359 (stale) Mirror 13: 0xf2a9f359 (stale) Mirror 14: 0xf2a9f359 (stale) Mirror 17: 0xf2a9f359 (stale) Mirror 18: 0xf2a9f359 (stale) Mirror 19: 0xf2a9f359 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29fb2: skipped 000816 1790098075.346587 write 0x002c94 thru 0x0035c4 (0x931 bytes) 000817 1790098075.357290 MIRROR_RESYNC 8 mirrors 000818 1790098077.240892 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) 000819 1790098077.795280 trunc from 0x029fb2 to 0x0201e4 (0x9dcd bytes) 000821 1790098077.974065 write 0x000262 thru 0x0061c9 (0x5f68 bytes) 000822 1790098077.990894 mapread 0x01697e thru 0x0191ec (0x286f bytes) 000823 1790098078.013818 punch from 0x01c925 to 0x01fa7e (0x315a bytes) 000824 1790098078.027949 mapwrite 0x012fd6 thru 0x015557 (0x2582 bytes) 000825 1790098078.119189 read 0x00b68f thru 0x00fd52 (0x46c4 bytes) 000826 1790098078.153835 mapread 0x015879 thru 0x01fb17 (0xa29f bytes) 000828 1790098078.173958 write 0x00a05d thru 0x013324 (0x92c8 bytes) 000830 1790098078.202949 MIRROR_RESYNC 7 mirrors 000831 1790098079.679498 write 0x012bef thru 0x01c664 (0x9a76 bytes) 000832 1790098079.764668 read 0x01cb3b thru 0x01eaaa (0x1f70 bytes) 000833 1790098079.775688 mapwrite 0x009c88 thru 0x01363d (0x99b6 bytes) 000834 1790098079.945605 trunc from 0x0201e5 to 0x03b5fb (0x1b417 bytes) 000835 1790098079.989008 MIRROR_RESYNC 7 mirrors 000836 1790098081.883973 write 0x0314b1 thru 0x035098 (0x3be8 bytes) 000838 1790098081.984244 write 0x03adc9 thru 0x03bd96 (0xfce bytes) 000839 1790098082.018847 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bd97): Mirror 12: 0x214c8d7f (stale) Mirror 13: 0x214c8d7f (stale) Mirror 14: 0x214c8d7f (stale) Mirror 17: 0x214c8d7f (stale) Mirror 18: 0x214c8d7f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd97): Mirror 12: 0x214c8d7f (stale) Mirror 13: 0x214c8d7f (stale) Mirror 14: 0x214c8d7f (stale) Mirror 17: 0x214c8d7f (stale) Mirror 18: 0x214c8d7f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd97: skipped 000840 1790098082.547058 read 0x03152d thru 0x03bd96 (0xa86a bytes) 000841 1790098082.566008 mapread 0x011688 thru 0x01d34d (0xbcc6 bytes) 000842 1790098082.608014 trunc from 0x03bd97 to 0x01e935 (0x1d461 bytes) 000843 1790098082.639656 punch from 0x010ecc to 0x019898 (0x89cd bytes) 000845 1790098082.646771 read 0x00e4db thru 0x012e92 (0x49b8 bytes) 000847 1790098082.668596 trunc from 0x01e936 to 0x015af4 (0x8e41 bytes) 000848 1790098082.691114 punch from 0x009af5 to 0x00a09e (0x5aa bytes) 000849 1790098082.700443 mapread 0x00285d thru 0x00aba7 (0x834b bytes) 000851 1790098082.717890 trunc from 0x015af5 to 0x005668 (0x1048c bytes) 000854 1790098082.740997 trunc from 0x005669 to 0x01ac68 (0x15600 bytes) 000855 1790098082.765322 punch from 0x0030d8 to 0x00fd15 (0xcc3e bytes) 000856 1790098082.772384 trunc from 0x01ac69 to 0x0311e1 (0x16579 bytes) 000857 1790098082.797472 MIRROR_RESYNC 6 mirrors 000859 1790098084.156219 mapread 0x00b8c5 thru 0x019003 (0xd73f bytes) 000860 1790098084.229989 write 0x035c74 thru 0x0366b1 (0xa3e bytes) 000861 1790098084.373670 mapread 0x024fd3 thru 0x026154 (0x1182 bytes) 000865 1790098084.402114 mapwrite 0x0303e3 thru 0x03fe65 (0xfa83 bytes) 000870 1790098084.686026 write 0x02e4e9 thru 0x036ba8 (0x86c0 bytes) 000872 1790098084.698769 MIRROR_RESYNC 6 mirrors 000873 1790098087.278141 write 0x032779 thru 0x036e0f (0x4697 bytes) 000875 1790098087.417621 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3fe66): Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fe66): Mirror 12: 0x350ccf53 (stale) Mirror 13: 0x350ccf53 (stale) Mirror 14: 0x350ccf53 (stale) Mirror 17: 0x350ccf53 (stale) Mirror 18: 0x350ccf53 (stale) CRC-32 checksum value for chunk [0, 0x3fe66): Mirror 1: 0xe0a0537f Mirror 19: 0xe0a0537f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fe66: skipped 000879 1790098088.464507 mapread 0x022e00 thru 0x02a444 (0x7645 bytes) 000882 1790098088.493056 mapread 0x011180 thru 0x01d7df (0xc660 bytes) 000884 1790098088.531179 punch from 0x034a97 to 0x03fe65 (0xb3cf bytes) 000885 1790098088.650351 read 0x0050b5 thru 0x00fb1f (0xaa6b bytes) 000886 1790098088.665811 write 0x0163ca thru 0x01d402 (0x7039 bytes) 000887 1790098088.714990 write 0x026d2f thru 0x02f62b (0x88fd bytes) 000888 1790098088.725486 read 0x0092ce thru 0x0180ed (0xee20 bytes) 000889 1790098088.737353 punch from 0x0054d5 to 0x00a9fe (0x552a bytes) 000890 1790098088.753274 trunc from 0x03fe66 to 0x030976 (0xf4ef bytes) 000891 1790098088.770056 read 0x00e0c8 thru 0x019905 (0xb83e bytes) 000894 1790098088.772812 trunc from 0x030977 to 0x000f57 (0x2fa1f bytes) 000895 1790098088.790493 read 0x0007cb thru 0x000f57 (0x78d bytes) 000897 1790098088.791520 write 0x00ec9b thru 0x01b35f (0xc6c5 bytes) 000898 1790098088.792571 trunc from 0x01b360 to 0x002a5a (0x18905 bytes) 000899 1790098088.809383 mapwrite 0x03ca01 thru 0x03ffff (0x35ff bytes) 000902 1790098088.885635 mapwrite 0x00147b thru 0x0098a6 (0x842c bytes) 000903 1790098088.931685 read 0x0055e3 thru 0x0151c0 (0xfbde bytes) 000904 1790098088.932438 write 0x0108b8 thru 0x01928d (0x89d6 bytes) 000905 1790098088.933487 write 0x000fd0 thru 0x010ceb (0xfd1c bytes) 000906 1790098088.937015 punch from 0x03cf58 to 0x03ffff (0x30a8 bytes) 000907 1790098088.947993 punch from 0x0055a5 to 0x0149a3 (0xf3ff bytes) 000908 1790098088.955451 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x99109202 (stale) Mirror 14: 0x99109202 (stale) Mirror 17: 0x99109202 (stale) Mirror 18: 0x99109202 (stale) Mirror 19: 0x58ce5768 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0x99109202 (stale) Mirror 14: 0x99109202 (stale) Mirror 17: 0x99109202 (stale) Mirror 18: 0x99109202 (stale) Mirror 19: 0x58ce5768 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000910 1790098089.383570 mapwrite 0x01f70c thru 0x02b78f (0xc084 bytes) 000911 1790098089.552892 read 0x02c728 thru 0x0379d3 (0xb2ac bytes) 000912 1790098089.571888 write 0x0384c7 thru 0x039843 (0x137d bytes) 000913 1790098089.594543 write 0x02e952 thru 0x0309db (0x208a bytes) 000914 1790098089.597881 read 0x02fb5e thru 0x03a38d (0xa830 bytes) 000915 1790098089.608465 read 0x01ac00 thru 0x02345f (0x8860 bytes) 000916 1790098089.609214 trunc from 0x040000 to 0x00592b (0x3a6d4 bytes) 000917 1790098089.640494 write 0x013b5f thru 0x01be1b (0x82bd bytes) 000918 1790098089.643459 punch from 0x0002fb to 0x0017d6 (0x14dc bytes) 000919 1790098089.676767 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1be1c): Mirror 14: 0x7488dadd (stale) Mirror 17: 0x7488dadd (stale) Mirror 18: 0x7488dadd (stale) Mirror 19: 0x7488dadd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1be1c): Mirror 14: 0x7488dadd (stale) Mirror 17: 0x7488dadd (stale) Mirror 18: 0x7488dadd (stale) Mirror 19: 0x7488dadd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1be1c: skipped 000920 1790098090.177337 MIRROR_RESYNC 5 mirrors 000921 1790098091.573425 punch from 0x00d264 to 0x00e37a (0x1117 bytes) 000923 1790098091.640966 mapread 0x011b0d thru 0x01b8ca (0x9dbe bytes) 000924 1790098091.658875 mapread 0x001d7f thru 0x00d25e (0xb4e0 bytes) 000925 1790098091.667753 mapread 0x008548 thru 0x013717 (0xb1d0 bytes) 000926 1790098091.669812 write 0x0001ad thru 0x00140b (0x125f bytes) 000930 1790098091.677688 mapread 0x003ee9 thru 0x011895 (0xd9ad bytes) 000933 1790098091.680740 write 0x03a5e9 thru 0x03ab0e (0x526 bytes) 000934 1790098091.681941 trunc from 0x03ab0f to 0x00554e (0x355c0 bytes) 000935 1790098091.710779 write 0x03489a thru 0x03bf23 (0x768a bytes) 000937 1790098091.717133 punch from 0x01b056 to 0x027d7c (0xcd27 bytes) 000938 1790098091.731972 write 0x02e09d thru 0x035d15 (0x7c79 bytes) 000940 1790098091.738636 punch from 0x0008bc to 0x00f8a7 (0xefec bytes) 000941 1790098091.754743 trunc from 0x03bf24 to 0x03722b (0x4cf8 bytes) 000942 1790098091.770076 mapwrite 0x0100fc thru 0x0131c5 (0x30ca bytes) 000943 1790098091.856354 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3722c): Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3722c): Mirror 14: 0x1a6c926 (stale) Mirror 17: 0x1a6c926 (stale) Mirror 18: 0x1a6c926 (stale) Mirror 19: 0x1a6c926 (stale) CRC-32 checksum value for chunk [0, 0x3722c): Mirror 1: 0xb4fc5420 Mirror 20: 0xb4fc5420 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3722c: skipped 000945 1790098093.124038 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3722c): Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3722c): Mirror 14: 0x1a6c926 (stale) Mirror 17: 0x1a6c926 (stale) Mirror 18: 0x1a6c926 (stale) Mirror 19: 0x1a6c926 (stale) CRC-32 checksum value for chunk [0, 0x3722c): Mirror 1: 0xb4fc5420 Mirror 20: 0xb4fc5420 Mirror 21: 0xb4fc5420 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3722c: skipped 000946 1790098094.316113 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3722c): Mirror 14: 0x1a6c926 (stale) Mirror 17: 0x1a6c926 (stale) Mirror 18: 0x1a6c926 (stale) Mirror 19: 0x1a6c926 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3722c): Mirror 14: 0x1a6c926 (stale) Mirror 17: 0x1a6c926 (stale) Mirror 18: 0x1a6c926 (stale) Mirror 19: 0x1a6c926 (stale) CRC-32 checksum value for chunk [0, 0x3722c): Mirror 1: 0xb4fc5420 Mirror 20: 0xb4fc5420 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3722c: skipped 000947 1790098095.204523 punch from 0x00bf05 to 0x010c7d (0x4d79 bytes) 000949 1790098095.390579 punch from 0x01334d to 0x016d11 (0x39c5 bytes) 000951 1790098095.400855 trunc from 0x03722c to 0x03a425 (0x31fa bytes) 000952 1790098095.457670 mapwrite 0x038bcd thru 0x03d633 (0x4a67 bytes) 000953 1790098095.728571 punch from 0x03c023 to 0x03d633 (0x1611 bytes) 000954 1790098095.738736 mapwrite 0x0093ff thru 0x00b49f (0x20a1 bytes) 000955 1790098095.892494 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d634): Mirror 14: 0x1f7048c6 (stale) Mirror 17: 0x1f7048c6 (stale) Mirror 19: 0x1f7048c6 (stale) Mirror 20: 0x55bc241b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d634): Mirror 14: 0x1f7048c6 (stale) Mirror 17: 0x1f7048c6 (stale) Mirror 19: 0x1f7048c6 (stale) Mirror 20: 0x55bc241b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d634: skipped 000956 1790098096.342084 write 0x031710 thru 0x0398ff (0x81f0 bytes) 000957 1790098096.352566 write 0x00b6db thru 0x0176fa (0xc020 bytes) 000958 1790098096.362470 read 0x0140d6 thru 0x0220c6 (0xdff1 bytes) 000959 1790098096.376279 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d634): Mirror 17: 0x1f7048c6 (stale) Mirror 19: 0x1f7048c6 (stale) Mirror 20: 0x55bc241b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d634): Mirror 17: 0x1f7048c6 (stale) Mirror 19: 0x1f7048c6 (stale) Mirror 20: 0x55bc241b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d634: skipped 000960 1790098096.691569 read 0x025982 thru 0x031d95 (0xc414 bytes) 000961 1790098096.699613 write 0x0324b4 thru 0x03ffff (0xdb4c bytes) 000963 1790098096.705146 trunc from 0x040000 to 0x00b355 (0x34caa bytes) 000964 1790098096.718500 read 0x00a383 thru 0x00b355 (0xfd3 bytes) 000965 1790098096.736115 punch from 0x0022d8 to 0x005c49 (0x3972 bytes) 000966 1790098096.740413 punch from 0x002849 to 0x005252 (0x2a0a bytes) 000967 1790098096.744509 trunc from 0x00b356 to 0x0228e8 (0x17593 bytes) 000969 1790098096.759218 mapread 0x00b922 thru 0x00ba51 (0x130 bytes) 000972 1790098096.762508 mapwrite 0x012810 thru 0x020f8d (0xe77e bytes) 000975 1790098096.833878 mapread 0x0218e0 thru 0x0228e8 (0x1009 bytes) 000976 1790098096.834988 punch from 0x021bb9 to 0x0228e8 (0xd30 bytes) 000977 1790098096.839416 trunc from 0x0228e9 to 0x03e4a8 (0x1bbc0 bytes) 000978 1790098096.854330 mapwrite 0x02f44c thru 0x037642 (0x81f7 bytes) 000979 1790098096.900869 mapwrite 0x02c96e thru 0x03aed1 (0xe564 bytes) 000981 1790098096.947869 trunc from 0x03e4a9 to 0x03c5cf (0x1ed9 bytes) 000982 1790098096.964950 mapread 0x009e5d thru 0x018730 (0xe8d4 bytes) 000983 1790098096.970304 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x23a:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c5d0): Mirror 17: 0x6b93da4d (stale) Mirror 20: 0x3f7e4852 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c5d0): Mirror 17: 0x6b93da4d (stale) Mirror 20: 0x3f7e4852 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c5d0: skipped 000984 1790098097.287544 read 0x028470 thru 0x02dbdc (0x576d bytes) 000985 1790098097.296561 punch from 0x01e2b8 to 0x02cbb1 (0xe8fa bytes) 000990 1790098097.302215 mapwrite 0x0337f9 thru 0x035536 (0x1d3e bytes) 000991 1790098097.355351 write 0x033234 thru 0x037160 (0x3f2d bytes) 000992 1790098097.356363 mapwrite 0x02ffb3 thru 0x03208f (0x20dd bytes) 000993 1790098097.398360 write 0x033210 thru 0x03834d (0x513e bytes) 000994 1790098097.399763 read 0x01714b thru 0x0232f8 (0xc1ae bytes) 000995 1790098097.423750 mapread 0x017077 thru 0x0245e9 (0xd573 bytes) 000996 1790098097.424815 write 0x00d67c thru 0x01ca16 (0xf39b bytes) 000997 1790098097.430274 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c5d0): Mirror 17: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c5d0): Mirror 17: 0x6b93da4d (stale) Mirror 20: 0x3f7e4852 (stale) CRC-32 checksum value for chunk [0, 0x3c5d0): Mirror 1: 0x45251bb Mirror 21: 0x45251bb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c5d0: skipped 000998 1790098098.160797 read 0x023702 thru 0x02d6f6 (0x9ff5 bytes) All operations completed A-OK! PASS 70a (113s) == sanity-flr test 71: check mirror extend parent fid ==== 13:28:24 (1790098104) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 13:28:28 (1790098108) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x350:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x334:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.206332 s, 10.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x350:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x351:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x334:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157312 s, 6.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x350:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x351:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x335:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x334:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x350:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x351:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x335:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x334:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x352:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (9s) == sanity-flr test 200a: stress test ===================== 13:28:37 (1790098117) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre2 Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6808704 .. Extending file size to 7553504 .. Extending file size to 7388192 .. Extending file size to 8648576 .. Extending file size to 7581472 .. Extending file size to 8752544 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8087616 .. Extending file size to 8426432 .. Extending file size to 8731008 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8767072 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8835712 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8982624 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 58207 58208 58209 58211 58213 192.168.202.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg201-client.virtnet /mnt/lustre2 (opts:) 192.168.202.101@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg201-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 13:29:57 (1790098197) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre2 Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1119168 .. Extending file size to 3632704 .. Extending file size to 7522208 .. Extending file size to 9034208 .. Extending file size to 4223776 .. Extending file size to 5716256 .. Extending file size to 6217280 .. Extending file size to 6564224 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8177664 .. Extending file size to 8510752 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8793056 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9004160 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 59837 59838 59839 59840 59843 failed 192.168.202.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg201-client.virtnet /mnt/lustre2 (opts:) 192.168.202.101@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg201-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (84s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 13:31:21 (1790098281) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.146802 s, 14.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.30826 s, 6.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x95d3eec (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg201-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 13:31:44 (1790098304) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12036 s, 17.4 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 13:31:51 (1790098311) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x28c:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34c:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x367:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 13:31:58 (1790098318) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.313187 s, 6.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x368:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x369:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 13:32:06 (1790098326) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170535 s, 12.3 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x34f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36a:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36b:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 13:32:14 (1790098334) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.265267 s, 7.9 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x350:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x350:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36d:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 13:32:47 (1790098367) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg201-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107637 s, 9.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x351:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x351:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x36e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x352:0x0] } PASS 204d (55s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 13:33:42 (1790098422) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg201-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x370:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x371:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.244358 s, 8.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x370:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x371:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x353:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x370:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x371:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x353:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 13:34:38 (1790098478) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg201-server mds-ost sync done. Creating new pool test_204f oleg201-server: Pool lustre.test_204f created Adding targets to pool oleg201-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg201-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.24845 s, 8.4 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x372:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x354:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x372:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x354:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg201-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg201-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg201-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 13:35:53 (1790098553) PASS 205a (6s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 13:35:59 (1790098559) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 13:36:07 (1790098567) oleg201-server: Pool lustre.test_206 created Adding targets to pool oleg201-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg201-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg201-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg201-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg201-server: Pool lustre.test_206 destroyed PASS 206 (23s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 13:36:30 (1790098590) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782779 s, 13.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379173 s, 27.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x467913c2 Mirror 3: 0x467913c2 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 13:36:40 (1790098600) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 13:36:43 (1790098603) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 13:36:46 (1790098606) Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.12208 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1080000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2660000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2660000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2660000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1210000 Read this (4 bytes): BBBB 192.168.202.101@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg201-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 13:37:20 (1790098640) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0299828 s, 273 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 13:37:26 (1790098646) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0325779 s, 32.2 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x2b1 lmm_fid: [0x200000404:0x2b1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x2b1 lmm_fid: [0x200000404:0x2b1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x35e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834609 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-22 13:37:27.000000000 -0400 Modify: 2026-09-22 13:37:27.000000000 -0400 Change: 2026-09-22 13:37:28.000000000 -0400 Birth: 2026-09-22 13:37:27.000000000 -0400 PASS 210a (7s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 13:37:33 (1790098653) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 13:37:37 (1790098657) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.487407 s, 84.0 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0391362 s, 105 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 22 13:37 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x2b3:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 22 13:37 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 13:37:44 (1790098664) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 13:37:52 (1790098672) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1812 sec =========== 13:38:00 (1790098680) === sanity-flr: start cleanup 13:38:01 (1790098681) === === sanity-flr: finish cleanup 13:38:07 (1790098687) ===