-----============= acceptance-small: sanity-flr ============----- Tue Jun 30 20:31:03 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg336-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 20:31:10 (1782865870) === oleg336-client.virtnet: executing check_config_client /mnt/lustre oleg336-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg336-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa0f088d48000.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0f088d48000.idle_timeout=debug disable quota as required oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 20:31:22 (1782865882) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8235] == sanity-flr test 0a: lfs mirror create with -N option == 20:31:23 (1782865883) PASS 0a (5s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 20:31:28 (1782865888) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (1s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 20:31:29 (1782865889) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (1s) == sanity-flr test 0d: lfs mirror extend with -N option == 20:31:30 (1782865890) PASS 0d (7s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 20:31:37 (1782865897) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (1s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 20:31:38 (1782865898) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (1s) == sanity-flr test 0g: lfs mirror create flags support === 20:31:39 (1782865899) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffffa0f088d48000.stats=clear osc.lustre-OST0001-osc-ffffa0f088d48000.stats=clear PASS 0g (2s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 20:31:42 (1782865902) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (4s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 20:31:45 (1782865905) PASS 0j (2s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 20:31:47 (1782865907) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (1s) == sanity-flr test 0m: mirror extend with -N= option ===== 20:31:48 (1782865908) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (3s) == sanity-flr test 1: create components with setstripe options ========================================================== 20:31:51 (1782865911) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (3s) == sanity-flr test 2: create components from existing files ========================================================== 20:31:54 (1782865914) PASS 2 (2s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 20:31:56 (1782865916) PASS 3 (2s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 20:31:58 (1782865918) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (2s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 20:32:00 (1782865920) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (2s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 20:32:03 (1782865923) PASS 7 (3s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 20:32:06 (1782865926) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0431758 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209104 s, 50.1 MB/s PASS 21 (2s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 20:32:08 (1782865928) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0969433 s, 108 MB/s PASS 22 (2s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 20:32:10 (1782865930) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (3s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 20:32:13 (1782865933) Stopping /mnt/lustre-ost1 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0f088d48000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0f088d48000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg336-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa0f088d48000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0f088d48000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg336-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (22s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 20:32:35 (1782865955) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.138231 s, 53.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0f088d48000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0f088d48000.ost_server_uuid in DISCONN state after 2 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg336-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (15s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 20:32:50 (1782865970) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0f088d48000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0f088d48000.ost_server_uuid in DISCONN state after 3 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg336-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa0f088d48000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0f088d48000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg336-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (26s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 20:33:16 (1782865996) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.762842 s, 68.7 MB/s -rw-r--r-- 1 root root 52428800 Jun 30 20:33 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.227s user 0m0.001s sys 0m0.136s Stopping /mnt/lustre-ost1 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffffa0f088d48000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffffa0f088d48000.ost_server_uuid in DISCONN state after 2 sec reading file (data should be provided by ost2)... real 0m0.263s user 0m0.001s sys 0m0.161s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg336-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg336-server oleg336-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffffa0f088d48000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0f088d48000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.259s user 0m0.000s sys 0m0.137s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg336-server: oleg336-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg336-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg336-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0f088d48000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffffa0f088d48000.ost_server_uuid in FULL state after 0 sec PASS 33b (28s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 20:33:44 (1782866024) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (1s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 20:33:45 (1782866025) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 20:33:46 (1782866026) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 20:33:47 (1782866027) PASS 35 (2s) == sanity-flr test 36a: write to mirrored files ========== 20:33:49 (1782866029) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.137257 s, 91.7 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (4s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 20:33:53 (1782866033) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011038 s, 95.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154776 s, 67.7 MB/s PASS 36b (2s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 20:33:55 (1782866035) PASS 36c (3s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 20:33:58 (1782866038) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000564117 s, 1.8 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000535803 s, 1.9 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000515761 s, 2.0 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000486631 s, 2.1 MB/s flv=10 olv=10 PASS 36d (2s) == sanity-flr test 37: mirror I/O API verification ======= 20:34:00 (1782866040) create test files with size 15744900 .. f1be4a31c59d1f7bd29ffd51cbaa46da - 71d38784df20e6bfd88d306321735e32 - 9ab04ca12323a78aa4fca95638f9f010 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg336-client.virtnet /mnt/lustre (opts:) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre PASS 37 (5s) == sanity-flr test 38: resync ============================ 20:34:05 (1782866045) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (4s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 20:34:09 (1782866049) PASS 39 (2s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 20:34:11 (1782866051) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.026804 s, 39.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0144404 s, 72.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (3s) == sanity-flr test 41: lfs mirror resync check =========== 20:34:14 (1782866054) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.025618 s, 81.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.0891336 s, 47.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0484085 s, 43.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (4s) == sanity-flr test 42: lfs mirror verify ================= 20:34:18 (1782866058) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (1s) == sanity-flr test 43a: mirror pick on write ============= 20:34:19 (1782866059) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 20:34:20 (1782866060) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130625 s, 80.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 20:34:23 (1782866063) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0794308 s, 39.6 MB/s PASS 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 20:34:26 (1782866066) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (2s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 20:34:29 (1782866069) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.108476 s, 96.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg336-client.virtnet /mnt/lustre (opts:) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (19s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 20:34:48 (1782866088) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffffa0f090c5c800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0970872 s, 108 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg336-client.virtnet /mnt/lustre (opts:) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffffa0f088d49800.xattr_cache=1 PASS 44e (4s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 20:34:52 (1782866092) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (10s) == sanity-flr test 46: Verify setstripe --copy option ==== 20:35:02 (1782866102) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (10s) == sanity-flr test 47: Verify mirror obj alloc =========== 20:35:12 (1782866112) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 20:35:13 (1782866113) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0723038 s, 43.5 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0534852 s, 58.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x1cca136c (nosync,stale) Mirror 2: 0x1cca136c (nosync,stale) ** verify mirror 2 doesn't change original checksum: 7c3d5332f858551d5dbeaed8ff84a8cf - mirror 1 checksum: d9315a1d2c681e2ed7c9ca8050a868cd - mirror 2 checksum: 7c3d5332f858551d5dbeaed8ff84a8cf - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-30T20:35:13-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-30T20:35:13-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 7c3d5332f858551d5dbeaed8ff84a8cf - mirror 1 checksum: d9315a1d2c681e2ed7c9ca8050a868cd - mirror 2 checksum: d9315a1d2c681e2ed7c9ca8050a868cd - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-30T20:35:13-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-30T20:35:13-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcc:0x0] } PASS 48 (3s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 20:35:16 (1782866116) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106411 s, 98.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (2s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 20:35:18 (1782866118) Failing mds1 on oleg336-server Stopping /mnt/lustre-mds1 (opts:) on oleg336-server 20:35:25 (1782866125) shut down facet: mds1 facet_host: oleg336-server facet_failover_host: oleg336-server Failover mds1 to oleg336-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg336-server: oleg336-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0000 20:35:39 (1782866139) targets are mounted 20:35:39 (1782866139) facet_failover done oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.009555 s, 110 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } PASS 50A (30s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 20:35:48 (1782866148) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00919141 s, 2.2 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00461955 s, 4.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (3s) == sanity-flr test 50b: mirror rsync handles sparseness == 20:35:51 (1782866151) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00948915 s, 2.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00305584 s, 6.7 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (3s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 20:35:54 (1782866154) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00267338 s, 6.1 MB/s PASS 50c (2s) == sanity-flr test 50d: mirror rsync keep holes ========== 20:35:56 (1782866156) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00913758 s, 2.2 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 20:35:59 (1782866159) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00183018 s, 2.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00846328 s, 484 kB/s PASS 60a (2s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 20:36:02 (1782866162) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (5s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 20:36:07 (1782866167) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (19s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 20:36:26 (1782866186) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (17s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 20:36:43 (1782866203) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 20:37:01 (1782866221) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 20:37:03 (1782866223) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57296 mirror split pid 57297 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57296 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5040: 57297 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (62s) == sanity-flr test 70a: flr mode fsx test ================ 20:38:05 (1782866285) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 6286 000001 1782866286.553653 mapwrite 0x0395b0 thru 0x03f731 (0x6182 bytes) 000004 1782866286.570270 punch from 0x035327 to 0x03f731 (0xa40b bytes) 000005 1782866286.571934 write 0x023b23 thru 0x0254fb (0x19d9 bytes) 000006 1782866286.574814 punch from 0x021516 to 0x025cc9 (0x47b4 bytes) 000007 1782866286.577720 trunc from 0x035ae1 to 0x03f731 (0x9c51 bytes) 000008 1782866286.579199 mapread 0x03d4c8 thru 0x03f731 (0x226a bytes) 000009 1782866286.581958 punch from 0x0355fc to 0x03d67e (0x8083 bytes) 000010 1782866286.583707 write 0x026140 thru 0x0311f6 (0xb0b7 bytes) 000011 1782866286.587139 write 0x006656 thru 0x00f655 (0x9000 bytes) 000012 1782866286.590261 read 0x031d7d thru 0x03f731 (0xd9b5 bytes) 000013 1782866286.592926 mapread 0x029d8d thru 0x02f1e7 (0x545b bytes) 000014 1782866286.593730 read 0x01d477 thru 0x01ff3c (0x2ac6 bytes) zero_range to largest ever: 0x3f732 000015 1782866286.596797 trunc from 0x0329a5 to 0x03f731 (0xcd8d bytes) 000016 1782866286.598309 fallocate 0x02d3fa thru 0x0348ad (0x74b4 bytes) 000017 1782866286.600005 punch from 0x013b9b to 0x01e1af (0xa615 bytes) 000018 1782866286.601588 punch from 0x02d5c9 to 0x03155c (0x3f94 bytes) 000019 1782866286.605781 mapread 0x02933e thru 0x038950 (0xf613 bytes) 000020 1782866286.620949 mapread 0x00c1aa thru 0x01960c (0xd463 bytes) 000022 1782866286.630211 fallocate 0x00066c thru 0x003892 (0x3227 bytes) 000023 1782866286.633304 MIRROR_EXTEND to 2 mirrors 000024 1782866286.817906 punch from 0x032d26 to 0x038ef7 (0x61d2 bytes) 000025 1782866286.828127 fallocate 0x002a3d thru 0x009e31 (0x73f5 bytes) 000026 1782866286.829956 trunc from 0x03f732 to 0x03f6ab (0x86 bytes) 000027 1782866286.839676 mapread 0x0331d1 thru 0x03f6ab (0xc4db bytes) 000028 1782866286.846581 fallocate 0x032400 thru 0x03b3a0 (0x8fa1 bytes) 000029 1782866286.848318 punch from 0x02ec67 to 0x030518 (0x18b2 bytes) 000030 1782866286.849881 punch from 0x02dfec to 0x031525 (0x353a bytes) 000031 1782866286.851426 write 0x01cca3 thru 0x01d27d (0x5db bytes) 000032 1782866286.854211 mapread 0x015e56 thru 0x021d08 (0xbeb3 bytes) 000033 1782866286.859263 mapread 0x027d9a thru 0x0305ec (0x8853 bytes) 000034 1782866286.863672 mapwrite 0x02347c thru 0x0290fa (0x5c7f bytes) 000035 1782866286.869983 mapwrite 0x02332c thru 0x03076d (0xd442 bytes) 000036 1782866286.876899 trunc from 0x03f6ac to 0x0157e5 (0x29ec6 bytes) 000037 1782866286.881344 mapwrite 0x03d0b2 thru 0x03f60b (0x255a bytes) 000038 1782866286.892356 mapread 0x00a95d thru 0x019f28 (0xf5cc bytes) 000039 1782866286.897719 write 0x00e366 thru 0x01741d (0x90b8 bytes) 000040 1782866286.899151 fallocate 0x02a711 thru 0x02b787 (0x1077 bytes) 000042 1782866286.901643 write 0x03947b thru 0x03ffff (0x6b85 bytes) 000043 1782866286.902356 punch from 0x00611d to 0x009d1e (0x3c02 bytes) 000044 1782866286.904117 mapread 0x0218b5 thru 0x022d38 (0x1484 bytes) 000046 1782866286.907347 trunc from 0x040000 to 0x015665 (0x2a99a bytes) 000047 1782866286.911684 trunc from 0x001db0 to 0x00df88 (0xc1d9 bytes) 000051 1782866286.914509 read 0x000bc0 thru 0x006cd8 (0x6119 bytes) 000052 1782866286.917111 trunc from 0x006ebb to 0x015665 (0xe7ab bytes) 000053 1782866286.921211 trunc from 0x010eae to 0x015665 (0x47b8 bytes) 000054 1782866286.922960 mapwrite 0x025193 thru 0x02c485 (0x72f3 bytes) 000055 1782866286.936431 mapread 0x005f3e thru 0x0153ab (0xf46e bytes) 000056 1782866286.943591 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000057 1782866286.978181 trunc from 0x00c546 to 0x018324 (0xbddf bytes) 000058 1782866286.979977 mapread 0x0283ff thru 0x02b3fc (0x2ffe bytes) 000059 1782866286.985333 read 0x0195ef thru 0x01f0f2 (0x5b04 bytes) 000060 1782866286.988181 trunc from 0x0090df to 0x012a92 (0x99b4 bytes) 000061 1782866286.989834 fallocate 0x000f59 thru 0x0059c2 (0x4a6a bytes) 000062 1782866286.991604 fallocate 0x033638 thru 0x03573c (0x2105 bytes) 000063 1782866286.993258 trunc from 0x01b10a to 0x02094a (0x5841 bytes) 000064 1782866286.994889 fallocate 0x01f713 thru 0x021ba9 (0x2497 bytes) 000065 1782866286.996719 mapwrite 0x031d54 thru 0x03ca4e (0xacfb bytes) 000066 1782866287.012475 fallocate 0x023a60 thru 0x031b48 (0xe0e9 bytes) 000069 1782866287.015026 fallocate 0x007b43 thru 0x0113e6 (0x98a4 bytes) 000070 1782866287.016684 mapwrite 0x03d248 thru 0x03ffff (0x2db8 bytes) 000071 1782866287.025690 trunc from 0x00616d to 0x0082bc (0x2150 bytes) 000072 1782866287.027167 read 0x01c1a0 thru 0x02a62b (0xe48c bytes) 000073 1782866287.031552 MIRROR_EXTEND to 2 mirrors 000074 1782866287.203559 trunc from 0x040000 to 0x0127c8 (0x2d837 bytes) 000075 1782866287.220308 trunc from 0x0060d3 to 0x0127c8 (0xc6f6 bytes) 000076 1782866287.222150 punch from 0x01243e to 0x0127c8 (0x38b bytes) 000077 1782866287.223838 write 0x02fef8 thru 0x0341ab (0x42b4 bytes) 000078 1782866287.224834 trunc from 0x0341ac to 0x0334bf (0xcec bytes) 000079 1782866287.231834 MIRROR_RESYNC 2 mirrors 000080 1782866287.385040 trunc from 0x0212ef to 0x029fa3 (0x8cb5 bytes) 000081 1782866287.393818 mapwrite 0x0335a0 thru 0x03e989 (0xb3ea bytes) 000082 1782866287.414467 mapread 0x02831c thru 0x02866c (0x351 bytes) 000083 1782866287.417160 mapwrite 0x039c6d thru 0x03ef40 (0x52d4 bytes) 000084 1782866287.426202 trunc from 0x03ef41 to 0x0221c8 (0x1cd78 bytes) 000085 1782866287.430181 trunc from 0x0221c9 to 0x032295 (0x100cd bytes) 000086 1782866287.433764 read 0x011fa0 thru 0x01b48d (0x94ee bytes) 000087 1782866287.436471 mapwrite 0x0044d3 thru 0x008d20 (0x484e bytes) 000088 1782866287.443824 trunc from 0x00f09f to 0x0188be (0x9820 bytes) 000089 1782866287.445629 punch from 0x019d90 to 0x01fb3f (0x5db0 bytes) 000090 1782866287.447177 read 0x007d97 thru 0x010200 (0x846a bytes) 000091 1782866287.450368 trunc from 0x02a78e to 0x032295 (0x7b08 bytes) 000092 1782866287.453190 MIRROR_RESYNC 2 mirrors 000093 1782866287.613128 fallocate 0x01cdbf thru 0x0242ce (0x7510 bytes) 000094 1782866287.621509 write 0x0170ea thru 0x0255aa (0xe4c1 bytes) 000095 1782866287.624823 write 0x03b528 thru 0x03ffff (0x4ad8 bytes) 000096 1782866287.626919 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000097 1782866287.660445 punch from 0x00198c to 0x00ffc7 (0xe63c bytes) 000098 1782866287.662573 trunc from 0x040000 to 0x03345f (0xcba0 bytes) 000099 1782866287.672368 mapwrite 0x01148d thru 0x01b9da (0xa54e bytes) 000100 1782866287.683621 read 0x01940c thru 0x01b294 (0x1e89 bytes) 000102 1782866287.684736 mapwrite 0x00184a thru 0x00fc67 (0xe41e bytes) 000103 1782866287.693922 trunc from 0x01b86a to 0x02575c (0x9ef3 bytes) 000104 1782866287.695588 trunc from 0x033460 to 0x001a96 (0x319c9 bytes) 000105 1782866287.699999 trunc from 0x001a97 to 0x022a91 (0x20ffb bytes) 000107 1782866287.704206 write 0x0168a8 thru 0x016d85 (0x4de bytes) 000108 1782866287.706044 write 0x01a293 thru 0x01a91a (0x688 bytes) 000109 1782866287.707834 trunc from 0x002300 to 0x009f14 (0x7c15 bytes) 000110 1782866287.709305 punch from 0x01c9f8 to 0x01e989 (0x1f92 bytes) 000111 1782866287.710865 fallocate 0x021728 thru 0x029b29 (0x8402 bytes) 000112 1782866287.712712 read 0x01a0f4 thru 0x022a91 (0x899e bytes) 000113 1782866287.715655 mapread 0x015bb8 thru 0x0220ea (0xc533 bytes) 000114 1782866287.716471 write 0x004b1a thru 0x006d17 (0x21fe bytes) 000115 1782866287.717117 trunc from 0x022a92 to 0x02d38c (0xa8fb bytes) 000116 1782866287.721119 mapread 0x019859 thru 0x020467 (0x6c0f bytes) 000117 1782866287.721831 read 0x016d99 thru 0x01f564 (0x87cc bytes) 000118 1782866287.722256 mapwrite 0x028938 thru 0x02b5ac (0x2c75 bytes) 000120 1782866287.730142 trunc from 0x012b86 to 0x017006 (0x4481 bytes) 000122 1782866287.732187 mapread 0x01744a thru 0x01d2c0 (0x5e77 bytes) 000124 1782866287.734524 trunc from 0x025e7c to 0x02d38c (0x7511 bytes) 000126 1782866287.736430 trunc from 0x02d38d to 0x03b1c1 (0xde35 bytes) 000127 1782866287.740519 trunc from 0x01df9e to 0x026946 (0x89a9 bytes) 000128 1782866287.742154 trunc from 0x03b1c2 to 0x03227f (0x8f42 bytes) 000130 1782866287.748741 read 0x00a6fe thru 0x00d77f (0x3082 bytes) 000131 1782866287.749167 mapwrite 0x0141a9 thru 0x01de96 (0x9cee bytes) 000132 1782866287.757962 mapwrite 0x0300ed thru 0x0349d9 (0x48ed bytes) 000133 1782866287.768356 punch from 0x0025a3 to 0x007021 (0x4a7f bytes) 000134 1782866287.770256 mapwrite 0x008bdc thru 0x018bb6 (0xffdb bytes) 000135 1782866287.777679 fallocate 0x026595 thru 0x02ecee (0x875a bytes) fallocating to largest ever: 0x40000 000136 1782866287.779532 fallocate 0x039153 thru 0x03ffff (0x6ead bytes) 000138 1782866287.781857 read 0x0392ef thru 0x03ffff (0x6d11 bytes) 000139 1782866287.784807 punch from 0x0387c3 to 0x03ffff (0x783d bytes) 000140 1782866287.786287 read 0x03de4a thru 0x03ffff (0x21b6 bytes) 000142 1782866287.789275 mapwrite 0x0153c2 thru 0x01b598 (0x61d7 bytes) 000143 1782866287.794938 fallocate 0x02bc55 thru 0x037faf (0xc35b bytes) 000144 1782866287.796797 read 0x03e148 thru 0x03ffff (0x1eb8 bytes) 000145 1782866287.797192 trunc from 0x040000 to 0x02ef1f (0x110e0 bytes) 000146 1782866287.801412 mapwrite 0x0092fe thru 0x00a58d (0x1290 bytes) 000147 1782866287.805602 punch from 0x0057af to 0x0071d6 (0x1a28 bytes) 000148 1782866287.807261 trunc from 0x02ef20 to 0x01024a (0x1ecd5 bytes) 000150 1782866287.811923 fallocate 0x01fe3b thru 0x0242e5 (0x44ab bytes) 000151 1782866287.813930 trunc from 0x01fe6b to 0x0242e5 (0x447b bytes) 000152 1782866287.815370 punch from 0x01e5c8 to 0x021180 (0x2bb9 bytes) 000153 1782866287.816993 fallocate 0x0327f0 thru 0x03aa27 (0x8238 bytes) 000154 1782866287.818714 mapread 0x01afef thru 0x01f217 (0x4229 bytes) 000155 1782866287.821976 punch from 0x01ce7b to 0x01f3dd (0x2563 bytes) 000156 1782866287.823446 trunc from 0x0242e6 to 0x00aa9d (0x19848 bytes) 000157 1782866287.829465 MIRROR_EXTEND to 2 mirrors 000158 1782866288.011648 write 0x03898b thru 0x03ffff (0x7675 bytes) 000159 1782866288.021909 fallocate 0x02cea1 thru 0x036c1c (0x9d7c bytes) 000160 1782866288.023781 write 0x00d023 thru 0x0148c7 (0x78a5 bytes) 000161 1782866288.027153 write 0x03ba96 thru 0x03ffff (0x456a bytes) 000162 1782866288.027870 write 0x02d0ed thru 0x03159f (0x44b3 bytes) 000163 1782866288.030758 trunc from 0x040000 to 0x024473 (0x1bb8c bytes) 000164 1782866288.040258 fallocate 0x00c312 thru 0x010b2c (0x481b bytes) 000165 1782866288.042095 punch from 0x007fb4 to 0x0128d5 (0xa922 bytes) 000166 1782866288.046004 mapwrite 0x0086bc thru 0x01142c (0x8d71 bytes) 000167 1782866288.056949 fallocate 0x0385bf thru 0x03d7ef (0x5231 bytes) 000168 1782866288.058800 write 0x006766 thru 0x01263f (0xbeda bytes) 000169 1782866288.059869 write 0x0007e4 thru 0x00dde9 (0xd606 bytes) 000170 1782866288.061970 mapread 0x029cf3 thru 0x02c3c7 (0x26d5 bytes) 000171 1782866288.064990 trunc from 0x03d7f0 to 0x005198 (0x38657 bytes) 000172 1782866288.069665 mapread 0x004756 thru 0x004f1b (0x7c6 bytes) 000173 1782866288.070784 trunc from 0x005199 to 0x034e63 (0x2fccb bytes) 000174 1782866288.075071 read 0x02caf2 thru 0x032198 (0x56a7 bytes) 000175 1782866288.077695 write 0x01f139 thru 0x02e72e (0xf5f6 bytes) 000176 1782866288.080056 mapread 0x003ed7 thru 0x00811b (0x4245 bytes) 000177 1782866288.082916 mapwrite 0x00b3bc thru 0x01a539 (0xf17e bytes) 000178 1782866288.096066 trunc from 0x034e64 to 0x039f8d (0x512a bytes) 000179 1782866288.100723 fallocate 0x03ec12 thru 0x03ffff (0x13ee bytes) 000180 1782866288.102619 trunc from 0x018ca8 to 0x0258ba (0xcc13 bytes) 000181 1782866288.104616 fallocate 0x024265 thru 0x0328ba (0xe656 bytes) 000182 1782866288.106727 trunc from 0x039f8e to 0x033dda (0x61b3 bytes) 000183 1782866288.110773 mapread 0x00164e thru 0x010323 (0xecd6 bytes) 000184 1782866288.112339 trunc from 0x033ddb to 0x025082 (0xed58 bytes) 000185 1782866288.117217 write 0x02be9e thru 0x037d8f (0xbef2 bytes) 000187 1782866288.118775 write 0x0357f8 thru 0x039998 (0x41a1 bytes) 000188 1782866288.119585 trunc from 0x01d3fb to 0x01db66 (0x76c bytes) 000189 1782866288.121298 trunc from 0x022d77 to 0x02d6a6 (0xa930 bytes) 000190 1782866288.126739 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39999): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39999): Mirror 2: 0x608e06da (stale) CRC-32 checksum value for chunk [0, 0x39999): Mirror 1: 0x2f7c2ffe Mirror 3: 0x2f7c2ffe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39999: skipped 000191 1782866288.380373 trunc from 0x008b50 to 0x00b4c3 (0x2974 bytes) 000192 1782866288.390501 read 0x0316b0 thru 0x039998 (0x82e9 bytes) 000193 1782866288.393618 punch from 0x000e75 to 0x0072d9 (0x6465 bytes) 000194 1782866288.395378 fallocate 0x03419b thru 0x039ef7 (0x5d5d bytes) 000195 1782866288.398690 MIRROR_RESYNC 3 mirrors 000196 1782866288.597369 trunc from 0x02c270 to 0x039904 (0xd695 bytes) 000197 1782866288.606620 fallocate 0x028099 thru 0x0326cd (0xa635 bytes) 000198 1782866288.608426 punch from 0x011548 to 0x01acd0 (0x9789 bytes) 000199 1782866288.611478 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39999): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39999): Mirror 2: 0xe2d2e215 (stale) Mirror 3: 0xe2d2e215 (stale) CRC-32 checksum value for chunk [0, 0x39999): Mirror 1: 0x4dfb5321 Mirror 4: 0x4dfb5321 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39999: skipped 000200 1782866288.827947 trunc from 0x039999 to 0x01144b (0x2854d bytes) 000201 1782866288.845119 mapwrite 0x035333 thru 0x036fe3 (0x1cb1 bytes) 000202 1782866288.857947 trunc from 0x036fe4 to 0x03e95e (0x797b bytes) 000203 1782866288.862170 mapwrite 0x000b79 thru 0x003657 (0x2adf bytes) 000204 1782866288.868893 mapread 0x038f77 thru 0x03e95e (0x59e8 bytes) 000205 1782866288.871776 mapwrite 0x02edde thru 0x036f64 (0x8187 bytes) 000206 1782866288.879966 mapwrite 0x023564 thru 0x02c2cf (0x8d6c bytes) 000208 1782866288.888282 mapread 0x00435c thru 0x010651 (0xc2f6 bytes) 000209 1782866288.891135 trunc from 0x0060c3 to 0x007359 (0x1297 bytes) 000211 1782866288.893133 mapread 0x016b2c thru 0x01a58b (0x3a60 bytes) 000212 1782866288.895313 trunc from 0x03e95f to 0x02b71a (0x13244 bytes) 000213 1782866288.900331 mapwrite 0x0111c0 thru 0x0179b9 (0x67fa bytes) 000214 1782866288.906949 punch from 0x0087b1 to 0x0117d2 (0x9022 bytes) 000215 1782866288.909055 trunc from 0x02b71b to 0x004d67 (0x269b3 bytes) 000216 1782866288.913680 trunc from 0x002cea to 0x004d67 (0x207e bytes) 000217 1782866288.915342 mapread 0x002d1f thru 0x004d67 (0x2049 bytes) 000218 1782866288.917645 punch from 0x0024dd to 0x004d67 (0x288b bytes) 000219 1782866288.920632 MIRROR_RESYNC 4 mirrors 000220 1782866289.111612 punch from 0x0032f3 to 0x004d67 (0x1a75 bytes) 000221 1782866289.120167 punch from 0x003755 to 0x004d67 (0x1613 bytes) 000222 1782866289.121663 write 0x019fc4 thru 0x01d703 (0x3740 bytes) 000223 1782866289.122565 trunc from 0x01a162 to 0x01d703 (0x35a2 bytes) 000224 1782866289.126216 mapread 0x01cd1d thru 0x01d703 (0x9e7 bytes) 000225 1782866289.129683 punch from 0x0198a4 to 0x01d703 (0x3e60 bytes) 000226 1782866289.131375 fallocate 0x00d9b3 thru 0x00f85a (0x1ea8 bytes) 000227 1782866289.133292 punch from 0x014e7f to 0x01d703 (0x8885 bytes) 000228 1782866289.135082 punch from 0x00b6c8 to 0x017c3a (0xc573 bytes) 000229 1782866289.136844 punch from 0x00bc8a to 0x013b46 (0x7ebd bytes) 000230 1782866289.138363 punch from 0x0195bd to 0x01d703 (0x4147 bytes) 000231 1782866289.140085 fallocate 0x02e532 thru 0x03a83a (0xc309 bytes) 000232 1782866289.141875 fallocate 0x00d001 thru 0x017e53 (0xae53 bytes) 000233 1782866289.143713 mapwrite 0x0229a1 thru 0x030068 (0xd6c8 bytes) 000234 1782866289.155347 write 0x029c1c thru 0x034dc2 (0xb1a7 bytes) 000235 1782866289.157405 read 0x03876f thru 0x03a83a (0x20cc bytes) 000236 1782866289.159326 read 0x014932 thru 0x014951 (0x20 bytes) 000237 1782866289.162326 mapwrite 0x00d7bd thru 0x01c914 (0xf158 bytes) 000238 1782866289.169833 trunc from 0x017c3f to 0x01cb53 (0x4f15 bytes) 000239 1782866289.171668 write 0x034226 thru 0x03c311 (0x80ec bytes) 000240 1782866289.173973 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c312): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c312): Mirror 2: 0xa70da82d (stale) Mirror 3: 0xa70da82d (stale) Mirror 4: 0xa70da82d (stale) CRC-32 checksum value for chunk [0, 0x3c312): Mirror 1: 0x8feb3290 Mirror 5: 0x8feb3290 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c312: skipped 000241 1782866289.413740 punch from 0x0326a8 to 0x03be0e (0x9767 bytes) 000242 1782866289.423825 punch from 0x01ccae to 0x027eb7 (0xb20a bytes) 000243 1782866289.425686 fallocate 0x02febd thru 0x036574 (0x66b8 bytes) 000244 1782866289.427629 mapwrite 0x0257d1 thru 0x027a89 (0x22b9 bytes) 000245 1782866289.437859 trunc from 0x038e40 to 0x03c311 (0x34d2 bytes) 000246 1782866289.439372 trunc from 0x03bfec to 0x03c311 (0x326 bytes) 000247 1782866289.441004 read 0x020ff6 thru 0x026ad1 (0x5adc bytes) 000248 1782866289.441470 read 0x0012d1 thru 0x00975c (0x848c bytes) 000249 1782866289.444819 read 0x019f4a thru 0x0201fa (0x62b1 bytes) 000250 1782866289.445250 punch from 0x03aa85 to 0x03c311 (0x188d bytes) 000251 1782866289.447025 mapwrite 0x03476e thru 0x03ffff (0xb892 bytes) 000252 1782866289.466985 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x650339aa (stale) Mirror 3: 0x650339aa (stale) Mirror 4: 0x650339aa (stale) Mirror 5: 0xf4b3ec6b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2b4369d Mirror 6: 0x2b4369d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000253 1782866289.748467 read 0x03a98a thru 0x03f4ee (0x4b65 bytes) 000254 1782866289.752649 punch from 0x018977 to 0x027671 (0xecfb bytes) 000256 1782866289.763481 read 0x02ecaa thru 0x03c408 (0xd75f bytes) 000257 1782866289.766963 mapwrite 0x0338d1 thru 0x03b54b (0x7c7b bytes) 000258 1782866289.773771 punch from 0x00beb6 to 0x014b3e (0x8c89 bytes) 000259 1782866289.775729 trunc from 0x039425 to 0x03f894 (0x6470 bytes) 000260 1782866289.777475 trunc from 0x040000 to 0x003910 (0x3c6ef bytes) 000261 1782866289.788475 punch from 0x001a25 to 0x003910 (0x1eec bytes) 000262 1782866289.789981 trunc from 0x0038b0 to 0x003910 (0x61 bytes) 000263 1782866289.791567 trunc from 0x003911 to 0x01e523 (0x1ac13 bytes) 000264 1782866289.795400 trunc from 0x01e524 to 0x01b191 (0x3392 bytes) 000265 1782866289.799571 trunc from 0x01b192 to 0x001c6f (0x19522 bytes) 000266 1782866289.803567 read 0x001394 thru 0x001c6f (0x8dc bytes) 000268 1782866289.805902 punch from 0x000e1d to 0x001c6f (0xe53 bytes) 000270 1782866289.807841 trunc from 0x0014bb to 0x001c6f (0x7b5 bytes) 000271 1782866289.809470 read 0x0016d8 thru 0x001c6f (0x598 bytes) 000272 1782866289.812642 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c70): Mirror 2: 0xf3fe9c6f (stale) Mirror 3: 0xf3fe9c6f (stale) Mirror 4: 0xf3fe9c6f (stale) Mirror 6: 0xf3fe9c6f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c70): Mirror 2: 0xf3fe9c6f (stale) Mirror 3: 0xf3fe9c6f (stale) Mirror 4: 0xf3fe9c6f (stale) Mirror 6: 0xf3fe9c6f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c70: skipped 000273 1782866289.891615 mapread 0x0016d7 thru 0x001c6f (0x599 bytes) 000274 1782866289.893783 trunc from 0x001055 to 0x001c6f (0xc1b bytes) 000275 1782866289.895454 punch from 0x001bd5 to 0x001c6f (0x9b bytes) 000276 1782866289.896907 mapwrite 0x00abcf thru 0x00f651 (0x4a83 bytes) 000277 1782866289.910165 trunc from 0x00f652 to 0x029e1b (0x1a7ca bytes) 000278 1782866289.914304 write 0x03fde5 thru 0x03ffff (0x21b bytes) 000279 1782866289.915006 mapwrite 0x009789 thru 0x01257b (0x8df3 bytes) 000280 1782866289.923597 write 0x03baa0 thru 0x03ffff (0x4560 bytes) 000281 1782866289.925677 fallocate 0x0114b5 thru 0x01e89c (0xd3e8 bytes) 000282 1782866289.927582 trunc from 0x040000 to 0x03c99a (0x3665 bytes) 000283 1782866289.931870 fallocate 0x038bec thru 0x03ffff (0x7414 bytes) 000284 1782866289.933689 mapwrite 0x00f25e thru 0x01afae (0xbd51 bytes) 000285 1782866289.943986 fallocate 0x01aeb3 thru 0x0211ab (0x62f9 bytes) 000286 1782866289.946056 punch from 0x00fac2 to 0x013d1f (0x425e bytes) 000287 1782866289.948005 trunc from 0x03c99b to 0x0233df (0x195bb bytes) 000288 1782866289.952672 trunc from 0x01e3ed to 0x01e667 (0x27b bytes) 000289 1782866289.954560 read 0x0010f7 thru 0x005182 (0x408c bytes) 000290 1782866289.956723 trunc from 0x0233e0 to 0x020843 (0x2b9c bytes) 000291 1782866289.961217 mapwrite 0x001f78 thru 0x00427c (0x2305 bytes) 000292 1782866289.966349 fallocate 0x02c446 thru 0x034c8f (0x884a bytes) 000293 1782866289.969745 MIRROR_RESYNC 5 mirrors 000294 1782866290.302076 mapwrite 0x01001a thru 0x01f1bb (0xf1a2 bytes) 000295 1782866290.339612 trunc from 0x020844 to 0x01777f (0x90c4 bytes) 000296 1782866290.344864 mapwrite 0x01ac5c thru 0x01b526 (0x8cb bytes) 000297 1782866290.353985 fallocate 0x00793f thru 0x00c119 (0x47db bytes) 000298 1782866290.355903 mapwrite 0x02ba84 thru 0x036f11 (0xb48e bytes) 000299 1782866290.370674 fallocate 0x025e4f thru 0x02eb31 (0x8ce3 bytes) 000300 1782866290.372678 trunc from 0x00813a to 0x00af20 (0x2de7 bytes) 000301 1782866290.374559 trunc from 0x036f12 to 0x029425 (0xdaec bytes) 000302 1782866290.378799 trunc from 0x0221cc to 0x029425 (0x725a bytes) 000303 1782866290.380393 punch from 0x017a54 to 0x0273ca (0xf977 bytes) 000304 1782866290.382167 trunc from 0x029426 to 0x011bae (0x17877 bytes) 000305 1782866290.386328 fallocate 0x00082b thru 0x00b134 (0xa90a bytes) 000306 1782866290.388089 punch from 0x003890 to 0x00d8a8 (0xa019 bytes) 000307 1782866290.389804 read 0x0019f3 thru 0x004f75 (0x3583 bytes) 000309 1782866290.392204 write 0x029934 thru 0x02b2d5 (0x19a2 bytes) 000310 1782866290.393002 mapread 0x019286 thru 0x02317f (0x9efa bytes) 000311 1782866290.398117 mapread 0x00b745 thru 0x01a474 (0xed30 bytes) 000312 1782866290.401439 punch from 0x014441 to 0x01751e (0x30de bytes) 000313 1782866290.403153 fallocate 0x0066bc thru 0x00e629 (0x7f6e bytes) 000314 1782866290.405036 trunc from 0x00a23b to 0x0180e2 (0xdea8 bytes) 000315 1782866290.406753 punch from 0x0181f9 to 0x01f1d9 (0x6fe1 bytes) 000316 1782866290.408262 trunc from 0x02b2d6 to 0x01438c (0x16f49 bytes) 000318 1782866290.413043 trunc from 0x009dbe to 0x01438c (0xa5cf bytes) 000319 1782866290.414719 trunc from 0x00d7c7 to 0x0113a3 (0x3bdd bytes) 000320 1782866290.416466 trunc from 0x0004d0 to 0x0059e5 (0x5516 bytes) 000321 1782866290.419681 MIRROR_RESYNC 5 mirrors 000322 1782866290.625766 trunc from 0x01438d to 0x004d0e (0xf67e bytes) 000324 1782866290.641054 mapread 0x001ba3 thru 0x0049a3 (0x2e01 bytes) 000325 1782866290.645767 read 0x001104 thru 0x004d0e (0x3c0b bytes) 000326 1782866290.646293 trunc from 0x004d0f to 0x036501 (0x317f3 bytes) 000327 1782866290.651629 trunc from 0x00fee3 to 0x00ff39 (0x57 bytes) 000328 1782866290.654739 MIRROR_RESYNC 5 mirrors 000330 1782866290.857524 read 0x00d593 thru 0x0146a0 (0x710e bytes) 000331 1782866290.859928 punch from 0x02e51d to 0x035833 (0x7317 bytes) 000332 1782866290.868291 mapread 0x001341 thru 0x00676e (0x542e bytes) 000333 1782866290.873209 punch from 0x021c1b to 0x0288f2 (0x6cd8 bytes) 000334 1782866290.874699 write 0x028eef thru 0x03595d (0xca6f bytes) 000335 1782866290.877881 fallocate 0x027da1 thru 0x031c81 (0x9ee1 bytes) 000336 1782866290.880119 punch from 0x02861c to 0x034000 (0xb9e5 bytes) 000337 1782866290.885759 MIRROR_RESYNC 5 mirrors 000338 1782866291.176670 MIRROR_RESYNC 5 mirrors 000339 1782866291.301033 mapwrite 0x02c4ec thru 0x03bc85 (0xf79a bytes) 000340 1782866291.324783 mapwrite 0x02466f thru 0x032adb (0xe46d bytes) 000341 1782866291.334061 mapwrite 0x03d9fb thru 0x03eea8 (0x14ae bytes) 000342 1782866291.344178 mapread 0x00ad69 thru 0x016f5b (0xc1f3 bytes) 000343 1782866291.349668 read 0x0104c7 thru 0x01f197 (0xecd1 bytes) 000344 1782866291.352871 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3eea9): Mirror 2: 0xa8bbab1 (stale) Mirror 4: 0xa8bbab1 (stale) Mirror 6: 0xa8bbab1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3eea9): Mirror 2: 0xa8bbab1 (stale) Mirror 4: 0xa8bbab1 (stale) Mirror 6: 0xa8bbab1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3eea9: skipped 000345 1782866291.435959 trunc from 0x03eea9 to 0x0207a6 (0x1e702 bytes) 000346 1782866291.440680 trunc from 0x0207a7 to 0x010cb5 (0xfaf1 bytes) 000347 1782866291.444333 mapwrite 0x0353b6 thru 0x03ffff (0xac4a bytes) 000348 1782866291.459062 trunc from 0x040000 to 0x02b557 (0x14aa8 bytes) 000349 1782866291.464250 MIRROR_RESYNC 4 mirrors 000350 1782866291.642659 mapread 0x026c69 thru 0x02b557 (0x48ef bytes) 000352 1782866291.647226 read 0x00667b thru 0x012b8b (0xc511 bytes) 000353 1782866291.649637 mapwrite 0x018f1f thru 0x01fe54 (0x6f36 bytes) 000354 1782866291.680716 punch from 0x020028 to 0x02b557 (0xb530 bytes) 000355 1782866291.682678 mapwrite 0x00434e thru 0x009de1 (0x5a94 bytes) 000356 1782866291.690528 mapwrite 0x03a95d thru 0x03de48 (0x34ec bytes) 000357 1782866291.706812 trunc from 0x03de49 to 0x0182fc (0x25b4c bytes) 000358 1782866291.711620 trunc from 0x01083c to 0x0116fc (0xec1 bytes) 000359 1782866291.714252 punch from 0x0141d4 to 0x0182fc (0x4129 bytes) 000360 1782866291.715895 punch from 0x0026e4 to 0x008aff (0x641c bytes) 000361 1782866291.717721 trunc from 0x0182fd to 0x028ccd (0x109d1 bytes) 000362 1782866291.723221 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28cce): Mirror 2: 0xbdd64a7c (stale) Mirror 4: 0xbdd64a7c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28cce): Mirror 2: 0xbdd64a7c (stale) Mirror 4: 0xbdd64a7c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28cce: skipped 000363 1782866291.800435 write 0x027672 thru 0x0282f6 (0xc85 bytes) 000364 1782866291.804132 fallocate 0x03b9de thru 0x03efe4 (0x3607 bytes) 000365 1782866291.806073 trunc from 0x03efe5 to 0x03ce40 (0x21a4 bytes) 000366 1782866291.809959 fallocate 0x007b15 thru 0x0107fd (0x8ce9 bytes) 000367 1782866291.811723 trunc from 0x0255f1 to 0x02be40 (0x6850 bytes) 000369 1782866291.815003 mapread 0x012a3d thru 0x01d53a (0xaafe bytes) 000370 1782866291.822163 mapwrite 0x00c98d thru 0x0129dd (0x6051 bytes) 000371 1782866291.829917 read 0x032119 thru 0x03ca6f (0xa957 bytes) 000372 1782866291.832328 mapwrite 0x02bce3 thru 0x02ee12 (0x3130 bytes) 000373 1782866291.839014 trunc from 0x007d68 to 0x00b37c (0x3615 bytes) 000374 1782866291.840825 trunc from 0x02e64a to 0x033b7d (0x5534 bytes) 000375 1782866291.842568 mapwrite 0x033518 thru 0x0352d5 (0x1dbe bytes) 000376 1782866291.849037 trunc from 0x03ce41 to 0x01b3b8 (0x21a88 bytes) 000377 1782866291.853716 mapread 0x008e4f thru 0x00b385 (0x2537 bytes) 000378 1782866291.856734 mapwrite 0x03258f thru 0x0365c1 (0x4033 bytes) 000379 1782866291.868311 fallocate 0x0242db thru 0x02f3ff (0xb125 bytes) 000381 1782866291.870638 trunc from 0x0080a8 to 0x0154b8 (0xd411 bytes) 000383 1782866291.872860 mapwrite 0x038be0 thru 0x039129 (0x54a bytes) 000384 1782866291.883799 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3912a): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3912a): Mirror 2: 0xccdbb496 (stale) Mirror 4: 0xccdbb496 (stale) CRC-32 checksum value for chunk [0, 0x3912a): Mirror 1: 0x637efcce Mirror 5: 0x637efcce lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3912a: skipped 000385 1782866292.115612 trunc from 0x03513d to 0x039129 (0x3fed bytes) 000386 1782866292.125717 trunc from 0x0010bb to 0x0076cf (0x6615 bytes) 000388 1782866292.128230 write 0x02d6fb thru 0x0333a3 (0x5ca9 bytes) 000389 1782866292.132699 mapwrite 0x03da7b thru 0x03ffff (0x2585 bytes) 000390 1782866292.151142 mapread 0x00b507 thru 0x0121ad (0x6ca7 bytes) 000391 1782866292.154346 fallocate 0x02e64c thru 0x03b5cf (0xcf84 bytes) 000392 1782866292.156626 read 0x03250f thru 0x03ae01 (0x88f3 bytes) 000393 1782866292.158746 write 0x0208fa thru 0x026d4b (0x6452 bytes) 000394 1782866292.161903 fallocate 0x026d2f thru 0x02a253 (0x3525 bytes) 000395 1782866292.163838 write 0x004319 thru 0x0118b9 (0xd5a1 bytes) 000396 1782866292.165145 punch from 0x010c7e to 0x015f6f (0x52f2 bytes) 000397 1782866292.169002 trunc from 0x008561 to 0x00f507 (0x6fa7 bytes) 000398 1782866292.170750 mapwrite 0x00c6fa thru 0x00f129 (0x2a30 bytes) 000399 1782866292.177299 write 0x02778b thru 0x034b67 (0xd3dd bytes) 000400 1782866292.179353 read 0x01e79b thru 0x023608 (0x4e6e bytes) 000401 1782866292.181984 trunc from 0x008047 to 0x00a956 (0x2910 bytes) 000402 1782866292.183626 read 0x03e6da thru 0x03ffff (0x1926 bytes) 000404 1782866292.184439 fallocate 0x00994e thru 0x01647d (0xcb30 bytes) 000405 1782866292.186230 mapread 0x00926b thru 0x00d062 (0x3df8 bytes) 000407 1782866292.188636 write 0x01f2a1 thru 0x02acd8 (0xba38 bytes) 000409 1782866292.189906 fallocate 0x036ec1 thru 0x037b08 (0xc48 bytes) 000411 1782866292.192051 mapwrite 0x00177f thru 0x001aad (0x32f bytes) 000412 1782866292.197088 trunc from 0x0313ee to 0x03c591 (0xb1a4 bytes) 000413 1782866292.198745 fallocate 0x02355a thru 0x028c97 (0x573e bytes) 000414 1782866292.200482 fallocate 0x00cd12 thru 0x01bc50 (0xef3f bytes) 000415 1782866292.202397 punch from 0x005678 to 0x011a5a (0xc3e3 bytes) 000417 1782866292.204583 trunc from 0x040000 to 0x01efa8 (0x21057 bytes) 000418 1782866292.208833 punch from 0x01bb82 to 0x01efa8 (0x3427 bytes) 000419 1782866292.210604 trunc from 0x01efa9 to 0x03c1f3 (0x1d24b bytes) 000420 1782866292.215954 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c1f4): Mirror 4: 0xa3dc7235 (stale) Mirror 5: 0xb44548ad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c1f4): Mirror 4: 0xa3dc7235 (stale) Mirror 5: 0xb44548ad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c1f4: skipped 000421 1782866292.292319 trunc from 0x03c1f4 to 0x01cdf6 (0x1f3fd bytes) 000422 1782866292.297454 trunc from 0x01cdf7 to 0x029aa4 (0xccae bytes) 000423 1782866292.301697 fallocate 0x0070e2 thru 0x00f1a4 (0x80c3 bytes) 000424 1782866292.303939 write 0x02532b thru 0x031f7b (0xcc51 bytes) 000425 1782866292.306787 mapread 0x019010 thru 0x02814f (0xf140 bytes) 000426 1782866292.314038 fallocate 0x001a8b thru 0x00479f (0x2d15 bytes) 000427 1782866292.317446 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x31f7c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31f7c): Mirror 4: 0x17abe2f1 (stale) Mirror 5: 0x17abe2f1 (stale) CRC-32 checksum value for chunk [0, 0x31f7c): Mirror 1: 0x15f643ec Mirror 6: 0x15f643ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31f7c: skipped 000428 1782866292.559980 write 0x0341a2 thru 0x0358e7 (0x1746 bytes) 000429 1782866292.571616 MIRROR_RESYNC 4 mirrors 000430 1782866292.875881 mapwrite 0x017a10 thru 0x0255d4 (0xdbc5 bytes) 000431 1782866292.912416 mapread 0x0078d5 thru 0x00a127 (0x2853 bytes) 000432 1782866292.915519 read 0x02b212 thru 0x03346c (0x825b bytes) 000433 1782866292.917988 punch from 0x005ef1 to 0x00ed23 (0x8e33 bytes) 000434 1782866292.921037 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x358e8): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x358e8): Mirror 4: 0x82242ec6 (stale) Mirror 5: 0x82242ec6 (stale) Mirror 6: 0x82242ec6 (stale) CRC-32 checksum value for chunk [0, 0x358e8): Mirror 1: 0x55c911c3 Mirror 7: 0x55c911c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x358e8: skipped 000435 1782866293.162023 trunc from 0x0358e8 to 0x01a2b0 (0x1b637 bytes) 000436 1782866293.173690 mapread 0x0139a8 thru 0x018836 (0x4e8f bytes) 000437 1782866293.178987 punch from 0x000635 to 0x008902 (0x82ce bytes) 000438 1782866293.180678 mapread 0x0088ec thru 0x00ceec (0x4601 bytes) 000439 1782866293.185223 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x1a2b1): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a2b1): Mirror 4: 0x7c052e70 (stale) Mirror 5: 0x7c052e70 (stale) Mirror 6: 0x7c052e70 (stale) Mirror 7: 0x6a6e25d1 (stale) CRC-32 checksum value for chunk [0, 0x1a2b1): Mirror 1: 0x1f54d431 Mirror 8: 0x1f54d431 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a2b1: skipped 000440 1782866293.403784 trunc from 0x01a2b1 to 0x031c96 (0x179e6 bytes) 000441 1782866293.419746 read 0x00cb70 thru 0x01c5a2 (0xfa33 bytes) 000442 1782866293.422483 fallocate 0x032d2a thru 0x033cdb (0xfb2 bytes) 000443 1782866293.424331 read 0x029fa8 thru 0x02f317 (0x5370 bytes) 000444 1782866293.428268 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33cdc): Mirror 4: 0xc23e1989 (stale) Mirror 5: 0xc23e1989 (stale) Mirror 6: 0xc23e1989 (stale) Mirror 7: 0x4befbe18 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33cdc): Mirror 4: 0xc23e1989 (stale) Mirror 5: 0xc23e1989 (stale) Mirror 6: 0xc23e1989 (stale) Mirror 7: 0x4befbe18 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33cdc: skipped 000445 1782866293.520655 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x33cdc): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33cdc): Mirror 4: 0xc23e1989 (stale) Mirror 5: 0xc23e1989 (stale) Mirror 6: 0xc23e1989 (stale) Mirror 7: 0x4befbe18 (stale) CRC-32 checksum value for chunk [0, 0x33cdc): Mirror 1: 0xccdeb59b Mirror 8: 0xccdeb59b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33cdc: skipped 000446 1782866293.757749 read 0x018899 thru 0x019da9 (0x1511 bytes) 000447 1782866293.761843 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x33cdc): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33cdc): Mirror 4: 0xc23e1989 (stale) Mirror 5: 0xc23e1989 (stale) Mirror 6: 0xc23e1989 (stale) Mirror 7: 0x4befbe18 (stale) CRC-32 checksum value for chunk [0, 0x33cdc): Mirror 1: 0xccdeb59b Mirror 8: 0xccdeb59b Mirror 9: 0xccdeb59b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33cdc: skipped 000448 1782866293.996529 trunc from 0x033cdc to 0x00ee1f (0x24ebc bytes) 000449 1782866294.013793 write 0x029f2d thru 0x032258 (0x832c bytes) 000450 1782866294.016048 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x32259): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32259): Mirror 4: 0x2b1b737d (stale) Mirror 5: 0x2b1b737d (stale) Mirror 6: 0x2b1b737d (stale) Mirror 7: 0xb956e995 (stale) Mirror 8: 0xfa846d97 (stale) Mirror 9: 0xfa846d97 (stale) CRC-32 checksum value for chunk [0, 0x32259): Mirror 1: 0xcd3d3e63 Mirror 10: 0xcd3d3e63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32259: skipped 000451 1782866294.272367 fallocate 0x006c11 thru 0x012651 (0xba41 bytes) 000452 1782866294.283055 write 0x03ea26 thru 0x03ffff (0x15da bytes) 000453 1782866294.285096 trunc from 0x040000 to 0x02ce20 (0x131df bytes) 000454 1782866294.290254 write 0x032821 thru 0x036cb8 (0x4498 bytes) 000455 1782866294.291093 write 0x010b73 thru 0x01fe15 (0xf2a3 bytes) 000458 1782866294.295178 punch from 0x0243f2 to 0x02f105 (0xad14 bytes) 000459 1782866294.296899 write 0x01a848 thru 0x0239fd (0x91b6 bytes) 000460 1782866294.300873 MIRROR_RESYNC 8 mirrors 000461 1782866294.677569 punch from 0x00116e to 0x00de2f (0xccc2 bytes) 000462 1782866294.687169 trunc from 0x036cb9 to 0x03bfbb (0x5303 bytes) 000463 1782866294.692463 mapwrite 0x0001de thru 0x00a965 (0xa788 bytes) 000464 1782866294.705681 write 0x024922 thru 0x02e39a (0x9a79 bytes) 000466 1782866294.709113 mapwrite 0x008b03 thru 0x01172c (0x8c2a bytes) 000467 1782866294.718137 read 0x02d4a4 thru 0x02fd23 (0x2880 bytes) 000468 1782866294.719930 mapread 0x02a03d thru 0x032c32 (0x8bf6 bytes) 000471 1782866294.723534 mapwrite 0x014f6d thru 0x016ac4 (0x1b58 bytes) 000472 1782866294.727732 punch from 0x0013e4 to 0x008caa (0x78c7 bytes) 000474 1782866294.730185 write 0x01b7c0 thru 0x024570 (0x8db1 bytes) 000475 1782866294.732294 trunc from 0x03bfbc to 0x003052 (0x38f69 bytes) 000476 1782866294.737058 fallocate 0x0225e0 thru 0x02d213 (0xac34 bytes) 000478 1782866294.739073 write 0x00ac6c thru 0x019e92 (0xf227 bytes) 000479 1782866294.740235 write 0x021627 thru 0x02be2e (0xa808 bytes) 000480 1782866294.741198 trunc from 0x02be2f to 0x028084 (0x3daa bytes) 000481 1782866294.745405 trunc from 0x00a659 to 0x014e62 (0xa80a bytes) 000482 1782866294.749404 trunc from 0x028085 to 0x00b377 (0x1cd0d bytes) 000483 1782866294.753704 fallocate 0x03603a thru 0x039b55 (0x3b1c bytes) 000484 1782866294.755319 read 0x00282f thru 0x002927 (0xf9 bytes) 000485 1782866294.758952 mapwrite 0x03674a thru 0x03ffff (0x98b6 bytes) 000486 1782866294.771730 trunc from 0x040000 to 0x009eef (0x36110 bytes) 000487 1782866294.776010 mapread 0x001e39 thru 0x009e19 (0x7fe1 bytes) 000488 1782866294.776826 write 0x01fc2d thru 0x0213b3 (0x1787 bytes) 000490 1782866294.778223 mapread 0x00aa13 thru 0x00d148 (0x2736 bytes) 000491 1782866294.784193 trunc from 0x010293 to 0x01d12d (0xce9b bytes) 000492 1782866294.785942 mapwrite 0x00e990 thru 0x01207d (0x36ee bytes) 000493 1782866294.798122 fallocate 0x0118ff thru 0x014992 (0x3094 bytes) 000494 1782866294.800142 read 0x01d449 thru 0x0210e8 (0x3ca0 bytes) 000495 1782866294.804368 MIRROR_RESYNC 8 mirrors 000496 1782866295.164990 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000497 1782866295.267422 trunc from 0x0213b4 to 0x02530a (0x3f57 bytes) 000498 1782866295.279220 punch from 0x01fb97 to 0x02530a (0x5774 bytes) 000499 1782866295.280850 fallocate 0x02ad62 thru 0x03a629 (0xf8c8 bytes) 000500 1782866295.282896 trunc from 0x020bff to 0x02530a (0x470c bytes) 000502 1782866295.285391 trunc from 0x00baeb to 0x010234 (0x474a bytes) 000503 1782866295.287366 read 0x01abc5 thru 0x02530a (0xa746 bytes) 000504 1782866295.290416 mapwrite 0x0199a8 thru 0x027536 (0xdb8f bytes) 000505 1782866295.308407 read 0x009c31 thru 0x00cd2a (0x30fa bytes) 000506 1782866295.311209 read 0x01d6fd thru 0x0255af (0x7eb3 bytes) 000508 1782866295.312080 fallocate 0x00d41e thru 0x010b78 (0x375b bytes) 000509 1782866295.313861 fallocate 0x02a482 thru 0x02e84e (0x43cd bytes) 000510 1782866295.317090 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x27537): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27537): Mirror 5: 0x8c75994a (stale) Mirror 6: 0x8c75994a (stale) Mirror 7: 0x8c75994a (stale) Mirror 8: 0x8c75994a (stale) Mirror 9: 0x8c75994a (stale) Mirror 10: 0x8c75994a (stale) CRC-32 checksum value for chunk [0, 0x27537): Mirror 1: 0x8a5c8d1e Mirror 11: 0x8a5c8d1e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27537: skipped 000511 1782866295.553669 punch from 0x00a478 to 0x0104ab (0x6034 bytes) 000512 1782866295.564384 punch from 0x008324 to 0x009c69 (0x1946 bytes) 000513 1782866295.565937 fallocate 0x01ff0e thru 0x02e69d (0xe790 bytes) 000514 1782866295.569000 MIRROR_RESYNC 8 mirrors 000515 1782866295.975367 mapread 0x01490c thru 0x01ab18 (0x620d bytes) 000516 1782866295.980309 write 0x026a54 thru 0x02b0f3 (0x46a0 bytes) 000518 1782866295.992184 fallocate 0x012650 thru 0x016760 (0x4111 bytes) 000519 1782866295.993826 read 0x0124e6 thru 0x02012e (0xdc49 bytes) 000520 1782866295.996300 trunc from 0x028204 to 0x02ae79 (0x2c76 bytes) 000522 1782866296.000465 punch from 0x01a691 to 0x028515 (0xde85 bytes) 000523 1782866296.002244 read 0x02987d thru 0x02e69d (0x4e21 bytes) 000524 1782866296.005286 write 0x0251c6 thru 0x030709 (0xb544 bytes) 000525 1782866296.006274 mapread 0x0239ee thru 0x025781 (0x1d94 bytes) 000526 1782866296.006971 punch from 0x012e8a to 0x019045 (0x61bc bytes) 000527 1782866296.009872 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3070a): Mirror 5: 0x9fa7baee (stale) Mirror 6: 0x9fa7baee (stale) Mirror 7: 0x9fa7baee (stale) Mirror 8: 0x9fa7baee (stale) Mirror 9: 0x9fa7baee (stale) Mirror 11: 0x9fa7baee (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3070a): Mirror 5: 0x9fa7baee (stale) Mirror 6: 0x9fa7baee (stale) Mirror 7: 0x9fa7baee (stale) Mirror 8: 0x9fa7baee (stale) Mirror 9: 0x9fa7baee (stale) Mirror 11: 0x9fa7baee (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3070a: skipped 000528 1782866296.119055 trunc from 0x03070a to 0x00e856 (0x21eb3 bytes) 000529 1782866296.127804 fallocate 0x007443 thru 0x01688e (0xf44c bytes) 000530 1782866296.129626 mapwrite 0x00bf29 thru 0x019c27 (0xdcff bytes) 000531 1782866296.148307 read 0x003fd0 thru 0x004101 (0x132 bytes) 000532 1782866296.150179 mapread 0x00a4c2 thru 0x017b56 (0xd695 bytes) 000533 1782866296.151002 fallocate 0x01b684 thru 0x025abd (0xa43a bytes) 000534 1782866296.152769 trunc from 0x025abe to 0x01d5db (0x84e2 bytes) 000535 1782866296.157389 write 0x01a0aa thru 0x01cc93 (0x2bea bytes) 000536 1782866296.160732 punch from 0x009c4e to 0x017b2f (0xdee2 bytes) 000537 1782866296.162832 read 0x000f82 thru 0x00dd2f (0xcdae bytes) 000538 1782866296.166221 mapread 0x0093fc thru 0x00b19b (0x1da0 bytes) 000539 1782866296.168900 MIRROR_RESYNC 7 mirrors 000540 1782866296.427440 fallocate 0x003caf thru 0x00a1e6 (0x6538 bytes) 000541 1782866296.437595 trunc from 0x01d5dc to 0x039ec2 (0x1c8e7 bytes) 000543 1782866296.446735 trunc from 0x023ede to 0x024a5e (0xb81 bytes) 000544 1782866296.448398 mapwrite 0x01ad95 thru 0x029ccd (0xef39 bytes) 000545 1782866296.464190 MIRROR_RESYNC 7 mirrors 000546 1782866296.799784 trunc from 0x039a85 to 0x039ec2 (0x43e bytes) 000547 1782866296.810096 punch from 0x030087 to 0x031e84 (0x1dfe bytes) 000548 1782866296.811890 fallocate 0x004ae0 thru 0x0126ad (0xdbce bytes) 000549 1782866296.813922 trunc from 0x039ec3 to 0x03342b (0x6a97 bytes) 000550 1782866296.826425 write 0x0149c2 thru 0x01989a (0x4ed9 bytes) 000551 1782866296.830288 trunc from 0x02ce63 to 0x03342b (0x65c9 bytes) 000552 1782866296.834618 MIRROR_RESYNC 7 mirrors 000553 1782866297.347109 trunc from 0x03342c to 0x0286b4 (0xad77 bytes) 000554 1782866297.364783 trunc from 0x023c2c to 0x0286b4 (0x4a89 bytes) 000555 1782866297.367774 trunc from 0x0286b5 to 0x00acd1 (0x1d9e3 bytes) 000556 1782866297.375428 punch from 0x002f56 to 0x003e6f (0xf1a bytes) 000557 1782866297.377443 trunc from 0x00acd2 to 0x02086f (0x15b9e bytes) 000558 1782866297.383087 read 0x01e639 thru 0x02086f (0x2237 bytes) 000559 1782866297.386421 fallocate 0x014853 thru 0x020bdf (0xc38d bytes) 000560 1782866297.388982 read 0x011b9e thru 0x0161c1 (0x4624 bytes) 000561 1782866297.393697 read 0x016ca4 thru 0x01efa4 (0x8301 bytes) 000562 1782866297.397550 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x20870): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20870): Mirror 5: 0x55d022ee (stale) Mirror 6: 0x55d022ee (stale) Mirror 7: 0x55d022ee (stale) Mirror 8: 0x55d022ee (stale) Mirror 9: 0x55d022ee (stale) Mirror 11: 0x55d022ee (stale) CRC-32 checksum value for chunk [0, 0x20870): Mirror 1: 0x848967c2 Mirror 12: 0x848967c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20870: skipped 000563 1782866297.776038 read 0x01223b thru 0x018bf6 (0x69bc bytes) 000564 1782866297.785178 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x20870): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20870): Mirror 5: 0x55d022ee (stale) Mirror 6: 0x55d022ee (stale) Mirror 7: 0x55d022ee (stale) Mirror 8: 0x55d022ee (stale) Mirror 9: 0x55d022ee (stale) Mirror 11: 0x55d022ee (stale) CRC-32 checksum value for chunk [0, 0x20870): Mirror 1: 0x848967c2 Mirror 12: 0x848967c2 Mirror 13: 0x848967c2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20870: skipped 000565 1782866298.104745 punch from 0x01ab9e to 0x01b5bc (0xa1f bytes) 000567 1782866298.126174 mapread 0x004438 thru 0x00abc6 (0x678f bytes) 000568 1782866298.136634 trunc from 0x020870 to 0x01791c (0x8f53 bytes) 000569 1782866298.155538 write 0x02f23d thru 0x03a103 (0xaec7 bytes) 000570 1782866298.157300 punch from 0x031327 to 0x039ff2 (0x8ccc bytes) 000571 1782866298.163236 trunc from 0x00ea01 to 0x01575b (0x6d5b bytes) 000574 1782866298.166450 trunc from 0x03a104 to 0x026bbc (0x13547 bytes) 000575 1782866298.173175 fallocate 0x00aa92 thru 0x0189dd (0xdf4c bytes) 000576 1782866298.176108 mapwrite 0x023e93 thru 0x02dbc8 (0x9d36 bytes) 000577 1782866298.198150 trunc from 0x0172f6 to 0x01dffd (0x6d08 bytes) 000578 1782866298.200400 mapwrite 0x02b032 thru 0x02e9d6 (0x39a5 bytes) 000579 1782866298.217339 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2e9d7): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e9d7): Mirror 5: 0xc34eefad (stale) Mirror 6: 0xc34eefad (stale) Mirror 7: 0xc34eefad (stale) Mirror 8: 0xc34eefad (stale) Mirror 9: 0xc34eefad (stale) Mirror 11: 0xc34eefad (stale) Mirror 12: 0xdfc1079b (stale) Mirror 13: 0xdfc1079b (stale) CRC-32 checksum value for chunk [0, 0x2e9d7): Mirror 1: 0x77f73478 Mirror 14: 0x77f73478 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e9d7: skipped 000580 1782866298.540907 trunc from 0x02e9d7 to 0x014c33 (0x19da3 bytes) 000581 1782866298.569058 mapwrite 0x026beb thru 0x02ad9f (0x41b5 bytes) 000582 1782866298.589809 write 0x03474c thru 0x03cc4e (0x8503 bytes) 000583 1782866298.590779 punch from 0x0187f3 to 0x01d454 (0x4c62 bytes) 000584 1782866298.592920 write 0x024636 thru 0x02dd81 (0x974c bytes) 000586 1782866298.596095 mapwrite 0x0298a6 thru 0x0305b8 (0x6d13 bytes) 000588 1782866298.610759 write 0x01d2c0 thru 0x02ac34 (0xd975 bytes) 000589 1782866298.614882 mapread 0x003401 thru 0x007414 (0x4014 bytes) 000590 1782866298.619874 fallocate 0x02dc7a thru 0x0329b7 (0x4d3e bytes) 000591 1782866298.622574 fallocate 0x00df6d thru 0x016b84 (0x8c18 bytes) 000592 1782866298.625041 mapwrite 0x03f2e0 thru 0x03ffff (0xd20 bytes) 000593 1782866298.640552 mapread 0x019e37 thru 0x025a3b (0xbc05 bytes) 000594 1782866298.644592 fallocate 0x00e8f1 thru 0x01c880 (0xdf90 bytes) 000595 1782866298.647004 fallocate 0x022c78 thru 0x023877 (0xc00 bytes) 000596 1782866298.649280 trunc from 0x008603 to 0x00b493 (0x2e91 bytes) 000597 1782866298.651702 mapwrite 0x00d823 thru 0x019495 (0xbc73 bytes) 000598 1782866298.664381 trunc from 0x022c32 to 0x0271ac (0x457b bytes) 000599 1782866298.666755 mapread 0x01139e thru 0x015c4d (0x48b0 bytes) 000600 1782866298.667426 fallocate 0x0260ba thru 0x032c72 (0xcbb9 bytes) 000601 1782866298.669882 mapwrite 0x009eb7 thru 0x00a195 (0x2df bytes) 000602 1782866298.677265 trunc from 0x0238f2 to 0x03177b (0xde8a bytes) 000603 1782866298.679349 read 0x01292d thru 0x018a2f (0x6103 bytes) 000604 1782866298.679831 mapread 0x03182b thru 0x038ade (0x72b4 bytes) 000605 1782866298.683031 write 0x02d42b thru 0x02f00b (0x1be1 bytes) 000606 1782866298.683779 mapread 0x006ccd thru 0x014f30 (0xe264 bytes) 000607 1782866298.684615 mapread 0x02ef24 thru 0x038a59 (0x9b36 bytes) 000608 1782866298.685346 read 0x030d9b thru 0x03b2ea (0xa550 bytes) 000610 1782866298.686164 punch from 0x00cbba to 0x00ed73 (0x21ba bytes) 000611 1782866298.689689 MIRROR_RESYNC 10 mirrors 000612 1782866299.362061 fallocate 0x004d2f thru 0x00d83c (0x8b0e bytes) 000613 1782866299.371871 fallocate 0x032970 thru 0x03ca5a (0xa0eb bytes) 000614 1782866299.374140 trunc from 0x040000 to 0x00ab90 (0x3546f bytes) 000615 1782866299.381931 write 0x02e086 thru 0x03c607 (0xe582 bytes) 000616 1782866299.384076 mapwrite 0x00eaff thru 0x014bfb (0x60fd bytes) 000618 1782866299.397263 fallocate 0x033ad5 thru 0x0385eb (0x4b17 bytes) 000619 1782866299.399362 trunc from 0x03c608 to 0x02d0c4 (0xf543 bytes) 000620 1782866299.404227 trunc from 0x00c971 to 0x00ca13 (0xa3 bytes) 000621 1782866299.406350 mapread 0x0055bf thru 0x012350 (0xcd92 bytes) 000622 1782866299.409869 trunc from 0x02d0c5 to 0x022cef (0xa3d5 bytes) 000623 1782866299.414935 fallocate 0x0109fc thru 0x01b990 (0xaf95 bytes) 000624 1782866299.417165 punch from 0x00756b to 0x00f90b (0x83a1 bytes) 000625 1782866299.419185 punch from 0x015e5f to 0x01a0f9 (0x429b bytes) 000626 1782866299.421320 trunc from 0x0142f2 to 0x014b90 (0x89f bytes) 000627 1782866299.423327 trunc from 0x004fa8 to 0x0094db (0x4534 bytes) 000628 1782866299.425411 punch from 0x0016f4 to 0x006c3e (0x554b bytes) 000629 1782866299.429336 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x22cf0): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22cf0): Mirror 5: 0xcc3d774a (stale) Mirror 6: 0xcc3d774a (stale) Mirror 7: 0xcc3d774a (stale) Mirror 8: 0xcc3d774a (stale) Mirror 9: 0xcc3d774a (stale) Mirror 11: 0xcc3d774a (stale) Mirror 12: 0xcc3d774a (stale) Mirror 13: 0xcc3d774a (stale) Mirror 14: 0xcc3d774a (stale) CRC-32 checksum value for chunk [0, 0x22cf0): Mirror 1: 0xd26bcae2 Mirror 15: 0xd26bcae2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22cf0: skipped 000630 1782866299.813151 fallocate 0x02a793 thru 0x039e0e (0xf67c bytes) 000631 1782866299.829244 read 0x0182c4 thru 0x022cef (0xaa2c bytes) 000632 1782866299.833459 fallocate 0x02b018 thru 0x02fdd9 (0x4dc2 bytes) 000633 1782866299.836543 write 0x02ad6f thru 0x031abf (0x6d51 bytes) 000634 1782866299.839883 mapwrite 0x022d5a thru 0x02e21a (0xb4c1 bytes) 000636 1782866299.859059 mapwrite 0x01aa9f thru 0x027212 (0xc774 bytes) 000638 1782866299.871729 trunc from 0x031ac0 to 0x014345 (0x1d77a bytes) 000639 1782866299.881965 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x14346): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14346): Mirror 5: 0x18c97378 (stale) Mirror 6: 0x18c97378 (stale) Mirror 7: 0x18c97378 (stale) Mirror 8: 0x18c97378 (stale) Mirror 9: 0x18c97378 (stale) Mirror 11: 0x18c97378 (stale) Mirror 12: 0x18c97378 (stale) Mirror 13: 0x18c97378 (stale) Mirror 14: 0x18c97378 (stale) Mirror 15: 0xfd417a53 (stale) CRC-32 checksum value for chunk [0, 0x14346): Mirror 1: 0xfd417a53 Mirror 16: 0xfd417a53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14346: skipped 000640 1782866300.144876 MIRROR_RESYNC 12 mirrors 000641 1782866300.530060 mapread 0x003186 thru 0x011e2a (0xeca5 bytes) 000642 1782866300.539147 trunc from 0x014346 to 0x034e8a (0x20b45 bytes) 000643 1782866300.559537 read 0x008426 thru 0x01741e (0xeff9 bytes) 000644 1782866300.564331 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34e8b): Mirror 5: 0x18d211a7 (stale) Mirror 6: 0x18d211a7 (stale) Mirror 8: 0x18d211a7 (stale) Mirror 9: 0x18d211a7 (stale) Mirror 11: 0x18d211a7 (stale) Mirror 12: 0x18d211a7 (stale) Mirror 13: 0x18d211a7 (stale) Mirror 14: 0x18d211a7 (stale) Mirror 15: 0x18d211a7 (stale) Mirror 16: 0x18d211a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34e8b): Mirror 5: 0x18d211a7 (stale) Mirror 6: 0x18d211a7 (stale) Mirror 8: 0x18d211a7 (stale) Mirror 9: 0x18d211a7 (stale) Mirror 11: 0x18d211a7 (stale) Mirror 12: 0x18d211a7 (stale) Mirror 13: 0x18d211a7 (stale) Mirror 14: 0x18d211a7 (stale) Mirror 15: 0x18d211a7 (stale) Mirror 16: 0x18d211a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34e8b: skipped 000645 1782866300.761597 mapwrite 0x03722b thru 0x03ffff (0x8dd5 bytes) 000646 1782866300.786249 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x31ffd4db (stale) Mirror 6: 0x31ffd4db (stale) Mirror 8: 0x31ffd4db (stale) Mirror 9: 0x31ffd4db (stale) Mirror 11: 0x31ffd4db (stale) Mirror 12: 0x31ffd4db (stale) Mirror 13: 0x31ffd4db (stale) Mirror 14: 0x31ffd4db (stale) Mirror 15: 0x31ffd4db (stale) Mirror 16: 0x31ffd4db (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa75a2fab Mirror 17: 0xa75a2fab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000647 1782866301.146777 mapwrite 0x03c31d thru 0x03e852 (0x2536 bytes) 000648 1782866301.180382 fallocate 0x004549 thru 0x006cd8 (0x2790 bytes) 000649 1782866301.182164 mapwrite 0x019916 thru 0x025331 (0xba1c bytes) 000650 1782866301.192737 read 0x03f89c thru 0x03ffff (0x764 bytes) 000651 1782866301.193152 trunc from 0x040000 to 0x03d060 (0x2f9f bytes) 000652 1782866301.203737 read 0x022178 thru 0x027e4c (0x5cd5 bytes) 000653 1782866301.204211 write 0x0385bb thru 0x03ffff (0x7a45 bytes) 000654 1782866301.205138 punch from 0x021573 to 0x02ff4b (0xe9d9 bytes) 000655 1782866301.206883 read 0x03b1bd thru 0x03ffff (0x4e43 bytes) 000656 1782866301.207358 trunc from 0x040000 to 0x011b4a (0x2e4b5 bytes) 000658 1782866301.211942 read 0x009d11 thru 0x00a1c6 (0x4b6 bytes) 000659 1782866301.214678 trunc from 0x011b4b to 0x00ade6 (0x6d64 bytes) 000660 1782866301.218741 trunc from 0x008238 to 0x00ade6 (0x2baf bytes) 000661 1782866301.220412 fallocate 0x002555 thru 0x008937 (0x63e3 bytes) 000662 1782866301.222215 mapwrite 0x0378fc thru 0x03ffff (0x8704 bytes) 000663 1782866301.234239 trunc from 0x002a08 to 0x002f3b (0x534 bytes) 000664 1782866301.235811 trunc from 0x03509a to 0x03bb41 (0x6aa8 bytes) 000665 1782866301.238903 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x31ffd4db (stale) Mirror 6: 0x31ffd4db (stale) Mirror 8: 0x31ffd4db (stale) Mirror 9: 0x31ffd4db (stale) Mirror 11: 0x31ffd4db (stale) Mirror 12: 0x31ffd4db (stale) Mirror 13: 0x31ffd4db (stale) Mirror 14: 0x31ffd4db (stale) Mirror 15: 0x31ffd4db (stale) Mirror 16: 0x31ffd4db (stale) Mirror 17: 0xa75a2fab (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4d06e2d3 Mirror 18: 0x4d06e2d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000666 1782866301.511955 mapread 0x0115fb thru 0x0188e0 (0x72e6 bytes) 000667 1782866301.525611 read 0x031102 thru 0x035c11 (0x4b10 bytes) 000668 1782866301.530900 punch from 0x00b096 to 0x00ef43 (0x3eae bytes) 000670 1782866301.549449 fallocate 0x00437c thru 0x00bf54 (0x7bd9 bytes) 000671 1782866301.551905 read 0x00b1a9 thru 0x01842e (0xd286 bytes) 000673 1782866301.556369 mapread 0x00dbe4 thru 0x014e18 (0x7235 bytes) 000674 1782866301.557115 trunc from 0x03cc4c to 0x03ffff (0x33b4 bytes) 000675 1782866301.560995 MIRROR_RESYNC 13 mirrors 000676 1782866302.099944 MIRROR_SPLIT mirror 9 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000677 1782866302.221605 fallocate 0x021662 thru 0x02f189 (0xdb28 bytes) 000678 1782866302.232169 mapread 0x037a5c thru 0x039cd6 (0x227b bytes) 000679 1782866302.237475 write 0x01925c thru 0x01a318 (0x10bd bytes) 000680 1782866302.241215 trunc from 0x040000 to 0x01d8d1 (0x2272e bytes) 000681 1782866302.250854 read 0x0120d3 thru 0x01c942 (0xa870 bytes) 000683 1782866302.253612 read 0x0149cd thru 0x01c114 (0x7748 bytes) 000684 1782866302.254020 read 0x014fa1 thru 0x01a57f (0x55df bytes) 000685 1782866302.254434 mapwrite 0x032082 thru 0x034c3e (0x2bbd bytes) 000686 1782866302.266337 write 0x01f15d thru 0x0247c6 (0x566a bytes) 000687 1782866302.267070 mapread 0x009e52 thru 0x00fe87 (0x6036 bytes) 000688 1782866302.269028 MIRROR_RESYNC 12 mirrors 000689 1782866302.948977 punch from 0x001261 to 0x003a1a (0x27ba bytes) 000690 1782866302.968034 mapread 0x028c5d thru 0x031d06 (0x90aa bytes) 000691 1782866302.978070 mapwrite 0x001de8 thru 0x004e99 (0x30b2 bytes) 000692 1782866302.989152 mapwrite 0x01e980 thru 0x01feaa (0x152b bytes) 000693 1782866303.000096 trunc from 0x034c3f to 0x02906b (0xbbd3 bytes) 000694 1782866303.017065 trunc from 0x015cbe to 0x022071 (0xc3b4 bytes) 000695 1782866303.019854 trunc from 0x02906c to 0x0247f1 (0x487a bytes) 000696 1782866303.026043 punch from 0x0073b2 to 0x00facb (0x871a bytes) 000697 1782866303.028330 trunc from 0x0247f2 to 0x023198 (0x1659 bytes) 000698 1782866303.034944 fallocate 0x018197 thru 0x022c49 (0xaab3 bytes) 000699 1782866303.037249 mapread 0x017ef0 thru 0x023198 (0xb2a9 bytes) 000700 1782866303.044595 punch from 0x01fb38 to 0x023198 (0x3661 bytes) 000701 1782866303.047337 read 0x021345 thru 0x0230fb (0x1db7 bytes) 000702 1782866303.050358 trunc from 0x023199 to 0x03bd59 (0x18bc1 bytes) 000703 1782866303.059594 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3bd5a): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd5a): Mirror 5: 0x6d09e626 (stale) Mirror 6: 0x6d09e626 (stale) Mirror 8: 0x6d09e626 (stale) Mirror 9: 0x6d09e626 (stale) Mirror 11: 0x6d09e626 (stale) Mirror 12: 0x6d09e626 (stale) Mirror 13: 0x6d09e626 (stale) Mirror 14: 0x6d09e626 (stale) Mirror 16: 0x6d09e626 (stale) Mirror 17: 0x6d09e626 (stale) Mirror 18: 0x6d09e626 (stale) CRC-32 checksum value for chunk [0, 0x3bd5a): Mirror 1: 0x8f5e1ca0 Mirror 19: 0x8f5e1ca0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd5a: skipped 000704 1782866303.453651 write 0x0262b2 thru 0x028790 (0x24df bytes) 000705 1782866303.478302 mapread 0x01a936 thru 0x020cec (0x63b7 bytes) 000706 1782866303.487049 trunc from 0x01997a to 0x022273 (0x88fa bytes) 000707 1782866303.489667 punch from 0x03b10d to 0x03bd59 (0xc4d bytes) 000708 1782866303.491939 mapwrite 0x01bec7 thru 0x01e5fb (0x2735 bytes) 000709 1782866303.503298 fallocate 0x025476 thru 0x02c171 (0x6cfc bytes) 000710 1782866303.505908 read 0x005c08 thru 0x014e59 (0xf252 bytes) 000711 1782866303.509864 punch from 0x016c68 to 0x021a42 (0xaddb bytes) 000712 1782866303.514521 MIRROR_RESYNC 13 mirrors 000713 1782866304.415071 mapread 0x010e50 thru 0x017eae (0x705f bytes) 000714 1782866304.423071 trunc from 0x00f963 to 0x016188 (0x6826 bytes) 000715 1782866304.438582 read 0x0088e1 thru 0x017b8b (0xf2ab bytes) 000716 1782866304.443678 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3bd5a): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bd5a): Mirror 5: 0xbe6cf21a (stale) Mirror 6: 0xbe6cf21a (stale) Mirror 8: 0xbe6cf21a (stale) Mirror 9: 0xbe6cf21a (stale) Mirror 11: 0xbe6cf21a (stale) Mirror 12: 0xbe6cf21a (stale) Mirror 13: 0xbe6cf21a (stale) Mirror 14: 0xbe6cf21a (stale) Mirror 16: 0xbe6cf21a (stale) Mirror 17: 0xbe6cf21a (stale) Mirror 18: 0xbe6cf21a (stale) Mirror 19: 0xbe6cf21a (stale) CRC-32 checksum value for chunk [0, 0x3bd5a): Mirror 1: 0xbe6cf21a Mirror 20: 0xbe6cf21a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bd5a: skipped 000718 1782866304.902098 mapwrite 0x00a975 thru 0x010a24 (0x60b0 bytes) 000719 1782866304.957192 read 0x024e0b thru 0x02c781 (0x7977 bytes) 000720 1782866304.961674 trunc from 0x03bd5a to 0x030046 (0xbd13 bytes) 000721 1782866304.974174 mapwrite 0x03e8a9 thru 0x03ffff (0x1757 bytes) 000722 1782866304.994908 MIRROR_RESYNC 14 mirrors 000723 1782866306.063929 fallocate 0x03fffc thru 0x03ffff (0x4 bytes) 000726 1782866306.074094 fallocate 0x025122 thru 0x02de98 (0x8d77 bytes) 000727 1782866306.076047 trunc from 0x010df5 to 0x0141a4 (0x33b0 bytes) 000728 1782866306.077885 trunc from 0x01155b to 0x01f52b (0xdfd1 bytes) 000729 1782866306.079443 punch from 0x01fcad to 0x02d4b2 (0xd806 bytes) 000730 1782866306.081203 mapwrite 0x00e211 thru 0x015093 (0x6e83 bytes) 000731 1782866306.093040 punch from 0x0066e2 to 0x014765 (0xe084 bytes) 000733 1782866306.095461 punch from 0x00a08e to 0x00ab55 (0xac8 bytes) 000734 1782866306.097129 punch from 0x021092 to 0x026544 (0x54b3 bytes) 000735 1782866306.098788 trunc from 0x040000 to 0x01c20f (0x23df0 bytes) 000736 1782866306.104654 punch from 0x01333c to 0x01c20f (0x8ed4 bytes) 000737 1782866306.106558 write 0x011ac7 thru 0x01cf4f (0xb489 bytes) 000738 1782866306.109933 trunc from 0x01cf50 to 0x034976 (0x17a27 bytes) 000739 1782866306.114241 write 0x03e8f5 thru 0x03ffff (0x170b bytes) 000740 1782866306.115020 trunc from 0x040000 to 0x0034e1 (0x3cb1e bytes) 000741 1782866306.119546 punch from 0x002696 to 0x0034e1 (0xe4c bytes) 000742 1782866306.121208 trunc from 0x0034e2 to 0x003a0c (0x52b bytes) 000743 1782866306.126605 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3a0d): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a0d): Mirror 5: 0x29029010 (stale) Mirror 6: 0x29029010 (stale) Mirror 8: 0x29029010 (stale) Mirror 9: 0x29029010 (stale) Mirror 11: 0x29029010 (stale) Mirror 12: 0x29029010 (stale) Mirror 13: 0x29029010 (stale) Mirror 14: 0x29029010 (stale) Mirror 16: 0x29029010 (stale) Mirror 17: 0x29029010 (stale) Mirror 18: 0x29029010 (stale) Mirror 19: 0x29029010 (stale) Mirror 20: 0x29029010 (stale) CRC-32 checksum value for chunk [0, 0x3a0d): Mirror 1: 0x7dffd2d1 Mirror 21: 0x7dffd2d1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a0d: skipped 000744 1782866306.375877 write 0x003762 thru 0x007f3f (0x47de bytes) 000745 1782866306.389907 mapread 0x00081b thru 0x005135 (0x491b bytes) 000746 1782866306.394483 trunc from 0x003519 to 0x007f3f (0x4a27 bytes) 000747 1782866306.398474 trunc from 0x0063b4 to 0x007f3f (0x1b8c bytes) 000749 1782866306.400617 read 0x002293 thru 0x002588 (0x2f6 bytes) 000750 1782866306.401077 read 0x0053c7 thru 0x007f3f (0x2b79 bytes) 000751 1782866306.404058 trunc from 0x003d02 to 0x007f3f (0x423e bytes) 000753 1782866306.406854 write 0x02161c thru 0x026c5f (0x5644 bytes) 000754 1782866306.408125 trunc from 0x026c60 to 0x00646b (0x207f4 bytes) 000755 1782866306.415133 MIRROR_SPLIT mirror 4 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x646c): Mirror 5: 0xbba7397d (stale) Mirror 6: 0xbba7397d (stale) Mirror 8: 0xbba7397d (stale) Mirror 11: 0xbba7397d (stale) Mirror 12: 0xbba7397d (stale) Mirror 13: 0xbba7397d (stale) Mirror 14: 0xbba7397d (stale) Mirror 16: 0xbba7397d (stale) Mirror 17: 0xbba7397d (stale) Mirror 18: 0xbba7397d (stale) Mirror 19: 0xbba7397d (stale) Mirror 20: 0xbba7397d (stale) Mirror 21: 0x8e178e9a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x646c): Mirror 5: 0xbba7397d (stale) Mirror 6: 0xbba7397d (stale) Mirror 8: 0xbba7397d (stale) Mirror 11: 0xbba7397d (stale) Mirror 12: 0xbba7397d (stale) Mirror 13: 0xbba7397d (stale) Mirror 14: 0xbba7397d (stale) Mirror 16: 0xbba7397d (stale) Mirror 17: 0xbba7397d (stale) Mirror 18: 0xbba7397d (stale) Mirror 19: 0xbba7397d (stale) Mirror 20: 0xbba7397d (stale) Mirror 21: 0x8e178e9a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x646c: skipped 000756 1782866306.557637 write 0x016db5 thru 0x022010 (0xb25c bytes) 000757 1782866306.559807 punch from 0x010e7f to 0x0194d4 (0x8656 bytes) 000758 1782866306.564136 write 0x030242 thru 0x038d8b (0x8b4a bytes) 000759 1782866306.565079 fallocate 0x028612 thru 0x036de8 (0xe7d7 bytes) 000760 1782866306.567012 mapwrite 0x01e6fe thru 0x02187a (0x317d bytes) 000761 1782866306.573357 mapread 0x029023 thru 0x0292e0 (0x2be bytes) 000762 1782866306.575453 trunc from 0x038d8c to 0x003091 (0x35cfa bytes) 000763 1782866306.580221 punch from 0x002dd2 to 0x003091 (0x2c0 bytes) 000764 1782866306.581945 read 0x002aa5 thru 0x003091 (0x5ed bytes) 000765 1782866306.583742 mapwrite 0x02e825 thru 0x0334de (0x4cba bytes) 000766 1782866306.598878 trunc from 0x00c905 to 0x00e48c (0x1b88 bytes) 000767 1782866306.600684 trunc from 0x0334df to 0x00d5cc (0x25f12 bytes) 000768 1782866306.605192 punch from 0x0078b0 to 0x00c6a5 (0x4df6 bytes) 000769 1782866306.606809 write 0x027b1a thru 0x02e2fe (0x67e5 bytes) 000770 1782866306.607733 read 0x00be7d thru 0x014cdc (0x8e60 bytes) 000771 1782866306.610844 fallocate 0x038c47 thru 0x03ffff (0x73b9 bytes) 000772 1782866306.614454 MIRROR_RESYNC 14 mirrors 000773 1782866307.068115 MIRROR_EXTEND to 15 mirrors 000774 1782866307.283595 MIRROR_SPLIT mirror 8 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) 000775 1782866307.423711 MIRROR_RESYNC 14 mirrors 000776 1782866307.613729 trunc from 0x0210c3 to 0x0264fe (0x543c bytes) 000778 1782866307.626426 trunc from 0x02e2ff to 0x02162d (0xccd1 bytes) 000781 1782866307.640100 fallocate 0x03311f thru 0x03e2d3 (0xb1b5 bytes) 000782 1782866307.642176 fallocate 0x018714 thru 0x01f0fd (0x69ea bytes) 000783 1782866307.644538 mapwrite 0x005a4f thru 0x00a305 (0x48b7 bytes) 000784 1782866307.655768 punch from 0x021fe4 to 0x027ca6 (0x5cc3 bytes) 000786 1782866307.658724 read 0x0136fb thru 0x013860 (0x166 bytes) 000787 1782866307.660688 trunc from 0x03e2d4 to 0x003dc7 (0x3a50c bytes) 000788 1782866307.665951 punch from 0x0036d1 to 0x003dc7 (0x6f7 bytes) 000790 1782866307.668766 read 0x001f69 thru 0x003dc7 (0x1e5f bytes) 000791 1782866307.671708 mapread 0x0026fd thru 0x003dc7 (0x16cb bytes) 000792 1782866307.672558 mapread 0x001f4b thru 0x003dc7 (0x1e7d bytes) 000793 1782866307.673704 mapwrite 0x01c260 thru 0x027123 (0xaec4 bytes) 000794 1782866307.692324 MIRROR_RESYNC 14 mirrors 000795 1782866308.168737 write 0x03ac3e thru 0x03ffff (0x53c2 bytes) 000796 1782866308.177738 trunc from 0x026575 to 0x032da7 (0xc833 bytes) 000797 1782866308.179899 trunc from 0x002ada to 0x0084b0 (0x59d7 bytes) 000798 1782866308.181860 mapwrite 0x036230 thru 0x03ffff (0x9dd0 bytes) 000799 1782866308.194077 read 0x028916 thru 0x03118f (0x887a bytes) 000800 1782866308.196818 write 0x023a16 thru 0x0323fa (0xe9e5 bytes) 000801 1782866308.199349 mapwrite 0x034358 thru 0x03bc04 (0x78ad bytes) 000802 1782866308.206328 mapwrite 0x025567 thru 0x0275a4 (0x203e bytes) 000803 1782866308.211191 punch from 0x0234c5 to 0x02ceb0 (0x99ec bytes) 000804 1782866308.213418 read 0x004e92 thru 0x00e9bc (0x9b2b bytes) 000805 1782866308.217339 write 0x015906 thru 0x0207f0 (0xaeeb bytes) 000807 1782866308.219408 fallocate 0x03aa38 thru 0x03ffff (0x55c8 bytes) 000808 1782866308.221655 mapread 0x016ffd thru 0x02277f (0xb783 bytes) 000809 1782866308.222939 punch from 0x00faa5 to 0x01b2b5 (0xb811 bytes) 000810 1782866308.227076 punch from 0x02d5c9 to 0x033aba (0x64f2 bytes) 000811 1782866308.228821 trunc from 0x01805e to 0x0212e5 (0x9288 bytes) 000812 1782866308.230843 mapread 0x01bfaf thru 0x01c9b7 (0xa09 bytes) 000814 1782866308.235422 trunc from 0x01cfc9 to 0x02ac72 (0xdcaa bytes) 000815 1782866308.237587 mapread 0x0226ed thru 0x0244bb (0x1dcf bytes) 000816 1782866308.242931 trunc from 0x040000 to 0x02f53b (0x10ac4 bytes) 000817 1782866308.249601 mapwrite 0x022c54 thru 0x028456 (0x5803 bytes) 000818 1782866308.263089 read 0x00f174 thru 0x0159b7 (0x6844 bytes) 000820 1782866308.267525 write 0x010eba thru 0x02005e (0xf1a5 bytes) 000822 1782866308.269977 mapwrite 0x01d014 thru 0x01db82 (0xb6f bytes) 000823 1782866308.275988 punch from 0x0177db to 0x0183ab (0xbd1 bytes) 000824 1782866308.277990 mapwrite 0x029137 thru 0x035952 (0xc81c bytes) 000825 1782866308.297829 trunc from 0x010363 to 0x01a634 (0xa2d2 bytes) 000826 1782866308.300141 mapwrite 0x02c8de thru 0x02e276 (0x1999 bytes) 000827 1782866308.305614 punch from 0x020f6e to 0x023f1a (0x2fad bytes) 000828 1782866308.307708 fallocate 0x00d98c thru 0x015c77 (0x82ec bytes) 000829 1782866308.310086 mapwrite 0x0064e7 thru 0x0121b1 (0xbccb bytes) 000830 1782866308.322988 trunc from 0x02f275 to 0x035952 (0x66de bytes) 000831 1782866308.326219 MIRROR_RESYNC 14 mirrors 000832 1782866309.159813 trunc from 0x01cce0 to 0x028bde (0xbeff bytes) 000833 1782866309.170273 trunc from 0x035953 to 0x017ef0 (0x1da62 bytes) 000834 1782866309.175939 trunc from 0x017ef1 to 0x025e24 (0xdf34 bytes) 000835 1782866309.180478 mapread 0x000af6 thru 0x00a752 (0x9c5d bytes) 000836 1782866309.187395 punch from 0x014428 to 0x0224c5 (0xe09e bytes) 000837 1782866309.190387 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x25e25): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25e25): Mirror 5: 0x8a621b0 (stale) Mirror 6: 0x8a621b0 (stale) Mirror 8: 0x8a621b0 (stale) Mirror 11: 0x8a621b0 (stale) Mirror 12: 0x8a621b0 (stale) Mirror 13: 0x8a621b0 (stale) Mirror 14: 0x8a621b0 (stale) Mirror 17: 0x8a621b0 (stale) Mirror 18: 0x8a621b0 (stale) Mirror 19: 0x8a621b0 (stale) Mirror 20: 0x8a621b0 (stale) Mirror 21: 0x8a621b0 (stale) Mirror 22: 0x8a621b0 (stale) CRC-32 checksum value for chunk [0, 0x25e25): Mirror 1: 0x6b4fc1f4 Mirror 23: 0x6b4fc1f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25e25: skipped 000838 1782866309.496915 punch from 0x01934d to 0x01f9ae (0x6662 bytes) 000839 1782866309.509277 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x25e25): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25e25): Mirror 5: 0x8a621b0 (stale) Mirror 6: 0x8a621b0 (stale) Mirror 8: 0x8a621b0 (stale) Mirror 11: 0x8a621b0 (stale) Mirror 12: 0x8a621b0 (stale) Mirror 13: 0x8a621b0 (stale) Mirror 14: 0x8a621b0 (stale) Mirror 17: 0x8a621b0 (stale) Mirror 18: 0x8a621b0 (stale) Mirror 19: 0x8a621b0 (stale) Mirror 20: 0x8a621b0 (stale) Mirror 21: 0x8a621b0 (stale) Mirror 22: 0x8a621b0 (stale) Mirror 23: 0x6b4fc1f4 (stale) CRC-32 checksum value for chunk [0, 0x25e25): Mirror 1: 0x6b4fc1f4 Mirror 24: 0x6b4fc1f4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25e25: skipped 000840 1782866309.816779 write 0x02fd70 thru 0x030c77 (0xf08 bytes) 000841 1782866309.841557 trunc from 0x030c78 to 0x021d92 (0xeee5 bytes) 000842 1782866309.858152 trunc from 0x021d93 to 0x02d322 (0xb590 bytes) 000843 1782866309.865589 write 0x00cb3d thru 0x0167fd (0x9cc1 bytes) 000844 1782866309.871126 read 0x00db79 thru 0x010412 (0x289a bytes) 000845 1782866309.871961 read 0x003156 thru 0x012889 (0xf734 bytes) 000846 1782866309.880574 MIRROR_SPLIT mirror 3 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d323): Mirror 5: 0xac781410 (stale) Mirror 6: 0xac781410 (stale) Mirror 11: 0xac781410 (stale) Mirror 12: 0xac781410 (stale) Mirror 13: 0xac781410 (stale) Mirror 14: 0xac781410 (stale) Mirror 17: 0xac781410 (stale) Mirror 18: 0xac781410 (stale) Mirror 19: 0xac781410 (stale) Mirror 20: 0xac781410 (stale) Mirror 21: 0xac781410 (stale) Mirror 22: 0xac781410 (stale) Mirror 23: 0xb438bf06 (stale) Mirror 24: 0xb438bf06 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d323): Mirror 5: 0xac781410 (stale) Mirror 6: 0xac781410 (stale) Mirror 11: 0xac781410 (stale) Mirror 12: 0xac781410 (stale) Mirror 13: 0xac781410 (stale) Mirror 14: 0xac781410 (stale) Mirror 17: 0xac781410 (stale) Mirror 18: 0xac781410 (stale) Mirror 19: 0xac781410 (stale) Mirror 20: 0xac781410 (stale) Mirror 21: 0xac781410 (stale) Mirror 22: 0xac781410 (stale) Mirror 23: 0xb438bf06 (stale) Mirror 24: 0xb438bf06 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d323: skipped 000847 1782866310.088608 trunc from 0x02d323 to 0x00961d (0x23d05 bytes) 000848 1782866310.094649 mapread 0x00424d thru 0x007334 (0x30e8 bytes) 000849 1782866310.100963 punch from 0x002995 to 0x00961d (0x6c89 bytes) 000851 1782866310.103443 mapread 0x00267b thru 0x00961d (0x6fa3 bytes) 000852 1782866310.106933 punch from 0x0005df to 0x002752 (0x2174 bytes) 000854 1782866310.109146 trunc from 0x00961e to 0x02bc56 (0x22639 bytes) 000855 1782866310.113909 punch from 0x0150a5 to 0x021cf7 (0xcc53 bytes) 000856 1782866310.115796 punch from 0x0130d1 to 0x01a117 (0x7047 bytes) 000857 1782866310.117759 fallocate 0x0095b5 thru 0x00b773 (0x21bf bytes) 000858 1782866310.119899 write 0x017983 thru 0x01ad3e (0x33bc bytes) 000859 1782866310.123337 trunc from 0x02bc57 to 0x015250 (0x16a06 bytes) 000860 1782866310.127957 write 0x03104a thru 0x03a0f6 (0x90ad bytes) 000861 1782866310.129006 trunc from 0x017aa0 to 0x022988 (0xaee9 bytes) 000862 1782866310.130654 write 0x0289b9 thru 0x02c41e (0x3a66 bytes) 000863 1782866310.134764 MIRROR_RESYNC 15 mirrors 000864 1782866311.003784 punch from 0x0177e1 to 0x019b8e (0x23ae bytes) 000865 1782866311.013273 read 0x00d179 thru 0x00d748 (0x5d0 bytes) 000866 1782866311.015040 trunc from 0x0017b9 to 0x00ef54 (0xd79c bytes) 000867 1782866311.016815 trunc from 0x03a0f7 to 0x027a65 (0x12691 bytes) 000868 1782866311.023933 MIRROR_RESYNC 15 mirrors 000869 1782866311.436539 trunc from 0x027a66 to 0x00e184 (0x198e1 bytes) 000870 1782866311.452746 read 0x0051e8 thru 0x006a0a (0x1823 bytes) 000871 1782866311.454608 read 0x003228 thru 0x00718e (0x3f67 bytes) 000872 1782866311.457301 trunc from 0x00e185 to 0x03ca4e (0x2e8ca bytes) 000873 1782866311.462893 trunc from 0x03ca4f to 0x021659 (0x1b3f5 bytes) 000874 1782866311.466556 trunc from 0x008ea1 to 0x00fd04 (0x6e64 bytes) 000875 1782866311.468538 punch from 0x004751 to 0x008a8d (0x433d bytes) 000876 1782866311.470116 fallocate 0x0391e1 thru 0x03ae53 (0x1c73 bytes) 000878 1782866311.472357 trunc from 0x00529f to 0x00e492 (0x91f4 bytes) 000879 1782866311.474071 mapwrite 0x02df2f thru 0x032d7d (0x4e4f bytes) 000880 1782866311.487825 write 0x014539 thru 0x01dc53 (0x971b bytes) 000881 1782866311.489022 punch from 0x032a97 to 0x032d7d (0x2e7 bytes) 000882 1782866311.490808 trunc from 0x032d7e to 0x016eb7 (0x1bec6 bytes) 000883 1782866311.495459 punch from 0x0140ef to 0x016eb7 (0x2dc9 bytes) 000884 1782866311.498892 punch from 0x014b12 to 0x016eb7 (0x23a6 bytes) 000885 1782866311.500617 mapread 0x00d215 thru 0x016eb7 (0x9ca3 bytes) 000886 1782866311.508009 trunc from 0x016eb8 to 0x00ecd2 (0x81e5 bytes) 000887 1782866311.512380 trunc from 0x00ecd3 to 0x03493e (0x25c6c bytes) 000888 1782866311.516817 fallocate 0x01c107 thru 0x026eab (0xada5 bytes) 000889 1782866311.518631 fallocate 0x02dce5 thru 0x036835 (0x8b51 bytes) 000890 1782866311.520532 trunc from 0x02fcdf to 0x036835 (0x6b57 bytes) 000891 1782866311.522403 read 0x0346fa thru 0x036835 (0x213c bytes) 000892 1782866311.526067 trunc from 0x032879 to 0x036835 (0x3fbd bytes) 000893 1782866311.527772 trunc from 0x036836 to 0x03fb83 (0x934e bytes) 000894 1782866311.531813 trunc from 0x00d091 to 0x018fe5 (0xbf55 bytes) 000895 1782866311.533464 fallocate 0x00c25a thru 0x00d3bb (0x1162 bytes) 000896 1782866311.535158 write 0x0130e2 thru 0x01362c (0x54b bytes) 000897 1782866311.538093 fallocate 0x0299ad thru 0x0326a5 (0x8cf9 bytes) 000898 1782866311.539899 punch from 0x00926e to 0x00980e (0x5a1 bytes) 000899 1782866311.541456 read 0x03d7d3 thru 0x03fb83 (0x23b1 bytes) 000900 1782866311.541894 fallocate 0x0071cd thru 0x00913c (0x1f70 bytes) 000902 1782866311.543925 punch from 0x038d04 to 0x03eb54 (0x5e51 bytes) 000903 1782866311.545545 mapread 0x02a928 thru 0x02c151 (0x182a bytes) 000904 1782866311.546251 punch from 0x0169c3 to 0x01c073 (0x56b1 bytes) 000906 1782866311.548310 punch from 0x01b58f to 0x025bdc (0xa64e bytes) 000907 1782866311.549812 read 0x030a32 thru 0x0385b9 (0x7b88 bytes) 000908 1782866311.552926 trunc from 0x034267 to 0x03f452 (0xb1ec bytes) 000909 1782866311.554625 trunc from 0x0367a8 to 0x03baf3 (0x534c bytes) 000910 1782866311.556109 read 0x00638b thru 0x00704f (0xcc5 bytes) 000912 1782866311.558688 MIRROR_RESYNC 15 mirrors 000913 1782866312.075727 mapwrite 0x019d54 thru 0x01da72 (0x3d1f bytes) 000915 1782866312.108712 fallocate 0x027095 thru 0x02b718 (0x4684 bytes) 000916 1782866312.110688 trunc from 0x0321af to 0x035654 (0x34a6 bytes) 000917 1782866312.112250 read 0x01ea46 thru 0x02b508 (0xcac3 bytes) 000918 1782866312.114796 write 0x002b54 thru 0x0094a9 (0x6956 bytes) 000919 1782866312.117620 trunc from 0x03fb84 to 0x02e0d4 (0x11aaf bytes) 000920 1782866312.126719 trunc from 0x02c138 to 0x02e0d4 (0x1f9d bytes) 000921 1782866312.128195 mapwrite 0x013d73 thru 0x01ce90 (0x911e bytes) 000922 1782866312.134841 trunc from 0x00ca1b to 0x019484 (0xca6a bytes) 000923 1782866312.136811 write 0x02cbd9 thru 0x0335bd (0x69e5 bytes) 000924 1782866312.138902 write 0x001d9f thru 0x010003 (0xe265 bytes) 000925 1782866312.142258 fallocate 0x03e4aa thru 0x03ffff (0x1b56 bytes) 000926 1782866312.144249 punch from 0x02a141 to 0x0335bd (0x947d bytes) 000928 1782866312.148858 trunc from 0x0335be to 0x015370 (0x1e24d bytes) 000929 1782866312.153181 mapread 0x012afb thru 0x015370 (0x2876 bytes) 000930 1782866312.155779 write 0x0393bd thru 0x03ffff (0x6c43 bytes) 000931 1782866312.156674 trunc from 0x040000 to 0x027126 (0x18ed9 bytes) 000932 1782866312.160696 mapread 0x00b464 thru 0x013970 (0x850d bytes) 000933 1782866312.161623 write 0x013eaf thru 0x01dd5b (0x9ead bytes) 000934 1782866312.163637 mapwrite 0x013987 thru 0x017d1b (0x4395 bytes) 000935 1782866312.168743 mapread 0x020127 thru 0x0214f4 (0x13ce bytes) 000936 1782866312.171555 read 0x01b622 thru 0x027126 (0xbb05 bytes) 000938 1782866312.172370 mapwrite 0x007ec0 thru 0x012cf8 (0xae39 bytes) 000939 1782866312.178628 write 0x01f40c thru 0x02a8f9 (0xb4ee bytes) 000940 1782866312.179642 trunc from 0x00f9e2 to 0x01cc19 (0xd238 bytes) 000941 1782866312.181639 write 0x026748 thru 0x02913b (0x29f4 bytes) 000942 1782866312.182325 mapread 0x00dcce thru 0x012559 (0x488c bytes) 000943 1782866312.185092 mapread 0x00b056 thru 0x0189ac (0xd957 bytes) 000944 1782866312.185929 mapwrite 0x002baa thru 0x00d9d1 (0xae28 bytes) 000945 1782866312.194573 MIRROR_RESYNC 15 mirrors 000946 1782866312.756848 MIRROR_RESYNC 15 mirrors 000947 1782866312.929907 write 0x00e43d thru 0x013a72 (0x5636 bytes) 000948 1782866312.943610 mapwrite 0x021d3a thru 0x02957f (0x7846 bytes) 000949 1782866312.957440 MIRROR_SPLIT mirror 6 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x52:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a8fa): Mirror 5: 0x50174745 (stale) Mirror 6: 0x50174745 (stale) Mirror 11: 0x50174745 (stale) Mirror 12: 0x50174745 (stale) Mirror 13: 0x50174745 (stale) Mirror 17: 0x50174745 (stale) Mirror 18: 0x50174745 (stale) Mirror 19: 0x50174745 (stale) Mirror 20: 0x50174745 (stale) Mirror 21: 0x50174745 (stale) Mirror 22: 0x50174745 (stale) Mirror 23: 0x50174745 (stale) Mirror 24: 0x50174745 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a8fa): Mirror 5: 0x50174745 (stale) Mirror 6: 0x50174745 (stale) Mirror 11: 0x50174745 (stale) Mirror 12: 0x50174745 (stale) Mirror 13: 0x50174745 (stale) Mirror 17: 0x50174745 (stale) Mirror 18: 0x50174745 (stale) Mirror 19: 0x50174745 (stale) Mirror 20: 0x50174745 (stale) Mirror 21: 0x50174745 (stale) Mirror 22: 0x50174745 (stale) Mirror 23: 0x50174745 (stale) Mirror 24: 0x50174745 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a8fa: skipped 000950 1782866313.113673 read 0x008af6 thru 0x00fb9e (0x70a9 bytes) 000951 1782866313.116191 write 0x0017b1 thru 0x0031b7 (0x1a07 bytes) 000953 1782866313.120411 write 0x0365be thru 0x03c01a (0x5a5d bytes) 000954 1782866313.121232 read 0x02949e thru 0x033db9 (0xa91c bytes) 000955 1782866313.124372 punch from 0x000ca2 to 0x004ac8 (0x3e27 bytes) 000956 1782866313.128316 read 0x003d8c thru 0x010172 (0xc3e7 bytes) 000957 1782866313.130833 trunc from 0x00ea6a to 0x01b63f (0xcbd6 bytes) 000958 1782866313.132701 write 0x0374e1 thru 0x038a6c (0x158c bytes) 000959 1782866313.133407 punch from 0x01e8c8 to 0x02d4fa (0xec33 bytes) 000960 1782866313.135224 fallocate 0x033f44 thru 0x03cda7 (0x8e64 bytes) 000961 1782866313.137110 mapread 0x01ef4f thru 0x0297ab (0xa85d bytes) 000962 1782866313.153555 mapwrite 0x012a5d thru 0x0224c7 (0xfa6b bytes) 000963 1782866313.174660 trunc from 0x00f887 to 0x01099f (0x1119 bytes) 000964 1782866313.176198 mapwrite 0x002686 thru 0x011717 (0xf092 bytes) 000965 1782866313.189691 write 0x0381d9 thru 0x03ffff (0x7e27 bytes) 000966 1782866313.190878 mapwrite 0x02c38a thru 0x02ebe9 (0x2860 bytes) 000967 1782866313.199066 punch from 0x022d73 to 0x024a0c (0x1c9a bytes) 000968 1782866313.200944 read 0x010613 thru 0x010b4c (0x53a bytes) 000969 1782866313.202951 MIRROR_RESYNC 14 mirrors 000970 1782866313.866408 trunc from 0x03d2c5 to 0x03ffff (0x2d3b bytes) 000971 1782866313.876889 trunc from 0x035daa to 0x03a5cd (0x4824 bytes) 000972 1782866313.878468 mapread 0x02653a thru 0x031aca (0xb591 bytes) 000973 1782866313.885386 punch from 0x02c694 to 0x034c65 (0x85d2 bytes) 000974 1782866313.887407 mapwrite 0x008542 thru 0x01211a (0x9bd9 bytes) 000975 1782866313.897928 trunc from 0x01baed to 0x022aa2 (0x6fb6 bytes) 000978 1782866313.900334 trunc from 0x01fd7c to 0x02cb80 (0xce05 bytes) 000979 1782866313.902174 trunc from 0x028b42 to 0x02a2a9 (0x1768 bytes) 000981 1782866313.904125 write 0x027675 thru 0x028b7f (0x150b bytes) 000982 1782866313.907196 trunc from 0x01fe93 to 0x023bae (0x3d1c bytes) 000983 1782866313.908810 trunc from 0x01298b to 0x021a7b (0xf0f1 bytes) 000984 1782866313.910760 punch from 0x01ca7e to 0x023af5 (0x7078 bytes) 000985 1782866313.912599 mapread 0x01997a thru 0x024a54 (0xb0db bytes) 000986 1782866313.917458 punch from 0x0334be to 0x03fd8c (0xc8cf bytes) 000987 1782866313.919160 fallocate 0x030cc0 thru 0x03ae8c (0xa1cd bytes) 000988 1782866313.921051 fallocate 0x036df2 thru 0x03ffff (0x920e bytes) 000989 1782866313.924072 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x17fb82b (stale) Mirror 6: 0x17fb82b (stale) Mirror 11: 0x17fb82b (stale) Mirror 12: 0x17fb82b (stale) Mirror 13: 0x17fb82b (stale) Mirror 17: 0x17fb82b (stale) Mirror 18: 0x17fb82b (stale) Mirror 19: 0x17fb82b (stale) Mirror 20: 0x17fb82b (stale) Mirror 21: 0x17fb82b (stale) Mirror 22: 0x17fb82b (stale) Mirror 23: 0x17fb82b (stale) Mirror 24: 0x17fb82b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3d353b8a Mirror 25: 0x3d353b8a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000990 1782866314.218841 mapwrite 0x02f833 thru 0x032d0e (0x34dc bytes) 000991 1782866314.256483 write 0x004e8b thru 0x00e92f (0x9aa5 bytes) 000992 1782866314.261039 punch from 0x02111a to 0x0251fb (0x40e2 bytes) 000993 1782866314.262821 write 0x026268 thru 0x0332e1 (0xd07a bytes) 000994 1782866314.265409 mapread 0x02e7cc thru 0x03a94a (0xc17f bytes) 000995 1782866314.268619 mapread 0x03a033 thru 0x03ffff (0x5fcd bytes) 000996 1782866314.269461 read 0x03c327 thru 0x03d3d8 (0x10b2 bytes) 000998 1782866314.270232 read 0x02b49b thru 0x03a98b (0xf4f1 bytes) 000999 1782866314.272046 MIRROR_RESYNC 15 mirrors 001000 1782866314.953965 fallocate 0x025380 thru 0x030290 (0xaf11 bytes) All operations completed A-OK! PASS 70a (31s) == sanity-flr test 71: check mirror extend parent fid ==== 20:38:36 (1782866316) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x81:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 72: check append on FLR file ========== 20:38:39 (1782866319) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0562153 s, 37.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319127 s, 32.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 20:38:42 (1782866322) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 7724800 .. Extending file size to 2080896 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7808960 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 5766112 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6901088 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8656096 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8500320 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8929152 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9027808 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8578528 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 66016 66017 66019 66021 66025 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) 192.168.203.136@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (64s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 20:39:46 (1782866386) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5190752 .. Extending file size to 3029312 .. failed Extending file size to 4102560 .. Extending file size to 8538432 .. Extending file size to 7392576 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8404096 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8617856 .. failed Extending file size to 8779840 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8843744 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9032000 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8548768 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8609920 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9178400 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done Extending file size to 8737344 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8931520 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9222688 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9348320 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9390880 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 69466 69467 69469 69471 69473 done 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) 192.168.203.136@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 20:40:51 (1782866451) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2804 1268500 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2924 1268380 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26244 3580776 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18032 3588988 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 44276 7169764 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2804 1268500 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2924 1268380 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 26244 3580776 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18032 3588988 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 44276 7169764 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0391917 s, 53.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0648383 s, 32.3 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x8643e74e (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) PASS 200c (7s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 20:40:59 (1782866459) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0339896 s, 61.7 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 20:41:02 (1782866462) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x90:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12b:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12b:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x14b:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 20:41:04 (1782866464) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0195424 s, 107 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 20:41:06 (1782866466) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0212794 s, 98.6 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } PASS 204b (2s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 20:41:08 (1782866468) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0189157 s, 111 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbfd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbfe:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbfd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbfe:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xbff:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 20:41:32 (1782866492) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg336-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0164059 s, 63.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbfe:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc01:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc00:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbfe:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc01:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0xc00:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0xbff:0x0] } PASS 204d (35s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 20:42:07 (1782866527) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg336-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0196076 s, 107 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (33s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 20:42:40 (1782866560) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg336-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg336-server: Pool lustre.test_204f created Adding targets to pool oleg336-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg336-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0279961 s, 74.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x152:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg336-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg336-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg336-server: Pool lustre.test_204f destroyed PASS 204f (47s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 20:43:27 (1782866607) PASS 205a (2s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 20:43:29 (1782866609) PASS 205b (2s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 20:43:31 (1782866611) oleg336-server: Pool lustre.test_206 created Adding targets to pool oleg336-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg336-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg336-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg336-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg336-server: Pool lustre.test_206 destroyed PASS 206 (15s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 20:43:46 (1782866626) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0304778 s, 34.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0114689 s, 91.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x3249f756 Mirror 3: 0x3249f756 PASS 207 (3s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 20:43:49 (1782866629) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (1s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 20:43:50 (1782866630) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (1s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 20:43:51 (1782866631) Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0100314 s, 0.4 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xa60000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x20f0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x20f0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x20f0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1d20000 Read this (4 bytes): BBBB 192.168.203.136@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg336-client.virtnet /mnt/lustre2 (opts:) PASS 209a (16s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 20:44:07 (1782866647) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00333299 s, 2.5 MB/s the very first read read with warm cache PASS 209b (2s) == sanity-flr test 210a: handle broken mirrored lovea ==== 20:44:09 (1782866649) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0150287 s, 69.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb1 lmm_fid: [0x200000405:0xb1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb1 lmm_fid: [0x200000405:0xb1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611313 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-30 20:44:09.000000000 -0400 Modify: 2026-06-30 20:44:09.000000000 -0400 Change: 2026-06-30 20:44:10.000000000 -0400 Birth: 2026-06-30 20:44:09.000000000 -0400 PASS 210a (2s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 20:44:11 (1782866651) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0128375 s, 81.7 MB/s oleg336-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg336-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg336-server: O/280000401/d28/15c: File not found by ext2_lookup PASS 210b (12s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 20:44:23 (1782866663) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.035063 s, 1.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00471705 s, 868 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 30 20:44 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xb6:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 30 20:44 /mnt/lustre/f211.sanity-flr PASS 211 (2s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 20:44:25 (1782866665) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (2s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 20:44:27 (1782866667) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (3s) == sanity-flr test complete, duration 806 sec ============ 20:44:30 (1782866670) === sanity-flr: start cleanup 20:44:30 (1782866670) === === sanity-flr: finish cleanup 20:44:32 (1782866672) ===