-----============= acceptance-small: sanity-flr ============----- Wed Aug 26 18:31:47 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg240-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 18:32:09 (1787783529) === oleg240-client.virtnet: executing check_config_client /mnt/lustre oleg240-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg240-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff94ff12f71800.idle_timeout=debug osc.lustre-OST0001-osc-ffff94ff12f71800.idle_timeout=debug disable quota as required oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 18:32:51 (1787783571) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8245] == sanity-flr test 0a: lfs mirror create with -N option == 18:32:54 (1787783574) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 18:33:13 (1787783593) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 18:33:16 (1787783596) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 18:33:20 (1787783600) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 18:33:43 (1787783623) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 18:33:46 (1787783626) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 18:33:49 (1787783629) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff94ff12f71800.stats=clear osc.lustre-OST0001-osc-ffff94ff12f71800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 18:33:58 (1787783638) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (10s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 18:34:08 (1787783648) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 18:34:16 (1787783656) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 18:34:20 (1787783660) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 18:34:27 (1787783667) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 18:34:37 (1787783677) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 18:34:44 (1787783684) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 18:34:52 (1787783692) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 18:34:59 (1787783699) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 18:35:09 (1787783709) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 18:35:16 (1787783716) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 1.80397 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132847 s, 7.9 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 18:35:27 (1787783727) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.387095 s, 35.2 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 18:35:38 (1787783738) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (14s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 18:35:52 (1787783752) Stopping /mnt/lustre-ost1 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94ff12f71800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94ff12f71800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg240-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94ff12f71800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94ff12f71800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg240-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (81s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 18:37:13 (1787783833) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.389231 s, 13.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94ff12f71800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94ff12f71800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg240-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 18:37:59 (1787783879) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94ff12f71800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94ff12f71800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg240-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94ff12f71800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94ff12f71800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg240-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 18:39:21 (1787783961) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.53985 s, 20.6 MB/s -rw-r--r-- 1 root root 52428800 Aug 26 18:39 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.266s user 0m0.000s sys 0m0.184s Stopping /mnt/lustre-ost1 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff94ff12f71800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff94ff12f71800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.175s user 0m0.001s sys 0m0.151s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg240-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg240-server oleg240-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff94ff12f71800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94ff12f71800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.250s user 0m0.001s sys 0m0.184s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg240-server: oleg240-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg240-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg240-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff94ff12f71800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff94ff12f71800.ost_server_uuid in FULL state after 0 sec PASS 33b (89s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 18:40:50 (1787784050) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 18:40:54 (1787784054) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 18:40:57 (1787784057) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 18:41:02 (1787784062) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 18:41:10 (1787784070) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.863112 s, 14.6 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 18:41:23 (1787784083) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383023 s, 27.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0358823 s, 29.2 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 18:41:32 (1787784092) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 18:41:41 (1787784101) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00219856 s, 466 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00196934 s, 520 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0011885 s, 862 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00149659 s, 684 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 18:41:52 (1787784112) create test files with size 15750220 .. a95b86aa6d784a9eb0c02832cdd48c48 - 3be949ae092b8ee7081c4d45b9d3b7fa - 417164da9797bef54f70d4a6c36d47af - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg240-client.virtnet /mnt/lustre (opts:) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 18:42:19 (1787784139) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 18:42:38 (1787784158) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 18:42:47 (1787784167) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915815 s, 11.4 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283822 s, 36.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 18:42:57 (1787784177) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0746032 s, 28.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.469095 s, 8.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.187333 s, 11.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 18:43:16 (1787784196) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 18:43:21 (1787784201) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 18:43:26 (1787784206) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0518718 s, 20.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (11s) == sanity-flr test 43c: read prefer uncompressed mirror == 18:43:36 (1787784216) llite.lustre-ffff94ff067a5800.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (8s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 18:43:45 (1787784225) llite.lustre-ffff94ff067a5800.enable_compression=1 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526181 s, 19.9 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774406 s, 13.5 MB/s **verify components PASS 43d (10s) == sanity-flr test 44a: lfs mirror split check =========== 18:43:55 (1787784235) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.292235 s, 10.8 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 18:44:06 (1787784246) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 18:44:18 (1787784258) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.250729 s, 41.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x8:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg240-client.virtnet /mnt/lustre (opts:) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (22s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 18:44:40 (1787784280) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff94ff05790800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.490688 s, 21.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg240-client.virtnet /mnt/lustre (opts:) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff94ff11875000.xattr_cache=1 PASS 44e (21s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 18:45:01 (1787784301) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (43s) == sanity-flr test 46: Verify setstripe --copy option ==== 18:45:44 (1787784344) striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (45s) == sanity-flr test 47: Verify mirror obj alloc =========== 18:46:29 (1787784389) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 18:46:33 (1787784393) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.293784 s, 10.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.184839 s, 17.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x8e476ad0 (nosync,stale) Mirror 2: 0x8e476ad0 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 48d0e6f663e826c0050ca847c8a27e11 - mirror 1 checksum: 88103cc800711c3fcf6d1b1e52a24b43 - mirror 2 checksum: 48d0e6f663e826c0050ca847c8a27e11 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T18:46:37-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-08-26T18:46:37-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 48d0e6f663e826c0050ca847c8a27e11 - mirror 1 checksum: 88103cc800711c3fcf6d1b1e52a24b43 - mirror 2 checksum: 88103cc800711c3fcf6d1b1e52a24b43 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T18:46:37-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-08-26T18:46:37-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 48 (13s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 18:46:46 (1787784406) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732495 s, 14.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (10s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 18:46:56 (1787784416) Failing mds1 on oleg240-server Stopping /mnt/lustre-mds1 (opts:) on oleg240-server 18:47:10 (1787784430) shut down facet: mds1 facet_host: oleg240-server facet_failover_host: oleg240-server Failover mds1 to oleg240-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg240-server: oleg240-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-MDT0000 18:47:48 (1787784468) targets are mounted 18:47:48 (1787784468) facet_failover done oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.035618 s, 29.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 50A (72s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 18:48:09 (1787784489) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0443785 s, 461 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0183653 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 18:48:19 (1787784499) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0905681 s, 226 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0248266 s, 825 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 18:48:30 (1787784510) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0106178 s, 1.5 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 18:48:39 (1787784519) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0523556 s, 391 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 18:48:49 (1787784529) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0151088 s, 271 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0489424 s, 83.7 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 18:48:59 (1787784539) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x13:0x0]: Invalid argument (22) PASS 60b (23s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 18:49:22 (1787784562) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 18:49:52 (1787784592) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1e:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 18:50:17 (1787784617) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 18:50:41 (1787784641) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 18:50:51 (1787784651) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 58872 mirror split pid 58874 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58872 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5089: 58874 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 18:51:59 (1787784719) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 4721 skipping zero length punch hole skipping zero length zero range skipping zero size read 000004 1787784721.835349 write 0x03338c thru 0x036d26 (0x399b bytes) 000005 1787784721.838350 punch from 0x010885 to 0x01c550 (0xbccc bytes) 000006 1787784721.844229 punch from 0x029a24 to 0x031058 (0x7635 bytes) 000007 1787784721.853218 write 0x00105a thru 0x00452a (0x34d1 bytes) 000008 1787784721.868450 MIRROR_EXTEND to 2 mirrors 000009 1787784723.022581 MIRROR_RESYNC 2 mirrors 000010 1787784723.328629 trunc from 0x0051de to 0x0145a0 (0xf3c3 bytes) fallocating to largest ever: 0x13fc7 000011 1787784723.390936 fallocate 0x01094e thru 0x013fc6 (0x3679 bytes) 000012 1787784723.397120 punch from 0x003290 to 0x00e1e9 (0xaf5a bytes) 000013 1787784723.403190 punch from 0x01cf26 to 0x021eac (0x4f87 bytes) 000014 1787784723.409181 read 0x00fd43 thru 0x0110c4 (0x1382 bytes) 000016 1787784723.420580 write 0x021bc1 thru 0x02998d (0x7dcd bytes) truncating to largest ever: 0x15f94 000017 1787784723.447730 trunc from 0x036d27 to 0x015f93 (0x20d93 bytes) 000018 1787784723.484425 read 0x00db5c thru 0x015f93 (0x8438 bytes) 000019 1787784723.502732 trunc from 0x015f94 to 0x00a042 (0xbf51 bytes) 000020 1787784723.529805 trunc from 0x005faf to 0x00a042 (0x4094 bytes) 000021 1787784723.536711 mapwrite 0x03d437 thru 0x03e0ee (0xcb8 bytes) 000023 1787784723.595172 write 0x0038ea thru 0x009cbe (0x63d5 bytes) truncating to largest ever: 0x267d4 000025 1787784723.604238 trunc from 0x03e0ef to 0x0267d3 (0x1791b bytes) 000026 1787784723.623712 mapread 0x0207c1 thru 0x02660b (0x5e4b bytes) fallocating to largest ever: 0x3340d 000027 1787784723.626624 fallocate 0x0269e9 thru 0x03340c (0xca24 bytes) 000028 1787784723.633427 read 0x00f238 thru 0x016db8 (0x7b81 bytes) 000029 1787784723.645375 mapread 0x02b587 thru 0x03340c (0x7e86 bytes) 000030 1787784723.647450 punch from 0x029085 to 0x03340c (0xa388 bytes) 000031 1787784723.656608 fallocate 0x021a3b thru 0x030746 (0xed0c bytes) 000032 1787784723.680949 write 0x02f213 thru 0x03dc62 (0xea50 bytes) 000033 1787784723.697503 mapwrite 0x03b682 thru 0x03ffff (0x497e bytes) 000035 1787784723.798363 read 0x03e6a2 thru 0x03ffff (0x195e bytes) 000036 1787784723.801657 trunc from 0x0037af to 0x0041c3 (0xa15 bytes) 000039 1787784723.816678 punch from 0x01cfd0 to 0x01ebfb (0x1c2c bytes) 000040 1787784723.822098 read 0x00a3bb thru 0x00ca5a (0x26a0 bytes) 000041 1787784723.823196 read 0x01b0e1 thru 0x0243d9 (0x92f9 bytes) 000042 1787784723.850210 punch from 0x039ab3 to 0x03af7d (0x14cb bytes) 000043 1787784723.854023 fallocate 0x003405 thru 0x00cd10 (0x990c bytes) 000044 1787784723.859195 punch from 0x00172c to 0x006e64 (0x5739 bytes) 000045 1787784723.867113 punch from 0x028bd4 to 0x029507 (0x934 bytes) 000046 1787784723.872799 trunc from 0x040000 to 0x003f65 (0x3c09a bytes) 000047 1787784723.898359 trunc from 0x0009ef to 0x003f65 (0x3577 bytes) 000048 1787784723.902233 trunc from 0x003f66 to 0x0120aa (0xe145 bytes) fallocating to largest ever: 0x35ce3 000049 1787784723.913752 fallocate 0x02d2c1 thru 0x035ce2 (0x8a22 bytes) 000050 1787784723.919446 fallocate 0x004eb9 thru 0x012352 (0xd49a bytes) 000051 1787784723.924307 mapread 0x02688f thru 0x030831 (0x9fa3 bytes) 000052 1787784723.989891 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) 000053 1787784724.148513 read 0x004dff thru 0x0050db (0x2dd bytes) 000054 1787784724.155280 punch from 0x029547 to 0x035ce2 (0xc79c bytes) 000055 1787784724.164539 punch from 0x002594 to 0x010c71 (0xe6de bytes) 000057 1787784724.173385 trunc from 0x035ce3 to 0x0172c1 (0x1ea21 bytes) 000058 1787784724.217380 mapwrite 0x007760 thru 0x010557 (0x8df8 bytes) 000059 1787784724.288193 write 0x016039 thru 0x01b2a2 (0x526a bytes) 000060 1787784724.297172 mapread 0x001f9a thru 0x00e9fe (0xca65 bytes) fallocating to largest ever: 0x40000 000061 1787784724.307362 fallocate 0x03f515 thru 0x03ffff (0xaeb bytes) 000064 1787784724.316670 mapwrite 0x036c85 thru 0x03ffff (0x937b bytes) 000065 1787784724.368335 mapwrite 0x00f28a thru 0x015d35 (0x6aac bytes) 000066 1787784724.402431 trunc from 0x00be55 to 0x01b734 (0xf8e0 bytes) 000067 1787784724.410076 write 0x0170c8 thru 0x0245f4 (0xd52d bytes) 000068 1787784724.433776 fallocate 0x0149a8 thru 0x01a35b (0x59b4 bytes) 000069 1787784724.441336 punch from 0x00b370 to 0x00c460 (0x10f1 bytes) 000070 1787784724.448049 fallocate 0x010f58 thru 0x01ea3c (0xdae5 bytes) 000071 1787784724.454702 punch from 0x032ae4 to 0x03c069 (0x9586 bytes) 000072 1787784724.464070 trunc from 0x040000 to 0x01c29d (0x23d62 bytes) 000073 1787784724.487211 punch from 0x0113ad to 0x01c29d (0xaef1 bytes) 000074 1787784724.507656 read 0x01bd19 thru 0x01c29d (0x585 bytes) 000075 1787784724.535497 trunc from 0x01c29e to 0x014c7c (0x7621 bytes) 000076 1787784724.563393 mapwrite 0x03f94b thru 0x03ffff (0x6b5 bytes) 000077 1787784724.611880 mapwrite 0x0133a3 thru 0x015597 (0x21f5 bytes) 000078 1787784724.641920 fallocate 0x0079ca thru 0x00a679 (0x2cb0 bytes) 000079 1787784724.651799 write 0x00efba thru 0x0193f0 (0xa437 bytes) 000081 1787784724.668746 punch from 0x023f0d to 0x0313b9 (0xd4ad bytes) 000082 1787784724.687182 fallocate 0x02649f thru 0x02d6bd (0x721f bytes) 000083 1787784724.698232 fallocate 0x00484f thru 0x008fd7 (0x4789 bytes) 000084 1787784724.711374 trunc from 0x040000 to 0x00ca05 (0x335fa bytes) 000086 1787784724.740128 trunc from 0x00ca06 to 0x001be2 (0xae23 bytes) 000087 1787784724.769173 trunc from 0x001038 to 0x001441 (0x40a bytes) 000088 1787784724.781239 fallocate 0x02e263 thru 0x037038 (0x8dd6 bytes) 000089 1787784724.788873 trunc from 0x001be3 to 0x038bb1 (0x36fcf bytes) 000090 1787784724.819221 write 0x025c64 thru 0x031d86 (0xc123 bytes) 000092 1787784724.845729 fallocate 0x02ec9d thru 0x03b090 (0xc3f4 bytes) 000093 1787784724.861468 punch from 0x032384 to 0x038334 (0x5fb1 bytes) 000094 1787784724.873563 read 0x02e984 thru 0x03b090 (0xc70d bytes) 000095 1787784724.892780 trunc from 0x03b091 to 0x035cde (0x53b2 bytes) 000097 1787784724.910328 trunc from 0x035cdf to 0x0176ff (0x1e5df bytes) 000098 1787784724.937254 fallocate 0x03a715 thru 0x03ffff (0x58eb bytes) 000099 1787784724.947589 mapread 0x01462f thru 0x0224a3 (0xde75 bytes) 000100 1787784725.067539 punch from 0x0241b9 to 0x025633 (0x147b bytes) 000101 1787784725.074638 punch from 0x005aa1 to 0x00abf3 (0x5153 bytes) 000103 1787784725.085238 write 0x027a6b thru 0x02e8f6 (0x6e8c bytes) 000105 1787784725.104317 write 0x026ff0 thru 0x02febf (0x8ed0 bytes) 000106 1787784725.115772 trunc from 0x0144dd to 0x02323e (0xed62 bytes) 000107 1787784725.137881 fallocate 0x036a88 thru 0x03ffff (0x9578 bytes) 000108 1787784725.145183 mapread 0x00c182 thru 0x015bae (0x9a2d bytes) 000110 1787784725.163268 write 0x01b378 thru 0x020cfc (0x5985 bytes) 000111 1787784725.181548 fallocate 0x01ef9d thru 0x028dcc (0x9e30 bytes) 000112 1787784725.189139 write 0x00b40a thru 0x014176 (0x8d6d bytes) 000113 1787784725.191847 write 0x028b01 thru 0x038006 (0xf506 bytes) 000114 1787784725.202463 write 0x012e30 thru 0x01b9c5 (0x8b96 bytes) 000117 1787784725.204951 punch from 0x019a50 to 0x0261b8 (0xc769 bytes) 000118 1787784725.229886 punch from 0x0153e2 to 0x01638b (0xfaa bytes) 000119 1787784725.234521 trunc from 0x027e12 to 0x027fa0 (0x18f bytes) 000120 1787784725.247491 trunc from 0x002eb7 to 0x004bf7 (0x1d41 bytes) 000121 1787784725.257788 MIRROR_EXTEND to 2 mirrors 000122 1787784725.947697 trunc from 0x040000 to 0x03bf1e (0x40e1 bytes) 000123 1787784726.116578 fallocate 0x027ce6 thru 0x031997 (0x9cb2 bytes) 000124 1787784726.146670 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) 000125 1787784726.389917 mapread 0x020d47 thru 0x023167 (0x2421 bytes) 000126 1787784726.409875 punch from 0x022b80 to 0x0259f2 (0x2e73 bytes) 000127 1787784726.414577 mapwrite 0x005fed thru 0x008d36 (0x2d4a bytes) 000128 1787784726.450752 read 0x02b701 thru 0x02fa36 (0x4336 bytes) 000129 1787784726.470680 fallocate 0x00c9ef thru 0x010c04 (0x4216 bytes) 000130 1787784726.512180 punch from 0x01b20d to 0x027dce (0xcbc2 bytes) 000131 1787784726.527650 fallocate 0x028932 thru 0x0368dc (0xdfab bytes) 000132 1787784726.538269 trunc from 0x01a767 to 0x021201 (0x6a9b bytes) 000133 1787784726.552706 MIRROR_EXTEND to 2 mirrors 000134 1787784727.319518 mapwrite 0x00489f thru 0x01002c (0xb78e bytes) 000135 1787784727.492139 mapwrite 0x024bdd thru 0x02eb70 (0x9f94 bytes) 000137 1787784727.612936 write 0x02e469 thru 0x039700 (0xb298 bytes) 000139 1787784727.629493 mapwrite 0x03a5af thru 0x03ffff (0x5a51 bytes) 000141 1787784727.703241 read 0x037bfc thru 0x03be5a (0x425f bytes) 000142 1787784727.704114 mapwrite 0x01f5c9 thru 0x029cac (0xa6e4 bytes) 000143 1787784727.739470 write 0x02fb80 thru 0x039a7d (0x9efe bytes) 000145 1787784727.741263 fallocate 0x004bad thru 0x01301a (0xe46e bytes) 000146 1787784727.748273 mapwrite 0x010ce3 thru 0x012919 (0x1c37 bytes) 000147 1787784727.790117 write 0x026aab thru 0x028d02 (0x2258 bytes) 000148 1787784727.791418 mapwrite 0x038c26 thru 0x03c89e (0x3c79 bytes) 000150 1787784727.830457 fallocate 0x003065 thru 0x00cb43 (0x9adf bytes) 000151 1787784727.844948 mapread 0x03ece0 thru 0x03ffff (0x1320 bytes) 000153 1787784727.847299 write 0x032ece thru 0x03bf79 (0x90ac bytes) 000154 1787784727.856242 punch from 0x03b97d to 0x03ffff (0x4683 bytes) 000156 1787784727.882117 punch from 0x02ca30 to 0x03b2ef (0xe8c0 bytes) 000157 1787784727.894716 mapwrite 0x0222b4 thru 0x02ab15 (0x8862 bytes) 000158 1787784727.943300 trunc from 0x0169f7 to 0x019f33 (0x353d bytes) 000160 1787784727.951370 trunc from 0x02de92 to 0x037fb0 (0xa11f bytes) 000161 1787784727.957668 trunc from 0x016ea4 to 0x019ee4 (0x3041 bytes) 000162 1787784727.969951 read 0x0247a0 thru 0x024ad2 (0x333 bytes) 000163 1787784727.970656 fallocate 0x03ddc3 thru 0x03ffff (0x223d bytes) 000164 1787784727.979216 trunc from 0x02aad6 to 0x03383e (0x8d69 bytes) 000165 1787784727.991499 punch from 0x0091f7 to 0x0178b4 (0xe6be bytes) 000166 1787784727.998648 mapread 0x016627 thru 0x01e1f4 (0x7bce bytes) 000167 1787784728.023137 mapwrite 0x009592 thru 0x0135ac (0xa01b bytes) 000168 1787784728.075946 fallocate 0x018db7 thru 0x01c743 (0x398d bytes) 000169 1787784728.081598 mapread 0x010540 thru 0x01f9a6 (0xf467 bytes) 000170 1787784728.082566 read 0x02c3f2 thru 0x038844 (0xc453 bytes) 000171 1787784728.083199 trunc from 0x00dd17 to 0x017336 (0x9620 bytes) 000172 1787784728.089200 mapwrite 0x0012ef thru 0x002a77 (0x1789 bytes) 000173 1787784728.103424 write 0x03f658 thru 0x03ffff (0x9a8 bytes) 000175 1787784728.105301 trunc from 0x040000 to 0x0220d3 (0x1df2c bytes) 000176 1787784728.130653 punch from 0x018d05 to 0x0208b9 (0x7bb5 bytes) 000177 1787784728.138424 mapwrite 0x01ee0a thru 0x027b6b (0x8d62 bytes) 000178 1787784728.191975 trunc from 0x0093e6 to 0x01724e (0xde69 bytes) 000179 1787784728.199661 trunc from 0x027b6c to 0x00656b (0x21600 bytes) 000180 1787784728.222089 mapwrite 0x034a63 thru 0x038d4a (0x42e8 bytes) 000182 1787784728.284728 punch from 0x02a2bb to 0x031b6c (0x78b2 bytes) 000183 1787784728.293506 write 0x03d7ef thru 0x03ffff (0x2811 bytes) 000184 1787784728.302515 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) 000185 1787784728.541063 mapread 0x034577 thru 0x035f40 (0x19ca bytes) 000186 1787784728.569276 read 0x01d01c thru 0x028c4b (0xbc30 bytes) 000187 1787784728.588135 fallocate 0x020910 thru 0x02abc6 (0xa2b7 bytes) 000188 1787784728.600523 punch from 0x02a143 to 0x02f202 (0x50c0 bytes) 000189 1787784728.615472 mapwrite 0x033848 thru 0x03b504 (0x7cbd bytes) 000190 1787784728.689501 mapwrite 0x02c65d thru 0x038847 (0xc1eb bytes) 000191 1787784728.713997 MIRROR_EXTEND to 2 mirrors 000192 1787784729.667463 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) 000193 1787784729.824946 write 0x008027 thru 0x00e358 (0x6332 bytes) 000194 1787784729.849588 read 0x01c70c thru 0x02629d (0x9b92 bytes) 000196 1787784729.863072 read 0x0278e6 thru 0x037533 (0xfc4e bytes) 000198 1787784729.874985 punch from 0x020e5b to 0x02553b (0x46e1 bytes) 000199 1787784729.880058 trunc from 0x009b2e to 0x015a13 (0xbee6 bytes) 000200 1787784729.897788 punch from 0x00a0eb to 0x00cd8c (0x2ca2 bytes) 000201 1787784729.902227 punch from 0x00828b to 0x00fee1 (0x7c57 bytes) 000202 1787784729.908697 mapread 0x021e94 thru 0x02fe8d (0xdffa bytes) 000203 1787784729.948240 trunc from 0x040000 to 0x02a7a3 (0x1585c bytes) 000204 1787784729.975713 mapwrite 0x0258ee thru 0x02abbb (0x52ce bytes) 000205 1787784730.052328 fallocate 0x015c61 thru 0x020650 (0xa9f0 bytes) 000206 1787784730.057734 read 0x010743 thru 0x0194b4 (0x8d72 bytes) 000207 1787784730.071295 fallocate 0x01a8ae thru 0x01bb22 (0x1275 bytes) 000209 1787784730.079768 read 0x021cc6 thru 0x028cec (0x7027 bytes) 000211 1787784730.082304 trunc from 0x02abbc to 0x017c90 (0x12f2b bytes) 000212 1787784730.097290 trunc from 0x017c91 to 0x0359ad (0x1dd1d bytes) 000213 1787784730.120632 write 0x03bfd5 thru 0x03ffff (0x402b bytes) 000214 1787784730.122640 fallocate 0x0345c0 thru 0x03fd09 (0xb74a bytes) 000215 1787784730.127779 write 0x013741 thru 0x0174e6 (0x3da6 bytes) 000216 1787784730.129321 trunc from 0x040000 to 0x01084c (0x2f7b3 bytes) 000218 1787784730.148352 trunc from 0x01084d to 0x023ed4 (0x13688 bytes) 000220 1787784730.161316 mapread 0x00f5ce thru 0x01e386 (0xedb9 bytes) 000221 1787784730.168947 mapread 0x020248 thru 0x023ed4 (0x3c8d bytes) 000222 1787784730.169672 trunc from 0x001e3a to 0x00a0b2 (0x8279 bytes) 000223 1787784730.173311 write 0x0279e2 thru 0x034625 (0xcc44 bytes) 000225 1787784730.175089 mapwrite 0x006cc6 thru 0x011381 (0xa6bc bytes) 000226 1787784730.222168 mapwrite 0x0093c4 thru 0x00b3cc (0x2009 bytes) 000227 1787784730.239416 read 0x01d02a thru 0x023c67 (0x6c3e bytes) 000228 1787784730.243639 mapwrite 0x02aa53 thru 0x032669 (0x7c17 bytes) 000229 1787784730.262094 trunc from 0x034626 to 0x022017 (0x1260e bytes) 000230 1787784730.277788 write 0x01d57f thru 0x02445e (0x6ee0 bytes) 000231 1787784730.279321 punch from 0x008440 to 0x016af2 (0xe6b3 bytes) 000232 1787784730.285390 trunc from 0x01e2ed to 0x02445e (0x6172 bytes) 000233 1787784730.303722 trunc from 0x01fcf7 to 0x0229f0 (0x2cfa bytes) 000234 1787784730.312255 MIRROR_EXTEND to 2 mirrors 000235 1787784730.956636 write 0x03bd6a thru 0x03ffff (0x4296 bytes) 000236 1787784731.028264 trunc from 0x040000 to 0x01de93 (0x2216c bytes) 000237 1787784731.096181 mapwrite 0x01bb52 thru 0x01d76f (0x1c1e bytes) 000239 1787784731.139285 fallocate 0x014f80 thru 0x019ad7 (0x4b58 bytes) 000240 1787784731.144797 trunc from 0x003964 to 0x005115 (0x17b2 bytes) 000241 1787784731.150582 punch from 0x005717 to 0x01198b (0xc275 bytes) 000242 1787784731.157224 mapread 0x01c6ae thru 0x01de93 (0x17e6 bytes) 000243 1787784731.158699 mapread 0x003028 thru 0x012164 (0xf13d bytes) 000244 1787784731.182182 read 0x00fd23 thru 0x01b34f (0xb62d bytes) 000245 1787784731.189506 MIRROR_RESYNC 2 mirrors 000246 1787784731.862382 trunc from 0x01de94 to 0x014c77 (0x921c bytes) 000247 1787784731.982466 read 0x014413 thru 0x014c77 (0x865 bytes) 000248 1787784731.988511 trunc from 0x00e6ab to 0x0134a1 (0x4df7 bytes) 000249 1787784732.011349 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) 000250 1787784732.314318 trunc from 0x014c78 to 0x00cce1 (0x7f96 bytes) 000251 1787784732.344199 trunc from 0x002289 to 0x004fbe (0x2d36 bytes) 000252 1787784732.352264 punch from 0x0044e0 to 0x00b756 (0x7277 bytes) 000253 1787784732.362964 mapread 0x008adf thru 0x00cce1 (0x4203 bytes) 000254 1787784732.400708 mapread 0x001ce9 thru 0x00accb (0x8fe3 bytes) 000255 1787784732.405767 trunc from 0x00cce2 to 0x02cc5b (0x1ff7a bytes) 000256 1787784732.435750 mapread 0x02948a thru 0x02c2ea (0x2e61 bytes) 000257 1787784732.460552 write 0x0094c8 thru 0x0186d5 (0xf20e bytes) 000258 1787784732.474204 punch from 0x026488 to 0x02cc5b (0x67d4 bytes) 000260 1787784732.491060 mapwrite 0x031e36 thru 0x03bf77 (0xa142 bytes) 000261 1787784732.585726 mapwrite 0x035a83 thru 0x03ffff (0xa57d bytes) 000262 1787784732.683667 mapwrite 0x03306a thru 0x03bdd5 (0x8d6c bytes) 000263 1787784732.731598 punch from 0x02864f to 0x02c150 (0x3b02 bytes) 000264 1787784732.746351 trunc from 0x040000 to 0x00f4eb (0x30b14 bytes) 000265 1787784732.774897 fallocate 0x02d519 thru 0x02e90c (0x13f4 bytes) 000266 1787784732.784191 trunc from 0x02e90d to 0x03b289 (0xc97d bytes) 000267 1787784732.797633 mapwrite 0x008e1c thru 0x01495e (0xbb43 bytes) 000268 1787784732.844267 mapwrite 0x02f234 thru 0x03b72e (0xc4fb bytes) 000270 1787784732.895425 punch from 0x0395c8 to 0x03b72e (0x2167 bytes) 000271 1787784732.901358 mapread 0x03643d thru 0x036eb2 (0xa76 bytes) 000272 1787784732.904662 trunc from 0x008e58 to 0x00c614 (0x37bd bytes) 000273 1787784732.911076 fallocate 0x0062cd thru 0x00b1a5 (0x4ed9 bytes) 000274 1787784732.916149 trunc from 0x00623f to 0x015bb0 (0xf972 bytes) 000276 1787784732.923476 punch from 0x00735c to 0x00a791 (0x3436 bytes) 000277 1787784732.931090 read 0x0133cf thru 0x018909 (0x553b bytes) 000278 1787784732.936696 punch from 0x00b85b to 0x00d18c (0x1932 bytes) 000279 1787784732.942312 trunc from 0x03b72f to 0x03dc86 (0x2558 bytes) 000280 1787784732.965236 mapread 0x021fab thru 0x029688 (0x76de bytes) 000281 1787784732.977517 trunc from 0x00aac8 to 0x018318 (0xd851 bytes) 000282 1787784732.984739 read 0x0277b5 thru 0x035809 (0xe055 bytes) 000283 1787784732.985440 trunc from 0x03dc87 to 0x01b4b6 (0x227d0 bytes) 000284 1787784733.005307 punch from 0x001a34 to 0x0040c0 (0x268d bytes) 000286 1787784733.012244 read 0x00d3fe thru 0x010caf (0x38b2 bytes) 000287 1787784733.040720 MIRROR_EXTEND to 2 mirrors 000288 1787784733.922162 read 0x008247 thru 0x009e86 (0x1c40 bytes) 000289 1787784733.935869 write 0x02be72 thru 0x039b0f (0xdc9e bytes) 000290 1787784734.022509 MIRROR_RESYNC 2 mirrors 000291 1787784734.927473 mapread 0x01cf4b thru 0x0226a9 (0x575f bytes) 000292 1787784734.963601 fallocate 0x0392c3 thru 0x039a3c (0x77a bytes) 000293 1787784735.033169 read 0x0153d5 thru 0x01ae64 (0x5a90 bytes) 000295 1787784735.049767 fallocate 0x005103 thru 0x00cdcc (0x7cca bytes) 000296 1787784735.058249 trunc from 0x00c4d4 to 0x00d996 (0x14c3 bytes) 000297 1787784735.066681 trunc from 0x039b10 to 0x01416d (0x259a2 bytes) 000298 1787784735.109281 mapwrite 0x005500 thru 0x00b430 (0x5f31 bytes) 000299 1787784735.171197 fallocate 0x00d61b thru 0x01482d (0x7213 bytes) 000300 1787784735.182038 fallocate 0x026197 thru 0x02c6cd (0x6537 bytes) 000301 1787784735.195550 trunc from 0x02c6ce to 0x0179bb (0x14d12 bytes) 000302 1787784735.225428 read 0x01709b thru 0x0179bb (0x921 bytes) 000303 1787784735.236367 trunc from 0x00aa40 to 0x0179bb (0xcf7c bytes) 000305 1787784735.244576 read 0x00407b thru 0x005290 (0x1216 bytes) 000306 1787784735.245192 mapwrite 0x020453 thru 0x022483 (0x2031 bytes) 000307 1787784735.294181 mapwrite 0x0114e9 thru 0x0126b6 (0x11ce bytes) 000308 1787784735.314597 MIRROR_RESYNC 2 mirrors 000309 1787784736.273482 trunc from 0x021661 to 0x022483 (0xe23 bytes) 000310 1787784736.341402 read 0x00422d thru 0x009ead (0x5c81 bytes) 000311 1787784736.356281 trunc from 0x021848 to 0x022483 (0xc3c bytes) 000312 1787784736.360892 trunc from 0x010794 to 0x014337 (0x3ba4 bytes) 000313 1787784736.366880 trunc from 0x022484 to 0x03fa2a (0x1d5a7 bytes) 000314 1787784736.413882 write 0x00acc9 thru 0x01a2e9 (0xf621 bytes) 000315 1787784736.423646 mapread 0x0126a5 thru 0x016fe2 (0x493e bytes) 000316 1787784736.429362 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) 000319 1787784736.618649 trunc from 0x03fa2b to 0x018251 (0x277d9 bytes) 000320 1787784736.653379 write 0x033351 thru 0x03d2fd (0x9fad bytes) 000321 1787784736.655474 trunc from 0x03d2fe to 0x034aa6 (0x8857 bytes) 000322 1787784736.677626 punch from 0x027af2 to 0x034aa6 (0xcfb5 bytes) 000323 1787784736.691061 mapread 0x013a2c thru 0x0182a7 (0x487c bytes) 000324 1787784736.719331 punch from 0x027366 to 0x034aa6 (0xd741 bytes) 000326 1787784736.725474 mapread 0x010474 thru 0x020378 (0xff05 bytes) 000327 1787784736.737376 fallocate 0x037624 thru 0x03ffff (0x89dc bytes) 000328 1787784736.743148 write 0x013852 thru 0x01ebd7 (0xb386 bytes) 000330 1787784736.746649 mapread 0x025875 thru 0x02b568 (0x5cf4 bytes) 000331 1787784736.761341 trunc from 0x034aa7 to 0x00f0c6 (0x259e0 bytes) 000332 1787784736.785189 mapread 0x0059b1 thru 0x00f0c6 (0x9716 bytes) 000333 1787784736.799597 punch from 0x005f6a to 0x00f0c6 (0x915d bytes) 000334 1787784736.805198 mapwrite 0x026a5e thru 0x029acb (0x306e bytes) 000335 1787784736.853613 MIRROR_EXTEND to 2 mirrors 000336 1787784737.715497 mapwrite 0x008109 thru 0x013c43 (0xbb3b bytes) 000337 1787784737.895540 mapwrite 0x005f25 thru 0x0068de (0x9ba bytes) 000338 1787784737.942079 trunc from 0x024721 to 0x028672 (0x3f52 bytes) 000339 1787784737.977374 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) 000340 1787784738.184269 fallocate 0x01891f thru 0x019910 (0xff2 bytes) 000341 1787784738.195151 fallocate 0x02d48a thru 0x03b96e (0xe4e5 bytes) 000342 1787784738.204096 trunc from 0x00bfee to 0x0106af (0x46c2 bytes) 000343 1787784738.209435 write 0x03d4d6 thru 0x03ffff (0x2b2a bytes) 000344 1787784738.221192 write 0x00aa2d thru 0x019da8 (0xf37c bytes) 000345 1787784738.240655 write 0x02b235 thru 0x02e8db (0x36a7 bytes) 000346 1787784738.252247 mapread 0x027b04 thru 0x0344b0 (0xc9ad bytes) 000347 1787784738.281392 trunc from 0x02a500 to 0x02f9df (0x54e0 bytes) 000348 1787784738.296760 mapread 0x010591 thru 0x01b748 (0xb1b8 bytes) 000349 1787784738.313075 read 0x00115f thru 0x00647b (0x531d bytes) 000350 1787784738.322343 trunc from 0x0154ea to 0x01dca6 (0x87bd bytes) 000351 1787784738.328784 trunc from 0x0344a8 to 0x03fa9e (0xb5f7 bytes) 000355 1787784738.339801 fallocate 0x02eec1 thru 0x0368cb (0x7a0b bytes) 000356 1787784738.344943 mapread 0x004967 thru 0x011949 (0xcfe3 bytes) 000359 1787784738.366992 write 0x034721 thru 0x035a13 (0x12f3 bytes) 000360 1787784738.394627 write 0x037b0e thru 0x03ffff (0x84f2 bytes) 000361 1787784738.406632 write 0x026476 thru 0x032ba7 (0xc732 bytes) 000362 1787784738.408061 trunc from 0x002aa0 to 0x008161 (0x56c2 bytes) 000365 1787784738.428782 trunc from 0x040000 to 0x015a61 (0x2a59e bytes) 000366 1787784738.483938 trunc from 0x00a7c1 to 0x00cdf3 (0x2633 bytes) 000367 1787784738.490910 mapwrite 0x038436 thru 0x03ffff (0x7bca bytes) 000369 1787784738.584097 write 0x03c440 thru 0x03ffff (0x3bc0 bytes) 000371 1787784738.585619 trunc from 0x01d387 to 0x029ae8 (0xc762 bytes) 000372 1787784738.623364 trunc from 0x040000 to 0x000e81 (0x3f17e bytes) 000373 1787784738.662133 fallocate 0x00d5b5 thru 0x018515 (0xaf61 bytes) 000374 1787784738.673078 read 0x00a985 thru 0x0148fb (0x9f77 bytes) 000375 1787784738.708370 write 0x028c7e thru 0x02bb47 (0x2eca bytes) 000376 1787784738.712344 write 0x033455 thru 0x038725 (0x52d1 bytes) 000377 1787784738.719550 read 0x0288c1 thru 0x030c83 (0x83c3 bytes) 000378 1787784738.746768 fallocate 0x0261c6 thru 0x02bc40 (0x5a7b bytes) 000379 1787784738.765621 fallocate 0x033b89 thru 0x039849 (0x5cc1 bytes) 000380 1787784738.772227 mapwrite 0x01f557 thru 0x020de3 (0x188d bytes) 000381 1787784738.813368 mapwrite 0x002345 thru 0x00b742 (0x93fe bytes) 000383 1787784738.884023 MIRROR_EXTEND to 2 mirrors 000384 1787784740.145064 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) 000385 1787784740.282495 read 0x02d3e3 thru 0x02fb97 (0x27b5 bytes) 000386 1787784740.288099 write 0x029989 thru 0x036973 (0xcfeb bytes) 000387 1787784740.310637 trunc from 0x03984a to 0x02bc6b (0xdbde bytes) 000388 1787784740.329355 trunc from 0x001383 to 0x00d690 (0xc30e bytes) 000390 1787784740.335339 read 0x013978 thru 0x01b615 (0x7c9e bytes) 000391 1787784740.349130 mapread 0x011c56 thru 0x017c47 (0x5ff2 bytes) 000392 1787784740.350157 trunc from 0x0214b1 to 0x02bc6b (0xa7bb bytes) 000393 1787784740.360852 trunc from 0x02bc6c to 0x037556 (0xb8eb bytes) 000394 1787784740.384674 trunc from 0x037557 to 0x030834 (0x6d22 bytes) 000395 1787784740.404747 MIRROR_EXTEND to 2 mirrors 000396 1787784740.996347 MIRROR_EXTEND to 3 mirrors 000397 1787784742.040650 punch from 0x012c17 to 0x015923 (0x2d0d bytes) 000398 1787784742.088907 trunc from 0x030835 to 0x030745 (0xef bytes) 000399 1787784742.121156 punch from 0x017a61 to 0x0192fa (0x189a bytes) 000400 1787784742.134699 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30746): Mirror 2: 0x7ff9d2cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30746): Mirror 2: 0x7ff9d2cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30746: skipped 000401 1787784742.559735 fallocate 0x02f810 thru 0x03914e (0x993f bytes) 000402 1787784742.572633 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30746): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30746): Mirror 2: 0x7ff9d2cd (stale) CRC-32 checksum value for chunk [0, 0x30746): Mirror 1: 0x7ff9d2cd Mirror 3: 0x7ff9d2cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30746: skipped 000403 1787784743.532071 mapwrite 0x02b049 thru 0x02fbf1 (0x4ba9 bytes) 000404 1787784743.665214 MIRROR_RESYNC 3 mirrors 000405 1787784745.010694 mapread 0x00b755 thru 0x00db1c (0x23c8 bytes) 000407 1787784745.071044 MIRROR_EXTEND to 4 mirrors 000408 1787784746.281933 write 0x025b8b thru 0x02f991 (0x9e07 bytes) 000409 1787784746.345767 trunc from 0x00e05c to 0x01cf78 (0xef1d bytes) 000411 1787784746.352656 mapread 0x00ceca thru 0x01acb7 (0xddee bytes) 000412 1787784746.391783 read 0x0094f9 thru 0x0148e4 (0xb3ec bytes) 000413 1787784746.392737 mapread 0x00ce0b thru 0x01615f (0x9355 bytes) 000414 1787784746.394992 mapread 0x0223a6 thru 0x028295 (0x5ef0 bytes) 000415 1787784746.406188 mapread 0x021ffc thru 0x02c8ce (0xa8d3 bytes) 000416 1787784746.408590 fallocate 0x007866 thru 0x00c8c8 (0x5063 bytes) 000417 1787784746.414143 fallocate 0x01f2fb thru 0x021cc8 (0x29ce bytes) 000418 1787784746.420671 fallocate 0x03debe thru 0x03ffff (0x2142 bytes) 000419 1787784746.427204 mapread 0x00c4dd thru 0x01751e (0xb042 bytes) 000420 1787784746.429587 trunc from 0x03333b to 0x037dc1 (0x4a87 bytes) 000422 1787784746.436683 trunc from 0x040000 to 0x029989 (0x16676 bytes) 000423 1787784746.499235 mapread 0x00b48c thru 0x01a643 (0xf1b8 bytes) 000424 1787784746.501073 punch from 0x014ad4 to 0x02310c (0xe639 bytes) 000426 1787784746.509949 mapread 0x00c3f0 thru 0x01c14c (0xfd5d bytes) 000427 1787784746.523574 write 0x0118ca thru 0x01ffc6 (0xe6fd bytes) 000428 1787784746.531474 read 0x01225b thru 0x01aa38 (0x87de bytes) 000429 1787784746.532207 mapwrite 0x00dc80 thru 0x016661 (0x89e2 bytes) 000431 1787784746.580550 read 0x01e4b0 thru 0x0226b7 (0x4208 bytes) 000433 1787784746.594196 fallocate 0x031c4b thru 0x03f2b1 (0xd667 bytes) 000434 1787784746.602254 mapwrite 0x008c3e thru 0x016a4e (0xde11 bytes) 000435 1787784746.636952 trunc from 0x02998a to 0x038f0e (0xf585 bytes) 000436 1787784746.657657 mapread 0x032d17 thru 0x038f0e (0x61f8 bytes) 000437 1787784746.672175 mapwrite 0x00a3f4 thru 0x00ceb1 (0x2abe bytes) 000438 1787784746.687186 trunc from 0x029f93 to 0x02bf24 (0x1f92 bytes) 000439 1787784746.695652 trunc from 0x019d87 to 0x01aed2 (0x114c bytes) 000440 1787784746.706199 fallocate 0x037b97 thru 0x03bbe4 (0x404e bytes) 000441 1787784746.712986 mapwrite 0x0207e2 thru 0x0242f2 (0x3b11 bytes) 000442 1787784746.736770 mapread 0x001302 thru 0x00e2dc (0xcfdb bytes) 000443 1787784746.744595 mapwrite 0x02d91d thru 0x036358 (0x8a3c bytes) 000444 1787784746.783752 fallocate 0x03c9a2 thru 0x03d166 (0x7c5 bytes) 000445 1787784746.789235 mapwrite 0x01ff9d thru 0x0214b0 (0x1514 bytes) 000446 1787784746.803152 read 0x016652 thru 0x023637 (0xcfe6 bytes) 000447 1787784746.813793 mapread 0x03ccae thru 0x03d166 (0x4b9 bytes) 000448 1787784746.814716 punch from 0x018bd2 to 0x0194bb (0x8ea bytes) 000450 1787784746.822320 mapwrite 0x03524e thru 0x03b7e0 (0x6593 bytes) 000451 1787784746.846177 punch from 0x011368 to 0x01309b (0x1d34 bytes) 000452 1787784746.854044 punch from 0x00a0e6 to 0x013e4d (0x9d68 bytes) 000453 1787784746.860791 punch from 0x014dc5 to 0x01ca95 (0x7cd1 bytes) 000454 1787784746.868302 trunc from 0x03d167 to 0x01a431 (0x22d35 bytes) 000455 1787784746.902487 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a432): Mirror 2: 0x36359713 (stale) Mirror 3: 0x36359713 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a432): Mirror 2: 0x36359713 (stale) Mirror 3: 0x36359713 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a432: skipped 000456 1787784747.376605 read 0x00d59e thru 0x0171eb (0x9c4e bytes) 000457 1787784747.388195 trunc from 0x0014c2 to 0x00a270 (0x8daf bytes) 000458 1787784747.394068 fallocate 0x00bed9 thru 0x0128d3 (0x69fb bytes) 000459 1787784747.398768 punch from 0x00cddc to 0x017102 (0xa327 bytes) 000460 1787784747.404204 mapread 0x00ded5 thru 0x0109b3 (0x2adf bytes) 000462 1787784747.437595 write 0x02d2c8 thru 0x032df1 (0x5b2a bytes) 000463 1787784747.450327 fallocate 0x02969d thru 0x034441 (0xada5 bytes) 000464 1787784747.457191 read 0x030ffe thru 0x032df1 (0x1df4 bytes) 000465 1787784747.458870 mapwrite 0x01dcb9 thru 0x026096 (0x83de bytes) 000466 1787784747.506285 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32df2): Mirror 3: 0x48669188 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32df2): Mirror 3: 0x48669188 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32df2: skipped 000468 1787784747.867230 write 0x007911 thru 0x00cff8 (0x56e8 bytes) 000469 1787784747.902783 write 0x005483 thru 0x00ee86 (0x9a04 bytes) 000470 1787784747.926981 read 0x0313af thru 0x032df1 (0x1a43 bytes) 000471 1787784747.937977 write 0x01a10a thru 0x0222b1 (0x81a8 bytes) 000473 1787784747.956627 MIRROR_RESYNC 2 mirrors 000474 1787784748.886941 trunc from 0x02a248 to 0x02f202 (0x4fbb bytes) 000475 1787784748.924929 punch from 0x0109ef to 0x01657a (0x5b8c bytes) 000476 1787784748.931335 mapwrite 0x003ea6 thru 0x00f37d (0xb4d8 bytes) 000477 1787784748.972550 read 0x0145bf thru 0x01ef97 (0xa9d9 bytes) 000478 1787784748.981571 mapread 0x01f7c8 thru 0x02dc8f (0xe4c8 bytes) 000479 1787784748.997413 trunc from 0x02b5f4 to 0x032df1 (0x77fe bytes) 000480 1787784749.002550 read 0x02a529 thru 0x02bdfa (0x18d2 bytes) 000481 1787784749.012650 write 0x010dbc thru 0x018f90 (0x81d5 bytes) 000482 1787784749.014192 write 0x031f8b thru 0x03407e (0x20f4 bytes) 000483 1787784749.022160 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) 000484 1787784749.311591 write 0x0020f6 thru 0x011261 (0xf16c bytes) 000485 1787784749.334061 trunc from 0x027dfc to 0x02ba26 (0x3c2b bytes) 000486 1787784749.340236 trunc from 0x00d9fb to 0x019599 (0xbb9f bytes) 000487 1787784749.357332 trunc from 0x03407f to 0x03892c (0x48ae bytes) 000488 1787784749.377219 punch from 0x02faaf to 0x036a47 (0x6f99 bytes) 000489 1787784749.385397 read 0x01fc1a thru 0x02d710 (0xdaf7 bytes) 000490 1787784749.396302 trunc from 0x03892d to 0x01d448 (0x1b4e4 bytes) 000491 1787784749.413302 write 0x001c2f thru 0x00f2f4 (0xd6c6 bytes) 000492 1787784749.429313 punch from 0x018d6d to 0x01d448 (0x46dc bytes) 000493 1787784749.435530 write 0x022642 thru 0x02594e (0x330d bytes) 000495 1787784749.437134 fallocate 0x03e8d4 thru 0x03ffff (0x172c bytes) 000496 1787784749.443448 trunc from 0x02594f to 0x030ed4 (0xb586 bytes) 000497 1787784749.464164 fallocate 0x037433 thru 0x03dc8c (0x685a bytes) 000498 1787784749.475913 MIRROR_EXTEND to 2 mirrors 000499 1787784750.143776 punch from 0x01958a to 0x01f8e5 (0x635c bytes) 000501 1787784750.187543 mapread 0x02ba45 thru 0x0301a8 (0x4764 bytes) 000502 1787784750.209661 trunc from 0x027f33 to 0x037020 (0xf0ee bytes) 000503 1787784750.215099 trunc from 0x03dc8d to 0x007bbe (0x360ce bytes) 000504 1787784750.272939 mapread 0x00624a thru 0x007536 (0x12ed bytes) 000505 1787784750.283905 punch from 0x003341 to 0x007bbe (0x487e bytes) 000506 1787784750.293229 MIRROR_RESYNC 2 mirrors 000507 1787784750.890634 mapread 0x001db8 thru 0x002f51 (0x119a bytes) 000508 1787784750.902445 read 0x001cef thru 0x00412f (0x2441 bytes) 000509 1787784750.916337 MIRROR_RESYNC 2 mirrors 000510 1787784751.329115 trunc from 0x001615 to 0x006c60 (0x564c bytes) 000511 1787784751.394585 mapwrite 0x030d8b thru 0x036c70 (0x5ee6 bytes) 000512 1787784751.502120 read 0x02ea3a thru 0x036c70 (0x8237 bytes) 000513 1787784751.503131 trunc from 0x011571 to 0x01c36d (0xadfd bytes) 000514 1787784751.514199 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36c71): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36c71): Mirror 2: 0xd840c3e2 (stale) CRC-32 checksum value for chunk [0, 0x36c71): Mirror 1: 0x1a18a366 Mirror 3: 0x1a18a366 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36c71: skipped 000515 1787784752.459333 trunc from 0x032025 to 0x03459e (0x257a bytes) 000516 1787784752.534106 mapread 0x003596 thru 0x00c269 (0x8cd4 bytes) 000517 1787784752.552510 mapread 0x01f558 thru 0x021998 (0x2441 bytes) 000519 1787784752.561494 write 0x020445 thru 0x020e87 (0xa43 bytes) 000520 1787784752.566132 read 0x02f60c thru 0x036c70 (0x7665 bytes) 000522 1787784752.575212 trunc from 0x01edcd to 0x02af9f (0xc1d3 bytes) 000523 1787784752.585965 mapwrite 0x00cb68 thru 0x013169 (0x6602 bytes) 000524 1787784752.615669 mapread 0x02c245 thru 0x02f8a3 (0x365f bytes) 000525 1787784752.643206 read 0x02b214 thru 0x036c70 (0xba5d bytes) 000526 1787784752.647522 fallocate 0x01ec26 thru 0x02a40c (0xb7e7 bytes) 000527 1787784752.671305 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x36c71): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36c71): Mirror 2: 0xd840c3e2 (stale) Mirror 3: 0x1a18a366 (stale) CRC-32 checksum value for chunk [0, 0x36c71): Mirror 1: 0xf2cf46e6 Mirror 4: 0xf2cf46e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36c71: skipped 000528 1787784753.941768 trunc from 0x013278 to 0x014064 (0xded bytes) 000530 1787784754.029521 read 0x01d280 thru 0x02bc0a (0xe98b bytes) 000531 1787784754.042716 mapwrite 0x01a63f thru 0x0295ac (0xef6e bytes) 000532 1787784754.095956 mapread 0x001383 thru 0x004cc2 (0x3940 bytes) 000534 1787784754.108513 fallocate 0x011491 thru 0x01249c (0x100c bytes) 000535 1787784754.123777 write 0x014d55 thru 0x01832d (0x35d9 bytes) 000536 1787784754.138298 trunc from 0x014dd1 to 0x021912 (0xcb42 bytes) 000537 1787784754.162202 fallocate 0x0139f5 thru 0x016545 (0x2b51 bytes) 000538 1787784754.167572 trunc from 0x036c71 to 0x01d445 (0x1982b bytes) 000539 1787784754.206137 punch from 0x0181ac to 0x01d445 (0x529a bytes) 000540 1787784754.218023 read 0x008155 thru 0x010439 (0x82e5 bytes) 000542 1787784754.240608 mapread 0x015c28 thru 0x01728e (0x1667 bytes) 000543 1787784754.249124 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1d446): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d446): Mirror 2: 0xb1dbd53e (stale) Mirror 3: 0x58de7a64 (stale) Mirror 4: 0x28313945 (stale) CRC-32 checksum value for chunk [0, 0x1d446): Mirror 1: 0x43a47671 Mirror 5: 0x43a47671 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d446: skipped 000544 1787784755.387583 trunc from 0x008274 to 0x016873 (0xe600 bytes) 000545 1787784755.456783 punch from 0x00a467 to 0x01664b (0xc1e5 bytes) 000547 1787784755.465556 read 0x0057db thru 0x010228 (0xaa4e bytes) 000548 1787784755.479915 trunc from 0x01d446 to 0x03a949 (0x1d504 bytes) 000549 1787784755.534087 punch from 0x01aa63 to 0x01dc83 (0x3221 bytes) 000550 1787784755.542789 mapwrite 0x00468f thru 0x007562 (0x2ed4 bytes) 000551 1787784755.564850 fallocate 0x03f661 thru 0x03ffff (0x99f bytes) 000552 1787784755.570503 punch from 0x005f1c to 0x00a083 (0x4168 bytes) 000553 1787784755.575469 trunc from 0x03a94a to 0x01d689 (0x1d2c0 bytes) 000554 1787784755.590266 write 0x01a7c7 thru 0x01ef4f (0x4789 bytes) 000555 1787784755.596739 mapwrite 0x021b2a thru 0x029495 (0x796c bytes) 000556 1787784755.654839 write 0x009095 thru 0x0162ff (0xd26b bytes) 000557 1787784755.672970 write 0x018c17 thru 0x01f2ad (0x6697 bytes) 000558 1787784755.679044 write 0x013fd7 thru 0x015854 (0x187e bytes) 000559 1787784755.680322 trunc from 0x013a0f to 0x01a378 (0x696a bytes) 000560 1787784755.698255 punch from 0x0129e5 to 0x01c01c (0x9638 bytes) 000561 1787784755.704457 trunc from 0x029496 to 0x02dd54 (0x48bf bytes) 000562 1787784755.720772 read 0x00e645 thru 0x01aace (0xc48a bytes) 000563 1787784755.732144 mapwrite 0x02ff33 thru 0x03f38f (0xf45d bytes) 000564 1787784755.786179 MIRROR_RESYNC 5 mirrors 000566 1787784757.732458 punch from 0x035bd1 to 0x03cf69 (0x7399 bytes) 000567 1787784757.773106 punch from 0x03b210 to 0x03f38f (0x4180 bytes) 000568 1787784757.779522 read 0x007c68 thru 0x012e00 (0xb199 bytes) 000569 1787784757.810151 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f390): Mirror 2: 0x433d2694 (stale) Mirror 4: 0x433d2694 (stale) Mirror 5: 0x433d2694 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f390): Mirror 2: 0x433d2694 (stale) Mirror 4: 0x433d2694 (stale) Mirror 5: 0x433d2694 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f390: skipped 000571 1787784758.208409 fallocate 0x03928d thru 0x03ffff (0x6d73 bytes) 000572 1787784758.219368 MIRROR_RESYNC 4 mirrors 000574 1787784759.852602 trunc from 0x00b902 to 0x00bca3 (0x3a2 bytes) 000575 1787784759.896697 read 0x02dbf5 thru 0x03093d (0x2d49 bytes) 000576 1787784759.905958 fallocate 0x00bf88 thru 0x01bd4d (0xfdc6 bytes) 000577 1787784759.915754 write 0x00ca61 thru 0x0122ca (0x586a bytes) 000578 1787784759.927195 read 0x003454 thru 0x00cbef (0x979c bytes) 000579 1787784759.944097 MIRROR_RESYNC 4 mirrors 000580 1787784761.713887 mapwrite 0x03135c thru 0x03a344 (0x8fe9 bytes) 000581 1787784761.877019 punch from 0x00cb49 to 0x017cc0 (0xb178 bytes) 000582 1787784761.889448 fallocate 0x031cd8 thru 0x03a98b (0x8cb4 bytes) 000583 1787784761.897363 fallocate 0x026520 thru 0x032f2e (0xca0f bytes) 000584 1787784761.918220 trunc from 0x03f390 to 0x006b9d (0x387f2 bytes) 000585 1787784761.955394 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x6b9e): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6b9e): Mirror 2: 0x3090e97c (stale) Mirror 4: 0x3090e97c (stale) Mirror 5: 0x3090e97c (stale) CRC-32 checksum value for chunk [0, 0x6b9e): Mirror 1: 0x3090e97c Mirror 6: 0x3090e97c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6b9e: skipped 000586 1787784763.469721 write 0x017436 thru 0x01bd16 (0x48e1 bytes) 000587 1787784763.529915 write 0x02bafc thru 0x0381c6 (0xc6cb bytes) 000588 1787784763.555108 MIRROR_RESYNC 5 mirrors 000589 1787784765.466160 mapread 0x01efd9 thru 0x02378d (0x47b5 bytes) 000590 1787784765.506421 write 0x012ba8 thru 0x022264 (0xf6bd bytes) 000591 1787784765.610790 fallocate 0x025f7f thru 0x02af6f (0x4ff1 bytes) 000592 1787784765.628275 trunc from 0x02d6c5 to 0x0381c6 (0xab02 bytes) 000593 1787784765.646060 read 0x00abf8 thru 0x0100ac (0x54b5 bytes) 000594 1787784765.669376 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x381c7): Mirror 2: 0xc566dcea (stale) Mirror 4: 0xc566dcea (stale) Mirror 5: 0xc566dcea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x381c7): Mirror 2: 0xc566dcea (stale) Mirror 4: 0xc566dcea (stale) Mirror 5: 0xc566dcea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x381c7: skipped 000595 1787784766.360507 trunc from 0x01a968 to 0x01c7ab (0x1e44 bytes) 000596 1787784766.368044 trunc from 0x0105dd to 0x01ae0b (0xa82f bytes) 000597 1787784766.379757 trunc from 0x0381c7 to 0x034fad (0x3219 bytes) 000598 1787784766.424766 read 0x03189c thru 0x034fad (0x3712 bytes) 000599 1787784766.446343 read 0x0316ee thru 0x034fad (0x38c0 bytes) 000600 1787784766.447019 mapread 0x001e2a thru 0x00d11b (0xb2f2 bytes) 000601 1787784766.499908 trunc from 0x034fae to 0x00325e (0x31d4f bytes) 000602 1787784766.543872 trunc from 0x00325f to 0x035e5d (0x32bff bytes) 000603 1787784766.569032 trunc from 0x022233 to 0x0267ba (0x4588 bytes) 000604 1787784766.581379 trunc from 0x029902 to 0x035130 (0xb82f bytes) 000605 1787784766.591960 write 0x01c639 thru 0x01f892 (0x325a bytes) 000606 1787784766.617508 trunc from 0x035e5e to 0x005f38 (0x2ff25 bytes) 000607 1787784766.652116 trunc from 0x0008db to 0x005f38 (0x565e bytes) 000608 1787784766.662633 write 0x032ac1 thru 0x03866d (0x5bad bytes) 000609 1787784766.666233 trunc from 0x02189d to 0x022ec0 (0x1624 bytes) 000610 1787784766.682594 mapread 0x02cbdc thru 0x03238f (0x57b4 bytes) 000611 1787784766.760129 read 0x03151d thru 0x034af9 (0x35dd bytes) 000612 1787784766.778319 MIRROR_RESYNC 4 mirrors 000613 1787784767.604050 trunc from 0x03866e to 0x0391fe (0xb91 bytes) 000614 1787784767.663688 write 0x027bd3 thru 0x03279e (0xabcc bytes) 000615 1787784767.677063 trunc from 0x0391ff to 0x03433b (0x4ec3 bytes) 000616 1787784767.719413 punch from 0x01d601 to 0x02be4c (0xe84c bytes) 000617 1787784767.734011 fallocate 0x017918 thru 0x0262fa (0xe9e3 bytes) 000618 1787784767.738964 write 0x037789 thru 0x03841d (0xc95 bytes) 000619 1787784767.740420 punch from 0x0136fe to 0x018171 (0x4a74 bytes) 000620 1787784767.745798 read 0x0329d0 thru 0x03841d (0x5a4e bytes) 000622 1787784767.752703 fallocate 0x02abab thru 0x034c9b (0xa0f1 bytes) 000624 1787784767.758708 trunc from 0x03841e to 0x02ed49 (0x96d4 bytes) 000625 1787784767.782317 trunc from 0x019a40 to 0x01a7b8 (0xd79 bytes) 000627 1787784767.787412 fallocate 0x02f38e thru 0x03d2b1 (0xdf24 bytes) 000628 1787784767.792339 mapread 0x0273ec thru 0x02d886 (0x649b bytes) 000629 1787784767.812399 write 0x000f13 thru 0x0047d9 (0x38c7 bytes) 000630 1787784767.836305 trunc from 0x02ed4a to 0x03a4c9 (0xb780 bytes) 000631 1787784767.862092 punch from 0x00ef7e to 0x01bd64 (0xcde7 bytes) 000632 1787784767.868728 punch from 0x003b03 to 0x01156e (0xda6c bytes) 000633 1787784767.887840 mapread 0x0138a3 thru 0x0189b1 (0x510f bytes) 000634 1787784767.899952 fallocate 0x034f59 thru 0x03d9c4 (0x8a6c bytes) 000635 1787784767.908096 trunc from 0x03d9c5 to 0x034f48 (0x8a7c bytes) 000636 1787784767.930602 punch from 0x004cb4 to 0x00fc3b (0xaf88 bytes) 000637 1787784767.939126 mapread 0x00d540 thru 0x01779c (0xa25d bytes) 000638 1787784767.956398 trunc from 0x034f49 to 0x01e7b2 (0x16796 bytes) 000639 1787784767.983600 mapwrite 0x00af3f thru 0x00f65c (0x471e bytes) 000640 1787784768.005178 mapwrite 0x037f4f thru 0x03ffff (0x80b1 bytes) 000641 1787784768.062632 mapwrite 0x03506a thru 0x038bee (0x3b85 bytes) 000642 1787784768.077200 trunc from 0x040000 to 0x005906 (0x3a6f9 bytes) 000643 1787784768.096180 fallocate 0x012549 thru 0x019605 (0x70bd bytes) 000644 1787784768.102105 trunc from 0x001cdd to 0x002179 (0x49d bytes) 000645 1787784768.107528 mapwrite 0x00e9a2 thru 0x019b5f (0xb1be bytes) 000646 1787784768.163301 trunc from 0x005c72 to 0x00e8a0 (0x8c2f bytes) 000647 1787784768.170472 trunc from 0x00befc to 0x014a47 (0x8b4c bytes) 000648 1787784768.176014 punch from 0x0117fa to 0x018b81 (0x7388 bytes) 000649 1787784768.181661 fallocate 0x005093 thru 0x012aab (0xda19 bytes) 000651 1787784768.191322 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19b60): Mirror 2: 0x1700dcb2 (stale) Mirror 5: 0x1700dcb2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19b60): Mirror 2: 0x1700dcb2 (stale) Mirror 5: 0x1700dcb2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19b60: skipped 000652 1787784768.572709 mapread 0x0059e3 thru 0x009b26 (0x4144 bytes) 000653 1787784768.592981 punch from 0x019aab to 0x019b5f (0xb5 bytes) 000654 1787784768.610918 trunc from 0x00c4a7 to 0x0176d2 (0xb22c bytes) 000655 1787784768.628334 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x19b60): Mirror 2: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19b60): Mirror 2: 0x1700dcb2 (stale) Mirror 5: 0x1700dcb2 (stale) CRC-32 checksum value for chunk [0, 0x19b60): Mirror 1: 0x42942a97 Mirror 6: 0x42942a97 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19b60: skipped 000656 1787784770.004722 read 0x012ee3 thru 0x019b5f (0x6c7d bytes) 000657 1787784770.029728 read 0x010bea thru 0x019b5f (0x8f76 bytes) 000658 1787784770.059614 trunc from 0x00f2cd to 0x015ecc (0x6c00 bytes) 000659 1787784770.136789 write 0x0217ed thru 0x02da65 (0xc279 bytes) 000660 1787784770.150752 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2da66): Mirror 2: 0xaa799940 (stale) Mirror 6: 0x1a961fa3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2da66): Mirror 2: 0xaa799940 (stale) Mirror 6: 0x1a961fa3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2da66: skipped 000661 1787784770.710001 fallocate 0x01c6c3 thru 0x026517 (0x9e55 bytes) 000662 1787784770.720508 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2da66): Mirror 2: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2da66): Mirror 2: 0xaa799940 (stale) Mirror 6: 0x1a961fa3 (stale) CRC-32 checksum value for chunk [0, 0x2da66): Mirror 1: 0x9c716bec Mirror 7: 0x9c716bec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2da66: skipped 000663 1787784772.330353 fallocate 0x008ea1 thru 0x00b498 (0x25f8 bytes) 000664 1787784772.443084 trunc from 0x02da66 to 0x013e1b (0x19c4a bytes) 000665 1787784772.493722 fallocate 0x015079 thru 0x022218 (0xd1a0 bytes) 000666 1787784772.506008 read 0x00a30e thru 0x00c0d7 (0x1dca bytes) 000667 1787784772.515149 fallocate 0x01a4d7 thru 0x0240d7 (0x9c01 bytes) 000668 1787784772.521358 write 0x00682e thru 0x00deac (0x767f bytes) 000669 1787784772.533991 read 0x00430a thru 0x013407 (0xf0fe bytes) 000670 1787784772.547431 read 0x0089cf thru 0x013e1b (0xb44d bytes) 000671 1787784772.549688 write 0x03af0f thru 0x03ffff (0x50f1 bytes) 000672 1787784772.552363 trunc from 0x02a21f to 0x0372cf (0xd0b1 bytes) 000673 1787784772.559081 fallocate 0x005e66 thru 0x00bc76 (0x5e11 bytes) 000674 1787784772.567749 trunc from 0x0374c1 to 0x03ffff (0x8b3f bytes) 000675 1787784772.583048 write 0x01f450 thru 0x020bbd (0x176e bytes) 000676 1787784772.595087 fallocate 0x038ed0 thru 0x03d89b (0x49cc bytes) 000677 1787784772.602413 trunc from 0x040000 to 0x036223 (0x9ddc bytes) 000678 1787784772.625020 punch from 0x022170 to 0x02e46b (0xc2fc bytes) 000679 1787784772.632401 fallocate 0x00277c thru 0x00cab9 (0xa33e bytes) 000680 1787784772.639937 read 0x035b10 thru 0x036223 (0x714 bytes) 000681 1787784772.642052 fallocate 0x00d03b thru 0x019388 (0xc34e bytes) 000682 1787784772.648078 punch from 0x00a5dc to 0x010893 (0x62b8 bytes) 000684 1787784772.656398 punch from 0x01995c to 0x025ed9 (0xc57e bytes) 000685 1787784772.670305 write 0x004770 thru 0x00c1c5 (0x7a56 bytes) 000686 1787784772.680079 mapwrite 0x02a736 thru 0x037130 (0xc9fb bytes) 000689 1787784772.758709 write 0x02cc75 thru 0x031e0c (0x5198 bytes) 000691 1787784772.762866 punch from 0x0232a1 to 0x02938b (0x60eb bytes) 000693 1787784772.778052 trunc from 0x01bf0d to 0x024ce4 (0x8dd8 bytes) 000694 1787784772.784154 punch from 0x003407 to 0x003bba (0x7b4 bytes) 000695 1787784772.790262 mapread 0x0187da thru 0x027faa (0xf7d1 bytes) 000696 1787784772.861930 mapread 0x003bd0 thru 0x0092b6 (0x56e7 bytes) 000697 1787784772.868661 trunc from 0x01effb to 0x025af2 (0x6af8 bytes) 000698 1787784772.875020 trunc from 0x02e28c to 0x037130 (0x8ea5 bytes) 000699 1787784772.890803 write 0x0326ab thru 0x03ffff (0xd955 bytes) 000700 1787784772.898554 read 0x00efa6 thru 0x012c85 (0x3ce0 bytes) 000701 1787784772.904246 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe2cbde2f (stale) Mirror 7: 0x33390fb1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xe2cbde2f (stale) Mirror 7: 0x33390fb1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000702 1787784773.285100 punch from 0x00da52 to 0x01646f (0x8a1e bytes) 000703 1787784773.292405 mapread 0x009e46 thru 0x00b56f (0x172a bytes) 000704 1787784773.313869 write 0x035855 thru 0x039fb6 (0x4762 bytes) 000705 1787784773.332070 trunc from 0x035be4 to 0x03ffff (0xa41c bytes) 000706 1787784773.345245 trunc from 0x040000 to 0x032df6 (0xd209 bytes) 000707 1787784773.360282 write 0x02dcd3 thru 0x02dfd7 (0x305 bytes) 000708 1787784773.372412 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32df7): Mirror 6: 0x9e9b369e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32df7): Mirror 6: 0x9e9b369e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32df7: skipped 000709 1787784773.644085 punch from 0x0054ff to 0x009b09 (0x460b bytes) 000710 1787784773.650352 mapwrite 0x008408 thru 0x00bf18 (0x3b11 bytes) 000711 1787784773.692077 punch from 0x026fc8 to 0x02facb (0x8b04 bytes) 000712 1787784773.697525 mapread 0x0145a8 thru 0x018093 (0x3aec bytes) 000713 1787784773.706334 trunc from 0x00f527 to 0x015ce8 (0x67c2 bytes) 000714 1787784773.713003 write 0x018e18 thru 0x025409 (0xc5f2 bytes) 000715 1787784773.725172 mapread 0x011edf thru 0x01b3d7 (0x94f9 bytes) 000716 1787784773.747321 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) 000718 1787784773.903002 trunc from 0x032df7 to 0x0052a7 (0x2db4f bytes) 000719 1787784773.930687 write 0x020f39 thru 0x02d205 (0xc2cd bytes) 000720 1787784773.933522 punch from 0x011ca8 to 0x01b4a3 (0x97fc bytes) 000721 1787784773.939682 trunc from 0x01f7ab to 0x0281c7 (0x8a1d bytes) 000722 1787784773.959564 mapread 0x006d8e thru 0x011d58 (0xafcb bytes) 000723 1787784773.996837 trunc from 0x000e1e to 0x003131 (0x2314 bytes) 000724 1787784774.002766 fallocate 0x0382af thru 0x03ffff (0x7d51 bytes) 000725 1787784774.008040 trunc from 0x02d206 to 0x024e59 (0x83ac bytes) 000726 1787784774.024721 trunc from 0x0118d7 to 0x01aee0 (0x960a bytes) 000727 1787784774.032086 mapwrite 0x008fa6 thru 0x00de6e (0x4ec9 bytes) 000728 1787784774.051029 punch from 0x01cc5e to 0x024e59 (0x81fc bytes) 000729 1787784774.056578 trunc from 0x01581f to 0x01dfd3 (0x87b5 bytes) 000730 1787784774.061945 trunc from 0x01e2d7 to 0x02287e (0x45a8 bytes) 000733 1787784774.068077 fallocate 0x018f85 thru 0x0245e2 (0xb65e bytes) 000735 1787784774.074964 write 0x001250 thru 0x00f1f8 (0xdfa9 bytes) 000736 1787784774.082746 mapwrite 0x036aa3 thru 0x03ffff (0x955d bytes) 000737 1787784774.153922 read 0x0162e7 thru 0x017050 (0xd6a bytes) 000738 1787784774.160190 mapwrite 0x007eb8 thru 0x01184d (0x9996 bytes) 000739 1787784774.191060 read 0x0108df thru 0x0158db (0x4ffd bytes) 000741 1787784774.193988 punch from 0x03806b to 0x03ffff (0x7f95 bytes) 000742 1787784774.200077 fallocate 0x02e54a thru 0x03a8f1 (0xc3a8 bytes) 000744 1787784774.206107 trunc from 0x040000 to 0x029f66 (0x16099 bytes) 000745 1787784774.224080 trunc from 0x029f67 to 0x039b65 (0xfbff bytes) 000746 1787784774.243582 trunc from 0x039b66 to 0x027560 (0x12605 bytes) 000748 1787784774.265649 mapread 0x021526 thru 0x02473c (0x3217 bytes) 000750 1787784774.283639 mapwrite 0x02b0b3 thru 0x02d4bb (0x2409 bytes) 000751 1787784774.326008 fallocate 0x021895 thru 0x0299f7 (0x8163 bytes) 000752 1787784774.333328 mapwrite 0x01b2b4 thru 0x01ef96 (0x3ce3 bytes) 000753 1787784774.352891 mapwrite 0x02723d thru 0x02a11b (0x2edf bytes) 000754 1787784774.370334 fallocate 0x018236 thru 0x021646 (0x9411 bytes) 000755 1787784774.377589 read 0x00ca01 thru 0x01468c (0x7c8c bytes) 000756 1787784774.379679 punch from 0x021b02 to 0x028d11 (0x7210 bytes) 000757 1787784774.387654 mapread 0x02449c thru 0x029544 (0x50a9 bytes) 000758 1787784774.401416 punch from 0x003e14 to 0x01113d (0xd32a bytes) 000759 1787784774.408294 punch from 0x012498 to 0x01e009 (0xbb72 bytes) 000760 1787784774.417365 read 0x018eeb thru 0x0232f3 (0xa409 bytes) 000761 1787784774.433260 mapwrite 0x029e7c thru 0x0342cc (0xa451 bytes) 000762 1787784774.526115 trunc from 0x017568 to 0x0261ed (0xec86 bytes) 000763 1787784774.532676 trunc from 0x013499 to 0x01c61c (0x9184 bytes) 000764 1787784774.539697 read 0x0051e9 thru 0x01000c (0xae24 bytes) 000766 1787784774.548116 MIRROR_EXTEND to 2 mirrors 000767 1787784775.518047 MIRROR_RESYNC 2 mirrors 000768 1787784776.081284 trunc from 0x0241ce to 0x02d198 (0x8fcb bytes) 000769 1787784776.146221 mapwrite 0x00afaa thru 0x0174aa (0xc501 bytes) 000770 1787784776.255424 mapwrite 0x01198b thru 0x0156ad (0x3d23 bytes) 000771 1787784776.295684 trunc from 0x01ab7c to 0x02363f (0x8ac4 bytes) 000772 1787784776.311052 trunc from 0x028997 to 0x0342cc (0xb936 bytes) 000773 1787784776.320493 trunc from 0x0342cd to 0x007c6b (0x2c661 bytes) 000774 1787784776.393642 punch from 0x00727c to 0x007c6b (0x9f0 bytes) 000775 1787784776.437599 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) 000776 1787784776.696202 punch from 0x0018a9 to 0x002f69 (0x16c1 bytes) 000777 1787784776.708798 fallocate 0x022265 thru 0x0247b4 (0x2550 bytes) 000778 1787784776.725214 mapwrite 0x01ee64 thru 0x027a0a (0x8ba7 bytes) 000779 1787784776.837405 write 0x03d7ab thru 0x03ffff (0x2855 bytes) 000780 1787784776.842912 read 0x01e6c5 thru 0x01f5c3 (0xeff bytes) 000781 1787784776.846404 trunc from 0x015e11 to 0x019a68 (0x3c58 bytes) 000782 1787784776.857348 mapwrite 0x016753 thru 0x02416a (0xda18 bytes) 000783 1787784776.917054 punch from 0x0132b0 to 0x013f0d (0xc5e bytes) 000784 1787784776.926575 write 0x02df7c thru 0x037b24 (0x9ba9 bytes) 000786 1787784776.949695 fallocate 0x0103ee thru 0x0108f5 (0x508 bytes) 000787 1787784776.959075 write 0x0334ea thru 0x03e93a (0xb451 bytes) 000788 1787784776.961180 read 0x011d2d thru 0x01db04 (0xbdd8 bytes) 000789 1787784776.979620 mapread 0x00a9cb thru 0x014671 (0x9ca7 bytes) 000790 1787784776.983326 read 0x031e53 thru 0x03c82f (0xa9dd bytes) 000791 1787784776.984616 fallocate 0x021ff3 thru 0x02f609 (0xd617 bytes) 000792 1787784776.993112 trunc from 0x01817b to 0x021d20 (0x9ba6 bytes) 000793 1787784777.010520 read 0x0091c7 thru 0x0099a2 (0x7dc bytes) 000794 1787784777.011514 read 0x001b51 thru 0x008007 (0x64b7 bytes) 000795 1787784777.019523 trunc from 0x040000 to 0x009c26 (0x363d9 bytes) 000796 1787784777.063038 write 0x020426 thru 0x02d773 (0xd34e bytes) 000797 1787784777.074394 trunc from 0x01026c to 0x01b76f (0xb504 bytes) 000798 1787784777.088490 mapwrite 0x013f55 thru 0x020b0a (0xcbb6 bytes) 000799 1787784777.181483 mapwrite 0x012841 thru 0x0128cb (0x8b bytes) 000800 1787784777.202085 trunc from 0x02d774 to 0x01c881 (0x10ef2 bytes) 000801 1787784777.221662 trunc from 0x01c882 to 0x0029b3 (0x19ece bytes) 000803 1787784777.240770 mapread 0x000673 thru 0x0029b3 (0x2341 bytes) 000805 1787784777.244594 trunc from 0x002847 to 0x0029b3 (0x16d bytes) 000806 1787784777.250505 trunc from 0x0029b4 to 0x02749e (0x24aeb bytes) 000807 1787784777.273327 write 0x01b552 thru 0x026d14 (0xb7c3 bytes) 000808 1787784777.288746 write 0x00af6c thru 0x0109a8 (0x5a3d bytes) 000809 1787784777.299927 read 0x000c1b thru 0x00b702 (0xaae8 bytes) 000811 1787784777.312181 mapwrite 0x003597 thru 0x004be5 (0x164f bytes) 000812 1787784777.329633 write 0x011f86 thru 0x014e13 (0x2e8e bytes) 000813 1787784777.331936 trunc from 0x02749f to 0x02bdea (0x494c bytes) 000814 1787784777.350293 mapwrite 0x03a700 thru 0x03ed29 (0x462a bytes) 000815 1787784777.412192 trunc from 0x03ed2a to 0x02287f (0x1c4aa bytes) 000816 1787784777.446385 fallocate 0x037ab3 thru 0x03ffff (0x854d bytes) 000817 1787784777.458769 write 0x03ae20 thru 0x03ffff (0x51e0 bytes) 000819 1787784777.482438 fallocate 0x006103 thru 0x008179 (0x2077 bytes) 000820 1787784777.490761 read 0x03c510 thru 0x03ffff (0x3af0 bytes) 000822 1787784777.492278 fallocate 0x009494 thru 0x009765 (0x2d2 bytes) 000823 1787784777.506337 fallocate 0x02f6b4 thru 0x036a2f (0x737c bytes) 000824 1787784777.514257 fallocate 0x023cc6 thru 0x032227 (0xe562 bytes) 000825 1787784777.526268 write 0x011b8a thru 0x01b177 (0x95ee bytes) 000826 1787784777.529240 fallocate 0x031b1b thru 0x037856 (0x5d3c bytes) 000827 1787784777.546322 write 0x003b76 thru 0x00ddf0 (0xa27b bytes) 000828 1787784777.552707 read 0x034d05 thru 0x03ffff (0xb2fb bytes) 000829 1787784777.571442 punch from 0x014dc8 to 0x0189dd (0x3c16 bytes) 000830 1787784777.592696 write 0x015d92 thru 0x02028c (0xa4fb bytes) 000831 1787784777.602902 mapwrite 0x02e0c4 thru 0x03d748 (0xf685 bytes) 000832 1787784777.639151 mapwrite 0x03f3ca thru 0x03ffff (0xc36 bytes) 000834 1787784777.665195 trunc from 0x01fd70 to 0x028270 (0x8501 bytes) 000837 1787784777.682500 mapwrite 0x021374 thru 0x025730 (0x43bd bytes) 000838 1787784777.791159 fallocate 0x02d2cf thru 0x03d025 (0xfd57 bytes) 000840 1787784777.816855 trunc from 0x040000 to 0x0244cc (0x1bb33 bytes) 000841 1787784777.869036 read 0x00194c thru 0x00fd26 (0xe3db bytes) 000842 1787784777.874385 punch from 0x022013 to 0x0244cc (0x24ba bytes) 000843 1787784777.898686 mapwrite 0x00b38f thru 0x019ba5 (0xe817 bytes) 000844 1787784777.961666 trunc from 0x00332e to 0x003915 (0x5e8 bytes) 000845 1787784777.981004 write 0x02bb02 thru 0x02d029 (0x1528 bytes) 000846 1787784777.983727 punch from 0x02315e to 0x028352 (0x51f5 bytes) 000847 1787784777.991351 mapread 0x026221 thru 0x02d029 (0x6e09 bytes) 000848 1787784778.003694 fallocate 0x03885d thru 0x03ffff (0x77a3 bytes) 000849 1787784778.014668 mapread 0x0286cb thru 0x033e00 (0xb736 bytes) 000850 1787784778.031787 write 0x001beb thru 0x00c7cb (0xabe1 bytes) 000851 1787784778.043773 MIRROR_EXTEND to 2 mirrors 000852 1787784779.034028 trunc from 0x00bcf6 to 0x013311 (0x761c bytes) 000853 1787784779.126384 punch from 0x014b95 to 0x01c8a5 (0x7d11 bytes) 000854 1787784779.134882 trunc from 0x0008b2 to 0x0047f2 (0x3f41 bytes) 000855 1787784779.141969 mapread 0x037c8f thru 0x038c94 (0x1006 bytes) 000857 1787784779.163202 fallocate 0x005fd7 thru 0x00bf88 (0x5fb2 bytes) 000858 1787784779.188875 read 0x003c6a thru 0x00be38 (0x81cf bytes) 000859 1787784779.214106 write 0x01e527 thru 0x01ff53 (0x1a2d bytes) 000860 1787784779.234437 write 0x01822d thru 0x01beeb (0x3cbf bytes) 000862 1787784779.265288 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc919d036 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7858d202 Mirror 3: 0x7858d202 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000863 1787784780.737420 mapread 0x00b7d3 thru 0x012509 (0x6d37 bytes) 000865 1787784780.767337 fallocate 0x024d7c thru 0x02e756 (0x99db bytes) 000866 1787784780.817883 trunc from 0x040000 to 0x037d22 (0x82dd bytes) 000867 1787784780.847620 punch from 0x02c896 to 0x0300b9 (0x3824 bytes) 000868 1787784780.853165 read 0x00efdd thru 0x01c49a (0xd4be bytes) 000869 1787784780.862999 trunc from 0x02e9f4 to 0x037d22 (0x932f bytes) 000870 1787784780.868050 fallocate 0x03ba1b thru 0x03ffff (0x45e5 bytes) 000871 1787784780.873469 fallocate 0x03bc5e thru 0x03ffff (0x43a2 bytes) 000872 1787784780.887031 MIRROR_RESYNC 3 mirrors 000873 1787784782.234639 punch from 0x0298d4 to 0x036a3d (0xd16a bytes) 000874 1787784782.277208 punch from 0x03a9e0 to 0x03ffff (0x5620 bytes) 000875 1787784782.282453 fallocate 0x02808a thru 0x02e4bc (0x6433 bytes) 000876 1787784782.288253 punch from 0x01ae0d to 0x02844e (0xd642 bytes) 000877 1787784782.293480 read 0x01f849 thru 0x02a98a (0xb142 bytes) 000878 1787784782.307978 punch from 0x02489b to 0x02dbe0 (0x9346 bytes) 000879 1787784782.312645 write 0x015555 thru 0x02517f (0xfc2b bytes) 000880 1787784782.328275 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x28e8241a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x28e8241a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000881 1787784782.671979 write 0x032e8a thru 0x03e70f (0xb886 bytes) 000882 1787784782.688053 trunc from 0x01721c to 0x01c428 (0x520d bytes) 000883 1787784782.695517 mapwrite 0x01d249 thru 0x01fd90 (0x2b48 bytes) 000884 1787784782.736742 trunc from 0x02cd56 to 0x03640f (0x96ba bytes) 000885 1787784782.743516 fallocate 0x0187b9 thru 0x0215a0 (0x8de8 bytes) 000888 1787784782.749755 trunc from 0x040000 to 0x019530 (0x26acf bytes) 000889 1787784782.771297 read 0x006f2c thru 0x0074ef (0x5c4 bytes) 000890 1787784782.779404 mapwrite 0x03fc4a thru 0x03ffff (0x3b6 bytes) 000891 1787784782.813640 punch from 0x03454c to 0x03caba (0x856f bytes) 000892 1787784782.819463 read 0x028abb thru 0x033c2a (0xb170 bytes) 000893 1787784782.830642 write 0x026f81 thru 0x03171a (0xa79a bytes) 000895 1787784782.833339 trunc from 0x02bba2 to 0x031c72 (0x60d1 bytes) 000896 1787784782.853937 write 0x0154a6 thru 0x02409d (0xebf8 bytes) 000897 1787784782.855896 mapread 0x036f71 thru 0x03f912 (0x89a2 bytes) 000898 1787784782.857880 fallocate 0x02bfd9 thru 0x03b636 (0xf65e bytes) 000899 1787784782.863671 mapread 0x034fa5 thru 0x03ffff (0xb05b bytes) 000900 1787784782.865347 write 0x00ff2a thru 0x019424 (0x94fb bytes) 000901 1787784782.867262 mapwrite 0x01666f thru 0x026531 (0xfec3 bytes) 000902 1787784782.888977 mapwrite 0x0377da thru 0x03ffff (0x8826 bytes) 000903 1787784782.908173 mapread 0x02d8a1 thru 0x03040e (0x2b6e bytes) 000904 1787784782.928781 write 0x02a752 thru 0x02e224 (0x3ad3 bytes) 000905 1787784782.930514 trunc from 0x00df7f to 0x014401 (0x6483 bytes) 000906 1787784782.936014 mapread 0x00331c thru 0x012914 (0xf5f9 bytes) 000907 1787784782.979053 read 0x038932 thru 0x03ffff (0x76ce bytes) 000908 1787784782.983806 MIRROR_RESYNC 2 mirrors 000909 1787784783.752430 mapwrite 0x010989 thru 0x01dd6c (0xd3e4 bytes) 000910 1787784783.848523 trunc from 0x01851a to 0x018d9b (0x882 bytes) 000911 1787784783.852350 mapread 0x03b044 thru 0x03e8fc (0x38b9 bytes) 000912 1787784783.858318 mapread 0x024521 thru 0x02e9af (0xa48f bytes) 000913 1787784783.871953 trunc from 0x00bc0e to 0x012b43 (0x6f36 bytes) 000914 1787784783.876632 trunc from 0x0362e7 to 0x03ffff (0x9d19 bytes) 000915 1787784783.880711 mapread 0x0253c9 thru 0x02e032 (0x8c6a bytes) 000916 1787784783.884934 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xccb2342d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6ee43338 Mirror 3: 0x6ee43338 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000917 1787784785.104790 fallocate 0x0031f5 thru 0x00f17b (0xbf87 bytes) 000918 1787784785.187546 mapread 0x02c4d6 thru 0x02fdaa (0x38d5 bytes) 000919 1787784785.255872 MIRROR_RESYNC 3 mirrors 000920 1787784786.795252 mapwrite 0x0386a1 thru 0x03ffff (0x795f bytes) 000921 1787784786.909080 fallocate 0x020e1c thru 0x02c998 (0xbb7d bytes) 000922 1787784786.917577 punch from 0x019c1b to 0x024233 (0xa619 bytes) 000923 1787784786.922473 fallocate 0x00d583 thru 0x010f33 (0x39b1 bytes) 000924 1787784786.931057 punch from 0x02ba20 to 0x038639 (0xcc1a bytes) 000926 1787784786.938973 trunc from 0x00e2f5 to 0x011ff0 (0x3cfc bytes) 000927 1787784786.944720 fallocate 0x00fbde thru 0x011b16 (0x1f39 bytes) 000928 1787784786.950460 trunc from 0x040000 to 0x03b9b7 (0x4648 bytes) 000929 1787784786.972115 mapread 0x030c33 thru 0x03b9b7 (0xad85 bytes) 000930 1787784786.979322 trunc from 0x03b9b8 to 0x013e00 (0x27bb7 bytes) 000931 1787784787.003272 write 0x01b81b thru 0x01ca51 (0x1237 bytes) 000932 1787784787.006292 punch from 0x015503 to 0x01ca51 (0x754f bytes) 000933 1787784787.054769 trunc from 0x014578 to 0x0163c6 (0x1e4f bytes) 000934 1787784787.072587 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ca52): Mirror 3: 0x34499cea (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ca52): Mirror 3: 0x34499cea (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ca52: skipped 000935 1787784787.528701 punch from 0x000327 to 0x003a69 (0x3743 bytes) 000936 1787784787.546564 write 0x03df83 thru 0x03ffff (0x207d bytes) 000937 1787784787.560244 fallocate 0x020783 thru 0x02566d (0x4eeb bytes) 000938 1787784787.592087 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) 000939 1787784787.815051 read 0x02cf99 thru 0x030428 (0x3490 bytes) 000940 1787784787.845591 punch from 0x0266ea to 0x034d70 (0xe687 bytes) 000941 1787784787.857999 trunc from 0x040000 to 0x0336b2 (0xc94d bytes) 000942 1787784787.918318 read 0x01f04a thru 0x0203d7 (0x138e bytes) 000943 1787784787.939775 trunc from 0x0083d4 to 0x0095fe (0x122b bytes) 000944 1787784787.961979 fallocate 0x023b18 thru 0x0334d0 (0xf9b9 bytes) 000945 1787784787.976619 trunc from 0x0336b3 to 0x03a07d (0x69cb bytes) 000946 1787784788.002778 mapread 0x00e320 thru 0x01e011 (0xfcf2 bytes) 000947 1787784788.011832 trunc from 0x03a07e to 0x005a56 (0x34627 bytes) 000949 1787784788.061573 fallocate 0x019088 thru 0x027186 (0xe0ff bytes) 000950 1787784788.067385 trunc from 0x00b024 to 0x01a452 (0xf42f bytes) 000951 1787784788.081803 write 0x023cb9 thru 0x028a06 (0x4d4e bytes) 000952 1787784788.096024 MIRROR_EXTEND to 2 mirrors 000953 1787784789.149953 trunc from 0x028a07 to 0x01a4e0 (0xe526 bytes) 000954 1787784789.226340 mapwrite 0x03700e thru 0x03ffff (0x8ff2 bytes) 000955 1787784789.298422 trunc from 0x03bf88 to 0x03ffff (0x4078 bytes) 000956 1787784789.305310 mapread 0x01dee6 thru 0x023d1e (0x5e39 bytes) 000957 1787784789.315302 trunc from 0x00f503 to 0x012bbd (0x36bb bytes) 000958 1787784789.321693 mapwrite 0x00d71f thru 0x0199ae (0xc290 bytes) 000959 1787784789.361219 mapwrite 0x027ff4 thru 0x02fc83 (0x7c90 bytes) 000960 1787784789.417648 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x47:0x0]: Invalid argument (22) 000961 1787784789.614453 punch from 0x003dcf to 0x006247 (0x2479 bytes) 000962 1787784789.618918 mapread 0x0127b6 thru 0x01367d (0xec8 bytes) 000963 1787784789.653589 punch from 0x0344bc to 0x03f81b (0xb360 bytes) 000965 1787784789.659687 mapwrite 0x02b49c thru 0x02dc31 (0x2796 bytes) 000966 1787784789.689882 fallocate 0x007faf thru 0x0145cd (0xc61f bytes) 000967 1787784789.696248 punch from 0x030418 to 0x0306f2 (0x2db bytes) 000968 1787784789.702256 fallocate 0x021052 thru 0x027f8d (0x6f3c bytes) 000969 1787784789.709188 trunc from 0x037392 to 0x03d50f (0x617e bytes) 000970 1787784789.714099 trunc from 0x03cb24 to 0x03ffff (0x34dc bytes) 000971 1787784789.720197 write 0x010d33 thru 0x01d018 (0xc2e6 bytes) 000972 1787784789.734436 write 0x004b5e thru 0x00a14f (0x55f2 bytes) 000973 1787784789.746258 trunc from 0x040000 to 0x031a5a (0xe5a5 bytes) 000974 1787784789.769725 trunc from 0x031a5b to 0x036d2e (0x52d4 bytes) 000975 1787784789.782518 trunc from 0x036d2f to 0x021e3f (0x14eef bytes) 000978 1787784789.796524 read 0x021dcf thru 0x021e3f (0x71 bytes) 000980 1787784789.804965 punch from 0x005d97 to 0x014237 (0xe4a1 bytes) 000981 1787784789.820113 mapread 0x01f354 thru 0x021e3f (0x2aec bytes) 000982 1787784789.827631 mapwrite 0x01c7fc thru 0x021f07 (0x570c bytes) 000983 1787784789.870408 read 0x013175 thru 0x019613 (0x649f bytes) 000986 1787784789.882270 mapwrite 0x00896d thru 0x00ba81 (0x3115 bytes) 000987 1787784789.897200 trunc from 0x01d7fd to 0x020461 (0x2c65 bytes) 000988 1787784789.902547 mapread 0x001ab0 thru 0x00e39b (0xc8ec bytes) 000989 1787784789.904972 mapwrite 0x02fd45 thru 0x0331bb (0x3477 bytes) 000990 1787784789.950523 trunc from 0x0331bc to 0x009bfc (0x295bf bytes) 000991 1787784789.967066 fallocate 0x00a2c8 thru 0x00b822 (0x155b bytes) 000992 1787784789.976298 read 0x00040c thru 0x009bfc (0x97f1 bytes) 000993 1787784789.977015 read 0x0080ca thru 0x009bfc (0x1b33 bytes) 000994 1787784789.977576 read 0x0060c3 thru 0x009bfc (0x3b3a bytes) 000997 1787784789.983645 punch from 0x008f72 to 0x009bfc (0xc8b bytes) 000998 1787784789.990996 fallocate 0x03b2f6 thru 0x03dbfc (0x2907 bytes) 000999 1787784789.997111 write 0x02e078 thru 0x03a5e0 (0xc569 bytes) 001000 1787784790.011545 mapread 0x02a405 thru 0x030b35 (0x6731 bytes) All operations completed A-OK! PASS 70a (76s) == sanity-flr test 71: check mirror extend parent fid ==== 18:53:15 (1787784795) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x7c:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 18:53:27 (1787784807) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2a6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.223742 s, 9.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2a6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2a7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122474 s, 8.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2a6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2a7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2a6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2a7:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a9:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2a8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2a8:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (12s) == sanity-flr test 200a: stress test ===================== 18:53:39 (1787784819) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4451840 .. Extending file size to 4503328 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7802912 .. Extending file size to 4555200 .. Extending file size to 5941056 .. Extending file size to 8161984 .. Extending file size to 9015296 .. Extending file size to 8538464 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8654112 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8821088 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 62482 62483 62485 62486 62489 failed 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) 192.168.202.140@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 18:55:02 (1787784902) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5204320 .. Extending file size to 294528 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2881664 .. Extending file size to 4039520 .. Extending file size to 7236256 .. failed Extending file size to 5790848 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8803552 .. Extending file size to 7125344 .. Extending file size to 8513728 .. Extending file size to 9345920 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8640288 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9178624 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 64125 64126 64128 64131 64132 done 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) 192.168.202.140@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 18:56:24 (1787784984) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1268856 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1948 1269356 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25728 3581292 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17868 3589152 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43596 7170444 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1268856 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1948 1269356 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25728 3581292 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17868 3589152 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43596 7170444 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.134934 s, 15.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.258574 s, 8.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x5cd95af6 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 18:56:49 (1787785009) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.114137 s, 18.4 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 18:56:57 (1787785017) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x172:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c5:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c5:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c6:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x2c3:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 18:57:07 (1787785027) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108781 s, 19.3 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x51:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x51:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x52:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 18:57:16 (1787785036) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0955378 s, 22.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c4:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c5:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 18:57:25 (1787785045) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.226162 s, 9.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x52:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x53:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } PASS 204c (37s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 18:58:02 (1787785082) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg240-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811169 s, 12.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x53:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x56:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x53:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x56:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x55:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x54:0x0] } PASS 204d (53s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 18:58:55 (1787785135) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg240-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0994153 s, 21.1 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2c7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2c9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (48s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 18:59:43 (1787785183) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg240-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg240-server: Pool lustre.test_204f created Adding targets to pool oleg240-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg240-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.131885 s, 15.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x57:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x55:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x57:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x55:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg240-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg240-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg240-server: Pool lustre.test_204f destroyed PASS 204f (79s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:01:02 (1787785262) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:01:12 (1787785272) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:01:22 (1787785282) oleg240-server: Pool lustre.test_206 created Adding targets to pool oleg240-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg240-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg240-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg240-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg240-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:01:52 (1787785312) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910152 s, 11.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640742 s, 16.4 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf0c7d4a4 Mirror 3: 0xf0c7d4a4 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:02:05 (1787785325) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:02:09 (1787785329) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:02:14 (1787785334) Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0523018 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x24a0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1940000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1940000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1940000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1120000 Read this (4 bytes): BBBB 192.168.202.140@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg240-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:02:54 (1787785374) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0304966 s, 269 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:03:03 (1787785383) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122427 s, 8.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x18f lmm_fid: [0x200000405:0x18f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x18f lmm_fid: [0x200000405:0x18f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2d1:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611535 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 19:03:05.000000000 -0400 Modify: 2026-08-26 19:03:05.000000000 -0400 Change: 2026-08-26 19:03:07.000000000 -0400 Birth: 2026-08-26 19:03:05.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:03:13 (1787785393) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573999 s, 18.3 MB/s oleg240-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg240-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg240-server: O/280000401/d18/2d2: File not found by ext2_lookup PASS 210b (27s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:03:40 (1787785420) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.273439 s, 150 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0212441 s, 193 kB/s size after second write -rw-r--r-- 1 root root 4096 Aug 26 19:03 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x194:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Aug 26 19:03 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 19:03:50 (1787785430) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 19:03:59 (1787785439) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (12s) == sanity-flr test complete, duration 1942 sec =========== 19:04:11 (1787785451) === sanity-flr: start cleanup 19:04:13 (1787785453) === === sanity-flr: finish cleanup 19:04:23 (1787785463) ===