-----============= acceptance-small: sanity-flr ============----- Thu Jun 18 17:18:02 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg405-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 17:18:11 (1781817491) === oleg405-client.virtnet: executing check_config_client /mnt/lustre oleg405-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg405-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff951d45421000.idle_timeout=debug osc.lustre-OST0001-osc-ffff951d45421000.idle_timeout=debug disable quota as required oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 17:18:31 (1781817511) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8206] == sanity-flr test 0a: lfs mirror create with -N option == 17:18:32 (1781817512) PASS 0a (7s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 17:18:39 (1781817519) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (2s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 17:18:41 (1781817521) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 17:18:43 (1781817523) PASS 0d (12s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 17:18:55 (1781817535) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (2s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 17:18:57 (1781817537) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (2s) == sanity-flr test 0g: lfs mirror create flags support === 17:18:59 (1781817539) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff951d45421000.stats=clear osc.lustre-OST0001-osc-ffff951d45421000.stats=clear PASS 0g (4s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 17:19:03 (1781817543) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (6s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 17:19:09 (1781817549) PASS 0j (4s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 17:19:13 (1781817553) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (2s) == sanity-flr test 0m: mirror extend with -N= option ===== 17:19:15 (1781817555) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (4s) == sanity-flr test 1: create components with setstripe options ========================================================== 17:19:19 (1781817559) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (4s) == sanity-flr test 2: create components from existing files ========================================================== 17:19:23 (1781817563) PASS 2 (4s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 17:19:27 (1781817567) PASS 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 17:19:31 (1781817571) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (3s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 17:19:34 (1781817574) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (4s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 17:19:39 (1781817579) PASS 7 (4s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 17:19:43 (1781817583) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.206216 s, 30.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473839 s, 22.1 MB/s PASS 21 (3s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 17:19:46 (1781817586) 18+0 records in 18+0 records out 18874368 bytes (19 MB, 18 MiB) copied, 0.217327 s, 86.8 MB/s PASS 22 (4s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 17:19:50 (1781817590) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (5s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 17:19:56 (1781817596) Stopping /mnt/lustre-ost1 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff951d45421000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff951d45421000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg405-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff951d45421000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff951d45421000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg405-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (39s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 17:20:35 (1781817635) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.231296 s, 40.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff951d45421000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff951d45421000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg405-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (19s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 17:20:54 (1781817654) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff951d45421000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff951d45421000.ost_server_uuid in DISCONN state after 1 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg405-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff951d45421000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff951d45421000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg405-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (41s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 17:21:35 (1781817695) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.45754 s, 36.0 MB/s -rw-r--r-- 1 root root 52428800 Jun 18 17:21 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.476s user 0m0.000s sys 0m0.173s Stopping /mnt/lustre-ost1 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff951d45421000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff951d45421000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.510s user 0m0.002s sys 0m0.181s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg405-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg405-server oleg405-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff951d45421000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff951d45421000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m0.594s user 0m0.001s sys 0m0.174s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg405-server: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg405-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg405-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff951d45421000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff951d45421000.ost_server_uuid in FULL state after 0 sec PASS 33b (46s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 17:22:21 (1781817741) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (2s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 17:22:23 (1781817743) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (1s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 17:22:25 (1781817745) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (1s) == sanity-flr test 35: allow to write to mirrored files == 17:22:26 (1781817746) PASS 35 (4s) == sanity-flr test 36a: write to mirrored files ========== 17:22:30 (1781817750) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.285594 s, 44.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (7s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 17:22:37 (1781817757) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.019299 s, 54.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0147044 s, 71.3 MB/s PASS 36b (3s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 17:22:40 (1781817760) PASS 36c (5s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 17:22:45 (1781817765) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00141124 s, 726 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0007633 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00090897 s, 1.1 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000985532 s, 1.0 MB/s flv=10 olv=10 PASS 36d (4s) == sanity-flr test 37: mirror I/O API verification ======= 17:22:49 (1781817769) create test files with size 15733677 .. dfd6c1a7943154e9a9b3c44c9ff812a6 - 0741f2046f45467b83d3193c14edf58b - a86ab57254eb0eafee84782c0b8672c1 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg405-client.virtnet /mnt/lustre (opts:) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre PASS 37 (9s) == sanity-flr test 38: resync ============================ 17:22:58 (1781817778) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (6s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 17:23:04 (1781817784) PASS 39 (3s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 17:23:07 (1781817787) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426652 s, 24.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0177525 s, 59.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (4s) == sanity-flr test 41: lfs mirror resync check =========== 17:23:11 (1781817791) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0418883 s, 50.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.166186 s, 25.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0883652 s, 23.7 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (7s) == sanity-flr test 42: lfs mirror verify ================= 17:23:18 (1781817798) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (2s) == sanity-flr test 43a: mirror pick on write ============= 17:23:20 (1781817800) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (1s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 17:23:21 (1781817801) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173741 s, 60.4 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (3s) == sanity-flr test 44a: lfs mirror split check =========== 17:23:25 (1781817805) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.136874 s, 23.0 MB/s PASS 44a (5s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 17:23:29 (1781817809) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (3s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 17:23:33 (1781817813) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.133531 s, 78.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg405-client.virtnet /mnt/lustre (opts:) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 17:23:46 (1781817826) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff951d43546000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.123574 s, 84.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg405-client.virtnet /mnt/lustre (opts:) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff951d44d23000.xattr_cache=1 PASS 44e (6s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 17:23:52 (1781817832) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (19s) == sanity-flr test 46: Verify setstripe --copy option ==== 17:24:11 (1781817851) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (17s) == sanity-flr test 47: Verify mirror obj alloc =========== 17:24:28 (1781817868) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (1s) == sanity-flr test 48: Verify snapshot mirror ============ 17:24:29 (1781817869) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.10426 s, 30.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.0783416 s, 40.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd8ecad61 (nosync,stale) Mirror 2: 0xd8ecad61 (nosync,stale) ** verify mirror 2 doesn't change original checksum: b8f6bcbeac684322a084a1c00db777f1 - mirror 1 checksum: c1ed86654178097a49f729cc6cca125e - mirror 2 checksum: b8f6bcbeac684322a084a1c00db777f1 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-18T17:24:31-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-06-18T17:24:31-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b8f6bcbeac684322a084a1c00db777f1 - mirror 1 checksum: c1ed86654178097a49f729cc6cca125e - mirror 2 checksum: c1ed86654178097a49f729cc6cca125e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-18T17:24:31-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-06-18T17:24:31-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd9:0x0] } PASS 48 (5s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 17:24:34 (1781817874) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0184417 s, 56.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (3s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 17:24:37 (1781817877) Failing mds1 on oleg405-server Stopping /mnt/lustre-mds1 (opts:) on oleg405-server 17:24:45 (1781817885) shut down facet: mds1 facet_host: oleg405-server facet_failover_host: oleg405-server Failover mds1 to oleg405-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg405-server: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-MDT0000 17:25:00 (1781817900) targets are mounted 17:25:00 (1781817900) facet_failover done oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0145377 s, 72.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd6:0x0] } PASS 50A (32s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 17:25:09 (1781817909) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0177109 s, 1.2 MB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00765199 s, 2.7 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 17:25:13 (1781817913) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0208415 s, 983 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00496057 s, 4.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 17:25:17 (1781817917) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00709298 s, 2.3 MB/s PASS 50c (3s) == sanity-flr test 50d: mirror rsync keep holes ========== 17:25:20 (1781817920) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0174989 s, 1.2 MB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (3s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 17:25:23 (1781817923) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0034088 s, 1.2 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0206618 s, 198 kB/s PASS 60a (4s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 17:25:27 (1781817927) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1d:0x0]: Invalid argument (22) PASS 60b (10s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 17:25:37 (1781817937) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (20s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 17:25:57 (1781817957) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x47:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (19s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 17:26:16 (1781817976) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (18s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 17:26:34 (1781817994) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (2s) == sanity-flr test 70: mirror create and split race ====== 17:26:36 (1781817996) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 57280 mirror split pid 57281 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5023: 57280 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5023: 57281 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (63s) == sanity-flr test 70a: flr mode fsx test ================ 17:27:39 (1781818059) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 8060 truncating to largest ever: 0x39ab2 000002 1781818060.120682 trunc from 00000000 to 0x039ab1 (0x39ab2 bytes) 000003 1781818060.126402 mapwrite 0x030741 thru 0x039dd0 (0x9690 bytes) 000004 1781818060.152876 punch from 0x032d74 to 0x033114 (0x3a1 bytes) 000005 1781818060.155835 read 0x028b89 thru 0x0348d2 (0xbd4a bytes) 000006 1781818060.159425 punch from 0x01f799 to 0x029cd1 (0xa539 bytes) 000007 1781818060.161690 trunc from 0x039dd1 to 0x00b328 (0x2eaa8 bytes) 000008 1781818060.167935 mapread 0x001060 thru 0x002ab3 (0x1a54 bytes) 000009 1781818060.171763 read 0x008af8 thru 0x008d88 (0x291 bytes) 000010 1781818060.172419 write 0x01693d thru 0x020805 (0x9ec9 bytes) 000011 1781818060.173528 read 0x017879 thru 0x020805 (0x8f8d bytes) 000012 1781818060.174183 read 0x01bcfa thru 0x01c080 (0x387 bytes) 000013 1781818060.174790 mapwrite 0x000965 thru 0x00c1ca (0xb866 bytes) 000014 1781818060.186781 mapwrite 0x010767 thru 0x01f13b (0xe9d5 bytes) 000015 1781818060.196361 trunc from 0x020806 to 0x00c19d (0x14668 bytes) 000016 1781818060.201681 trunc from 0x00c19e to 0x003771 (0x8a2c bytes) 000017 1781818060.206936 trunc from 0x00244c to 0x00353d (0x10f2 bytes) 000018 1781818060.209103 trunc from 0x00375e to 0x003771 (0x14 bytes) 000019 1781818060.210767 trunc from 0x000e00 to 0x003771 (0x2972 bytes) 000020 1781818060.212508 mapread 0x000110 thru 0x003771 (0x3662 bytes) fallocating to largest ever: 0x40000 000022 1781818060.220000 fallocate 0x03b840 thru 0x03ffff (0x47c0 bytes) 000024 1781818060.223402 trunc from 0x02cfb9 to 0x0316b2 (0x46fa bytes) 000025 1781818060.225710 fallocate 0x03b589 thru 0x03dd34 (0x27ac bytes) 000028 1781818060.229233 trunc from 0x040000 to 0x02c73b (0x138c4 bytes) 000029 1781818060.234701 trunc from 0x02c73c to 0x019e32 (0x12909 bytes) 000030 1781818060.239698 read 0x01091a thru 0x019e32 (0x9519 bytes) 000031 1781818060.243239 trunc from 0x0141c5 to 0x019e32 (0x5c6e bytes) 000032 1781818060.245545 mapwrite 0x00859d thru 0x00bf07 (0x396b bytes) 000033 1781818060.251349 mapwrite 0x011ab7 thru 0x01402e (0x2578 bytes) 000034 1781818060.258845 punch from 0x010dbd to 0x0183a4 (0x75e8 bytes) 000035 1781818060.261228 write 0x01f856 thru 0x0215ad (0x1d58 bytes) 000036 1781818060.262050 trunc from 0x0215ae to 0x019a57 (0x7b56 bytes) 000037 1781818060.267661 mapread 0x00ec8a thru 0x014969 (0x5ce0 bytes) 000038 1781818060.275701 fallocate 0x0328f7 thru 0x035737 (0x2e41 bytes) 000039 1781818060.277971 trunc from 0x035738 to 0x03a0d3 (0x499c bytes) 000041 1781818060.283799 mapwrite 0x026572 thru 0x02adb8 (0x4847 bytes) 000042 1781818060.293461 write 0x01f771 thru 0x0244d6 (0x4d66 bytes) 000044 1781818060.295187 mapwrite 0x02221b thru 0x03090d (0xe6f3 bytes) 000045 1781818060.304085 trunc from 0x01b3b3 to 0x023f3b (0x8b89 bytes) 000046 1781818060.306711 fallocate 0x027109 thru 0x02ce81 (0x5d79 bytes) 000048 1781818060.309809 fallocate 0x01169c thru 0x013ca2 (0x2607 bytes) 000049 1781818060.311898 fallocate 0x0250f6 thru 0x0326fd (0xd608 bytes) 000050 1781818060.315301 MIRROR_EXTEND to 2 mirrors 000052 1781818060.530662 punch from 0x030375 to 0x03321e (0x2eaa bytes) 000053 1781818060.544697 fallocate 0x020380 thru 0x02db5d (0xd7de bytes) 000054 1781818060.547406 punch from 0x03970d to 0x03a0d3 (0x9c7 bytes) 000055 1781818060.549711 fallocate 0x00ff06 thru 0x01c8f8 (0xc9f3 bytes) 000056 1781818060.552595 trunc from 0x01885c to 0x026575 (0xdd1a bytes) 000057 1781818060.555044 trunc from 0x03a0d4 to 0x03b54a (0x1477 bytes) 000058 1781818060.569466 read 0x0232a3 thru 0x03033d (0xd09b bytes) 000059 1781818060.573723 trunc from 0x012359 to 0x0137a6 (0x144e bytes) 000061 1781818060.576613 trunc from 0x03b54b to 0x01a97b (0x20bcf bytes) 000062 1781818060.581314 trunc from 0x018c48 to 0x01a97b (0x1d34 bytes) 000063 1781818060.583275 mapwrite 0x008dd8 thru 0x018831 (0xfa5a bytes) 000064 1781818060.599411 fallocate 0x00313d thru 0x008ed9 (0x5d9d bytes) 000065 1781818060.601661 mapread 0x0143d4 thru 0x01a97b (0x65a8 bytes) 000066 1781818060.602450 trunc from 0x01a97c to 0x0090c7 (0x118b4 bytes) 000067 1781818060.607625 punch from 0x004c22 to 0x0090c7 (0x44a6 bytes) 000068 1781818060.610701 MIRROR_RESYNC 2 mirrors 000069 1781818060.789674 mapwrite 0x028c9c thru 0x0368d5 (0xdc3a bytes) 000072 1781818060.829829 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x368d6): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x368d6): Mirror 2: 0x51304e1f (stale) CRC-32 checksum value for chunk [0, 0x368d6): Mirror 1: 0xeae2b1e7 Mirror 3: 0xeae2b1e7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x368d6: skipped 000073 1781818061.102211 mapwrite 0x011787 thru 0x01bcef (0xa569 bytes) 000074 1781818061.154981 trunc from 0x02f4b3 to 0x0368d5 (0x7423 bytes) 000075 1781818061.157930 trunc from 0x007e31 to 0x01369f (0xb86f bytes) 000076 1781818061.160857 mapread 0x019580 thru 0x01ddc8 (0x4849 bytes) 000077 1781818061.161932 read 0x03370a thru 0x0368d5 (0x31cc bytes) 000078 1781818061.164880 write 0x00b254 thru 0x018015 (0xcdc2 bytes) 000079 1781818061.169977 mapread 0x001913 thru 0x00f135 (0xd823 bytes) 000080 1781818061.174177 trunc from 0x0368d6 to 0x018e57 (0x1da7e bytes) 000081 1781818061.190725 write 0x00781f thru 0x00b267 (0x3a49 bytes) 000082 1781818061.191696 trunc from 0x01222d to 0x017296 (0x506a bytes) 000083 1781818061.198803 trunc from 0x018e58 to 0x033346 (0x1a4ef bytes) 000084 1781818061.204204 fallocate 0x02d25e thru 0x039e2c (0xcbcf bytes) 000086 1781818061.206837 write 0x006253 thru 0x00779a (0x1548 bytes) 000087 1781818061.209172 MIRROR_RESYNC 3 mirrors 000088 1781818061.466022 mapwrite 0x03470b thru 0x036810 (0x2106 bytes) 000089 1781818061.500488 read 0x024740 thru 0x02db5f (0x9420 bytes) 000090 1781818061.504026 fallocate 0x01782a thru 0x01af29 (0x3700 bytes) 000091 1781818061.508202 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39e2d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39e2d): Mirror 2: 0xa90d4a46 (stale) Mirror 3: 0xa90d4a46 (stale) CRC-32 checksum value for chunk [0, 0x39e2d): Mirror 1: 0x2d2ac896 Mirror 4: 0x2d2ac896 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39e2d: skipped 000093 1781818061.789684 punch from 0x00bec9 to 0x0105bc (0x46f4 bytes) 000094 1781818061.803481 write 0x017c45 thru 0x01bce8 (0x40a4 bytes) 000095 1781818061.809165 read 0x01aadf thru 0x020ca3 (0x61c5 bytes) 000096 1781818061.812608 trunc from 0x039e2d to 0x01109f (0x28d8d bytes) 000098 1781818061.825535 mapwrite 0x0243c7 thru 0x031e8d (0xdac7 bytes) 000099 1781818061.845724 punch from 0x02f600 to 0x031e8d (0x288e bytes) 000100 1781818061.847759 write 0x019b0c thru 0x01ed6a (0x525f bytes) 000101 1781818061.851351 mapwrite 0x0073e6 thru 0x01602e (0xec49 bytes) 000102 1781818061.865269 punch from 0x0222c9 to 0x02f1a6 (0xcede bytes) 000103 1781818061.867721 read 0x00f9ac thru 0x00fa58 (0xad bytes) 000104 1781818061.868195 read 0x010b64 thru 0x01b4e1 (0xa97e bytes) 000105 1781818061.870578 MIRROR_RESYNC 4 mirrors 000107 1781818062.288343 fallocate 0x015b4b thru 0x021964 (0xbe1a bytes) 000108 1781818062.297753 write 0x01a2c6 thru 0x01b4c1 (0x11fc bytes) 000109 1781818062.301043 fallocate 0x0169ee thru 0x02488f (0xdea2 bytes) 000110 1781818062.302990 trunc from 0x031e8e to 0x03ca1d (0xab90 bytes) 000111 1781818062.309035 mapread 0x01a7ac thru 0x02965c (0xeeb1 bytes) 000112 1781818062.317743 fallocate 0x032e93 thru 0x03ffff (0xd16d bytes) 000113 1781818062.319760 read 0x002927 thru 0x00d19d (0xa877 bytes) 000114 1781818062.322743 fallocate 0x03fbfb thru 0x03ffff (0x405 bytes) 000115 1781818062.324748 punch from 0x0179c6 to 0x01d90a (0x5f45 bytes) 000116 1781818062.328300 mapread 0x015d33 thru 0x017fbd (0x228b bytes) 000117 1781818062.331855 mapread 0x00fbd6 thru 0x01b61f (0xba4a bytes) 000118 1781818062.332713 mapread 0x00c620 thru 0x017cb1 (0xb692 bytes) 000119 1781818062.333637 fallocate 0x01b9e7 thru 0x01fefa (0x4514 bytes) 000120 1781818062.336334 write 0x0004a9 thru 0x001011 (0xb69 bytes) 000121 1781818062.340137 write 0x034cb2 thru 0x03f202 (0xa551 bytes) 000122 1781818062.344257 mapwrite 0x02ba21 thru 0x03b20a (0xf7ea bytes) 000123 1781818062.356362 punch from 0x002361 to 0x007fb9 (0x5c59 bytes) 000124 1781818062.360442 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f203): Mirror 2: 0x43661345 (stale) Mirror 4: 0x43661345 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f203): Mirror 2: 0x43661345 (stale) Mirror 4: 0x43661345 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f203: skipped 000125 1781818062.471409 mapwrite 0x017494 thru 0x020a55 (0x95c2 bytes) 000126 1781818062.492388 mapread 0x03df6d thru 0x03f202 (0x1296 bytes) 000127 1781818062.496502 write 0x0197a8 thru 0x02493d (0xb196 bytes) 000128 1781818062.500138 write 0x0023b0 thru 0x00908c (0x6cdd bytes) 000129 1781818062.503955 mapwrite 0x0338fd thru 0x03ffff (0xc703 bytes) 000130 1781818062.525297 trunc from 0x040000 to 0x01c781 (0x2387e bytes) 000131 1781818062.534028 MIRROR_RESYNC 3 mirrors 000132 1781818062.747233 fallocate 0x019fc5 thru 0x0289ff (0xea3b bytes) 000133 1781818062.759736 mapwrite 0x00b3da thru 0x015c98 (0xa8bf bytes) 000134 1781818062.774232 write 0x01fecf thru 0x025f50 (0x6082 bytes) 000135 1781818062.778615 mapread 0x002c05 thru 0x006cc0 (0x40bc bytes) 000136 1781818062.781694 write 0x0318fb thru 0x03dea4 (0xc5aa bytes) 000137 1781818062.782941 write 0x02bac2 thru 0x033cbe (0x81fd bytes) 000138 1781818062.785388 write 0x01ac42 thru 0x025d86 (0xb145 bytes) 000139 1781818062.787821 mapread 0x017683 thru 0x01b77e (0x40fc bytes) 000140 1781818062.791144 write 0x01cdc4 thru 0x01e839 (0x1a76 bytes) 000142 1781818062.792795 fallocate 0x005e95 thru 0x00e9cd (0x8b39 bytes) 000143 1781818062.795711 mapread 0x0203ae thru 0x02eae5 (0xe738 bytes) 000144 1781818062.802682 MIRROR_RESYNC 3 mirrors 000145 1781818063.047669 mapwrite 0x0386f2 thru 0x03f5c4 (0x6ed3 bytes) 000146 1781818063.081654 fallocate 0x02d14f thru 0x02f24d (0x20ff bytes) 000147 1781818063.083991 read 0x02a3c3 thru 0x02b30a (0xf48 bytes) 000148 1781818063.086796 fallocate 0x0007ee thru 0x00a1a7 (0x99ba bytes) 000149 1781818063.090480 read 0x039ae9 thru 0x03c117 (0x262f bytes) 000150 1781818063.091059 trunc from 0x03f5c5 to 0x023710 (0x1beb4 bytes) 000151 1781818063.099036 mapwrite 0x0153fc thru 0x01f8cc (0xa4d1 bytes) 000153 1781818063.112689 mapwrite 0x0184ce thru 0x01bb1c (0x364f bytes) 000154 1781818063.121591 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x23711): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23711): Mirror 2: 0x5d6a7820 (stale) Mirror 4: 0x5d6a7820 (stale) CRC-32 checksum value for chunk [0, 0x23711): Mirror 1: 0xd38df623 Mirror 5: 0xd38df623 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23711: skipped 000155 1781818063.379969 write 0x031b77 thru 0x03fa34 (0xdebe bytes) 000156 1781818063.395343 punch from 0x02bc61 to 0x0386bf (0xca5f bytes) 000157 1781818063.404211 write 0x01a1df thru 0x02032c (0x614e bytes) 000158 1781818063.409845 write 0x00d225 thru 0x01061c (0x33f8 bytes) 000159 1781818063.419677 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3fa35): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fa35): Mirror 2: 0x5d3b20fb (stale) Mirror 4: 0x5d3b20fb (stale) Mirror 5: 0xf8d0c617 (stale) CRC-32 checksum value for chunk [0, 0x3fa35): Mirror 1: 0x5ab6fb2 Mirror 6: 0x5ab6fb2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fa35: skipped 000160 1781818063.708007 fallocate 0x028f7b thru 0x02c46f (0x34f5 bytes) 000162 1781818063.723629 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3fa35): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fa35): Mirror 2: 0x5d3b20fb (stale) Mirror 4: 0x5d3b20fb (stale) Mirror 5: 0xf8d0c617 (stale) Mirror 6: 0x5ab6fb2 (stale) CRC-32 checksum value for chunk [0, 0x3fa35): Mirror 1: 0x5ab6fb2 Mirror 7: 0x5ab6fb2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fa35: skipped 000163 1781818064.000639 trunc from 0x031f41 to 0x033751 (0x1811 bytes) 000164 1781818064.013344 punch from 0x028987 to 0x0300c1 (0x773b bytes) 000165 1781818064.015077 read 0x015b74 thru 0x021171 (0xb5fe bytes) 000166 1781818064.018342 read 0x006f3b thru 0x00f923 (0x89e9 bytes) 000167 1781818064.022730 punch from 0x005658 to 0x00804e (0x29f7 bytes) 000168 1781818064.025079 read 0x01c5ff thru 0x02b526 (0xef28 bytes) 000169 1781818064.025801 read 0x01d11d thru 0x0268da (0x97be bytes) 000170 1781818064.028078 MIRROR_RESYNC 6 mirrors 000171 1781818064.428688 mapwrite 0x0241b6 thru 0x026864 (0x26af bytes) 000172 1781818064.462274 write 0x02fee7 thru 0x034579 (0x4693 bytes) 000173 1781818064.466803 MIRROR_RESYNC 6 mirrors 000174 1781818064.862356 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) 000175 1781818064.983229 read 0x0141b6 thru 0x01ae12 (0x6c5d bytes) 000178 1781818064.986941 write 0x017eb4 thru 0x025a15 (0xdb62 bytes) 000179 1781818065.003890 write 0x017a5c thru 0x019e87 (0x242c bytes) 000180 1781818065.004882 write 0x03cd89 thru 0x03ffff (0x3277 bytes) 000181 1781818065.007456 punch from 0x02ea70 to 0x03832e (0x98bf bytes) 000182 1781818065.009963 punch from 0x0225c5 to 0x02e7ad (0xc1e9 bytes) 000183 1781818065.014770 trunc from 0x040000 to 0x029770 (0x1688f bytes) 000184 1781818065.026359 mapread 0x01acc9 thru 0x01ff4a (0x5282 bytes) 000185 1781818065.027413 trunc from 0x029771 to 0x025891 (0x3edf bytes) 000186 1781818065.032937 trunc from 0x025892 to 0x034287 (0xe9f6 bytes) 000187 1781818065.038644 read 0x01a794 thru 0x029629 (0xee96 bytes) 000188 1781818065.042874 trunc from 0x021e39 to 0x02698f (0x4b57 bytes) 000189 1781818065.045350 punch from 0x022260 to 0x0294bf (0x7260 bytes) 000190 1781818065.047807 trunc from 0x034288 to 0x03a005 (0x5d7e bytes) 000191 1781818065.053990 mapread 0x015445 thru 0x0177f7 (0x23b3 bytes) 000192 1781818065.054793 write 0x012798 thru 0x0220e6 (0xf94f bytes) 000193 1781818065.058458 fallocate 0x036058 thru 0x03b22a (0x51d3 bytes) 000194 1781818065.060964 fallocate 0x003d59 thru 0x00de72 (0xa11a bytes) 000195 1781818065.063330 write 0x0090a1 thru 0x00f4e6 (0x6446 bytes) 000196 1781818065.065956 MIRROR_RESYNC 5 mirrors 000197 1781818065.355767 trunc from 0x025732 to 0x026c02 (0x14d1 bytes) 000198 1781818065.367082 trunc from 0x03a006 to 0x00aa66 (0x2f59f bytes) 000199 1781818065.378585 mapwrite 0x013823 thru 0x0223bf (0xeb9d bytes) 000200 1781818065.398348 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x223c0): Mirror 2: 0x30f4c76a (stale) Mirror 5: 0x30f4c76a (stale) Mirror 7: 0x30f4c76a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x223c0): Mirror 2: 0x30f4c76a (stale) Mirror 5: 0x30f4c76a (stale) Mirror 7: 0x30f4c76a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x223c0: skipped 000201 1781818065.502758 fallocate 0x0376df thru 0x03c9f1 (0x5313 bytes) 000202 1781818065.506848 read 0x0195af thru 0x01ed3f (0x5791 bytes) 000203 1781818065.510993 fallocate 0x0022e7 thru 0x00f991 (0xd6ab bytes) 000204 1781818065.515945 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3c9f2): Mirror 2: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c9f2): Mirror 2: 0x6c8d39ca (stale) Mirror 5: 0x6c8d39ca (stale) Mirror 7: 0x6c8d39ca (stale) CRC-32 checksum value for chunk [0, 0x3c9f2): Mirror 1: 0x4648c16f Mirror 8: 0x4648c16f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c9f2: skipped 000205 1781818065.763208 mapwrite 0x00f4aa thru 0x016f85 (0x7adc bytes) 000206 1781818065.806466 mapread 0x029f04 thru 0x02ed30 (0x4e2d bytes) 000207 1781818065.810598 mapread 0x031b58 thru 0x031d22 (0x1cb bytes) 000208 1781818065.811216 write 0x019b4a thru 0x0231b0 (0x9667 bytes) 000209 1781818065.814857 write 0x01ff04 thru 0x02a19e (0xa29b bytes) 000211 1781818065.816496 fallocate 0x010e51 thru 0x0173c4 (0x6574 bytes) 000212 1781818065.818498 mapwrite 0x032122 thru 0x03c399 (0xa278 bytes) 000213 1781818065.829149 mapwrite 0x003284 thru 0x00e555 (0xb2d2 bytes) 000215 1781818065.839443 read 0x01a21a thru 0x01a883 (0x66a bytes) 000216 1781818065.841334 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c9f2): Mirror 5: 0x6c8d39ca (stale) Mirror 7: 0x6c8d39ca (stale) Mirror 8: 0x4648c16f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c9f2): Mirror 5: 0x6c8d39ca (stale) Mirror 7: 0x6c8d39ca (stale) Mirror 8: 0x4648c16f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c9f2: skipped 000217 1781818065.928541 read 0x01f321 thru 0x02cdcd (0xdaad bytes) 000218 1781818065.932077 fallocate 0x020140 thru 0x028a3d (0x88fe bytes) 000219 1781818065.935302 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c9f2): Mirror 5: 0x6c8d39ca (stale) Mirror 7: 0x6c8d39ca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c9f2): Mirror 5: 0x6c8d39ca (stale) Mirror 7: 0x6c8d39ca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c9f2: skipped 000220 1781818066.024528 write 0x023960 thru 0x032c64 (0xf305 bytes) 000221 1781818066.028642 mapread 0x00a91e thru 0x01699e (0xc081 bytes) 000222 1781818066.037267 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c9f2): Mirror 7: 0x6c8d39ca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c9f2): Mirror 7: 0x6c8d39ca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c9f2: skipped 000223 1781818066.128954 punch from 0x003f31 to 0x007bcf (0x3c9f bytes) 000224 1781818066.131875 mapread 0x01cd1d thru 0x01d4b8 (0x79c bytes) 000225 1781818066.136465 mapread 0x0147df thru 0x01ac07 (0x6429 bytes) 000226 1781818066.140242 trunc from 0x03c9f2 to 0x0371b6 (0x583b bytes) 000227 1781818066.152906 mapread 0x01ef1c thru 0x02a6b4 (0xb799 bytes) 000228 1781818066.159302 mapread 0x00357d thru 0x00e3e8 (0xae6c bytes) 000229 1781818066.163153 mapread 0x01ecae thru 0x01f9c2 (0xd15 bytes) 000230 1781818066.164033 trunc from 0x0371b7 to 0x02f54c (0x7c6a bytes) 000232 1781818066.172057 write 0x00c790 thru 0x016663 (0x9ed4 bytes) 000233 1781818066.173534 mapwrite 0x03f13f thru 0x03ffff (0xec1 bytes) 000234 1781818066.190426 trunc from 0x040000 to 0x038868 (0x7797 bytes) 000235 1781818066.197389 MIRROR_RESYNC 2 mirrors 000236 1781818066.378294 fallocate 0x01d14d thru 0x02c0e6 (0xef9a bytes) 000237 1781818066.388243 trunc from 0x038869 to 0x03bea3 (0x363b bytes) 000238 1781818066.394333 trunc from 0x03bea4 to 0x008e2f (0x33074 bytes) 000239 1781818066.400112 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) 000240 1781818066.440826 read 0x007276 thru 0x008e2f (0x1bba bytes) 000243 1781818066.446812 MIRROR_EXTEND to 2 mirrors 000244 1781818066.627661 mapwrite 0x020c54 thru 0x0272b6 (0x6663 bytes) 000245 1781818066.657447 trunc from 0x0272b7 to 0x038959 (0x116a3 bytes) 000246 1781818066.664010 trunc from 0x03895a to 0x006b8a (0x31dcf bytes) 000247 1781818066.669324 write 0x037e49 thru 0x03bfde (0x4196 bytes) 000248 1781818066.670491 trunc from 0x0272cc to 0x036779 (0xf4ae bytes) 000249 1781818066.672880 mapread 0x0112b0 thru 0x01baf1 (0xa842 bytes) 000250 1781818066.679871 punch from 0x03b9f8 to 0x03bfde (0x5e7 bytes) 000251 1781818066.686128 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bfdf): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfdf): Mirror 2: 0x632268c4 (stale) CRC-32 checksum value for chunk [0, 0x3bfdf): Mirror 1: 0xbccee63d Mirror 3: 0xbccee63d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfdf: skipped 000254 1781818066.917716 trunc from 0x03bfdf to 0x019ac2 (0x2251c bytes) 000255 1781818066.933374 fallocate 0x01ad0b thru 0x01cf25 (0x221b bytes) 000256 1781818066.935185 write 0x02266f thru 0x029943 (0x72d5 bytes) 000257 1781818066.936477 read 0x002a89 thru 0x0064c2 (0x3a3a bytes) 000258 1781818066.940659 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x29944): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29944): Mirror 2: 0xb2219eb2 (stale) Mirror 3: 0x7921dc2 (stale) CRC-32 checksum value for chunk [0, 0x29944): Mirror 1: 0x7a1253d9 Mirror 4: 0x7a1253d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29944: skipped 000259 1781818067.200683 trunc from 0x011e0f to 0x017c4e (0x5e40 bytes) 000260 1781818067.213680 read 0x02133d thru 0x0219e3 (0x6a7 bytes) 000261 1781818067.215836 mapwrite 0x014895 thru 0x015e1f (0x158b bytes) 000263 1781818067.226235 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x29944): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29944): Mirror 2: 0xb2219eb2 (stale) Mirror 3: 0x7921dc2 (stale) Mirror 4: 0x7a1253d9 (stale) CRC-32 checksum value for chunk [0, 0x29944): Mirror 1: 0x48e1bd14 Mirror 5: 0x48e1bd14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29944: skipped 000264 1781818067.486410 punch from 0x02468e to 0x029561 (0x4ed4 bytes) 000265 1781818067.497076 write 0x00f134 thru 0x010982 (0x184f bytes) 000266 1781818067.501733 mapread 0x0052aa thru 0x00e36d (0x90c4 bytes) 000267 1781818067.508797 MIRROR_RESYNC 5 mirrors 000268 1781818067.865287 mapread 0x0066a4 thru 0x010c21 (0xa57e bytes) 000269 1781818067.871919 mapwrite 0x02f13d thru 0x02fadb (0x99f bytes) 000270 1781818067.900739 punch from 0x000875 to 0x003830 (0x2fbc bytes) 000271 1781818067.902699 trunc from 0x02fadc to 0x0281ec (0x78ef bytes) 000272 1781818067.910447 fallocate 0x01e2f2 thru 0x02b1db (0xceea bytes) 000273 1781818067.913112 write 0x0199c2 thru 0x0226ac (0x8ceb bytes) 000274 1781818067.919919 trunc from 0x005e5b to 0x00e1f7 (0x839d bytes) 000275 1781818067.921875 mapread 0x0167e3 thru 0x01a083 (0x38a1 bytes) 000276 1781818067.925082 trunc from 0x01446f to 0x01c2e7 (0x7e79 bytes) 000277 1781818067.929315 punch from 0x01fc6d to 0x02b1db (0xb56f bytes) 000279 1781818067.931956 mapread 0x0079cc thru 0x00dafa (0x612f bytes) 000280 1781818067.936724 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2b1dc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b1dc): Mirror 2: 0x3f889e1 (stale) Mirror 3: 0x3f889e1 (stale) Mirror 4: 0x3f889e1 (stale) Mirror 5: 0x3f889e1 (stale) CRC-32 checksum value for chunk [0, 0x2b1dc): Mirror 1: 0x39f4c8e9 Mirror 6: 0x39f4c8e9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b1dc: skipped 000281 1781818068.251434 write 0x02268f thru 0x027ae7 (0x5459 bytes) 000283 1781818068.268620 mapwrite 0x010cec thru 0x015096 (0x43ab bytes) 000284 1781818068.283327 fallocate 0x02fc4a thru 0x030aaf (0xe66 bytes) 000287 1781818068.287313 read 0x016300 thru 0x01f54f (0x9250 bytes) 000288 1781818068.291155 mapwrite 0x00e00c thru 0x00f8d6 (0x18cb bytes) 000289 1781818068.296639 trunc from 0x01006c to 0x01c161 (0xc0f6 bytes) 000290 1781818068.300608 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30ab0): Mirror 2: 0xbaaadab7 (stale) Mirror 3: 0xbaaadab7 (stale) Mirror 4: 0xbaaadab7 (stale) Mirror 6: 0xbcb275a0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30ab0): Mirror 2: 0xbaaadab7 (stale) Mirror 3: 0xbaaadab7 (stale) Mirror 4: 0xbaaadab7 (stale) Mirror 6: 0xbcb275a0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30ab0: skipped 000291 1781818068.417674 trunc from 0x030ab0 to 0x03bbe4 (0xb135 bytes) 000292 1781818068.424358 write 0x010fd8 thru 0x019206 (0x822f bytes) 000293 1781818068.428692 mapwrite 0x0068b0 thru 0x012276 (0xb9c7 bytes) 000295 1781818068.442922 fallocate 0x036ec2 thru 0x03ffff (0x913e bytes) 000296 1781818068.446510 punch from 0x0132ab to 0x022b86 (0xf8dc bytes) 000297 1781818068.448334 trunc from 0x01c7a1 to 0x02a702 (0xdf62 bytes) 000298 1781818068.449853 trunc from 0x022dfc to 0x02ddd1 (0xafd6 bytes) 000299 1781818068.452880 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4bcc1e76 (stale) Mirror 3: 0x4bcc1e76 (stale) Mirror 4: 0x4bcc1e76 (stale) Mirror 6: 0x7bd4b786 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x427ad77f Mirror 7: 0x427ad77f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000300 1781818068.712149 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4bcc1e76 (stale) Mirror 4: 0x4bcc1e76 (stale) Mirror 6: 0x7bd4b786 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x4bcc1e76 (stale) Mirror 4: 0x4bcc1e76 (stale) Mirror 6: 0x7bd4b786 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x427ad77f Mirror 7: 0x427ad77f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000301 1781818068.918625 punch from 0x03600f to 0x03b6b9 (0x56ab bytes) 000302 1781818068.930415 mapwrite 0x035a43 thru 0x03ffff (0xa5bd bytes) 000303 1781818068.944270 mapread 0x0180f7 thru 0x01a05a (0x1f64 bytes) 000304 1781818068.947509 punch from 0x0178e4 to 0x017909 (0x26 bytes) 000305 1781818068.949327 fallocate 0x02758a thru 0x03609b (0xeb12 bytes) 000306 1781818068.951267 write 0x014d24 thru 0x01b19d (0x647a bytes) 000307 1781818068.955399 MIRROR_RESYNC 5 mirrors 000308 1781818069.329148 punch from 0x00260a to 0x00a054 (0x7a4b bytes) 000309 1781818069.337163 write 0x01121e thru 0x012c52 (0x1a35 bytes) 000310 1781818069.340277 fallocate 0x007f7e thru 0x00c0ab (0x412e bytes) 000311 1781818069.342158 trunc from 0x040000 to 0x01a596 (0x25a69 bytes) 000312 1781818069.347188 trunc from 0x019174 to 0x01a596 (0x1423 bytes) 000313 1781818069.348662 trunc from 0x01a597 to 0x03691b (0x1c385 bytes) 000314 1781818069.353417 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3691c): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3691c): Mirror 3: 0x60354acc (stale) Mirror 4: 0x60354acc (stale) Mirror 6: 0x60354acc (stale) Mirror 7: 0x60354acc (stale) CRC-32 checksum value for chunk [0, 0x3691c): Mirror 1: 0x302c6c53 Mirror 8: 0x302c6c53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3691c: skipped 000315 1781818069.585547 mapread 0x032c80 thru 0x035c02 (0x2f83 bytes) 000316 1781818069.593949 trunc from 0x03038a to 0x03691b (0x6592 bytes) 000317 1781818069.604899 punch from 0x02720b to 0x031b86 (0xa97c bytes) 000318 1781818069.606477 trunc from 0x00bf95 to 0x015c86 (0x9cf2 bytes) 000319 1781818069.608133 trunc from 0x0084b7 to 0x00e8a6 (0x63f0 bytes) 000321 1781818069.610062 trunc from 0x0318f5 to 0x03691b (0x5027 bytes) 000322 1781818069.611326 fallocate 0x00b3aa thru 0x019087 (0xdcde bytes) 000324 1781818069.613414 mapread 0x031378 thru 0x03691b (0x55a4 bytes) 000327 1781818069.617296 trunc from 0x03691c to 0x02db13 (0x8e08 bytes) 000328 1781818069.625155 MIRROR_RESYNC 6 mirrors 000329 1781818069.899879 write 0x018887 thru 0x01efcf (0x6749 bytes) 000330 1781818069.908941 mapwrite 0x00ace9 thru 0x0170bf (0xc3d7 bytes) 000331 1781818069.922284 punch from 0x013f71 to 0x0173d8 (0x3468 bytes) 000332 1781818069.923939 punch from 0x016b0f to 0x018186 (0x1678 bytes) 000333 1781818069.925208 write 0x00e390 thru 0x015d41 (0x79b2 bytes) 000334 1781818069.927349 fallocate 0x030bbb thru 0x0396d7 (0x8b1d bytes) 000335 1781818069.929383 mapread 0x03724a thru 0x0396d7 (0x248e bytes) 000338 1781818069.934361 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x396d8): Mirror 4: 0x5ffd1331 (stale) Mirror 6: 0x5ffd1331 (stale) Mirror 7: 0x5ffd1331 (stale) Mirror 8: 0x5ffd1331 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x396d8): Mirror 4: 0x5ffd1331 (stale) Mirror 6: 0x5ffd1331 (stale) Mirror 7: 0x5ffd1331 (stale) Mirror 8: 0x5ffd1331 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x396d8: skipped 000339 1781818070.038103 punch from 0x01ac52 to 0x023195 (0x8544 bytes) 000340 1781818070.040040 trunc from 0x0143ae to 0x021d6d (0xd9c0 bytes) 000341 1781818070.041607 punch from 0x016af6 to 0x0237b4 (0xccbf bytes) 000343 1781818070.043437 mapwrite 0x017d61 thru 0x021fba (0xa25a bytes) 000344 1781818070.056677 punch from 0x00f11f to 0x011c83 (0x2b65 bytes) 000345 1781818070.060121 mapread 0x008d85 thru 0x0164e4 (0xd760 bytes) 000346 1781818070.066413 MIRROR_RESYNC 5 mirrors 000347 1781818070.402836 trunc from 0x0099f8 to 0x0192f0 (0xf8f9 bytes) 000348 1781818070.412999 read 0x02480b thru 0x032660 (0xde56 bytes) 000349 1781818070.417149 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x396d8): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x396d8): Mirror 4: 0xc273e028 (stale) Mirror 6: 0xc273e028 (stale) Mirror 7: 0xc273e028 (stale) Mirror 8: 0xc273e028 (stale) CRC-32 checksum value for chunk [0, 0x396d8): Mirror 1: 0x6813d240 Mirror 9: 0x6813d240 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x396d8: skipped 000350 1781818070.683678 mapwrite 0x03935b thru 0x03c23a (0x2ee0 bytes) 000351 1781818070.710451 mapwrite 0x035e1f thru 0x03ffff (0xa1e1 bytes) 000352 1781818070.725204 mapwrite 0x00b694 thru 0x016849 (0xb1b6 bytes) 000353 1781818070.739971 read 0x010513 thru 0x012060 (0x1b4e bytes) 000354 1781818070.740648 fallocate 0x01cfc2 thru 0x020317 (0x3356 bytes) 000355 1781818070.744516 MIRROR_RESYNC 6 mirrors 000356 1781818071.123514 punch from 0x022833 to 0x02e63d (0xbe0b bytes) 000357 1781818071.131640 trunc from 0x039ccd to 0x03a0c5 (0x3f9 bytes) 000358 1781818071.133073 mapwrite 0x01fdef thru 0x028d3e (0x8f50 bytes) 000359 1781818071.143867 trunc from 0x040000 to 0x02e21f (0x11de0 bytes) 000360 1781818071.149790 trunc from 0x00daab to 0x017fc9 (0xa51f bytes) 000362 1781818071.153859 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e220): Mirror 6: 0xcf1ac835 (stale) Mirror 7: 0xcf1ac835 (stale) Mirror 8: 0xcf1ac835 (stale) Mirror 9: 0xcf1ac835 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e220): Mirror 6: 0xcf1ac835 (stale) Mirror 7: 0xcf1ac835 (stale) Mirror 8: 0xcf1ac835 (stale) Mirror 9: 0xcf1ac835 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e220: skipped 000363 1781818071.263439 mapwrite 0x02e900 thru 0x0380f0 (0x97f1 bytes) 000364 1781818071.280022 fallocate 0x004e44 thru 0x00d3a9 (0x8566 bytes) 000365 1781818071.282217 mapread 0x001be9 thru 0x009650 (0x7a68 bytes) 000367 1781818071.288887 MIRROR_RESYNC 5 mirrors 000368 1781818071.601800 fallocate 0x00de22 thru 0x01d6b4 (0xf893 bytes) 000369 1781818071.612012 read 0x02993b thru 0x02ea68 (0x512e bytes) 000370 1781818071.615170 mapwrite 0x02733c thru 0x03090b (0x95d0 bytes) 000372 1781818071.632541 trunc from 0x01872a to 0x02053b (0x7e12 bytes) 000373 1781818071.634460 mapread 0x023385 thru 0x031b1b (0xe797 bytes) 000374 1781818071.639795 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x380f1): Mirror 6: 0x2713395e (stale) Mirror 7: 0x2713395e (stale) Mirror 8: 0x2713395e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x380f1): Mirror 6: 0x2713395e (stale) Mirror 7: 0x2713395e (stale) Mirror 8: 0x2713395e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x380f1: skipped 000375 1781818071.732456 trunc from 0x0380f1 to 0x034b79 (0x3577 bytes) 000376 1781818071.742503 read 0x02e7f9 thru 0x034b79 (0x6381 bytes) 000378 1781818071.745804 mapwrite 0x01f649 thru 0x02881f (0x91d7 bytes) 000379 1781818071.756227 write 0x03616d thru 0x03ffff (0x9e93 bytes) 000381 1781818071.757756 write 0x031b4e thru 0x03ed47 (0xd1fa bytes) 000382 1781818071.758727 trunc from 0x040000 to 0x02d212 (0x12ded bytes) 000383 1781818071.763542 trunc from 0x00a93e to 0x00bf41 (0x1604 bytes) 000385 1781818071.767300 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2d213): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d213): Mirror 6: 0xa44cfe2e (stale) Mirror 7: 0xa44cfe2e (stale) Mirror 8: 0xa44cfe2e (stale) CRC-32 checksum value for chunk [0, 0x2d213): Mirror 1: 0x4a56bf67 Mirror 9: 0x4a56bf67 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d213: skipped 000386 1781818071.998656 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2d213): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d213): Mirror 6: 0xa44cfe2e (stale) Mirror 7: 0xa44cfe2e (stale) Mirror 8: 0xa44cfe2e (stale) CRC-32 checksum value for chunk [0, 0x2d213): Mirror 1: 0x4a56bf67 Mirror 9: 0x4a56bf67 Mirror 10: 0x4a56bf67 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d213: skipped 000387 1781818072.240589 punch from 0x029369 to 0x02c9e3 (0x367b bytes) 000389 1781818072.251105 mapwrite 0x003fcd thru 0x013455 (0xf489 bytes) 000390 1781818072.269112 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2d213): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d213): Mirror 6: 0xa44cfe2e (stale) Mirror 7: 0xa44cfe2e (stale) Mirror 8: 0xa44cfe2e (stale) Mirror 9: 0x4a56bf67 (stale) Mirror 10: 0x4a56bf67 (stale) CRC-32 checksum value for chunk [0, 0x2d213): Mirror 1: 0x65055400 Mirror 11: 0x65055400 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d213: skipped 000391 1781818072.518177 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d213): Mirror 6: 0xa44cfe2e (stale) Mirror 7: 0xa44cfe2e (stale) Mirror 8: 0xa44cfe2e (stale) Mirror 9: 0x4a56bf67 (stale) Mirror 10: 0x4a56bf67 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d213): Mirror 6: 0xa44cfe2e (stale) Mirror 7: 0xa44cfe2e (stale) Mirror 8: 0xa44cfe2e (stale) Mirror 9: 0x4a56bf67 (stale) Mirror 10: 0x4a56bf67 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d213: skipped 000392 1781818072.636192 mapread 0x026d6c thru 0x02bad7 (0x4d6c bytes) 000393 1781818072.643004 trunc from 0x01fc69 to 0x029184 (0x951c bytes) 000394 1781818072.650007 punch from 0x02ae6e to 0x02d212 (0x23a5 bytes) 000395 1781818072.652221 fallocate 0x032fe4 thru 0x03bde8 (0x8e05 bytes) 000397 1781818072.655010 punch from 0x028a67 to 0x02d212 (0x47ac bytes) 000398 1781818072.657412 trunc from 0x01800e to 0x01bfae (0x3fa1 bytes) 000400 1781818072.659927 mapwrite 0x035d03 thru 0x037c32 (0x1f30 bytes) 000401 1781818072.679984 trunc from 0x037c33 to 0x02c448 (0xb7ea bytes) 000402 1781818072.684623 read 0x02864d thru 0x02c448 (0x3dfc bytes) 000403 1781818072.687755 mapwrite 0x0080f1 thru 0x00eb46 (0x6a56 bytes) 000405 1781818072.698534 mapwrite 0x034ff0 thru 0x03ffff (0xb010 bytes) 000406 1781818072.716582 mapread 0x038ebd thru 0x03ffff (0x7143 bytes) 000409 1781818072.718141 write 0x018e91 thru 0x01b5a0 (0x2710 bytes) 000410 1781818072.721465 punch from 0x031cd6 to 0x039c13 (0x7f3e bytes) 000411 1781818072.723796 trunc from 0x02e233 to 0x030a70 (0x283e bytes) 000412 1781818072.725924 write 0x01a659 thru 0x02611a (0xbac2 bytes) 000413 1781818072.728626 read 0x023f40 thru 0x02c886 (0x8947 bytes) 000414 1781818072.731801 punch from 0x01071a to 0x015c4d (0x5534 bytes) 000415 1781818072.733644 mapwrite 0x0178e6 thru 0x02419c (0xc8b7 bytes) 000416 1781818072.742788 mapread 0x0233b6 thru 0x025bda (0x2825 bytes) 000417 1781818072.743749 trunc from 0x0173cf to 0x019e43 (0x2a75 bytes) 000418 1781818072.745982 trunc from 0x019a7c to 0x028bd8 (0xf15d bytes) 000419 1781818072.747934 trunc from 0x021e35 to 0x02c09c (0xa268 bytes) 000420 1781818072.749649 mapread 0x0285fc thru 0x03001e (0x7a23 bytes) 000421 1781818072.756330 mapread 0x0257aa thru 0x02dab0 (0x8307 bytes) 000422 1781818072.761536 fallocate 0x01642f thru 0x01ef80 (0x8b52 bytes) 000423 1781818072.763663 trunc from 0x040000 to 0x02c030 (0x13fcf bytes) 000424 1781818072.769375 punch from 0x01b3e2 to 0x02733d (0xbf5c bytes) 000425 1781818072.771728 punch from 0x00c307 to 0x00eafc (0x27f6 bytes) 000426 1781818072.773910 punch from 0x01395b to 0x01a443 (0x6ae9 bytes) 000427 1781818072.775806 fallocate 0x0245e4 thru 0x02715f (0x2b7c bytes) 000428 1781818072.777947 fallocate 0x030aa8 thru 0x0314be (0xa17 bytes) 000429 1781818072.780183 trunc from 0x0314bf to 0x02a4e4 (0x6fda bytes) 000430 1781818072.788030 write 0x009bc1 thru 0x00a980 (0xdc0 bytes) 000432 1781818072.789351 write 0x02a561 thru 0x033b50 (0x95f0 bytes) 000433 1781818072.792776 MIRROR_RESYNC 6 mirrors 000434 1781818073.190009 punch from 0x01b60c to 0x023541 (0x7f36 bytes) 000435 1781818073.198203 write 0x00bd44 thru 0x0190ad (0xd36a bytes) 000436 1781818073.201477 fallocate 0x0146ec thru 0x01b9f5 (0x730a bytes) 000437 1781818073.203440 punch from 0x01b520 to 0x0263a7 (0xae88 bytes) 000438 1781818073.205104 mapwrite 0x014512 thru 0x019896 (0x5385 bytes) 000439 1781818073.211506 mapread 0x0142ed thru 0x019ade (0x57f2 bytes) 000440 1781818073.212586 trunc from 0x033b51 to 0x0214a9 (0x126a7 bytes) 000442 1781818073.219268 mapread 0x020edd thru 0x0214a9 (0x5cd bytes) 000443 1781818073.224470 read 0x008d95 thru 0x012e2f (0xa09b bytes) 000444 1781818073.228493 MIRROR_RESYNC 6 mirrors 000445 1781818073.516104 write 0x030324 thru 0x0380d2 (0x7daf bytes) 000446 1781818073.527431 write 0x03937b thru 0x03fc6c (0x68f2 bytes) 000447 1781818073.528648 read 0x035d46 thru 0x037000 (0x12bb bytes) 000448 1781818073.529181 fallocate 0x027d6b thru 0x03274d (0xa9e3 bytes) 000449 1781818073.533204 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fc6d): Mirror 6: 0xd09e6b8e (stale) Mirror 8: 0xd09e6b8e (stale) Mirror 9: 0xd09e6b8e (stale) Mirror 10: 0xd09e6b8e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc6d): Mirror 6: 0xd09e6b8e (stale) Mirror 8: 0xd09e6b8e (stale) Mirror 9: 0xd09e6b8e (stale) Mirror 10: 0xd09e6b8e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc6d: skipped 000450 1781818073.649142 MIRROR_RESYNC 5 mirrors 000452 1781818073.939396 trunc from 0x03fc6d to 0x03de85 (0x1de7 bytes) 000453 1781818073.955033 write 0x0316d0 thru 0x038f39 (0x786a bytes) 000454 1781818073.958423 read 0x01dfdf thru 0x0267e1 (0x8803 bytes) 000455 1781818073.961257 mapread 0x00d640 thru 0x01d21c (0xfbdd bytes) 000456 1781818073.968862 write 0x02e742 thru 0x034fa3 (0x6862 bytes) 000457 1781818073.970812 trunc from 0x00574a to 0x012eb3 (0xd76a bytes) 000458 1781818073.972575 trunc from 0x00a4a1 to 0x00c2c3 (0x1e23 bytes) 000459 1781818073.974089 read 0x0248aa thru 0x02f039 (0xa790 bytes) 000460 1781818073.977349 punch from 0x001233 to 0x004576 (0x3344 bytes) 000461 1781818073.979293 mapread 0x02798c thru 0x02c909 (0x4f7e bytes) 000462 1781818073.980118 write 0x006414 thru 0x0079e6 (0x15d3 bytes) 000464 1781818073.981375 mapread 0x01f739 thru 0x02f656 (0xff1e bytes) 000465 1781818073.982607 trunc from 0x00be80 to 0x01a10d (0xe28e bytes) 000466 1781818073.984671 write 0x03695e thru 0x03ffff (0x96a2 bytes) 000467 1781818073.985532 punch from 0x0375f4 to 0x03ed2e (0x773b bytes) 000468 1781818073.991864 mapwrite 0x0326f9 thru 0x03c5d4 (0x9edc bytes) 000470 1781818074.010246 read 0x03cdb1 thru 0x03ffff (0x324f bytes) 000471 1781818074.013959 trunc from 0x040000 to 0x021fca (0x1e035 bytes) 000472 1781818074.020358 read 0x006fa8 thru 0x00bfef (0x5048 bytes) 000473 1781818074.020888 trunc from 0x021fcb to 0x02d311 (0xb347 bytes) 000474 1781818074.025591 mapread 0x0117be thru 0x01c106 (0xa949 bytes) 000475 1781818074.026522 trunc from 0x025bba to 0x02813d (0x2584 bytes) 000476 1781818074.028234 trunc from 0x004851 to 0x00d728 (0x8ed8 bytes) 000477 1781818074.031751 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2d312): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d312): Mirror 6: 0x12424e91 (stale) Mirror 8: 0x12424e91 (stale) Mirror 9: 0x12424e91 (stale) Mirror 10: 0x12424e91 (stale) CRC-32 checksum value for chunk [0, 0x2d312): Mirror 1: 0x3b11a7a8 Mirror 11: 0x3b11a7a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d312: skipped 000478 1781818074.323469 mapread 0x02a2c4 thru 0x02d311 (0x304e bytes) 000479 1781818074.333193 fallocate 0x03e234 thru 0x03ffff (0x1dcc bytes) 000480 1781818074.345169 write 0x038a07 thru 0x03ffff (0x75f9 bytes) 000482 1781818074.349215 trunc from 0x040000 to 0x029d5e (0x162a1 bytes) 000483 1781818074.365511 fallocate 0x002fd3 thru 0x0059c3 (0x29f1 bytes) 000484 1781818074.368389 punch from 0x0116c6 to 0x01714d (0x5a88 bytes) 000485 1781818074.370601 fallocate 0x0058e9 thru 0x011ff2 (0xc70a bytes) 000486 1781818074.372771 read 0x00ce53 thru 0x00cfa7 (0x155 bytes) 000487 1781818074.374919 read 0x00c5c5 thru 0x010f21 (0x495d bytes) 000488 1781818074.378510 trunc from 0x00fcbd to 0x01a7ec (0xab30 bytes) 000489 1781818074.380589 trunc from 0x003c68 to 0x012cb8 (0xf051 bytes) 000490 1781818074.383951 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x29d5f): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29d5f): Mirror 6: 0xdef7a6fb (stale) Mirror 8: 0xdef7a6fb (stale) Mirror 9: 0xdef7a6fb (stale) Mirror 10: 0xdef7a6fb (stale) Mirror 11: 0x93761b90 (stale) CRC-32 checksum value for chunk [0, 0x29d5f): Mirror 1: 0x8b2af8db Mirror 12: 0x8b2af8db lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29d5f: skipped 000491 1781818074.652498 mapread 0x01bbfa thru 0x01d804 (0x1c0b bytes) 000492 1781818074.660900 write 0x030ec5 thru 0x03c314 (0xb450 bytes) 000493 1781818074.672196 write 0x02295a thru 0x029f22 (0x75c9 bytes) 000494 1781818074.675223 mapread 0x006fab thru 0x0164a2 (0xf4f8 bytes) 000495 1781818074.680802 write 0x01ff02 thru 0x02b34b (0xb44a bytes) 000496 1781818074.683222 mapwrite 0x03fa7b thru 0x03ffff (0x585 bytes) 000497 1781818074.700215 mapwrite 0x00330c thru 0x004a97 (0x178c bytes) 000500 1781818074.705811 trunc from 0x0115cd to 0x01b1b6 (0x9bea bytes) 000501 1781818074.707737 mapread 0x0385f1 thru 0x03ffff (0x7a0f bytes) 000502 1781818074.708755 punch from 0x03efff to 0x03ffff (0x1001 bytes) 000503 1781818074.710535 read 0x036583 thru 0x039433 (0x2eb1 bytes) 000504 1781818074.711193 read 0x025a8f thru 0x02a0f0 (0x4662 bytes) 000505 1781818074.711844 punch from 0x02ed16 to 0x038979 (0x9c64 bytes) 000506 1781818074.713972 fallocate 0x023271 thru 0x03286c (0xf5fc bytes) 000507 1781818074.716017 mapwrite 0x002ee3 thru 0x009bd2 (0x6cf0 bytes) 000508 1781818074.723285 write 0x00ec68 thru 0x0117a4 (0x2b3d bytes) 000509 1781818074.726192 mapread 0x023752 thru 0x030478 (0xcd27 bytes) 000510 1781818074.730615 read 0x01fe53 thru 0x02dac6 (0xdc74 bytes) 000511 1781818074.731159 write 0x01631e thru 0x0235cb (0xd2ae bytes) 000512 1781818074.732229 punch from 0x038ea9 to 0x03f26c (0x63c4 bytes) 000513 1781818074.734414 trunc from 0x020045 to 0x02c29c (0xc258 bytes) 000514 1781818074.739939 mapwrite 0x039fd2 thru 0x03ffff (0x602e bytes) 000515 1781818074.749480 read 0x03ee5e thru 0x03ffff (0x11a2 bytes) 000516 1781818074.750038 trunc from 0x040000 to 0x011ffc (0x2e003 bytes) 000518 1781818074.756505 trunc from 0x00920e to 0x011ffc (0x8def bytes) 000520 1781818074.760054 write 0x028977 thru 0x02c298 (0x3922 bytes) 000521 1781818074.760970 trunc from 0x02c299 to 0x02bafb (0x79d bytes) 000522 1781818074.766892 punch from 0x008c9d to 0x00d125 (0x4489 bytes) 000523 1781818074.769264 read 0x027ca6 thru 0x02bafb (0x3e56 bytes) 000524 1781818074.773284 mapread 0x004420 thru 0x006337 (0x1f18 bytes) 000525 1781818074.774046 write 0x01884e thru 0x0242d9 (0xba8c bytes) 000527 1781818074.775539 write 0x004f07 thru 0x007539 (0x2633 bytes) 000528 1781818074.776275 mapread 0x0070d2 thru 0x00fff1 (0x8f20 bytes) 000529 1781818074.777255 trunc from 0x020b6e to 0x022b07 (0x1f9a bytes) 000530 1781818074.782353 fallocate 0x010f6f thru 0x0161f5 (0x5287 bytes) 000531 1781818074.784253 read 0x018433 thru 0x02507b (0xcc49 bytes) 000532 1781818074.786548 mapwrite 0x014c68 thru 0x014d8d (0x126 bytes) 000533 1781818074.792312 mapwrite 0x013612 thru 0x01a316 (0x6d05 bytes) 000534 1781818074.800188 punch from 0x02007f to 0x027991 (0x7913 bytes) 000535 1781818074.802432 trunc from 0x02bafc to 0x01a890 (0x1126b bytes) 000536 1781818074.807994 trunc from 0x00d543 to 0x015280 (0x7d3e bytes) 000537 1781818074.809992 trunc from 0x012361 to 0x01a890 (0x8530 bytes) 000538 1781818074.811943 write 0x00970b thru 0x0140d7 (0xa9cd bytes) 000539 1781818074.814946 trunc from 0x00d3c5 to 0x019a30 (0xc66c bytes) 000540 1781818074.820392 fallocate 0x01c93c thru 0x0240eb (0x77b0 bytes) 000541 1781818074.822955 read 0x009f27 thru 0x00a41e (0x4f8 bytes) 000542 1781818074.825303 MIRROR_RESYNC 7 mirrors 000543 1781818075.290835 write 0x03911e thru 0x03ffff (0x6ee2 bytes) 000544 1781818075.303790 trunc from 0x040000 to 0x014ae7 (0x2b518 bytes) 000545 1781818075.311174 mapwrite 0x005714 thru 0x00dbce (0x84bb bytes) 000546 1781818075.327445 trunc from 0x00509f to 0x01006a (0xafcc bytes) 000547 1781818075.330372 mapread 0x009ce0 thru 0x01010d (0x642e bytes) 000548 1781818075.334510 fallocate 0x024374 thru 0x029568 (0x51f5 bytes) 000549 1781818075.340122 MIRROR_RESYNC 7 mirrors 000550 1781818075.772712 punch from 0x008a0f to 0x00a784 (0x1d76 bytes) 000551 1781818075.785780 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x14ae8): Mirror 8: 0x90884fdb (stale) Mirror 9: 0x90884fdb (stale) Mirror 10: 0x90884fdb (stale) Mirror 11: 0x90884fdb (stale) Mirror 12: 0x90884fdb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14ae8): Mirror 8: 0x90884fdb (stale) Mirror 9: 0x90884fdb (stale) Mirror 10: 0x90884fdb (stale) Mirror 11: 0x90884fdb (stale) Mirror 12: 0x90884fdb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14ae8: skipped 000552 1781818075.908941 mapread 0x002e80 thru 0x00d264 (0xa3e5 bytes) 000553 1781818075.914578 trunc from 0x0047f6 to 0x00828e (0x3a99 bytes) 000554 1781818075.916537 mapread 0x00a527 thru 0x00cb79 (0x2653 bytes) 000555 1781818075.917462 punch from 0x00e43f to 0x014ae7 (0x66a9 bytes) 000556 1781818075.919427 mapwrite 0x02bbed thru 0x031c64 (0x6078 bytes) 000557 1781818075.940203 write 0x029321 thru 0x02f3a6 (0x6086 bytes) 000558 1781818075.941149 write 0x034e0c thru 0x03e10c (0x9301 bytes) 000559 1781818075.942542 write 0x02dcef thru 0x0390a6 (0xb3b8 bytes) 000561 1781818075.943968 punch from 0x03c49d to 0x03e10c (0x1c70 bytes) 000562 1781818075.952789 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e10d): Mirror 9: 0x512ff6c4 (stale) Mirror 10: 0x512ff6c4 (stale) Mirror 11: 0x512ff6c4 (stale) Mirror 12: 0x512ff6c4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e10d): Mirror 9: 0x512ff6c4 (stale) Mirror 10: 0x512ff6c4 (stale) Mirror 11: 0x512ff6c4 (stale) Mirror 12: 0x512ff6c4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e10d: skipped 000563 1781818076.110280 punch from 0x02611d to 0x02d73b (0x761f bytes) 000564 1781818076.113197 read 0x023bbd thru 0x025455 (0x1899 bytes) 000565 1781818076.116628 punch from 0x01912c to 0x01c735 (0x360a bytes) 000566 1781818076.120944 MIRROR_RESYNC 5 mirrors 000567 1781818076.547850 mapwrite 0x03beec thru 0x03ffff (0x4114 bytes) 000568 1781818076.575525 mapwrite 0x029932 thru 0x03038a (0x6a59 bytes) 000569 1781818076.587546 trunc from 0x0351d3 to 0x03a0dc (0x4f0a bytes) 000571 1781818076.590805 mapread 0x01924a thru 0x02317e (0x9f35 bytes) 000572 1781818076.595351 write 0x0005d0 thru 0x00eee9 (0xe91a bytes) 000574 1781818076.600080 punch from 0x022d3d to 0x0308f1 (0xdbb5 bytes) 000575 1781818076.602765 mapread 0x03677e thru 0x036b66 (0x3e9 bytes) 000576 1781818076.606968 trunc from 0x00d508 to 0x010a7e (0x3577 bytes) 000577 1781818076.613932 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x9dac02a9 (stale) Mirror 11: 0x9dac02a9 (stale) Mirror 12: 0x9dac02a9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x9dac02a9 (stale) Mirror 11: 0x9dac02a9 (stale) Mirror 12: 0x9dac02a9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000578 1781818076.729973 mapread 0x0250bb thru 0x02e523 (0x9469 bytes) 000579 1781818076.736897 read 0x0356a6 thru 0x03e26b (0x8bc6 bytes) 000580 1781818076.740955 read 0x016530 thru 0x022ea1 (0xc972 bytes) 000581 1781818076.744213 trunc from 0x02ee4b to 0x03151e (0x26d4 bytes) 000582 1781818076.746106 read 0x03f07b thru 0x03ffff (0xf85 bytes) 000583 1781818076.746587 read 0x01f5a9 thru 0x02147d (0x1ed5 bytes) 000584 1781818076.746997 read 0x02bf50 thru 0x02bfba (0x6b bytes) 000585 1781818076.747383 trunc from 0x031686 to 0x038b2f (0x74aa bytes) 000586 1781818076.749368 trunc from 0x00da67 to 0x013820 (0x5dba bytes) 000587 1781818076.751354 trunc from 0x018a96 to 0x01ec62 (0x61cd bytes) 000588 1781818076.753083 mapwrite 0x02b7d8 thru 0x035df9 (0xa622 bytes) 000589 1781818076.763195 fallocate 0x005678 thru 0x011112 (0xba9b bytes) 000590 1781818076.765364 trunc from 0x040000 to 0x0208ba (0x1f745 bytes) 000591 1781818076.771462 mapwrite 0x019064 thru 0x025882 (0xc81f bytes) 000592 1781818076.793466 write 0x00cc66 thru 0x01b614 (0xe9af bytes) 000593 1781818076.794518 mapread 0x00b745 thru 0x017272 (0xbb2e bytes) 000594 1781818076.795360 mapwrite 0x02afeb thru 0x034f87 (0x9f9d bytes) 000595 1781818076.829035 punch from 0x00c259 to 0x0118b3 (0x565b bytes) 000596 1781818076.831368 trunc from 0x034f88 to 0x0058e2 (0x2f6a5 bytes) 000597 1781818076.836829 write 0x03714c thru 0x0383e3 (0x1298 bytes) 000598 1781818076.839582 MIRROR_RESYNC 4 mirrors 000599 1781818077.081522 trunc from 0x00602c to 0x006449 (0x41e bytes) 000600 1781818077.097032 punch from 0x01ba7e to 0x024729 (0x8cac bytes) 000601 1781818077.101496 MIRROR_RESYNC 4 mirrors 000602 1781818077.361751 mapread 0x023b43 thru 0x02c0c9 (0x8587 bytes) 000603 1781818077.369436 MIRROR_EXTEND to 5 mirrors 000605 1781818077.583825 mapread 0x034076 thru 0x0383e3 (0x436e bytes) 000606 1781818077.593298 trunc from 0x0383e4 to 0x0137a0 (0x24c43 bytes) 000607 1781818077.607454 write 0x0171a1 thru 0x01ef98 (0x7df8 bytes) 000608 1781818077.608814 read 0x016bed thru 0x01a050 (0x3464 bytes) 000609 1781818077.610967 trunc from 0x01ef99 to 0x00695b (0x1863d bytes) 000610 1781818077.618295 write 0x030aac thru 0x037f34 (0x7489 bytes) 000611 1781818077.619733 trunc from 0x037f35 to 0x0070bd (0x30e77 bytes) 000612 1781818077.626136 punch from 0x00161e to 0x0070bd (0x5aa0 bytes) 000613 1781818077.628470 mapread 0x0012a2 thru 0x0066fa (0x5459 bytes) 000614 1781818077.634732 MIRROR_RESYNC 5 mirrors 000615 1781818077.860094 write 0x0370fe thru 0x03ffff (0x8f02 bytes) 000616 1781818077.870686 fallocate 0x00689a thru 0x00f84f (0x8fb6 bytes) 000618 1781818077.873400 trunc from 0x040000 to 0x00e615 (0x319ea bytes) 000619 1781818077.887613 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0xe616): Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe616): Mirror 10: 0xe69ae641 (stale) Mirror 11: 0xe69ae641 (stale) Mirror 12: 0xe69ae641 (stale) Mirror 13: 0xe69ae641 (stale) CRC-32 checksum value for chunk [0, 0xe616): Mirror 1: 0xe69ae641 Mirror 14: 0xe69ae641 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe616: skipped 000620 1781818078.132770 mapread 0x00db6b thru 0x00e615 (0xaab bytes) 000622 1781818078.144023 write 0x01d22a thru 0x026c5d (0x9a34 bytes) 000623 1781818078.158294 read 0x0177fa thru 0x01810a (0x911 bytes) 000625 1781818078.160850 write 0x01fa67 thru 0x02040b (0x9a5 bytes) 000626 1781818078.161432 trunc from 0x026c5e to 0x0338ca (0xcc6d bytes) 000627 1781818078.172221 mapwrite 0x007701 thru 0x00dce5 (0x65e5 bytes) 000628 1781818078.184035 trunc from 0x005c5f to 0x007a39 (0x1ddb bytes) 000629 1781818078.186108 punch from 0x007b66 to 0x008177 (0x612 bytes) 000630 1781818078.189523 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x338cb): Mirror 10: 0x40461aa6 (stale) Mirror 11: 0x40461aa6 (stale) Mirror 12: 0x40461aa6 (stale) Mirror 13: 0x40461aa6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x338cb): Mirror 10: 0x40461aa6 (stale) Mirror 11: 0x40461aa6 (stale) Mirror 12: 0x40461aa6 (stale) Mirror 13: 0x40461aa6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x338cb: skipped 000631 1781818078.296331 punch from 0x000522 to 0x006b10 (0x65ef bytes) 000632 1781818078.298224 mapwrite 0x038f70 thru 0x0393c1 (0x452 bytes) 000633 1781818078.310908 write 0x01bee8 thru 0x023ec8 (0x7fe1 bytes) 000634 1781818078.314221 trunc from 0x0377ba to 0x0393c1 (0x1c08 bytes) 000635 1781818078.316203 fallocate 0x0057be thru 0x00d6fc (0x7f3f bytes) 000636 1781818078.318252 read 0x008914 thru 0x016aff (0xe1ec bytes) 000637 1781818078.321425 punch from 0x025834 to 0x02d7f6 (0x7fc3 bytes) 000639 1781818078.323413 trunc from 0x0393c2 to 0x03ae77 (0x1ab6 bytes) 000640 1781818078.327926 punch from 0x00c0df to 0x017913 (0xb835 bytes) 000641 1781818078.330061 fallocate 0x01ca02 thru 0x021be5 (0x51e4 bytes) 000642 1781818078.332303 trunc from 0x01eb93 to 0x022cac (0x411a bytes) 000643 1781818078.336630 write 0x0090d6 thru 0x00e099 (0x4fc4 bytes) 000644 1781818078.338821 fallocate 0x03f514 thru 0x03ffff (0xaec bytes) 000646 1781818078.341361 mapwrite 0x010789 thru 0x015067 (0x48df bytes) 000647 1781818078.350403 read 0x02bc9f thru 0x038a76 (0xcdd8 bytes) 000649 1781818078.356719 MIRROR_RESYNC 5 mirrors 000650 1781818078.817803 mapread 0x02b044 thru 0x02cc51 (0x1c0e bytes) 000651 1781818078.821225 mapwrite 0x022072 thru 0x026f5d (0x4eec bytes) 000652 1781818078.856265 mapwrite 0x03b60f thru 0x03ffff (0x49f1 bytes) 000653 1781818078.877710 read 0x035a94 thru 0x03e035 (0x85a2 bytes) 000654 1781818078.880308 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x676beb5d (stale) Mirror 11: 0x676beb5d (stale) Mirror 13: 0x676beb5d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 10: 0x676beb5d (stale) Mirror 11: 0x676beb5d (stale) Mirror 13: 0x676beb5d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000655 1781818078.996789 trunc from 0x040000 to 0x00fa1f (0x305e0 bytes) 000656 1781818079.002313 read 0x006604 thru 0x006733 (0x130 bytes) 000658 1781818079.004808 trunc from 0x00fa20 to 0x02e1e0 (0x1e7c1 bytes) 000659 1781818079.009110 trunc from 0x02e1e1 to 0x033ffa (0x5e1a bytes) 000661 1781818079.013318 write 0x02143b thru 0x02acc8 (0x988e bytes) 000662 1781818079.016215 mapread 0x00395c thru 0x00488b (0xf30 bytes) 000664 1781818079.019882 mapwrite 0x00f573 thru 0x01a8a9 (0xb337 bytes) 000665 1781818079.032775 write 0x0001b9 thru 0x00f5dd (0xf425 bytes) 000666 1781818079.035766 punch from 0x02ef71 to 0x033ffa (0x508a bytes) 000667 1781818079.037979 mapwrite 0x03786f thru 0x03eb87 (0x7319 bytes) 000668 1781818079.053887 mapread 0x03966e thru 0x03eb87 (0x551a bytes) 000669 1781818079.054707 trunc from 0x02b4ab to 0x034768 (0x92be bytes) 000670 1781818079.056956 write 0x028b4b thru 0x030f67 (0x841d bytes) 000672 1781818079.059926 mapread 0x01b0b6 thru 0x029c8f (0xebda bytes) 000673 1781818079.060798 write 0x029d95 thru 0x038c04 (0xee70 bytes) 000674 1781818079.061875 mapread 0x02a5e6 thru 0x039a49 (0xf464 bytes) 000675 1781818079.062671 write 0x02b285 thru 0x035eb2 (0xac2e bytes) 000676 1781818079.063607 write 0x0129ed thru 0x014759 (0x1d6d bytes) 000677 1781818079.064307 mapread 0x022127 thru 0x02c705 (0xa5df bytes) 000678 1781818079.065331 mapread 0x004a5c thru 0x00eb92 (0xa137 bytes) 000679 1781818079.066391 mapwrite 0x03e703 thru 0x03ffff (0x18fd bytes) 000680 1781818079.078245 write 0x02fc03 thru 0x037971 (0x7d6f bytes) 000681 1781818079.079259 read 0x028f54 thru 0x02ad85 (0x1e32 bytes) 000682 1781818079.079909 mapread 0x0046a4 thru 0x0056eb (0x1048 bytes) 000683 1781818079.080858 punch from 0x02a20d to 0x032efb (0x8cef bytes) 000685 1781818079.086274 write 0x011a05 thru 0x01bfae (0xa5aa bytes) 000688 1781818079.088753 write 0x01351a thru 0x01bb29 (0x8610 bytes) 000689 1781818079.089774 trunc from 0x01d68b to 0x0224b6 (0x4e2c bytes) 000690 1781818079.094431 MIRROR_RESYNC 4 mirrors 000691 1781818079.350287 MIRROR_RESYNC 4 mirrors 000692 1781818079.479921 trunc from 0x01d849 to 0x025029 (0x77e1 bytes) 000694 1781818079.492823 trunc from 0x02af33 to 0x02d98c (0x2a5a bytes) 000695 1781818079.495293 fallocate 0x004a78 thru 0x009557 (0x4ae0 bytes) 000696 1781818079.497520 read 0x018b65 thru 0x01c702 (0x3b9e bytes) 000697 1781818079.501013 trunc from 0x030ab1 to 0x03ffff (0xf54f bytes) 000698 1781818079.503330 read 0x02b527 thru 0x0316d2 (0x61ac bytes) 000699 1781818079.508195 trunc from 0x020a54 to 0x02ec26 (0xe1d3 bytes) 000701 1781818079.511683 mapread 0x000093 thru 0x0072f8 (0x7266 bytes) 000702 1781818079.512738 fallocate 0x02b9a8 thru 0x03092a (0x4f83 bytes) 000703 1781818079.515358 read 0x03190f thru 0x03daa0 (0xc192 bytes) 000704 1781818079.516030 trunc from 0x040000 to 0x0201a5 (0x1fe5a bytes) 000705 1781818079.527464 fallocate 0x01ca73 thru 0x02be09 (0xf397 bytes) 000706 1781818079.529456 trunc from 0x02ac4d to 0x02be09 (0x11bd bytes) 000707 1781818079.531232 mapwrite 0x0074be thru 0x0147d1 (0xd314 bytes) 000708 1781818079.539135 fallocate 0x021ab9 thru 0x02adcf (0x9317 bytes) 000709 1781818079.542873 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2be0a): Mirror 11: 0xedbc8ecb (stale) Mirror 13: 0xedbc8ecb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2be0a): Mirror 11: 0xedbc8ecb (stale) Mirror 13: 0xedbc8ecb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2be0a: skipped 000710 1781818079.630638 read 0x01a571 thru 0x021a49 (0x74d9 bytes) 000711 1781818079.633748 write 0x0038ae thru 0x00c288 (0x89db bytes) 000713 1781818079.639035 trunc from 0x01315f to 0x020778 (0xd61a bytes) 000714 1781818079.641265 trunc from 0x02be0a to 0x01c155 (0xfcb4 bytes) 000715 1781818079.645850 mapread 0x00bb03 thru 0x00f1f1 (0x36ef bytes) 000717 1781818079.651718 mapwrite 0x0112e1 thru 0x019b09 (0x8829 bytes) 000718 1781818079.659925 fallocate 0x0378fc thru 0x038561 (0xc66 bytes) 000719 1781818079.662914 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x38562): Mirror 11: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38562): Mirror 11: 0xae106e96 (stale) Mirror 13: 0xae106e96 (stale) CRC-32 checksum value for chunk [0, 0x38562): Mirror 1: 0x8aea2a71 Mirror 14: 0x8aea2a71 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38562: skipped 000720 1781818079.882851 write 0x00fc52 thru 0x01973b (0x9aea bytes) 000721 1781818079.899802 read 0x01b4ae thru 0x01bf90 (0xae3 bytes) 000722 1781818079.902932 write 0x008b34 thru 0x00f3db (0x68a8 bytes) 000723 1781818079.907869 MIRROR_RESYNC 4 mirrors 000724 1781818080.149239 trunc from 0x038562 to 0x00e24e (0x2a313 bytes) 000725 1781818080.162189 fallocate 0x025c77 thru 0x02d6f6 (0x7a80 bytes) 000727 1781818080.165051 fallocate 0x01c028 thru 0x0280c1 (0xc09a bytes) 000730 1781818080.168416 mapread 0x02a571 thru 0x02cf3b (0x29cb bytes) 000731 1781818080.175250 fallocate 0x03cda4 thru 0x03ffff (0x325c bytes) 000732 1781818080.178069 punch from 0x00e46a to 0x01b00f (0xcba6 bytes) 000733 1781818080.180259 trunc from 0x015905 to 0x021fae (0xc6aa bytes) 000735 1781818080.182545 mapread 0x0208ad thru 0x02f178 (0xe8cc bytes) 000736 1781818080.189106 write 0x0283b2 thru 0x033e8a (0xbad9 bytes) 000737 1781818080.190335 write 0x026a87 thru 0x02baa1 (0x501b bytes) 000738 1781818080.191289 write 0x03f1a6 thru 0x03ffff (0xe5a bytes) 000739 1781818080.193849 mapread 0x00d1c8 thru 0x016365 (0x919e bytes) 000740 1781818080.198888 mapwrite 0x032341 thru 0x036ef5 (0x4bb5 bytes) 000741 1781818080.205975 fallocate 0x0373ec thru 0x03ffff (0x8c14 bytes) 000744 1781818080.209209 mapwrite 0x03c420 thru 0x03ffff (0x3be0 bytes) 000745 1781818080.216998 mapwrite 0x03a941 thru 0x03fd76 (0x5436 bytes) 000746 1781818080.223695 mapread 0x0026d0 thru 0x00d94c (0xb27d bytes) 000748 1781818080.227604 mapread 0x02bb09 thru 0x030b41 (0x5039 bytes) 000749 1781818080.228544 fallocate 0x005f52 thru 0x00e298 (0x8347 bytes) 000750 1781818080.230895 mapwrite 0x00a408 thru 0x01123f (0x6e38 bytes) 000751 1781818080.238086 read 0x01815b thru 0x01e447 (0x62ed bytes) 000752 1781818080.238678 trunc from 0x040000 to 0x009a16 (0x365e9 bytes) 000753 1781818080.245379 trunc from 0x003e89 to 0x009a16 (0x5b8e bytes) 000754 1781818080.247406 write 0x006733 thru 0x00c572 (0x5e40 bytes) 000755 1781818080.249848 trunc from 0x00c573 to 0x03cb1c (0x305aa bytes) 000756 1781818080.254529 fallocate 0x033b5f thru 0x03801b (0x44bd bytes) 000757 1781818080.256801 trunc from 0x03cb1d to 0x036c04 (0x5f18 bytes) 000758 1781818080.261402 fallocate 0x000af2 thru 0x00afc7 (0xa4d6 bytes) 000759 1781818080.263431 trunc from 0x02b983 to 0x036c04 (0xb282 bytes) 000760 1781818080.265340 mapwrite 0x0175aa thru 0x02103f (0x9a96 bytes) 000761 1781818080.280338 fallocate 0x035717 thru 0x03ffff (0xa8e9 bytes) 000763 1781818080.282859 read 0x031111 thru 0x0362a3 (0x5193 bytes) 000764 1781818080.288236 MIRROR_RESYNC 4 mirrors 000766 1781818080.567374 mapread 0x02ddf4 thru 0x0361e2 (0x83ef bytes) 000767 1781818080.574070 mapwrite 0x00070b thru 0x001ea7 (0x179d bytes) 000768 1781818080.611040 trunc from 0x036c05 to 0x02395b (0x132a9 bytes) 000769 1781818080.623058 trunc from 0x011d53 to 0x01e6d8 (0xc986 bytes) 000770 1781818080.624891 punch from 0x01b221 to 0x01fb79 (0x4959 bytes) 000771 1781818080.626398 write 0x027688 thru 0x029e84 (0x27fd bytes) 000772 1781818080.627349 trunc from 0x029e85 to 0x002d59 (0x2712b bytes) 000773 1781818080.634062 MIRROR_RESYNC 4 mirrors 000774 1781818080.823998 punch from 0x001333 to 0x0022e9 (0xfb7 bytes) 000775 1781818080.834619 mapread 0x00197f thru 0x002d59 (0x13db bytes) 000776 1781818080.837955 fallocate 0x02b0b6 thru 0x02fb20 (0x4a6b bytes) 000777 1781818080.839796 trunc from 0x01f610 to 0x02dbda (0xe5cb bytes) 000778 1781818080.841354 trunc from 0x02fb21 to 0x01fffc (0xfb24 bytes) 000779 1781818080.850783 read 0x01feaf thru 0x01fffc (0x14e bytes) 000780 1781818080.852730 mapread 0x00dc43 thru 0x0106bf (0x2a7d bytes) 000781 1781818080.856239 trunc from 0x006272 to 0x013c8d (0xda1c bytes) 000782 1781818080.858105 fallocate 0x027da6 thru 0x032411 (0xa66c bytes) 000783 1781818080.861332 read 0x010976 thru 0x01cf74 (0xc5ff bytes) 000784 1781818080.866091 fallocate 0x03f5fa thru 0x03ffff (0xa06 bytes) 000785 1781818080.868680 read 0x02fbd1 thru 0x032411 (0x2841 bytes) 000786 1781818080.869311 mapwrite 0x00acf5 thru 0x012ac3 (0x7dcf bytes) 000787 1781818080.878274 fallocate 0x013c80 thru 0x01f9e0 (0xbd61 bytes) 000789 1781818080.881318 trunc from 0x0172cf to 0x01feef (0x8c21 bytes) 000790 1781818080.883755 read 0x00aa35 thru 0x00bc50 (0x121c bytes) 000791 1781818080.884180 punch from 0x02a6b7 to 0x02d55a (0x2ea4 bytes) 000792 1781818080.886490 mapread 0x02c229 thru 0x02cc9e (0xa76 bytes) 000793 1781818080.890854 punch from 0x0194b0 to 0x01f9f2 (0x6543 bytes) 000794 1781818080.893346 trunc from 0x0067c0 to 0x012d66 (0xc5a7 bytes) 000795 1781818080.896020 punch from 0x0087de to 0x00e25a (0x5a7d bytes) 000796 1781818080.898300 mapwrite 0x01981c thru 0x021dc0 (0x85a5 bytes) 000797 1781818080.918445 mapread 0x0134fa thru 0x01ca08 (0x950f bytes) 000798 1781818080.919414 fallocate 0x0188f7 thru 0x02786d (0xef77 bytes) 000799 1781818080.922041 punch from 0x01b54c to 0x0234df (0x7f94 bytes) 000800 1781818080.924352 mapwrite 0x016615 thru 0x02482f (0xe21b bytes) 000802 1781818080.946942 trunc from 0x032412 to 0x02a5bb (0x7e56 bytes) 000803 1781818080.952093 trunc from 0x02528c to 0x029cb0 (0x4a25 bytes) 000804 1781818080.954450 mapread 0x0075cb thru 0x009957 (0x238d bytes) 000806 1781818080.960367 punch from 0x002b62 to 0x00ad8f (0x822e bytes) 000807 1781818080.962390 write 0x00d5d4 thru 0x012d43 (0x5770 bytes) 000809 1781818080.966651 trunc from 0x02a5bc to 0x0139a7 (0x16c14 bytes) 000810 1781818080.972550 punch from 0x00ad7e to 0x011171 (0x63f4 bytes) 000812 1781818080.977904 read 0x002ed2 thru 0x006700 (0x382f bytes) 000813 1781818080.981425 punch from 0x007cd5 to 0x012651 (0xa97d bytes) 000814 1781818080.983882 write 0x00891f thru 0x0107a1 (0x7e83 bytes) 000815 1781818080.988149 fallocate 0x01193c thru 0x0186b9 (0x6d7e bytes) 000816 1781818080.990403 punch from 0x009152 to 0x011060 (0x7f0f bytes) 000817 1781818080.995503 mapread 0x0049b2 thru 0x00e77e (0x9dcd bytes) 000818 1781818081.006418 mapwrite 0x01ce09 thru 0x023001 (0x61f9 bytes) 000819 1781818081.021991 mapwrite 0x0336b8 thru 0x0367ee (0x3137 bytes) 000820 1781818081.037420 write 0x0008e0 thru 0x00409e (0x37bf bytes) 000821 1781818081.038496 write 0x0141fd thru 0x01ac84 (0x6a88 bytes) 000822 1781818081.039360 trunc from 0x0367ef to 0x03b907 (0x5119 bytes) 000823 1781818081.044895 read 0x020093 thru 0x02857e (0x84ec bytes) 000824 1781818081.045403 trunc from 0x03b908 to 0x0022dd (0x3962a bytes) 000826 1781818081.051718 mapwrite 0x029e6c thru 0x030659 (0x67ee bytes) 000827 1781818081.071774 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3065a): Mirror 13: 0xba1aa607 (stale) Mirror 14: 0xba1aa607 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3065a): Mirror 13: 0xba1aa607 (stale) Mirror 14: 0xba1aa607 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3065a: skipped 000828 1781818081.155621 write 0x03ffd8 thru 0x03ffff (0x28 bytes) 000829 1781818081.157172 read 0x03da2c thru 0x03ffff (0x25d4 bytes) 000830 1781818081.158910 punch from 0x0127f1 to 0x01802d (0x583d bytes) 000831 1781818081.160652 trunc from 0x040000 to 0x007f01 (0x380fe bytes) 000832 1781818081.165077 trunc from 0x007f02 to 0x01d454 (0x15553 bytes) 000834 1781818081.170331 fallocate 0x027d8f thru 0x02f2c3 (0x7535 bytes) 000835 1781818081.174330 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2f2c4): Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f2c4): Mirror 13: 0x29be318e (stale) Mirror 14: 0x29be318e (stale) CRC-32 checksum value for chunk [0, 0x2f2c4): Mirror 1: 0xfad1bb13 Mirror 15: 0xfad1bb13 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f2c4: skipped 000836 1781818081.409366 trunc from 0x02970e to 0x02f2c3 (0x5bb6 bytes) 000837 1781818081.419594 write 0x007130 thru 0x00d012 (0x5ee3 bytes) 000838 1781818081.424701 fallocate 0x008f72 thru 0x015772 (0xc801 bytes) 000839 1781818081.426672 punch from 0x004ba8 to 0x009aa2 (0x4efb bytes) 000841 1781818081.431312 mapwrite 0x0045d8 thru 0x01140f (0xce38 bytes) 000842 1781818081.445880 mapwrite 0x019013 thru 0x01bd69 (0x2d57 bytes) 000844 1781818081.453233 trunc from 0x02f2c4 to 0x009a97 (0x2582c bytes) 000845 1781818081.463186 mapread 0x007df7 thru 0x009a97 (0x1ca1 bytes) 000846 1781818081.463878 trunc from 0x003461 to 0x009a97 (0x6637 bytes) 000847 1781818081.465894 mapwrite 0x00bca3 thru 0x0146f6 (0x8a54 bytes) 000848 1781818081.482489 trunc from 0x0059e4 to 0x009fa4 (0x45c1 bytes) 000850 1781818081.485047 mapwrite 0x03b133 thru 0x03ffff (0x4ecd bytes) 000851 1781818081.499672 trunc from 0x0321a4 to 0x034ad6 (0x2933 bytes) 000852 1781818081.501691 trunc from 0x0369b8 to 0x03ffff (0x9648 bytes) 000853 1781818081.503720 punch from 0x005b99 to 0x013469 (0xd8d1 bytes) 000854 1781818081.505797 read 0x030b6a thru 0x033814 (0x2cab bytes) 000855 1781818081.508350 punch from 0x014a32 to 0x02242f (0xd9fe bytes) 000856 1781818081.510436 mapread 0x00bb80 thru 0x00fad1 (0x3f52 bytes) 000857 1781818081.515050 mapwrite 0x019a59 thru 0x01f871 (0x5e19 bytes) 000858 1781818081.526932 write 0x039204 thru 0x03ffff (0x6dfc bytes) 000859 1781818081.529894 mapread 0x027ddf thru 0x0353de (0xd600 bytes) 000860 1781818081.536779 fallocate 0x01c2f6 thru 0x024ece (0x8bd9 bytes) 000862 1781818081.539179 punch from 0x0241d2 to 0x02d74f (0x957e bytes) 000863 1781818081.542651 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xd0b3c5c6 (stale) Mirror 14: 0xd0b3c5c6 (stale) Mirror 15: 0xc34d0a5e (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7b2c187c Mirror 16: 0x7b2c187c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000864 1781818081.786443 read 0x023095 thru 0x02e994 (0xb900 bytes) 000865 1781818081.791334 mapread 0x03cd99 thru 0x03ffff (0x3267 bytes) 000866 1781818081.801763 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xd0b3c5c6 (stale) Mirror 14: 0xd0b3c5c6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xd0b3c5c6 (stale) Mirror 14: 0xd0b3c5c6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7b2c187c Mirror 16: 0x7b2c187c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000867 1781818081.982479 trunc from 0x040000 to 0x00722a (0x38dd5 bytes) 000868 1781818082.001803 trunc from 0x00722b to 0x010488 (0x925e bytes) 000870 1781818082.008604 write 0x000287 thru 0x0032fc (0x3076 bytes) 000871 1781818082.012056 trunc from 0x010489 to 0x037761 (0x272d9 bytes) 000873 1781818082.017058 mapread 0x02d344 thru 0x02eb0e (0x17cb bytes) 000874 1781818082.020434 trunc from 0x037762 to 0x008604 (0x2f15d bytes) 000875 1781818082.024842 fallocate 0x021542 thru 0x024c06 (0x36c5 bytes) 000876 1781818082.026957 trunc from 0x00ffc5 to 0x01599e (0x59da bytes) 000877 1781818082.028608 fallocate 0x02e0c1 thru 0x035e28 (0x7d68 bytes) 000880 1781818082.031426 read 0x0220b5 thru 0x02966d (0x75b9 bytes) 000881 1781818082.034244 mapwrite 0x03eb3f thru 0x03ffff (0x14c1 bytes) 000882 1781818082.046438 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 13: 0xd0b3c5c6 (stale) Mirror 14: 0xd0b3c5c6 (stale) Mirror 16: 0x7b2c187c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x43f2654c Mirror 17: 0x43f2654c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000883 1781818082.274289 mapwrite 0x03d63d thru 0x03e14f (0xb13 bytes) 000884 1781818082.313165 punch from 0x039888 to 0x03d8f1 (0x406a bytes) 000886 1781818082.316014 fallocate 0x016b99 thru 0x0181b2 (0x161a bytes) 000887 1781818082.319656 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xd0b3c5c6 (stale) Mirror 16: 0x7b2c187c (stale) Mirror 17: 0x43f2654c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 14: 0xd0b3c5c6 (stale) Mirror 16: 0x7b2c187c (stale) Mirror 17: 0x43f2654c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000889 1781818082.415820 trunc from 0x01cd1b to 0x029b23 (0xce09 bytes) 000890 1781818082.417636 trunc from 0x040000 to 0x038ba0 (0x745f bytes) 000891 1781818082.429354 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38ba1): Mirror 14: 0xb28eed1 (stale) Mirror 17: 0xa977de71 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ba1): Mirror 14: 0xb28eed1 (stale) Mirror 17: 0xa977de71 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ba1: skipped 000892 1781818082.519238 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38ba1): Mirror 17: 0xa977de71 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38ba1): Mirror 17: 0xa977de71 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38ba1: skipped 000893 1781818082.600889 read 0x00c511 thru 0x00d90c (0x13fc bytes) 000894 1781818082.604598 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x615:0x0]: Invalid argument (22) 000895 1781818082.639894 fallocate 0x01d551 thru 0x02d341 (0xfdf1 bytes) 000896 1781818082.641847 mapread 0x02a969 thru 0x03390e (0x8fa6 bytes) 000897 1781818082.647299 write 0x01d47e thru 0x01f28c (0x1e0f bytes) 000898 1781818082.651195 read 0x01fccf thru 0x027608 (0x793a bytes) 000899 1781818082.654065 trunc from 0x00ae6f to 0x018b3a (0xdccc bytes) 000900 1781818082.655814 write 0x016012 thru 0x01a8a8 (0x4897 bytes) 000902 1781818082.658877 mapread 0x01846a thru 0x01c80d (0x43a4 bytes) 000903 1781818082.661549 write 0x037a2c thru 0x03ffff (0x85d4 bytes) 000904 1781818082.663422 read 0x008b32 thru 0x0169c3 (0xde92 bytes) 000905 1781818082.665947 trunc from 0x00ba4e to 0x01b672 (0xfc25 bytes) 000906 1781818082.670057 punch from 0x02180c to 0x02a627 (0x8e1c bytes) 000907 1781818082.671662 fallocate 0x01a145 thru 0x023cf0 (0x9bac bytes) 000908 1781818082.673443 trunc from 0x00fbc2 to 0x01e340 (0xe77f bytes) 000911 1781818082.676697 trunc from 0x040000 to 0x037c05 (0x83fa bytes) 000912 1781818082.681417 read 0x02d263 thru 0x032eee (0x5c8c bytes) 000913 1781818082.682151 trunc from 0x0068a0 to 0x0138e0 (0xd041 bytes) 000914 1781818082.684150 mapwrite 0x00f2d3 thru 0x01a9e2 (0xb710 bytes) 000915 1781818082.694399 mapread 0x01f279 thru 0x02294e (0x36d6 bytes) 000916 1781818082.695439 mapread 0x01cdb3 thru 0x022fbe (0x620c bytes) 000917 1781818082.696474 trunc from 0x02c3d6 to 0x02cfe5 (0xc10 bytes) 000918 1781818082.698595 mapwrite 0x0130be thru 0x014c5f (0x1ba2 bytes) 000919 1781818082.703214 fallocate 0x01277d thru 0x020c4e (0xe4d2 bytes) 000920 1781818082.705508 mapwrite 0x00c146 thru 0x011292 (0x514d bytes) 000921 1781818082.712695 fallocate 0x02f933 thru 0x03184c (0x1f1a bytes) 000922 1781818082.714702 mapwrite 0x015596 thru 0x020952 (0xb3bd bytes) 000924 1781818082.723342 mapwrite 0x03101d thru 0x03f102 (0xe0e6 bytes) 000925 1781818082.739274 trunc from 0x011c82 to 0x01855d (0x68dc bytes) 000926 1781818082.741373 mapwrite 0x0391f8 thru 0x03ffff (0x6e08 bytes) 000928 1781818082.752752 mapwrite 0x013b78 thru 0x02151a (0xd9a3 bytes) 000929 1781818082.768938 fallocate 0x01605f thru 0x0199e9 (0x398b bytes) 000930 1781818082.771066 mapread 0x0001bf thru 0x00106c (0xeae bytes) 000931 1781818082.772031 mapwrite 0x005502 thru 0x010053 (0xab52 bytes) 000932 1781818082.779736 fallocate 0x037a27 thru 0x03ffff (0x85d9 bytes) 000933 1781818082.781688 read 0x034dd3 thru 0x03909d (0x42cb bytes) 000934 1781818082.782277 fallocate 0x00d162 thru 0x0115a0 (0x443f bytes) 000935 1781818082.784068 fallocate 0x022dc3 thru 0x0281b9 (0x53f7 bytes) 000936 1781818082.786158 fallocate 0x011368 thru 0x0139ee (0x2687 bytes) 000937 1781818082.788008 mapread 0x0017a3 thru 0x004e49 (0x36a7 bytes) 000939 1781818082.788930 mapwrite 0x034d3b thru 0x03a196 (0x545c bytes) 000941 1781818082.795984 mapread 0x0306b8 thru 0x0366c3 (0x600c bytes) 000942 1781818082.796743 read 0x035bd4 thru 0x038084 (0x24b1 bytes) 000943 1781818082.797119 read 0x025947 thru 0x026eb0 (0x156a bytes) 000944 1781818082.797500 mapwrite 0x03a02e thru 0x03ffff (0x5fd2 bytes) 000946 1781818082.803889 trunc from 0x040000 to 0x00c1a7 (0x33e58 bytes) 000947 1781818082.808834 trunc from 0x0086a8 to 0x00c1a7 (0x3b00 bytes) 000948 1781818082.810667 trunc from 0x00c1a8 to 0x024e2a (0x18c83 bytes) 000949 1781818082.814847 punch from 0x005ab9 to 0x0066c2 (0xc0a bytes) 000950 1781818082.816700 fallocate 0x00a3f2 thru 0x00b106 (0xd15 bytes) 000951 1781818082.818691 write 0x01483d thru 0x0168e8 (0x20ac bytes) 000952 1781818082.822735 trunc from 0x024e2b to 0x012fc4 (0x11e66 bytes) 000954 1781818082.828181 write 0x02df2e thru 0x0399f8 (0xbacb bytes) 000955 1781818082.829348 fallocate 0x01afb6 thru 0x0211a0 (0x61eb bytes) 000956 1781818082.831328 punch from 0x031055 to 0x0340cd (0x3079 bytes) 000957 1781818082.835670 punch from 0x026963 to 0x02d140 (0x67de bytes) 000958 1781818082.837528 punch from 0x0002fd to 0x002d2e (0x2a32 bytes) 000959 1781818082.839208 write 0x03b855 thru 0x03ffff (0x47ab bytes) 000963 1781818082.842780 MIRROR_EXTEND to 2 mirrors 000964 1781818083.047583 MIRROR_RESYNC 2 mirrors 000965 1781818083.153403 mapwrite 0x0069f5 thru 0x00a652 (0x3c5e bytes) 000966 1781818083.188857 mapwrite 0x014731 thru 0x015537 (0xe07 bytes) 000967 1781818083.196762 fallocate 0x02f8fc thru 0x038567 (0x8c6c bytes) 000968 1781818083.199200 trunc from 0x040000 to 0x02a460 (0x15b9f bytes) 000969 1781818083.210715 write 0x0068b7 thru 0x00cae9 (0x6233 bytes) 000970 1781818083.211809 trunc from 0x020e18 to 0x02a460 (0x9649 bytes) 000971 1781818083.213500 read 0x003f72 thru 0x0111ee (0xd27d bytes) 000973 1781818083.214364 trunc from 0x019502 to 0x0226b4 (0x91b3 bytes) 000974 1781818083.216260 fallocate 0x03372b thru 0x035dbf (0x2695 bytes) 000975 1781818083.217922 read 0x01141b thru 0x014aeb (0x36d1 bytes) 000976 1781818083.218330 write 0x016b01 thru 0x01a991 (0x3e91 bytes) 000977 1781818083.221281 punch from 0x00108c to 0x00143d (0x3b2 bytes) 000978 1781818083.222980 fallocate 0x02332b thru 0x02b3e8 (0x80be bytes) 000979 1781818083.224950 punch from 0x01516d to 0x01dbc5 (0x8a59 bytes) 000980 1781818083.229283 write 0x031264 thru 0x03d162 (0xbeff bytes) 000981 1781818083.230456 punch from 0x01f663 to 0x02a2b8 (0xac56 bytes) 000982 1781818083.232525 punch from 0x03696d to 0x03d162 (0x67f6 bytes) 000983 1781818083.236461 write 0x0082ee thru 0x0110e8 (0x8dfb bytes) 000984 1781818083.237656 trunc from 0x02304e to 0x02fbaf (0xcb62 bytes) 000985 1781818083.239356 write 0x00c898 thru 0x011e34 (0x559d bytes) 000986 1781818083.240248 read 0x03bd57 thru 0x03d162 (0x140c bytes) 000987 1781818083.244128 write 0x024224 thru 0x02ed37 (0xab14 bytes) 000988 1781818083.245318 trunc from 0x03c360 to 0x03d162 (0xe03 bytes) 000989 1781818083.247429 write 0x00771c thru 0x00d384 (0x5c69 bytes) 000990 1781818083.248323 fallocate 0x028495 thru 0x029f0d (0x1a79 bytes) 000991 1781818083.250341 mapread 0x03c714 thru 0x03d162 (0xa4f bytes) 000992 1781818083.254179 read 0x0141ef thru 0x016ec7 (0x2cd9 bytes) 000993 1781818083.256408 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3d163): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d163): Mirror 2: 0x4c27a35b (stale) CRC-32 checksum value for chunk [0, 0x3d163): Mirror 1: 0x7a3ae7ec Mirror 3: 0x7a3ae7ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d163: skipped 000994 1781818083.545527 mapwrite 0x032558 thru 0x0327eb (0x294 bytes) 000995 1781818083.585067 trunc from 0x010fda to 0x01c2d6 (0xb2fd bytes) 000996 1781818083.587854 mapwrite 0x002192 thru 0x0026c5 (0x534 bytes) 000997 1781818083.595890 trunc from 0x03d163 to 0x015f3a (0x27228 bytes) 000998 1781818083.609599 mapread 0x014a28 thru 0x015f3a (0x1513 bytes) 000999 1781818083.613606 trunc from 0x00bcfd to 0x015f3a (0xa23e bytes) All operations completed A-OK! PASS 70a (26s) == sanity-flr test 71: check mirror extend parent fid ==== 17:28:05 (1781818085) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x650:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (3s) == sanity-flr test 72: check append on FLR file ========== 17:28:08 (1781818088) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0674052 s, 31.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041068 s, 25.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x112:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (3s) == sanity-flr test 200a: stress test ===================== 17:28:11 (1781818091) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre2 Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5700544 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2902976 .. done Extending file size to 4926016 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8667040 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5447616 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8160160 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8405568 .. Extending file size to 8489120 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9275808 .. Extending file size to 8908448 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9090080 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 9312864 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 64090 64091 64093 64095 64097 192.168.204.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre2 (opts:) 192.168.204.105@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (65s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 17:29:16 (1781818156) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre2 Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 Extending file size to 2073408 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 2688960 .. Extending file size to 5338048 .. Extending file size to 4044800 .. done Extending file size to 4798528 .. Extending file size to 6925216 .. Extending file size to 6741344 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7164416 .. done Extending file size to 8701856 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8278560 .. Extending file size to 9221120 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8524192 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8545632 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8562368 .. Extending file size to 8578528 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9281408 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Waiting 67343 67344 67346 67347 67350 192.168.204.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre2 (opts:) 192.168.204.105@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (65s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 17:30:21 (1781818221) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2588 1268716 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2832 1268472 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 20904 3586116 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 23052 3583968 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43956 7170084 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2588 1268716 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2832 1268472 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 20904 3586116 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 23052 3583968 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43956 7170084 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0337558 s, 62.1 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0502389 s, 41.7 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x4db0d41 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre2 (opts:) PASS 200c (5s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 17:30:27 (1781818227) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0344847 s, 60.8 MB/s PASS 202 (2s) == sanity-flr test 203: mirror file preserve mirror ID === 17:30:29 (1781818229) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x129:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } PASS 203 (2s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 17:30:31 (1781818231) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0204983 s, 102 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x8ba:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x8b8:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x8b9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (2s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 17:30:34 (1781818234) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0283576 s, 74.0 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } PASS 204b (3s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 17:30:36 (1781818236) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0235489 s, 89.1 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } PASS 204c (24s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 17:31:00 (1781818260) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg405-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243274 s, 43.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } PASS 204d (35s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 17:31:35 (1781818295) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg405-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0429165 s, 48.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x130:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x133:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (44s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 17:32:19 (1781818339) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg405-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg405-server: Pool lustre.test_204f created Adding targets to pool oleg405-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg405-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0754393 s, 27.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x134:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg405-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg405-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg405-server: Pool lustre.test_204f destroyed PASS 204f (48s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 17:33:07 (1781818387) PASS 205a (3s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 17:33:10 (1781818390) PASS 205b (3s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 17:33:13 (1781818393) oleg405-server: Pool lustre.test_206 created Adding targets to pool oleg405-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg405-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg405-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg405-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg405-server: Pool lustre.test_206 destroyed PASS 206 (19s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 17:33:32 (1781818412) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0404331 s, 25.9 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0246257 s, 42.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xd149d43d Mirror 3: 0xd149d43d PASS 207 (5s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 17:33:37 (1781818417) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (2s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 17:33:39 (1781818419) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (2s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 17:33:41 (1781818421) Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0315739 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1760000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xdb0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xdb0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xdb0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x8c0000 Read this (4 bytes): BBBB 192.168.204.105@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg405-client.virtnet /mnt/lustre2 (opts:) PASS 209a (21s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 17:34:02 (1781818442) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.00514263 s, 1.6 MB/s the very first read read with warm cache PASS 209b (3s) == sanity-flr test 210a: handle broken mirrored lovea ==== 17:34:05 (1781818445) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260349 s, 40.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x82 lmm_fid: [0x200000405:0x82:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611266 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-18 17:34:06.000000000 -0400 Modify: 2026-06-18 17:34:06.000000000 -0400 Change: 2026-06-18 17:34:07.000000000 -0400 Birth: 2026-06-18 17:34:07.000000000 -0400 PASS 210a (4s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 17:34:09 (1781818449) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0154615 s, 67.8 MB/s oleg405-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg405-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg405-server: O/280000401/d29/13d: File not found by ext2_lookup PASS 210b (16s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 17:34:25 (1781818465) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.118475 s, 346 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0174637 s, 235 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 18 17:34 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x87:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 18 17:34 /mnt/lustre/f211.sanity-flr PASS 211 (4s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 17:34:29 (1781818469) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (3s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 17:34:32 (1781818472) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (5s) == sanity-flr test complete, duration 994 sec ============ 17:34:37 (1781818477) === sanity-flr: start cleanup 17:34:37 (1781818477) === === sanity-flr: finish cleanup 17:34:42 (1781818482) ===