-----============= acceptance-small: sanity-flr ============----- Tue Sep 29 22:46:58 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITY_FLR_ONLY='' SANITY_FLR_EXCEPT='' oleg343-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 22:47:16 (1790736436) === oleg343-client.virtnet: executing check_config_client /mnt/lustre oleg343-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg343-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8dd0917f3000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8dd0917f3000.idle_timeout=debug disable quota as required oleg343-server: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 22:47:53 (1790736473) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8190] == sanity-flr test 0a: lfs mirror create with -N option == 22:47:55 (1790736475) PASS 0a (15s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 22:48:10 (1790736490) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 22:48:13 (1790736493) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (2s) == sanity-flr test 0d: lfs mirror extend with -N option == 22:48:16 (1790736496) PASS 0d (19s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 22:48:35 (1790736515) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 22:48:39 (1790736519) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 22:48:42 (1790736522) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8dd0917f3000.stats=clear osc.lustre-OST0001-osc-ffff8dd0917f3000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 22:48:51 (1790736531) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--ec-map] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 22:49:00 (1790736540) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 22:49:07 (1790736547) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 22:49:10 (1790736550) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 22:49:17 (1790736557) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 22:49:25 (1790736565) PASS 2 (6s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 22:49:32 (1790736572) PASS 3 (6s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 22:49:39 (1790736579) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (6s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 22:49:45 (1790736585) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (5s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 22:49:53 (1790736593) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 22:49:59 (1790736599) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.482046 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0612889 s, 17.1 MB/s PASS 21 (7s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 22:50:06 (1790736606) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.263703 s, 31.8 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 22:50:13 (1790736613) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 22:50:23 (1790736623) Stopping /mnt/lustre-ost1 (opts:) on oleg343-server oleg343-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8dd0917f3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8dd0917f3000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg343-server: oleg343-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg343-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg343-server oleg343-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8dd0917f3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8dd0917f3000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg343-server: oleg343-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg343-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (71s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 22:51:34 (1790736694) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.579982 s, 19.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg343-server oleg343-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8dd0917f3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8dd0917f3000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg343-server: oleg343-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg343-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (38s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 22:52:12 (1790736732) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg343-server oleg343-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8dd0917f3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8dd0917f3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg343-server: oleg343-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg343-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg343-server oleg343-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8dd0917f3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8dd0917f3000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg343-server: oleg343-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg343-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (71s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 22:53:24 (1790736804) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.90632 s, 18.0 MB/s -rw-r--r-- 1 root root 52428800 Sep 29 22:53 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.800s user 0m0.001s sys 0m0.167s Stopping /mnt/lustre-ost1 (opts:) on oleg343-server oleg343-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8dd0917f3000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8dd0917f3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.992s user 0m0.000s sys 0m0.169s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg343-server: oleg343-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg343-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg343-server oleg343-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8dd0917f3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8dd0917f3000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.035s user 0m0.000s sys 0m0.145s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg343-server: oleg343-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg343-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg343-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8dd0917f3000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8dd0917f3000.ost_server_uuid in FULL state after 0 sec PASS 33b (82s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 22:54:47 (1790736887) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 22:54:50 (1790736890) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (2s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 22:54:52 (1790736892) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 22:54:56 (1790736896) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 22:55:03 (1790736903) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.665809 s, 18.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 22:55:15 (1790736915) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047963 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519265 s, 20.2 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 22:55:23 (1790736923) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 22:55:30 (1790736930) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00837523 s, 122 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00412521 s, 248 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00239732 s, 427 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00686258 s, 149 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 22:55:39 (1790736939) create test files with size 15741095 .. b176c82c44d17d7018109f5782913ae2 - 18d50fb26ec5198160e99141d610af2c - 5ce502492857dcfb865be6dc8b208cb1 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg343-client.virtnet /mnt/lustre (opts:) Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 22:56:00 (1790736960) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 22:56:12 (1790736972) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 22:56:19 (1790736979) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841761 s, 12.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219309 s, 47.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 22:56:28 (1790736988) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132207 s, 15.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.383157 s, 10.9 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.182017 s, 11.5 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (12s) == sanity-flr test 42: lfs mirror verify ================= 22:56:40 (1790737000) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 22:56:43 (1790737003) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 22:56:46 (1790737006) striped dir -i1 -c2 -H crush /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359552 s, 29.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (6s) == sanity-flr test 44a: lfs mirror split check =========== 22:56:52 (1790737012) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.420656 s, 7.5 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 22:57:02 (1790737022) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xcb:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc7:0x0] } split mirror_id 2 id 131073 error lfs split: file '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' has 1 component, cannot split make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 22:57:12 (1790737032) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.363128 s, 28.9 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg343-client.virtnet /mnt/lustre (opts:) Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 22:57:22 (1790737042) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8dd09098e000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.313629 s, 33.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg343-client.virtnet /mnt/lustre (opts:) Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8dd083795000.xattr_cache=1 PASS 44e (12s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 22:57:34 (1790737054) error lfs split: create victim file failed: File exists PASS 44f (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 22:57:50 (1790737070) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (36s) == sanity-flr test 46: Verify setstripe --copy option ==== 22:58:26 (1790737106) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 22:59:04 (1790737144) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 22:59:07 (1790737147) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.230449 s, 13.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.121644 s, 25.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x47551646 (nosync,stale) Mirror 2: 0x47551646 (nosync,stale) ** verify mirror 2 doesn't change original checksum: b224bce589fd67718151059ce64b50fa - mirror 1 checksum: 57289b666d0e3a5502bdb2e3fd872b30 - mirror 2 checksum: b224bce589fd67718151059ce64b50fa - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-29T22:59:10-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xfc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-29T22:59:10-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xfd:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: b224bce589fd67718151059ce64b50fa - mirror 1 checksum: 57289b666d0e3a5502bdb2e3fd872b30 - mirror 2 checksum: 57289b666d0e3a5502bdb2e3fd872b30 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-29T22:59:10-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xfc:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-29T22:59:10-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xfd:0x0] } PASS 48 (10s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 22:59:17 (1790737157) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357669 s, 29.3 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 22:59:24 (1790737164) Failing mds1 on oleg343-server Stopping /mnt/lustre-mds1 (opts:) on oleg343-server 22:59:30 (1790737170) shut down facet: mds1 facet_host: oleg343-server facet_failover_host: oleg343-server Failover mds1 to oleg343-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg343-server: oleg343-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-MDT0000 22:59:53 (1790737193) targets are mounted 22:59:53 (1790737193) facet_failover done oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472764 s, 22.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x240000404 lmm_object_id: 0x8 lmm_fid: [0x240000404:0x8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:00:12 (1790737212) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0262684 s, 780 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0111291 s, 1.8 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:00:22 (1790737222) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0502015 s, 408 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00993828 s, 2.1 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:00:31 (1790737231) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00601126 s, 2.7 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:00:38 (1790737238) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0860859 s, 238 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:00:47 (1790737247) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00674562 s, 607 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0281705 s, 145 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:00:53 (1790737253) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:01:13 (1790737273) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (27s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:01:40 (1790737300) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (22s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:02:02 (1790737322) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:02:26 (1790737346) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 23:02:33 (1790737353) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 58222 mirror split pid 58223 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 58222 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 58223 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (66s) == sanity-flr test 70a: flr mode fsx test ================ 23:03:39 (1790737419) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 7421 skipping zero size read skipping zero length zero range skipping zero length zero range skipping zero length zero range skipping zero length punch hole 000006 1790737421.624486 MIRROR_EXTEND to 2 mirrors skipping zero length punch hole skipping zero size read 000009 1790737421.927331 write 0x018a20 thru 0x022090 (0x9671 bytes) 000010 1790737421.965177 mapwrite 0x00fe23 thru 0x01e774 (0xe952 bytes) 000011 1790737422.002133 punch from 0x00d9d6 to 0x01cd07 (0xf332 bytes) 000012 1790737422.011238 write 0x035d90 thru 0x03c6b2 (0x6923 bytes) fallocating to largest ever: 0x23498 000013 1790737422.014583 fallocate 0x014577 thru 0x023497 (0xef21 bytes) 000014 1790737422.027033 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3c6b3): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c6b3): Mirror 2: 0xac864fe7 (stale) CRC-32 checksum value for chunk [0, 0x3c6b3): Mirror 1: 0x6d6e39a2 Mirror 3: 0x6d6e39a2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c6b3: skipped 000015 1790737422.828431 mapread 0x0050fa thru 0x0092bc (0x41c3 bytes) 000016 1790737422.848107 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) 000017 1790737423.194587 mapread 0x03a09f thru 0x03c6b2 (0x2614 bytes) 000018 1790737423.242880 punch from 0x0377c4 to 0x03b672 (0x3eaf bytes) 000019 1790737423.289328 mapread 0x018184 thru 0x018bcb (0xa48 bytes) 000021 1790737423.299968 read 0x0007d9 thru 0x001c16 (0x143e bytes) 000022 1790737423.304925 mapwrite 0x010c30 thru 0x01dd3e (0xd10f bytes) 000023 1790737423.335958 MIRROR_RESYNC 2 mirrors 000024 1790737423.840586 read 0x000099 thru 0x008bbd (0x8b25 bytes) 000025 1790737423.848768 mapwrite 0x036df3 thru 0x03ffff (0x920d bytes) 000026 1790737423.945638 mapwrite 0x012933 thru 0x01b1e6 (0x88b4 bytes) 000027 1790737424.000512 punch from 0x012894 to 0x01b00a (0x8777 bytes) 000030 1790737424.006988 punch from 0x033889 to 0x03ffff (0xc777 bytes) zero_range to largest ever: 0x309b7 000031 1790737424.014444 trunc from 0x02eb0c to 0x0309b6 (0x1eab bytes) 000032 1790737424.021121 trunc from 0x01e89c to 0x021ac6 (0x322b bytes) 000033 1790737424.028146 read 0x0037f2 thru 0x00a15a (0x6969 bytes) 000034 1790737424.045792 trunc from 0x02fbdb to 0x03d33d (0xd763 bytes) truncating to largest ever: 0x3e59a 000036 1790737424.057193 trunc from 0x040000 to 0x03e599 (0x1a66 bytes) 000037 1790737424.077623 trunc from 0x03e59a to 0x03bd53 (0x2846 bytes) 000038 1790737424.089953 trunc from 0x03bd54 to 0x03e13d (0x23ea bytes) 000039 1790737424.107856 trunc from 0x03e13e to 0x03a8e5 (0x3858 bytes) 000040 1790737424.129944 write 0x00b34a thru 0x016883 (0xb53a bytes) truncating to largest ever: 0x3e8c0 000041 1790737424.133388 trunc from 0x03a8e6 to 0x03e8bf (0x3fda bytes) 000044 1790737424.162598 write 0x0001ba thru 0x00c50f (0xc356 bytes) 000045 1790737424.164722 trunc from 0x001227 to 0x004669 (0x3443 bytes) 000046 1790737424.191064 punch from 0x01aa47 to 0x028d76 (0xe330 bytes) 000047 1790737424.195141 mapwrite 0x00b611 thru 0x0185b6 (0xcfa6 bytes) 000048 1790737424.212729 trunc from 0x03e8c0 to 0x031595 (0xd32a bytes) 000049 1790737424.227163 mapwrite 0x004940 thru 0x011025 (0xc6e6 bytes) 000050 1790737424.253445 MIRROR_RESYNC 2 mirrors 000051 1790737424.871961 write 0x035327 thru 0x03ffff (0xacd9 bytes) 000052 1790737424.918677 trunc from 0x0321dd to 0x03c74d (0xa571 bytes) 000054 1790737424.935947 trunc from 0x0046e5 to 0x00c742 (0x805e bytes) 000055 1790737424.944875 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xec59eefd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xac3b658a Mirror 4: 0xac3b658a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000057 1790737426.017617 mapwrite 0x000f9b thru 0x0031e5 (0x224b bytes) 000058 1790737426.158504 trunc from 0x040000 to 0x0399b5 (0x664a bytes) 000059 1790737426.185212 fallocate 0x021e89 thru 0x02d6b3 (0xb82b bytes) 000060 1790737426.193312 punch from 0x02af1a to 0x032895 (0x797c bytes) 000061 1790737426.201915 mapread 0x0172d6 thru 0x01cd66 (0x5a91 bytes) 000062 1790737426.212631 trunc from 0x0399b6 to 0x002b79 (0x36e3c bytes) 000063 1790737426.229978 trunc from 0x0024cd to 0x002b79 (0x6ad bytes) 000064 1790737426.245920 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b7a): Mirror 4: 0x13410de5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b7a): Mirror 4: 0x13410de5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b7a: skipped 000065 1790737426.517763 mapwrite 0x02c695 thru 0x0342e5 (0x7c51 bytes) 000066 1790737426.580398 trunc from 0x0342e6 to 0x02c198 (0x814d bytes) 000067 1790737426.604674 trunc from 0x014b23 to 0x024a91 (0xff6f bytes) 000068 1790737426.609684 mapread 0x014305 thru 0x01896e (0x466a bytes) 000069 1790737426.624037 mapread 0x007e0f thru 0x013504 (0xb6f6 bytes) 000070 1790737426.648459 read 0x0195b0 thru 0x01fe13 (0x6864 bytes) 000071 1790737426.656086 trunc from 0x02c199 to 0x032da5 (0x6c0d bytes) 000072 1790737426.694643 mapwrite 0x028bfe thru 0x029533 (0x936 bytes) 000074 1790737426.708197 mapwrite 0x00ccf0 thru 0x01b6e0 (0xe9f1 bytes) 000075 1790737426.725724 punch from 0x010335 to 0x01dc81 (0xd94d bytes) 000076 1790737426.731740 write 0x036da2 thru 0x03ffff (0x925e bytes) 000077 1790737426.735084 trunc from 0x040000 to 0x037193 (0x8e6c bytes) 000078 1790737426.748053 fallocate 0x02085b thru 0x02304e (0x27f4 bytes) 000079 1790737426.755138 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) 000080 1790737426.915811 fallocate 0x0202ab thru 0x02ad6f (0xaac5 bytes) 000081 1790737426.938462 MIRROR_EXTEND to 2 mirrors 000082 1790737427.869096 trunc from 0x013be4 to 0x01b2e9 (0x7706 bytes) 000083 1790737427.908886 fallocate 0x020c9d thru 0x021396 (0x6fa bytes) 000084 1790737427.918598 MIRROR_RESYNC 2 mirrors 000085 1790737428.728426 punch from 0x022c7f to 0x02fc3c (0xcfbe bytes) 000087 1790737428.761953 MIRROR_RESYNC 2 mirrors 000088 1790737429.310877 mapwrite 0x006c9d thru 0x0115b0 (0xa914 bytes) 000089 1790737429.419213 mapwrite 0x01063d thru 0x01f1b1 (0xeb75 bytes) 000091 1790737429.458319 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) 000092 1790737429.663064 read 0x01c289 thru 0x0269e8 (0xa760 bytes) 000094 1790737429.675238 mapread 0x027fc2 thru 0x032fa1 (0xafe0 bytes) 000095 1790737429.707186 punch from 0x010a51 to 0x01c27b (0xb82b bytes) 000096 1790737429.713903 trunc from 0x02705c to 0x02d77f (0x6724 bytes) 000098 1790737429.719754 mapread 0x01a633 thru 0x01d65d (0x302b bytes) 000099 1790737429.733057 punch from 0x02cd21 to 0x02fd1a (0x2ffa bytes) 000100 1790737429.742903 punch from 0x01d230 to 0x0291a1 (0xbf72 bytes) 000101 1790737429.749936 fallocate 0x01c469 thru 0x02b890 (0xf428 bytes) 000103 1790737429.756442 fallocate 0x011a2b thru 0x01af68 (0x953e bytes) 000104 1790737429.761567 write 0x01db92 thru 0x0238d9 (0x5d48 bytes) 000106 1790737429.781855 read 0x01da27 thru 0x023203 (0x57dd bytes) 000107 1790737429.783148 trunc from 0x02bca6 to 0x035f74 (0xa2cf bytes) 000108 1790737429.790140 fallocate 0x0216b2 thru 0x02f467 (0xddb6 bytes) 000109 1790737429.797558 read 0x01afbe thru 0x01cd31 (0x1d74 bytes) 000113 1790737429.802072 trunc from 0x037194 to 0x01f67d (0x17b16 bytes) 000114 1790737429.826048 read 0x00340e thru 0x010e6f (0xda62 bytes) 000115 1790737429.841692 punch from 0x00d42b to 0x014a62 (0x7638 bytes) 000116 1790737429.849566 mapread 0x003e29 thru 0x0055a4 (0x177c bytes) 000118 1790737429.852520 punch from 0x014d42 to 0x01c957 (0x7c16 bytes) 000119 1790737429.859541 mapread 0x01d285 thru 0x01f67d (0x23f9 bytes) 000120 1790737429.861495 trunc from 0x013b9e to 0x01d4a7 (0x990a bytes) 000121 1790737429.873477 mapwrite 0x00c93e thru 0x01a849 (0xdf0c bytes) 000122 1790737429.968093 trunc from 0x01f67e to 0x007fde (0x1769f bytes) 000123 1790737429.982921 read 0x004ccc thru 0x007fde (0x3313 bytes) 000124 1790737429.983582 trunc from 0x007fdf to 0x01940c (0x1142e bytes) 000125 1790737429.999335 trunc from 0x0110ad to 0x0149e8 (0x393c bytes) 000126 1790737430.005025 trunc from 0x01564f to 0x01940c (0x3dbe bytes) 000127 1790737430.011694 trunc from 0x001d01 to 0x00c87b (0xab7b bytes) 000128 1790737430.019216 mapread 0x015ad8 thru 0x015d84 (0x2ad bytes) 000129 1790737430.038201 trunc from 0x00fa6c to 0x017855 (0x7dea bytes) 000130 1790737430.049624 trunc from 0x009a84 to 0x00da18 (0x3f95 bytes) 000131 1790737430.061469 fallocate 0x026157 thru 0x0262fd (0x1a7 bytes) 000133 1790737430.088705 mapread 0x0248a1 thru 0x0262fd (0x1a5d bytes) 000135 1790737430.107086 mapwrite 0x00f7c8 thru 0x013825 (0x405e bytes) 000136 1790737430.185233 trunc from 0x0262fe to 0x017f51 (0xe3ac bytes) 000137 1790737430.207989 read 0x012e05 thru 0x017f51 (0x514d bytes) fallocating to largest ever: 0x40000 000138 1790737430.218108 fallocate 0x0371e2 thru 0x03ffff (0x8e1e bytes) 000139 1790737430.223357 mapwrite 0x005d29 thru 0x0060e4 (0x3bc bytes) 000140 1790737430.238575 punch from 0x001470 to 0x007cbf (0x6850 bytes) 000141 1790737430.246920 mapread 0x001d9d thru 0x00f8e4 (0xdb48 bytes) 000143 1790737430.286506 read 0x00edeb thru 0x017f51 (0x9167 bytes) 000144 1790737430.287177 mapwrite 0x019e4c thru 0x02549c (0xb651 bytes) 000145 1790737430.335508 read 0x00905a thru 0x012215 (0x91bc bytes) 000146 1790737430.336226 punch from 0x017c45 to 0x0252d0 (0xd68c bytes) 000147 1790737430.341511 read 0x002b69 thru 0x008456 (0x58ee bytes) 000148 1790737430.342557 mapwrite 0x028a42 thru 0x0294c2 (0xa81 bytes) 000149 1790737430.379538 read 0x00e42d thru 0x015b00 (0x76d4 bytes) 000150 1790737430.380490 fallocate 0x018286 thru 0x026847 (0xe5c2 bytes) 000151 1790737430.385109 read 0x00bc96 thru 0x01a09d (0xe408 bytes) 000152 1790737430.386089 fallocate 0x00c1a7 thru 0x019c4f (0xdaa9 bytes) 000153 1790737430.390181 fallocate 0x02854a thru 0x02f464 (0x6f1b bytes) 000154 1790737430.394488 read 0x01addb thru 0x01ed31 (0x3f57 bytes) 000155 1790737430.395786 trunc from 0x0294c3 to 0x01abfa (0xe8c8 bytes) 000156 1790737430.418784 mapread 0x00e08d thru 0x0173f1 (0x9365 bytes) 000158 1790737430.421814 mapwrite 0x03c759 thru 0x03eade (0x2386 bytes) 000159 1790737430.463076 write 0x0063bf thru 0x00d203 (0x6e45 bytes) 000161 1790737430.465195 trunc from 0x03eadf to 0x00a883 (0x3425b bytes) 000162 1790737430.483601 trunc from 0x00a884 to 0x024e8e (0x1a60b bytes) 000163 1790737430.504030 trunc from 0x024e8f to 0x02608c (0x11fe bytes) 000164 1790737430.519156 MIRROR_EXTEND to 2 mirrors 000165 1790737431.141949 MIRROR_EXTEND to 3 mirrors 000166 1790737431.946564 mapwrite 0x03910b thru 0x0395ae (0x4a4 bytes) 000167 1790737432.045919 punch from 0x0060c9 to 0x00a5c0 (0x44f8 bytes) 000168 1790737432.051136 mapwrite 0x033398 thru 0x039c8a (0x68f3 bytes) 000170 1790737432.107220 trunc from 0x039c8b to 0x019983 (0x20307 bytes) 000171 1790737432.131406 MIRROR_RESYNC 3 mirrors 000172 1790737432.994998 read 0x0050eb thru 0x013ef6 (0xee0c bytes) 000173 1790737433.010472 trunc from 0x003711 to 0x011757 (0xe047 bytes) 000174 1790737433.046128 trunc from 0x012341 to 0x019983 (0x7643 bytes) 000175 1790737433.053529 trunc from 0x0110fd to 0x015622 (0x4526 bytes) 000176 1790737433.058441 mapread 0x009673 thru 0x00acbb (0x1649 bytes) 000177 1790737433.066600 trunc from 0x019984 to 0x010cb7 (0x8ccc bytes) 000178 1790737433.082778 fallocate 0x01d2f2 thru 0x02ba5b (0xe76a bytes) 000179 1790737433.087119 read 0x024c30 thru 0x02b7cb (0x6b9c bytes) 000180 1790737433.108087 MIRROR_RESYNC 3 mirrors 000181 1790737433.763501 punch from 0x019925 to 0x01d5d4 (0x3cb0 bytes) 000183 1790737433.824547 trunc from 0x01436c to 0x017417 (0x30ac bytes) 000184 1790737433.829618 mapread 0x00c14f thru 0x0161f4 (0xa0a6 bytes) 000185 1790737433.864417 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ba5c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ba5c): Mirror 2: 0x89a73763 (stale) Mirror 3: 0x89a73763 (stale) CRC-32 checksum value for chunk [0, 0x2ba5c): Mirror 1: 0x89a73763 Mirror 4: 0x89a73763 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ba5c: skipped 000186 1790737434.774615 trunc from 0x0040d3 to 0x00cfe5 (0x8f13 bytes) 000187 1790737434.829613 MIRROR_RESYNC 4 mirrors 000189 1790737435.583314 mapread 0x024d32 thru 0x02ba5b (0x6d2a bytes) 000190 1790737435.601092 read 0x028d55 thru 0x02ba5b (0x2d07 bytes) 000191 1790737435.601486 read 0x00837b thru 0x014fae (0xcc34 bytes) 000192 1790737435.610694 write 0x010cc0 thru 0x01df08 (0xd249 bytes) 000193 1790737435.641293 read 0x00f848 thru 0x01357d (0x3d36 bytes) 000194 1790737435.653233 punch from 0x01c24c to 0x020c01 (0x49b6 bytes) 000195 1790737435.666381 punch from 0x01702c to 0x01f8b7 (0x888c bytes) 000196 1790737435.671022 trunc from 0x02ba5c to 0x0281d0 (0x388b bytes) 000197 1790737435.689714 trunc from 0x019b1e to 0x01f208 (0x56eb bytes) 000198 1790737435.695102 trunc from 0x0281d1 to 0x0210dd (0x70f3 bytes) 000199 1790737435.707990 trunc from 0x009c33 to 0x00e1d9 (0x45a7 bytes) 000201 1790737435.713010 trunc from 0x00b795 to 0x00df48 (0x27b4 bytes) 000202 1790737435.717036 trunc from 0x0210de to 0x01b3d8 (0x5d05 bytes) 000204 1790737435.730256 mapread 0x01130d thru 0x01226c (0xf60 bytes) 000205 1790737435.734884 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b3d9): Mirror 2: 0xb83efac2 (stale) Mirror 4: 0xb83efac2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b3d9): Mirror 2: 0xb83efac2 (stale) Mirror 4: 0xb83efac2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b3d9: skipped 000207 1790737436.000679 trunc from 0x00c843 to 0x012011 (0x57cf bytes) 000208 1790737436.006118 fallocate 0x03204c thru 0x035bb6 (0x3b6b bytes) 000209 1790737436.012237 read 0x005ed1 thru 0x014ba1 (0xecd1 bytes) 000210 1790737436.021576 fallocate 0x013b51 thru 0x0237e7 (0xfc97 bytes) 000211 1790737436.029701 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x237e8): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x237e8): Mirror 2: 0xdf8fb10 (stale) Mirror 4: 0xdf8fb10 (stale) CRC-32 checksum value for chunk [0, 0x237e8): Mirror 1: 0xede45cd6 Mirror 5: 0xede45cd6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x237e8: skipped 000212 1790737436.945542 write 0x03617d thru 0x03ffff (0x9e83 bytes) 000214 1790737436.989785 punch from 0x03b14d to 0x03ffff (0x4eb3 bytes) 000215 1790737437.008302 trunc from 0x019845 to 0x01ab7f (0x133b bytes) 000216 1790737437.012816 punch from 0x0061af to 0x01429f (0xe0f1 bytes) 000217 1790737437.017711 mapwrite 0x026358 thru 0x029582 (0x322b bytes) 000218 1790737437.052640 punch from 0x03f819 to 0x03ffff (0x7e7 bytes) 000219 1790737437.057072 punch from 0x03abd5 to 0x03c2a9 (0x16d5 bytes) 000220 1790737437.062401 mapwrite 0x0146c7 thru 0x023a31 (0xf36b bytes) 000222 1790737437.094151 fallocate 0x03776b thru 0x03ffff (0x8895 bytes) 000223 1790737437.099636 write 0x01cbdb thru 0x01f3b5 (0x27db bytes) 000224 1790737437.104595 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4234a4f2 (stale) Mirror 4: 0x4234a4f2 (stale) Mirror 5: 0xa77ab082 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1c0578c8 Mirror 6: 0x1c0578c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000225 1790737438.062892 mapread 0x02b983 thru 0x02e18c (0x280a bytes) 000226 1790737438.119537 write 0x033168 thru 0x03be7b (0x8d14 bytes) 000227 1790737438.183820 punch from 0x01faea to 0x025e40 (0x6357 bytes) 000229 1790737438.190165 read 0x02e2d1 thru 0x02ead3 (0x803 bytes) 000230 1790737438.197523 mapwrite 0x031b27 thru 0x037063 (0x553d bytes) 000231 1790737438.226638 punch from 0x01ba41 to 0x021d98 (0x6358 bytes) 000232 1790737438.231087 write 0x00a72e thru 0x0119f5 (0x72c8 bytes) 000234 1790737438.238453 trunc from 0x03c400 to 0x03ffff (0x3c00 bytes) 000235 1790737438.247891 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4234a4f2 (stale) Mirror 4: 0x4234a4f2 (stale) Mirror 5: 0xa77ab082 (stale) Mirror 6: 0x1c0578c8 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x8d238b0d Mirror 7: 0x8d238b0d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000236 1790737439.267326 trunc from 0x040000 to 0x02cff8 (0x13007 bytes) 000237 1790737439.356571 mapwrite 0x01c7fe thru 0x026ccf (0xa4d2 bytes) 000238 1790737439.426670 trunc from 0x010ca2 to 0x0156da (0x4a39 bytes) 000239 1790737439.435614 write 0x014ef0 thru 0x01f810 (0xa921 bytes) 000240 1790737439.444546 trunc from 0x02cff9 to 0x022774 (0xa884 bytes) 000241 1790737439.480792 punch from 0x007841 to 0x00df0c (0x66cc bytes) 000243 1790737439.488260 punch from 0x00e141 to 0x01372a (0x55ea bytes) 000244 1790737439.492577 write 0x02f6b2 thru 0x038639 (0x8f88 bytes) 000245 1790737439.498413 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3863a): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3863a): Mirror 2: 0x3a7e866e (stale) Mirror 4: 0x3a7e866e (stale) Mirror 5: 0xb4c27c32 (stale) Mirror 6: 0x966137 (stale) Mirror 7: 0x8c850aaa (stale) CRC-32 checksum value for chunk [0, 0x3863a): Mirror 1: 0xbfa84d07 Mirror 8: 0xbfa84d07 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3863a: skipped 000246 1790737440.516476 fallocate 0x00b8d4 thru 0x01af0a (0xf637 bytes) 000247 1790737440.556178 write 0x03e3db thru 0x03ffff (0x1c25 bytes) 000248 1790737440.566521 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4234a4f2 (stale) Mirror 4: 0x4234a4f2 (stale) Mirror 5: 0xa77ab082 (stale) Mirror 7: 0x8d238b0d (stale) Mirror 8: 0xcdd5f758 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4234a4f2 (stale) Mirror 4: 0x4234a4f2 (stale) Mirror 5: 0xa77ab082 (stale) Mirror 7: 0x8d238b0d (stale) Mirror 8: 0xcdd5f758 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000249 1790737440.941559 trunc from 0x00b4d6 to 0x00b882 (0x3ad bytes) 000250 1790737440.950521 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4234a4f2 (stale) Mirror 4: 0x4234a4f2 (stale) Mirror 5: 0xa77ab082 (stale) Mirror 7: 0x8d238b0d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4234a4f2 (stale) Mirror 4: 0x4234a4f2 (stale) Mirror 5: 0xa77ab082 (stale) Mirror 7: 0x8d238b0d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000251 1790737441.284977 mapread 0x00a0bc thru 0x013e6c (0x9db1 bytes) 000252 1790737441.300476 trunc from 0x02b636 to 0x0352cc (0x9c97 bytes) 000253 1790737441.310020 trunc from 0x03ba18 to 0x03ffff (0x45e8 bytes) 000254 1790737441.333278 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4234a4f2 (stale) Mirror 4: 0x4234a4f2 (stale) Mirror 5: 0xa77ab082 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4234a4f2 (stale) Mirror 4: 0x4234a4f2 (stale) Mirror 5: 0xa77ab082 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000255 1790737441.668505 fallocate 0x01dd2d thru 0x021942 (0x3c16 bytes) 000256 1790737441.673641 punch from 0x02bff9 to 0x0371a7 (0xb1af bytes) 000257 1790737441.678607 write 0x03ae72 thru 0x03ffff (0x518e bytes) 000258 1790737441.691718 mapwrite 0x0003a1 thru 0x0017a9 (0x1409 bytes) 000259 1790737441.722387 write 0x0053f4 thru 0x0092df (0x3eec bytes) 000260 1790737441.734745 fallocate 0x0087a2 thru 0x008db8 (0x617 bytes) 000261 1790737441.748163 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4234a4f2 (stale) Mirror 5: 0xa77ab082 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4234a4f2 (stale) Mirror 5: 0xa77ab082 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000262 1790737442.095618 write 0x006b7f thru 0x0140c5 (0xd547 bytes) 000263 1790737442.116585 mapread 0x03cf8c thru 0x03ffff (0x3074 bytes) 000264 1790737442.147730 punch from 0x017e24 to 0x0260d3 (0xe2b0 bytes) 000265 1790737442.151070 trunc from 0x03b912 to 0x03ffff (0x46ee bytes) 000266 1790737442.154456 trunc from 0x030e5d to 0x032f53 (0x20f7 bytes) 000267 1790737442.157442 trunc from 0x040000 to 0x02e4a6 (0x11b59 bytes) 000268 1790737442.187997 trunc from 0x02e4a7 to 0x01e7be (0xfce8 bytes) 000269 1790737442.204906 punch from 0x019f55 to 0x01acb4 (0xd60 bytes) 000270 1790737442.214094 punch from 0x0070bf to 0x01610c (0xf04e bytes) 000271 1790737442.235383 punch from 0x0184e2 to 0x01be66 (0x3985 bytes) 000272 1790737442.245459 punch from 0x01afe0 to 0x01e7be (0x37df bytes) 000273 1790737442.255600 trunc from 0x0198a0 to 0x01e7be (0x4f1f bytes) 000274 1790737442.271761 trunc from 0x01cc87 to 0x01e7be (0x1b38 bytes) 000275 1790737442.286193 trunc from 0x00df09 to 0x011c03 (0x3cfb bytes) 000276 1790737442.304399 trunc from 0x00739b to 0x0077a6 (0x40c bytes) 000277 1790737442.324027 trunc from 0x01e7bf to 0x009062 (0x1575c bytes) 000278 1790737442.360710 punch from 0x0044cd to 0x009062 (0x4b96 bytes) 000279 1790737442.375781 punch from 0x0018fa to 0x009062 (0x7769 bytes) 000280 1790737442.392760 mapwrite 0x00e66a thru 0x01943d (0xadd4 bytes) 000281 1790737442.481060 trunc from 0x01213c to 0x01943d (0x7302 bytes) 000282 1790737442.486214 trunc from 0x01943e to 0x03f35f (0x25f22 bytes) 000283 1790737442.504941 trunc from 0x03f360 to 0x0077cc (0x37b93 bytes) 000284 1790737442.530637 trunc from 0x007541 to 0x0077cc (0x28c bytes) 000285 1790737442.538687 trunc from 0x0077cd to 0x03432b (0x2cb5f bytes) 000286 1790737442.595573 mapread 0x03061d thru 0x031528 (0xf0c bytes) 000288 1790737442.613535 read 0x015215 thru 0x021ad3 (0xc8bf bytes) 000289 1790737442.639943 fallocate 0x02ae59 thru 0x03ac3f (0xfde7 bytes) 000290 1790737442.650503 punch from 0x0315b5 to 0x03432b (0x2d77 bytes) 000291 1790737442.656976 trunc from 0x01cc2c to 0x022caa (0x607f bytes) 000292 1790737442.667931 mapwrite 0x000abd thru 0x00e9ec (0xdf30 bytes) 000293 1790737442.724891 fallocate 0x00a0ce thru 0x010ffe (0x6f31 bytes) 000294 1790737442.733202 write 0x03bd84 thru 0x03ffff (0x427c bytes) 000295 1790737442.747007 trunc from 0x00dc18 to 0x01ab1b (0xcf04 bytes) 000297 1790737442.761125 write 0x027ed2 thru 0x02b5fc (0x372b bytes) 000298 1790737442.766425 trunc from 0x02ad7d to 0x03012e (0x53b2 bytes) 000299 1790737442.790787 MIRROR_RESYNC 3 mirrors 000300 1790737443.841645 trunc from 0x040000 to 0x01ba8d (0x24572 bytes) 000301 1790737443.920438 mapread 0x018052 thru 0x01ba8d (0x3a3c bytes) 000302 1790737443.940723 write 0x03cc30 thru 0x03ffff (0x33d0 bytes) 000303 1790737443.942354 write 0x03ff85 thru 0x03ffff (0x7b bytes) 000304 1790737443.943130 mapread 0x010380 thru 0x010478 (0xf9 bytes) 000305 1790737443.961720 trunc from 0x02f583 to 0x0338a0 (0x431e bytes) 000306 1790737443.973551 read 0x000628 thru 0x007017 (0x69f0 bytes) 000307 1790737443.993904 trunc from 0x004d0f to 0x00e154 (0x9446 bytes) 000308 1790737444.005328 punch from 0x02a6b5 to 0x038c43 (0xe58f bytes) 000310 1790737444.014400 fallocate 0x01fcb3 thru 0x0282a0 (0x85ee bytes) 000311 1790737444.046068 MIRROR_RESYNC 3 mirrors 000312 1790737444.662437 fallocate 0x03b87a thru 0x03ffff (0x4786 bytes) 000313 1790737444.692567 trunc from 0x040000 to 0x018fc4 (0x2703b bytes) 000315 1790737444.713305 mapread 0x01593c thru 0x018fc4 (0x3689 bytes) 000317 1790737444.732463 read 0x0016c5 thru 0x004fff (0x393b bytes) 000318 1790737444.737753 mapread 0x00da8d thru 0x011fb8 (0x452c bytes) 000319 1790737444.759024 write 0x01df45 thru 0x022104 (0x41c0 bytes) 000320 1790737444.762347 mapwrite 0x029830 thru 0x02a686 (0xe57 bytes) 000321 1790737444.808170 fallocate 0x00b1f9 thru 0x00bb9a (0x9a2 bytes) 000323 1790737444.821420 mapwrite 0x025358 thru 0x0351ed (0xfe96 bytes) 000324 1790737444.927619 trunc from 0x0351ee to 0x02ce63 (0x838a bytes) 000325 1790737444.959549 write 0x01b9cf thru 0x023029 (0x765b bytes) 000326 1790737444.983708 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ce64): Mirror 5: 0x8fddfc17 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ce64): Mirror 5: 0x8fddfc17 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ce64: skipped 000327 1790737445.362289 fallocate 0x010fd5 thru 0x0175d4 (0x6600 bytes) 000328 1790737445.366979 trunc from 0x02ce64 to 0x02edf3 (0x1f90 bytes) 000329 1790737445.382229 write 0x00ef28 thru 0x017306 (0x83df bytes) 000330 1790737445.391761 read 0x01ad5d thru 0x021828 (0x6acc bytes) 000331 1790737445.400023 read 0x017b4a thru 0x01f404 (0x78bb bytes) 000332 1790737445.409371 trunc from 0x01ffb4 to 0x02e8d5 (0xe922 bytes) 000333 1790737445.414310 mapread 0x0212aa thru 0x022523 (0x127a bytes) 000334 1790737445.422302 fallocate 0x002b26 thru 0x007548 (0x4a23 bytes) 000335 1790737445.427004 mapread 0x0172f3 thru 0x024abb (0xd7c9 bytes) 000336 1790737445.442474 trunc from 0x02edf4 to 0x027c15 (0x71de bytes) 000337 1790737445.455444 fallocate 0x003108 thru 0x00ce27 (0x9d20 bytes) 000338 1790737445.459801 read 0x01ce2c thru 0x027c15 (0xadea bytes) 000339 1790737445.464915 mapwrite 0x00dc70 thru 0x014edb (0x726c bytes) 000340 1790737445.480409 mapread 0x00c5e3 thru 0x019284 (0xcca2 bytes) 000341 1790737445.488347 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) 000342 1790737445.628753 fallocate 0x00804a thru 0x015a2e (0xd9e5 bytes) 000344 1790737445.633629 trunc from 0x027c16 to 0x00db1c (0x1a0f9 bytes) 000346 1790737445.661808 trunc from 0x00db1d to 0x0019ce (0xc14e bytes) 000347 1790737445.680278 mapread 0x000e48 thru 0x0019ce (0xb87 bytes) 000348 1790737445.690615 write 0x018bd3 thru 0x01e0f7 (0x5525 bytes) 000349 1790737445.692337 fallocate 0x02b2d7 thru 0x0323ee (0x7118 bytes) 000352 1790737445.697179 read 0x006c93 thru 0x01532c (0xe69a bytes) 000353 1790737445.703500 write 0x01781c thru 0x018a9e (0x1283 bytes) 000354 1790737445.709021 trunc from 0x01d92b to 0x01e0f7 (0x7cd bytes) 000355 1790737445.724570 mapwrite 0x0286c5 thru 0x02c44e (0x3d8a bytes) 000356 1790737445.785060 punch from 0x00eeaf to 0x01c34b (0xd49d bytes) 000357 1790737445.791145 fallocate 0x03e74e thru 0x03ffff (0x18b2 bytes) 000358 1790737445.795387 punch from 0x00dba7 to 0x019262 (0xb6bc bytes) 000359 1790737445.804645 punch from 0x03ec60 to 0x03f868 (0xc09 bytes) 000360 1790737445.818059 read 0x03c03c thru 0x03ffff (0x3fc4 bytes) 000361 1790737445.829682 write 0x014742 thru 0x018c93 (0x4552 bytes) 000362 1790737445.830803 trunc from 0x003987 to 0x004614 (0xc8e bytes) 000364 1790737445.836503 punch from 0x01552e to 0x021361 (0xbe34 bytes) 000365 1790737445.846326 mapwrite 0x036084 thru 0x03b2ef (0x526c bytes) 000366 1790737445.864425 trunc from 0x027ed8 to 0x032351 (0xa47a bytes) 000367 1790737445.876092 trunc from 0x040000 to 0x03bf3b (0x40c4 bytes) 000369 1790737445.908463 trunc from 0x03bf3c to 0x03853c (0x39ff bytes) 000370 1790737445.928197 trunc from 0x03853d to 0x00d226 (0x2b316 bytes) 000372 1790737445.957948 mapread 0x004a90 thru 0x007027 (0x2598 bytes) 000373 1790737445.963419 read 0x00cb60 thru 0x00d226 (0x6c7 bytes) 000374 1790737445.964192 trunc from 0x00d227 to 0x00b694 (0x1b92 bytes) 000375 1790737445.976376 write 0x004e3a thru 0x0055d9 (0x7a0 bytes) 000376 1790737445.980381 punch from 0x0044dd to 0x008079 (0x3b9d bytes) 000377 1790737446.013856 write 0x001dfe thru 0x00c72d (0xa930 bytes) 000378 1790737446.021792 mapwrite 0x020062 thru 0x028f88 (0x8f27 bytes) 000379 1790737446.083444 trunc from 0x028f89 to 0x000b61 (0x28427 bytes) 000380 1790737446.103312 mapread 0x00085b thru 0x000b61 (0x307 bytes) 000381 1790737446.107085 write 0x01da94 thru 0x02422d (0x679a bytes) 000383 1790737446.112303 trunc from 0x004517 to 0x00576b (0x1255 bytes) 000385 1790737446.130092 fallocate 0x009421 thru 0x012b68 (0x9748 bytes) 000386 1790737446.138958 write 0x027e05 thru 0x02ab63 (0x2d5f bytes) 000387 1790737446.140378 fallocate 0x021e6a thru 0x030703 (0xe89a bytes) 000389 1790737446.151580 fallocate 0x022153 thru 0x02479b (0x2649 bytes) 000390 1790737446.169382 fallocate 0x018cb0 thru 0x01e9c0 (0x5d11 bytes) 000391 1790737446.174997 read 0x01f92c thru 0x021a64 (0x2139 bytes) 000392 1790737446.175944 mapread 0x0242d1 thru 0x02ab63 (0x6893 bytes) 000393 1790737446.185977 mapwrite 0x0012ee thru 0x008a78 (0x778b bytes) 000394 1790737446.206496 trunc from 0x02ab64 to 0x005072 (0x25af1 bytes) 000396 1790737446.228495 write 0x0163c4 thru 0x01a792 (0x43cf bytes) 000397 1790737446.230920 read 0x019e35 thru 0x01a792 (0x95e bytes) 000398 1790737446.232478 read 0x00037e thru 0x00f4a5 (0xf128 bytes) 000399 1790737446.239472 mapread 0x0025ce thru 0x00c302 (0x9d35 bytes) 000400 1790737446.241001 punch from 0x009b5a to 0x00b87a (0x1d21 bytes) 000401 1790737446.244756 write 0x02b3de thru 0x033b9a (0x87bd bytes) 000403 1790737446.248008 trunc from 0x033b9b to 0x016f47 (0x1cc53 bytes) 000404 1790737446.267285 mapwrite 0x010eef thru 0x01d8e4 (0xc9f6 bytes) 000405 1790737446.367240 read 0x01b9dc thru 0x01d8e4 (0x1f09 bytes) 000406 1790737446.370778 mapwrite 0x00fd03 thru 0x011735 (0x1a33 bytes) 000407 1790737446.401077 MIRROR_EXTEND to 2 mirrors 000408 1790737446.992727 fallocate 0x0221ac thru 0x024902 (0x2757 bytes) 000409 1790737447.040995 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x24903): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24903): Mirror 2: 0xe4d37662 (stale) CRC-32 checksum value for chunk [0, 0x24903): Mirror 1: 0xe4d37662 Mirror 3: 0xe4d37662 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24903: skipped 000410 1790737447.918792 fallocate 0x006742 thru 0x00e1d4 (0x7a93 bytes) 000411 1790737447.970348 read 0x01db5e thru 0x024902 (0x6da5 bytes) 000412 1790737447.983702 trunc from 0x0171dc to 0x02007b (0x8ea0 bytes) 000413 1790737447.991550 punch from 0x019f38 to 0x022b8e (0x8c57 bytes) 000414 1790737448.001043 trunc from 0x021d57 to 0x024902 (0x2bac bytes) 000415 1790737448.011474 write 0x03df83 thru 0x03ffff (0x207d bytes) 000416 1790737448.018708 mapwrite 0x007ad2 thru 0x00ccaf (0x51de bytes) 000417 1790737448.106287 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa602ba6c (stale) Mirror 3: 0xa602ba6c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaaed578a Mirror 4: 0xaaed578a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000418 1790737449.093486 fallocate 0x007c53 thru 0x013689 (0xba37 bytes) 000419 1790737449.138481 trunc from 0x029fe5 to 0x02a487 (0x4a3 bytes) 000420 1790737449.146363 write 0x01340f thru 0x013e00 (0x9f2 bytes) 000421 1790737449.162810 trunc from 0x040000 to 0x01a6fc (0x25903 bytes) 000423 1790737449.228561 read 0x00170a thru 0x006497 (0x4d8e bytes) 000425 1790737449.247316 mapread 0x003609 thru 0x008f82 (0x597a bytes) 000426 1790737449.263810 write 0x00c661 thru 0x01b669 (0xf009 bytes) 000427 1790737449.269523 trunc from 0x01b66a to 0x03f48e (0x23e25 bytes) 000428 1790737449.293996 punch from 0x01c10e to 0x0244bc (0x83af bytes) 000430 1790737449.302532 fallocate 0x02979f thru 0x033a90 (0xa2f2 bytes) 000431 1790737449.314778 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3f48f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f48f): Mirror 2: 0x37584206 (stale) Mirror 3: 0x37584206 (stale) Mirror 4: 0x7e017d59 (stale) CRC-32 checksum value for chunk [0, 0x3f48f): Mirror 1: 0xf5ff9772 Mirror 5: 0xf5ff9772 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f48f: skipped 000432 1790737450.132942 trunc from 0x03f48f to 0x01685f (0x28c2f bytes) 000433 1790737450.192005 fallocate 0x00fb57 thru 0x01b30c (0xb7b6 bytes) 000434 1790737450.203745 write 0x017ebb thru 0x025beb (0xdd31 bytes) 000435 1790737450.213809 trunc from 0x00a215 to 0x01a024 (0xfe10 bytes) 000436 1790737450.240163 punch from 0x017b04 to 0x023ba6 (0xc0a3 bytes) 000437 1790737450.245356 trunc from 0x025bec to 0x02353f (0x26ac bytes) 000438 1790737450.290296 read 0x00e37a thru 0x01551e (0x71a5 bytes) 000439 1790737450.298475 mapwrite 0x009e73 thru 0x01129e (0x742c bytes) 000440 1790737450.373594 punch from 0x0097cf to 0x00f2af (0x5ae1 bytes) 000441 1790737450.389052 trunc from 0x00503e to 0x006421 (0x13e4 bytes) 000442 1790737450.404615 write 0x0164a6 thru 0x017ca9 (0x1804 bytes) 000443 1790737450.431084 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23540): Mirror 3: 0x55b96e94 (stale) Mirror 4: 0x38831f2d (stale) Mirror 5: 0x15301641 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23540): Mirror 3: 0x55b96e94 (stale) Mirror 4: 0x38831f2d (stale) Mirror 5: 0x15301641 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23540: skipped 000444 1790737450.948289 fallocate 0x001443 thru 0x0059d8 (0x4596 bytes) 000446 1790737450.952677 read 0x0079ba thru 0x008f77 (0x15be bytes) 000447 1790737450.956259 mapwrite 0x02504c thru 0x02a86f (0x5824 bytes) 000448 1790737451.013691 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2a870): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a870): Mirror 3: 0xa6cf50a8 (stale) Mirror 4: 0xe1119642 (stale) Mirror 5: 0xc432edd (stale) CRC-32 checksum value for chunk [0, 0x2a870): Mirror 1: 0xbc9d6e3f Mirror 6: 0xbc9d6e3f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a870: skipped 000449 1790737452.444772 write 0x015280 thru 0x02122c (0xbfad bytes) 000450 1790737452.522030 mapwrite 0x0367f0 thru 0x0397b8 (0x2fc9 bytes) 000452 1790737452.589930 fallocate 0x02d4c5 thru 0x03a050 (0xcb8c bytes) 000453 1790737452.596294 mapread 0x01d58c thru 0x02bedd (0xe952 bytes) 000454 1790737452.616629 read 0x003e6f thru 0x006308 (0x249a bytes) 000455 1790737452.625519 trunc from 0x022f0b to 0x02b2ad (0x83a3 bytes) 000456 1790737452.630042 read 0x01b75a thru 0x01c87a (0x1121 bytes) 000457 1790737452.630886 mapwrite 0x00f179 thru 0x01cbd3 (0xda5b bytes) 000458 1790737452.695368 read 0x0216c1 thru 0x02e765 (0xd0a5 bytes) 000459 1790737452.699417 fallocate 0x0315c7 thru 0x03c102 (0xab3c bytes) 000460 1790737452.706316 fallocate 0x027521 thru 0x034fdd (0xdabd bytes) 000461 1790737452.712961 punch from 0x00d826 to 0x012ab0 (0x528b bytes) 000462 1790737452.718664 read 0x01db00 thru 0x029df0 (0xc2f1 bytes) 000463 1790737452.719587 fallocate 0x02311c thru 0x02f677 (0xc55c bytes) 000464 1790737452.735748 trunc from 0x03977f to 0x03c102 (0x2984 bytes) 000465 1790737452.742008 trunc from 0x03c103 to 0x0322bc (0x9e46 bytes) 000466 1790737452.759280 mapwrite 0x033a08 thru 0x03ffff (0xc5f8 bytes) 000467 1790737452.877670 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xa602ba6c (stale) Mirror 4: 0xaaed578a (stale) Mirror 5: 0x8d064144 (stale) Mirror 6: 0x74fe8060 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x495736b5 Mirror 7: 0x495736b5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000468 1790737454.149201 mapread 0x02d5f7 thru 0x02ecfc (0x1706 bytes) 000469 1790737454.191491 mapread 0x025708 thru 0x0297d1 (0x40ca bytes) 000470 1790737454.234888 mapread 0x00eabd thru 0x016747 (0x7c8b bytes) 000471 1790737454.265757 mapwrite 0x00ef9c thru 0x01c29c (0xd301 bytes) 000472 1790737454.401791 mapread 0x02e9ee thru 0x037772 (0x8d85 bytes) 000473 1790737454.453360 mapwrite 0x0164a9 thru 0x02088a (0xa3e2 bytes) 000474 1790737454.503745 punch from 0x01a74b to 0x01e95c (0x4212 bytes) 000475 1790737454.519331 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xaaed578a (stale) Mirror 5: 0x8d064144 (stale) Mirror 6: 0x74fe8060 (stale) Mirror 7: 0x495736b5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xaaed578a (stale) Mirror 5: 0x8d064144 (stale) Mirror 6: 0x74fe8060 (stale) Mirror 7: 0x495736b5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000476 1790737455.028383 write 0x01d409 thru 0x024468 (0x7060 bytes) 000477 1790737455.045630 mapread 0x03eabb thru 0x03ffff (0x1545 bytes) 000478 1790737455.058404 trunc from 0x028379 to 0x031b3b (0x97c3 bytes) 000479 1790737455.062957 write 0x022448 thru 0x02bcce (0x9887 bytes) 000480 1790737455.068305 mapwrite 0x01879d thru 0x02568e (0xcef2 bytes) 000481 1790737455.098116 mapwrite 0x016133 thru 0x01c1a3 (0x6071 bytes) 000482 1790737455.112152 trunc from 0x040000 to 0x00ac21 (0x353de bytes) 000483 1790737455.124698 fallocate 0x005958 thru 0x00cb02 (0x71ab bytes) 000484 1790737455.129400 read 0x0064ab thru 0x00948b (0x2fe1 bytes) 000485 1790737455.135976 read 0x002879 thru 0x0070df (0x4867 bytes) 000486 1790737455.144602 mapread 0x003aa4 thru 0x006d32 (0x328f bytes) 000487 1790737455.145669 fallocate 0x030f75 thru 0x03b4ac (0xa538 bytes) 000488 1790737455.150540 write 0x0010a5 thru 0x005edb (0x4e37 bytes) 000489 1790737455.152323 punch from 0x002cd4 to 0x00cb02 (0x9e2f bytes) 000490 1790737455.163582 fallocate 0x02ae45 thru 0x0329ac (0x7b68 bytes) 000491 1790737455.167959 read 0x01481a thru 0x02083c (0xc023 bytes) 000492 1790737455.181622 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x329ad): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x329ad): Mirror 4: 0x3d9d0889 (stale) Mirror 5: 0xd468209b (stale) Mirror 6: 0xd18ed998 (stale) Mirror 7: 0x7ab6d9a4 (stale) CRC-32 checksum value for chunk [0, 0x329ad): Mirror 1: 0x381dd782 Mirror 8: 0x381dd782 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x329ad: skipped 000494 1790737456.112979 read 0x01a713 thru 0x01e20c (0x3afa bytes) 000495 1790737456.141460 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x329ad): Mirror 5: 0xd468209b (stale) Mirror 6: 0xd18ed998 (stale) Mirror 7: 0x7ab6d9a4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x329ad): Mirror 5: 0xd468209b (stale) Mirror 6: 0xd18ed998 (stale) Mirror 7: 0x7ab6d9a4 (stale) CRC-32 checksum value for chunk [0, 0x329ad): Mirror 1: 0x381dd782 Mirror 8: 0x381dd782 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x329ad: skipped 000496 1790737457.064426 punch from 0x01bb87 to 0x0223e7 (0x6861 bytes) 000497 1790737457.121707 mapwrite 0x01b327 thru 0x027402 (0xc0dc bytes) 000498 1790737457.233393 read 0x01cb62 thru 0x01f920 (0x2dbf bytes) 000499 1790737457.234230 punch from 0x02b3f2 to 0x0329ac (0x75bb bytes) 000500 1790737457.241232 read 0x0078d5 thru 0x00e13c (0x6868 bytes) 000501 1790737457.271758 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x329ad): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x329ad): Mirror 5: 0xd468209b (stale) Mirror 6: 0xd18ed998 (stale) Mirror 7: 0x7ab6d9a4 (stale) Mirror 8: 0x381dd782 (stale) CRC-32 checksum value for chunk [0, 0x329ad): Mirror 1: 0xb8cff4ec Mirror 9: 0xb8cff4ec lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x329ad: skipped 000502 1790737458.495515 read 0x02348a thru 0x032974 (0xf4eb bytes) 000503 1790737458.512935 trunc from 0x010f5f to 0x01830a (0x73ac bytes) 000504 1790737458.567553 mapwrite 0x02f6fe thru 0x03a721 (0xb024 bytes) 000505 1790737458.739901 fallocate 0x02e2d2 thru 0x0314ab (0x31da bytes) 000506 1790737458.759894 read 0x015181 thru 0x0165db (0x145b bytes) 000507 1790737458.827418 trunc from 0x00c23a to 0x01ab00 (0xe8c7 bytes) 000508 1790737458.842573 trunc from 0x03a722 to 0x002d04 (0x37a1d bytes) 000509 1790737458.879760 fallocate 0x00c1df thru 0x00e08e (0x1eb0 bytes) 000510 1790737458.892320 read 0x0016b9 thru 0x002d04 (0x164c bytes) 000511 1790737458.893076 punch from 0x000f91 to 0x002d04 (0x1d74 bytes) 000512 1790737458.920441 fallocate 0x01ea56 thru 0x024604 (0x5baf bytes) 000515 1790737458.935099 mapwrite 0x01837f thru 0x026810 (0xe492 bytes) 000516 1790737459.062461 mapwrite 0x03e04d thru 0x03e0d2 (0x86 bytes) 000517 1790737459.109423 fallocate 0x02d93f thru 0x03358d (0x5c4f bytes) 000518 1790737459.114617 mapread 0x01cb11 thru 0x024472 (0x7962 bytes) 000519 1790737459.119900 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3e0d3): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e0d3): Mirror 5: 0x9795d930 (stale) Mirror 6: 0x29843c2 (stale) Mirror 7: 0x42e8910c (stale) Mirror 8: 0xd23327b3 (stale) Mirror 9: 0xc388b4c5 (stale) CRC-32 checksum value for chunk [0, 0x3e0d3): Mirror 1: 0x7f190e Mirror 10: 0x7f190e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e0d3: skipped 000520 1790737460.110808 mapread 0x01def6 thru 0x02dc96 (0xfda1 bytes) 000521 1790737460.216994 trunc from 0x039ced to 0x03e0d2 (0x43e6 bytes) 000523 1790737460.292941 write 0x024d60 thru 0x026601 (0x18a2 bytes) 000524 1790737460.313942 mapwrite 0x00c750 thru 0x0196bb (0xcf6c bytes) 000525 1790737460.362217 read 0x0254e2 thru 0x02ebc0 (0x96df bytes) 000526 1790737460.377645 mapread 0x034a8a thru 0x0367c7 (0x1d3e bytes) 000527 1790737460.390715 write 0x00d2e8 thru 0x017859 (0xa572 bytes) 000528 1790737460.392945 punch from 0x03bed9 to 0x03e0d2 (0x21fa bytes) 000529 1790737460.402279 write 0x00e506 thru 0x014344 (0x5e3f bytes) 000530 1790737460.403585 trunc from 0x012850 to 0x013d0e (0x14bf bytes) 000531 1790737460.427691 mapwrite 0x0385c0 thru 0x03c988 (0x43c9 bytes) 000533 1790737460.456660 fallocate 0x00d44a thru 0x0101b8 (0x2d6f bytes) 000534 1790737460.463575 trunc from 0x03e0d3 to 0x019020 (0x250b2 bytes) 000535 1790737460.564920 mapwrite 0x00214c thru 0x00680e (0x46c3 bytes) 000536 1790737460.604030 MIRROR_RESYNC 7 mirrors 000537 1790737462.091915 fallocate 0x034901 thru 0x03ffff (0xb6ff bytes) 000538 1790737462.129015 punch from 0x036f1c to 0x03ffff (0x90e4 bytes) 000539 1790737462.135610 read 0x032a54 thru 0x03baf1 (0x909e bytes) 000540 1790737462.149297 punch from 0x00ba39 to 0x01538d (0x9955 bytes) 000541 1790737462.154504 mapwrite 0x034318 thru 0x03ffff (0xbce8 bytes) 000543 1790737462.203739 fallocate 0x013826 thru 0x01ddfa (0xa5d5 bytes) 000544 1790737462.207720 trunc from 0x040000 to 0x01cc0f (0x233f0 bytes) 000545 1790737462.230653 trunc from 0x00d384 to 0x00d568 (0x1e5 bytes) 000546 1790737462.238633 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1cc10): Mirror 5: 0x9f87c03c (stale) Mirror 6: 0x9f87c03c (stale) Mirror 7: 0x9f87c03c (stale) Mirror 8: 0x9f87c03c (stale) Mirror 10: 0x9f87c03c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cc10): Mirror 5: 0x9f87c03c (stale) Mirror 6: 0x9f87c03c (stale) Mirror 7: 0x9f87c03c (stale) Mirror 8: 0x9f87c03c (stale) Mirror 10: 0x9f87c03c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cc10: skipped 000547 1790737462.654801 trunc from 0x01cc10 to 0x015dca (0x6e45 bytes) 000548 1790737462.674747 trunc from 0x015dcb to 0x02ef06 (0x1913c bytes) 000549 1790737462.694439 read 0x009af2 thru 0x013d72 (0xa281 bytes) 000550 1790737462.704169 read 0x01ca7d thru 0x02c1b1 (0xf735 bytes) 000552 1790737462.714281 write 0x031e04 thru 0x03b6c0 (0x98bd bytes) 000553 1790737462.715924 trunc from 0x03b6c1 to 0x025b2f (0x15b91 bytes) 000554 1790737462.733310 trunc from 0x00c5b9 to 0x00d5f6 (0x103e bytes) 000555 1790737462.740209 trunc from 0x025b30 to 0x03e056 (0x18527 bytes) 000556 1790737462.760000 fallocate 0x036312 thru 0x03b251 (0x4f40 bytes) 000557 1790737462.770420 read 0x0193d2 thru 0x01b62c (0x225b bytes) 000558 1790737462.772655 fallocate 0x01b565 thru 0x026421 (0xaebd bytes) 000559 1790737462.782922 MIRROR_RESYNC 6 mirrors 000560 1790737464.344239 punch from 0x01c939 to 0x01ca56 (0x11e bytes) 000561 1790737464.404518 mapread 0x025c7a thru 0x02e093 (0x841a bytes) 000563 1790737464.450726 trunc from 0x03e057 to 0x006ad5 (0x37581 bytes) 000564 1790737464.471724 write 0x039ebd thru 0x03ffff (0x6143 bytes) 000565 1790737464.474738 trunc from 0x01624c to 0x01c378 (0x612d bytes) 000566 1790737464.479881 write 0x003763 thru 0x0078e9 (0x4187 bytes) 000567 1790737464.495932 read 0x02eed0 thru 0x02f2c1 (0x3f2 bytes) 000568 1790737464.506677 trunc from 0x040000 to 0x0176e1 (0x2891e bytes) 000569 1790737464.548982 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x176e2): Mirror 5: 0x52840321 (stale) Mirror 6: 0x52840321 (stale) Mirror 7: 0x52840321 (stale) Mirror 10: 0x52840321 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x176e2): Mirror 5: 0x52840321 (stale) Mirror 6: 0x52840321 (stale) Mirror 7: 0x52840321 (stale) Mirror 10: 0x52840321 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x176e2: skipped 000570 1790737465.007418 trunc from 0x0176e2 to 0x017eaf (0x7ce bytes) 000571 1790737465.032689 trunc from 0x00ea6e to 0x014015 (0x55a8 bytes) 000572 1790737465.041920 read 0x007a1e thru 0x009c30 (0x2213 bytes) 000573 1790737465.049443 fallocate 0x03c2ad thru 0x03ffff (0x3d53 bytes) 000574 1790737465.054382 fallocate 0x02ce28 thru 0x03ae7c (0xe055 bytes) 000575 1790737465.059799 mapread 0x02a5c2 thru 0x038fcd (0xea0c bytes) 000576 1790737465.108646 punch from 0x039051 to 0x03ffff (0x6faf bytes) 000577 1790737465.128171 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x80316b42 (stale) Mirror 6: 0x80316b42 (stale) Mirror 7: 0x80316b42 (stale) Mirror 10: 0x80316b42 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1c39e14c Mirror 11: 0x1c39e14c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000578 1790737466.003354 mapwrite 0x002133 thru 0x00e8a6 (0xc774 bytes) 000581 1790737466.166756 read 0x014e95 thru 0x01f66f (0xa7db bytes) 000582 1790737466.190519 write 0x00582b thru 0x006fe9 (0x17bf bytes) 000583 1790737466.196634 mapwrite 0x0177fb thru 0x018294 (0xa9a bytes) 000584 1790737466.219534 mapwrite 0x028b2a thru 0x02ee1d (0x62f4 bytes) 000585 1790737466.254514 mapread 0x00f2ea thru 0x01799c (0x86b3 bytes) 000586 1790737466.264791 fallocate 0x020339 thru 0x02d956 (0xd61e bytes) 000587 1790737466.272637 mapread 0x0216b1 thru 0x0243de (0x2d2e bytes) 000588 1790737466.273967 fallocate 0x036a5d thru 0x03cf5f (0x6503 bytes) 000590 1790737466.278964 write 0x026ca1 thru 0x032e2b (0xc18b bytes) 000591 1790737466.285344 fallocate 0x03e293 thru 0x03ffff (0x1d6d bytes) 000592 1790737466.290144 mapwrite 0x039ee2 thru 0x03ffff (0x611e bytes) 000593 1790737466.329742 fallocate 0x0032b8 thru 0x00f695 (0xc3de bytes) 000594 1790737466.334527 trunc from 0x029775 to 0x02bf0a (0x2796 bytes) 000595 1790737466.339405 mapread 0x027c53 thru 0x02aed4 (0x3282 bytes) 000596 1790737466.345520 write 0x036a8a thru 0x03c665 (0x5bdc bytes) 000597 1790737466.350754 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x80316b42 (stale) Mirror 6: 0x80316b42 (stale) Mirror 7: 0x80316b42 (stale) Mirror 10: 0x80316b42 (stale) Mirror 11: 0x1c39e14c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x531cbe67 Mirror 12: 0x531cbe67 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000598 1790737467.435437 trunc from 0x040000 to 0x03fb43 (0x4bc bytes) 000599 1790737467.520892 read 0x012830 thru 0x012ebb (0x68c bytes) 000600 1790737467.526743 punch from 0x01995a to 0x021b31 (0x81d8 bytes) 000601 1790737467.531940 mapwrite 0x0283df thru 0x02c7a2 (0x43c4 bytes) 000602 1790737467.604863 mapwrite 0x032716 thru 0x03ea92 (0xc37d bytes) 000604 1790737467.678961 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3fb44): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb44): Mirror 5: 0x986b3143 (stale) Mirror 6: 0x986b3143 (stale) Mirror 7: 0x986b3143 (stale) Mirror 10: 0x986b3143 (stale) Mirror 11: 0x7580d9b1 (stale) Mirror 12: 0x28f5297f (stale) CRC-32 checksum value for chunk [0, 0x3fb44): Mirror 1: 0x532fffb8 Mirror 13: 0x532fffb8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb44: skipped 000605 1790737468.797701 write 0x01e8bf thru 0x0277c7 (0x8f09 bytes) 000606 1790737468.874963 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3fb44): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb44): Mirror 5: 0x986b3143 (stale) Mirror 6: 0x986b3143 (stale) Mirror 7: 0x986b3143 (stale) Mirror 10: 0x986b3143 (stale) Mirror 11: 0x7580d9b1 (stale) Mirror 12: 0x28f5297f (stale) Mirror 13: 0x532fffb8 (stale) CRC-32 checksum value for chunk [0, 0x3fb44): Mirror 1: 0xaeb6dee2 Mirror 14: 0xaeb6dee2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb44: skipped 000607 1790737470.214365 fallocate 0x00b3b1 thru 0x013d10 (0x8960 bytes) 000609 1790737470.329358 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3fb44): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb44): Mirror 5: 0x986b3143 (stale) Mirror 6: 0x986b3143 (stale) Mirror 7: 0x986b3143 (stale) Mirror 10: 0x986b3143 (stale) Mirror 11: 0x7580d9b1 (stale) Mirror 12: 0x28f5297f (stale) Mirror 13: 0x532fffb8 (stale) Mirror 14: 0xaeb6dee2 (stale) CRC-32 checksum value for chunk [0, 0x3fb44): Mirror 1: 0xaeb6dee2 Mirror 15: 0xaeb6dee2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb44: skipped 000610 1790737471.747725 read 0x03d746 thru 0x03fb43 (0x23fe bytes) 000611 1790737471.761668 mapwrite 0x032fe9 thru 0x03ffff (0xd017 bytes) 000612 1790737471.881525 mapread 0x03934c thru 0x03db8f (0x4844 bytes) 000613 1790737471.882738 write 0x018be2 thru 0x028094 (0xf4b3 bytes) 000614 1790737471.893422 mapwrite 0x008c97 thru 0x0138a3 (0xac0d bytes) 000615 1790737471.935523 read 0x03cb14 thru 0x03cd40 (0x22d bytes) 000617 1790737471.942272 MIRROR_RESYNC 10 mirrors 000618 1790737474.066927 fallocate 0x01d655 thru 0x026789 (0x9135 bytes) 000619 1790737474.115288 write 0x00fa77 thru 0x01dea0 (0xe42a bytes) 000620 1790737474.126788 mapread 0x01d8c1 thru 0x020837 (0x2f77 bytes) 000622 1790737474.154367 mapread 0x032bd6 thru 0x033cea (0x1115 bytes) 000624 1790737474.169510 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xcd205525 (stale) Mirror 6: 0xcd205525 (stale) Mirror 7: 0xcd205525 (stale) Mirror 10: 0xcd205525 (stale) Mirror 11: 0xcd205525 (stale) Mirror 12: 0xcd205525 (stale) Mirror 13: 0xcd205525 (stale) Mirror 15: 0xcd205525 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xcd205525 (stale) Mirror 6: 0xcd205525 (stale) Mirror 7: 0xcd205525 (stale) Mirror 10: 0xcd205525 (stale) Mirror 11: 0xcd205525 (stale) Mirror 12: 0xcd205525 (stale) Mirror 13: 0xcd205525 (stale) Mirror 15: 0xcd205525 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000626 1790737474.842455 trunc from 0x040000 to 0x00eb81 (0x3147e bytes) 000627 1790737474.875938 mapread 0x0086e7 thru 0x00eb81 (0x649b bytes) 000630 1790737474.912881 MIRROR_RESYNC 9 mirrors 000631 1790737476.605795 mapwrite 0x02d075 thru 0x0320af (0x503b bytes) 000632 1790737476.771105 trunc from 0x0320b0 to 0x0396bc (0x760d bytes) 000633 1790737476.811802 trunc from 0x0396bd to 0x036121 (0x359b bytes) 000634 1790737476.862606 trunc from 0x00a38f to 0x016d45 (0xc9b7 bytes) 000635 1790737476.876757 mapwrite 0x037814 thru 0x038cc4 (0x14b1 bytes) 000636 1790737476.948418 trunc from 0x038cc5 to 0x00dc91 (0x2b033 bytes) 000637 1790737476.975804 trunc from 0x00dc92 to 0x02a9e3 (0x1cd52 bytes) 000638 1790737476.994336 write 0x03d788 thru 0x03ffff (0x2878 bytes) 000639 1790737476.996525 read 0x00789f thru 0x0079b9 (0x11b bytes) 000640 1790737477.002735 trunc from 0x040000 to 0x0332f1 (0xcd0e bytes) 000641 1790737477.025403 write 0x00b07d thru 0x01349b (0x841f bytes) 000642 1790737477.043755 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x332f2): Mirror 6: 0x6db97c15 (stale) Mirror 7: 0x6db97c15 (stale) Mirror 10: 0x6db97c15 (stale) Mirror 11: 0x6db97c15 (stale) Mirror 12: 0x6db97c15 (stale) Mirror 13: 0x6db97c15 (stale) Mirror 15: 0x6db97c15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x332f2): Mirror 6: 0x6db97c15 (stale) Mirror 7: 0x6db97c15 (stale) Mirror 10: 0x6db97c15 (stale) Mirror 11: 0x6db97c15 (stale) Mirror 12: 0x6db97c15 (stale) Mirror 13: 0x6db97c15 (stale) Mirror 15: 0x6db97c15 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x332f2: skipped 000643 1790737477.708806 read 0x02914d thru 0x0332f1 (0xa1a5 bytes) 000644 1790737477.724134 read 0x020bfb thru 0x02a288 (0x968e bytes) 000645 1790737477.753226 trunc from 0x0280a4 to 0x02d780 (0x56dd bytes) 000646 1790737477.765860 read 0x025a67 thru 0x0332f1 (0xd88b bytes) 000647 1790737477.815592 mapread 0x01bade thru 0x01c329 (0x84c bytes) 000648 1790737477.817679 read 0x022553 thru 0x030503 (0xdfb1 bytes) 000649 1790737477.818901 write 0x01fafb thru 0x026d16 (0x721c bytes) 000650 1790737477.834469 trunc from 0x0332f2 to 0x02d86a (0x5a87 bytes) 000651 1790737477.886213 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2d86b): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d86b): Mirror 6: 0x2551b457 (stale) Mirror 7: 0x2551b457 (stale) Mirror 10: 0x2551b457 (stale) Mirror 11: 0x2551b457 (stale) Mirror 12: 0x2551b457 (stale) Mirror 13: 0x2551b457 (stale) Mirror 15: 0x2551b457 (stale) CRC-32 checksum value for chunk [0, 0x2d86b): Mirror 1: 0xe3c8db03 Mirror 16: 0xe3c8db03 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d86b: skipped 000652 1790737478.948342 mapread 0x029ce2 thru 0x02d86a (0x3b89 bytes) 000653 1790737478.966222 trunc from 0x02d86b to 0x03322a (0x59c0 bytes) 000654 1790737479.009056 write 0x00910f thru 0x00b0b9 (0x1fab bytes) 000655 1790737479.022237 MIRROR_RESYNC 9 mirrors 000656 1790737481.299591 write 0x017144 thru 0x025903 (0xe7c0 bytes) 000658 1790737481.340060 trunc from 0x03322b to 0x00a5ff (0x28c2b bytes) 000660 1790737481.371936 trunc from 0x00a600 to 0x025217 (0x1ac18 bytes) 000661 1790737481.393471 read 0x016aa3 thru 0x0204f4 (0x9a52 bytes) 000662 1790737481.408688 trunc from 0x02497c to 0x025217 (0x89c bytes) 000663 1790737481.418206 MIRROR_RESYNC 9 mirrors 000665 1790737483.009385 trunc from 0x011eab to 0x01aa66 (0x8bbc bytes) 000666 1790737483.109111 fallocate 0x02cfe7 thru 0x0336a3 (0x66bd bytes) 000667 1790737483.121174 read 0x01261a thru 0x01b88e (0x9275 bytes) 000668 1790737483.127506 read 0x021290 thru 0x023449 (0x21ba bytes) 000669 1790737483.143390 trunc from 0x025218 to 0x01b074 (0xa1a3 bytes) 000670 1790737483.167795 trunc from 0x01b075 to 0x038f5d (0x1dee9 bytes) 000671 1790737483.180132 mapread 0x01e45f thru 0x02d4e2 (0xf084 bytes) 000672 1790737483.218456 read 0x032dcd thru 0x038f5d (0x6191 bytes) 000673 1790737483.219056 mapread 0x01bd32 thru 0x022238 (0x6507 bytes) 000675 1790737483.226314 write 0x02fbc4 thru 0x02ffb3 (0x3f0 bytes) 000676 1790737483.227419 mapwrite 0x006992 thru 0x00e6c2 (0x7d31 bytes) 000677 1790737483.249427 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x38f5e): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38f5e): Mirror 6: 0xdc0eb18d (stale) Mirror 7: 0xdc0eb18d (stale) Mirror 10: 0xdc0eb18d (stale) Mirror 11: 0xdc0eb18d (stale) Mirror 12: 0xdc0eb18d (stale) Mirror 13: 0xdc0eb18d (stale) Mirror 15: 0xdc0eb18d (stale) Mirror 16: 0xdc0eb18d (stale) CRC-32 checksum value for chunk [0, 0x38f5e): Mirror 1: 0x240ed534 Mirror 17: 0x240ed534 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38f5e: skipped 000678 1790737484.378361 fallocate 0x02b5dd thru 0x02fdb4 (0x47d8 bytes) 000679 1790737484.436230 trunc from 0x038f5e to 0x028ec0 (0x1009d bytes) 000680 1790737484.478758 read 0x001562 thru 0x007663 (0x6102 bytes) 000681 1790737484.502303 write 0x01ebce thru 0x02812b (0x955e bytes) 000682 1790737484.518228 mapwrite 0x02e34a thru 0x02ea2b (0x6e2 bytes) 000683 1790737484.569043 trunc from 0x023459 to 0x02ea2b (0xb5d3 bytes) 000685 1790737484.580552 mapread 0x00838a thru 0x00e729 (0x63a0 bytes) 000686 1790737484.598052 write 0x020f79 thru 0x02bcb8 (0xad40 bytes) 000687 1790737484.604434 write 0x0202c2 thru 0x021fdf (0x1d1e bytes) 000688 1790737484.606688 mapread 0x02329c thru 0x0275b8 (0x431d bytes) 000689 1790737484.608760 punch from 0x0139b6 to 0x017917 (0x3f62 bytes) 000690 1790737484.614290 mapwrite 0x01467c thru 0x02237c (0xdd01 bytes) 000691 1790737484.676137 mapwrite 0x0311aa thru 0x036f91 (0x5de8 bytes) 000692 1790737484.769849 trunc from 0x01212d to 0x01eebf (0xcd93 bytes) 000693 1790737484.787277 MIRROR_RESYNC 10 mirrors 000694 1790737487.594175 punch from 0x02326a to 0x02bf3f (0x8cd6 bytes) 000695 1790737487.629998 fallocate 0x0244db thru 0x02b0f0 (0x6c16 bytes) 000696 1790737487.641446 trunc from 0x036f92 to 0x01f7a8 (0x177e9 bytes) 000697 1790737487.664110 read 0x01e897 thru 0x01f7a8 (0xf12 bytes) 000698 1790737487.669725 mapwrite 0x010a67 thru 0x01b11e (0xa6b8 bytes) 000700 1790737487.718554 read 0x01f6a8 thru 0x01f7a8 (0x101 bytes) 000701 1790737487.719372 punch from 0x00f7a6 to 0x00fada (0x335 bytes) 000702 1790737487.723811 mapwrite 0x02811e thru 0x03229d (0xa180 bytes) 000703 1790737487.773482 MIRROR_RESYNC 10 mirrors 000704 1790737490.254057 mapwrite 0x0326ca thru 0x03e2b1 (0xbbe8 bytes) 000705 1790737490.342591 mapwrite 0x0077a6 thru 0x00b81e (0x4079 bytes) 000706 1790737490.387768 punch from 0x0027d3 to 0x00bd38 (0x9566 bytes) 000707 1790737490.401820 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3e2b2): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e2b2): Mirror 6: 0x454d994d (stale) Mirror 7: 0x454d994d (stale) Mirror 10: 0x454d994d (stale) Mirror 11: 0x454d994d (stale) Mirror 12: 0x454d994d (stale) Mirror 13: 0x454d994d (stale) Mirror 15: 0x454d994d (stale) Mirror 16: 0x454d994d (stale) Mirror 17: 0x454d994d (stale) CRC-32 checksum value for chunk [0, 0x3e2b2): Mirror 1: 0x400527f2 Mirror 18: 0x400527f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e2b2: skipped 000708 1790737491.406238 mapread 0x009452 thru 0x00c96e (0x351d bytes) 000709 1790737491.435673 fallocate 0x013e35 thru 0x017c58 (0x3e24 bytes) 000710 1790737491.472732 punch from 0x0074da to 0x00cbcd (0x56f4 bytes) 000711 1790737491.477986 write 0x03ccae thru 0x03ffff (0x3352 bytes) 000712 1790737491.486507 write 0x022128 thru 0x02d976 (0xb84f bytes) 000713 1790737491.495580 trunc from 0x0319b1 to 0x031e45 (0x495 bytes) 000714 1790737491.498925 mapread 0x012a33 thru 0x01c50e (0x9adc bytes) 000715 1790737491.517367 trunc from 0x040000 to 0x03ece6 (0x1319 bytes) 000717 1790737491.556007 fallocate 0x014252 thru 0x022a81 (0xe830 bytes) 000718 1790737491.560711 mapread 0x03eaeb thru 0x03ece6 (0x1fc bytes) 000719 1790737491.561941 trunc from 0x018f26 to 0x025a67 (0xcb42 bytes) 000720 1790737491.576206 read 0x001ece thru 0x00b246 (0x9379 bytes) 000721 1790737491.583953 trunc from 0x03ece7 to 0x02215f (0x1cb87 bytes) 000722 1790737491.602545 trunc from 0x022160 to 0x001af7 (0x20668 bytes) 000723 1790737491.614554 trunc from 0x001af8 to 0x00ed9f (0xd2a8 bytes) 000724 1790737491.630375 read 0x00b975 thru 0x00ed9f (0x342b bytes) 000725 1790737491.637723 read 0x0085f5 thru 0x0099ef (0x13fb bytes) 000728 1790737491.639311 write 0x02db9b thru 0x02fa11 (0x1e77 bytes) 000729 1790737491.641502 punch from 0x016f24 to 0x020f30 (0xa00d bytes) 000730 1790737491.645280 fallocate 0x03fa67 thru 0x03ffff (0x599 bytes) 000731 1790737491.649652 mapread 0x02d841 thru 0x0387fa (0xafba bytes) 000732 1790737491.671338 mapwrite 0x025fd6 thru 0x02656f (0x59a bytes) 000733 1790737491.683681 read 0x007e34 thru 0x00f104 (0x72d1 bytes) 000734 1790737491.688159 MIRROR_RESYNC 11 mirrors 000735 1790737494.424714 read 0x03ec5c thru 0x03ffff (0x13a4 bytes) 000736 1790737494.447238 mapwrite 0x0089a2 thru 0x0101aa (0x7809 bytes) 000737 1790737494.616705 mapwrite 0x00c948 thru 0x0112eb (0x49a4 bytes) 000738 1790737494.649918 trunc from 0x040000 to 0x012611 (0x2d9ee bytes) 000740 1790737494.675191 read 0x008dc0 thru 0x00e9d1 (0x5c12 bytes) 000741 1790737494.675754 read 0x008bee thru 0x00f6a9 (0x6abc bytes) 000742 1790737494.676240 write 0x02f307 thru 0x03e9b2 (0xf6ac bytes) 000744 1790737494.680113 fallocate 0x0178fa thru 0x0236f9 (0xbe00 bytes) 000745 1790737494.686649 punch from 0x00c1c1 to 0x019dad (0xdbed bytes) 000746 1790737494.691960 trunc from 0x03e9b3 to 0x02896a (0x16048 bytes) 000747 1790737494.705271 write 0x03b9b4 thru 0x03ffff (0x464c bytes) 000749 1790737494.708462 fallocate 0x029a35 thru 0x02ce1e (0x33ea bytes) 000750 1790737494.733663 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x2cf83b35 (stale) Mirror 7: 0x2cf83b35 (stale) Mirror 10: 0x2cf83b35 (stale) Mirror 11: 0x2cf83b35 (stale) Mirror 12: 0x2cf83b35 (stale) Mirror 13: 0x2cf83b35 (stale) Mirror 15: 0x2cf83b35 (stale) Mirror 16: 0x2cf83b35 (stale) Mirror 17: 0x2cf83b35 (stale) Mirror 18: 0x2cf83b35 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x67a1aa47 Mirror 19: 0x67a1aa47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000751 1790737495.874477 trunc from 0x004dea to 0x00d3e1 (0x85f8 bytes) 000752 1790737495.948479 fallocate 0x01f5ca thru 0x02c12f (0xcb66 bytes) 000753 1790737495.955433 mapwrite 0x037f92 thru 0x03e13b (0x61aa bytes) 000754 1790737496.032552 punch from 0x036407 to 0x03f741 (0x933b bytes) 000755 1790737496.046985 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x2cf83b35 (stale) Mirror 7: 0x2cf83b35 (stale) Mirror 10: 0x2cf83b35 (stale) Mirror 11: 0x2cf83b35 (stale) Mirror 12: 0x2cf83b35 (stale) Mirror 13: 0x2cf83b35 (stale) Mirror 15: 0x2cf83b35 (stale) Mirror 16: 0x2cf83b35 (stale) Mirror 17: 0x2cf83b35 (stale) Mirror 18: 0x2cf83b35 (stale) Mirror 19: 0x67a1aa47 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x726d5a6c Mirror 20: 0x726d5a6c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000756 1790737497.109148 read 0x007018 thru 0x00aebb (0x3ea4 bytes) 000757 1790737497.119921 mapread 0x00867a thru 0x0117e9 (0x9170 bytes) 000758 1790737497.144520 fallocate 0x0225a2 thru 0x02c716 (0xa175 bytes) 000759 1790737497.193346 trunc from 0x040000 to 0x01f9ad (0x20652 bytes) 000760 1790737497.220679 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x1f9ae): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f9ae): Mirror 6: 0xe20c0950 (stale) Mirror 7: 0xe20c0950 (stale) Mirror 10: 0xe20c0950 (stale) Mirror 11: 0xe20c0950 (stale) Mirror 12: 0xe20c0950 (stale) Mirror 13: 0xe20c0950 (stale) Mirror 15: 0xe20c0950 (stale) Mirror 16: 0xe20c0950 (stale) Mirror 17: 0xe20c0950 (stale) Mirror 18: 0xe20c0950 (stale) Mirror 19: 0xb10612d4 (stale) Mirror 20: 0xe20c0950 (stale) CRC-32 checksum value for chunk [0, 0x1f9ae): Mirror 1: 0xe20c0950 Mirror 21: 0xe20c0950 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f9ae: skipped 000761 1790737498.307314 punch from 0x01555f to 0x01f9ad (0xa44f bytes) 000762 1790737498.353265 read 0x00d47c thru 0x01bf46 (0xeacb bytes) 000763 1790737498.363011 mapread 0x00303f thru 0x010298 (0xd25a bytes) 000764 1790737498.379134 mapread 0x0052b1 thru 0x013efb (0xec4b bytes) 000765 1790737498.385807 MIRROR_SPLIT mirror 11 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f9ae): Mirror 6: 0xe20c0950 (stale) Mirror 7: 0xe20c0950 (stale) Mirror 10: 0xe20c0950 (stale) Mirror 11: 0xe20c0950 (stale) Mirror 12: 0xe20c0950 (stale) Mirror 13: 0xe20c0950 (stale) Mirror 15: 0xe20c0950 (stale) Mirror 16: 0xe20c0950 (stale) Mirror 17: 0xe20c0950 (stale) Mirror 18: 0xe20c0950 (stale) Mirror 20: 0xe20c0950 (stale) Mirror 21: 0xe20c0950 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f9ae): Mirror 6: 0xe20c0950 (stale) Mirror 7: 0xe20c0950 (stale) Mirror 10: 0xe20c0950 (stale) Mirror 11: 0xe20c0950 (stale) Mirror 12: 0xe20c0950 (stale) Mirror 13: 0xe20c0950 (stale) Mirror 15: 0xe20c0950 (stale) Mirror 16: 0xe20c0950 (stale) Mirror 17: 0xe20c0950 (stale) Mirror 18: 0xe20c0950 (stale) Mirror 20: 0xe20c0950 (stale) Mirror 21: 0xe20c0950 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f9ae: skipped 000766 1790737499.036375 trunc from 0x003e4a to 0x00dbc1 (0x9d78 bytes) 000767 1790737499.062320 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x1f9ae): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f9ae): Mirror 6: 0xe20c0950 (stale) Mirror 7: 0xe20c0950 (stale) Mirror 10: 0xe20c0950 (stale) Mirror 11: 0xe20c0950 (stale) Mirror 12: 0xe20c0950 (stale) Mirror 13: 0xe20c0950 (stale) Mirror 15: 0xe20c0950 (stale) Mirror 16: 0xe20c0950 (stale) Mirror 17: 0xe20c0950 (stale) Mirror 18: 0xe20c0950 (stale) Mirror 20: 0xe20c0950 (stale) Mirror 21: 0xe20c0950 (stale) CRC-32 checksum value for chunk [0, 0x1f9ae): Mirror 1: 0xe20c0950 Mirror 22: 0xe20c0950 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f9ae: skipped 000768 1790737500.118576 write 0x005ef1 thru 0x00a003 (0x4113 bytes) 000769 1790737500.198169 mapwrite 0x002e08 thru 0x002fc6 (0x1bf bytes) 000770 1790737500.235751 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x1f9ae): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f9ae): Mirror 6: 0xe20c0950 (stale) Mirror 7: 0xe20c0950 (stale) Mirror 10: 0xe20c0950 (stale) Mirror 11: 0xe20c0950 (stale) Mirror 12: 0xe20c0950 (stale) Mirror 13: 0xe20c0950 (stale) Mirror 15: 0xe20c0950 (stale) Mirror 16: 0xe20c0950 (stale) Mirror 17: 0xe20c0950 (stale) Mirror 18: 0xe20c0950 (stale) Mirror 20: 0xe20c0950 (stale) Mirror 21: 0xe20c0950 (stale) Mirror 22: 0xe20c0950 (stale) CRC-32 checksum value for chunk [0, 0x1f9ae): Mirror 1: 0xdf791af Mirror 23: 0xdf791af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f9ae: skipped 000771 1790737501.339639 punch from 0x008a41 to 0x014bc0 (0xc180 bytes) 000772 1790737501.402608 read 0x0138d5 thru 0x0169d1 (0x30fd bytes) 000773 1790737501.408056 fallocate 0x007ca3 thru 0x0151eb (0xd549 bytes) 000775 1790737501.414078 trunc from 0x01f9ae to 0x0280f9 (0x874c bytes) 000776 1790737501.459154 trunc from 0x0280fa to 0x03a27f (0x12186 bytes) 000777 1790737501.467366 read 0x02a95b thru 0x0318ef (0x6f95 bytes) 000778 1790737501.474493 mapread 0x037a0a thru 0x03a27f (0x2876 bytes) 000779 1790737501.475331 punch from 0x037aaa to 0x03a27f (0x27d6 bytes) 000780 1790737501.485585 MIRROR_RESYNC 15 mirrors 000781 1790737504.558088 MIRROR_RESYNC 15 mirrors 000782 1790737505.229646 trunc from 0x03a280 to 0x03c2a8 (0x2029 bytes) 000784 1790737505.277260 read 0x0036d3 thru 0x005680 (0x1fae bytes) 000785 1790737505.282399 read 0x035c99 thru 0x039c74 (0x3fdc bytes) 000786 1790737505.291646 read 0x003086 thru 0x00a0cf (0x704a bytes) 000787 1790737505.292412 punch from 0x008022 to 0x00f95b (0x793a bytes) 000788 1790737505.296680 mapread 0x0054b2 thru 0x012265 (0xcdb4 bytes) 000789 1790737505.308096 read 0x02075c thru 0x02359e (0x2e43 bytes) 000790 1790737505.308703 read 0x035bea thru 0x03c2a8 (0x66bf bytes) 000792 1790737505.309731 write 0x033568 thru 0x03ffff (0xca98 bytes) 000794 1790737505.313171 trunc from 0x040000 to 0x032615 (0xd9ea bytes) 000795 1790737505.345168 trunc from 0x032616 to 0x00e7df (0x23e36 bytes) 000796 1790737505.360034 trunc from 0x00e7e0 to 0x02fbd3 (0x213f4 bytes) 000798 1790737505.375669 trunc from 0x01f1f7 to 0x025577 (0x6381 bytes) 000800 1790737505.379530 fallocate 0x0366f2 thru 0x03ca9c (0x63ab bytes) 000801 1790737505.383188 trunc from 0x03231e to 0x03b290 (0x8f73 bytes) 000802 1790737505.389033 punch from 0x039114 to 0x03ca9c (0x3989 bytes) 000803 1790737505.393538 mapread 0x03604a thru 0x03ca9c (0x6a53 bytes) 000804 1790737505.405086 trunc from 0x026673 to 0x03444d (0xdddb bytes) 000805 1790737505.409787 read 0x00e347 thru 0x015f12 (0x7bcc bytes) 000806 1790737505.413199 MIRROR_RESYNC 15 mirrors 000807 1790737508.880696 write 0x000205 thru 0x00fd77 (0xfb73 bytes) 000808 1790737508.946748 read 0x03607b thru 0x03ca9c (0x6a22 bytes) 000809 1790737508.958294 punch from 0x02137b to 0x029f11 (0x8b97 bytes) 000810 1790737508.971994 mapread 0x02ada9 thru 0x030123 (0x537b bytes) 000811 1790737509.004540 trunc from 0x03ca9d to 0x027f18 (0x14b84 bytes) 000812 1790737509.036391 MIRROR_RESYNC 15 mirrors 000813 1790737511.204027 read 0x0087a1 thru 0x013a18 (0xb278 bytes) 000814 1790737511.214795 mapread 0x01d6a5 thru 0x01ef63 (0x18bf bytes) 000815 1790737511.226873 punch from 0x00d21d to 0x01c337 (0xf11b bytes) 000817 1790737511.304293 trunc from 0x007062 to 0x013100 (0xc09f bytes) 000818 1790737511.320911 trunc from 0x02025e to 0x027f18 (0x7cbb bytes) 000820 1790737511.326877 mapread 0x0148f2 thru 0x02308d (0xe79c bytes) 000822 1790737511.385002 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x27f19): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27f19): Mirror 6: 0x356ccc25 (stale) Mirror 7: 0x356ccc25 (stale) Mirror 10: 0x356ccc25 (stale) Mirror 11: 0x356ccc25 (stale) Mirror 12: 0x356ccc25 (stale) Mirror 13: 0x356ccc25 (stale) Mirror 15: 0x356ccc25 (stale) Mirror 16: 0x356ccc25 (stale) Mirror 17: 0x356ccc25 (stale) Mirror 18: 0x356ccc25 (stale) Mirror 20: 0x356ccc25 (stale) Mirror 21: 0x356ccc25 (stale) Mirror 22: 0x356ccc25 (stale) Mirror 23: 0x356ccc25 (stale) CRC-32 checksum value for chunk [0, 0x27f19): Mirror 1: 0xf0790be0 Mirror 24: 0xf0790be0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27f19: skipped 000823 1790737513.056520 MIRROR_RESYNC 16 mirrors 000824 1790737515.135239 trunc from 0x00cc76 to 0x00eb4a (0x1ed5 bytes) 000825 1790737515.190249 write 0x01e5d5 thru 0x02cc3c (0xe668 bytes) 000826 1790737515.196197 fallocate 0x028638 thru 0x0364b3 (0xde7c bytes) 000827 1790737515.200508 write 0x0281db thru 0x0368d5 (0xe6fb bytes) 000828 1790737515.202256 mapread 0x0313de thru 0x0368d5 (0x54f8 bytes) 000829 1790737515.206183 trunc from 0x025ca1 to 0x0325df (0xc93f bytes) 000830 1790737515.222639 fallocate 0x00d546 thru 0x01c995 (0xf450 bytes) 000831 1790737515.248976 MIRROR_RESYNC 16 mirrors 000832 1790737519.547167 trunc from 0x008498 to 0x00feb9 (0x7a22 bytes) 000833 1790737519.604969 write 0x029788 thru 0x03294a (0x91c3 bytes) 000834 1790737519.617042 write 0x01d25a thru 0x0284b8 (0xb25f bytes) 000835 1790737519.631976 MIRROR_SPLIT mirror 2 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x368d6): Mirror 6: 0xdb4a6b90 (stale) Mirror 10: 0xdb4a6b90 (stale) Mirror 11: 0xdb4a6b90 (stale) Mirror 12: 0xdb4a6b90 (stale) Mirror 13: 0xdb4a6b90 (stale) Mirror 15: 0xdb4a6b90 (stale) Mirror 16: 0xdb4a6b90 (stale) Mirror 17: 0xdb4a6b90 (stale) Mirror 18: 0xdb4a6b90 (stale) Mirror 20: 0xdb4a6b90 (stale) Mirror 21: 0xdb4a6b90 (stale) Mirror 22: 0xdb4a6b90 (stale) Mirror 23: 0xdb4a6b90 (stale) Mirror 24: 0xdb4a6b90 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x368d6): Mirror 6: 0xdb4a6b90 (stale) Mirror 10: 0xdb4a6b90 (stale) Mirror 11: 0xdb4a6b90 (stale) Mirror 12: 0xdb4a6b90 (stale) Mirror 13: 0xdb4a6b90 (stale) Mirror 15: 0xdb4a6b90 (stale) Mirror 16: 0xdb4a6b90 (stale) Mirror 17: 0xdb4a6b90 (stale) Mirror 18: 0xdb4a6b90 (stale) Mirror 20: 0xdb4a6b90 (stale) Mirror 21: 0xdb4a6b90 (stale) Mirror 22: 0xdb4a6b90 (stale) Mirror 23: 0xdb4a6b90 (stale) Mirror 24: 0xdb4a6b90 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x368d6: skipped 000836 1790737520.710751 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x368d6): Mirror 6: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x368d6): Mirror 6: 0xdb4a6b90 (stale) Mirror 10: 0xdb4a6b90 (stale) Mirror 11: 0xdb4a6b90 (stale) Mirror 12: 0xdb4a6b90 (stale) Mirror 13: 0xdb4a6b90 (stale) Mirror 15: 0xdb4a6b90 (stale) Mirror 16: 0xdb4a6b90 (stale) Mirror 17: 0xdb4a6b90 (stale) Mirror 18: 0xdb4a6b90 (stale) Mirror 20: 0xdb4a6b90 (stale) Mirror 21: 0xdb4a6b90 (stale) Mirror 22: 0xdb4a6b90 (stale) Mirror 23: 0xdb4a6b90 (stale) Mirror 24: 0xdb4a6b90 (stale) CRC-32 checksum value for chunk [0, 0x368d6): Mirror 1: 0xf026e504 Mirror 25: 0xf026e504 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x368d6: skipped 000837 1790737522.415061 trunc from 0x0368d6 to 0x0290a8 (0xd82d bytes) 000838 1790737522.478140 trunc from 0x001599 to 0x0024ee (0xf56 bytes) 000839 1790737522.483702 write 0x00bcbe thru 0x0126bc (0x69ff bytes) 000840 1790737522.496070 mapwrite 0x0323a8 thru 0x036752 (0x43ab bytes) 000841 1790737522.555869 fallocate 0x03c2a8 thru 0x03cb63 (0x8bc bytes) 000842 1790737522.561583 fallocate 0x01f7b0 thru 0x028cda (0x952b bytes) 000843 1790737522.571965 punch from 0x00aed7 to 0x0176fd (0xc827 bytes) 000844 1790737522.584433 fallocate 0x00d5ca thru 0x01c728 (0xf15f bytes) 000845 1790737522.590205 trunc from 0x036753 to 0x0304c9 (0x6289 bytes) 000847 1790737522.621345 mapread 0x02b0d1 thru 0x02ba40 (0x970 bytes) 000849 1790737522.640853 fallocate 0x03828c thru 0x03d558 (0x52cd bytes) 000850 1790737522.650974 mapread 0x00808c thru 0x017a03 (0xf978 bytes) 000851 1790737522.680217 write 0x02b972 thru 0x0302a7 (0x4936 bytes) 000852 1790737522.682307 mapwrite 0x000339 thru 0x00470a (0x43d2 bytes) 000853 1790737522.713248 mapread 0x018695 thru 0x01c9f5 (0x4361 bytes) 000854 1790737522.729923 trunc from 0x03d559 to 0x008ca8 (0x348b0 bytes) 000855 1790737522.759352 fallocate 0x02195b thru 0x03049d (0xeb43 bytes) 000857 1790737522.776075 fallocate 0x0141ab thru 0x01f3d3 (0xb229 bytes) 000858 1790737522.785186 read 0x007422 thru 0x01705e (0xfc3d bytes) 000860 1790737522.804059 fallocate 0x01ce83 thru 0x01fd6f (0x2eed bytes) 000861 1790737522.822685 punch from 0x017e61 to 0x019f35 (0x20d5 bytes) 000862 1790737522.834504 write 0x008fdf thru 0x017b21 (0xeb43 bytes) 000863 1790737522.853518 punch from 0x007f01 to 0x01371d (0xb81d bytes) 000864 1790737522.899167 read 0x01fa33 thru 0x01fd6f (0x33d bytes) 000865 1790737522.920911 punch from 0x019ed2 to 0x01db24 (0x3c53 bytes) 000866 1790737522.927898 read 0x00c936 thru 0x00cb88 (0x253 bytes) 000867 1790737522.944935 mapread 0x00b757 thru 0x01209b (0x6945 bytes) 000868 1790737522.950449 mapwrite 0x02c01e thru 0x03aeda (0xeebd bytes) 000869 1790737523.042271 mapwrite 0x024d27 thru 0x02f132 (0xa40c bytes) 000870 1790737523.081466 write 0x01bd9a thru 0x01bf23 (0x18a bytes) 000871 1790737523.086422 trunc from 0x03aedb to 0x01ddc2 (0x1d118 bytes) 000872 1790737523.113355 mapread 0x004614 thru 0x009f3e (0x592b bytes) 000873 1790737523.115141 mapread 0x004b26 thru 0x01082d (0xbd08 bytes) 000874 1790737523.119042 mapread 0x00e93c thru 0x0141df (0x58a4 bytes) 000875 1790737523.122525 mapread 0x014137 thru 0x0177bb (0x3685 bytes) 000876 1790737523.125175 trunc from 0x01b8cd to 0x01ddc2 (0x24f6 bytes) 000877 1790737523.137796 mapread 0x0173f0 thru 0x01ddc2 (0x69d3 bytes) 000878 1790737523.162113 mapread 0x017de1 thru 0x019601 (0x1821 bytes) 000879 1790737523.163510 trunc from 0x00d4d9 to 0x01263c (0x5164 bytes) 000880 1790737523.192317 MIRROR_RESYNC 16 mirrors 000882 1790737527.171457 read 0x0199e5 thru 0x01ddc2 (0x43de bytes) 000883 1790737527.180909 fallocate 0x01a12f thru 0x01ab54 (0xa26 bytes) 000884 1790737527.218001 read 0x0107c7 thru 0x01ddc2 (0xd5fc bytes) 000885 1790737527.234477 MIRROR_SPLIT mirror 15 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ddc3): Mirror 6: 0x4bc22185 (stale) Mirror 10: 0x4bc22185 (stale) Mirror 11: 0x4bc22185 (stale) Mirror 12: 0x4bc22185 (stale) Mirror 13: 0x4bc22185 (stale) Mirror 15: 0x4bc22185 (stale) Mirror 16: 0x4bc22185 (stale) Mirror 17: 0x4bc22185 (stale) Mirror 18: 0x4bc22185 (stale) Mirror 20: 0x4bc22185 (stale) Mirror 21: 0x4bc22185 (stale) Mirror 22: 0x4bc22185 (stale) Mirror 23: 0x4bc22185 (stale) Mirror 24: 0x4bc22185 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ddc3): Mirror 6: 0x4bc22185 (stale) Mirror 10: 0x4bc22185 (stale) Mirror 11: 0x4bc22185 (stale) Mirror 12: 0x4bc22185 (stale) Mirror 13: 0x4bc22185 (stale) Mirror 15: 0x4bc22185 (stale) Mirror 16: 0x4bc22185 (stale) Mirror 17: 0x4bc22185 (stale) Mirror 18: 0x4bc22185 (stale) Mirror 20: 0x4bc22185 (stale) Mirror 21: 0x4bc22185 (stale) Mirror 22: 0x4bc22185 (stale) Mirror 23: 0x4bc22185 (stale) Mirror 24: 0x4bc22185 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ddc3: skipped 000886 1790737528.015808 read 0x008209 thru 0x017cb4 (0xfaac bytes) 000887 1790737528.030916 trunc from 0x0093da to 0x00b482 (0x20a9 bytes) 000888 1790737528.040303 MIRROR_SPLIT mirror 2 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ddc3): Mirror 6: 0x4bc22185 (stale) Mirror 11: 0x4bc22185 (stale) Mirror 12: 0x4bc22185 (stale) Mirror 13: 0x4bc22185 (stale) Mirror 15: 0x4bc22185 (stale) Mirror 16: 0x4bc22185 (stale) Mirror 17: 0x4bc22185 (stale) Mirror 18: 0x4bc22185 (stale) Mirror 20: 0x4bc22185 (stale) Mirror 21: 0x4bc22185 (stale) Mirror 22: 0x4bc22185 (stale) Mirror 23: 0x4bc22185 (stale) Mirror 24: 0x4bc22185 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ddc3): Mirror 6: 0x4bc22185 (stale) Mirror 11: 0x4bc22185 (stale) Mirror 12: 0x4bc22185 (stale) Mirror 13: 0x4bc22185 (stale) Mirror 15: 0x4bc22185 (stale) Mirror 16: 0x4bc22185 (stale) Mirror 17: 0x4bc22185 (stale) Mirror 18: 0x4bc22185 (stale) Mirror 20: 0x4bc22185 (stale) Mirror 21: 0x4bc22185 (stale) Mirror 22: 0x4bc22185 (stale) Mirror 23: 0x4bc22185 (stale) Mirror 24: 0x4bc22185 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ddc3: skipped 000889 1790737528.688251 mapwrite 0x016e23 thru 0x01ef38 (0x8116 bytes) 000890 1790737528.777116 write 0x00e3b1 thru 0x01db90 (0xf7e0 bytes) 000891 1790737528.804949 MIRROR_RESYNC 14 mirrors 000892 1790737531.068315 mapread 0x00c3fc thru 0x01313a (0x6d3f bytes) 000893 1790737531.088928 trunc from 0x01c909 to 0x01ef38 (0x2630 bytes) 000894 1790737531.162865 read 0x00ef35 thru 0x01d144 (0xe210 bytes) 000895 1790737531.169377 trunc from 0x01ef39 to 0x03a027 (0x1b0ef bytes) 000896 1790737531.190076 fallocate 0x03c517 thru 0x03ffff (0x3ae9 bytes) 000897 1790737531.206240 fallocate 0x033c0c thru 0x03ffff (0xc3f4 bytes) 000898 1790737531.218515 read 0x01c868 thru 0x0219d1 (0x516a bytes) 000899 1790737531.248960 punch from 0x01b4af to 0x01dc09 (0x275b bytes) 000900 1790737531.254794 fallocate 0x030f14 thru 0x0366fe (0x57eb bytes) 000901 1790737531.260257 trunc from 0x008a3d to 0x016e42 (0xe406 bytes) 000902 1790737531.266783 fallocate 0x026b07 thru 0x030529 (0x9a23 bytes) 000903 1790737531.272250 read 0x006ac0 thru 0x00e6bf (0x7c00 bytes) 000904 1790737531.284945 read 0x005949 thru 0x006448 (0xb00 bytes) 000905 1790737531.287482 mapwrite 0x00e6ed thru 0x01687f (0x8193 bytes) 000906 1790737531.313478 write 0x01e38b thru 0x02bc99 (0xd90f bytes) 000907 1790737531.315517 read 0x0041cd thru 0x00c81f (0x8653 bytes) 000908 1790737531.316815 fallocate 0x00d7cb thru 0x013cb1 (0x64e7 bytes) 000909 1790737531.331466 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x4089f380 (stale) Mirror 11: 0x4089f380 (stale) Mirror 12: 0x4089f380 (stale) Mirror 13: 0x4089f380 (stale) Mirror 15: 0x4089f380 (stale) Mirror 16: 0x4089f380 (stale) Mirror 17: 0x4089f380 (stale) Mirror 18: 0x4089f380 (stale) Mirror 20: 0x4089f380 (stale) Mirror 21: 0x4089f380 (stale) Mirror 22: 0x4089f380 (stale) Mirror 23: 0x4089f380 (stale) Mirror 24: 0x4089f380 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x375d2546 Mirror 25: 0x375d2546 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000910 1790737532.819143 punch from 0x0119fa to 0x01899d (0x6fa4 bytes) 000911 1790737532.904978 trunc from 0x008021 to 0x00e01a (0x5ffa bytes) 000912 1790737532.911193 trunc from 0x0387c7 to 0x03c23e (0x3a78 bytes) 000913 1790737532.922857 fallocate 0x02c19d thru 0x02f083 (0x2ee7 bytes) 000914 1790737532.935761 fallocate 0x0339c1 thru 0x03a563 (0x6ba3 bytes) 000915 1790737532.941298 trunc from 0x040000 to 0x01a793 (0x2586c bytes) 000916 1790737533.014668 trunc from 0x01a794 to 0x03e331 (0x23b9e bytes) 000917 1790737533.031273 write 0x038fae thru 0x03ffff (0x7052 bytes) 000918 1790737533.041661 write 0x002f4a thru 0x010f14 (0xdfcb bytes) 000919 1790737533.056338 write 0x0114c5 thru 0x01dcf2 (0xc82e bytes) 000920 1790737533.089143 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x4089f380 (stale) Mirror 11: 0x4089f380 (stale) Mirror 12: 0x4089f380 (stale) Mirror 13: 0x4089f380 (stale) Mirror 15: 0x4089f380 (stale) Mirror 16: 0x4089f380 (stale) Mirror 17: 0x4089f380 (stale) Mirror 18: 0x4089f380 (stale) Mirror 20: 0x4089f380 (stale) Mirror 21: 0x4089f380 (stale) Mirror 22: 0x4089f380 (stale) Mirror 23: 0x4089f380 (stale) Mirror 24: 0x4089f380 (stale) Mirror 25: 0x375d2546 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdbb3885c Mirror 26: 0xdbb3885c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000921 1790737534.811106 trunc from 0x040000 to 0x020eb4 (0x1f14b bytes) 000922 1790737534.877656 trunc from 0x020eb5 to 0x0390dc (0x18228 bytes) 000924 1790737534.900274 fallocate 0x03b461 thru 0x03ffff (0x4b9f bytes) 000925 1790737534.909691 trunc from 0x0390dd to 0x01bfa1 (0x1d13b bytes) 000926 1790737534.930990 write 0x017e39 thru 0x01ad9b (0x2f63 bytes) 000927 1790737534.942927 trunc from 0x01bfa2 to 0x017bfd (0x43a4 bytes) 000928 1790737534.956024 write 0x033ac6 thru 0x03ffff (0xc53a bytes) 000929 1790737534.965293 trunc from 0x028202 to 0x037dce (0xfbcd bytes) 000931 1790737534.991406 mapread 0x002298 thru 0x0102d2 (0xe03b bytes) 000932 1790737535.011954 write 0x033026 thru 0x03f48e (0xc469 bytes) 000933 1790737535.026535 MIRROR_RESYNC 16 mirrors 000934 1790737537.867133 write 0x00077e thru 0x00efbd (0xe840 bytes) 000935 1790737537.948449 MIRROR_RESYNC 16 mirrors 000936 1790737540.874758 trunc from 0x000029 to 0x00ea61 (0xea39 bytes) 000937 1790737540.941426 fallocate 0x028beb thru 0x02e3e4 (0x57fa bytes) 000939 1790737540.948563 fallocate 0x015e65 thru 0x016dab (0xf47 bytes) 000940 1790737540.960231 mapwrite 0x025662 thru 0x02f687 (0xa026 bytes) 000941 1790737541.035077 mapwrite 0x003002 thru 0x00eb0e (0xbb0d bytes) 000942 1790737541.092939 mapwrite 0x02eeb0 thru 0x038a48 (0x9b99 bytes) 000943 1790737541.117776 read 0x03bfc2 thru 0x03ffff (0x403e bytes) 000944 1790737541.135194 read 0x035400 thru 0x03a2e1 (0x4ee2 bytes) 000945 1790737541.154314 fallocate 0x02b2af thru 0x035936 (0xa688 bytes) 000946 1790737541.161478 MIRROR_SPLIT mirror 7 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x6cf523d (stale) Mirror 11: 0x6cf523d (stale) Mirror 12: 0x6cf523d (stale) Mirror 13: 0x6cf523d (stale) Mirror 15: 0x6cf523d (stale) Mirror 16: 0x6cf523d (stale) Mirror 18: 0x6cf523d (stale) Mirror 20: 0x6cf523d (stale) Mirror 21: 0x6cf523d (stale) Mirror 22: 0x6cf523d (stale) Mirror 23: 0x6cf523d (stale) Mirror 24: 0x6cf523d (stale) Mirror 25: 0x6cf523d (stale) Mirror 26: 0x6cf523d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x6cf523d (stale) Mirror 11: 0x6cf523d (stale) Mirror 12: 0x6cf523d (stale) Mirror 13: 0x6cf523d (stale) Mirror 15: 0x6cf523d (stale) Mirror 16: 0x6cf523d (stale) Mirror 18: 0x6cf523d (stale) Mirror 20: 0x6cf523d (stale) Mirror 21: 0x6cf523d (stale) Mirror 22: 0x6cf523d (stale) Mirror 23: 0x6cf523d (stale) Mirror 24: 0x6cf523d (stale) Mirror 25: 0x6cf523d (stale) Mirror 26: 0x6cf523d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000947 1790737541.889007 trunc from 0x040000 to 0x03810e (0x7ef1 bytes) 000948 1790737541.926136 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3810f): Mirror 6: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3810f): Mirror 6: 0xe34f6dc9 (stale) Mirror 11: 0xe34f6dc9 (stale) Mirror 12: 0xe34f6dc9 (stale) Mirror 13: 0xe34f6dc9 (stale) Mirror 15: 0xe34f6dc9 (stale) Mirror 16: 0xe34f6dc9 (stale) Mirror 18: 0xe34f6dc9 (stale) Mirror 20: 0xe34f6dc9 (stale) Mirror 21: 0xe34f6dc9 (stale) Mirror 22: 0xe34f6dc9 (stale) Mirror 23: 0xe34f6dc9 (stale) Mirror 24: 0xe34f6dc9 (stale) Mirror 25: 0xe34f6dc9 (stale) Mirror 26: 0xe34f6dc9 (stale) CRC-32 checksum value for chunk [0, 0x3810f): Mirror 1: 0x31dd6e25 Mirror 27: 0x31dd6e25 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3810f: skipped 000949 1790737543.364503 mapread 0x02e51a thru 0x037dd9 (0x98c0 bytes) 000950 1790737543.408910 fallocate 0x01de60 thru 0x026b58 (0x8cf9 bytes) 000951 1790737543.521609 punch from 0x023cdb to 0x026c9f (0x2fc5 bytes) 000952 1790737543.531281 write 0x00bf1e thru 0x016fa4 (0xb087 bytes) 000953 1790737543.586089 fallocate 0x030ed0 thru 0x039a3a (0x8b6b bytes) 000954 1790737543.608124 mapread 0x018bee thru 0x0245ff (0xba12 bytes) 000955 1790737543.651170 fallocate 0x02c878 thru 0x032596 (0x5d1f bytes) 000956 1790737543.662253 read 0x009a60 thru 0x00a986 (0xf27 bytes) 000958 1790737543.688546 mapwrite 0x010f6f thru 0x01bc0f (0xaca1 bytes) 000959 1790737543.734174 trunc from 0x03810f to 0x00a490 (0x2dc7e bytes) 000960 1790737543.797482 MIRROR_SPLIT mirror 3 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa491): Mirror 6: 0x7a841412 (stale) Mirror 11: 0x7a841412 (stale) Mirror 13: 0x7a841412 (stale) Mirror 15: 0x7a841412 (stale) Mirror 16: 0x7a841412 (stale) Mirror 18: 0x7a841412 (stale) Mirror 20: 0x7a841412 (stale) Mirror 21: 0x7a841412 (stale) Mirror 22: 0x7a841412 (stale) Mirror 23: 0x7a841412 (stale) Mirror 24: 0x7a841412 (stale) Mirror 25: 0x7a841412 (stale) Mirror 26: 0x7a841412 (stale) Mirror 27: 0x22492a1b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa491): Mirror 6: 0x7a841412 (stale) Mirror 11: 0x7a841412 (stale) Mirror 13: 0x7a841412 (stale) Mirror 15: 0x7a841412 (stale) Mirror 16: 0x7a841412 (stale) Mirror 18: 0x7a841412 (stale) Mirror 20: 0x7a841412 (stale) Mirror 21: 0x7a841412 (stale) Mirror 22: 0x7a841412 (stale) Mirror 23: 0x7a841412 (stale) Mirror 24: 0x7a841412 (stale) Mirror 25: 0x7a841412 (stale) Mirror 26: 0x7a841412 (stale) Mirror 27: 0x22492a1b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa491: skipped 000961 1790737544.498755 trunc from 0x004a43 to 0x00a490 (0x5a4e bytes) 000962 1790737544.503592 read 0x0055eb thru 0x00a2c6 (0x4cdc bytes) 000963 1790737544.510754 trunc from 0x005640 to 0x00a490 (0x4e51 bytes) 000964 1790737544.515704 trunc from 0x008ff6 to 0x00a490 (0x149b bytes) 000965 1790737544.519697 mapwrite 0x014c60 thru 0x01eb28 (0x9ec9 bytes) 000966 1790737544.590461 read 0x00b5d1 thru 0x01345c (0x7e8c bytes) 000967 1790737544.603036 fallocate 0x034684 thru 0x037aef (0x346c bytes) 000968 1790737544.606803 read 0x0360d4 thru 0x037aef (0x1a1c bytes) 000970 1790737544.624113 trunc from 0x037af0 to 0x02879e (0xf351 bytes) 000971 1790737544.640801 mapwrite 0x0179d2 thru 0x02027d (0x88ac bytes) 000972 1790737544.657036 trunc from 0x02879f to 0x008925 (0x1fe79 bytes) 000973 1790737544.671797 punch from 0x004ad7 to 0x008925 (0x3e4f bytes) 000974 1790737544.676677 punch from 0x001374 to 0x0040a2 (0x2d2f bytes) 000975 1790737544.681629 mapread 0x0088c3 thru 0x008925 (0x63 bytes) 000976 1790737544.690735 trunc from 0x008926 to 0x012e3c (0xa517 bytes) 000977 1790737544.708239 mapread 0x001940 thru 0x002758 (0xe19 bytes) 000978 1790737544.718678 trunc from 0x012e3d to 0x004155 (0xece7 bytes) 000979 1790737544.731605 write 0x02f41a thru 0x033b7a (0x4761 bytes) 000980 1790737544.733197 mapread 0x01f3bc thru 0x02a553 (0xb198 bytes) 000981 1790737544.809037 fallocate 0x009898 thru 0x00a113 (0x87c bytes) 000982 1790737544.817916 punch from 0x022901 to 0x02cb7f (0xa27f bytes) 000985 1790737544.826150 fallocate 0x02950e thru 0x02e5bd (0x50b0 bytes) 000986 1790737544.834805 mapread 0x0048d3 thru 0x00fc23 (0xb351 bytes) 000987 1790737544.904442 MIRROR_SPLIT mirror 8 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33b7b): Mirror 6: 0x5a8a6744 (stale) Mirror 11: 0x5a8a6744 (stale) Mirror 13: 0x5a8a6744 (stale) Mirror 15: 0x5a8a6744 (stale) Mirror 16: 0x5a8a6744 (stale) Mirror 18: 0x5a8a6744 (stale) Mirror 20: 0x5a8a6744 (stale) Mirror 22: 0x5a8a6744 (stale) Mirror 23: 0x5a8a6744 (stale) Mirror 24: 0x5a8a6744 (stale) Mirror 25: 0x5a8a6744 (stale) Mirror 26: 0x5a8a6744 (stale) Mirror 27: 0x7630cf0e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33b7b): Mirror 6: 0x5a8a6744 (stale) Mirror 11: 0x5a8a6744 (stale) Mirror 13: 0x5a8a6744 (stale) Mirror 15: 0x5a8a6744 (stale) Mirror 16: 0x5a8a6744 (stale) Mirror 18: 0x5a8a6744 (stale) Mirror 20: 0x5a8a6744 (stale) Mirror 22: 0x5a8a6744 (stale) Mirror 23: 0x5a8a6744 (stale) Mirror 24: 0x5a8a6744 (stale) Mirror 25: 0x5a8a6744 (stale) Mirror 26: 0x5a8a6744 (stale) Mirror 27: 0x7630cf0e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33b7b: skipped 000988 1790737545.630650 punch from 0x02456f to 0x02e5ca (0xa05c bytes) 000989 1790737545.648125 trunc from 0x033b7b to 0x0187ef (0x1b38b bytes) 000990 1790737545.687567 punch from 0x014414 to 0x017488 (0x3075 bytes) 000991 1790737545.694711 trunc from 0x0187f0 to 0x02254b (0x9d5c bytes) 000992 1790737545.736793 read 0x015262 thru 0x01c250 (0x6fef bytes) 000993 1790737545.748884 write 0x00f7d9 thru 0x0111e0 (0x1a08 bytes) 000995 1790737545.757492 trunc from 0x02254c to 0x01a3a5 (0x81a6 bytes) 000996 1790737545.785572 MIRROR_SPLIT mirror 6 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x247:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a3a6): Mirror 6: 0x55a778a4 (stale) Mirror 11: 0x55a778a4 (stale) Mirror 13: 0x55a778a4 (stale) Mirror 15: 0x55a778a4 (stale) Mirror 16: 0x55a778a4 (stale) Mirror 20: 0x55a778a4 (stale) Mirror 22: 0x55a778a4 (stale) Mirror 23: 0x55a778a4 (stale) Mirror 24: 0x55a778a4 (stale) Mirror 25: 0x55a778a4 (stale) Mirror 26: 0x55a778a4 (stale) Mirror 27: 0x2be5431a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a3a6): Mirror 6: 0x55a778a4 (stale) Mirror 11: 0x55a778a4 (stale) Mirror 13: 0x55a778a4 (stale) Mirror 15: 0x55a778a4 (stale) Mirror 16: 0x55a778a4 (stale) Mirror 20: 0x55a778a4 (stale) Mirror 22: 0x55a778a4 (stale) Mirror 23: 0x55a778a4 (stale) Mirror 24: 0x55a778a4 (stale) Mirror 25: 0x55a778a4 (stale) Mirror 26: 0x55a778a4 (stale) Mirror 27: 0x2be5431a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a3a6: skipped 000997 1790737546.382160 read 0x006d38 thru 0x00eb31 (0x7dfa bytes) 000999 1790737546.388673 mapwrite 0x037545 thru 0x03ffff (0x8abb bytes) 001000 1790737546.433316 mapread 0x031713 thru 0x039a5f (0x834d bytes) All operations completed A-OK! PASS 70a (133s) == sanity-flr test 71: check mirror extend parent fid ==== 23:05:52 (1790737552) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x28a:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (9s) == sanity-flr test 72: check append on FLR file ========== 23:06:01 (1790737561) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x383:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x37f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.262547 s, 8.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x383:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x384:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x37f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106536 s, 9.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x383:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x384:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x380:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x37f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x383:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x384:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x380:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x37f:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x385:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 23:06:09 (1790737569) Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre2 Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 4102112 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 2603808 .. failed Extending file size to 7935616 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8128288 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8607264 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done Extending file size to 8667776 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8919200 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8975968 .. Extending file size to 8877664 .. Extending file size to 9044128 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9089440 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62191 62192 62193 62195 62197 done 192.168.203.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg343-client.virtnet /mnt/lustre2 (opts:) 192.168.203.143@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg343-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (81s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:07:30 (1790737650) Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre2 Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6132192 .. Extending file size to 6036608 .. Extending file size to 8595168 .. Extending file size to 6443040 .. Extending file size to 6763296 .. Extending file size to 8814976 .. Extending file size to 8310592 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8359136 .. Extending file size to 8585120 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9062624 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 63685 63686 63687 63688 63689 192.168.203.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg343-client.virtnet /mnt/lustre2 (opts:) 192.168.203.143@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg343-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:08:50 (1790737730) Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2708 1268596 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1864 1269440 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 17704 3589316 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25372 3581648 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43076 7170964 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2708 1268596 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1864 1269440 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 17704 3589316 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25372 3581648 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43076 7170964 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0691043 s, 30.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.229378 s, 9.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xeac2193a (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg343-client.virtnet /mnt/lustre2 (opts:) PASS 200c (15s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:09:07 (1790737747) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113035 s, 18.6 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 23:09:14 (1790737754) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x299:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x396:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x396:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x397:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x39a:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:09:21 (1790737761) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0855728 s, 24.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2d:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (7s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:09:28 (1790737768) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0849686 s, 24.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x398:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x399:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x39b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x398:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x399:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x39b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x39c:0x0] } PASS 204b (6s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:09:35 (1790737775) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0840384 s, 25.0 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x39a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x39d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x39a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x39d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x39e:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:10:08 (1790737808) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg343-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859138 s, 12.2 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2b:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2c:0x0] } PASS 204d (52s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:11:00 (1790737860) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg343-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x39b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x39c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0854794 s, 24.5 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x39b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x39c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x39f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x39b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x39c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x39f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (45s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:11:46 (1790737906) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg343-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg343-server: Pool lustre.test_204f created Adding targets to pool oleg343-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg343-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0905584 s, 23.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x30:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg343-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg343-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg343-server: Pool lustre.test_204f destroyed PASS 204f (71s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:12:57 (1790737977) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:13:04 (1790737984) PASS 205b (6s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:13:11 (1790737991) oleg343-server: Pool lustre.test_206 created Adding targets to pool oleg343-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg343-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg343-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg343-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg343-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:13:36 (1790738016) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0500299 s, 21.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0385029 s, 27.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xaf15ce52 Mirror 3: 0xaf15ce52 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:13:46 (1790738026) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:13:49 (1790738029) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:13:53 (1790738033) Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0822675 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2620000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2270000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2270000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2270000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xd40000 Read this (4 bytes): BBBB 192.168.203.143@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg343-client.virtnet /mnt/lustre2 (opts:) PASS 209a (34s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:14:26 (1790738066) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.020038 s, 409 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:14:32 (1790738072) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948889 s, 11.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x2b8 lmm_fid: [0x200000405:0x2b8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x2b8 lmm_fid: [0x200000405:0x2b8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x3a6:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611832 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 23:14:34.000000000 -0400 Modify: 2026-09-29 23:14:34.000000000 -0400 Change: 2026-09-29 23:14:35.000000000 -0400 Birth: 2026-09-29 23:14:34.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:14:42 (1790738082) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273101 s, 38.4 MB/s oleg343-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg343-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg343-server: O/280000401/d7/3a7: File not found by ext2_lookup PASS 210b (25s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:15:07 (1790738107) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.285519 s, 143 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0169765 s, 241 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 29 23:15 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x2bd:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 29 23:15 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 23:15:14 (1790738114) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (6s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 23:15:20 (1790738120) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (7s) == sanity-flr test complete, duration 1707 sec =========== 23:15:27 (1790738127) === sanity-flr: start cleanup 23:15:29 (1790738129) === === sanity-flr: finish cleanup 23:15:37 (1790738137) ===