-----============= acceptance-small: sanity-flr ============----- Tue Sep 29 15:47:20 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITY_FLR_ONLY='' SANITY_FLR_EXCEPT='' oleg333-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 15:47:46 (1790711266) === oleg333-client.virtnet: executing check_config_client /mnt/lustre oleg333-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg333-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8df450128000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8df450128000.idle_timeout=debug disable quota as required oleg333-server: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 15:48:31 (1790711311) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8205] == sanity-flr test 0a: lfs mirror create with -N option == 15:48:33 (1790711313) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:48:50 (1790711330) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:48:54 (1790711334) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:48:58 (1790711338) PASS 0d (22s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:49:20 (1790711360) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:49:23 (1790711363) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 15:49:27 (1790711367) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8df450128000.stats=clear osc.lustre-OST0001-osc-ffff8df450128000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:49:35 (1790711375) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--ec-map] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:49:45 (1790711385) PASS 0j (7s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:49:52 (1790711392) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 15:49:56 (1790711396) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:50:04 (1790711404) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (8s) == sanity-flr test 2: create components from existing files ========================================================== 15:50:13 (1790711413) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:50:21 (1790711421) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:50:29 (1790711429) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:50:38 (1790711438) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:50:48 (1790711448) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:50:55 (1790711455) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 1.65525 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551519 s, 19.0 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:51:05 (1790711465) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.32192 s, 29.3 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 15:51:13 (1790711473) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:51:24 (1790711484) Stopping /mnt/lustre-ost1 (opts:) on oleg333-server oleg333-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8df450128000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8df450128000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg333-server: oleg333-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg333-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg333-server oleg333-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8df450128000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8df450128000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg333-server: oleg333-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg333-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (78s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:52:42 (1790711562) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.701279 s, 16.4 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg333-server oleg333-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8df450128000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8df450128000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg333-server: oleg333-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg333-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (53s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:53:35 (1790711615) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg333-server oleg333-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8df450128000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8df450128000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg333-server: oleg333-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg333-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg333-server oleg333-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8df450128000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8df450128000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg333-server: oleg333-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg333-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (90s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:55:05 (1790711705) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.50768 s, 14.9 MB/s -rw-r--r-- 1 root root 52428800 Sep 29 15:55 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.251s user 0m0.000s sys 0m0.211s Stopping /mnt/lustre-ost1 (opts:) on oleg333-server oleg333-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8df450128000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8df450128000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.477s user 0m0.000s sys 0m0.187s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg333-server: oleg333-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg333-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg333-server oleg333-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8df450128000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8df450128000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.467s user 0m0.001s sys 0m0.159s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg333-server: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg333-server: oleg333-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg333-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg333-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8df450128000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8df450128000.ost_server_uuid in FULL state after 0 sec PASS 33b (104s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:56:50 (1790711810) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:56:53 (1790711813) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:56:57 (1790711817) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 15:57:00 (1790711820) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 15:57:08 (1790711828) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.704632 s, 17.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:57:21 (1790711841) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0200159 s, 52.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0302667 s, 34.6 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:57:30 (1790711850) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:57:37 (1790711857) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00102801 s, 996 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00079725 s, 1.3 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000842608 s, 1.2 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00082209 s, 1.2 MB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 15:57:46 (1790711866) create test files with size 15759251 .. e4a4daf4da0b23addd7634eada76dcfe - f8edc8388f83e0f7de27a6373b649186 - 965f29fc6379c9dd36d24b0b277d4093 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg333-client.virtnet /mnt/lustre (opts:) Starting client: oleg333-client.virtnet: -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre PASS 37 (25s) == sanity-flr test 38: resync ============================ 15:58:11 (1790711891) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:58:27 (1790711907) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:58:34 (1790711914) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.089653 s, 11.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xae:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0255565 s, 41.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xce:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 15:58:43 (1790711923) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.142739 s, 14.7 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.320806 s, 13.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.229536 s, 9.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 15:58:58 (1790711938) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 15:59:01 (1790711941) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:59:05 (1790711945) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294386 s, 35.6 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 15:59:13 (1790711953) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.33577 s, 9.4 MB/s PASS 44a (11s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:59:24 (1790711964) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb9:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd9:0x0] } split mirror_id 2 id 131073 error lfs split: file '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' has 1 component, cannot split make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:59:36 (1790711976) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.401143 s, 26.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg333-client.virtnet /mnt/lustre (opts:) Starting client: oleg333-client.virtnet: -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:59:50 (1790711990) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff8df442c36800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.195126 s, 53.7 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg333-client.virtnet /mnt/lustre (opts:) Starting client: oleg333-client.virtnet: -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff8df44b1bb000.xattr_cache=1 PASS 44e (20s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 16:00:10 (1790712010) error lfs split: create victim file failed: File exists PASS 44f (19s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:00:29 (1790712029) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (57s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:01:27 (1790712087) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (49s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:02:17 (1790712137) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 16:02:20 (1790712140) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.349608 s, 9.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.119554 s, 26.3 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xd6a6ee48 (nosync,stale) Mirror 2: 0xd6a6ee48 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 8bcee423fe87a490d8d004ceacadc84a - mirror 1 checksum: 48a5738f561b893934f123594affae5e - mirror 2 checksum: 8bcee423fe87a490d8d004ceacadc84a - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-29T16:02:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-29T16:02:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8bcee423fe87a490d8d004ceacadc84a - mirror 1 checksum: 48a5738f561b893934f123594affae5e - mirror 2 checksum: 48a5738f561b893934f123594affae5e - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-29T16:02:24-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-29T16:02:24-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xed:0x0] } PASS 48 (12s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 16:02:32 (1790712152) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.043966 s, 23.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:02:40 (1790712160) Failing mds1 on oleg333-server Stopping /mnt/lustre-mds1 (opts:) on oleg333-server 16:02:47 (1790712167) shut down facet: mds1 facet_host: oleg333-server facet_failover_host: oleg333-server Failover mds1 to oleg333-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg333-server: oleg333-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg333-client: oleg333-server: ssh exited with exit code 1 Started lustre-MDT0000 16:03:24 (1790712204) targets are mounted 16:03:24 (1790712204) facet_failover done oleg333-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x9:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441742 s, 23.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x240000405 lmm_object_id: 0x9 lmm_fid: [0x240000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x21:0x0] } PASS 50A (61s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:03:42 (1790712222) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0375003 s, 546 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0246988 s, 829 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:03:53 (1790712233) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.106172 s, 193 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00710651 s, 2.9 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:04:04 (1790712244) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0133745 s, 1.2 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:04:14 (1790712254) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0629923 s, 325 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (7s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:04:21 (1790712261) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0199841 s, 205 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0369458 s, 111 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:04:29 (1790712269) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:04:50 (1790712290) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (28s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:05:18 (1790712318) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:05:42 (1790712342) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:06:07 (1790712367) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 16:06:15 (1790712375) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 58243 mirror split pid 58245 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 58243 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 58245 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 16:07:24 (1790712444) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 2446 000002 1790712447.165417 MIRROR_EXTEND to 2 mirrors fallocating to largest ever: 0x1ec8d 000003 1790712447.723859 fallocate 0x00ecca thru 0x01ec8c (0xffc3 bytes) 000004 1790712447.751647 trunc from 0x01ec8d to 0x00024f (0x1ea3d bytes) 000005 1790712447.776863 read 0x000034 thru 0x00024f (0x21c bytes) 000006 1790712447.783798 punch from 0x0000ee to 0x00024f (0x162 bytes) 000007 1790712447.804193 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) 000009 1790712448.003133 read 0x00023a thru 0x00024f (0x16 bytes) 000010 1790712448.013663 read 0x000213 thru 0x00024f (0x3d bytes) 000011 1790712448.015240 fallocate 0x017601 thru 0x01c182 (0x4b82 bytes) 000012 1790712448.023043 mapwrite 0x02b443 thru 0x033189 (0x7d47 bytes) 000013 1790712448.171476 trunc from 0x00ebc7 to 0x0187f1 (0x9c2b bytes) fallocating to largest ever: 0x38555 000014 1790712448.182852 fallocate 0x0355f0 thru 0x038554 (0x2f65 bytes) 000015 1790712448.189140 punch from 0x027d7f to 0x0325d9 (0xa85b bytes) 000016 1790712448.202498 punch from 0x01ded1 to 0x027d6d (0x9e9d bytes) 000017 1790712448.208207 fallocate 0x02b514 thru 0x035c43 (0xa730 bytes) 000018 1790712448.221842 fallocate 0x02d449 thru 0x02e3fc (0xfb4 bytes) 000019 1790712448.235135 write 0x03df20 thru 0x03ffff (0x20e0 bytes) 000020 1790712448.240638 punch from 0x01e9f5 to 0x022cce (0x42da bytes) 000021 1790712448.245762 punch from 0x009502 to 0x0178d2 (0xe3d1 bytes) 000022 1790712448.254421 fallocate 0x02a30c thru 0x037681 (0xd376 bytes) 000023 1790712448.268743 write 0x01b12b thru 0x01f544 (0x441a bytes) 000024 1790712448.287683 fallocate 0x020999 thru 0x02c061 (0xb6c9 bytes) 000025 1790712448.301449 punch from 0x02f52f to 0x031bd5 (0x26a7 bytes) 000026 1790712448.326235 mapwrite 0x00556e thru 0x014d07 (0xf79a bytes) 000027 1790712448.565286 write 0x0181c7 thru 0x01eda4 (0x6bde bytes) 000028 1790712448.573648 trunc from 0x040000 to 0x030720 (0xf8df bytes) 000029 1790712448.614379 trunc from 0x030721 to 0x023a9a (0xcc86 bytes) 000030 1790712448.640768 mapwrite 0x01b662 thru 0x01eb5e (0x34fd bytes) 000031 1790712448.686315 read 0x01cc8f thru 0x023a9a (0x6e0c bytes) 000033 1790712448.705673 MIRROR_EXTEND to 2 mirrors 000034 1790712449.860556 write 0x03b92e thru 0x03ffff (0x46d2 bytes) 000035 1790712449.936543 punch from 0x02b9cd to 0x039415 (0xda49 bytes) zero_range to largest ever: 0x40000 000036 1790712449.946920 trunc from 0x039f28 to 0x03ffff (0x60d8 bytes) 000037 1790712449.976763 write 0x0075bc thru 0x0085da (0x101f bytes) 000038 1790712450.001978 trunc from 0x040000 to 0x039c75 (0x638a bytes) 000039 1790712450.046017 mapwrite 0x00f6a0 thru 0x018703 (0x9064 bytes) 000040 1790712450.121950 mapwrite 0x002212 thru 0x011943 (0xf732 bytes) 000041 1790712450.222382 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) 000042 1790712450.441534 trunc from 0x035c88 to 0x039c75 (0x3fee bytes) 000043 1790712450.454142 mapwrite 0x004063 thru 0x013a9a (0xfa38 bytes) 000044 1790712450.532767 fallocate 0x03a27c thru 0x03ffff (0x5d84 bytes) 000045 1790712450.543513 trunc from 0x0069fb to 0x011fc3 (0xb5c9 bytes) 000046 1790712450.549779 write 0x02e5ad thru 0x03a26d (0xbcc1 bytes) 000047 1790712450.567410 mapread 0x002f76 thru 0x00803f (0x50ca bytes) 000048 1790712450.578217 mapread 0x0120bb thru 0x01ebe5 (0xcb2b bytes) 000049 1790712450.594572 trunc from 0x040000 to 0x023410 (0x1cbef bytes) 000050 1790712450.617791 mapread 0x00ddbe thru 0x017c30 (0x9e73 bytes) 000051 1790712450.625627 mapread 0x02065e thru 0x023410 (0x2db3 bytes) 000052 1790712450.633378 read 0x020115 thru 0x023410 (0x32fc bytes) 000053 1790712450.635133 write 0x01ddd1 thru 0x0241e7 (0x6417 bytes) 000054 1790712450.636834 mapread 0x01d619 thru 0x020990 (0x3378 bytes) 000055 1790712450.639598 trunc from 0x0241e8 to 0x0157bd (0xea2a bytes) 000056 1790712450.665316 fallocate 0x02964c thru 0x032aaa (0x945f bytes) 000057 1790712450.671700 punch from 0x000496 to 0x0072e5 (0x6e50 bytes) 000058 1790712450.679716 mapread 0x014f71 thru 0x021ad7 (0xcb67 bytes) 000059 1790712450.702808 mapwrite 0x03eb86 thru 0x03ffff (0x147a bytes) 000060 1790712450.749803 mapread 0x0039cc thru 0x0057fa (0x1e2f bytes) 000061 1790712450.779217 trunc from 0x01de27 to 0x023959 (0x5b33 bytes) 000062 1790712450.800638 trunc from 0x0320ba to 0x03ffff (0xdf46 bytes) 000064 1790712450.806873 punch from 0x01f3b7 to 0x02a109 (0xad53 bytes) 000065 1790712450.813441 mapread 0x02db38 thru 0x0376d7 (0x9ba0 bytes) 000066 1790712450.869388 fallocate 0x03c801 thru 0x03ffff (0x37ff bytes) 000067 1790712450.877231 mapread 0x02c80b thru 0x035531 (0x8d27 bytes) 000068 1790712450.878760 mapread 0x016075 thru 0x01cafb (0x6a87 bytes) 000069 1790712450.880224 trunc from 0x00d025 to 0x01b61b (0xe5f7 bytes) 000070 1790712450.886196 mapread 0x02d2a4 thru 0x033977 (0x66d4 bytes) 000072 1790712450.888841 fallocate 0x0092e8 thru 0x01728e (0xdfa7 bytes) 000073 1790712450.895617 trunc from 0x040000 to 0x021d50 (0x1e2af bytes) 000076 1790712450.917349 mapread 0x01ba2a thru 0x021d50 (0x6327 bytes) 000077 1790712450.930111 read 0x001c50 thru 0x0093e4 (0x7795 bytes) 000078 1790712450.930644 read 0x01af9c thru 0x01ce1d (0x1e82 bytes) 000079 1790712450.931155 mapwrite 0x028674 thru 0x036ad4 (0xe461 bytes) 000080 1790712451.015691 write 0x01d36e thru 0x02663c (0x92cf bytes) 000081 1790712451.018556 read 0x02ec96 thru 0x035997 (0x6d02 bytes) 000082 1790712451.020416 mapwrite 0x02d2b6 thru 0x031bfd (0x4948 bytes) 000083 1790712451.081912 mapwrite 0x00a3fe thru 0x00bd6d (0x1970 bytes) 000085 1790712451.115218 mapread 0x021e8d thru 0x02e2e3 (0xc457 bytes) 000087 1790712451.118484 punch from 0x0221f1 to 0x0295d3 (0x73e3 bytes) 000088 1790712451.125815 write 0x01fc26 thru 0x022794 (0x2b6f bytes) 000089 1790712451.133325 mapread 0x01b957 thru 0x01cc71 (0x131b bytes) 000090 1790712451.134285 fallocate 0x026f5b thru 0x027174 (0x21a bytes) 000091 1790712451.144999 fallocate 0x00f2ad thru 0x01a098 (0xadec bytes) 000092 1790712451.150690 mapread 0x01534d thru 0x020323 (0xafd7 bytes) 000093 1790712451.151606 punch from 0x008069 to 0x00f627 (0x75bf bytes) 000094 1790712451.162540 trunc from 0x036ad5 to 0x02e59d (0x8537 bytes) 000095 1790712451.189020 trunc from 0x02e59e to 0x030021 (0x1a84 bytes) 000097 1790712451.211027 trunc from 0x030022 to 0x011c28 (0x1e3f9 bytes) 000098 1790712451.237954 punch from 0x00a598 to 0x011648 (0x70b1 bytes) 000099 1790712451.248267 punch from 0x006290 to 0x011c28 (0xb999 bytes) 000100 1790712451.255009 trunc from 0x0070e8 to 0x011c28 (0xab41 bytes) 000101 1790712451.275952 write 0x0300eb thru 0x033423 (0x3339 bytes) 000102 1790712451.277112 trunc from 0x02dc63 to 0x02e593 (0x931 bytes) 000103 1790712451.294882 fallocate 0x029378 thru 0x029448 (0xd1 bytes) 000105 1790712451.323857 punch from 0x01c8f8 to 0x029e01 (0xd50a bytes) 000106 1790712451.371388 MIRROR_EXTEND to 2 mirrors 000107 1790712451.969675 read 0x01aa5b thru 0x0245d8 (0x9b7e bytes) 000108 1790712451.981972 read 0x000cdb thru 0x003215 (0x253b bytes) 000109 1790712452.004120 mapwrite 0x01f25a thru 0x0263af (0x7156 bytes) 000110 1790712452.237544 mapwrite 0x010ce9 thru 0x0114ef (0x807 bytes) 000111 1790712452.310951 mapwrite 0x036867 thru 0x03ffff (0x9799 bytes) 000112 1790712452.443233 fallocate 0x014ef3 thru 0x0217f7 (0xc905 bytes) 000114 1790712452.453782 mapread 0x02555f thru 0x0262a6 (0xd48 bytes) 000115 1790712452.465031 mapwrite 0x0308a1 thru 0x03f939 (0xf099 bytes) 000116 1790712452.571840 read 0x00a120 thru 0x014f51 (0xae32 bytes) 000117 1790712452.599942 mapread 0x01e215 thru 0x024b9f (0x698b bytes) 000118 1790712452.643932 MIRROR_RESYNC 2 mirrors 000119 1790712453.593134 mapread 0x0120b1 thru 0x01ba5c (0x99ac bytes) 000120 1790712453.636708 MIRROR_RESYNC 2 mirrors 000122 1790712454.241527 trunc from 0x040000 to 0x02acbd (0x15342 bytes) 000124 1790712454.327335 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) 000125 1790712454.608675 write 0x016545 thru 0x01a5b0 (0x406c bytes) 000126 1790712454.631694 write 0x006faa thru 0x00af30 (0x3f87 bytes) 000127 1790712454.672515 write 0x01b86a thru 0x02734d (0xbae4 bytes) 000128 1790712454.697799 fallocate 0x01df3a thru 0x02456f (0x6636 bytes) 000130 1790712454.713373 trunc from 0x02acbe to 0x02738a (0x3933 bytes) 000131 1790712454.733503 write 0x037a9c thru 0x039a45 (0x1faa bytes) 000134 1790712454.736005 read 0x00af53 thru 0x014245 (0x92f3 bytes) 000135 1790712454.768440 trunc from 0x039a46 to 0x022a54 (0x16ff1 bytes) 000136 1790712454.793360 punch from 0x00347d to 0x004aaf (0x1633 bytes) 000137 1790712454.801029 trunc from 0x007ad8 to 0x007f4e (0x477 bytes) 000138 1790712454.831848 trunc from 0x00d3f9 to 0x01c637 (0xf23f bytes) 000139 1790712454.837329 punch from 0x00cb5d to 0x01307f (0x6523 bytes) 000140 1790712454.843573 fallocate 0x02cfa5 thru 0x02d297 (0x2f3 bytes) 000141 1790712454.850407 trunc from 0x014e57 to 0x01c572 (0x771c bytes) 000142 1790712454.856931 read 0x008eee thru 0x016643 (0xd756 bytes) 000143 1790712454.908673 trunc from 0x003d71 to 0x00de1d (0xa0ad bytes) 000144 1790712454.935864 mapwrite 0x039d01 thru 0x03af6d (0x126d bytes) 000145 1790712455.040875 write 0x02083f thru 0x02b444 (0xac06 bytes) 000146 1790712455.042196 trunc from 0x02fd89 to 0x038a19 (0x8c91 bytes) 000147 1790712455.055325 fallocate 0x01a0e6 thru 0x023034 (0x8f4f bytes) 000148 1790712455.072986 write 0x02dfd5 thru 0x0334ec (0x5518 bytes) 000149 1790712455.080980 trunc from 0x03af6e to 0x009fdb (0x30f92 bytes) 000150 1790712455.114383 mapread 0x00039b thru 0x009fdb (0x9c41 bytes) 000151 1790712455.170593 mapwrite 0x000003 thru 0x00dc28 (0xdc26 bytes) 000152 1790712455.269896 MIRROR_EXTEND to 2 mirrors 000153 1790712456.079295 mapread 0x00554b thru 0x00968e (0x4144 bytes) 000154 1790712456.117986 trunc from 0x00d7c2 to 0x00dc28 (0x467 bytes) 000155 1790712456.178939 fallocate 0x00bd75 thru 0x00c30f (0x59b bytes) 000157 1790712456.197035 trunc from 0x00dc29 to 0x011abf (0x3e97 bytes) 000158 1790712456.274576 read 0x00f84a thru 0x011abf (0x2276 bytes) 000159 1790712456.295895 punch from 0x00a09f to 0x011abf (0x7a21 bytes) 000160 1790712456.325536 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) 000161 1790712456.515460 mapread 0x00b3e4 thru 0x00d6b2 (0x22cf bytes) 000163 1790712456.540630 mapwrite 0x000b50 thru 0x0098c9 (0x8d7a bytes) 000165 1790712456.588254 fallocate 0x01681e thru 0x021419 (0xabfc bytes) 000166 1790712456.594524 fallocate 0x01bd5a thru 0x029402 (0xd6a9 bytes) 000168 1790712456.607807 read 0x0070a1 thru 0x00f396 (0x82f6 bytes) 000169 1790712456.613050 write 0x00c506 thru 0x0165d4 (0xa0cf bytes) 000170 1790712456.639665 punch from 0x019df1 to 0x01fcda (0x5eea bytes) 000171 1790712456.662419 read 0x00b641 thru 0x016275 (0xac35 bytes) 000173 1790712456.664487 mapread 0x00131a thru 0x01118b (0xfe72 bytes) 000174 1790712456.666698 fallocate 0x010d06 thru 0x01f8f5 (0xebf0 bytes) 000175 1790712456.685522 MIRROR_EXTEND to 2 mirrors 000176 1790712458.347580 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) 000177 1790712458.514135 mapread 0x0068d9 thru 0x00f371 (0x8a99 bytes) 000178 1790712458.542549 mapread 0x00f035 thru 0x01a712 (0xb6de bytes) 000179 1790712458.570715 write 0x01a713 thru 0x021892 (0x7180 bytes) 000180 1790712458.605356 mapread 0x002b77 thru 0x00a96d (0x7df7 bytes) 000181 1790712458.615317 punch from 0x0112c8 to 0x01c6e6 (0xb41f bytes) 000182 1790712458.636967 trunc from 0x00a248 to 0x019102 (0xeebb bytes) 000183 1790712458.643496 trunc from 0x021893 to 0x012337 (0xf55b bytes) 000184 1790712458.663992 mapread 0x003ba6 thru 0x0081c0 (0x461b bytes) 000186 1790712458.675318 trunc from 0x012338 to 0x009aaf (0x8888 bytes) 000187 1790712458.698544 write 0x0367fb thru 0x03ffff (0x9805 bytes) 000188 1790712458.702870 mapread 0x03d455 thru 0x03ffff (0x2bab bytes) 000189 1790712458.704716 fallocate 0x009aec thru 0x010bbb (0x70d0 bytes) 000190 1790712458.711417 punch from 0x01bfdc to 0x020d8d (0x4db2 bytes) 000191 1790712458.717203 trunc from 0x0038f1 to 0x00d550 (0x9c60 bytes) 000192 1790712458.729109 MIRROR_EXTEND to 2 mirrors 000193 1790712459.538948 fallocate 0x004efd thru 0x00a208 (0x530c bytes) 000194 1790712459.562653 write 0x03aee2 thru 0x03d8c0 (0x29df bytes) 000195 1790712459.588683 mapwrite 0x00bc4f thru 0x012310 (0x66c2 bytes) 000197 1790712459.699236 read 0x00ea34 thru 0x010c5d (0x222a bytes) 000199 1790712459.700328 trunc from 0x031ae6 to 0x034c05 (0x3120 bytes) 000200 1790712459.706776 mapread 0x02ced3 thru 0x02e088 (0x11b6 bytes) 000201 1790712459.733168 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) 000202 1790712459.989318 write 0x0018b9 thru 0x00a23e (0x8986 bytes) 000203 1790712460.025121 trunc from 0x040000 to 0x00f09a (0x30f65 bytes) 000204 1790712460.050702 fallocate 0x00c577 thru 0x014335 (0x7dbf bytes) 000205 1790712460.061408 mapwrite 0x011f1e thru 0x012b2d (0xc10 bytes) 000206 1790712460.122309 write 0x021277 thru 0x02dd9e (0xcb28 bytes) 000208 1790712460.125986 fallocate 0x009d29 thru 0x009eb0 (0x188 bytes) 000209 1790712460.131273 trunc from 0x011148 to 0x017f99 (0x6e52 bytes) 000212 1790712460.138550 trunc from 0x02dd9f to 0x0254bd (0x88e1 bytes) 000213 1790712460.162826 mapwrite 0x0353a1 thru 0x03c2d0 (0x6f30 bytes) 000215 1790712460.223662 mapwrite 0x031ac3 thru 0x039052 (0x7590 bytes) 000216 1790712460.258353 trunc from 0x00b998 to 0x0197fa (0xde63 bytes) 000219 1790712460.268714 punch from 0x0180e5 to 0x01f538 (0x7454 bytes) 000221 1790712460.280478 punch from 0x01f3aa to 0x02de4b (0xeaa2 bytes) 000222 1790712460.285389 fallocate 0x003752 thru 0x008d1b (0x55ca bytes) 000224 1790712460.300393 mapwrite 0x025de0 thru 0x0340d9 (0xe2fa bytes) 000225 1790712460.343249 MIRROR_EXTEND to 2 mirrors 000226 1790712460.866833 punch from 0x03ac81 to 0x03c2d0 (0x1650 bytes) 000227 1790712460.891557 mapwrite 0x030305 thru 0x039f6d (0x9c69 bytes) 000228 1790712460.963415 read 0x018c6f thru 0x025601 (0xc993 bytes) 000229 1790712460.974857 trunc from 0x03c2d1 to 0x020520 (0x1bdb0 bytes) 000230 1790712461.039411 read 0x01e60d thru 0x020520 (0x1f14 bytes) 000231 1790712461.040212 trunc from 0x020521 to 0x01fe97 (0x689 bytes) 000232 1790712461.075757 trunc from 0x00b508 to 0x012770 (0x7269 bytes) 000234 1790712461.086543 read 0x00db2a thru 0x011f67 (0x443e bytes) 000235 1790712461.106070 punch from 0x00d447 to 0x0104f8 (0x30b2 bytes) 000236 1790712461.118597 trunc from 0x01fe98 to 0x035ff2 (0x1615b bytes) 000237 1790712461.140477 trunc from 0x035ff3 to 0x008740 (0x2d8b2 bytes) 000239 1790712461.159641 mapwrite 0x01a6c7 thru 0x01fa0c (0x5346 bytes) 000240 1790712461.226582 write 0x03751d thru 0x03ffff (0x8ae3 bytes) 000241 1790712461.233417 trunc from 0x040000 to 0x016f05 (0x290fa bytes) 000242 1790712461.260266 write 0x003d79 thru 0x006eff (0x3187 bytes) 000243 1790712461.266049 fallocate 0x01413b thru 0x0166c2 (0x2588 bytes) 000246 1790712461.292871 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x16f06): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16f06): Mirror 2: 0xfd6102d1 (stale) CRC-32 checksum value for chunk [0, 0x16f06): Mirror 1: 0x9256d8d9 Mirror 3: 0x9256d8d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16f06: skipped 000247 1790712462.418042 write 0x03543a thru 0x03c7f7 (0x73be bytes) 000248 1790712462.514350 mapwrite 0x0357a8 thru 0x03ffff (0xa858 bytes) 000249 1790712462.684915 fallocate 0x01476d thru 0x01e3d7 (0x9c6b bytes) 000250 1790712462.694965 read 0x03c38b thru 0x03ffff (0x3c75 bytes) 000251 1790712462.697268 punch from 0x025134 to 0x02b373 (0x6240 bytes) 000252 1790712462.702180 fallocate 0x02f163 thru 0x036734 (0x75d2 bytes) 000253 1790712462.707966 mapwrite 0x034754 thru 0x03df11 (0x97be bytes) 000254 1790712462.735861 trunc from 0x040000 to 0x01c6f5 (0x2390a bytes) 000255 1790712462.760182 read 0x00f797 thru 0x01bf8f (0xc7f9 bytes) 000256 1790712462.770399 mapread 0x001e7a thru 0x00f8f1 (0xda78 bytes) 000257 1790712462.771575 read 0x00e367 thru 0x018935 (0xa5cf bytes) 000258 1790712462.772467 mapwrite 0x030eb2 thru 0x03d6fa (0xc849 bytes) 000259 1790712462.851542 write 0x035d7e thru 0x039b01 (0x3d84 bytes) 000261 1790712462.854535 mapwrite 0x02b103 thru 0x0380f4 (0xcff2 bytes) 000262 1790712462.880716 trunc from 0x00bf4b to 0x01366d (0x7723 bytes) 000263 1790712462.889599 trunc from 0x021699 to 0x02999d (0x8305 bytes) 000264 1790712462.894649 write 0x02d950 thru 0x02f4be (0x1b6f bytes) 000265 1790712462.898765 mapwrite 0x01f940 thru 0x02e0dc (0xe79d bytes) 000266 1790712463.036547 mapwrite 0x00ea20 thru 0x0106b9 (0x1c9a bytes) 000267 1790712463.098501 mapread 0x031430 thru 0x0386aa (0x727b bytes) 000268 1790712463.099612 punch from 0x025dfa to 0x0322bd (0xc4c4 bytes) 000269 1790712463.121346 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3d6fb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d6fb): Mirror 2: 0xf3e7d634 (stale) Mirror 3: 0x40e02fc8 (stale) CRC-32 checksum value for chunk [0, 0x3d6fb): Mirror 1: 0x305cd495 Mirror 4: 0x305cd495 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d6fb: skipped 000270 1790712464.222689 mapread 0x02384c thru 0x033582 (0xfd37 bytes) 000271 1790712464.281873 mapwrite 0x006838 thru 0x007280 (0xa49 bytes) 000272 1790712464.430017 mapread 0x035c3c thru 0x03c618 (0x69dd bytes) 000273 1790712464.441593 mapwrite 0x00ffa8 thru 0x011f34 (0x1f8d bytes) 000274 1790712464.478982 trunc from 0x03d6fb to 0x031987 (0xbd73 bytes) 000275 1790712464.556368 write 0x0165be thru 0x023905 (0xd348 bytes) 000276 1790712464.586817 punch from 0x025b60 to 0x031987 (0xbe28 bytes) 000277 1790712464.596974 trunc from 0x02e867 to 0x02f907 (0x10a1 bytes) 000278 1790712464.608871 write 0x02f138 thru 0x03c3a4 (0xd26d bytes) 000279 1790712464.620626 trunc from 0x03c3a5 to 0x024c88 (0x1771c bytes) 000280 1790712464.636473 trunc from 0x024c89 to 0x03e438 (0x197b0 bytes) 000281 1790712464.654086 punch from 0x00cbb2 to 0x017361 (0xa7b0 bytes) 000283 1790712464.683667 mapread 0x035934 thru 0x03e438 (0x8b05 bytes) 000285 1790712464.720106 read 0x02379f thru 0x0310b0 (0xd912 bytes) 000286 1790712464.730581 trunc from 0x03e439 to 0x017257 (0x271e1 bytes) 000288 1790712464.754696 read 0x00d542 thru 0x017257 (0x9d16 bytes) 000289 1790712464.768978 trunc from 0x010b3f to 0x013c48 (0x310a bytes) 000291 1790712464.775708 trunc from 0x0138d3 to 0x0155bc (0x1cea bytes) 000292 1790712464.781702 trunc from 0x017258 to 0x0246c9 (0xd472 bytes) 000294 1790712464.832107 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x246ca): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x246ca): Mirror 2: 0xe09ff414 (stale) Mirror 3: 0x4d7913e1 (stale) Mirror 4: 0xc0795102 (stale) CRC-32 checksum value for chunk [0, 0x246ca): Mirror 1: 0x319ce0a0 Mirror 5: 0x319ce0a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x246ca: skipped 000295 1790712465.906361 punch from 0x0206f9 to 0x0246c9 (0x3fd1 bytes) 000296 1790712465.981507 read 0x0119b2 thru 0x01c7a0 (0xadef bytes) 000297 1790712466.000326 read 0x01d138 thru 0x02257c (0x5445 bytes) 000298 1790712466.023410 read 0x00171e thru 0x0036fa (0x1fdd bytes) 000299 1790712466.025353 trunc from 0x012361 to 0x01b54f (0x91ef bytes) 000300 1790712466.033652 write 0x01002f thru 0x01a1c4 (0xa196 bytes) 000301 1790712466.044886 read 0x004737 thru 0x00e00e (0x98d8 bytes) 000302 1790712466.049923 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x246ca): Mirror 2: 0xe09ff414 (stale) Mirror 4: 0xc0795102 (stale) Mirror 5: 0x319ce0a0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x246ca): Mirror 2: 0xe09ff414 (stale) Mirror 4: 0xc0795102 (stale) Mirror 5: 0x319ce0a0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x246ca: skipped 000304 1790712466.546615 mapread 0x021488 thru 0x0246c9 (0x3242 bytes) 000305 1790712466.611674 write 0x00db99 thru 0x01db69 (0xffd1 bytes) 000306 1790712466.642507 fallocate 0x03f108 thru 0x03ffff (0xef8 bytes) 000307 1790712466.652341 trunc from 0x040000 to 0x009c2f (0x363d0 bytes) 000309 1790712466.673969 mapread 0x007ee0 thru 0x009c2f (0x1d50 bytes) 000311 1790712466.714813 read 0x003ed3 thru 0x005a33 (0x1b61 bytes) 000312 1790712466.715400 trunc from 0x009c30 to 0x02ea13 (0x24de4 bytes) 000313 1790712466.736924 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ea14): Mirror 4: 0xc5a22aed (stale) Mirror 5: 0x4188c6ba (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ea14): Mirror 4: 0xc5a22aed (stale) Mirror 5: 0x4188c6ba (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ea14: skipped 000315 1790712467.147692 mapread 0x01ab58 thru 0x02608b (0xb534 bytes) 000316 1790712467.176606 write 0x008ac3 thru 0x00dbf3 (0x5131 bytes) 000318 1790712467.189405 read 0x01425c thru 0x022c3a (0xe9df bytes) 000319 1790712467.195477 trunc from 0x0104d5 to 0x014f19 (0x4a45 bytes) 000320 1790712467.200797 punch from 0x019dd7 to 0x01f9e3 (0x5c0d bytes) 000321 1790712467.204819 trunc from 0x01dd6e to 0x028fff (0xb292 bytes) 000323 1790712467.209483 trunc from 0x02ea14 to 0x034610 (0x5bfd bytes) 000324 1790712467.223368 mapwrite 0x03b65a thru 0x03ffff (0x49a6 bytes) 000325 1790712467.255005 mapread 0x0047f5 thru 0x00ac6f (0x647b bytes) 000326 1790712467.266556 read 0x00a449 thru 0x00c599 (0x2151 bytes) 000327 1790712467.267180 punch from 0x0002a8 to 0x007809 (0x7562 bytes) 000329 1790712467.271633 trunc from 0x040000 to 0x033760 (0xc89f bytes) 000330 1790712467.286196 trunc from 0x033761 to 0x020835 (0x12f2b bytes) 000331 1790712467.315641 mapread 0x01fb4e thru 0x020835 (0xce8 bytes) 000332 1790712467.331585 write 0x032a3f thru 0x036ce6 (0x42a8 bytes) 000333 1790712467.334047 trunc from 0x02523c to 0x02697c (0x1741 bytes) 000334 1790712467.350003 trunc from 0x036ce7 to 0x002bd9 (0x3410d bytes) 000335 1790712467.364656 read 0x00137b thru 0x002bd9 (0x185f bytes) 000336 1790712467.365561 trunc from 0x002bda to 0x0183ce (0x157f5 bytes) 000337 1790712467.386508 fallocate 0x0111da thru 0x015cec (0x4b13 bytes) 000338 1790712467.391054 trunc from 0x004456 to 0x00bae6 (0x7691 bytes) 000339 1790712467.399298 mapwrite 0x016b39 thru 0x0249aa (0xde72 bytes) 000340 1790712467.458460 mapread 0x007856 thru 0x01362e (0xbdd9 bytes) 000341 1790712467.460456 read 0x002686 thru 0x003c82 (0x15fd bytes) 000342 1790712467.465470 read 0x006fdb thru 0x00bd5e (0x4d84 bytes) 000343 1790712467.469355 mapwrite 0x00efc2 thru 0x019fbb (0xaffa bytes) 000344 1790712467.509866 mapwrite 0x000a68 thru 0x006e7f (0x6418 bytes) 000346 1790712467.536671 mapwrite 0x01ef15 thru 0x023f77 (0x5063 bytes) 000348 1790712467.557661 mapread 0x011296 thru 0x013dc4 (0x2b2f bytes) 000350 1790712467.559159 mapread 0x00ee40 thru 0x014565 (0x5726 bytes) 000351 1790712467.560838 write 0x0021d4 thru 0x00fea4 (0xdcd1 bytes) 000352 1790712467.562321 mapread 0x00e4d8 thru 0x013a1c (0x5545 bytes) 000353 1790712467.563652 punch from 0x0128ea to 0x01e4f8 (0xbc0f bytes) 000354 1790712467.568102 punch from 0x00a53a to 0x00dd84 (0x384b bytes) 000355 1790712467.575374 mapwrite 0x0278f5 thru 0x029fef (0x26fb bytes) 000356 1790712467.617195 fallocate 0x02335c thru 0x02839b (0x5040 bytes) 000357 1790712467.624692 MIRROR_RESYNC 3 mirrors 000358 1790712468.766624 fallocate 0x01b34a thru 0x02a4b7 (0xf16e bytes) 000359 1790712468.795020 trunc from 0x011d50 to 0x01fc7f (0xdf30 bytes) 000361 1790712468.807607 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a4b8): Mirror 5: 0xd5e053b7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a4b8): Mirror 5: 0xd5e053b7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a4b8: skipped 000362 1790712469.114936 trunc from 0x02a4b8 to 0x00ee97 (0x1b620 bytes) 000363 1790712469.147280 trunc from 0x00ee98 to 0x00c6d7 (0x27c0 bytes) 000364 1790712469.171716 mapwrite 0x0380d6 thru 0x038d2f (0xc5a bytes) 000365 1790712469.235327 trunc from 0x038d30 to 0x012d46 (0x25fe9 bytes) 000366 1790712469.251935 read 0x00a94a thru 0x00e3a7 (0x3a5e bytes) 000367 1790712469.263888 trunc from 0x010f32 to 0x011720 (0x7ef bytes) 000368 1790712469.269244 trunc from 0x009fce to 0x012d46 (0x8d79 bytes) 000369 1790712469.275286 write 0x031a37 thru 0x0364a5 (0x4a6f bytes) 000370 1790712469.276475 mapread 0x0054a9 thru 0x00b5b0 (0x6108 bytes) 000371 1790712469.292239 trunc from 0x00ec85 to 0x013f49 (0x52c5 bytes) 000372 1790712469.300344 fallocate 0x01fef8 thru 0x024e08 (0x4f11 bytes) 000373 1790712469.309886 trunc from 0x0364a6 to 0x033f1c (0x2589 bytes) 000374 1790712469.330397 trunc from 0x01006b to 0x01d70e (0xd6a4 bytes) 000375 1790712469.335637 mapread 0x02795a thru 0x033f1c (0xc5c3 bytes) 000376 1790712469.356339 trunc from 0x00721c to 0x010b46 (0x992b bytes) 000378 1790712469.365491 punch from 0x00a805 to 0x00fd8c (0x5588 bytes) 000380 1790712469.371090 read 0x000293 thru 0x005e7e (0x5bec bytes) 000381 1790712469.380383 fallocate 0x01326f thru 0x021bf8 (0xe98a bytes) 000382 1790712469.385261 punch from 0x0019c2 to 0x003076 (0x16b5 bytes) 000383 1790712469.390237 punch from 0x0002c1 to 0x0018b3 (0x15f3 bytes) 000384 1790712469.395851 punch from 0x02eecb to 0x033f1c (0x5052 bytes) 000385 1790712469.405461 read 0x006687 thru 0x009b67 (0x34e1 bytes) 000386 1790712469.406357 punch from 0x016fa2 to 0x0211ff (0xa25e bytes) 000387 1790712469.412367 read 0x024f75 thru 0x031ce8 (0xcd74 bytes) 000388 1790712469.432396 write 0x032eee thru 0x03de1f (0xaf32 bytes) 000389 1790712469.435266 fallocate 0x00296e thru 0x006593 (0x3c26 bytes) 000390 1790712469.444412 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3de20): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3de20): Mirror 5: 0xc3125ba9 (stale) CRC-32 checksum value for chunk [0, 0x3de20): Mirror 1: 0x74606b11 Mirror 6: 0x74606b11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3de20: skipped 000391 1790712470.543476 MIRROR_RESYNC 3 mirrors 000392 1790712471.386341 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) 000393 1790712471.854513 mapread 0x02399c thru 0x028241 (0x48a6 bytes) 000394 1790712471.926417 trunc from 0x03de20 to 0x033055 (0xadca bytes) 000395 1790712472.033781 mapwrite 0x025e3b thru 0x03157e (0xb744 bytes) 000396 1790712472.094479 trunc from 0x00ab16 to 0x00c9db (0x1ec6 bytes) 000397 1790712472.103056 punch from 0x01d6c0 to 0x027ea4 (0xa7e5 bytes) 000398 1790712472.114428 mapwrite 0x017ae1 thru 0x0279e5 (0xff05 bytes) 000399 1790712472.164158 fallocate 0x0201c9 thru 0x0222c1 (0x20f9 bytes) 000400 1790712472.170201 read 0x02e52f thru 0x0321b1 (0x3c83 bytes) 000401 1790712472.176902 fallocate 0x010c8f thru 0x01828f (0x7601 bytes) 000402 1790712472.192642 mapread 0x00094e thru 0x000cbc (0x36f bytes) 000403 1790712472.224485 write 0x00c430 thru 0x0113db (0x4fac bytes) 000404 1790712472.228424 fallocate 0x0156b0 thru 0x022ad4 (0xd425 bytes) 000405 1790712472.247273 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x33056): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33056): Mirror 6: 0xde82b38e (stale) CRC-32 checksum value for chunk [0, 0x33056): Mirror 1: 0x72700ec3 Mirror 7: 0x72700ec3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33056: skipped 000406 1790712473.412512 mapread 0x005251 thru 0x01202d (0xcddd bytes) 000407 1790712473.450400 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33056): Mirror 6: 0xde82b38e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33056): Mirror 6: 0xde82b38e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33056: skipped 000408 1790712474.017307 fallocate 0x00adf1 thru 0x015ed1 (0xb0e1 bytes) 000409 1790712474.042872 write 0x02cfd7 thru 0x038f60 (0xbf8a bytes) 000410 1790712474.093067 MIRROR_RESYNC 2 mirrors 000412 1790712475.459348 MIRROR_EXTEND to 3 mirrors 000413 1790712476.200597 trunc from 0x038f61 to 0x003b9a (0x353c6 bytes) 000414 1790712476.265280 trunc from 0x002c1b to 0x003b9a (0xf80 bytes) 000415 1790712476.274624 mapwrite 0x01e678 thru 0x021d22 (0x36ab bytes) 000417 1790712476.371098 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21d23): Mirror 6: 0x544c9f32 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21d23): Mirror 6: 0x544c9f32 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21d23: skipped 000419 1790712476.620654 mapread 0x0036bb thru 0x00d488 (0x9dce bytes) 000420 1790712476.643719 trunc from 0x02082a to 0x021d22 (0x14f9 bytes) 000422 1790712476.648875 trunc from 0x021d23 to 0x011037 (0x10ceb bytes) 000423 1790712476.672347 fallocate 0x0203f9 thru 0x026262 (0x5e6a bytes) 000424 1790712476.678201 read 0x0039a3 thru 0x0040e2 (0x740 bytes) 000425 1790712476.679196 write 0x03463e thru 0x0373ed (0x2db0 bytes) 000426 1790712476.681606 read 0x01b0c4 thru 0x01b62e (0x56b bytes) 000427 1790712476.690640 mapwrite 0x0220e8 thru 0x02edf8 (0xcd11 bytes) 000428 1790712476.714495 mapread 0x02101c thru 0x0275ae (0x6593 bytes) 000429 1790712476.719486 MIRROR_RESYNC 2 mirrors 000430 1790712477.575626 punch from 0x018ff7 to 0x0230f0 (0xa0fa bytes) 000431 1790712477.601980 trunc from 0x0373ee to 0x030476 (0x6f77 bytes) 000432 1790712477.623290 punch from 0x028195 to 0x029f88 (0x1df4 bytes) 000433 1790712477.627555 mapread 0x01d17c thru 0x021314 (0x4199 bytes) 000434 1790712477.650102 fallocate 0x00ce3f thru 0x01bcf4 (0xeeb6 bytes) 000435 1790712477.657027 trunc from 0x030477 to 0x005dcb (0x2a6ab bytes) 000436 1790712477.685457 mapwrite 0x035b5c thru 0x03dc39 (0x80de bytes) 000437 1790712477.759183 trunc from 0x017f70 to 0x01cf4a (0x4fdb bytes) 000438 1790712477.766065 mapwrite 0x00da3b thru 0x019cab (0xc271 bytes) 000440 1790712477.847180 trunc from 0x037a05 to 0x03dc39 (0x6235 bytes) 000441 1790712477.865150 mapread 0x02bf88 thru 0x032f02 (0x6f7b bytes) 000442 1790712477.885270 fallocate 0x028f52 thru 0x02f526 (0x65d5 bytes) 000443 1790712477.911842 fallocate 0x036b9f thru 0x037ac0 (0xf22 bytes) 000444 1790712477.918875 read 0x03b1d3 thru 0x03dc39 (0x2a67 bytes) 000445 1790712477.927428 mapwrite 0x00346b thru 0x005083 (0x1c19 bytes) 000446 1790712477.971622 punch from 0x01212b to 0x01250f (0x3e5 bytes) 000447 1790712477.976928 trunc from 0x03dc3a to 0x021dc6 (0x1be73 bytes) 000448 1790712477.990650 read 0x010eba thru 0x01aab6 (0x9bfd bytes) 000449 1790712477.998804 write 0x021dd3 thru 0x02708a (0x52b8 bytes) 000451 1790712478.001960 mapread 0x018ed3 thru 0x021215 (0x8343 bytes) 000452 1790712478.003350 read 0x001176 thru 0x0097b3 (0x863e bytes) 000453 1790712478.004496 trunc from 0x02708b to 0x024e51 (0x2239 bytes) 000454 1790712478.023433 write 0x02ad6d thru 0x03a7ff (0xfa93 bytes) 000455 1790712478.028448 trunc from 0x03a800 to 0x017306 (0x234f9 bytes) 000456 1790712478.068595 trunc from 0x00af91 to 0x0162d4 (0xb344 bytes) 000457 1790712478.080924 fallocate 0x007c88 thru 0x017703 (0xfa7c bytes) 000458 1790712478.086158 fallocate 0x027296 thru 0x02887c (0x15e7 bytes) 000459 1790712478.092593 mapwrite 0x03d796 thru 0x03ffff (0x286a bytes) 000460 1790712478.137456 mapread 0x03bee2 thru 0x03fac7 (0x3be6 bytes) 000461 1790712478.139229 mapwrite 0x02bdd9 thru 0x03b14f (0xf377 bytes) 000462 1790712478.182110 mapread 0x00ee55 thru 0x014285 (0x5431 bytes) 000463 1790712478.183036 punch from 0x039eb1 to 0x03c2e8 (0x2438 bytes) 000464 1790712478.193253 punch from 0x0050ef to 0x007dcf (0x2ce1 bytes) 000465 1790712478.199425 mapwrite 0x0301ae thru 0x034f15 (0x4d68 bytes) 000466 1790712478.229608 read 0x009f02 thru 0x00deb3 (0x3fb2 bytes) 000467 1790712478.230467 trunc from 0x035e12 to 0x03ffff (0xa1ee bytes) 000468 1790712478.241474 trunc from 0x029048 to 0x031ead (0x8e66 bytes) 000469 1790712478.255318 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0xc0966863 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x809b73d2 Mirror 7: 0x809b73d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000470 1790712479.182075 mapwrite 0x00b93d thru 0x019cef (0xe3b3 bytes) 000471 1790712479.289516 trunc from 0x01db35 to 0x023261 (0x572d bytes) 000472 1790712479.296629 mapwrite 0x03e66e thru 0x03fefa (0x188d bytes) 000473 1790712479.313855 read 0x0339bd thru 0x033b4e (0x192 bytes) 000474 1790712479.340437 MIRROR_RESYNC 3 mirrors 000475 1790712480.906389 write 0x00de07 thru 0x010a94 (0x2c8e bytes) 000476 1790712480.968375 mapwrite 0x0145b2 thru 0x020842 (0xc291 bytes) 000478 1790712481.100429 write 0x001dfc thru 0x009eea (0x80ef bytes) 000479 1790712481.141031 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x829daae7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0x829daae7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000480 1790712481.574321 punch from 0x01e337 to 0x0204a6 (0x2170 bytes) 000482 1790712481.586030 mapread 0x028981 thru 0x02c743 (0x3dc3 bytes) 000483 1790712481.632768 trunc from 0x019d38 to 0x022eab (0x9174 bytes) 000484 1790712481.655649 MIRROR_RESYNC 2 mirrors 000485 1790712482.852446 trunc from 0x02fe07 to 0x03b986 (0xbb80 bytes) 000486 1790712482.905681 MIRROR_RESYNC 2 mirrors 000487 1790712484.191944 read 0x03b10d thru 0x03ffff (0x4ef3 bytes) 000488 1790712484.208488 mapread 0x029efd thru 0x02a287 (0x38b bytes) 000489 1790712484.226054 mapwrite 0x03b188 thru 0x03ffff (0x4e78 bytes) 000490 1790712484.426515 trunc from 0x034fce to 0x03d71e (0x8751 bytes) 000491 1790712484.462637 MIRROR_RESYNC 2 mirrors 000493 1790712485.425233 write 0x03061b thru 0x039365 (0x8d4b bytes) 000494 1790712485.471414 MIRROR_RESYNC 2 mirrors 000495 1790712486.591751 mapwrite 0x0015f6 thru 0x009828 (0x8233 bytes) 000496 1790712486.728344 mapread 0x029b35 thru 0x02c485 (0x2951 bytes) 000497 1790712486.742107 read 0x0144f8 thru 0x0168ed (0x23f6 bytes) 000498 1790712486.753777 MIRROR_RESYNC 2 mirrors 000499 1790712487.529110 punch from 0x024980 to 0x02717d (0x27fe bytes) 000500 1790712487.566399 mapwrite 0x013fef thru 0x020683 (0xc695 bytes) 000501 1790712487.629426 fallocate 0x01666a thru 0x0234b5 (0xce4c bytes) 000502 1790712487.634323 read 0x0382c9 thru 0x03ffff (0x7d37 bytes) 000503 1790712487.645700 read 0x0013a8 thru 0x00c651 (0xb2aa bytes) 000504 1790712487.657273 punch from 0x02d9cf to 0x035af2 (0x8124 bytes) 000505 1790712487.663252 write 0x012647 thru 0x022068 (0xfa22 bytes) 000506 1790712487.667064 fallocate 0x00c76d thru 0x0106fa (0x3f8e bytes) 000507 1790712487.672195 punch from 0x01f853 to 0x02d147 (0xd8f5 bytes) 000508 1790712487.688681 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xf214768f (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4bd5c417 Mirror 8: 0x4bd5c417 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000509 1790712488.853101 trunc from 0x035c03 to 0x03ffff (0xa3fd bytes) 000510 1790712488.982720 punch from 0x0371cb to 0x03d21d (0x6053 bytes) 000511 1790712488.997474 trunc from 0x036100 to 0x03b1d5 (0x50d6 bytes) 000512 1790712489.009660 write 0x026419 thru 0x028574 (0x215c bytes) 000513 1790712489.066301 punch from 0x03eb8a to 0x03ffff (0x1476 bytes) 000514 1790712489.093020 trunc from 0x01e74f to 0x022dd7 (0x4689 bytes) 000516 1790712489.103980 trunc from 0x040000 to 0x00ac8e (0x35371 bytes) 000517 1790712489.145553 trunc from 0x000c5f to 0x0069af (0x5d51 bytes) 000518 1790712489.163047 trunc from 0x00ac8f to 0x03ea95 (0x33e07 bytes) 000519 1790712489.224277 read 0x02e744 thru 0x034a46 (0x6303 bytes) 000520 1790712489.266072 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3ea96): Mirror 7: 0x22f50c15 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ea96): Mirror 7: 0x22f50c15 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ea96: skipped 000522 1790712489.696604 trunc from 0x03ea96 to 0x009249 (0x3584c bytes) 000523 1790712489.728486 trunc from 0x00924a to 0x010439 (0x71f0 bytes) 000525 1790712489.742706 trunc from 0x01043a to 0x02b245 (0x1ae0c bytes) 000526 1790712489.766693 MIRROR_RESYNC 2 mirrors 000527 1790712490.486923 write 0x02945d thru 0x03367c (0xa220 bytes) 000528 1790712490.532196 fallocate 0x033d2a thru 0x03605d (0x2334 bytes) 000529 1790712490.543223 write 0x014f94 thru 0x01e021 (0x908e bytes) 000530 1790712490.564290 punch from 0x028de4 to 0x03367c (0xa899 bytes) 000531 1790712490.593209 punch from 0x010e3d to 0x01bdff (0xafc3 bytes) 000532 1790712490.599082 trunc from 0x03367d to 0x02bec5 (0x77b7 bytes) 000533 1790712490.629996 mapread 0x026487 thru 0x028bab (0x2725 bytes) 000534 1790712490.650655 mapread 0x0176af thru 0x01ad60 (0x36b2 bytes) 000535 1790712490.664105 fallocate 0x00fe4f thru 0x01addc (0xaf8e bytes) 000536 1790712490.669319 write 0x03a5f5 thru 0x03ee05 (0x4811 bytes) 000537 1790712490.671423 trunc from 0x03ee06 to 0x03e9f3 (0x412 bytes) 000538 1790712490.691204 trunc from 0x01e348 to 0x022891 (0x454a bytes) 000539 1790712490.697918 write 0x00adf2 thru 0x00d186 (0x2395 bytes) 000541 1790712490.708667 fallocate 0x0287df thru 0x028ac9 (0x2eb bytes) 000542 1790712490.718472 MIRROR_RESYNC 2 mirrors 000543 1790712491.289674 MIRROR_EXTEND to 3 mirrors 000544 1790712492.002705 mapwrite 0x0095b4 thru 0x0160a8 (0xcaf5 bytes) 000546 1790712492.105399 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3e9f4): Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e9f4): Mirror 7: 0x45023d2a (stale) Mirror 8: 0x45023d2a (stale) CRC-32 checksum value for chunk [0, 0x3e9f4): Mirror 1: 0xedf6d664 Mirror 9: 0xedf6d664 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e9f4: skipped 000547 1790712493.196220 mapwrite 0x023eba thru 0x025484 (0x15cb bytes) 000548 1790712493.318682 write 0x039059 thru 0x03ffff (0x6fa7 bytes) 000549 1790712493.332894 fallocate 0x00819f thru 0x00d1e2 (0x5044 bytes) 000550 1790712493.339199 punch from 0x00931d to 0x011883 (0x8567 bytes) 000551 1790712493.347863 trunc from 0x01b406 to 0x01cfce (0x1bc9 bytes) 000552 1790712493.355701 punch from 0x011861 to 0x01ee8f (0xd62f bytes) 000553 1790712493.363372 write 0x02e640 thru 0x034d56 (0x6717 bytes) 000555 1790712493.378572 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xf419c75b (stale) Mirror 8: 0xf419c75b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 7: 0xf419c75b (stale) Mirror 8: 0xf419c75b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000556 1790712493.710874 trunc from 0x03ef59 to 0x03fc87 (0xd2f bytes) 000557 1790712493.719854 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xf419c75b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0xf419c75b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000559 1790712493.993231 mapread 0x015209 thru 0x016603 (0x13fb bytes) 000560 1790712494.004020 trunc from 0x040000 to 0x02f307 (0x10cf8 bytes) 000561 1790712494.069917 trunc from 0x02f308 to 0x01f382 (0xff85 bytes) 000562 1790712494.101852 mapwrite 0x037120 thru 0x037291 (0x172 bytes) 000563 1790712494.147385 mapwrite 0x034146 thru 0x03ffff (0xbeba bytes) 000564 1790712494.211134 mapread 0x03ab8e thru 0x03ffff (0x5472 bytes) 000565 1790712494.212950 trunc from 0x0172c1 to 0x0201db (0x8f1b bytes) 000566 1790712494.218536 read 0x015260 thru 0x0199a8 (0x4749 bytes) 000567 1790712494.223627 read 0x00a897 thru 0x01a4ff (0xfc69 bytes) 000568 1790712494.234658 fallocate 0x017da3 thru 0x01e066 (0x62c4 bytes) 000569 1790712494.239279 punch from 0x001527 to 0x00529f (0x3d79 bytes) 000570 1790712494.244040 fallocate 0x000bb6 thru 0x010266 (0xf6b1 bytes) 000571 1790712494.249071 mapread 0x00c5eb thru 0x01c247 (0xfc5d bytes) 000572 1790712494.251713 write 0x0128f0 thru 0x01ea02 (0xc113 bytes) 000574 1790712494.255250 punch from 0x03267d to 0x03f48c (0xce10 bytes) 000575 1790712494.259976 fallocate 0x008e09 thru 0x00aa86 (0x1c7e bytes) 000576 1790712494.269091 punch from 0x012089 to 0x012d78 (0xcf0 bytes) 000577 1790712494.284812 write 0x03e4e9 thru 0x03ffff (0x1b17 bytes) 000578 1790712494.289606 read 0x03def2 thru 0x03ffff (0x210e bytes) 000580 1790712494.298108 mapwrite 0x033403 thru 0x03eecb (0xbac9 bytes) 000581 1790712494.315172 mapread 0x005c5d thru 0x00f526 (0x98ca bytes) 000582 1790712494.316120 punch from 0x02d68b to 0x030cc5 (0x363b bytes) 000583 1790712494.321138 trunc from 0x040000 to 0x007c6f (0x38390 bytes) 000584 1790712494.338608 mapread 0x006cf1 thru 0x007c6f (0xf7f bytes) 000585 1790712494.339604 mapwrite 0x022c3c thru 0x02e4ca (0xb88f bytes) 000586 1790712494.426683 mapread 0x017670 thru 0x02232c (0xacbd bytes) 000587 1790712494.478259 trunc from 0x0069b1 to 0x0101f1 (0x9841 bytes) 000588 1790712494.486612 trunc from 0x000753 to 0x009120 (0x89ce bytes) 000589 1790712494.496713 fallocate 0x017dcc thru 0x01e124 (0x6359 bytes) 000590 1790712494.503261 punch from 0x01767a to 0x019354 (0x1cdb bytes) 000591 1790712494.509991 read 0x0202e2 thru 0x0206e4 (0x403 bytes) 000592 1790712494.512104 mapread 0x02843d thru 0x02e4ca (0x608e bytes) 000593 1790712494.514138 mapwrite 0x02bc70 thru 0x03833f (0xc6d0 bytes) 000594 1790712494.574256 punch from 0x037423 to 0x03833f (0xf1d bytes) 000596 1790712494.580193 mapread 0x03615b thru 0x03833f (0x21e5 bytes) 000597 1790712494.590429 punch from 0x0094f6 to 0x00ac77 (0x1782 bytes) 000598 1790712494.595107 mapread 0x0194b7 thru 0x025b5d (0xc6a7 bytes) 000600 1790712494.596924 write 0x03be2e thru 0x03ffff (0x41d2 bytes) 000601 1790712494.599063 fallocate 0x0168f9 thru 0x01bb82 (0x528a bytes) 000602 1790712494.604389 mapwrite 0x00b153 thru 0x00dd94 (0x2c42 bytes) 000604 1790712494.622001 trunc from 0x040000 to 0x015377 (0x2ac88 bytes) 000605 1790712494.638675 read 0x004478 thru 0x00d60e (0x9197 bytes) 000606 1790712494.646005 trunc from 0x00aaf4 to 0x00b7ed (0xcfa bytes) 000607 1790712494.655513 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x15378): Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15378): Mirror 8: 0xc8068912 (stale) CRC-32 checksum value for chunk [0, 0x15378): Mirror 1: 0xe006dac1 Mirror 9: 0xe006dac1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15378: skipped 000609 1790712495.602991 fallocate 0x03a949 thru 0x03ffff (0x56b7 bytes) 000610 1790712495.664373 trunc from 0x0081cd to 0x00b6ad (0x34e1 bytes) 000611 1790712495.669240 trunc from 0x040000 to 0x02ecd1 (0x1132e bytes) 000612 1790712495.696143 mapread 0x00433c thru 0x00a5a7 (0x626c bytes) 000613 1790712495.733572 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ecd2): Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ecd2): Mirror 8: 0xa0b4eef8 (stale) Mirror 9: 0x4c494907 (stale) CRC-32 checksum value for chunk [0, 0x2ecd2): Mirror 1: 0x4c494907 Mirror 10: 0x4c494907 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ecd2: skipped 000614 1790712496.792959 write 0x00e662 thru 0x011ee2 (0x3881 bytes) 000615 1790712496.879327 read 0x01a434 thru 0x02571c (0xb2e9 bytes) 000616 1790712496.910119 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2ecd2): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ecd2): Mirror 8: 0xa0b4eef8 (stale) Mirror 9: 0x4c494907 (stale) Mirror 10: 0x4c494907 (stale) CRC-32 checksum value for chunk [0, 0x2ecd2): Mirror 1: 0xaafef7bf Mirror 11: 0xaafef7bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ecd2: skipped 000617 1790712497.949247 read 0x021871 thru 0x02b7da (0x9f6a bytes) 000618 1790712497.964418 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2ecd2): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ecd2): Mirror 8: 0xa0b4eef8 (stale) Mirror 9: 0x4c494907 (stale) Mirror 10: 0x4c494907 (stale) CRC-32 checksum value for chunk [0, 0x2ecd2): Mirror 1: 0xaafef7bf Mirror 11: 0xaafef7bf Mirror 12: 0xaafef7bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ecd2: skipped 000619 1790712499.205095 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2ecd2): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ecd2): Mirror 8: 0xa0b4eef8 (stale) Mirror 9: 0x4c494907 (stale) Mirror 10: 0x4c494907 (stale) CRC-32 checksum value for chunk [0, 0x2ecd2): Mirror 1: 0xaafef7bf Mirror 11: 0xaafef7bf Mirror 12: 0xaafef7bf Mirror 13: 0xaafef7bf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ecd2: skipped 000620 1790712500.191857 mapwrite 0x03cd5b thru 0x03ffff (0x32a5 bytes) 000621 1790712500.307155 fallocate 0x016fca thru 0x020cd3 (0x9d0a bytes) 000622 1790712500.316029 read 0x037e73 thru 0x03ffff (0x818d bytes) 000623 1790712500.316629 read 0x001945 thru 0x008101 (0x67bd bytes) 000625 1790712500.347120 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xbe613171 (stale) Mirror 10: 0xbe613171 (stale) Mirror 11: 0xb6ab589 (stale) Mirror 12: 0xb6ab589 (stale) Mirror 13: 0xb6ab589 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xbe613171 (stale) Mirror 10: 0xbe613171 (stale) Mirror 11: 0xb6ab589 (stale) Mirror 12: 0xb6ab589 (stale) Mirror 13: 0xb6ab589 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000626 1790712500.892492 trunc from 0x017591 to 0x022b85 (0xb5f5 bytes) 000627 1790712500.899337 write 0x01f0c7 thru 0x02d6e2 (0xe61c bytes) 000629 1790712500.915148 trunc from 0x023f22 to 0x02f714 (0xb7f3 bytes) 000630 1790712500.928483 trunc from 0x040000 to 0x018938 (0x276c7 bytes) 000631 1790712500.947449 punch from 0x0027f9 to 0x00e093 (0xb89b bytes) 000632 1790712500.953065 fallocate 0x018e85 thru 0x025966 (0xcae2 bytes) 000633 1790712500.959308 punch from 0x000cdd to 0x000d20 (0x44 bytes) 000634 1790712500.963529 mapwrite 0x03aaeb thru 0x03ffff (0x5515 bytes) 000635 1790712501.018369 mapread 0x002afd thru 0x00f460 (0xc964 bytes) 000636 1790712501.035251 read 0x011e89 thru 0x016164 (0x42dc bytes) 000637 1790712501.047978 mapread 0x021ca1 thru 0x0242ce (0x262e bytes) 000638 1790712501.062391 punch from 0x02deac to 0x030376 (0x24cb bytes) 000639 1790712501.067226 fallocate 0x0317a0 thru 0x0357c7 (0x4028 bytes) 000640 1790712501.071182 mapwrite 0x01e01f thru 0x026dcc (0x8dae bytes) 000641 1790712501.103152 trunc from 0x040000 to 0x03eb00 (0x14ff bytes) 000642 1790712501.142287 trunc from 0x03eb01 to 0x01954a (0x255b6 bytes) 000643 1790712501.163184 trunc from 0x01954b to 0x03c814 (0x232ca bytes) 000644 1790712501.182371 trunc from 0x03c815 to 0x023824 (0x18ff0 bytes) 000645 1790712501.210005 trunc from 0x023825 to 0x030dae (0xd58a bytes) 000646 1790712501.233001 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x30daf): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30daf): Mirror 9: 0x4b2777fb (stale) Mirror 10: 0x4b2777fb (stale) Mirror 11: 0x2bfd1436 (stale) Mirror 12: 0x2bfd1436 (stale) Mirror 13: 0x2bfd1436 (stale) CRC-32 checksum value for chunk [0, 0x30daf): Mirror 1: 0xb75d80bc Mirror 14: 0xb75d80bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30daf: skipped 000649 1790712502.167985 fallocate 0x00331d thru 0x0072a7 (0x3f8b bytes) 000651 1790712502.216286 trunc from 0x030daf to 0x034173 (0x33c5 bytes) 000652 1790712502.262699 fallocate 0x029f8f thru 0x02c669 (0x26db bytes) 000653 1790712502.269768 read 0x00c35d thru 0x0150b4 (0x8d58 bytes) 000654 1790712502.280935 read 0x031350 thru 0x034173 (0x2e24 bytes) 000655 1790712502.292559 trunc from 0x005fb0 to 0x00b5e0 (0x5631 bytes) 000656 1790712502.297959 write 0x002d56 thru 0x010559 (0xd804 bytes) 000657 1790712502.300574 read 0x022f78 thru 0x02edfe (0xbe87 bytes) 000658 1790712502.304275 punch from 0x01a171 to 0x01a816 (0x6a6 bytes) 000659 1790712502.316410 trunc from 0x034174 to 0x023d1c (0x10457 bytes) 000660 1790712502.340464 read 0x022900 thru 0x022deb (0x4ec bytes) 000661 1790712502.342481 punch from 0x0172b1 to 0x01e53b (0x728b bytes) 000663 1790712502.348410 mapwrite 0x03630c thru 0x03cb4d (0x6842 bytes) 000665 1790712502.412131 read 0x02b819 thru 0x032072 (0x685a bytes) 000666 1790712502.412837 read 0x01f334 thru 0x02982a (0xa4f7 bytes) 000667 1790712502.414919 write 0x0047e8 thru 0x00c130 (0x7949 bytes) 000668 1790712502.417135 trunc from 0x02d948 to 0x0339fc (0x60b5 bytes) 000669 1790712502.422277 punch from 0x030930 to 0x031ac2 (0x1193 bytes) 000670 1790712502.427031 fallocate 0x009cfa thru 0x00c6f9 (0x2a00 bytes) 000671 1790712502.431576 read 0x03290f thru 0x035fa1 (0x3693 bytes) 000672 1790712502.442088 write 0x003643 thru 0x012974 (0xf332 bytes) 000674 1790712502.445448 fallocate 0x01276f thru 0x017127 (0x49b9 bytes) 000675 1790712502.451124 mapread 0x00f6cd thru 0x01a599 (0xaecd bytes) 000676 1790712502.457927 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3cb4e): Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb4e): Mirror 9: 0x34d32702 (stale) Mirror 10: 0x34d32702 (stale) Mirror 11: 0x9d8536e0 (stale) Mirror 12: 0x9d8536e0 (stale) Mirror 13: 0x9d8536e0 (stale) Mirror 14: 0xa9abb54c (stale) CRC-32 checksum value for chunk [0, 0x3cb4e): Mirror 1: 0xe36a2ed Mirror 15: 0xe36a2ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb4e: skipped 000677 1790712503.585175 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cb4e): Mirror 9: 0x34d32702 (stale) Mirror 11: 0x9d8536e0 (stale) Mirror 12: 0x9d8536e0 (stale) Mirror 13: 0x9d8536e0 (stale) Mirror 14: 0xa9abb54c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb4e): Mirror 9: 0x34d32702 (stale) Mirror 11: 0x9d8536e0 (stale) Mirror 12: 0x9d8536e0 (stale) Mirror 13: 0x9d8536e0 (stale) Mirror 14: 0xa9abb54c (stale) CRC-32 checksum value for chunk [0, 0x3cb4e): Mirror 1: 0xe36a2ed Mirror 15: 0xe36a2ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb4e: skipped 000678 1790712504.191477 read 0x01b2c2 thru 0x021d29 (0x6a68 bytes) 000679 1790712504.225137 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3cb4e): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb4e): Mirror 9: 0x34d32702 (stale) Mirror 11: 0x9d8536e0 (stale) Mirror 12: 0x9d8536e0 (stale) Mirror 13: 0x9d8536e0 (stale) Mirror 14: 0xa9abb54c (stale) CRC-32 checksum value for chunk [0, 0x3cb4e): Mirror 1: 0xe36a2ed Mirror 15: 0xe36a2ed Mirror 16: 0xe36a2ed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb4e: skipped 000680 1790712505.410158 punch from 0x01b0df to 0x01c726 (0x1648 bytes) 000681 1790712505.479748 fallocate 0x034684 thru 0x03ffff (0xb97c bytes) 000682 1790712505.488787 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xbe613171 (stale) Mirror 11: 0xb6ab589 (stale) Mirror 12: 0xb6ab589 (stale) Mirror 13: 0xb6ab589 (stale) Mirror 14: 0x75e52839 (stale) Mirror 15: 0xdfa24792 (stale) Mirror 16: 0xdfa24792 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdfa24792 Mirror 17: 0xdfa24792 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000683 1790712506.824279 write 0x00bd1a thru 0x013f88 (0x826f bytes) 000684 1790712506.907371 mapread 0x03ecdc thru 0x03ffff (0x1324 bytes) 000685 1790712506.926301 mapwrite 0x009c49 thru 0x0107a5 (0x6b5d bytes) 000686 1790712506.962183 trunc from 0x040000 to 0x028f69 (0x17096 bytes) 000688 1790712507.003628 trunc from 0x028f6a to 0x01bb01 (0xd468 bytes) 000690 1790712507.056125 fallocate 0x01c885 thru 0x024c8d (0x8409 bytes) 000691 1790712507.064940 read 0x0138b8 thru 0x016002 (0x274b bytes) 000692 1790712507.084175 fallocate 0x01530f thru 0x017db2 (0x2aa4 bytes) 000693 1790712507.089321 read 0x001055 thru 0x004550 (0x34fc bytes) 000694 1790712507.112019 trunc from 0x01bb02 to 0x00cef2 (0xec0f bytes) 000695 1790712507.136701 read 0x002178 thru 0x00c594 (0xa41d bytes) 000696 1790712507.137404 trunc from 0x00cef3 to 0x03f627 (0x32735 bytes) 000697 1790712507.158118 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3f628): Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f628): Mirror 9: 0x160aee1b (stale) Mirror 11: 0xb9a88b5c (stale) Mirror 12: 0xb9a88b5c (stale) Mirror 13: 0xb9a88b5c (stale) Mirror 14: 0xa53890e (stale) Mirror 15: 0x8ac7d825 (stale) Mirror 16: 0x8ac7d825 (stale) Mirror 17: 0x8ac7d825 (stale) CRC-32 checksum value for chunk [0, 0x3f628): Mirror 1: 0xb05195c8 Mirror 18: 0xb05195c8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f628: skipped 000698 1790712508.149104 read 0x03d838 thru 0x03f627 (0x1df0 bytes) 000699 1790712508.162008 mapwrite 0x01b50f thru 0x028ea4 (0xd996 bytes) 000700 1790712508.343969 punch from 0x0373d5 to 0x03f627 (0x8253 bytes) 000701 1790712508.356852 MIRROR_RESYNC 10 mirrors 000702 1790712511.563977 write 0x02cf8a thru 0x036691 (0x9708 bytes) 000704 1790712511.623996 write 0x020c28 thru 0x02e81a (0xdbf3 bytes) 000705 1790712511.630708 punch from 0x03936e to 0x03edef (0x5a82 bytes) 000706 1790712511.650969 read 0x00fe63 thru 0x01513f (0x52dd bytes) 000707 1790712511.659799 trunc from 0x03f628 to 0x035b59 (0x9ace bytes) 000708 1790712511.701909 trunc from 0x035b5a to 0x01ddee (0x17d6b bytes) 000709 1790712511.725125 punch from 0x013f96 to 0x017d62 (0x3dcd bytes) 000710 1790712511.731159 write 0x011261 thru 0x01c036 (0xadd6 bytes) 000711 1790712511.738916 fallocate 0x00c404 thru 0x00c7f4 (0x3f1 bytes) 000712 1790712511.745437 trunc from 0x01c1e4 to 0x01ddee (0x1c0b bytes) 000713 1790712511.781464 MIRROR_RESYNC 10 mirrors 000714 1790712513.880868 punch from 0x00ff15 to 0x01da76 (0xdb62 bytes) 000715 1790712513.916998 write 0x016cbb thru 0x01bd8c (0x50d2 bytes) 000716 1790712513.928212 trunc from 0x008bb7 to 0x00f992 (0x6ddc bytes) 000717 1790712513.932313 fallocate 0x003b61 thru 0x00cc56 (0x90f6 bytes) 000718 1790712513.936428 mapwrite 0x00806b thru 0x00eaf1 (0x6a87 bytes) 000720 1790712513.968983 mapwrite 0x0127a2 thru 0x021a3a (0xf299 bytes) 000721 1790712514.022959 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21a3b): Mirror 11: 0xe02653a0 (stale) Mirror 12: 0xe02653a0 (stale) Mirror 13: 0xe02653a0 (stale) Mirror 14: 0xe02653a0 (stale) Mirror 15: 0xe02653a0 (stale) Mirror 16: 0xe02653a0 (stale) Mirror 17: 0xe02653a0 (stale) Mirror 18: 0xe02653a0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21a3b): Mirror 11: 0xe02653a0 (stale) Mirror 12: 0xe02653a0 (stale) Mirror 13: 0xe02653a0 (stale) Mirror 14: 0xe02653a0 (stale) Mirror 15: 0xe02653a0 (stale) Mirror 16: 0xe02653a0 (stale) Mirror 17: 0xe02653a0 (stale) Mirror 18: 0xe02653a0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21a3b: skipped 000722 1790712514.703897 trunc from 0x021a3b to 0x015b44 (0xbef6 bytes) 000723 1790712514.738241 trunc from 0x010855 to 0x013260 (0x2a0c bytes) 000724 1790712514.745553 fallocate 0x00469a thru 0x00884a (0x41b1 bytes) 000727 1790712514.752690 punch from 0x000b7f to 0x00b38f (0xa811 bytes) 000729 1790712514.764007 read 0x002e1b thru 0x004c86 (0x1e6c bytes) 000730 1790712514.771797 read 0x00370a thru 0x006bad (0x34a4 bytes) 000731 1790712514.783550 trunc from 0x015b45 to 0x032fe5 (0x1d4a1 bytes) 000732 1790712514.811242 punch from 0x024d8c to 0x029035 (0x42aa bytes) 000733 1790712514.825468 write 0x0321cc thru 0x0343c6 (0x21fb bytes) 000734 1790712514.863571 trunc from 0x015504 to 0x016f5a (0x1a57 bytes) 000735 1790712514.874044 mapread 0x0268ad thru 0x030f64 (0xa6b8 bytes) 000736 1790712514.875046 trunc from 0x0343c7 to 0x02f011 (0x53b5 bytes) 000737 1790712514.904294 punch from 0x02393c to 0x029c2b (0x62f0 bytes) 000738 1790712514.914295 trunc from 0x02f012 to 0x031ca9 (0x2c98 bytes) 000740 1790712515.002982 trunc from 0x01fcf8 to 0x02569a (0x59a3 bytes) 000741 1790712515.010134 read 0x013e90 thru 0x023b7e (0xfcef bytes) 000742 1790712515.041383 trunc from 0x02f301 to 0x031ca9 (0x29a9 bytes) 000743 1790712515.052281 read 0x01ad3d thru 0x0268e6 (0xbbaa bytes) 000744 1790712515.059138 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31caa): Mirror 11: 0x41752879 (stale) Mirror 12: 0x41752879 (stale) Mirror 14: 0x41752879 (stale) Mirror 15: 0x41752879 (stale) Mirror 16: 0x41752879 (stale) Mirror 17: 0x41752879 (stale) Mirror 18: 0x41752879 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31caa): Mirror 11: 0x41752879 (stale) Mirror 12: 0x41752879 (stale) Mirror 14: 0x41752879 (stale) Mirror 15: 0x41752879 (stale) Mirror 16: 0x41752879 (stale) Mirror 17: 0x41752879 (stale) Mirror 18: 0x41752879 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31caa: skipped 000745 1790712515.674650 punch from 0x009788 to 0x0155ac (0xbe25 bytes) 000746 1790712515.681519 mapread 0x02f866 thru 0x031ca9 (0x2444 bytes) 000747 1790712515.699296 trunc from 0x01c949 to 0x02434e (0x7a06 bytes) 000748 1790712515.703038 trunc from 0x001f1d to 0x005da6 (0x3e8a bytes) 000749 1790712515.708162 read 0x02ae76 thru 0x0318d3 (0x6a5e bytes) 000750 1790712515.708883 fallocate 0x01b6e3 thru 0x026787 (0xb0a5 bytes) 000751 1790712515.713489 fallocate 0x017fa0 thru 0x01fd6e (0x7dcf bytes) 000752 1790712515.717411 write 0x00084e thru 0x00c719 (0xbecc bytes) 000753 1790712515.729238 trunc from 0x00afa8 to 0x01628a (0xb2e3 bytes) 000754 1790712515.740532 mapwrite 0x031fc1 thru 0x03cc59 (0xac99 bytes) 000755 1790712515.822497 mapread 0x020eec thru 0x029faa (0x90bf bytes) 000756 1790712515.843013 write 0x002bfc thru 0x0116bb (0xeac0 bytes) 000757 1790712515.864549 mapread 0x037232 thru 0x03cc59 (0x5a28 bytes) 000758 1790712515.879924 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3cc5a): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc5a): Mirror 11: 0x64705ee2 (stale) Mirror 12: 0x64705ee2 (stale) Mirror 14: 0x64705ee2 (stale) Mirror 15: 0x64705ee2 (stale) Mirror 16: 0x64705ee2 (stale) Mirror 17: 0x64705ee2 (stale) Mirror 18: 0x64705ee2 (stale) CRC-32 checksum value for chunk [0, 0x3cc5a): Mirror 1: 0xd1f42bb0 Mirror 19: 0xd1f42bb0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc5a: skipped 000759 1790712517.048094 trunc from 0x0020bf to 0x00adfb (0x8d3d bytes) 000760 1790712517.082101 write 0x01b41b thru 0x01d33d (0x1f23 bytes) 000761 1790712517.100083 mapwrite 0x0158bb thru 0x01d4f5 (0x7c3b bytes) 000762 1790712517.141975 fallocate 0x02d1d0 thru 0x032a77 (0x58a8 bytes) 000763 1790712517.151962 fallocate 0x00def7 thru 0x014881 (0x698b bytes) 000764 1790712517.179334 punch from 0x017d2c to 0x024f89 (0xd25e bytes) 000765 1790712517.196443 punch from 0x0000e9 to 0x00a98d (0xa8a5 bytes) 000766 1790712517.214324 fallocate 0x008c50 thru 0x008e0a (0x1bb bytes) 000767 1790712517.234001 MIRROR_RESYNC 9 mirrors 000768 1790712519.710418 MIRROR_EXTEND to 10 mirrors 000769 1790712520.669013 punch from 0x024806 to 0x02a532 (0x5d2d bytes) 000770 1790712520.709969 punch from 0x004023 to 0x007617 (0x35f5 bytes) 000771 1790712520.715367 write 0x009bc3 thru 0x016037 (0xc475 bytes) 000772 1790712520.733002 write 0x027c48 thru 0x03146c (0x9825 bytes) 000773 1790712520.756928 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cc5a): Mirror 11: 0x3458ddb4 (stale) Mirror 12: 0x3458ddb4 (stale) Mirror 15: 0x3458ddb4 (stale) Mirror 16: 0x3458ddb4 (stale) Mirror 17: 0x3458ddb4 (stale) Mirror 18: 0x3458ddb4 (stale) Mirror 19: 0x3458ddb4 (stale) Mirror 20: 0x3458ddb4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc5a): Mirror 11: 0x3458ddb4 (stale) Mirror 12: 0x3458ddb4 (stale) Mirror 15: 0x3458ddb4 (stale) Mirror 16: 0x3458ddb4 (stale) Mirror 17: 0x3458ddb4 (stale) Mirror 18: 0x3458ddb4 (stale) Mirror 19: 0x3458ddb4 (stale) Mirror 20: 0x3458ddb4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc5a: skipped 000774 1790712521.221772 punch from 0x009738 to 0x011c6d (0x8536 bytes) 000775 1790712521.231019 MIRROR_SPLIT mirror 8 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cc5a): Mirror 11: 0x3458ddb4 (stale) Mirror 12: 0x3458ddb4 (stale) Mirror 15: 0x3458ddb4 (stale) Mirror 16: 0x3458ddb4 (stale) Mirror 17: 0x3458ddb4 (stale) Mirror 18: 0x3458ddb4 (stale) Mirror 19: 0x3458ddb4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc5a): Mirror 11: 0x3458ddb4 (stale) Mirror 12: 0x3458ddb4 (stale) Mirror 15: 0x3458ddb4 (stale) Mirror 16: 0x3458ddb4 (stale) Mirror 17: 0x3458ddb4 (stale) Mirror 18: 0x3458ddb4 (stale) Mirror 19: 0x3458ddb4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc5a: skipped 000776 1790712521.740338 fallocate 0x03a9a6 thru 0x03ffff (0x565a bytes) 000777 1790712521.748239 read 0x01a554 thru 0x021242 (0x6cef bytes) 000778 1790712521.758111 mapwrite 0x00df80 thru 0x00f11d (0x119e bytes) 000779 1790712521.791948 trunc from 0x02bd3d to 0x0384b6 (0xc77a bytes) 000780 1790712521.798120 mapread 0x02f084 thru 0x03cc59 (0xdbd6 bytes) 000781 1790712521.815852 mapread 0x02ae16 thru 0x039c45 (0xee30 bytes) 000782 1790712521.817512 trunc from 0x03cc5a to 0x02065d (0x1c5fc bytes) 000784 1790712521.859236 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2065e): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2065e): Mirror 11: 0xdc57a665 (stale) Mirror 12: 0xdc57a665 (stale) Mirror 15: 0xdc57a665 (stale) Mirror 16: 0xdc57a665 (stale) Mirror 17: 0xdc57a665 (stale) Mirror 18: 0xdc57a665 (stale) Mirror 19: 0xdc57a665 (stale) CRC-32 checksum value for chunk [0, 0x2065e): Mirror 1: 0xa2448cfb Mirror 20: 0xa2448cfb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2065e: skipped 000785 1790712523.283586 MIRROR_RESYNC 9 mirrors 000786 1790712525.532858 mapread 0x003c2f thru 0x005c83 (0x2055 bytes) 000787 1790712525.548872 mapwrite 0x038a81 thru 0x03bbba (0x313a bytes) 000788 1790712525.614812 trunc from 0x017407 to 0x024bcd (0xd7c7 bytes) 000789 1790712525.620044 fallocate 0x02c31c thru 0x0314ab (0x5190 bytes) 000790 1790712525.624207 write 0x027354 thru 0x031421 (0xa0ce bytes) 000791 1790712525.629491 mapread 0x007e44 thru 0x00ca29 (0x4be6 bytes) 000793 1790712525.643359 MIRROR_RESYNC 9 mirrors 000794 1790712528.963191 fallocate 0x013878 thru 0x01f8a6 (0xc02f bytes) 000795 1790712528.993257 trunc from 0x03bbbb to 0x0076f6 (0x344c4 bytes) 000796 1790712529.016021 read 0x0051c3 thru 0x0076f6 (0x2534 bytes) 000797 1790712529.021802 trunc from 0x003caf to 0x006a14 (0x2d66 bytes) 000798 1790712529.029132 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x76f7): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x76f7): Mirror 11: 0xa0a7e53a (stale) Mirror 12: 0xa0a7e53a (stale) Mirror 15: 0xa0a7e53a (stale) Mirror 16: 0xa0a7e53a (stale) Mirror 17: 0xa0a7e53a (stale) Mirror 18: 0xa0a7e53a (stale) Mirror 19: 0xa0a7e53a (stale) Mirror 20: 0xa0a7e53a (stale) CRC-32 checksum value for chunk [0, 0x76f7): Mirror 1: 0xa0a7e53a Mirror 21: 0xa0a7e53a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x76f7: skipped 000799 1790712529.778842 fallocate 0x008c33 thru 0x008dc8 (0x196 bytes) 000800 1790712529.812879 punch from 0x00804a to 0x008dc8 (0xd7f bytes) 000801 1790712529.820056 fallocate 0x002506 thru 0x0036d0 (0x11cb bytes) 000802 1790712529.824638 mapwrite 0x03eef6 thru 0x03ffff (0x110a bytes) 000803 1790712529.864840 mapread 0x03c07d thru 0x03f6b8 (0x363c bytes) 000804 1790712529.873157 mapread 0x03fe6d thru 0x03ffff (0x193 bytes) 000806 1790712529.874497 write 0x036c0e thru 0x03c5ee (0x59e1 bytes) 000807 1790712529.876159 read 0x02983f thru 0x02aea4 (0x1666 bytes) 000808 1790712529.880918 trunc from 0x02d65a to 0x031a97 (0x443e bytes) 000811 1790712529.892199 punch from 0x02ca9f to 0x037a59 (0xafbb bytes) 000812 1790712529.913008 mapread 0x011447 thru 0x01a15f (0x8d19 bytes) 000813 1790712529.929988 write 0x03fcde thru 0x03ffff (0x322 bytes) 000814 1790712529.930989 trunc from 0x012e2e to 0x01f74f (0xc922 bytes) 000815 1790712529.935374 fallocate 0x01ee87 thru 0x02af71 (0xc0eb bytes) 000816 1790712529.940066 trunc from 0x040000 to 0x02b3e5 (0x14c1a bytes) 000817 1790712529.958110 trunc from 0x02b3e6 to 0x0098dd (0x21b08 bytes) 000818 1790712529.980082 fallocate 0x02399d thru 0x0244cf (0xb33 bytes) 000819 1790712529.988579 MIRROR_RESYNC 10 mirrors 000820 1790712531.542234 mapread 0x0066d6 thru 0x0098dd (0x3208 bytes) 000821 1790712531.566183 punch from 0x001abb to 0x0098dd (0x7e23 bytes) 000822 1790712531.611625 write 0x008549 thru 0x0113a4 (0x8e5c bytes) 000823 1790712531.625035 trunc from 0x0113a5 to 0x0021a5 (0xf1ff bytes) 000824 1790712531.669323 mapwrite 0x0081a2 thru 0x013d2e (0xbb8d bytes) 000825 1790712531.806376 mapwrite 0x0223d3 thru 0x029bd5 (0x7803 bytes) 000827 1790712531.889406 trunc from 0x017ed9 to 0x019b30 (0x1c58 bytes) 000828 1790712531.900300 trunc from 0x005cb6 to 0x0067b2 (0xafd bytes) 000829 1790712531.906898 read 0x0046fb thru 0x004ce6 (0x5ec bytes) 000831 1790712531.913345 trunc from 0x01af59 to 0x01f7d4 (0x487c bytes) 000832 1790712531.922993 read 0x001266 thru 0x009449 (0x81e4 bytes) 000833 1790712531.944967 trunc from 0x029bd6 to 0x029284 (0x951 bytes) 000834 1790712531.965520 mapwrite 0x00bf79 thru 0x010086 (0x410e bytes) 000835 1790712532.014940 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x29285): Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29285): Mirror 11: 0xe99d4b35 (stale) Mirror 12: 0xe99d4b35 (stale) Mirror 15: 0xe99d4b35 (stale) Mirror 16: 0xe99d4b35 (stale) Mirror 17: 0xe99d4b35 (stale) Mirror 18: 0xe99d4b35 (stale) Mirror 19: 0xe99d4b35 (stale) Mirror 20: 0xe99d4b35 (stale) Mirror 21: 0xe99d4b35 (stale) CRC-32 checksum value for chunk [0, 0x29285): Mirror 1: 0xaa0b58aa Mirror 22: 0xaa0b58aa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29285: skipped 000836 1790712533.349618 mapwrite 0x01bd70 thru 0x0285b5 (0xc846 bytes) 000837 1790712533.521582 read 0x01e0b4 thru 0x026a62 (0x89af bytes) 000838 1790712533.523360 mapwrite 0x01d336 thru 0x02ac68 (0xd933 bytes) 000839 1790712533.613250 trunc from 0x02ac69 to 0x006474 (0x247f4 bytes) 000840 1790712533.632471 punch from 0x00481c to 0x006474 (0x1c59 bytes) 000841 1790712533.637165 trunc from 0x0000e7 to 0x0047a7 (0x46c1 bytes) 000842 1790712533.645164 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6475): Mirror 11: 0xfd4f02d9 (stale) Mirror 12: 0xfd4f02d9 (stale) Mirror 15: 0xfd4f02d9 (stale) Mirror 16: 0xfd4f02d9 (stale) Mirror 17: 0xfd4f02d9 (stale) Mirror 19: 0xfd4f02d9 (stale) Mirror 20: 0xfd4f02d9 (stale) Mirror 21: 0xfd4f02d9 (stale) Mirror 22: 0xfd4f02d9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6475): Mirror 11: 0xfd4f02d9 (stale) Mirror 12: 0xfd4f02d9 (stale) Mirror 15: 0xfd4f02d9 (stale) Mirror 16: 0xfd4f02d9 (stale) Mirror 17: 0xfd4f02d9 (stale) Mirror 19: 0xfd4f02d9 (stale) Mirror 20: 0xfd4f02d9 (stale) Mirror 21: 0xfd4f02d9 (stale) Mirror 22: 0xfd4f02d9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6475: skipped 000843 1790712534.127049 trunc from 0x0022b7 to 0x006474 (0x41be bytes) 000844 1790712534.136954 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x6475): Mirror 12: 0xfd4f02d9 (stale) Mirror 15: 0xfd4f02d9 (stale) Mirror 16: 0xfd4f02d9 (stale) Mirror 17: 0xfd4f02d9 (stale) Mirror 19: 0xfd4f02d9 (stale) Mirror 20: 0xfd4f02d9 (stale) Mirror 21: 0xfd4f02d9 (stale) Mirror 22: 0xfd4f02d9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x6475): Mirror 12: 0xfd4f02d9 (stale) Mirror 15: 0xfd4f02d9 (stale) Mirror 16: 0xfd4f02d9 (stale) Mirror 17: 0xfd4f02d9 (stale) Mirror 19: 0xfd4f02d9 (stale) Mirror 20: 0xfd4f02d9 (stale) Mirror 21: 0xfd4f02d9 (stale) Mirror 22: 0xfd4f02d9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x6475: skipped 000845 1790712534.663453 mapread 0x003d07 thru 0x0055d7 (0x18d1 bytes) 000846 1790712534.685478 read 0x000a5b thru 0x0063fc (0x59a2 bytes) 000847 1790712534.690202 write 0x039457 thru 0x03f0b4 (0x5c5e bytes) 000848 1790712534.696322 mapread 0x03b2fb thru 0x03d961 (0x2667 bytes) 000849 1790712534.697321 mapread 0x023cf8 thru 0x023e2e (0x137 bytes) 000850 1790712534.705209 punch from 0x02a73f to 0x02b776 (0x1038 bytes) 000851 1790712534.711395 write 0x03816c thru 0x03ffff (0x7e94 bytes) 000852 1790712534.720405 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x2ebf0110 (stale) Mirror 15: 0x2ebf0110 (stale) Mirror 17: 0x2ebf0110 (stale) Mirror 19: 0x2ebf0110 (stale) Mirror 20: 0x2ebf0110 (stale) Mirror 21: 0x2ebf0110 (stale) Mirror 22: 0xa93a2a69 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x2ebf0110 (stale) Mirror 15: 0x2ebf0110 (stale) Mirror 17: 0x2ebf0110 (stale) Mirror 19: 0x2ebf0110 (stale) Mirror 20: 0x2ebf0110 (stale) Mirror 21: 0x2ebf0110 (stale) Mirror 22: 0xa93a2a69 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000853 1790712535.110702 MIRROR_RESYNC 8 mirrors 000854 1790712536.514295 trunc from 0x040000 to 0x0175a1 (0x28a5e bytes) 000855 1790712536.551672 write 0x01f86e thru 0x024a72 (0x5205 bytes) 000856 1790712536.553221 punch from 0x013635 to 0x01dd8b (0xa757 bytes) 000857 1790712536.559326 fallocate 0x01d05e thru 0x02166f (0x4612 bytes) 000858 1790712536.564611 write 0x02c62a thru 0x032b45 (0x651c bytes) 000859 1790712536.565812 trunc from 0x032b46 to 0x006513 (0x2c632 bytes) 000860 1790712536.607091 mapwrite 0x01a14d thru 0x01e8cd (0x4781 bytes) 000861 1790712536.667027 read 0x01ca72 thru 0x01e8cd (0x1e5c bytes) 000862 1790712536.667598 trunc from 0x01e8ce to 0x026f2f (0x8662 bytes) 000863 1790712536.685286 trunc from 0x026f30 to 0x017e3a (0xf0f5 bytes) 000864 1790712536.709857 punch from 0x00ac27 to 0x017e3a (0xd214 bytes) 000865 1790712536.718365 mapwrite 0x0123f4 thru 0x01c42a (0xa037 bytes) 000866 1790712536.769169 punch from 0x00612c to 0x00bc7f (0x5b54 bytes) 000867 1790712536.781438 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x1c42b): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c42b): Mirror 12: 0xf8259750 (stale) Mirror 15: 0xf8259750 (stale) Mirror 17: 0xf8259750 (stale) Mirror 19: 0xf8259750 (stale) Mirror 20: 0xf8259750 (stale) Mirror 21: 0xf8259750 (stale) Mirror 22: 0xf8259750 (stale) CRC-32 checksum value for chunk [0, 0x1c42b): Mirror 1: 0x1e3cfdff Mirror 23: 0x1e3cfdff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c42b: skipped 000868 1790712537.967626 mapread 0x00fa6b thru 0x0185ee (0x8b84 bytes) 000869 1790712538.014799 mapread 0x014ff9 thru 0x017a58 (0x2a60 bytes) 000870 1790712538.017988 fallocate 0x03711d thru 0x03eb19 (0x79fd bytes) 000871 1790712538.068089 write 0x015069 thru 0x01a807 (0x579f bytes) 000872 1790712538.088401 punch from 0x01c1e7 to 0x01c42a (0x244 bytes) 000873 1790712538.096490 punch from 0x00e4f6 to 0x01b092 (0xcb9d bytes) 000874 1790712538.112324 read 0x018680 thru 0x01c42a (0x3dab bytes) 000875 1790712538.125036 MIRROR_RESYNC 9 mirrors 000876 1790712539.666090 trunc from 0x0099c0 to 0x00d4cc (0x3b0d bytes) 000877 1790712539.737057 write 0x01a82f thru 0x01ae6b (0x63d bytes) 000878 1790712539.755216 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1c42b): Mirror 12: 0xf4808fcf (stale) Mirror 15: 0xf4808fcf (stale) Mirror 19: 0xf4808fcf (stale) Mirror 20: 0xf4808fcf (stale) Mirror 21: 0xf4808fcf (stale) Mirror 22: 0xf4808fcf (stale) Mirror 23: 0xf4808fcf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1c42b): Mirror 12: 0xf4808fcf (stale) Mirror 15: 0xf4808fcf (stale) Mirror 19: 0xf4808fcf (stale) Mirror 20: 0xf4808fcf (stale) Mirror 21: 0xf4808fcf (stale) Mirror 22: 0xf4808fcf (stale) Mirror 23: 0xf4808fcf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1c42b: skipped 000879 1790712540.273125 mapread 0x00dc63 thru 0x019646 (0xb9e4 bytes) 000881 1790712540.308935 trunc from 0x001a13 to 0x00fc48 (0xe236 bytes) 000883 1790712540.317277 mapread 0x00845a thru 0x014718 (0xc2bf bytes) 000884 1790712540.328176 trunc from 0x01c42b to 0x00b8ae (0x10b7c bytes) 000885 1790712540.353288 read 0x0042cc thru 0x00a156 (0x5e8b bytes) 000886 1790712540.354077 trunc from 0x00360f to 0x0093dc (0x5dce bytes) 000887 1790712540.368922 read 0x004c48 thru 0x009b2e (0x4ee7 bytes) 000888 1790712540.390261 fallocate 0x004960 thru 0x00c132 (0x77d3 bytes) 000889 1790712540.402437 punch from 0x0010e3 to 0x00b8ae (0xa7cc bytes) 000890 1790712540.407554 read 0x003e7e thru 0x006d39 (0x2ebc bytes) 000891 1790712540.433577 punch from 0x0041e5 to 0x00b8ae (0x76ca bytes) 000892 1790712540.443715 mapwrite 0x034123 thru 0x03a4a9 (0x6387 bytes) 000893 1790712540.503182 fallocate 0x0233e6 thru 0x02cbad (0x97c8 bytes) 000894 1790712540.515413 write 0x034f2a thru 0x03f3b8 (0xa48f bytes) 000895 1790712540.519714 punch from 0x023c01 to 0x0255d5 (0x19d5 bytes) 000896 1790712540.525098 mapwrite 0x023cee thru 0x026625 (0x2938 bytes) 000897 1790712540.572022 write 0x022546 thru 0x02843e (0x5ef9 bytes) 000898 1790712540.573438 mapwrite 0x018c5f thru 0x01e828 (0x5bca bytes) 000899 1790712540.622944 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3f3b9): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f3b9): Mirror 12: 0xbbaa2723 (stale) Mirror 15: 0xbbaa2723 (stale) Mirror 19: 0xbbaa2723 (stale) Mirror 20: 0xbbaa2723 (stale) Mirror 21: 0xbbaa2723 (stale) Mirror 22: 0xbbaa2723 (stale) Mirror 23: 0xbbaa2723 (stale) CRC-32 checksum value for chunk [0, 0x3f3b9): Mirror 1: 0x32fea7e6 Mirror 24: 0x32fea7e6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f3b9: skipped 000900 1790712541.678284 write 0x0341d6 thru 0x039681 (0x54ac bytes) 000901 1790712541.772387 read 0x03a3cd thru 0x03c22c (0x1e60 bytes) 000902 1790712541.783122 mapread 0x008881 thru 0x00e8cb (0x604b bytes) 000903 1790712541.809120 punch from 0x0040c6 to 0x005e71 (0x1dac bytes) 000904 1790712541.819775 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3f3b9): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f3b9): Mirror 12: 0xbbaa2723 (stale) Mirror 15: 0xbbaa2723 (stale) Mirror 19: 0xbbaa2723 (stale) Mirror 20: 0xbbaa2723 (stale) Mirror 21: 0xbbaa2723 (stale) Mirror 22: 0xbbaa2723 (stale) Mirror 23: 0xbbaa2723 (stale) Mirror 24: 0x32fea7e6 (stale) CRC-32 checksum value for chunk [0, 0x3f3b9): Mirror 1: 0x21fa4861 Mirror 25: 0x21fa4861 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f3b9: skipped 000905 1790712543.389180 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3f3b9): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f3b9): Mirror 12: 0xbbaa2723 (stale) Mirror 15: 0xbbaa2723 (stale) Mirror 19: 0xbbaa2723 (stale) Mirror 20: 0xbbaa2723 (stale) Mirror 21: 0xbbaa2723 (stale) Mirror 22: 0xbbaa2723 (stale) Mirror 23: 0xbbaa2723 (stale) Mirror 24: 0x32fea7e6 (stale) CRC-32 checksum value for chunk [0, 0x3f3b9): Mirror 1: 0x21fa4861 Mirror 25: 0x21fa4861 Mirror 26: 0x21fa4861 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f3b9: skipped 000906 1790712544.917045 write 0x023d45 thru 0x033b29 (0xfde5 bytes) 000907 1790712545.034835 mapread 0x023df1 thru 0x032bec (0xedfc bytes) 000908 1790712545.037494 fallocate 0x00c004 thru 0x00ee89 (0x2e86 bytes) 000909 1790712545.043861 trunc from 0x03f3b9 to 0x0133d9 (0x2bfdf bytes) 000910 1790712545.065390 mapread 0x00c0e1 thru 0x0133d9 (0x72f9 bytes) 000911 1790712545.086696 trunc from 0x0133da to 0x013f77 (0xb9e bytes) 000912 1790712545.107187 trunc from 0x013f78 to 0x0108de (0x3699 bytes) 000913 1790712545.125905 write 0x002c6f thru 0x00843d (0x57cf bytes) 000915 1790712545.134255 mapwrite 0x036c1c thru 0x03ffff (0x93e4 bytes) 000916 1790712545.202716 read 0x029e4e thru 0x0327ea (0x899d bytes) 000918 1790712545.217902 read 0x03445f thru 0x03ffff (0xbba1 bytes) 000919 1790712545.219231 mapwrite 0x01aee9 thru 0x0296c1 (0xe7d9 bytes) 000920 1790712545.266965 fallocate 0x00e3bd thru 0x011804 (0x3448 bytes) 000921 1790712545.272787 mapwrite 0x02e59d thru 0x032680 (0x40e4 bytes) 000922 1790712545.298711 write 0x018581 thru 0x019bfc (0x167c bytes) 000923 1790712545.301211 trunc from 0x0187fe to 0x026cbd (0xe4c0 bytes) 000924 1790712545.312011 mapread 0x03b1cc thru 0x03ffff (0x4e34 bytes) 000926 1790712545.313451 fallocate 0x01edce thru 0x020a74 (0x1ca7 bytes) 000927 1790712545.320293 trunc from 0x007912 to 0x0114ef (0x9bde bytes) 000928 1790712545.328257 trunc from 0x00bc1b to 0x0162a2 (0xa688 bytes) 000929 1790712545.333229 mapwrite 0x0154ad thru 0x0166d2 (0x1226 bytes) 000930 1790712545.377999 mapwrite 0x0055f5 thru 0x009ef0 (0x48fc bytes) 000931 1790712545.413721 trunc from 0x040000 to 0x03de36 (0x21c9 bytes) 000932 1790712545.438453 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3de37): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3de37): Mirror 12: 0x15f67b54 (stale) Mirror 15: 0x15f67b54 (stale) Mirror 19: 0x15f67b54 (stale) Mirror 20: 0x15f67b54 (stale) Mirror 21: 0x15f67b54 (stale) Mirror 22: 0x15f67b54 (stale) Mirror 23: 0x15f67b54 (stale) Mirror 24: 0x20453078 (stale) Mirror 25: 0xa302804d (stale) Mirror 26: 0xa302804d (stale) CRC-32 checksum value for chunk [0, 0x3de37): Mirror 1: 0x1371ab1c Mirror 27: 0x1371ab1c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3de37: skipped 000933 1790712546.745722 punch from 0x026b4e to 0x032925 (0xbdd8 bytes) 000935 1790712546.795426 mapread 0x0337e3 thru 0x034d9e (0x15bc bytes) 000936 1790712546.816921 trunc from 0x03de37 to 0x02b02b (0x12e0b bytes) 000937 1790712546.845324 trunc from 0x02839e to 0x02a224 (0x1e87 bytes) 000939 1790712546.861958 fallocate 0x010508 thru 0x01b00e (0xab07 bytes) 000940 1790712546.869983 fallocate 0x00e2ea thru 0x01a543 (0xc25a bytes) 000941 1790712546.883582 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b02c): Mirror 12: 0x491afaf3 (stale) Mirror 15: 0x491afaf3 (stale) Mirror 19: 0x491afaf3 (stale) Mirror 20: 0x491afaf3 (stale) Mirror 21: 0x491afaf3 (stale) Mirror 22: 0x491afaf3 (stale) Mirror 23: 0x491afaf3 (stale) Mirror 24: 0x2c8b819 (stale) Mirror 25: 0x2c8b819 (stale) Mirror 27: 0x42a76b71 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b02c): Mirror 12: 0x491afaf3 (stale) Mirror 15: 0x491afaf3 (stale) Mirror 19: 0x491afaf3 (stale) Mirror 20: 0x491afaf3 (stale) Mirror 21: 0x491afaf3 (stale) Mirror 22: 0x491afaf3 (stale) Mirror 23: 0x491afaf3 (stale) Mirror 24: 0x2c8b819 (stale) Mirror 25: 0x2c8b819 (stale) Mirror 27: 0x42a76b71 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b02c: skipped 000942 1790712547.520326 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2b02c): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b02c): Mirror 12: 0x491afaf3 (stale) Mirror 15: 0x491afaf3 (stale) Mirror 19: 0x491afaf3 (stale) Mirror 20: 0x491afaf3 (stale) Mirror 21: 0x491afaf3 (stale) Mirror 22: 0x491afaf3 (stale) Mirror 23: 0x491afaf3 (stale) Mirror 24: 0x2c8b819 (stale) Mirror 25: 0x2c8b819 (stale) Mirror 27: 0x42a76b71 (stale) CRC-32 checksum value for chunk [0, 0x2b02c): Mirror 1: 0x7df94d2d Mirror 28: 0x7df94d2d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b02c: skipped 000944 1790712548.995814 write 0x0186aa thru 0x025dbd (0xd714 bytes) 000946 1790712549.098271 fallocate 0x0226b6 thru 0x02f844 (0xd18f bytes) 000947 1790712549.112339 mapwrite 0x023f40 thru 0x02b282 (0x7343 bytes) 000948 1790712549.201859 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f845): Mirror 12: 0xaa42dc3 (stale) Mirror 15: 0xaa42dc3 (stale) Mirror 19: 0xaa42dc3 (stale) Mirror 20: 0xaa42dc3 (stale) Mirror 21: 0xaa42dc3 (stale) Mirror 22: 0xaa42dc3 (stale) Mirror 23: 0xaa42dc3 (stale) Mirror 24: 0xad18edf8 (stale) Mirror 25: 0xad18edf8 (stale) Mirror 28: 0xbf548b1a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f845): Mirror 12: 0xaa42dc3 (stale) Mirror 15: 0xaa42dc3 (stale) Mirror 19: 0xaa42dc3 (stale) Mirror 20: 0xaa42dc3 (stale) Mirror 21: 0xaa42dc3 (stale) Mirror 22: 0xaa42dc3 (stale) Mirror 23: 0xaa42dc3 (stale) Mirror 24: 0xad18edf8 (stale) Mirror 25: 0xad18edf8 (stale) Mirror 28: 0xbf548b1a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f845: skipped 000949 1790712549.798133 fallocate 0x03f2d6 thru 0x03ffff (0xd2a bytes) 000950 1790712549.804620 trunc from 0x02f845 to 0x03a1e7 (0xa9a3 bytes) 000951 1790712549.843145 mapread 0x0121ca thru 0x01c809 (0xa640 bytes) 000952 1790712549.874425 punch from 0x033ee3 to 0x03a1e7 (0x6305 bytes) 000953 1790712549.884067 mapread 0x004c62 thru 0x00a3be (0x575d bytes) 000954 1790712549.901942 fallocate 0x0262cf thru 0x032132 (0xbe64 bytes) 000955 1790712549.910662 punch from 0x00702e to 0x012978 (0xb94b bytes) 000958 1790712549.917220 write 0x028805 thru 0x02cba0 (0x439c bytes) 000960 1790712549.927186 write 0x0231fb thru 0x03242f (0xf235 bytes) 000961 1790712549.944841 write 0x032fb6 thru 0x03ffff (0xd04a bytes) 000962 1790712549.947033 mapread 0x030eee thru 0x03d3ad (0xc4c0 bytes) 000963 1790712549.952764 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x3e00f64b (stale) Mirror 15: 0x3e00f64b (stale) Mirror 19: 0x3e00f64b (stale) Mirror 20: 0x3e00f64b (stale) Mirror 21: 0x3e00f64b (stale) Mirror 22: 0x3e00f64b (stale) Mirror 23: 0x3e00f64b (stale) Mirror 24: 0xb0ba3dee (stale) Mirror 25: 0xb7badec0 (stale) Mirror 28: 0xf92478f7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfeb28bed Mirror 29: 0xfeb28bed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000964 1790712551.178388 fallocate 0x03f8c3 thru 0x03ffff (0x73d bytes) 000965 1790712551.226220 mapwrite 0x002d0f thru 0x01132a (0xe61c bytes) 000966 1790712551.290895 mapread 0x022d1a thru 0x032b8a (0xfe71 bytes) 000967 1790712551.313289 write 0x03e8ba thru 0x03ffff (0x1746 bytes) 000968 1790712551.331434 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x3e00f64b (stale) Mirror 19: 0x3e00f64b (stale) Mirror 20: 0x3e00f64b (stale) Mirror 21: 0x3e00f64b (stale) Mirror 22: 0x3e00f64b (stale) Mirror 23: 0x3e00f64b (stale) Mirror 24: 0xb0ba3dee (stale) Mirror 25: 0xb7badec0 (stale) Mirror 28: 0xf92478f7 (stale) Mirror 29: 0xfeb28bed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 12: 0x3e00f64b (stale) Mirror 19: 0x3e00f64b (stale) Mirror 20: 0x3e00f64b (stale) Mirror 21: 0x3e00f64b (stale) Mirror 22: 0x3e00f64b (stale) Mirror 23: 0x3e00f64b (stale) Mirror 24: 0xb0ba3dee (stale) Mirror 25: 0xb7badec0 (stale) Mirror 28: 0xf92478f7 (stale) Mirror 29: 0xfeb28bed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000969 1790712551.971998 trunc from 0x040000 to 0x01565d (0x2a9a2 bytes) 000970 1790712551.998756 mapread 0x00f166 thru 0x01369f (0x453a bytes) 000972 1790712552.029197 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1565e): Mirror 12: 0x639fd56d (stale) Mirror 19: 0x639fd56d (stale) Mirror 20: 0x639fd56d (stale) Mirror 22: 0x639fd56d (stale) Mirror 23: 0x639fd56d (stale) Mirror 24: 0x639fd56d (stale) Mirror 25: 0x639fd56d (stale) Mirror 28: 0xd3db9611 (stale) Mirror 29: 0xfa4dfdcf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1565e): Mirror 12: 0x639fd56d (stale) Mirror 19: 0x639fd56d (stale) Mirror 20: 0x639fd56d (stale) Mirror 22: 0x639fd56d (stale) Mirror 23: 0x639fd56d (stale) Mirror 24: 0x639fd56d (stale) Mirror 25: 0x639fd56d (stale) Mirror 28: 0xd3db9611 (stale) Mirror 29: 0xfa4dfdcf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1565e: skipped 000973 1790712552.544095 write 0x02313f thru 0x029330 (0x61f2 bytes) 000975 1790712552.551250 mapread 0x018337 thru 0x01e417 (0x60e1 bytes) 000976 1790712552.586553 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x29331): Mirror 12: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29331): Mirror 12: 0xb1e9190e (stale) Mirror 19: 0xb1e9190e (stale) Mirror 20: 0xb1e9190e (stale) Mirror 22: 0xb1e9190e (stale) Mirror 23: 0xb1e9190e (stale) Mirror 24: 0xa7cd890a (stale) Mirror 25: 0xa7cd890a (stale) Mirror 28: 0x2ddffde6 (stale) Mirror 29: 0x5e248e4 (stale) CRC-32 checksum value for chunk [0, 0x29331): Mirror 1: 0x9e904fdc Mirror 30: 0x9e904fdc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29331: skipped 000977 1790712553.851386 write 0x01a406 thru 0x01aebe (0xab9 bytes) 000978 1790712553.929801 mapwrite 0x007e94 thru 0x00b844 (0x39b1 bytes) 000979 1790712553.971079 trunc from 0x029331 to 0x020868 (0x8ac8 bytes) 000980 1790712553.998046 mapwrite 0x004749 thru 0x006e4d (0x2705 bytes) 000981 1790712554.024788 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x20869): Mirror 12: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20869): Mirror 12: 0x7ed9595f (stale) Mirror 19: 0x7ed9595f (stale) Mirror 20: 0x7ed9595f (stale) Mirror 22: 0x7ed9595f (stale) Mirror 23: 0x7ed9595f (stale) Mirror 24: 0x2fda798c (stale) Mirror 25: 0x2fda798c (stale) Mirror 28: 0xe8d94669 (stale) Mirror 29: 0xf892fac2 (stale) Mirror 30: 0xb94384f9 (stale) CRC-32 checksum value for chunk [0, 0x20869): Mirror 1: 0x1731436 Mirror 31: 0x1731436 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20869: skipped 000982 1790712555.294061 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x18d:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20869): Mirror 12: 0x7ed9595f (stale) Mirror 19: 0x7ed9595f (stale) Mirror 20: 0x7ed9595f (stale) Mirror 22: 0x7ed9595f (stale) Mirror 23: 0x7ed9595f (stale) Mirror 24: 0x2fda798c (stale) Mirror 25: 0x2fda798c (stale) Mirror 28: 0xe8d94669 (stale) Mirror 29: 0xf892fac2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20869): Mirror 12: 0x7ed9595f (stale) Mirror 19: 0x7ed9595f (stale) Mirror 20: 0x7ed9595f (stale) Mirror 22: 0x7ed9595f (stale) Mirror 23: 0x7ed9595f (stale) Mirror 24: 0x2fda798c (stale) Mirror 25: 0x2fda798c (stale) Mirror 28: 0xe8d94669 (stale) Mirror 29: 0xf892fac2 (stale) CRC-32 checksum value for chunk [0, 0x20869): Mirror 1: 0x1731436 Mirror 31: 0x1731436 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20869: skipped 000983 1790712556.167447 mapwrite 0x0323c1 thru 0x034a7f (0x26bf bytes) 000984 1790712556.257630 MIRROR_RESYNC 11 mirrors 000985 1790712559.762193 fallocate 0x01c8c9 thru 0x01fc66 (0x339e bytes) 000986 1790712559.812255 punch from 0x009158 to 0x009ac8 (0x971 bytes) 000987 1790712559.824213 mapread 0x007b9e thru 0x017599 (0xf9fc bytes) 000988 1790712559.864589 trunc from 0x032b20 to 0x034a7f (0x1f60 bytes) 000989 1790712559.885459 trunc from 0x034a80 to 0x03d64e (0x8bcf bytes) 000990 1790712559.948343 trunc from 0x02ada9 to 0x036e86 (0xc0de bytes) 000991 1790712559.956077 read 0x00fcc5 thru 0x012542 (0x287e bytes) 000992 1790712559.965674 mapwrite 0x036839 thru 0x03a69f (0x3e67 bytes) 000993 1790712560.008162 trunc from 0x03d64f to 0x03f065 (0x1a17 bytes) 000994 1790712560.036981 write 0x0192e1 thru 0x019aa0 (0x7c0 bytes) 000995 1790712560.037913 fallocate 0x0252ab thru 0x034876 (0xf5cc bytes) 000996 1790712560.055013 trunc from 0x03f066 to 0x0394be (0x5ba7 bytes) 000997 1790712560.129331 trunc from 0x0394bf to 0x0294db (0xffe3 bytes) 000998 1790712560.155503 read 0x0204d7 thru 0x0258fb (0x5425 bytes) 000999 1790712560.178589 punch from 0x016393 to 0x022ade (0xc74c bytes) 001000 1790712560.200697 trunc from 0x0294dc to 0x016ccd (0x1280e bytes) All operations completed A-OK! PASS 70a (121s) == sanity-flr test 71: check mirror extend parent fid ==== 16:09:25 (1790712565) striped dir -i1 -c2 -H crush /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000405:0x1da:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (12s) == sanity-flr test 72: check append on FLR file ========== 16:09:37 (1790712577) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.220901 s, 9.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25781 s, 4.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (12s) == sanity-flr test 200a: stress test ===================== 16:09:50 (1790712590) Starting client: oleg333-client.virtnet: -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre2 Starting client: oleg333-client.virtnet: -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4084096 .. Extending file size to 7032160 .. done Extending file size to 6270880 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 7072448 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8834944 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8734592 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9142592 .. Extending file size to 8932160 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 62119 62120 62123 62124 62125 192.168.203.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg333-client.virtnet /mnt/lustre2 (opts:) 192.168.203.133@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg333-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:11:15 (1790712675) Starting client: oleg333-client.virtnet: -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre2 Starting client: oleg333-client.virtnet: -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3591712 .. Extending file size to 3623168 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 6314400 .. Extending file size to 5766880 .. Extending file size to 6622560 .. Extending file size to 5862432 .. Extending file size to 7499200 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7298208 .. Extending file size to 8511040 .. Extending file size to 9046336 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 63618 63619 63621 63622 63625 done 192.168.203.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg333-client.virtnet /mnt/lustre2 (opts:) 192.168.203.133@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg333-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (83s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:12:38 (1790712758) Starting client: oleg333-client.virtnet: -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2180 1269124 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2244 1269060 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18784 3588236 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24484 3582536 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43268 7170772 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2180 1269124 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2244 1269060 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18784 3588236 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24484 3582536 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43268 7170772 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.133603 s, 15.7 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.223862 s, 9.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x62d19a97 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.203.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg333-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:12:58 (1790712778) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.133516 s, 15.7 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 16:13:05 (1790712785) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xb2:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:13:13 (1790712793) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.162459 s, 12.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x240:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23e:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x23f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:13:21 (1790712801) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.133667 s, 15.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:13:28 (1790712808) striped dir -i0 -c2 -H crush /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.107128 s, 19.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:14:02 (1790712842) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg333-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715383 s, 14.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x241:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x241:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x240:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x241:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x241:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x240:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x242:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:14:56 (1790712896) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg333-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x242:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x243:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0877462 s, 23.9 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x242:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x243:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x243:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x242:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x243:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x243:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (51s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:15:47 (1790712947) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg333-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg333-server: Pool lustre.test_204f created Adding targets to pool oleg333-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg333-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.124139 s, 16.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x151:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x132:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg333-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg333-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg333-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:17:01 (1790713021) PASS 205a (7s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:17:08 (1790713028) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:17:15 (1790713035) oleg333-server: Pool lustre.test_206 created Adding targets to pool oleg333-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg333-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg333-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg333-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg333-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:17:41 (1790713061) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743816 s, 14.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362218 s, 28.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xeff56a95 Mirror 3: 0xeff56a95 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:17:51 (1790713071) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:17:55 (1790713075) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:17:58 (1790713078) Starting client: oleg333-client.virtnet: -o user_xattr,flock 192.168.203.133@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0580338 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1530000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1030000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1030000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1030000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xe80000 Read this (4 bytes): BBBB 192.168.203.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg333-client.virtnet /mnt/lustre2 (opts:) PASS 209a (39s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:18:37 (1790713117) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.015605 s, 525 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:18:45 (1790713125) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417918 s, 25.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd1 lmm_fid: [0x200000405:0xd1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd1 lmm_fid: [0x200000405:0xd1:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611345 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-29 16:18:46.000000000 -0400 Modify: 2026-09-29 16:18:46.000000000 -0400 Change: 2026-09-29 16:18:48.000000000 -0400 Birth: 2026-09-29 16:18:47.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:18:53 (1790713133) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240854 s, 43.5 MB/s oleg333-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg333-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg333-server: O/280000401/d28/15c: File not found by ext2_lookup PASS 210b (18s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:19:12 (1790713152) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.204768 s, 200 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0155595 s, 263 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 29 16:19 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xd6:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 29 16:19 /mnt/lustre/f211.sanity-flr PASS 211 (7s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:19:19 (1790713159) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:19:27 (1790713167) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1933 sec =========== 16:19:36 (1790713176) === sanity-flr: start cleanup 16:19:38 (1790713178) === === sanity-flr: finish cleanup 16:19:45 (1790713185) ===