-----============= acceptance-small: sanity-flr ============----- Fri Oct 2 16:06:05 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITY_FLR_ONLY='' SANITY_FLR_EXCEPT='' oleg142-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 16:06:27 (1790971587) === oleg142-client.virtnet: executing check_config_client /mnt/lustre oleg142-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg142-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93e40809e000.idle_timeout=debug osc.lustre-OST0001-osc-ffff93e40809e000.idle_timeout=debug disable quota as required oleg142-server: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 16:07:12 (1790971632) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8232] == sanity-flr test 0a: lfs mirror create with -N option == 16:07:15 (1790971635) PASS 0a (17s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 16:07:32 (1790971652) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 16:07:36 (1790971656) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 16:07:40 (1790971660) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 16:08:07 (1790971687) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 16:08:11 (1790971691) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 16:08:14 (1790971694) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff93e40809e000.stats=clear osc.lustre-OST0001-osc-ffff93e40809e000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 16:08:23 (1790971703) striped dir -i0 -c2 -H crush /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--ec-map] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 16:08:34 (1790971714) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 16:08:44 (1790971724) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 16:08:48 (1790971728) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (10s) == sanity-flr test 1: create components with setstripe options ========================================================== 16:08:58 (1790971738) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 16:09:10 (1790971750) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 16:09:19 (1790971759) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 16:09:28 (1790971768) striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (9s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 16:09:37 (1790971777) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 16:09:48 (1790971788) PASS 7 (6s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 16:09:54 (1790971794) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.765959 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742014 s, 14.1 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 16:10:03 (1790971803) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.856094 s, 20.8 MB/s PASS 22 (12s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 16:10:15 (1790971815) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 16:10:25 (1790971825) Stopping /mnt/lustre-ost1 (opts:) on oleg142-server oleg142-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93e40809e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93e40809e000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg142-server: oleg142-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg142-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg142-server oleg142-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93e40809e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93e40809e000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg142-server: oleg142-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg142-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (81s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 16:11:46 (1790971906) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.452634 s, 16.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg142-server oleg142-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93e40809e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93e40809e000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 7340032 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg142-server: oleg142-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg142-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (47s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 16:12:34 (1790971954) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg142-server oleg142-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93e40809e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93e40809e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg142-server: oleg142-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg142-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg142-server oleg142-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93e40809e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93e40809e000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg142-server: oleg142-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg142-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (84s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 16:13:58 (1790972038) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.53159 s, 20.7 MB/s -rw-r--r-- 1 root root 52428800 Oct 2 16:14 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.159s user 0m0.000s sys 0m0.145s Stopping /mnt/lustre-ost1 (opts:) on oleg142-server oleg142-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff93e40809e000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93e40809e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.084s user 0m0.000s sys 0m0.189s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg142-server: oleg142-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg142-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg142-server oleg142-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff93e40809e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93e40809e000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.164s user 0m0.002s sys 0m0.144s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg142-server: oleg142-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg142-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg142-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff93e40809e000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff93e40809e000.ost_server_uuid in FULL state after 0 sec PASS 33b (94s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 16:15:32 (1790972132) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 16:15:36 (1790972136) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 16:15:39 (1790972139) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 16:15:43 (1790972143) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 16:15:51 (1790972151) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.617068 s, 20.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 16:16:04 (1790972164) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565973 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372475 s, 28.2 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 16:16:11 (1790972171) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 16:16:19 (1790972179) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000755078 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00220477 s, 464 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000730959 s, 1.4 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00120619 s, 849 kB/s flv=10 olv=10 PASS 36d (9s) == sanity-flr test 37: mirror I/O API verification ======= 16:16:28 (1790972188) create test files with size 15749819 .. e9ef4506776f1195aa6387ecd3744372 - e4be82841ef35f451ed0ed8ea37027cf - 07f0eb40682b904f651b76660b607df8 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg142-client.virtnet /mnt/lustre (opts:) Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 16:16:51 (1790972211) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 16:17:03 (1790972223) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 16:17:10 (1790972230) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698821 s, 15.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xca:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268654 s, 39.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaa:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 16:17:18 (1790972238) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0810774 s, 25.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.203844 s, 20.6 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150986 s, 13.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (13s) == sanity-flr test 42: lfs mirror verify ================= 16:17:31 (1790972251) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 16:17:34 (1790972254) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 16:17:38 (1790972258) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040234 s, 26.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 16:17:45 (1790972265) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.21902 s, 14.4 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 16:17:54 (1790972274) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd7:0x0] } split mirror_id 2 id 131073 error lfs split: file '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' has 1 component, cannot split make sure there's no stale comp in the file PASS 44b (7s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 16:18:03 (1790972283) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.357199 s, 29.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg142-client.virtnet /mnt/lustre (opts:) Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (10s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 16:18:13 (1790972293) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff93e40552a800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.333627 s, 31.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg142-client.virtnet /mnt/lustre (opts:) Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff93e40885c800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 16:18:30 (1790972310) error lfs split: create victim file failed: File exists PASS 44f (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 16:18:43 (1790972323) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (39s) == sanity-flr test 46: Verify setstripe --copy option ==== 16:19:22 (1790972362) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 16:19:55 (1790972395) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 16:19:58 (1790972398) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.265442 s, 11.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.160897 s, 19.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xf3df731d (nosync,stale) Mirror 2: 0xf3df731d (nosync,stale) ** verify mirror 2 doesn't change original checksum: a415c95fd3510361440db80f88fe219b - mirror 1 checksum: e77382f8bc8b53b69f21c1c6dff0710f - mirror 2 checksum: a415c95fd3510361440db80f88fe219b - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-10-02T16:20:00-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5d lmm_fid: [0x200000405:0x5d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-10-02T16:20:00-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5d lmm_fid: [0x200000405:0x5d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: a415c95fd3510361440db80f88fe219b - mirror 1 checksum: e77382f8bc8b53b69f21c1c6dff0710f - mirror 2 checksum: e77382f8bc8b53b69f21c1c6dff0710f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-10-02T16:20:00-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5d lmm_fid: [0x200000405:0x5d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-10-02T16:20:00-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5d lmm_fid: [0x200000405:0x5d:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xe9:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 16:20:07 (1790972407) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379605 s, 27.6 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 16:20:14 (1790972414) Failing mds1 on oleg142-server Stopping /mnt/lustre-mds1 (opts:) on oleg142-server 16:20:19 (1790972419) shut down facet: mds1 facet_host: oleg142-server facet_failover_host: oleg142-server Failover mds1 to oleg142-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg142-server: oleg142-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-MDT0000 16:20:53 (1790972453) targets are mounted 16:20:53 (1790972453) facet_failover done oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x9:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268294 s, 39.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x240000405 lmm_object_id: 0x9 lmm_fid: [0x240000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1e:0x0] } PASS 50A (55s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 16:21:09 (1790972469) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0367809 s, 557 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0176097 s, 1.2 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 16:21:18 (1790972478) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0662673 s, 309 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00740883 s, 2.8 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (9s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 16:21:27 (1790972487) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0119086 s, 1.4 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 16:21:34 (1790972494) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0363318 s, 564 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (8s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 16:21:42 (1790972502) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00520878 s, 786 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0287817 s, 142 kB/s PASS 60a (6s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 16:21:49 (1790972509) striped dir -i0 -c2 -H all_char /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x15:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 16:22:09 (1790972529) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 16:22:38 (1790972558) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (24s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 16:23:02 (1790972582) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 16:23:27 (1790972607) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (7s) == sanity-flr test 70: mirror create and split race ====== 16:23:34 (1790972614) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 58267 mirror split pid 58268 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 58267 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 58268 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 16:24:42 (1790972682) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 2684 skipping zero length zero range 000002 1790972685.303884 MIRROR_EXTEND to 2 mirrors skipping zero size read skipping zero size read 000005 1790972685.987325 mapwrite 0x038208 thru 0x03b5f7 (0x33f0 bytes) 000006 1790972686.066739 mapwrite 0x03f582 thru 0x03ffff (0xa7e bytes) truncating to largest ever: 0x355dc 000007 1790972686.102722 trunc from 0x040000 to 0x0355db (0xaa24 bytes) 000008 1790972686.115160 write 0x03c945 thru 0x03ffff (0x36bb bytes) 000009 1790972686.116129 trunc from 0x040000 to 0x02ee6a (0x11195 bytes) 000010 1790972686.131380 trunc from 0x02ee6b to 0x010175 (0x1ecf5 bytes) 000011 1790972686.153658 write 0x035a2b thru 0x03ffff (0xa5d5 bytes) 000012 1790972686.155652 trunc from 0x02db99 to 0x035283 (0x76eb bytes) 000014 1790972686.205372 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb00656d9 Mirror 3: 0xb00656d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000015 1790972687.449082 trunc from 0x03d429 to 0x03ffff (0x2bd7 bytes) 000016 1790972687.531272 mapwrite 0x029c83 thru 0x02e497 (0x4815 bytes) 000017 1790972687.593393 mapread 0x002b1a thru 0x00b865 (0x8d4c bytes) 000018 1790972687.613670 mapread 0x031c1b thru 0x03a1e8 (0x85ce bytes) 000019 1790972687.626461 trunc from 0x040000 to 0x024b79 (0x1b486 bytes) 000020 1790972687.656651 read 0x0032f3 thru 0x006865 (0x3573 bytes) 000021 1790972687.657261 mapread 0x001686 thru 0x004121 (0x2a9c bytes) 000022 1790972687.658115 mapread 0x01124f thru 0x014edd (0x3c8f bytes) 000023 1790972687.671009 mapread 0x013fd8 thru 0x01c729 (0x8752 bytes) 000024 1790972687.686127 MIRROR_RESYNC 3 mirrors 000025 1790972688.227150 mapread 0x01ee82 thru 0x0246b8 (0x5837 bytes) 000026 1790972688.253031 read 0x00b5fe thru 0x00f623 (0x4026 bytes) 000027 1790972688.264135 mapwrite 0x01f951 thru 0x0205e5 (0xc95 bytes) 000028 1790972688.365017 write 0x00bbb6 thru 0x0154b7 (0x9902 bytes) 000029 1790972688.384725 fallocate 0x02a34b thru 0x033898 (0x954e bytes) 000030 1790972688.394255 mapwrite 0x015af4 thru 0x01a7a1 (0x4cae bytes) 000031 1790972688.445536 punch from 0x02dbb7 to 0x033898 (0x5ce2 bytes) 000032 1790972688.465343 mapread 0x0279f0 thru 0x028c78 (0x1289 bytes) 000033 1790972688.491641 trunc from 0x0226b9 to 0x02b847 (0x918f bytes) 000034 1790972688.503472 trunc from 0x03139d to 0x033898 (0x24fc bytes) 000035 1790972688.517665 mapwrite 0x03719f thru 0x03ffff (0x8e61 bytes) 000036 1790972688.621683 write 0x00fdb3 thru 0x01df2b (0xe179 bytes) 000037 1790972688.628180 trunc from 0x040000 to 0x01e0ad (0x21f52 bytes) 000038 1790972688.657446 write 0x02199a thru 0x021bee (0x255 bytes) 000039 1790972688.670886 trunc from 0x006067 to 0x01508b (0xf025 bytes) 000040 1790972688.714100 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x21bef): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21bef): Mirror 2: 0x60ac2154 (stale) Mirror 3: 0x60ac2154 (stale) CRC-32 checksum value for chunk [0, 0x21bef): Mirror 1: 0xe096fe72 Mirror 4: 0xe096fe72 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21bef: skipped 000041 1790972689.561740 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x21bef): Mirror 2: 0x60ac2154 (stale) Mirror 3: 0x60ac2154 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21bef): Mirror 2: 0x60ac2154 (stale) Mirror 3: 0x60ac2154 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21bef: skipped 000042 1790972689.851642 read 0x00a744 thru 0x00e606 (0x3ec3 bytes) fallocating to largest ever: 0x35c1f 000043 1790972689.859477 fallocate 0x034921 thru 0x035c1e (0x12fe bytes) 000044 1790972689.877453 write 0x0328d9 thru 0x035539 (0x2c61 bytes) 000046 1790972689.896805 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x35c1f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35c1f): Mirror 2: 0x5ddd96cd (stale) Mirror 3: 0x5ddd96cd (stale) CRC-32 checksum value for chunk [0, 0x35c1f): Mirror 1: 0xa528ef47 Mirror 4: 0xa528ef47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35c1f: skipped 000047 1790972690.708472 mapread 0x0227df thru 0x02f25c (0xca7e bytes) 000048 1790972690.755070 trunc from 0x00e44e to 0x017707 (0x92ba bytes) 000049 1790972690.797116 write 0x022fba thru 0x025eaa (0x2ef1 bytes) 000052 1790972690.809886 mapwrite 0x00502d thru 0x00d4a2 (0x8476 bytes) 000053 1790972690.834779 mapwrite 0x01e202 thru 0x02e01e (0xfe1d bytes) 000054 1790972690.875651 punch from 0x01af9c to 0x01b09f (0x104 bytes) 000055 1790972690.881194 fallocate 0x000d8f thru 0x00fa14 (0xec86 bytes) 000056 1790972690.887708 mapread 0x02306e thru 0x03270c (0xf69f bytes) 000057 1790972690.902707 punch from 0x0180b6 to 0x019056 (0xfa1 bytes) 000058 1790972690.909151 mapwrite 0x00349c thru 0x00b80c (0x8371 bytes) 000059 1790972690.928396 fallocate 0x02d331 thru 0x030ad7 (0x37a7 bytes) 000060 1790972690.934700 mapread 0x022c94 thru 0x02c2d8 (0x9645 bytes) fallocating to largest ever: 0x3a540 000061 1790972690.936886 fallocate 0x0384ec thru 0x03a53f (0x2054 bytes) 000062 1790972690.943102 trunc from 0x004052 to 0x01287e (0xe82d bytes) 000063 1790972690.948828 punch from 0x00799c to 0x017919 (0xff7e bytes) 000064 1790972690.954530 mapwrite 0x03a0d8 thru 0x03f344 (0x526d bytes) 000065 1790972691.004363 trunc from 0x03f345 to 0x037085 (0x82bf bytes) 000066 1790972691.024453 write 0x02d5a2 thru 0x038d5e (0xb7bd bytes) 000067 1790972691.027215 trunc from 0x038d5f to 0x010592 (0x287cc bytes) 000068 1790972691.049528 punch from 0x00e9b4 to 0x00fc39 (0x1286 bytes) 000070 1790972691.055168 mapread 0x003679 thru 0x00cdf1 (0x9779 bytes) 000072 1790972691.069505 trunc from 0x00b028 to 0x010592 (0x556b bytes) 000073 1790972691.074598 read 0x006627 thru 0x0083dd (0x1db7 bytes) 000074 1790972691.076038 mapread 0x00a03a thru 0x010592 (0x6559 bytes) 000075 1790972691.085131 write 0x000c72 thru 0x00e626 (0xd9b5 bytes) 000076 1790972691.091710 trunc from 0x010593 to 0x006110 (0xa482 bytes) 000077 1790972691.104766 trunc from 0x006111 to 0x03951c (0x3340c bytes) 000078 1790972691.129330 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3951d): Mirror 2: 0xc63a85cd (stale) Mirror 3: 0xc63a85cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3951d): Mirror 2: 0xc63a85cd (stale) Mirror 3: 0xc63a85cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3951d: skipped 000079 1790972691.579270 mapwrite 0x007c3d thru 0x0133f5 (0xb7b9 bytes) 000080 1790972691.631406 fallocate 0x02bfea thru 0x034797 (0x87ae bytes) 000081 1790972691.636676 mapwrite 0x01bb05 thru 0x02383c (0x7d38 bytes) 000082 1790972691.666262 punch from 0x001675 to 0x00ac0e (0x959a bytes) 000083 1790972691.671645 punch from 0x00e553 to 0x01974e (0xb1fc bytes) fallocating to largest ever: 0x3fdc2 000084 1790972691.677517 fallocate 0x035a64 thru 0x03fdc1 (0xa35e bytes) 000085 1790972691.683336 punch from 0x0123dc to 0x02115d (0xed82 bytes) 000086 1790972691.694265 MIRROR_RESYNC 3 mirrors 000087 1790972692.387791 trunc from 0x03951d to 0x0177ba (0x21d62 bytes) 000088 1790972692.427586 MIRROR_RESYNC 3 mirrors 000089 1790972693.058639 trunc from 0x014799 to 0x0177ba (0x3022 bytes) 000090 1790972693.087247 write 0x013f3a thru 0x0217e2 (0xd8a9 bytes) 000091 1790972693.098323 read 0x01f855 thru 0x0217e2 (0x1f8e bytes) 000092 1790972693.099304 punch from 0x01dae5 to 0x0217e2 (0x3cfe bytes) 000093 1790972693.113482 mapread 0x01ccad thru 0x0217e2 (0x4b36 bytes) 000095 1790972693.127523 write 0x015ebd thru 0x025c84 (0xfdc8 bytes) 000096 1790972693.129178 write 0x03d2a6 thru 0x03ffff (0x2d5a bytes) 000097 1790972693.130930 read 0x01a507 thru 0x021424 (0x6f1e bytes) 000098 1790972693.132192 mapwrite 0x01f2f8 thru 0x02cbdb (0xd8e4 bytes) 000099 1790972693.242063 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1c3d4a45 (stale) Mirror 3: 0x1c3d4a45 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xfabaf100 Mirror 4: 0xfabaf100 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000100 1790972694.256502 trunc from 0x0328e6 to 0x03ad2d (0x8448 bytes) 000102 1790972694.301882 write 0x00beab thru 0x010ce3 (0x4e39 bytes) 000103 1790972694.317534 write 0x021b34 thru 0x02a7a2 (0x8c6f bytes) 000104 1790972694.329261 fallocate 0x02f4fc thru 0x03ef2a (0xfa2f bytes) 000105 1790972694.333723 mapread 0x01501b thru 0x017121 (0x2107 bytes) 000106 1790972694.353973 punch from 0x0096a2 to 0x00e785 (0x50e4 bytes) 000107 1790972694.367868 mapwrite 0x01af15 thru 0x01f597 (0x4683 bytes) 000108 1790972694.384298 read 0x027a70 thru 0x0323d4 (0xa965 bytes) 000109 1790972694.400126 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1c3d4a45 (stale) Mirror 3: 0x1c3d4a45 (stale) Mirror 4: 0xfabaf100 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf1e690d Mirror 5: 0xf1e690d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000110 1790972695.654000 write 0x0102fb thru 0x01dd58 (0xda5e bytes) 000111 1790972695.717361 read 0x0117d3 thru 0x01c063 (0xa891 bytes) 000112 1790972695.726633 MIRROR_RESYNC 5 mirrors 000114 1790972696.958836 trunc from 0x01a261 to 0x028e10 (0xebb0 bytes) 000115 1790972696.999262 mapread 0x02aa34 thru 0x02cbcd (0x219a bytes) 000116 1790972697.020152 read 0x00bb59 thru 0x0190b0 (0xd558 bytes) 000117 1790972697.046533 MIRROR_RESYNC 5 mirrors 000118 1790972698.443722 mapwrite 0x02c12e thru 0x02efd6 (0x2ea9 bytes) 000119 1790972698.589748 write 0x03d2d3 thru 0x03ffff (0x2d2d bytes) 000121 1790972698.597294 write 0x0344aa thru 0x03df70 (0x9ac7 bytes) 000122 1790972698.603154 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x3338d051 (stale) Mirror 3: 0x3338d051 (stale) Mirror 4: 0x3338d051 (stale) Mirror 5: 0x3338d051 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9c047f18 Mirror 6: 0x9c047f18 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000123 1790972699.535474 trunc from 0x040000 to 0x02649a (0x19b65 bytes) 000124 1790972699.575696 mapwrite 0x014ed3 thru 0x017c33 (0x2d61 bytes) 000125 1790972699.637578 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2649b): Mirror 2: 0xe7c8798b (stale) Mirror 3: 0xe7c8798b (stale) Mirror 5: 0xe7c8798b (stale) Mirror 6: 0xe7c8798b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2649b): Mirror 2: 0xe7c8798b (stale) Mirror 3: 0xe7c8798b (stale) Mirror 5: 0xe7c8798b (stale) Mirror 6: 0xe7c8798b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2649b: skipped 000126 1790972700.170036 punch from 0x025e70 to 0x02649a (0x62b bytes) 000127 1790972700.181709 MIRROR_RESYNC 5 mirrors 000128 1790972701.512994 mapwrite 0x02b512 thru 0x032ad3 (0x75c2 bytes) 000129 1790972701.655551 mapread 0x01d54e thru 0x025de8 (0x889b bytes) 000130 1790972701.691580 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x32ad4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32ad4): Mirror 2: 0xfa3287a9 (stale) Mirror 3: 0xfa3287a9 (stale) Mirror 5: 0xfa3287a9 (stale) Mirror 6: 0xfa3287a9 (stale) CRC-32 checksum value for chunk [0, 0x32ad4): Mirror 1: 0xf9ebc624 Mirror 7: 0xf9ebc624 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32ad4: skipped 000131 1790972703.055747 write 0x033def thru 0x03ffff (0xc211 bytes) 000132 1790972703.087095 trunc from 0x01bc11 to 0x02a7dc (0xebcc bytes) 000133 1790972703.092166 mapwrite 0x0258e6 thru 0x034f87 (0xf6a2 bytes) 000134 1790972703.145331 mapwrite 0x026441 thru 0x031471 (0xb031 bytes) 000135 1790972703.184784 fallocate 0x030bde thru 0x0370c8 (0x64eb bytes) 000136 1790972703.189368 punch from 0x001b89 to 0x00a279 (0x86f1 bytes) 000137 1790972703.195293 fallocate 0x02c2b0 thru 0x031691 (0x53e2 bytes) 000138 1790972703.203812 MIRROR_RESYNC 6 mirrors 000139 1790972705.083044 trunc from 0x040000 to 0x03554c (0xaab3 bytes) 000140 1790972705.165714 MIRROR_RESYNC 6 mirrors 000141 1790972706.678216 punch from 0x027171 to 0x034c07 (0xda97 bytes) 000142 1790972706.768258 trunc from 0x03554d to 0x03450c (0x1040 bytes) 000143 1790972706.830081 read 0x02df44 thru 0x03450c (0x65c9 bytes) 000145 1790972706.897954 MIRROR_RESYNC 6 mirrors 000146 1790972708.207641 read 0x005f1a thru 0x014abb (0xeba2 bytes) 000147 1790972708.216933 trunc from 0x03450d to 0x00877a (0x2bd92 bytes) 000148 1790972708.257535 fallocate 0x019075 thru 0x0237ef (0xa77b bytes) 000149 1790972708.262837 trunc from 0x00df2f to 0x01c7ae (0xe880 bytes) 000150 1790972708.267777 mapwrite 0x0281b1 thru 0x036524 (0xe374 bytes) 000151 1790972708.331064 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x36525): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36525): Mirror 2: 0x7c30431c (stale) Mirror 3: 0x7c30431c (stale) Mirror 5: 0x7c30431c (stale) Mirror 6: 0x7c30431c (stale) Mirror 7: 0x7c30431c (stale) CRC-32 checksum value for chunk [0, 0x36525): Mirror 1: 0x5e455380 Mirror 8: 0x5e455380 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36525: skipped 000152 1790972709.110227 mapwrite 0x028b34 thru 0x02f674 (0x6b41 bytes) 000153 1790972709.197272 write 0x0093b5 thru 0x00a8e7 (0x1533 bytes) 000154 1790972709.215666 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36525): Mirror 2: 0x7c30431c (stale) Mirror 3: 0x7c30431c (stale) Mirror 5: 0x7c30431c (stale) Mirror 6: 0x7c30431c (stale) Mirror 7: 0x7c30431c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36525): Mirror 2: 0x7c30431c (stale) Mirror 3: 0x7c30431c (stale) Mirror 5: 0x7c30431c (stale) Mirror 6: 0x7c30431c (stale) Mirror 7: 0x7c30431c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36525: skipped 000155 1790972709.637702 punch from 0x034a59 to 0x036524 (0x1acc bytes) 000156 1790972709.644882 punch from 0x017912 to 0x0275b6 (0xfca5 bytes) 000157 1790972709.650161 mapwrite 0x02bdfa thru 0x02cc07 (0xe0e bytes) 000158 1790972709.682532 mapwrite 0x002500 thru 0x009a0c (0x750d bytes) 000159 1790972709.734158 mapwrite 0x030609 thru 0x038a36 (0x842e bytes) 000160 1790972709.821216 write 0x00f785 thru 0x01518a (0x5a06 bytes) 000161 1790972709.827558 trunc from 0x02038d to 0x0284f4 (0x8168 bytes) 000162 1790972709.833096 trunc from 0x01521f to 0x0222ec (0xd0ce bytes) 000163 1790972709.846637 fallocate 0x03ce7b thru 0x03de5b (0xfe1 bytes) 000164 1790972709.851858 trunc from 0x004072 to 0x00fd8c (0xbd1b bytes) 000165 1790972709.857354 write 0x00204a thru 0x00a302 (0x82b9 bytes) 000166 1790972709.867819 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38a37): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a37): Mirror 2: 0xcde759f4 (stale) Mirror 3: 0xcde759f4 (stale) Mirror 5: 0xcde759f4 (stale) Mirror 6: 0xcde759f4 (stale) Mirror 7: 0xcde759f4 (stale) CRC-32 checksum value for chunk [0, 0x38a37): Mirror 1: 0xe45b843a Mirror 8: 0xe45b843a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a37: skipped 000167 1790972710.856270 punch from 0x0318fb to 0x038a36 (0x713c bytes) 000168 1790972710.959083 mapwrite 0x019f3c thru 0x01a091 (0x156 bytes) 000169 1790972710.990583 write 0x027fe6 thru 0x034b75 (0xcb90 bytes) 000170 1790972711.007368 mapread 0x01dbdd thru 0x02805a (0xa47e bytes) 000171 1790972711.031329 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x38a37): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a37): Mirror 2: 0xcde759f4 (stale) Mirror 3: 0xcde759f4 (stale) Mirror 5: 0xcde759f4 (stale) Mirror 6: 0xcde759f4 (stale) Mirror 7: 0xcde759f4 (stale) Mirror 8: 0xe45b843a (stale) CRC-32 checksum value for chunk [0, 0x38a37): Mirror 1: 0x7d86553f Mirror 9: 0x7d86553f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a37: skipped 000172 1790972712.140121 fallocate 0x00c3c4 thru 0x00e318 (0x1f55 bytes) 000173 1790972712.193689 mapwrite 0x00c3b6 thru 0x01483d (0x8488 bytes) 000174 1790972712.235112 mapwrite 0x002b2a thru 0x005396 (0x286d bytes) 000176 1790972712.263218 read 0x00cdae thru 0x016e3d (0xa090 bytes) 000177 1790972712.264244 punch from 0x02383d to 0x026318 (0x2adc bytes) 000178 1790972712.270728 write 0x033243 thru 0x03f9db (0xc799 bytes) 000179 1790972712.287051 write 0x028ef8 thru 0x02ea6e (0x5b77 bytes) 000180 1790972712.311613 write 0x0046f0 thru 0x009135 (0x4a46 bytes) 000181 1790972712.314740 read 0x01a10b thru 0x0299e8 (0xf8de bytes) 000182 1790972712.345449 read 0x029290 thru 0x02f1dc (0x5f4d bytes) 000183 1790972712.351723 punch from 0x019c95 to 0x027c5f (0xdfcb bytes) 000184 1790972712.358784 trunc from 0x009497 to 0x017bce (0xe738 bytes) 000185 1790972712.393017 trunc from 0x03d48e to 0x03f9db (0x254e bytes) 000186 1790972712.400980 read 0x027122 thru 0x02ff60 (0x8e3f bytes) 000187 1790972712.422703 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3f9dc): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9dc): Mirror 2: 0xa7fa2992 (stale) Mirror 3: 0xa7fa2992 (stale) Mirror 5: 0xa7fa2992 (stale) Mirror 6: 0xa7fa2992 (stale) Mirror 7: 0xa7fa2992 (stale) Mirror 8: 0x607bc70 (stale) Mirror 9: 0x81977cb3 (stale) CRC-32 checksum value for chunk [0, 0x3f9dc): Mirror 1: 0x204c5f51 Mirror 10: 0x204c5f51 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9dc: skipped 000189 1790972713.485866 MIRROR_RESYNC 9 mirrors 000190 1790972715.713166 trunc from 0x014990 to 0x019beb (0x525c bytes) 000192 1790972715.765727 trunc from 0x03b96e to 0x03f9db (0x406e bytes) 000193 1790972715.776263 trunc from 0x03f9dc to 0x010c80 (0x2ed5b bytes) 000194 1790972715.833153 read 0x00c9f6 thru 0x010c80 (0x428b bytes) 000195 1790972715.853305 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x10c81): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10c81): Mirror 2: 0xe8187bf0 (stale) Mirror 3: 0xe8187bf0 (stale) Mirror 5: 0xe8187bf0 (stale) Mirror 6: 0xe8187bf0 (stale) Mirror 7: 0xe8187bf0 (stale) Mirror 8: 0xe8187bf0 (stale) Mirror 9: 0xe8187bf0 (stale) Mirror 10: 0xe8187bf0 (stale) CRC-32 checksum value for chunk [0, 0x10c81): Mirror 1: 0xe8187bf0 Mirror 11: 0xe8187bf0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10c81: skipped 000197 1790972716.814768 trunc from 0x00f931 to 0x010c80 (0x1350 bytes) 000198 1790972716.881791 trunc from 0x010c81 to 0x0261a4 (0x15524 bytes) 000199 1790972716.905429 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x261a5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x261a5): Mirror 2: 0x3dd05cb (stale) Mirror 3: 0x3dd05cb (stale) Mirror 5: 0x3dd05cb (stale) Mirror 6: 0x3dd05cb (stale) Mirror 7: 0x3dd05cb (stale) Mirror 8: 0x3dd05cb (stale) Mirror 9: 0x3dd05cb (stale) Mirror 10: 0x3dd05cb (stale) Mirror 11: 0x3dd05cb (stale) CRC-32 checksum value for chunk [0, 0x261a5): Mirror 1: 0x3dd05cb Mirror 12: 0x3dd05cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x261a5: skipped 000200 1790972717.954757 trunc from 0x0261a5 to 0x0039c4 (0x227e0 bytes) 000201 1790972718.040240 trunc from 0x0039c5 to 0x032558 (0x2eb94 bytes) 000203 1790972718.064835 trunc from 0x032559 to 0x0069b2 (0x2bba6 bytes) 000204 1790972718.088937 mapread 0x000a9e thru 0x0052cc (0x482f bytes) 000205 1790972718.151557 mapread 0x003267 thru 0x0069b2 (0x374c bytes) 000206 1790972718.152596 mapwrite 0x038d58 thru 0x03ffff (0x72a8 bytes) 000207 1790972718.213212 trunc from 0x040000 to 0x016cab (0x29354 bytes) 000208 1790972718.261076 trunc from 0x00909a to 0x01579a (0xc701 bytes) 000209 1790972718.277057 trunc from 0x00ce0c to 0x016cab (0x9ea0 bytes) 000211 1790972718.293360 trunc from 0x0078df to 0x008a35 (0x1157 bytes) 000212 1790972718.303167 mapwrite 0x0140c0 thru 0x01ca03 (0x8944 bytes) 000213 1790972718.388846 write 0x035d00 thru 0x03ffff (0xa300 bytes) 000215 1790972718.390951 mapwrite 0x01ffd8 thru 0x024742 (0x476b bytes) 000216 1790972718.437086 mapwrite 0x01f5ff thru 0x02634f (0x6d51 bytes) 000217 1790972718.475291 trunc from 0x040000 to 0x030167 (0xfe98 bytes) 000218 1790972718.517730 mapwrite 0x01f706 thru 0x0241d0 (0x4acb bytes) 000220 1790972718.553020 trunc from 0x030168 to 0x020c6a (0xf4fd bytes) 000221 1790972718.572484 punch from 0x01ba9a to 0x01f6a4 (0x3c0b bytes) 000222 1790972718.578094 read 0x01ff26 thru 0x020c6a (0xd45 bytes) 000223 1790972718.583884 mapread 0x01e353 thru 0x020c6a (0x2918 bytes) 000224 1790972718.595256 trunc from 0x007ee6 to 0x0107d9 (0x88f4 bytes) 000225 1790972718.599493 read 0x018c53 thru 0x01f194 (0x6542 bytes) 000226 1790972718.601212 trunc from 0x020c6b to 0x02b7f0 (0xab86 bytes) 000227 1790972718.627107 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b7f1): Mirror 2: 0xb851371a (stale) Mirror 3: 0xb851371a (stale) Mirror 5: 0xb851371a (stale) Mirror 6: 0xb851371a (stale) Mirror 7: 0xb851371a (stale) Mirror 8: 0xb851371a (stale) Mirror 9: 0xb851371a (stale) Mirror 10: 0xb851371a (stale) Mirror 12: 0x3b588401 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b7f1): Mirror 2: 0xb851371a (stale) Mirror 3: 0xb851371a (stale) Mirror 5: 0xb851371a (stale) Mirror 6: 0xb851371a (stale) Mirror 7: 0xb851371a (stale) Mirror 8: 0xb851371a (stale) Mirror 9: 0xb851371a (stale) Mirror 10: 0xb851371a (stale) Mirror 12: 0x3b588401 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b7f1: skipped 000228 1790972719.138403 mapread 0x02260a thru 0x02b7f0 (0x91e7 bytes) fallocating to largest ever: 0x40000 000229 1790972719.161836 fallocate 0x03a99a thru 0x03ffff (0x5666 bytes) 000230 1790972719.167601 mapread 0x031c9e thru 0x03e09f (0xc402 bytes) 000231 1790972719.183605 trunc from 0x03daa3 to 0x03ffff (0x255d bytes) 000232 1790972719.191629 MIRROR_RESYNC 10 mirrors 000233 1790972721.334641 trunc from 0x03a88b to 0x03ffff (0x5775 bytes) 000235 1790972721.365727 write 0x038bfe thru 0x03b0d1 (0x24d4 bytes) 000236 1790972721.375199 mapwrite 0x01adf7 thru 0x01b59f (0x7a9 bytes) 000239 1790972721.399297 trunc from 0x040000 to 0x0381d4 (0x7e2b bytes) 000241 1790972721.419729 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x381d5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x381d5): Mirror 2: 0xe25a2bd9 (stale) Mirror 3: 0xe25a2bd9 (stale) Mirror 5: 0xe25a2bd9 (stale) Mirror 6: 0xe25a2bd9 (stale) Mirror 7: 0xe25a2bd9 (stale) Mirror 8: 0xe25a2bd9 (stale) Mirror 9: 0xe25a2bd9 (stale) Mirror 10: 0xe25a2bd9 (stale) Mirror 12: 0xe25a2bd9 (stale) CRC-32 checksum value for chunk [0, 0x381d5): Mirror 1: 0x7292b56d Mirror 13: 0x7292b56d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x381d5: skipped 000244 1790972722.513830 punch from 0x0278c1 to 0x032912 (0xb052 bytes) 000245 1790972722.575885 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x381d5): Mirror 2: 0xe25a2bd9 (stale) Mirror 3: 0xe25a2bd9 (stale) Mirror 5: 0xe25a2bd9 (stale) Mirror 6: 0xe25a2bd9 (stale) Mirror 8: 0xe25a2bd9 (stale) Mirror 9: 0xe25a2bd9 (stale) Mirror 10: 0xe25a2bd9 (stale) Mirror 12: 0xe25a2bd9 (stale) Mirror 13: 0x7292b56d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x381d5): Mirror 2: 0xe25a2bd9 (stale) Mirror 3: 0xe25a2bd9 (stale) Mirror 5: 0xe25a2bd9 (stale) Mirror 6: 0xe25a2bd9 (stale) Mirror 8: 0xe25a2bd9 (stale) Mirror 9: 0xe25a2bd9 (stale) Mirror 10: 0xe25a2bd9 (stale) Mirror 12: 0xe25a2bd9 (stale) Mirror 13: 0x7292b56d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x381d5: skipped 000246 1790972723.185667 punch from 0x0268e3 to 0x0328b8 (0xbfd6 bytes) 000247 1790972723.189850 mapwrite 0x01a3c8 thru 0x028da9 (0xe9e2 bytes) 000248 1790972723.236503 punch from 0x010491 to 0x01674c (0x62bc bytes) 000249 1790972723.241338 trunc from 0x0381d5 to 0x02dd8d (0xa447 bytes) 000250 1790972723.278301 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2dd8e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd8e): Mirror 2: 0x89e691bc (stale) Mirror 3: 0x89e691bc (stale) Mirror 5: 0x89e691bc (stale) Mirror 6: 0x89e691bc (stale) Mirror 8: 0x89e691bc (stale) Mirror 9: 0x89e691bc (stale) Mirror 10: 0x89e691bc (stale) Mirror 12: 0x89e691bc (stale) Mirror 13: 0x7e0bc612 (stale) CRC-32 checksum value for chunk [0, 0x2dd8e): Mirror 1: 0x2b51433a Mirror 14: 0x2b51433a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd8e: skipped 000251 1790972724.172292 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dd8e): Mirror 2: 0x89e691bc (stale) Mirror 3: 0x89e691bc (stale) Mirror 6: 0x89e691bc (stale) Mirror 8: 0x89e691bc (stale) Mirror 9: 0x89e691bc (stale) Mirror 10: 0x89e691bc (stale) Mirror 12: 0x89e691bc (stale) Mirror 13: 0x7e0bc612 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd8e): Mirror 2: 0x89e691bc (stale) Mirror 3: 0x89e691bc (stale) Mirror 6: 0x89e691bc (stale) Mirror 8: 0x89e691bc (stale) Mirror 9: 0x89e691bc (stale) Mirror 10: 0x89e691bc (stale) Mirror 12: 0x89e691bc (stale) Mirror 13: 0x7e0bc612 (stale) CRC-32 checksum value for chunk [0, 0x2dd8e): Mirror 1: 0x2b51433a Mirror 14: 0x2b51433a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd8e: skipped 000254 1790972725.097836 mapread 0x003ea3 thru 0x00fdb1 (0xbf0f bytes) 000255 1790972725.138611 write 0x004271 thru 0x004fae (0xd3e bytes) 000256 1790972725.205717 MIRROR_RESYNC 10 mirrors 000257 1790972727.837393 punch from 0x00fc10 to 0x01ade8 (0xb1d9 bytes) 000258 1790972727.873385 write 0x01077f thru 0x019ca2 (0x9524 bytes) 000259 1790972727.883572 mapwrite 0x033372 thru 0x03c055 (0x8ce4 bytes) 000260 1790972727.958423 write 0x02b279 thru 0x0318e2 (0x666a bytes) 000261 1790972727.964555 trunc from 0x03c056 to 0x030183 (0xbed2 bytes) 000262 1790972727.978194 write 0x01dced thru 0x02ca21 (0xed35 bytes) 000263 1790972727.986546 write 0x03eca0 thru 0x03ffff (0x1360 bytes) 000264 1790972727.987727 read 0x0063e2 thru 0x00a30c (0x3f2b bytes) 000265 1790972727.999593 MIRROR_RESYNC 10 mirrors 000267 1790972730.610263 fallocate 0x03f388 thru 0x03ffff (0xc78 bytes) 000268 1790972730.656552 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6c265d88 (stale) Mirror 3: 0x6c265d88 (stale) Mirror 6: 0x6c265d88 (stale) Mirror 8: 0x6c265d88 (stale) Mirror 9: 0x6c265d88 (stale) Mirror 10: 0x6c265d88 (stale) Mirror 12: 0x6c265d88 (stale) Mirror 13: 0x6c265d88 (stale) Mirror 14: 0x6c265d88 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6c265d88 Mirror 15: 0x6c265d88 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000269 1790972731.884144 mapwrite 0x002543 thru 0x009fc3 (0x7a81 bytes) 000270 1790972732.061349 trunc from 0x040000 to 0x02e279 (0x11d86 bytes) 000271 1790972732.102985 punch from 0x02850d to 0x02e279 (0x5d6d bytes) 000272 1790972732.119411 trunc from 0x02e27a to 0x01a9f9 (0x13880 bytes) 000273 1790972732.159293 punch from 0x00bca1 to 0x00dcd2 (0x2032 bytes) 000274 1790972732.168675 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a9fa): Mirror 2: 0x78531eec (stale) Mirror 3: 0x78531eec (stale) Mirror 8: 0x78531eec (stale) Mirror 9: 0x78531eec (stale) Mirror 10: 0x78531eec (stale) Mirror 12: 0x78531eec (stale) Mirror 13: 0x78531eec (stale) Mirror 14: 0x78531eec (stale) Mirror 15: 0x78531eec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a9fa): Mirror 2: 0x78531eec (stale) Mirror 3: 0x78531eec (stale) Mirror 8: 0x78531eec (stale) Mirror 9: 0x78531eec (stale) Mirror 10: 0x78531eec (stale) Mirror 12: 0x78531eec (stale) Mirror 13: 0x78531eec (stale) Mirror 14: 0x78531eec (stale) Mirror 15: 0x78531eec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a9fa: skipped 000275 1790972732.984352 mapread 0x01527a thru 0x01a9f9 (0x5780 bytes) 000276 1790972733.017355 mapread 0x00e191 thru 0x01a9f9 (0xc869 bytes) 000277 1790972733.036746 punch from 0x006b2f to 0x013762 (0xcc34 bytes) 000278 1790972733.046286 punch from 0x005fed to 0x00cebe (0x6ed2 bytes) 000279 1790972733.074132 mapread 0x011587 thru 0x015fd6 (0x4a50 bytes) 000280 1790972733.088577 trunc from 0x01a9fa to 0x010268 (0xa791 bytes) 000282 1790972733.120243 write 0x03e565 thru 0x03ffff (0x1a9b bytes) 000283 1790972733.126595 write 0x011292 thru 0x01146d (0x1dc bytes) 000284 1790972733.146049 mapwrite 0x034911 thru 0x03ffff (0xb6ef bytes) 000286 1790972733.229766 mapwrite 0x037797 thru 0x03ffff (0x8869 bytes) 000287 1790972733.300136 read 0x02fde2 thru 0x03de19 (0xe038 bytes) 000289 1790972733.301199 punch from 0x008a1f to 0x015664 (0xcc46 bytes) 000290 1790972733.316413 punch from 0x008b8a to 0x00d002 (0x4479 bytes) 000291 1790972733.335646 punch from 0x02553c to 0x02dac5 (0x858a bytes) 000292 1790972733.358996 mapwrite 0x0086f4 thru 0x00b667 (0x2f74 bytes) 000293 1790972733.406674 read 0x0357e3 thru 0x039b70 (0x438e bytes) 000294 1790972733.407580 mapwrite 0x023f6c thru 0x02fb4d (0xbbe2 bytes) 000295 1790972733.488370 write 0x022332 thru 0x029a71 (0x7740 bytes) 000296 1790972733.489586 fallocate 0x01b75e thru 0x029fdd (0xe880 bytes) 000297 1790972733.497386 trunc from 0x040000 to 0x01cb1f (0x234e0 bytes) 000298 1790972733.513260 punch from 0x011ea9 to 0x01ca9b (0xabf3 bytes) 000300 1790972733.519660 trunc from 0x0146ab to 0x01cb1f (0x8475 bytes) 000301 1790972733.529766 read 0x0059b8 thru 0x00fcb2 (0xa2fb bytes) 000302 1790972733.530482 mapwrite 0x00e09e thru 0x01b633 (0xd596 bytes) 000303 1790972733.560832 MIRROR_RESYNC 10 mirrors 000304 1790972736.477608 write 0x02a827 thru 0x030c4f (0x6429 bytes) 000305 1790972736.523521 fallocate 0x01016c thru 0x0150d8 (0x4f6d bytes) 000306 1790972736.535362 trunc from 0x030c50 to 0x0203c2 (0x1088d bytes) 000307 1790972736.606385 read 0x018d61 thru 0x01a15c (0x13fc bytes) 000309 1790972736.622240 trunc from 0x012329 to 0x0203c2 (0xe09a bytes) 000310 1790972736.628341 punch from 0x001669 to 0x00607f (0x4a17 bytes) 000311 1790972736.641474 trunc from 0x0203c3 to 0x009e70 (0x16552 bytes) 000313 1790972736.674382 MIRROR_RESYNC 10 mirrors 000315 1790972738.327689 mapread 0x004f30 thru 0x009e70 (0x4f41 bytes) 000316 1790972738.354581 trunc from 0x009e71 to 0x01ec72 (0x14e02 bytes) 000317 1790972738.403341 mapwrite 0x028b13 thru 0x0366b4 (0xdba2 bytes) 000318 1790972738.463121 trunc from 0x0366b5 to 0x02dfad (0x8707 bytes) 000319 1790972738.480517 mapread 0x00099a thru 0x007647 (0x6cae bytes) 000321 1790972738.490447 punch from 0x018631 to 0x019a6b (0x143b bytes) 000322 1790972738.495270 trunc from 0x02dfae to 0x0054a4 (0x28b09 bytes) 000323 1790972738.525502 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x54a5): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x54a5): Mirror 2: 0x6600748 (stale) Mirror 3: 0x6600748 (stale) Mirror 8: 0x6600748 (stale) Mirror 9: 0x6600748 (stale) Mirror 10: 0x6600748 (stale) Mirror 12: 0x6600748 (stale) Mirror 13: 0x6600748 (stale) Mirror 14: 0x6600748 (stale) Mirror 15: 0x6600748 (stale) CRC-32 checksum value for chunk [0, 0x54a5): Mirror 1: 0x6600748 Mirror 16: 0x6600748 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x54a5: skipped 000324 1790972739.602194 write 0x016afb thru 0x0241f1 (0xd6f7 bytes) 000325 1790972739.704371 read 0x01c9b2 thru 0x023778 (0x6dc7 bytes) 000326 1790972739.705594 read 0x019f9b thru 0x0241f1 (0xa257 bytes) 000327 1790972739.709684 read 0x00dfa1 thru 0x011b39 (0x3b99 bytes) 000328 1790972739.726117 mapwrite 0x03fa32 thru 0x03ffff (0x5ce bytes) 000329 1790972739.782579 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x649acf21 (stale) Mirror 3: 0x649acf21 (stale) Mirror 8: 0x649acf21 (stale) Mirror 9: 0x649acf21 (stale) Mirror 10: 0x649acf21 (stale) Mirror 12: 0x649acf21 (stale) Mirror 13: 0x649acf21 (stale) Mirror 14: 0x649acf21 (stale) Mirror 15: 0x649acf21 (stale) Mirror 16: 0xcb709d89 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf42547e2 Mirror 17: 0xf42547e2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000330 1790972740.848792 punch from 0x01341a to 0x01c373 (0x8f5a bytes) 000331 1790972740.892426 mapwrite 0x02ef91 thru 0x0377f7 (0x8867 bytes) 000332 1790972740.941168 punch from 0x03f056 to 0x03ffff (0xfaa bytes) 000333 1790972740.945707 read 0x030d72 thru 0x032404 (0x1693 bytes) 000334 1790972740.946276 read 0x001e47 thru 0x00c867 (0xaa21 bytes) 000335 1790972740.974000 mapread 0x006112 thru 0x01353f (0xd42e bytes) 000336 1790972740.975103 fallocate 0x02bd0a thru 0x03ae29 (0xf120 bytes) 000337 1790972740.979808 write 0x029cfb thru 0x02d326 (0x362c bytes) 000338 1790972740.985275 mapread 0x025b6f thru 0x026aad (0xf3f bytes) 000339 1790972740.986305 trunc from 0x0006f1 to 0x005d41 (0x5651 bytes) 000340 1790972740.991633 punch from 0x002e5a to 0x00725e (0x4405 bytes) 000341 1790972740.995468 mapwrite 0x00011d thru 0x00f1cd (0xf0b1 bytes) 000342 1790972741.045401 mapwrite 0x00104a thru 0x00efa2 (0xdf59 bytes) 000343 1790972741.072266 MIRROR_RESYNC 12 mirrors 000344 1790972744.396578 write 0x00aeb9 thru 0x014456 (0x959e bytes) 000346 1790972744.473168 trunc from 0x0342ab to 0x0345d4 (0x32a bytes) 000347 1790972744.480418 read 0x01335f thru 0x020843 (0xd4e5 bytes) 000348 1790972744.498232 trunc from 0x040000 to 0x02a836 (0x157c9 bytes) 000349 1790972744.535103 trunc from 0x0178a4 to 0x01d79f (0x5efc bytes) 000350 1790972744.540388 punch from 0x008d7c to 0x00e240 (0x54c5 bytes) 000351 1790972744.566574 MIRROR_RESYNC 12 mirrors 000352 1790972747.414196 trunc from 0x007720 to 0x00cfe3 (0x58c4 bytes) 000353 1790972747.463712 punch from 0x014109 to 0x015a0f (0x1907 bytes) 000354 1790972747.472724 mapread 0x008218 thru 0x014b85 (0xc96e bytes) 000355 1790972747.511593 read 0x004145 thru 0x005d58 (0x1c14 bytes) 000356 1790972747.512661 read 0x013705 thru 0x01f7fb (0xc0f7 bytes) 000357 1790972747.530094 trunc from 0x02a837 to 0x0301cc (0x5996 bytes) 000358 1790972747.553922 punch from 0x02aca7 to 0x02d79a (0x2af4 bytes) 000359 1790972747.559736 trunc from 0x0301cd to 0x0089c2 (0x2780a bytes) 000360 1790972747.575136 write 0x02548e thru 0x02984c (0x43bf bytes) 000361 1790972747.578105 trunc from 0x02984d to 0x02aee1 (0x1695 bytes) 000362 1790972747.621505 trunc from 0x02aee2 to 0x0191b2 (0x11d2f bytes) 000363 1790972747.650661 trunc from 0x014858 to 0x0191b2 (0x495b bytes) 000364 1790972747.655671 mapwrite 0x03bbe9 thru 0x03ffff (0x4417 bytes) 000365 1790972747.703579 trunc from 0x040000 to 0x0248d1 (0x1b72e bytes) 000367 1790972747.718834 mapwrite 0x02790d thru 0x028b0a (0x11fe bytes) 000368 1790972747.756548 MIRROR_RESYNC 12 mirrors 000369 1790972749.494188 fallocate 0x03d187 thru 0x03ffff (0x2e79 bytes) 000370 1790972749.532531 mapread 0x001618 thru 0x00981b (0x8204 bytes) 000371 1790972749.567157 mapwrite 0x03e0d9 thru 0x03ffff (0x1f27 bytes) 000372 1790972749.622689 mapwrite 0x02a5fb thru 0x032442 (0x7e48 bytes) 000373 1790972749.669879 read 0x020cdf thru 0x02cca0 (0xbfc2 bytes) 000374 1790972749.678625 punch from 0x027730 to 0x035dfb (0xe6cc bytes) 000375 1790972749.684080 punch from 0x026db6 to 0x03052b (0x9776 bytes) 000376 1790972749.696789 fallocate 0x000cf8 thru 0x00946e (0x8777 bytes) 000377 1790972749.703104 read 0x024b37 thru 0x02d0b9 (0x8583 bytes) 000378 1790972749.716331 write 0x019dcc thru 0x027055 (0xd28a bytes) 000380 1790972749.727886 write 0x014fd1 thru 0x022500 (0xd530 bytes) 000381 1790972749.745605 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x141b39e5 (stale) Mirror 3: 0x141b39e5 (stale) Mirror 8: 0x141b39e5 (stale) Mirror 9: 0x141b39e5 (stale) Mirror 12: 0x141b39e5 (stale) Mirror 13: 0x141b39e5 (stale) Mirror 14: 0x141b39e5 (stale) Mirror 15: 0x141b39e5 (stale) Mirror 16: 0x141b39e5 (stale) Mirror 17: 0x141b39e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x141b39e5 (stale) Mirror 3: 0x141b39e5 (stale) Mirror 8: 0x141b39e5 (stale) Mirror 9: 0x141b39e5 (stale) Mirror 12: 0x141b39e5 (stale) Mirror 13: 0x141b39e5 (stale) Mirror 14: 0x141b39e5 (stale) Mirror 15: 0x141b39e5 (stale) Mirror 16: 0x141b39e5 (stale) Mirror 17: 0x141b39e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000382 1790972750.423079 trunc from 0x040000 to 0x000b80 (0x3f47f bytes) 000383 1790972750.449285 punch from 0x0005a0 to 0x000b80 (0x5e1 bytes) 000384 1790972750.454980 mapwrite 0x03d085 thru 0x03ffff (0x2f7b bytes) 000385 1790972750.507958 read 0x03130b thru 0x039764 (0x845a bytes) 000386 1790972750.518145 mapwrite 0x02ffb2 thru 0x03ae6c (0xaebb bytes) 000387 1790972750.552824 read 0x002191 thru 0x011c1c (0xfa8c bytes) 000388 1790972750.562073 punch from 0x03c65d to 0x03e529 (0x1ecd bytes) 000389 1790972750.569944 trunc from 0x040000 to 0x021237 (0x1edc8 bytes) 000390 1790972750.587796 mapwrite 0x012d6e thru 0x0216f4 (0xe987 bytes) 000391 1790972750.629688 write 0x03363c thru 0x03ec46 (0xb60b bytes) 000392 1790972750.631705 write 0x0378cd thru 0x03f713 (0x7e47 bytes) 000393 1790972750.634397 mapread 0x037969 thru 0x03d97f (0x6017 bytes) 000394 1790972750.640845 read 0x014f80 thru 0x01561d (0x69e bytes) 000395 1790972750.641541 trunc from 0x02b590 to 0x02d594 (0x2005 bytes) 000396 1790972750.652972 MIRROR_SPLIT mirror 10 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f714): Mirror 2: 0xb017f441 (stale) Mirror 3: 0xb017f441 (stale) Mirror 8: 0xb017f441 (stale) Mirror 9: 0xb017f441 (stale) Mirror 12: 0xb017f441 (stale) Mirror 13: 0xb017f441 (stale) Mirror 14: 0xb017f441 (stale) Mirror 15: 0xb017f441 (stale) Mirror 16: 0xb017f441 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f714): Mirror 2: 0xb017f441 (stale) Mirror 3: 0xb017f441 (stale) Mirror 8: 0xb017f441 (stale) Mirror 9: 0xb017f441 (stale) Mirror 12: 0xb017f441 (stale) Mirror 13: 0xb017f441 (stale) Mirror 14: 0xb017f441 (stale) Mirror 15: 0xb017f441 (stale) Mirror 16: 0xb017f441 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f714: skipped 000397 1790972751.192502 trunc from 0x00cfb1 to 0x013737 (0x6787 bytes) 000398 1790972751.197298 write 0x036ba3 thru 0x03ffff (0x945d bytes) 000399 1790972751.215156 MIRROR_RESYNC 10 mirrors 000400 1790972753.119669 read 0x02c61e thru 0x0352c4 (0x8ca7 bytes) 000401 1790972753.126636 mapread 0x024dba thru 0x0274fe (0x2745 bytes) 000402 1790972753.150734 read 0x028a7e thru 0x0354bc (0xca3f bytes) 000403 1790972753.151292 trunc from 0x040000 to 0x029525 (0x16ada bytes) 000405 1790972753.193515 punch from 0x024b83 to 0x029525 (0x49a3 bytes) 000406 1790972753.197883 punch from 0x00585d to 0x00c1ef (0x6993 bytes) 000407 1790972753.206085 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x29526): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29526): Mirror 2: 0xc6dc1053 (stale) Mirror 3: 0xc6dc1053 (stale) Mirror 8: 0xc6dc1053 (stale) Mirror 9: 0xc6dc1053 (stale) Mirror 12: 0xc6dc1053 (stale) Mirror 13: 0xc6dc1053 (stale) Mirror 14: 0xc6dc1053 (stale) Mirror 15: 0xc6dc1053 (stale) Mirror 16: 0xc6dc1053 (stale) CRC-32 checksum value for chunk [0, 0x29526): Mirror 1: 0xc6dc1053 Mirror 17: 0xc6dc1053 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29526: skipped 000408 1790972754.014162 MIRROR_RESYNC 11 mirrors 000409 1790972755.712179 read 0x0246cb thru 0x025031 (0x967 bytes) 000410 1790972755.718524 trunc from 0x029526 to 0x03eb46 (0x15621 bytes) 000411 1790972755.788623 mapwrite 0x00f154 thru 0x01975e (0xa60b bytes) 000412 1790972755.833001 trunc from 0x03eb47 to 0x00c3e4 (0x32762 bytes) 000413 1790972755.856345 MIRROR_SPLIT mirror 3 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc3e5): Mirror 2: 0xd5260b1 (stale) Mirror 3: 0xd5260b1 (stale) Mirror 9: 0xd5260b1 (stale) Mirror 12: 0xd5260b1 (stale) Mirror 13: 0xd5260b1 (stale) Mirror 14: 0xd5260b1 (stale) Mirror 15: 0xd5260b1 (stale) Mirror 16: 0xd5260b1 (stale) Mirror 17: 0xd5260b1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc3e5): Mirror 2: 0xd5260b1 (stale) Mirror 3: 0xd5260b1 (stale) Mirror 9: 0xd5260b1 (stale) Mirror 12: 0xd5260b1 (stale) Mirror 13: 0xd5260b1 (stale) Mirror 14: 0xd5260b1 (stale) Mirror 15: 0xd5260b1 (stale) Mirror 16: 0xd5260b1 (stale) Mirror 17: 0xd5260b1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc3e5: skipped 000414 1790972756.304347 fallocate 0x02cf26 thru 0x0351d2 (0x82ad bytes) 000415 1790972756.311698 read 0x007d66 thru 0x00c3e4 (0x467f bytes) 000418 1790972756.321243 mapwrite 0x021a51 thru 0x02c00f (0xa5bf bytes) 000419 1790972756.382776 read 0x01a65a thru 0x01dba0 (0x3547 bytes) 000420 1790972756.391254 trunc from 0x025195 to 0x027058 (0x1ec4 bytes) 000421 1790972756.395451 fallocate 0x0014b3 thru 0x0111a6 (0xfcf4 bytes) 000422 1790972756.399732 mapread 0x024a6a thru 0x027641 (0x2bd8 bytes) 000423 1790972756.417774 mapwrite 0x0015ee thru 0x00a479 (0x8e8c bytes) 000424 1790972756.441771 trunc from 0x00acf4 to 0x014732 (0x9a3f bytes) 000425 1790972756.446646 read 0x019eca thru 0x01fca3 (0x5dda bytes) 000426 1790972756.447279 mapwrite 0x0328a0 thru 0x03ffff (0xd760 bytes) 000427 1790972756.496218 mapwrite 0x0180eb thru 0x01cf7e (0x4e94 bytes) 000428 1790972756.514505 write 0x003e77 thru 0x0049a3 (0xb2d bytes) 000429 1790972756.516276 read 0x017c25 thru 0x01913f (0x151b bytes) 000430 1790972756.518056 mapwrite 0x02c753 thru 0x03b558 (0xee06 bytes) 000431 1790972756.538318 fallocate 0x03f1ac thru 0x03ffff (0xe54 bytes) 000432 1790972756.543559 write 0x00d380 thru 0x0182f8 (0xaf79 bytes) 000433 1790972756.545807 trunc from 0x02c82c to 0x030110 (0x38e5 bytes) 000434 1790972756.552215 write 0x03f33b thru 0x03ffff (0xcc5 bytes) 000435 1790972756.553569 fallocate 0x026c37 thru 0x02bad7 (0x4ea1 bytes) 000436 1790972756.559259 trunc from 0x00a3a3 to 0x00aaa3 (0x701 bytes) 000437 1790972756.564876 mapwrite 0x0222be thru 0x02f9f3 (0xd736 bytes) 000438 1790972756.596745 punch from 0x01b1f4 to 0x0244ca (0x92d7 bytes) 000439 1790972756.603474 fallocate 0x0041ed thru 0x00870f (0x4523 bytes) 000440 1790972756.608381 write 0x0215b5 thru 0x02f1fb (0xdc47 bytes) 000441 1790972756.619473 MIRROR_RESYNC 10 mirrors 000442 1790972758.933591 punch from 0x030f6a to 0x03666e (0x5705 bytes) 000443 1790972758.969413 trunc from 0x040000 to 0x01d2c2 (0x22d3d bytes) 000445 1790972759.000521 fallocate 0x0072f8 thru 0x012891 (0xb59a bytes) 000446 1790972759.005047 read 0x00eb35 thru 0x01c4ac (0xd978 bytes) 000447 1790972759.016806 punch from 0x00c590 to 0x010e2c (0x489d bytes) 000448 1790972759.023168 write 0x009f33 thru 0x00d911 (0x39df bytes) 000449 1790972759.033275 mapread 0x012f43 thru 0x01cc7b (0x9d39 bytes) 000450 1790972759.034834 write 0x018f10 thru 0x01fe07 (0x6ef8 bytes) 000451 1790972759.041566 MIRROR_RESYNC 10 mirrors 000452 1790972760.715647 mapread 0x0045c6 thru 0x00a05f (0x5a9a bytes) 000455 1790972760.743005 fallocate 0x000428 thru 0x00bd56 (0xb92f bytes) 000456 1790972760.798996 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x1fe08): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1fe08): Mirror 2: 0x3a36da44 (stale) Mirror 3: 0x3a36da44 (stale) Mirror 9: 0x3a36da44 (stale) Mirror 12: 0x3a36da44 (stale) Mirror 13: 0x3a36da44 (stale) Mirror 14: 0x3a36da44 (stale) Mirror 15: 0x3a36da44 (stale) Mirror 16: 0x3a36da44 (stale) Mirror 17: 0x3a36da44 (stale) CRC-32 checksum value for chunk [0, 0x1fe08): Mirror 1: 0x3a36da44 Mirror 18: 0x3a36da44 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1fe08: skipped 000457 1790972761.971557 trunc from 0x006c5b to 0x01126e (0xa614 bytes) 000458 1790972762.035197 write 0x0146d1 thru 0x01f6d0 (0xb000 bytes) 000459 1790972762.049383 mapwrite 0x021d16 thru 0x029e7f (0x816a bytes) 000460 1790972762.113233 read 0x01a7db thru 0x01cd3e (0x2564 bytes) 000461 1790972762.113842 write 0x01e30b thru 0x023a45 (0x573b bytes) 000462 1790972762.115000 fallocate 0x00516d thru 0x014847 (0xf6db bytes) 000463 1790972762.120634 fallocate 0x03c199 thru 0x03ffff (0x3e67 bytes) 000464 1790972762.125652 trunc from 0x040000 to 0x008ef5 (0x3710a bytes) 000467 1790972762.161367 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8ef6): Mirror 2: 0x18f8384 (stale) Mirror 3: 0x18f8384 (stale) Mirror 9: 0x18f8384 (stale) Mirror 12: 0x18f8384 (stale) Mirror 14: 0x18f8384 (stale) Mirror 15: 0x18f8384 (stale) Mirror 16: 0x18f8384 (stale) Mirror 17: 0x18f8384 (stale) Mirror 18: 0x18f8384 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8ef6): Mirror 2: 0x18f8384 (stale) Mirror 3: 0x18f8384 (stale) Mirror 9: 0x18f8384 (stale) Mirror 12: 0x18f8384 (stale) Mirror 14: 0x18f8384 (stale) Mirror 15: 0x18f8384 (stale) Mirror 16: 0x18f8384 (stale) Mirror 17: 0x18f8384 (stale) Mirror 18: 0x18f8384 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8ef6: skipped 000468 1790972762.596813 mapwrite 0x02e23f thru 0x032bbb (0x497d bytes) 000469 1790972762.644819 mapread 0x02d6bc thru 0x02dfed (0x932 bytes) 000470 1790972762.646430 write 0x03b11d thru 0x03ffff (0x4ee3 bytes) 000471 1790972762.648111 trunc from 0x040000 to 0x0102ce (0x2fd31 bytes) 000472 1790972762.662952 trunc from 0x001059 to 0x001cfb (0xca3 bytes) 000473 1790972762.668083 mapread 0x0084d6 thru 0x0102ce (0x7df9 bytes) 000474 1790972762.681248 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x102cf): Mirror 3: 0x74c2e053 (stale) Mirror 9: 0x74c2e053 (stale) Mirror 12: 0x74c2e053 (stale) Mirror 14: 0x74c2e053 (stale) Mirror 15: 0x74c2e053 (stale) Mirror 16: 0x74c2e053 (stale) Mirror 17: 0x74c2e053 (stale) Mirror 18: 0x74c2e053 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x102cf): Mirror 3: 0x74c2e053 (stale) Mirror 9: 0x74c2e053 (stale) Mirror 12: 0x74c2e053 (stale) Mirror 14: 0x74c2e053 (stale) Mirror 15: 0x74c2e053 (stale) Mirror 16: 0x74c2e053 (stale) Mirror 17: 0x74c2e053 (stale) Mirror 18: 0x74c2e053 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x102cf: skipped 000475 1790972763.102879 punch from 0x006161 to 0x00ab70 (0x4a10 bytes) 000476 1790972763.108083 trunc from 0x0102cf to 0x0092ec (0x6fe2 bytes) 000478 1790972763.129633 trunc from 0x00554c to 0x0092ec (0x3da1 bytes) 000479 1790972763.134933 trunc from 0x0092ed to 0x00c479 (0x318d bytes) 000480 1790972763.150767 trunc from 0x000c6b to 0x00c479 (0xb80f bytes) 000481 1790972763.156787 mapwrite 0x01ebf7 thru 0x021e98 (0x32a2 bytes) 000482 1790972763.204094 read 0x007d9c thru 0x0120c9 (0xa32e bytes) 000483 1790972763.213685 trunc from 0x005696 to 0x008dc2 (0x372d bytes) 000485 1790972763.219058 write 0x02c8e2 thru 0x02d6ca (0xde9 bytes) 000486 1790972763.220806 fallocate 0x0391ba thru 0x0399b0 (0x7f7 bytes) 000487 1790972763.225619 punch from 0x02827e to 0x02f12c (0x6eaf bytes) 000488 1790972763.234005 read 0x01fb5d thru 0x027dbf (0x8263 bytes) 000489 1790972763.262259 MIRROR_SPLIT mirror 7 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x399b1): Mirror 3: 0x36ad2485 (stale) Mirror 9: 0x36ad2485 (stale) Mirror 12: 0x36ad2485 (stale) Mirror 14: 0x36ad2485 (stale) Mirror 15: 0x36ad2485 (stale) Mirror 16: 0x36ad2485 (stale) Mirror 18: 0x36ad2485 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x399b1): Mirror 3: 0x36ad2485 (stale) Mirror 9: 0x36ad2485 (stale) Mirror 12: 0x36ad2485 (stale) Mirror 14: 0x36ad2485 (stale) Mirror 15: 0x36ad2485 (stale) Mirror 16: 0x36ad2485 (stale) Mirror 18: 0x36ad2485 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x399b1: skipped 000490 1790972763.984344 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x399b1): Mirror 3: 0x36ad2485 (stale) Mirror 9: 0x36ad2485 (stale) Mirror 12: 0x36ad2485 (stale) Mirror 14: 0x36ad2485 (stale) Mirror 15: 0x36ad2485 (stale) Mirror 16: 0x36ad2485 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x399b1): Mirror 3: 0x36ad2485 (stale) Mirror 9: 0x36ad2485 (stale) Mirror 12: 0x36ad2485 (stale) Mirror 14: 0x36ad2485 (stale) Mirror 15: 0x36ad2485 (stale) Mirror 16: 0x36ad2485 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x399b1: skipped 000491 1790972764.464473 trunc from 0x03151a to 0x0399b0 (0x8497 bytes) 000492 1790972764.472454 punch from 0x0333f1 to 0x0399b0 (0x65c0 bytes) 000493 1790972764.481434 punch from 0x0388b0 to 0x0399b0 (0x1101 bytes) 000494 1790972764.491396 mapread 0x026a9e thru 0x02d54f (0x6ab2 bytes) 000495 1790972764.524030 trunc from 0x00538b to 0x00f30c (0x9f82 bytes) 000496 1790972764.534771 trunc from 0x0399b1 to 0x0202fd (0x196b3 bytes) 000497 1790972764.553675 mapread 0x01543c thru 0x0202fd (0xaec2 bytes) 000498 1790972764.571648 trunc from 0x0202fe to 0x01e578 (0x1d85 bytes) 000500 1790972764.595378 fallocate 0x01b999 thru 0x01fdef (0x4457 bytes) 000501 1790972764.602240 read 0x00dbed thru 0x01af13 (0xd327 bytes) 000502 1790972764.617578 trunc from 0x0062aa to 0x0117a8 (0xb4ff bytes) 000503 1790972764.622828 mapwrite 0x03d290 thru 0x03ffff (0x2d70 bytes) 000504 1790972764.685878 read 0x036d84 thru 0x03ffff (0x927c bytes) 000505 1790972764.687683 mapwrite 0x031182 thru 0x0362c8 (0x5147 bytes) 000506 1790972764.769560 mapwrite 0x02f790 thru 0x03b202 (0xba73 bytes) 000507 1790972764.816242 read 0x031438 thru 0x03a294 (0x8e5d bytes) 000508 1790972764.818255 read 0x015537 thru 0x015bcd (0x697 bytes) 000509 1790972764.819638 fallocate 0x031aa2 thru 0x03ffff (0xe55e bytes) 000510 1790972764.833282 MIRROR_SPLIT mirror 4 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x43dfe762 (stale) Mirror 9: 0x43dfe762 (stale) Mirror 12: 0x43dfe762 (stale) Mirror 15: 0x43dfe762 (stale) Mirror 16: 0x43dfe762 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x43dfe762 (stale) Mirror 9: 0x43dfe762 (stale) Mirror 12: 0x43dfe762 (stale) Mirror 15: 0x43dfe762 (stale) Mirror 16: 0x43dfe762 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000511 1790972765.410333 read 0x018417 thru 0x02056a (0x8154 bytes) 000512 1790972765.421849 read 0x021a49 thru 0x02ff84 (0xe53c bytes) 000513 1790972765.467580 fallocate 0x03d31c thru 0x03ffff (0x2ce4 bytes) 000514 1790972765.473666 fallocate 0x02de33 thru 0x03bc0c (0xddda bytes) 000515 1790972765.479365 write 0x034aa5 thru 0x03db0f (0x906b bytes) 000516 1790972765.488829 punch from 0x033b8d to 0x03ffff (0xc473 bytes) 000518 1790972765.508629 punch from 0x00821c to 0x00b5a9 (0x338e bytes) 000519 1790972765.515081 mapwrite 0x0053a1 thru 0x00b1e2 (0x5e42 bytes) 000521 1790972765.580004 mapread 0x03edb4 thru 0x03ffff (0x124c bytes) 000522 1790972765.597809 mapread 0x03d57a thru 0x03ffff (0x2a86 bytes) 000523 1790972765.607042 mapwrite 0x02bf7c thru 0x02e859 (0x28de bytes) 000524 1790972765.627951 trunc from 0x036ef4 to 0x03b04b (0x4158 bytes) 000525 1790972765.633868 write 0x017d33 thru 0x01ca3a (0x4d08 bytes) 000526 1790972765.639789 read 0x033e3c thru 0x03cc62 (0x8e27 bytes) 000528 1790972765.651850 fallocate 0x0011c0 thru 0x010ab2 (0xf8f3 bytes) 000529 1790972765.657472 read 0x020d76 thru 0x025fec (0x5277 bytes) 000530 1790972765.663577 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x43dfe762 (stale) Mirror 9: 0x43dfe762 (stale) Mirror 12: 0x43dfe762 (stale) Mirror 15: 0x43dfe762 (stale) Mirror 16: 0x43dfe762 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3deed154 Mirror 17: 0x3deed154 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000531 1790972766.541644 trunc from 0x040000 to 0x035128 (0xaed7 bytes) 000532 1790972766.587883 read 0x0227f8 thru 0x02803a (0x5843 bytes) 000533 1790972766.597881 mapwrite 0x02338c thru 0x02d2ae (0x9f23 bytes) 000534 1790972766.633390 write 0x02c044 thru 0x030f0b (0x4ec8 bytes) 000535 1790972766.634602 trunc from 0x035129 to 0x035be7 (0xabf bytes) 000536 1790972766.655262 read 0x02ab4e thru 0x02c0c4 (0x1577 bytes) 000537 1790972766.656092 trunc from 0x02b029 to 0x035be7 (0xabbf bytes) 000538 1790972766.669568 trunc from 0x035be8 to 0x03f6b7 (0x9ad0 bytes) 000539 1790972766.686323 write 0x038662 thru 0x03f3d9 (0x6d78 bytes) 000540 1790972766.696285 trunc from 0x03f6b8 to 0x021709 (0x1dfae bytes) 000541 1790972766.710137 trunc from 0x02170a to 0x01b7e5 (0x5f24 bytes) 000542 1790972766.726234 trunc from 0x01b7e6 to 0x030da2 (0x155bd bytes) 000543 1790972766.742039 mapwrite 0x03107a thru 0x03ad91 (0x9d18 bytes) 000544 1790972766.812753 fallocate 0x010e02 thru 0x01a425 (0x9624 bytes) 000545 1790972766.821708 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3ad92): Mirror 3: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ad92): Mirror 3: 0xe8d7d571 (stale) Mirror 9: 0xe8d7d571 (stale) Mirror 12: 0xe8d7d571 (stale) Mirror 15: 0xe8d7d571 (stale) Mirror 16: 0xe8d7d571 (stale) Mirror 17: 0x958f8cf5 (stale) CRC-32 checksum value for chunk [0, 0x3ad92): Mirror 1: 0xb23526f1 Mirror 18: 0xb23526f1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ad92: skipped 000546 1790972767.674452 write 0x002594 thru 0x0124b4 (0xff21 bytes) 000547 1790972767.725660 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3ad92): Mirror 3: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ad92): Mirror 3: 0xe8d7d571 (stale) Mirror 9: 0xe8d7d571 (stale) Mirror 12: 0xe8d7d571 (stale) Mirror 15: 0xe8d7d571 (stale) Mirror 16: 0xe8d7d571 (stale) Mirror 17: 0x958f8cf5 (stale) Mirror 18: 0xb23526f1 (stale) CRC-32 checksum value for chunk [0, 0x3ad92): Mirror 1: 0x3e3ab2d3 Mirror 19: 0x3e3ab2d3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ad92: skipped 000548 1790972768.782938 MIRROR_RESYNC 9 mirrors 000549 1790972770.595807 MIRROR_EXTEND to 10 mirrors 000550 1790972771.382656 trunc from 0x0134b8 to 0x01e4b1 (0xaffa bytes) 000551 1790972771.437520 trunc from 0x03ad92 to 0x01044c (0x2a945 bytes) 000552 1790972771.483693 punch from 0x0036bb to 0x00c333 (0x8c79 bytes) 000553 1790972771.492257 trunc from 0x0059e7 to 0x00f894 (0x9eae bytes) 000554 1790972771.504761 write 0x031296 thru 0x0367bd (0x5528 bytes) 000555 1790972771.507654 write 0x026935 thru 0x02d46c (0x6b38 bytes) 000556 1790972771.524207 read 0x03594a thru 0x0367bd (0xe74 bytes) 000557 1790972771.534180 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x367be): Mirror 3: 0x8af652b2 (stale) Mirror 9: 0x8af652b2 (stale) Mirror 15: 0x8af652b2 (stale) Mirror 16: 0x8af652b2 (stale) Mirror 17: 0x8af652b2 (stale) Mirror 18: 0x8af652b2 (stale) Mirror 19: 0x8af652b2 (stale) Mirror 20: 0x8af652b2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x367be): Mirror 3: 0x8af652b2 (stale) Mirror 9: 0x8af652b2 (stale) Mirror 15: 0x8af652b2 (stale) Mirror 16: 0x8af652b2 (stale) Mirror 17: 0x8af652b2 (stale) Mirror 18: 0x8af652b2 (stale) Mirror 19: 0x8af652b2 (stale) Mirror 20: 0x8af652b2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x367be: skipped 000558 1790972772.352194 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x367be): Mirror 3: (stale - read error) Mirror 9: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x367be): Mirror 3: 0x8af652b2 (stale) Mirror 9: 0x8af652b2 (stale) Mirror 15: 0x8af652b2 (stale) Mirror 16: 0x8af652b2 (stale) Mirror 17: 0x8af652b2 (stale) Mirror 18: 0x8af652b2 (stale) Mirror 19: 0x8af652b2 (stale) Mirror 20: 0x8af652b2 (stale) CRC-32 checksum value for chunk [0, 0x367be): Mirror 1: 0x6880b55b Mirror 21: 0x6880b55b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x367be: skipped 000559 1790972773.738493 punch from 0x0292b2 to 0x02cdcc (0x3b1b bytes) 000560 1790972773.832987 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x367be): Mirror 3: (stale - read error) Mirror 9: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x367be): Mirror 3: 0x8af652b2 (stale) Mirror 9: 0x8af652b2 (stale) Mirror 15: 0x8af652b2 (stale) Mirror 16: 0x8af652b2 (stale) Mirror 17: 0x8af652b2 (stale) Mirror 18: 0x8af652b2 (stale) Mirror 19: 0x8af652b2 (stale) Mirror 20: 0x8af652b2 (stale) Mirror 21: 0x6880b55b (stale) CRC-32 checksum value for chunk [0, 0x367be): Mirror 1: 0x8b7ac0f5 Mirror 22: 0x8b7ac0f5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x367be: skipped 000561 1790972775.539249 write 0x00bac1 thru 0x0132e0 (0x7820 bytes) 000562 1790972775.600174 fallocate 0x002fc4 thru 0x00b762 (0x879f bytes) 000563 1790972775.607037 fallocate 0x00ca1f thru 0x01a3dd (0xd9bf bytes) 000564 1790972775.625656 read 0x02adef thru 0x0356e5 (0xa8f7 bytes) 000565 1790972775.655842 punch from 0x018470 to 0x0273af (0xef40 bytes) 000566 1790972775.661742 mapread 0x027284 thru 0x03629a (0xf017 bytes) 000568 1790972775.693678 write 0x038b61 thru 0x03d4b8 (0x4958 bytes) 000569 1790972775.695122 write 0x0258a0 thru 0x0304e5 (0xac46 bytes) 000570 1790972775.701776 mapread 0x0184f4 thru 0x0205d1 (0x80de bytes) 000571 1790972775.722784 mapread 0x00bb5d thru 0x01a772 (0xec16 bytes) 000572 1790972775.734638 MIRROR_SPLIT mirror 5 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d4b9): Mirror 3: 0x16c633c (stale) Mirror 9: 0x16c633c (stale) Mirror 15: 0x16c633c (stale) Mirror 16: 0x16c633c (stale) Mirror 18: 0x16c633c (stale) Mirror 19: 0x16c633c (stale) Mirror 20: 0x16c633c (stale) Mirror 21: 0x9acabc6c (stale) Mirror 22: 0x2d8958cd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d4b9): Mirror 3: 0x16c633c (stale) Mirror 9: 0x16c633c (stale) Mirror 15: 0x16c633c (stale) Mirror 16: 0x16c633c (stale) Mirror 18: 0x16c633c (stale) Mirror 19: 0x16c633c (stale) Mirror 20: 0x16c633c (stale) Mirror 21: 0x9acabc6c (stale) Mirror 22: 0x2d8958cd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d4b9: skipped 000573 1790972776.639654 fallocate 0x005f8e thru 0x010a12 (0xaa85 bytes) 000574 1790972776.646759 trunc from 0x03d4b9 to 0x0059a7 (0x37b11 bytes) 000575 1790972776.674765 mapwrite 0x02fc26 thru 0x03e708 (0xeae3 bytes) 000576 1790972776.755610 punch from 0x034603 to 0x03e708 (0xa106 bytes) 000577 1790972776.764566 trunc from 0x00f330 to 0x01b16e (0xbe3f bytes) 000578 1790972776.770165 mapwrite 0x00a866 thru 0x010f38 (0x66d3 bytes) 000580 1790972776.800498 mapread 0x02452c thru 0x024ce7 (0x7bc bytes) 000581 1790972776.810788 mapread 0x000bc3 thru 0x00f4b0 (0xe8ee bytes) 000583 1790972776.820388 write 0x007653 thru 0x00a8da (0x3288 bytes) 000584 1790972776.826839 MIRROR_RESYNC 10 mirrors 000585 1790972780.435362 trunc from 0x03e709 to 0x00ed91 (0x2f977 bytes) 000586 1790972780.473678 mapread 0x003210 thru 0x008eed (0x5cde bytes) 000587 1790972780.491599 punch from 0x000404 to 0x005c0a (0x5807 bytes) 000588 1790972780.496601 trunc from 0x00ed92 to 0x00c97b (0x2416 bytes) 000589 1790972780.514147 mapread 0x00ab96 thru 0x00c97b (0x1de6 bytes) 000591 1790972780.524264 MIRROR_SPLIT mirror 9 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc97c): Mirror 3: 0x9ec25db6 (stale) Mirror 9: 0x9ec25db6 (stale) Mirror 15: 0x9ec25db6 (stale) Mirror 16: 0x9ec25db6 (stale) Mirror 18: 0x9ec25db6 (stale) Mirror 19: 0x9ec25db6 (stale) Mirror 20: 0x9ec25db6 (stale) Mirror 21: 0x9ec25db6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc97c): Mirror 3: 0x9ec25db6 (stale) Mirror 9: 0x9ec25db6 (stale) Mirror 15: 0x9ec25db6 (stale) Mirror 16: 0x9ec25db6 (stale) Mirror 18: 0x9ec25db6 (stale) Mirror 19: 0x9ec25db6 (stale) Mirror 20: 0x9ec25db6 (stale) Mirror 21: 0x9ec25db6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc97c: skipped 000593 1790972780.920594 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0xc97c): Mirror 3: (stale - read error) Mirror 9: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc97c): Mirror 3: 0x9ec25db6 (stale) Mirror 9: 0x9ec25db6 (stale) Mirror 15: 0x9ec25db6 (stale) Mirror 16: 0x9ec25db6 (stale) Mirror 18: 0x9ec25db6 (stale) Mirror 19: 0x9ec25db6 (stale) Mirror 20: 0x9ec25db6 (stale) Mirror 21: 0x9ec25db6 (stale) CRC-32 checksum value for chunk [0, 0xc97c): Mirror 1: 0x79322d3f Mirror 22: 0x79322d3f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc97c: skipped 000594 1790972781.840763 read 0x00997c thru 0x00c97b (0x3000 bytes) 000596 1790972781.870440 read 0x0099b5 thru 0x00c97b (0x2fc7 bytes) 000597 1790972781.885225 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc97c): Mirror 9: 0x9ec25db6 (stale) Mirror 15: 0x9ec25db6 (stale) Mirror 16: 0x9ec25db6 (stale) Mirror 18: 0x9ec25db6 (stale) Mirror 19: 0x9ec25db6 (stale) Mirror 20: 0x9ec25db6 (stale) Mirror 21: 0x9ec25db6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc97c): Mirror 9: 0x9ec25db6 (stale) Mirror 15: 0x9ec25db6 (stale) Mirror 16: 0x9ec25db6 (stale) Mirror 18: 0x9ec25db6 (stale) Mirror 19: 0x9ec25db6 (stale) Mirror 20: 0x9ec25db6 (stale) Mirror 21: 0x9ec25db6 (stale) CRC-32 checksum value for chunk [0, 0xc97c): Mirror 1: 0x79322d3f Mirror 22: 0x79322d3f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc97c: skipped 000598 1790972782.668737 trunc from 0x00c0bb to 0x00c97b (0x8c1 bytes) 000599 1790972782.715850 punch from 0x00c739 to 0x00c97b (0x243 bytes) 000600 1790972782.720312 fallocate 0x00ae00 thru 0x00fe4e (0x504f bytes) 000601 1790972782.724952 read 0x00a5f1 thru 0x00bd65 (0x1775 bytes) 000602 1790972782.730526 write 0x00c1d5 thru 0x00d6ef (0x151b bytes) 000603 1790972782.743250 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xd6f0): Mirror 9: 0x99644c3c (stale) Mirror 16: 0x99644c3c (stale) Mirror 18: 0x99644c3c (stale) Mirror 19: 0x99644c3c (stale) Mirror 20: 0x99644c3c (stale) Mirror 21: 0x99644c3c (stale) Mirror 22: 0xf99e232c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd6f0): Mirror 9: 0x99644c3c (stale) Mirror 16: 0x99644c3c (stale) Mirror 18: 0x99644c3c (stale) Mirror 19: 0x99644c3c (stale) Mirror 20: 0x99644c3c (stale) Mirror 21: 0x99644c3c (stale) Mirror 22: 0xf99e232c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd6f0: skipped 000604 1790972783.077077 punch from 0x008025 to 0x00be15 (0x3df1 bytes) 000606 1790972783.090201 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xd6f0): Mirror 9: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd6f0): Mirror 9: 0x99644c3c (stale) Mirror 16: 0x99644c3c (stale) Mirror 18: 0x99644c3c (stale) Mirror 19: 0x99644c3c (stale) Mirror 20: 0x99644c3c (stale) Mirror 21: 0x99644c3c (stale) Mirror 22: 0xf99e232c (stale) CRC-32 checksum value for chunk [0, 0xd6f0): Mirror 1: 0xb1574ff6 Mirror 23: 0xb1574ff6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd6f0: skipped 000607 1790972783.852400 mapwrite 0x018512 thru 0x01ef82 (0x6a71 bytes) 000608 1790972783.944339 write 0x01ee84 thru 0x020573 (0x16f0 bytes) 000609 1790972783.945809 trunc from 0x01d8ae to 0x020573 (0x2cc6 bytes) 000610 1790972783.953766 punch from 0x00c369 to 0x01174a (0x53e2 bytes) 000611 1790972783.958671 trunc from 0x01d8c2 to 0x020573 (0x2cb2 bytes) 000612 1790972783.962612 mapwrite 0x007779 thru 0x00e190 (0x6a18 bytes) 000613 1790972783.990381 read 0x01a614 thru 0x01defe (0x38eb bytes) 000614 1790972783.998101 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x20574): Mirror 9: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20574): Mirror 9: 0x2edb06de (stale) Mirror 16: 0x2edb06de (stale) Mirror 18: 0x2edb06de (stale) Mirror 19: 0x2edb06de (stale) Mirror 20: 0x2edb06de (stale) Mirror 21: 0x2edb06de (stale) Mirror 22: 0x93f778b9 (stale) Mirror 23: 0x6b9f2641 (stale) CRC-32 checksum value for chunk [0, 0x20574): Mirror 1: 0x59813b06 Mirror 24: 0x59813b06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20574: skipped 000615 1790972784.756404 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x20574): Mirror 9: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20574): Mirror 9: 0x2edb06de (stale) Mirror 16: 0x2edb06de (stale) Mirror 18: 0x2edb06de (stale) Mirror 19: 0x2edb06de (stale) Mirror 20: 0x2edb06de (stale) Mirror 21: 0x2edb06de (stale) Mirror 22: 0x93f778b9 (stale) Mirror 23: 0x6b9f2641 (stale) CRC-32 checksum value for chunk [0, 0x20574): Mirror 1: 0x59813b06 Mirror 24: 0x59813b06 Mirror 25: 0x59813b06 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20574: skipped 000616 1790972785.608336 punch from 0x00d111 to 0x01c064 (0xef54 bytes) 000617 1790972785.713273 write 0x0388ad thru 0x03d9a3 (0x50f7 bytes) 000618 1790972785.725270 trunc from 0x02b9fe to 0x02d358 (0x195b bytes) 000619 1790972785.734474 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3d9a4): Mirror 9: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d9a4): Mirror 9: 0x4b3c4f5d (stale) Mirror 16: 0x4b3c4f5d (stale) Mirror 18: 0x4b3c4f5d (stale) Mirror 19: 0x4b3c4f5d (stale) Mirror 20: 0x4b3c4f5d (stale) Mirror 21: 0x4b3c4f5d (stale) Mirror 22: 0x91b53581 (stale) Mirror 23: 0x892a1738 (stale) Mirror 24: 0x4d6377eb (stale) Mirror 25: 0x4d6377eb (stale) CRC-32 checksum value for chunk [0, 0x3d9a4): Mirror 1: 0x41a99678 Mirror 26: 0x41a99678 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d9a4: skipped 000620 1790972786.873169 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3d9a4): Mirror 9: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d9a4): Mirror 9: 0x4b3c4f5d (stale) Mirror 16: 0x4b3c4f5d (stale) Mirror 18: 0x4b3c4f5d (stale) Mirror 19: 0x4b3c4f5d (stale) Mirror 20: 0x4b3c4f5d (stale) Mirror 21: 0x4b3c4f5d (stale) Mirror 22: 0x91b53581 (stale) Mirror 23: 0x892a1738 (stale) Mirror 24: 0x4d6377eb (stale) Mirror 25: 0x4d6377eb (stale) CRC-32 checksum value for chunk [0, 0x3d9a4): Mirror 1: 0x41a99678 Mirror 26: 0x41a99678 Mirror 27: 0x41a99678 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d9a4: skipped 000621 1790972787.926469 write 0x01a465 thru 0x025da4 (0xb940 bytes) 000622 1790972788.007389 trunc from 0x03779e to 0x03d9a3 (0x6206 bytes) 000624 1790972788.016836 read 0x011f2d thru 0x012a9f (0xb73 bytes) 000627 1790972788.026034 read 0x01bb35 thru 0x022d9b (0x7267 bytes) 000629 1790972788.027866 mapread 0x02e216 thru 0x033ec1 (0x5cac bytes) 000630 1790972788.053188 MIRROR_RESYNC 13 mirrors 000631 1790972790.755702 punch from 0x008122 to 0x009d99 (0x1c78 bytes) 000633 1790972790.791878 read 0x01d834 thru 0x0218fd (0x40ca bytes) 000634 1790972790.799315 trunc from 0x03d9a4 to 0x029a2e (0x13f75 bytes) 000635 1790972790.818964 trunc from 0x007eb7 to 0x00de13 (0x5f5d bytes) 000636 1790972790.823647 mapwrite 0x032d38 thru 0x032f60 (0x229 bytes) 000638 1790972790.857793 trunc from 0x009524 to 0x00c930 (0x340d bytes) 000639 1790972790.862192 mapwrite 0x03918f thru 0x03d3e1 (0x4253 bytes) 000640 1790972790.899569 mapread 0x00b8bd thru 0x00bbc4 (0x308 bytes) 000641 1790972790.912294 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3d3e2): Mirror 9: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d3e2): Mirror 9: 0xe3bda4f3 (stale) Mirror 16: 0xe3bda4f3 (stale) Mirror 18: 0xe3bda4f3 (stale) Mirror 19: 0xe3bda4f3 (stale) Mirror 20: 0xe3bda4f3 (stale) Mirror 21: 0xe3bda4f3 (stale) Mirror 22: 0xe3bda4f3 (stale) Mirror 23: 0xe3bda4f3 (stale) Mirror 24: 0xe3bda4f3 (stale) Mirror 25: 0xe3bda4f3 (stale) Mirror 26: 0xe3bda4f3 (stale) Mirror 27: 0xe3bda4f3 (stale) CRC-32 checksum value for chunk [0, 0x3d3e2): Mirror 1: 0x72d89ee5 Mirror 28: 0x72d89ee5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d3e2: skipped 000642 1790972792.153280 fallocate 0x0023ab thru 0x00933f (0x6f95 bytes) 000643 1790972792.236331 trunc from 0x02ce9c to 0x0320e9 (0x524e bytes) 000644 1790972792.242231 mapread 0x0242a0 thru 0x02b01d (0x6d7e bytes) 000645 1790972792.264196 write 0x02c610 thru 0x0398d8 (0xd2c9 bytes) 000646 1790972792.290054 MIRROR_SPLIT mirror 12 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d3e2): Mirror 9: 0xe3bda4f3 (stale) Mirror 16: 0xe3bda4f3 (stale) Mirror 18: 0xe3bda4f3 (stale) Mirror 19: 0xe3bda4f3 (stale) Mirror 20: 0xe3bda4f3 (stale) Mirror 21: 0xe3bda4f3 (stale) Mirror 22: 0xe3bda4f3 (stale) Mirror 23: 0xe3bda4f3 (stale) Mirror 24: 0xe3bda4f3 (stale) Mirror 25: 0xe3bda4f3 (stale) Mirror 26: 0xe3bda4f3 (stale) Mirror 28: 0x72d89ee5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d3e2): Mirror 9: 0xe3bda4f3 (stale) Mirror 16: 0xe3bda4f3 (stale) Mirror 18: 0xe3bda4f3 (stale) Mirror 19: 0xe3bda4f3 (stale) Mirror 20: 0xe3bda4f3 (stale) Mirror 21: 0xe3bda4f3 (stale) Mirror 22: 0xe3bda4f3 (stale) Mirror 23: 0xe3bda4f3 (stale) Mirror 24: 0xe3bda4f3 (stale) Mirror 25: 0xe3bda4f3 (stale) Mirror 26: 0xe3bda4f3 (stale) Mirror 28: 0x72d89ee5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d3e2: skipped 000647 1790972792.939037 mapread 0x002994 thru 0x003941 (0xfae bytes) 000648 1790972792.952429 trunc from 0x0213e5 to 0x02fa60 (0xe67c bytes) 000649 1790972792.958586 punch from 0x01b584 to 0x025cb2 (0xa72f bytes) 000650 1790972792.963571 mapread 0x03bf39 thru 0x03d3e1 (0x14a9 bytes) 000651 1790972792.973011 write 0x0050a5 thru 0x00db6c (0x8ac8 bytes) 000653 1790972792.988584 read 0x00cbdb thru 0x011a61 (0x4e87 bytes) 000654 1790972792.996269 punch from 0x00b682 to 0x011614 (0x5f93 bytes) 000655 1790972793.010801 punch from 0x00bc40 to 0x019d1d (0xe0de bytes) 000657 1790972793.014600 trunc from 0x02c762 to 0x02ce2e (0x6cd bytes) 000658 1790972793.018877 read 0x03bec9 thru 0x03d3e1 (0x1519 bytes) 000659 1790972793.019340 punch from 0x022d86 to 0x024958 (0x1bd3 bytes) 000660 1790972793.022748 mapwrite 0x027310 thru 0x02bfd5 (0x4cc6 bytes) 000661 1790972793.051304 write 0x021ec7 thru 0x028b44 (0x6c7e bytes) 000663 1790972793.052849 write 0x039457 thru 0x03ffff (0x6ba9 bytes) 000664 1790972793.055301 mapwrite 0x013e5f thru 0x020577 (0xc719 bytes) 000665 1790972793.079865 punch from 0x02cea4 to 0x032b99 (0x5cf6 bytes) 000666 1790972793.091180 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xc939e891 (stale) Mirror 16: 0xc939e891 (stale) Mirror 18: 0xc939e891 (stale) Mirror 19: 0xc939e891 (stale) Mirror 20: 0xc939e891 (stale) Mirror 21: 0xc939e891 (stale) Mirror 22: 0xc939e891 (stale) Mirror 23: 0xc939e891 (stale) Mirror 24: 0xc939e891 (stale) Mirror 25: 0xc939e891 (stale) Mirror 26: 0xc939e891 (stale) Mirror 28: 0x8b09b84d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2ec53906 Mirror 29: 0x2ec53906 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000667 1790972794.181641 MIRROR_SPLIT mirror 10 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xc939e891 (stale) Mirror 16: 0xc939e891 (stale) Mirror 18: 0xc939e891 (stale) Mirror 19: 0xc939e891 (stale) Mirror 20: 0xc939e891 (stale) Mirror 21: 0xc939e891 (stale) Mirror 22: 0xc939e891 (stale) Mirror 23: 0xc939e891 (stale) Mirror 24: 0xc939e891 (stale) Mirror 26: 0xc939e891 (stale) Mirror 28: 0x8b09b84d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0xc939e891 (stale) Mirror 16: 0xc939e891 (stale) Mirror 18: 0xc939e891 (stale) Mirror 19: 0xc939e891 (stale) Mirror 20: 0xc939e891 (stale) Mirror 21: 0xc939e891 (stale) Mirror 22: 0xc939e891 (stale) Mirror 23: 0xc939e891 (stale) Mirror 24: 0xc939e891 (stale) Mirror 26: 0xc939e891 (stale) Mirror 28: 0x8b09b84d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2ec53906 Mirror 29: 0x2ec53906 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000669 1790972795.330208 read 0x0367d6 thru 0x037585 (0xdb0 bytes) 000670 1790972795.355995 MIRROR_RESYNC 13 mirrors 000671 1790972798.229170 trunc from 0x040000 to 0x02013f (0x1fec0 bytes) 000673 1790972798.322690 read 0x00d09a thru 0x016789 (0x96f0 bytes) 000675 1790972798.331993 mapread 0x0134ee thru 0x01d9c7 (0xa4da bytes) 000676 1790972798.357967 trunc from 0x0065d5 to 0x007450 (0xe7c bytes) 000677 1790972798.364717 mapwrite 0x0238d0 thru 0x02e1c5 (0xa8f6 bytes) 000678 1790972798.460217 write 0x020867 thru 0x02093e (0xd8 bytes) 000679 1790972798.461764 write 0x032c69 thru 0x034944 (0x1cdc bytes) 000680 1790972798.472493 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34945): Mirror 16: 0xcbd40ae8 (stale) Mirror 18: 0xcbd40ae8 (stale) Mirror 19: 0xcbd40ae8 (stale) Mirror 20: 0xcbd40ae8 (stale) Mirror 21: 0xcbd40ae8 (stale) Mirror 22: 0xcbd40ae8 (stale) Mirror 23: 0xcbd40ae8 (stale) Mirror 24: 0xcbd40ae8 (stale) Mirror 26: 0xcbd40ae8 (stale) Mirror 28: 0xcbd40ae8 (stale) Mirror 29: 0xcbd40ae8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34945): Mirror 16: 0xcbd40ae8 (stale) Mirror 18: 0xcbd40ae8 (stale) Mirror 19: 0xcbd40ae8 (stale) Mirror 20: 0xcbd40ae8 (stale) Mirror 21: 0xcbd40ae8 (stale) Mirror 22: 0xcbd40ae8 (stale) Mirror 23: 0xcbd40ae8 (stale) Mirror 24: 0xcbd40ae8 (stale) Mirror 26: 0xcbd40ae8 (stale) Mirror 28: 0xcbd40ae8 (stale) Mirror 29: 0xcbd40ae8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34945: skipped 000681 1790972799.258468 mapwrite 0x017cfa thru 0x02657a (0xe881 bytes) 000682 1790972799.368720 read 0x02d2ee thru 0x02ef71 (0x1c84 bytes) 000683 1790972799.373492 trunc from 0x034945 to 0x03b38a (0x6a46 bytes) 000684 1790972799.403179 write 0x0300a3 thru 0x03a116 (0xa074 bytes) 000687 1790972799.422443 punch from 0x03674d to 0x037e13 (0x16c7 bytes) 000688 1790972799.442461 mapread 0x011be6 thru 0x0177d3 (0x5bee bytes) 000689 1790972799.452794 fallocate 0x0013ff thru 0x00fc4c (0xe84e bytes) 000690 1790972799.458209 trunc from 0x039158 to 0x03b38a (0x2233 bytes) 000691 1790972799.471770 read 0x00b5a1 thru 0x014d3a (0x979a bytes) 000692 1790972799.472631 mapread 0x01c72f thru 0x029034 (0xc906 bytes) 000693 1790972799.480270 MIRROR_SPLIT mirror 1 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b38b): Mirror 18: 0x689b3c1e (stale) Mirror 19: 0x689b3c1e (stale) Mirror 20: 0x689b3c1e (stale) Mirror 21: 0x689b3c1e (stale) Mirror 22: 0x689b3c1e (stale) Mirror 23: 0x689b3c1e (stale) Mirror 24: 0x689b3c1e (stale) Mirror 26: 0x689b3c1e (stale) Mirror 28: 0x689b3c1e (stale) Mirror 29: 0x689b3c1e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b38b): Mirror 18: 0x689b3c1e (stale) Mirror 19: 0x689b3c1e (stale) Mirror 20: 0x689b3c1e (stale) Mirror 21: 0x689b3c1e (stale) Mirror 22: 0x689b3c1e (stale) Mirror 23: 0x689b3c1e (stale) Mirror 24: 0x689b3c1e (stale) Mirror 26: 0x689b3c1e (stale) Mirror 28: 0x689b3c1e (stale) Mirror 29: 0x689b3c1e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b38b: skipped 000694 1790972800.419854 read 0x031356 thru 0x03b38a (0xa035 bytes) 000695 1790972800.431465 write 0x01c5f8 thru 0x023ca8 (0x76b1 bytes) 000696 1790972800.447989 write 0x029382 thru 0x038bcf (0xf84e bytes) 000697 1790972800.458234 read 0x02c986 thru 0x03b38a (0xea05 bytes) 000698 1790972800.463858 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3b38b): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b38b): Mirror 18: 0x689b3c1e (stale) Mirror 19: 0x689b3c1e (stale) Mirror 20: 0x689b3c1e (stale) Mirror 21: 0x689b3c1e (stale) Mirror 22: 0x689b3c1e (stale) Mirror 23: 0x689b3c1e (stale) Mirror 24: 0x689b3c1e (stale) Mirror 26: 0x689b3c1e (stale) Mirror 28: 0x689b3c1e (stale) Mirror 29: 0x689b3c1e (stale) CRC-32 checksum value for chunk [0, 0x3b38b): Mirror 1: 0x7a443050 Mirror 30: 0x7a443050 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b38b: skipped 000699 1790972801.415097 read 0x0153ed thru 0x01aa6e (0x5682 bytes) 000700 1790972801.439844 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3b38b): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b38b): Mirror 18: 0x689b3c1e (stale) Mirror 19: 0x689b3c1e (stale) Mirror 20: 0x689b3c1e (stale) Mirror 21: 0x689b3c1e (stale) Mirror 22: 0x689b3c1e (stale) Mirror 23: 0x689b3c1e (stale) Mirror 24: 0x689b3c1e (stale) Mirror 26: 0x689b3c1e (stale) Mirror 28: 0x689b3c1e (stale) Mirror 29: 0x689b3c1e (stale) CRC-32 checksum value for chunk [0, 0x3b38b): Mirror 1: 0x7a443050 Mirror 30: 0x7a443050 Mirror 31: 0x7a443050 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b38b: skipped 000701 1790972803.003180 mapwrite 0x0173b3 thru 0x0247d9 (0xd427 bytes) 000702 1790972803.116887 read 0x02c63e thru 0x03b38a (0xed4d bytes) 000703 1790972803.126197 trunc from 0x01e937 to 0x02d4e8 (0xebb2 bytes) 000704 1790972803.135121 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x3b38b): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b38b): Mirror 18: 0x689b3c1e (stale) Mirror 19: 0x689b3c1e (stale) Mirror 20: 0x689b3c1e (stale) Mirror 21: 0x689b3c1e (stale) Mirror 22: 0x689b3c1e (stale) Mirror 23: 0x689b3c1e (stale) Mirror 24: 0x689b3c1e (stale) Mirror 26: 0x689b3c1e (stale) Mirror 28: 0x689b3c1e (stale) Mirror 29: 0x689b3c1e (stale) Mirror 30: 0x7a443050 (stale) Mirror 31: 0x7a443050 (stale) CRC-32 checksum value for chunk [0, 0x3b38b): Mirror 1: 0xec781e09 Mirror 32: 0xec781e09 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b38b: skipped 000705 1790972804.177355 read 0x024eb1 thru 0x032caa (0xddfa bytes) 000706 1790972804.195164 punch from 0x00c196 to 0x01a37f (0xe1ea bytes) 000707 1790972804.256850 punch from 0x03a14d to 0x03b38a (0x123e bytes) 000709 1790972804.263302 mapwrite 0x02ab26 thru 0x02d79b (0x2c76 bytes) 000710 1790972804.291372 punch from 0x02b5fc to 0x02d80a (0x220f bytes) 000711 1790972804.296373 mapread 0x02cb96 thru 0x02e379 (0x17e4 bytes) 000712 1790972804.302379 write 0x02b3c0 thru 0x033287 (0x7ec8 bytes) 000713 1790972804.317011 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x3b38b): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b38b): Mirror 18: 0x689b3c1e (stale) Mirror 19: 0x689b3c1e (stale) Mirror 20: 0x689b3c1e (stale) Mirror 21: 0x689b3c1e (stale) Mirror 22: 0x689b3c1e (stale) Mirror 23: 0x689b3c1e (stale) Mirror 24: 0x689b3c1e (stale) Mirror 26: 0x689b3c1e (stale) Mirror 28: 0x689b3c1e (stale) Mirror 29: 0x689b3c1e (stale) Mirror 30: 0x7a443050 (stale) Mirror 31: 0x7a443050 (stale) Mirror 32: 0xec781e09 (stale) CRC-32 checksum value for chunk [0, 0x3b38b): Mirror 1: 0x9d1a2552 Mirror 33: 0x9d1a2552 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b38b: skipped 000714 1790972805.405021 mapread 0x0015e6 thru 0x00bd46 (0xa761 bytes) 000715 1790972805.438856 read 0x004c80 thru 0x012742 (0xdac3 bytes) 000716 1790972805.453612 mapread 0x011307 thru 0x01a190 (0x8e8a bytes) 000717 1790972805.467378 read 0x000051 thru 0x00ab25 (0xaad5 bytes) 000718 1790972805.477942 fallocate 0x03da96 thru 0x03ffff (0x256a bytes) 000720 1790972805.514610 write 0x0131e5 thru 0x0160ee (0x2f0a bytes) 000721 1790972805.528506 trunc from 0x01a5b7 to 0x024d90 (0xa7da bytes) 000722 1790972805.532984 fallocate 0x03ac49 thru 0x03ffff (0x53b7 bytes) 000723 1790972805.540216 write 0x01d49e thru 0x01e108 (0xc6b bytes) 000724 1790972805.557386 read 0x031005 thru 0x03947f (0x847b bytes) 000725 1790972805.575065 trunc from 0x040000 to 0x01c711 (0x238ee bytes) 000726 1790972805.649079 read 0x015451 thru 0x01c711 (0x72c1 bytes) 000727 1790972805.650130 write 0x027319 thru 0x02c261 (0x4f49 bytes) 000728 1790972805.653227 mapread 0x00824c thru 0x015ded (0xdba2 bytes) 000729 1790972805.657393 read 0x00c844 thru 0x0182eb (0xbaa8 bytes) 000730 1790972805.659133 write 0x02cd85 thru 0x0395e2 (0xc85e bytes) 000731 1790972805.663358 punch from 0x03899a to 0x0395e2 (0xc49 bytes) 000732 1790972805.687360 trunc from 0x0213b2 to 0x02dd7c (0xc9cb bytes) 000733 1790972805.694500 fallocate 0x004ab7 thru 0x00b2c0 (0x680a bytes) 000734 1790972805.705153 trunc from 0x039149 to 0x0395e2 (0x49a bytes) 000735 1790972805.722817 MIRROR_SPLIT mirror 8 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x395e3): Mirror 18: 0x89fecf3d (stale) Mirror 19: 0x89fecf3d (stale) Mirror 20: 0x89fecf3d (stale) Mirror 21: 0x89fecf3d (stale) Mirror 22: 0x89fecf3d (stale) Mirror 23: 0x89fecf3d (stale) Mirror 24: 0x89fecf3d (stale) Mirror 28: 0x89fecf3d (stale) Mirror 29: 0x89fecf3d (stale) Mirror 30: 0x152937a2 (stale) Mirror 31: 0x152937a2 (stale) Mirror 32: 0x1fd0b5da (stale) Mirror 33: 0x1c15c4bd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x395e3): Mirror 18: 0x89fecf3d (stale) Mirror 19: 0x89fecf3d (stale) Mirror 20: 0x89fecf3d (stale) Mirror 21: 0x89fecf3d (stale) Mirror 22: 0x89fecf3d (stale) Mirror 23: 0x89fecf3d (stale) Mirror 24: 0x89fecf3d (stale) Mirror 28: 0x89fecf3d (stale) Mirror 29: 0x89fecf3d (stale) Mirror 30: 0x152937a2 (stale) Mirror 31: 0x152937a2 (stale) Mirror 32: 0x1fd0b5da (stale) Mirror 33: 0x1c15c4bd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x395e3: skipped 000736 1790972806.792112 fallocate 0x01a64d thru 0x02819b (0xdb4f bytes) 000737 1790972806.798230 fallocate 0x01b5c0 thru 0x021788 (0x61c9 bytes) 000738 1790972806.804348 fallocate 0x0098b7 thru 0x00af3c (0x1686 bytes) 000739 1790972806.809993 read 0x037150 thru 0x0395e2 (0x2493 bytes) 000740 1790972806.817545 write 0x01580d thru 0x020abe (0xb2b2 bytes) 000741 1790972806.852717 MIRROR_RESYNC 14 mirrors 000742 1790972810.165391 mapread 0x02e3ab thru 0x0342fb (0x5f51 bytes) 000743 1790972810.178674 write 0x00d49f thru 0x012659 (0x51bb bytes) 000744 1790972810.204034 write 0x0389b0 thru 0x03ffff (0x7650 bytes) 000745 1790972810.208268 trunc from 0x040000 to 0x007031 (0x38fce bytes) 000746 1790972810.220755 mapread 0x004c7b thru 0x007031 (0x23b7 bytes) 000747 1790972810.228506 trunc from 0x007032 to 0x011698 (0xa667 bytes) 000748 1790972810.236963 write 0x014c00 thru 0x022984 (0xdd85 bytes) 000749 1790972810.238141 fallocate 0x01c7ac thru 0x02abc9 (0xe41e bytes) 000751 1790972810.241780 write 0x006b96 thru 0x00f922 (0x8d8d bytes) 000752 1790972810.248143 MIRROR_RESYNC 14 mirrors 000753 1790972812.391048 mapread 0x01429a thru 0x01630e (0x2075 bytes) 000754 1790972812.403474 punch from 0x016433 to 0x022984 (0xc552 bytes) 000755 1790972812.431478 trunc from 0x00ec97 to 0x01e1c0 (0xf52a bytes) 000756 1790972812.435395 read 0x01312d thru 0x020a4b (0xd91f bytes) 000758 1790972812.449008 read 0x000b30 thru 0x0022e6 (0x17b7 bytes) 000759 1790972812.467465 write 0x012081 thru 0x021e42 (0xfdc2 bytes) 000760 1790972812.468968 punch from 0x013397 to 0x01b5a8 (0x8212 bytes) 000761 1790972812.478375 trunc from 0x022985 to 0x007859 (0x1b12b bytes) 000762 1790972812.498301 read 0x006f23 thru 0x007859 (0x937 bytes) 000763 1790972812.499121 trunc from 0x000ffa to 0x004c0f (0x3c16 bytes) 000764 1790972812.504800 punch from 0x00004e to 0x007859 (0x780c bytes) 000765 1790972812.510490 fallocate 0x002062 thru 0x00d9d0 (0xb96f bytes) 000766 1790972812.516285 punch from 0x00c839 to 0x00d9d0 (0x1198 bytes) 000767 1790972812.519997 punch from 0x00513f to 0x00d9d0 (0x8892 bytes) 000768 1790972812.524262 trunc from 0x00d9d1 to 0x01aa6f (0xd09f bytes) 000769 1790972812.536798 trunc from 0x01aa70 to 0x037488 (0x1ca19 bytes) 000770 1790972812.544367 write 0x00cef0 thru 0x014120 (0x7231 bytes) 000771 1790972812.551251 fallocate 0x03db43 thru 0x03ffff (0x24bd bytes) 000773 1790972812.555890 fallocate 0x02e722 thru 0x03e5b8 (0xfe97 bytes) 000774 1790972812.561464 mapwrite 0x01cb35 thru 0x01d3d0 (0x89c bytes) 000775 1790972812.575703 trunc from 0x0191f8 to 0x01d4f6 (0x42ff bytes) 000776 1790972812.581879 fallocate 0x027951 thru 0x0316cb (0x9d7b bytes) 000777 1790972812.587594 mapread 0x00d909 thru 0x00f951 (0x2049 bytes) 000779 1790972812.591089 trunc from 0x013e05 to 0x02039e (0xc59a bytes) 000780 1790972812.596384 punch from 0x007175 to 0x00b2f5 (0x4181 bytes) 000781 1790972812.600381 mapread 0x026664 thru 0x033a9f (0xd43c bytes) 000782 1790972812.602278 punch from 0x03aad9 to 0x03ffff (0x5527 bytes) 000785 1790972812.607424 read 0x0072f8 thru 0x008bbb (0x18c4 bytes) 000786 1790972812.617466 read 0x012768 thru 0x01fb41 (0xd3da bytes) 000787 1790972812.618284 mapwrite 0x010ece thru 0x016449 (0x557c bytes) 000788 1790972812.637475 punch from 0x02f8a1 to 0x036350 (0x6ab0 bytes) 000789 1790972812.641086 trunc from 0x0229ae to 0x0306e5 (0xdd38 bytes) 000790 1790972812.645215 fallocate 0x0213c8 thru 0x0269d9 (0x5612 bytes) 000791 1790972812.649030 fallocate 0x016fbe thru 0x01dd53 (0x6d96 bytes) 000792 1790972812.652734 mapread 0x028954 thru 0x02a273 (0x1920 bytes) 000793 1790972812.665389 trunc from 0x0013b6 to 0x003b7f (0x27ca bytes) 000795 1790972812.672757 read 0x025c3d thru 0x027299 (0x165d bytes) 000796 1790972812.673234 write 0x02c3f6 thru 0x030985 (0x4590 bytes) 000797 1790972812.675235 mapread 0x006cd6 thru 0x008e73 (0x219e bytes) 000798 1790972812.676063 punch from 0x00adad to 0x01748e (0xc6e2 bytes) 000800 1790972812.682359 mapwrite 0x02bb44 thru 0x039ee3 (0xe3a0 bytes) 000801 1790972812.702740 fallocate 0x02994f thru 0x034add (0xb18f bytes) 000802 1790972812.711763 read 0x01452e thru 0x01e248 (0x9d1b bytes) 000803 1790972812.728978 mapwrite 0x0393bf thru 0x03ffff (0x6c41 bytes) 000804 1790972812.754468 trunc from 0x040000 to 0x03d82a (0x27d5 bytes) 000805 1790972812.787989 read 0x01ae82 thru 0x02a095 (0xf214 bytes) 000806 1790972812.794986 MIRROR_RESYNC 14 mirrors 000807 1790972815.039013 mapread 0x006dc7 thru 0x008ebe (0x20f8 bytes) 000808 1790972815.048228 write 0x01819d thru 0x025d31 (0xdb95 bytes) 000809 1790972815.089397 read 0x023e26 thru 0x02f4f0 (0xb6cb bytes) 000810 1790972815.090252 fallocate 0x0205de thru 0x02ff9b (0xf9be bytes) 000812 1790972815.096078 trunc from 0x03d82b to 0x037d95 (0x5a95 bytes) 000814 1790972815.115001 read 0x032c30 thru 0x037d95 (0x5166 bytes) 000815 1790972815.123734 read 0x01fe46 thru 0x025a92 (0x5c4d bytes) 000816 1790972815.130540 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x37d96): Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37d96): Mirror 18: 0x5d48ee86 (stale) Mirror 19: 0x5d48ee86 (stale) Mirror 20: 0x5d48ee86 (stale) Mirror 21: 0x5d48ee86 (stale) Mirror 22: 0x5d48ee86 (stale) Mirror 23: 0x5d48ee86 (stale) Mirror 24: 0x5d48ee86 (stale) Mirror 28: 0x5d48ee86 (stale) Mirror 29: 0x5d48ee86 (stale) Mirror 30: 0x5d48ee86 (stale) Mirror 31: 0x5d48ee86 (stale) Mirror 32: 0x5d48ee86 (stale) Mirror 33: 0x5d48ee86 (stale) CRC-32 checksum value for chunk [0, 0x37d96): Mirror 1: 0x5f196751 Mirror 34: 0x5f196751 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37d96: skipped 000817 1790972816.111652 punch from 0x000d62 to 0x00997d (0x8c1c bytes) 000818 1790972816.204086 MIRROR_RESYNC 15 mirrors 000819 1790972819.218179 read 0x019aad thru 0x0233cf (0x9923 bytes) 000820 1790972819.245114 trunc from 0x037d96 to 0x00cf5f (0x2ae36 bytes) 000821 1790972819.350845 trunc from 0x00cf60 to 0x0257c6 (0x18867 bytes) 000822 1790972819.375732 mapwrite 0x009bc0 thru 0x011082 (0x74c3 bytes) 000823 1790972819.427283 write 0x037a95 thru 0x03a1c5 (0x2731 bytes) 000824 1790972819.428512 fallocate 0x016e83 thru 0x01efc3 (0x8141 bytes) 000825 1790972819.436301 trunc from 0x03a1c6 to 0x004b48 (0x3567d bytes) 000826 1790972819.472325 fallocate 0x033398 thru 0x03c819 (0x9482 bytes) 000828 1790972819.480006 punch from 0x002838 to 0x004b48 (0x2311 bytes) 000829 1790972819.485643 trunc from 0x000ed4 to 0x004b48 (0x3c75 bytes) 000830 1790972819.500064 MIRROR_SPLIT mirror 4 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x4b49): Mirror 18: 0xf095635e (stale) Mirror 19: 0xf095635e (stale) Mirror 20: 0xf095635e (stale) Mirror 22: 0xf095635e (stale) Mirror 23: 0xf095635e (stale) Mirror 24: 0xf095635e (stale) Mirror 28: 0xf095635e (stale) Mirror 29: 0xf095635e (stale) Mirror 30: 0xf095635e (stale) Mirror 31: 0xf095635e (stale) Mirror 32: 0xf095635e (stale) Mirror 33: 0xf095635e (stale) Mirror 34: 0xf095635e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x4b49): Mirror 18: 0xf095635e (stale) Mirror 19: 0xf095635e (stale) Mirror 20: 0xf095635e (stale) Mirror 22: 0xf095635e (stale) Mirror 23: 0xf095635e (stale) Mirror 24: 0xf095635e (stale) Mirror 28: 0xf095635e (stale) Mirror 29: 0xf095635e (stale) Mirror 30: 0xf095635e (stale) Mirror 31: 0xf095635e (stale) Mirror 32: 0xf095635e (stale) Mirror 33: 0xf095635e (stale) Mirror 34: 0xf095635e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x4b49: skipped 000831 1790972820.420149 mapwrite 0x03c7cc thru 0x03ffff (0x3834 bytes) 000832 1790972820.469048 mapwrite 0x008a63 thru 0x0158f1 (0xce8f bytes) 000834 1790972820.508430 mapwrite 0x0105c3 thru 0x018442 (0x7e80 bytes) 000836 1790972820.524977 read 0x005214 thru 0x00a3b0 (0x519d bytes) 000837 1790972820.532500 MIRROR_RESYNC 14 mirrors 000838 1790972824.225456 mapread 0x02f977 thru 0x034bc0 (0x524a bytes) 000839 1790972824.256697 trunc from 0x023e14 to 0x031c91 (0xde7e bytes) 000840 1790972824.338077 trunc from 0x040000 to 0x021e23 (0x1e1dc bytes) 000841 1790972824.366458 read 0x00ebf7 thru 0x011c19 (0x3023 bytes) 000842 1790972824.372631 mapwrite 0x017df0 thru 0x01ac3a (0x2e4b bytes) 000843 1790972824.407049 punch from 0x02169c to 0x021e23 (0x788 bytes) 000844 1790972824.411977 mapwrite 0x01f36e thru 0x028527 (0x91ba bytes) 000845 1790972824.452076 trunc from 0x028528 to 0x03ae85 (0x1295e bytes) 000846 1790972824.465765 punch from 0x02135b to 0x02c1f4 (0xae9a bytes) 000847 1790972824.471379 mapwrite 0x03741b thru 0x03ffff (0x8be5 bytes) 000848 1790972824.535718 trunc from 0x040000 to 0x006891 (0x3976e bytes) 000849 1790972824.568937 fallocate 0x009894 thru 0x016e4c (0xd5b9 bytes) 000850 1790972824.576184 mapwrite 0x00d381 thru 0x015e9c (0x8b1c bytes) 000851 1790972824.618253 read 0x00d40f thru 0x01180c (0x43fe bytes) 000852 1790972824.630269 trunc from 0x00dd9d to 0x01138a (0x35ee bytes) 000853 1790972824.644216 mapwrite 0x0105ce thru 0x016a58 (0x648b bytes) 000854 1790972824.694373 punch from 0x016509 to 0x016e4c (0x944 bytes) 000856 1790972824.706685 punch from 0x0152fc to 0x016e4c (0x1b51 bytes) 000857 1790972824.723487 mapwrite 0x02d7fc thru 0x03d164 (0xf969 bytes) 000858 1790972824.842362 fallocate 0x0099bb thru 0x01323a (0x9880 bytes) 000859 1790972824.862690 trunc from 0x03d165 to 0x01393b (0x29829 bytes) 000860 1790972824.915737 trunc from 0x01393c to 0x01e1f9 (0xa8be bytes) 000861 1790972824.983273 fallocate 0x03c426 thru 0x03e182 (0x1d5d bytes) 000862 1790972824.993024 mapread 0x028787 thru 0x029bcd (0x1447 bytes) 000863 1790972825.014644 mapread 0x00d16a thru 0x01c009 (0xeea0 bytes) 000864 1790972825.069523 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e183): Mirror 18: 0xd8e23ceb (stale) Mirror 19: 0xd8e23ceb (stale) Mirror 20: 0xd8e23ceb (stale) Mirror 23: 0xd8e23ceb (stale) Mirror 24: 0xd8e23ceb (stale) Mirror 28: 0xd8e23ceb (stale) Mirror 29: 0xd8e23ceb (stale) Mirror 30: 0xd8e23ceb (stale) Mirror 31: 0xd8e23ceb (stale) Mirror 32: 0xd8e23ceb (stale) Mirror 33: 0xd8e23ceb (stale) Mirror 34: 0xd8e23ceb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e183): Mirror 18: 0xd8e23ceb (stale) Mirror 19: 0xd8e23ceb (stale) Mirror 20: 0xd8e23ceb (stale) Mirror 23: 0xd8e23ceb (stale) Mirror 24: 0xd8e23ceb (stale) Mirror 28: 0xd8e23ceb (stale) Mirror 29: 0xd8e23ceb (stale) Mirror 30: 0xd8e23ceb (stale) Mirror 31: 0xd8e23ceb (stale) Mirror 32: 0xd8e23ceb (stale) Mirror 33: 0xd8e23ceb (stale) Mirror 34: 0xd8e23ceb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e183: skipped 000865 1790972826.188372 read 0x0148d8 thru 0x016042 (0x176b bytes) 000866 1790972826.208356 write 0x02a14f thru 0x033c8e (0x9b40 bytes) 000867 1790972826.239079 read 0x014645 thru 0x022615 (0xdfd1 bytes) 000868 1790972826.263638 punch from 0x02d6c9 to 0x033601 (0x5f39 bytes) 000869 1790972826.295759 trunc from 0x03e183 to 0x008ee1 (0x352a1 bytes) 000870 1790972826.318521 read 0x008113 thru 0x008ee1 (0xdcf bytes) 000871 1790972826.319166 trunc from 0x008ee2 to 0x029fec (0x2110b bytes) 000872 1790972826.343825 trunc from 0x029fed to 0x03188d (0x78a1 bytes) 000873 1790972826.372370 read 0x0100f2 thru 0x018318 (0x8227 bytes) 000874 1790972826.397474 read 0x026b61 thru 0x02ec9f (0x813f bytes) 000875 1790972826.408230 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3188e): Mirror 19: 0x84029534 (stale) Mirror 20: 0x84029534 (stale) Mirror 23: 0x84029534 (stale) Mirror 24: 0x84029534 (stale) Mirror 28: 0x84029534 (stale) Mirror 29: 0x84029534 (stale) Mirror 30: 0x84029534 (stale) Mirror 31: 0x84029534 (stale) Mirror 32: 0x84029534 (stale) Mirror 33: 0x84029534 (stale) Mirror 34: 0x84029534 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3188e): Mirror 19: 0x84029534 (stale) Mirror 20: 0x84029534 (stale) Mirror 23: 0x84029534 (stale) Mirror 24: 0x84029534 (stale) Mirror 28: 0x84029534 (stale) Mirror 29: 0x84029534 (stale) Mirror 30: 0x84029534 (stale) Mirror 31: 0x84029534 (stale) Mirror 32: 0x84029534 (stale) Mirror 33: 0x84029534 (stale) Mirror 34: 0x84029534 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3188e: skipped 000876 1790972827.079611 mapwrite 0x003b27 thru 0x0084b6 (0x4990 bytes) 000877 1790972827.126749 trunc from 0x02e0c5 to 0x03188d (0x37c9 bytes) 000878 1790972827.140154 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3188e): Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3188e): Mirror 19: 0x84029534 (stale) Mirror 20: 0x84029534 (stale) Mirror 23: 0x84029534 (stale) Mirror 24: 0x84029534 (stale) Mirror 28: 0x84029534 (stale) Mirror 29: 0x84029534 (stale) Mirror 30: 0x84029534 (stale) Mirror 31: 0x84029534 (stale) Mirror 32: 0x84029534 (stale) Mirror 33: 0x84029534 (stale) Mirror 34: 0x84029534 (stale) CRC-32 checksum value for chunk [0, 0x3188e): Mirror 1: 0xe685cc47 Mirror 35: 0xe685cc47 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3188e: skipped 000879 1790972828.395338 fallocate 0x02545b thru 0x02a9d0 (0x5576 bytes) 000880 1790972828.467796 trunc from 0x02b2ad to 0x03188d (0x65e1 bytes) 000881 1790972828.479001 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3188e): Mirror 19: 0x84029534 (stale) Mirror 20: 0x84029534 (stale) Mirror 23: 0x84029534 (stale) Mirror 28: 0x84029534 (stale) Mirror 29: 0x84029534 (stale) Mirror 30: 0x84029534 (stale) Mirror 31: 0x84029534 (stale) Mirror 32: 0x84029534 (stale) Mirror 33: 0x84029534 (stale) Mirror 34: 0x84029534 (stale) Mirror 35: 0xe685cc47 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3188e): Mirror 19: 0x84029534 (stale) Mirror 20: 0x84029534 (stale) Mirror 23: 0x84029534 (stale) Mirror 28: 0x84029534 (stale) Mirror 29: 0x84029534 (stale) Mirror 30: 0x84029534 (stale) Mirror 31: 0x84029534 (stale) Mirror 32: 0x84029534 (stale) Mirror 33: 0x84029534 (stale) Mirror 34: 0x84029534 (stale) Mirror 35: 0xe685cc47 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3188e: skipped 000882 1790972829.659655 mapread 0x010018 thru 0x01828e (0x8277 bytes) 000884 1790972829.687151 fallocate 0x01e8b7 thru 0x028070 (0x97ba bytes) 000885 1790972829.693616 fallocate 0x01d333 thru 0x01e6b8 (0x1386 bytes) 000886 1790972829.699819 trunc from 0x02ff14 to 0x03188d (0x197a bytes) 000887 1790972829.707511 read 0x005013 thru 0x00fbc5 (0xabb3 bytes) 000888 1790972829.717405 write 0x00e37c thru 0x0121aa (0x3e2f bytes) 000889 1790972829.724564 trunc from 0x03188e to 0x02ad75 (0x6b18 bytes) 000890 1790972829.772984 write 0x02f303 thru 0x030a1d (0x171b bytes) 000891 1790972829.775403 punch from 0x02bd91 to 0x030a1d (0x4c8d bytes) 000892 1790972829.798458 MIRROR_SPLIT mirror 2 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x30a1e): Mirror 19: 0xf76ce0a9 (stale) Mirror 23: 0xf76ce0a9 (stale) Mirror 28: 0xf76ce0a9 (stale) Mirror 29: 0xf76ce0a9 (stale) Mirror 30: 0xf76ce0a9 (stale) Mirror 31: 0xf76ce0a9 (stale) Mirror 32: 0xf76ce0a9 (stale) Mirror 33: 0xf76ce0a9 (stale) Mirror 34: 0xf76ce0a9 (stale) Mirror 35: 0x86b354fd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30a1e): Mirror 19: 0xf76ce0a9 (stale) Mirror 23: 0xf76ce0a9 (stale) Mirror 28: 0xf76ce0a9 (stale) Mirror 29: 0xf76ce0a9 (stale) Mirror 30: 0xf76ce0a9 (stale) Mirror 31: 0xf76ce0a9 (stale) Mirror 32: 0xf76ce0a9 (stale) Mirror 33: 0xf76ce0a9 (stale) Mirror 34: 0xf76ce0a9 (stale) Mirror 35: 0x86b354fd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30a1e: skipped 000893 1790972830.503064 mapwrite 0x01a37a thru 0x027cce (0xd955 bytes) 000895 1790972830.566297 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x30a1e): Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30a1e): Mirror 19: 0xf76ce0a9 (stale) Mirror 23: 0xf76ce0a9 (stale) Mirror 28: 0xf76ce0a9 (stale) Mirror 29: 0xf76ce0a9 (stale) Mirror 30: 0xf76ce0a9 (stale) Mirror 31: 0xf76ce0a9 (stale) Mirror 32: 0xf76ce0a9 (stale) Mirror 33: 0xf76ce0a9 (stale) Mirror 34: 0xf76ce0a9 (stale) Mirror 35: 0x86b354fd (stale) CRC-32 checksum value for chunk [0, 0x30a1e): Mirror 1: 0x1a3d8f54 Mirror 36: 0x1a3d8f54 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30a1e: skipped 000896 1790972831.774885 mapwrite 0x026458 thru 0x031983 (0xb52c bytes) 000897 1790972832.016877 punch from 0x01f604 to 0x02bf7b (0xc978 bytes) 000898 1790972832.030535 trunc from 0x00cdae to 0x018830 (0xba83 bytes) 000899 1790972832.056957 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31984): Mirror 19: 0x87a48f77 (stale) Mirror 23: 0x87a48f77 (stale) Mirror 28: 0x87a48f77 (stale) Mirror 29: 0x87a48f77 (stale) Mirror 30: 0x87a48f77 (stale) Mirror 31: 0x87a48f77 (stale) Mirror 32: 0x87a48f77 (stale) Mirror 34: 0x87a48f77 (stale) Mirror 35: 0x1a19b2c9 (stale) Mirror 36: 0xf1e854b0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31984): Mirror 19: 0x87a48f77 (stale) Mirror 23: 0x87a48f77 (stale) Mirror 28: 0x87a48f77 (stale) Mirror 29: 0x87a48f77 (stale) Mirror 30: 0x87a48f77 (stale) Mirror 31: 0x87a48f77 (stale) Mirror 32: 0x87a48f77 (stale) Mirror 34: 0x87a48f77 (stale) Mirror 35: 0x1a19b2c9 (stale) Mirror 36: 0xf1e854b0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31984: skipped 000900 1790972832.922676 trunc from 0x031984 to 0x00d5b4 (0x243cf bytes) 000901 1790972832.941812 write 0x01a52e thru 0x01ffd8 (0x5aab bytes) 000902 1790972832.943172 write 0x030764 thru 0x031e61 (0x16fe bytes) 000903 1790972832.944078 punch from 0x009706 to 0x00c8a8 (0x31a3 bytes) 000904 1790972832.952267 trunc from 0x031e62 to 0x01fd07 (0x1215a bytes) 000905 1790972832.971105 mapwrite 0x02fc5c thru 0x03dbec (0xdf91 bytes) 000906 1790972833.037490 MIRROR_RESYNC 11 mirrors 000907 1790972835.689124 trunc from 0x03dbed to 0x029d8b (0x13e61 bytes) 000908 1790972835.721610 mapwrite 0x02172b thru 0x0223f1 (0xcc7 bytes) 000909 1790972835.743062 read 0x028d0c thru 0x029d8b (0x1080 bytes) 000910 1790972835.746358 fallocate 0x006b21 thru 0x00f21d (0x86fd bytes) 000911 1790972835.749213 trunc from 0x0199b4 to 0x01ce67 (0x34b4 bytes) 000912 1790972835.751517 mapwrite 0x01964c thru 0x019785 (0x13a bytes) 000913 1790972835.763462 mapread 0x01269b thru 0x01d1c9 (0xab2f bytes) 000915 1790972835.764514 trunc from 0x029d8c to 0x01921c (0x10b6f bytes) 000916 1790972835.780447 write 0x01de6a thru 0x021eef (0x4086 bytes) 000917 1790972835.786165 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x21ef0): Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21ef0): Mirror 19: 0x7f66b303 (stale) Mirror 23: 0x7f66b303 (stale) Mirror 28: 0x7f66b303 (stale) Mirror 29: 0x7f66b303 (stale) Mirror 30: 0x7f66b303 (stale) Mirror 31: 0x7f66b303 (stale) Mirror 32: 0x7f66b303 (stale) Mirror 34: 0x7f66b303 (stale) Mirror 35: 0x7f66b303 (stale) Mirror 36: 0x7f66b303 (stale) CRC-32 checksum value for chunk [0, 0x21ef0): Mirror 1: 0x46d7fe77 Mirror 37: 0x46d7fe77 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21ef0: skipped 000918 1790972836.553227 fallocate 0x034530 thru 0x03d163 (0x8c34 bytes) 000919 1790972836.592650 trunc from 0x01cc1c to 0x01f86d (0x2c52 bytes) 000921 1790972836.597059 write 0x031ce5 thru 0x0372ed (0x5609 bytes) 000922 1790972836.618736 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x3d164): Mirror 19: (stale - read error) Mirror 23: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d164): Mirror 19: 0x2af66a65 (stale) Mirror 23: 0x2af66a65 (stale) Mirror 28: 0x2af66a65 (stale) Mirror 29: 0x2af66a65 (stale) Mirror 30: 0x2af66a65 (stale) Mirror 31: 0x2af66a65 (stale) Mirror 32: 0x2af66a65 (stale) Mirror 34: 0x2af66a65 (stale) Mirror 35: 0x2af66a65 (stale) Mirror 36: 0x2af66a65 (stale) Mirror 37: 0x33ad9cc2 (stale) CRC-32 checksum value for chunk [0, 0x3d164): Mirror 1: 0x16fd4e8a Mirror 38: 0x16fd4e8a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d164: skipped 000923 1790972837.978181 mapwrite 0x01a3bf thru 0x021a30 (0x7672 bytes) 000924 1790972838.163740 fallocate 0x02e5f5 thru 0x0365d3 (0x7fdf bytes) 000925 1790972838.168263 trunc from 0x02b70e to 0x039d1c (0xe60f bytes) 000926 1790972838.177400 mapread 0x0008a7 thru 0x00a458 (0x9bb2 bytes) 000927 1790972838.200463 read 0x023f39 thru 0x02e867 (0xa92f bytes) 000928 1790972838.215499 mapread 0x013559 thru 0x01ddb3 (0xa85b bytes) 000929 1790972838.223401 write 0x029d8d thru 0x032387 (0x85fb bytes) 000930 1790972838.233755 read 0x0002a0 thru 0x003c77 (0x39d8 bytes) 000931 1790972838.234429 read 0x018468 thru 0x020304 (0x7e9d bytes) 000932 1790972838.241173 MIRROR_RESYNC 13 mirrors 000933 1790972840.608265 mapwrite 0x01dbb1 thru 0x02a828 (0xcc78 bytes) 000934 1790972840.785456 read 0x020cdb thru 0x028819 (0x7b3f bytes) 000936 1790972840.786532 mapread 0x022b59 thru 0x025b09 (0x2fb1 bytes) 000937 1790972840.796119 MIRROR_SPLIT mirror 1 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000405:0x49:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d164): Mirror 23: 0x8fe03f84 (stale) Mirror 28: 0x8fe03f84 (stale) Mirror 29: 0x8fe03f84 (stale) Mirror 30: 0x8fe03f84 (stale) Mirror 31: 0x8fe03f84 (stale) Mirror 32: 0x8fe03f84 (stale) Mirror 34: 0x8fe03f84 (stale) Mirror 35: 0x8fe03f84 (stale) Mirror 36: 0x8fe03f84 (stale) Mirror 37: 0x8fe03f84 (stale) Mirror 38: 0x8fe03f84 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d164): Mirror 23: 0x8fe03f84 (stale) Mirror 28: 0x8fe03f84 (stale) Mirror 29: 0x8fe03f84 (stale) Mirror 30: 0x8fe03f84 (stale) Mirror 31: 0x8fe03f84 (stale) Mirror 32: 0x8fe03f84 (stale) Mirror 34: 0x8fe03f84 (stale) Mirror 35: 0x8fe03f84 (stale) Mirror 36: 0x8fe03f84 (stale) Mirror 37: 0x8fe03f84 (stale) Mirror 38: 0x8fe03f84 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d164: skipped 000938 1790972841.598134 write 0x019046 thru 0x01f68a (0x6645 bytes) 000939 1790972841.610773 fallocate 0x019eff thru 0x025c3b (0xbd3d bytes) 000940 1790972841.616182 mapread 0x033c83 thru 0x03d163 (0x94e1 bytes) 000941 1790972841.636699 fallocate 0x01dc78 thru 0x020b2b (0x2eb4 bytes) 000942 1790972841.641482 fallocate 0x003d94 thru 0x00ec14 (0xae81 bytes) 000943 1790972841.646752 trunc from 0x01f816 to 0x02515d (0x5948 bytes) 000944 1790972841.660500 trunc from 0x03d164 to 0x0332e3 (0x9e80 bytes) 000945 1790972841.677351 read 0x0086a0 thru 0x00d011 (0x4972 bytes) 000946 1790972841.686314 read 0x006459 thru 0x00f595 (0x913d bytes) 000948 1790972841.696200 write 0x0260a8 thru 0x027f40 (0x1e99 bytes) 000949 1790972841.697545 trunc from 0x0332e4 to 0x034692 (0x13af bytes) 000951 1790972841.714716 write 0x016d36 thru 0x017981 (0xc4c bytes) 000952 1790972841.715780 fallocate 0x0038f9 thru 0x00bb87 (0x828f bytes) 000953 1790972841.720630 write 0x01bee7 thru 0x02537d (0x9497 bytes) 000954 1790972841.722477 trunc from 0x013f0c to 0x015104 (0x11f9 bytes) 000955 1790972841.726534 mapwrite 0x021250 thru 0x026ee8 (0x5c99 bytes) 000956 1790972841.741296 trunc from 0x034693 to 0x007a81 (0x2cc11 bytes) 000957 1790972841.760304 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x7a82): Mirror 23: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) Mirror 38: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x7a82): Mirror 23: 0xffd2929f (stale) Mirror 28: 0xffd2929f (stale) Mirror 29: 0xffd2929f (stale) Mirror 30: 0xffd2929f (stale) Mirror 31: 0xffd2929f (stale) Mirror 32: 0xffd2929f (stale) Mirror 34: 0xffd2929f (stale) Mirror 35: 0xffd2929f (stale) Mirror 36: 0xffd2929f (stale) Mirror 37: 0xffd2929f (stale) Mirror 38: 0xffd2929f (stale) CRC-32 checksum value for chunk [0, 0x7a82): Mirror 1: 0xffd2929f Mirror 39: 0xffd2929f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x7a82: skipped 000958 1790972842.683600 trunc from 0x007903 to 0x007a81 (0x17f bytes) 000959 1790972842.741158 mapwrite 0x00b984 thru 0x01225f (0x68dc bytes) 000961 1790972842.818582 mapwrite 0x02128c thru 0x026e7a (0x5bef bytes) 000962 1790972842.873311 mapwrite 0x0005ad thru 0x00df70 (0xd9c4 bytes) 000963 1790972842.908056 write 0x01ab28 thru 0x023e3b (0x9314 bytes) 000964 1790972842.910639 punch from 0x02348b to 0x026e7a (0x39f0 bytes) 000965 1790972842.931513 punch from 0x021247 to 0x026e7a (0x5c34 bytes) 000966 1790972842.936643 read 0x01695d thru 0x01c582 (0x5c26 bytes) 000967 1790972842.943650 mapwrite 0x015447 thru 0x019c0a (0x47c4 bytes) 000969 1790972842.967695 fallocate 0x000d34 thru 0x004b5a (0x3e27 bytes) 000970 1790972842.973587 punch from 0x01042f to 0x01f792 (0xf364 bytes) 000971 1790972842.979555 mapwrite 0x00c110 thru 0x00ec3e (0x2b2f bytes) 000972 1790972842.993528 punch from 0x01d596 to 0x025b74 (0x85df bytes) 000973 1790972842.998664 fallocate 0x035b70 thru 0x03a68c (0x4b1d bytes) 000974 1790972843.004601 trunc from 0x03a68d to 0x00a99a (0x2fcf2 bytes) 000975 1790972843.026629 mapread 0x004a28 thru 0x004e97 (0x470 bytes) 000976 1790972843.030039 write 0x00d555 thru 0x014910 (0x73bc bytes) 000977 1790972843.031696 mapread 0x004895 thru 0x0075b2 (0x2d1e bytes) 000978 1790972843.032723 trunc from 0x014911 to 0x0182c7 (0x39b7 bytes) 000979 1790972843.048887 read 0x003584 thru 0x013564 (0xffe1 bytes) 000980 1790972843.056194 fallocate 0x00c4ba thru 0x00ca34 (0x57b bytes) 000981 1790972843.060969 trunc from 0x0182c8 to 0x037bbc (0x1f8f5 bytes) 000982 1790972843.085696 mapread 0x018998 thru 0x018c36 (0x29f bytes) 000983 1790972843.087724 trunc from 0x037bbd to 0x03c8b0 (0x4cf4 bytes) 000984 1790972843.116464 read 0x02653b thru 0x0292be (0x2d84 bytes) 000985 1790972843.123660 trunc from 0x03c8b1 to 0x029142 (0x1376e bytes) 000986 1790972843.150930 fallocate 0x01af41 thru 0x021c1f (0x6cdf bytes) 000987 1790972843.156704 trunc from 0x01a37d to 0x027c32 (0xd8b6 bytes) 000989 1790972843.165726 mapwrite 0x00bcf3 thru 0x01026b (0x4579 bytes) 000990 1790972843.196972 trunc from 0x029143 to 0x035487 (0xc345 bytes) 000991 1790972843.226564 write 0x024fdd thru 0x02a161 (0x5185 bytes) 000992 1790972843.240221 mapread 0x0030f9 thru 0x00d3ff (0xa307 bytes) 000994 1790972843.242706 read 0x00b65e thru 0x011d00 (0x66a3 bytes) 000996 1790972843.247536 MIRROR_RESYNC 13 mirrors 000997 1790972845.381827 punch from 0x00ada9 to 0x01531b (0xa573 bytes) 000998 1790972845.425134 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x35488): Mirror 23: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) Mirror 36: (stale - read error) Mirror 37: (stale - read error) Mirror 38: (stale - read error) Mirror 39: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35488): Mirror 23: 0x4e1dd030 (stale) Mirror 28: 0x4e1dd030 (stale) Mirror 29: 0x4e1dd030 (stale) Mirror 30: 0x4e1dd030 (stale) Mirror 31: 0x4e1dd030 (stale) Mirror 32: 0x4e1dd030 (stale) Mirror 34: 0x4e1dd030 (stale) Mirror 35: 0x4e1dd030 (stale) Mirror 36: 0x4e1dd030 (stale) Mirror 37: 0x4e1dd030 (stale) Mirror 38: 0x4e1dd030 (stale) Mirror 39: 0x4e1dd030 (stale) CRC-32 checksum value for chunk [0, 0x35488): Mirror 1: 0x4c946223 Mirror 40: 0x4c946223 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35488: skipped 000999 1790972846.446939 write 0x0130ba thru 0x01bfe3 (0x8f2a bytes) 001000 1790972846.520469 punch from 0x010c07 to 0x019ec2 (0x92bc bytes) All operations completed A-OK! PASS 70a (169s) == sanity-flr test 71: check mirror extend parent fid ==== 16:27:32 (1790972852) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x22c:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (10s) == sanity-flr test 72: check append on FLR file ========== 16:27:43 (1790972863) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x337:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x356:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.28546 s, 7.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x337:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x338:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x356:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166566 s, 6.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x337:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x338:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x357:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x356:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x337:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x338:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x357:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x356:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x339:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 16:27:52 (1790972872) Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre2 Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 5881728 .. Extending file size to 2524608 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8370816 .. Extending file size to 3817312 .. Extending file size to 8550048 .. Extending file size to 6214336 .. Extending file size to 6653248 .. Extending file size to 8890976 .. Extending file size to 8291712 .. Extending file size to 8343200 .. Extending file size to 8435328 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8454656 .. Extending file size to 9079936 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 62271 62272 62273 62276 62278 192.168.201.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg142-client.virtnet /mnt/lustre2 (opts:) 192.168.201.142@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg142-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (89s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 16:29:21 (1790972961) Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre2 Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 1960960 .. Extending file size to 8932544 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6972640 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7257056 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8601504 .. Extending file size to 8920320 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8994144 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8944800 .. Waiting 63705 63706 63707 63708 63710 done 192.168.201.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg142-client.virtnet /mnt/lustre2 (opts:) 192.168.201.142@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg142-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 16:30:47 (1790973047) Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2416 1268888 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2248 1269056 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18740 3588280 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24384 3582636 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43124 7170916 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2416 1268888 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2248 1269056 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18740 3588280 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24384 3582636 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43124 7170916 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0667628 s, 31.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.304868 s, 6.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x9d8a2190 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg142-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 16:31:07 (1790973067) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.153622 s, 13.7 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 16:31:17 (1790973077) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x23b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x36d:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x36d:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x36e:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x34e:0x0] } PASS 203 (11s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 16:31:28 (1790973088) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.127552 s, 16.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x36f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x34f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x350:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 16:31:36 (1790973096) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113748 s, 18.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x370:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x371:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x351:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x370:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x371:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x351:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x352:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 16:31:45 (1790973105) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.101311 s, 20.7 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x66:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x65:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x66:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x65:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x66:0x0] } PASS 204c (34s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 16:32:19 (1790973139) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg142-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677482 s, 15.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x67:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x68:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x67:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x67:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x68:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x67:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x68:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 16:33:13 (1790973193) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg142-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x372:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x373:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.163892 s, 12.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x372:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x373:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x353:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x372:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x373:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x353:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (50s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 16:34:03 (1790973243) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg142-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg142-server: Pool lustre.test_204f created Adding targets to pool oleg142-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg142-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0770883 s, 27.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x374:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x354:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x374:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x354:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg142-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg142-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg142-server: Pool lustre.test_204f destroyed PASS 204f (80s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:35:23 (1790973323) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:35:32 (1790973332) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:35:39 (1790973339) oleg142-server: Pool lustre.test_206 created Adding targets to pool oleg142-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg142-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg142-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg142-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg142-server: Pool lustre.test_206 destroyed PASS 206 (31s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:36:10 (1790973370) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109998 s, 9.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0690297 s, 15.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x88aaf2ed Mirror 3: 0x88aaf2ed PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:36:21 (1790973381) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:36:26 (1790973386) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:36:30 (1790973390) Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0503949 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x11c0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1ca0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1ca0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1ca0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xe90000 Read this (4 bytes): BBBB 192.168.201.142@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg142-client.virtnet /mnt/lustre2 (opts:) PASS 209a (40s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:37:10 (1790973430) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0118873 s, 689 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:37:18 (1790973438) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0365161 s, 28.7 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x25c lmm_fid: [0x200000405:0x25c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x25c lmm_fid: [0x200000405:0x25c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x35e:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611740 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-10-02 16:37:19.000000000 -0400 Modify: 2026-10-02 16:37:19.000000000 -0400 Change: 2026-10-02 16:37:21.000000000 -0400 Birth: 2026-10-02 16:37:19.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:37:27 (1790973447) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481172 s, 21.8 MB/s oleg142-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg142-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg142-server: O/280000401/d31/37f: File not found by ext2_lookup PASS 210b (26s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:37:53 (1790973473) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.311235 s, 132 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0203181 s, 202 kB/s size after second write -rw-r--r-- 1 root root 4096 Oct 2 16:37 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x261:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Oct 2 16:37 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:38:03 (1790973483) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:38:13 (1790973493) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1935 sec =========== 16:38:23 (1790973503) === sanity-flr: start cleanup 16:38:25 (1790973505) === === sanity-flr: finish cleanup 16:38:36 (1790973516) ===