-----============= acceptance-small: sanity-flr ============----- Fri Sep 11 04:35:12 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg148-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 04:35:30 (1789115730) === oleg148-client.virtnet: executing check_config_client /mnt/lustre oleg148-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg148-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9a764638d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a764638d000.idle_timeout=debug disable quota as required oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 Setting lod.lustre-MDT0000-mdtlov.enable_compr_rotational from 0 to 1 Waiting 90s for '1' Updated after 2s: want '1' got '1' Setting lod.lustre-MDT0001-mdtlov.enable_compr_rotational from 0 to 1 Waiting 90s for '1' Updated after 2s: want '1' got '1' === sanity-flr: finish setup 04:36:22 (1789115782) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8212] == sanity-flr test 0a: lfs mirror create with -N option == 04:36:25 (1789115785) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 04:36:41 (1789115801) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 04:36:44 (1789115804) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 04:36:47 (1789115807) PASS 0d (19s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 04:37:07 (1789115827) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 04:37:10 (1789115830) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 04:37:13 (1789115833) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9a764638d000.stats=clear osc.lustre-OST0001-osc-ffff9a764638d000.stats=clear PASS 0g (7s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 04:37:21 (1789115841) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (9s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 04:37:30 (1789115850) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 04:37:38 (1789115858) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 04:37:42 (1789115862) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 04:37:49 (1789115869) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (12s) == sanity-flr test 2: create components from existing files ========================================================== 04:38:01 (1789115881) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 04:38:09 (1789115889) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 04:38:17 (1789115897) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 04:38:24 (1789115904) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 04:38:35 (1789115915) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 04:38:44 (1789115924) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.0956 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0938511 s, 11.2 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 04:38:54 (1789115934) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.666609 s, 18.9 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 04:39:01 (1789115941) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 04:39:12 (1789115952) Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a764638d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a764638d000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a764638d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a764638d000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (75s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 04:40:27 (1789116027) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.943957 s, 12.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a764638d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a764638d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 04:41:13 (1789116073) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a764638d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a764638d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a764638d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a764638d000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (78s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 04:42:31 (1789116151) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.3485 s, 22.3 MB/s -rw-r--r-- 1 root root 52428800 Sep 11 04:42 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.761s user 0m0.001s sys 0m0.172s Stopping /mnt/lustre-ost1 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a764638d000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a764638d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.183s user 0m0.001s sys 0m0.167s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg148-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg148-server oleg148-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a764638d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a764638d000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.994s user 0m0.000s sys 0m0.178s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg148-server: oleg148-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg148-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg148-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a764638d000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a764638d000.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 04:44:09 (1789116249) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 04:44:13 (1789116253) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 04:44:16 (1789116256) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 04:44:19 (1789116259) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 04:44:26 (1789116266) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.426565 s, 29.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 04:44:37 (1789116277) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386682 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281007 s, 37.3 MB/s PASS 36b (6s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 04:44:44 (1789116284) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 04:44:52 (1789116292) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00122151 s, 838 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0013418 s, 763 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00193644 s, 529 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00271521 s, 377 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 04:45:04 (1789116304) create test files with size 15748226 .. cf7db54a610341f57d36183174408baf - c9cd664d68c6f5598f1eb04aae9bc866 - b17424d36d6433b7c86599f743417fc6 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre PASS 37 (23s) == sanity-flr test 38: resync ============================ 04:45:28 (1789116328) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 04:45:41 (1789116341) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 04:45:50 (1789116350) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126632 s, 8.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0892691 s, 11.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xbe:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (13s) == sanity-flr test 41: lfs mirror resync check =========== 04:46:03 (1789116363) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0583545 s, 35.9 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.366138 s, 11.5 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.149809 s, 14.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (17s) == sanity-flr test 42: lfs mirror verify ================= 04:46:20 (1789116380) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 04:46:23 (1789116383) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 04:46:27 (1789116387) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292451 s, 35.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 43c: read prefer uncompressed mirror == 04:46:34 (1789116394) llite.lustre-ffff9a7649925000.enable_compression=1 striped dir -i1 -c2 -H crush2 /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (6s) == sanity-flr test 43d: write prefers uncompressed mirror ========================================================== 04:46:41 (1789116401) llite.lustre-ffff9a7649925000.enable_compression=1 striped dir -i1 -c2 -H all_char /mnt/lustre/d43d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020001 s, 52.4 MB/s **verify components 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234123 s, 44.8 MB/s **verify components PASS 43d (7s) == sanity-flr test 44a: lfs mirror split check =========== 04:46:48 (1789116408) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.191735 s, 16.4 MB/s PASS 44a (8s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 04:46:56 (1789116416) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc8:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc8:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (8s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 04:47:05 (1789116425) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.355186 s, 29.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0xd:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (11s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 04:47:16 (1789116436) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9a76485f0800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.36957 s, 28.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg148-client.virtnet /mnt/lustre (opts:) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9a76505b9000.xattr_cache=1 PASS 44e (18s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 04:47:34 (1789116454) striped dir -i1 -c2 -H all_char /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (46s) == sanity-flr test 46: Verify setstripe --copy option ==== 04:48:20 (1789116500) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (38s) == sanity-flr test 47: Verify mirror obj alloc =========== 04:48:58 (1789116538) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 04:49:02 (1789116542) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.26324 s, 12.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.171077 s, 18.4 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x6cdc7f8a (nosync,stale) Mirror 2: 0x6cdc7f8a (nosync,stale) ** verify mirror 2 doesn't change original checksum: 6ba9db2ed892b14f8af772559c04d3e3 - mirror 1 checksum: 010f7a18e282778840fbfb1d175a2366 - mirror 2 checksum: 6ba9db2ed892b14f8af772559c04d3e3 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T04:49:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-11T04:49:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6ba9db2ed892b14f8af772559c04d3e3 - mirror 1 checksum: 010f7a18e282778840fbfb1d175a2366 - mirror 2 checksum: 010f7a18e282778840fbfb1d175a2366 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T04:49:04-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-11T04:49:04-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xa lmm_fid: [0x200000405:0xa:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 04:49:13 (1789116553) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329303 s, 31.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (6s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 04:49:20 (1789116560) Failing mds1 on oleg148-server Stopping /mnt/lustre-mds1 (opts:) on oleg148-server 04:49:31 (1789116571) shut down facet: mds1 facet_host: oleg148-server facet_failover_host: oleg148-server Failover mds1 to oleg148-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 04:50:08 (1789116608) targets are mounted 04:50:08 (1789116608) facet_failover done oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x10:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291912 s, 35.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x10 lmm_fid: [0x200000405:0x10:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd4:0x0] } PASS 50A (66s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 04:50:25 (1789116625) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0283889 s, 721 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0150068 s, 1.4 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (9s) == sanity-flr test 50b: mirror rsync handles sparseness == 04:50:35 (1789116635) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0393901 s, 520 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0276234 s, 741 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 04:50:45 (1789116645) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00844798 s, 1.9 MB/s PASS 50c (8s) == sanity-flr test 50d: mirror rsync keep holes ========== 04:50:53 (1789116653) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0861275 s, 238 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (9s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 04:51:02 (1789116662) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00812763 s, 504 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0388726 s, 105 kB/s PASS 60a (7s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 04:51:09 (1789116669) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x21:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 04:51:29 (1789116689) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 04:52:02 (1789116722) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4b:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 04:52:26 (1789116746) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 04:52:52 (1789116772) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 04:53:00 (1789116780) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70.sanity-flr mirror create pid 59100 mirror split pid 59102 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 59100 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5142: 59102 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 04:54:08 (1789116848) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 6852 skipping zero size read skipping zero size read 000005 1789116852.683404 write 0x0262ad thru 0x026460 (0x1b4 bytes) zero_range to largest ever: 0x1dc02 000006 1789116852.698288 trunc from 0x00e3ea to 0x01dc01 (0xf818 bytes) 000007 1789116852.715581 read 0x007b8a thru 0x0147af (0xcc26 bytes) 000008 1789116852.742917 trunc from 0x0207cd to 0x023f87 (0x37bb bytes) fallocating to largest ever: 0x40000 000009 1789116852.766706 fallocate 0x038a3b thru 0x03ffff (0x75c5 bytes) 000010 1789116852.777730 read 0x00871d thru 0x00d9c0 (0x52a4 bytes) 000011 1789116852.779213 mapread 0x0016a2 thru 0x003105 (0x1a64 bytes) 000014 1789116852.805854 punch from 0x01d3a1 to 0x026460 (0x90c0 bytes) 000015 1789116852.833207 trunc from 0x026461 to 0x0235c5 (0x2e9b bytes) 000016 1789116852.859941 punch from 0x003df2 to 0x00d44d (0x965c bytes) 000017 1789116852.866277 mapwrite 0x013db1 thru 0x018730 (0x4980 bytes) 000018 1789116852.885635 read 0x00464f thru 0x011ad8 (0xd48a bytes) 000019 1789116852.897032 mapwrite 0x000904 thru 0x00cbc8 (0xc2c5 bytes) 000020 1789116852.914695 mapwrite 0x025b3b thru 0x02dd56 (0x821c bytes) 000022 1789116852.970376 mapwrite 0x01a96f thru 0x028860 (0xdef2 bytes) 000023 1789116853.008169 trunc from 0x02dd57 to 0x0243dd (0x9979 bytes) 000024 1789116853.030276 read 0x0052f7 thru 0x007d9e (0x2aa8 bytes) 000026 1789116853.034949 punch from 0x00af86 to 0x0187f0 (0xd86b bytes) 000027 1789116853.041207 write 0x013708 thru 0x016bd8 (0x34d1 bytes) 000029 1789116853.056404 mapread 0x021d8f thru 0x0234f1 (0x1763 bytes) 000030 1789116853.059095 write 0x010684 thru 0x015dbb (0x5738 bytes) 000031 1789116853.061204 fallocate 0x01f516 thru 0x029d2e (0xa819 bytes) 000032 1789116853.073142 MIRROR_EXTEND to 2 mirrors 000033 1789116853.922233 fallocate 0x01022f thru 0x01e52f (0xe301 bytes) 000034 1789116853.962377 fallocate 0x03fe81 thru 0x03ffff (0x17f bytes) 000035 1789116853.974186 punch from 0x0149f9 to 0x0213c8 (0xc9d0 bytes) 000036 1789116853.983204 read 0x023b4f thru 0x027de8 (0x429a bytes) 000037 1789116853.998983 punch from 0x0393fd to 0x03c37f (0x2f83 bytes) 000038 1789116854.007894 mapread 0x0208a1 thru 0x02d6e0 (0xce40 bytes) 000039 1789116854.057341 mapwrite 0x0285f2 thru 0x02c2c4 (0x3cd3 bytes) 000040 1789116854.076668 read 0x0399d0 thru 0x03ffff (0x6630 bytes) 000041 1789116854.090293 read 0x0022b6 thru 0x00d8c5 (0xb610 bytes) 000042 1789116854.091057 trunc from 0x040000 to 0x029eb2 (0x1614d bytes) 000043 1789116854.194602 MIRROR_RESYNC 2 mirrors 000044 1789116854.871607 write 0x03cb7f thru 0x03ffff (0x3481 bytes) 000045 1789116854.907032 mapwrite 0x00cfad thru 0x013320 (0x6374 bytes) 000046 1789116854.943009 fallocate 0x037053 thru 0x03ffff (0x8fad bytes) 000047 1789116854.947773 punch from 0x00a616 to 0x00aeba (0x8a5 bytes) 000049 1789116854.953618 trunc from 0x03b5a9 to 0x03ffff (0x4a57 bytes) 000050 1789116854.958592 mapwrite 0x006d8c thru 0x00ed40 (0x7fb5 bytes) 000051 1789116854.986587 write 0x0032c8 thru 0x005c21 (0x295a bytes) 000053 1789116854.988640 write 0x035112 thru 0x036426 (0x1315 bytes) 000055 1789116854.998369 trunc from 0x040000 to 0x028ad9 (0x17526 bytes) 000056 1789116855.050245 mapread 0x019298 thru 0x02424a (0xafb3 bytes) 000057 1789116855.082603 trunc from 0x028ada to 0x0249ab (0x412e bytes) 000058 1789116855.147971 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x249ac): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x249ac): Mirror 2: 0x4a735701 (stale) CRC-32 checksum value for chunk [0, 0x249ac): Mirror 1: 0xf0fd491b Mirror 3: 0xf0fd491b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x249ac: skipped 000059 1789116856.463945 mapread 0x01f2ab thru 0x0216d6 (0x242c bytes) 000060 1789116856.481894 write 0x02b13b thru 0x036033 (0xaef9 bytes) 000061 1789116856.543295 mapread 0x01d10a thru 0x0235cb (0x64c2 bytes) 000062 1789116856.562824 read 0x001a2e thru 0x00592f (0x3f02 bytes) 000063 1789116856.574473 write 0x03a68e thru 0x03fe93 (0x5806 bytes) 000064 1789116856.577286 mapread 0x03571c thru 0x03cece (0x77b3 bytes) 000065 1789116856.585638 trunc from 0x03fe94 to 0x024baa (0x1b2e9 bytes) 000066 1789116856.610219 mapwrite 0x015496 thru 0x024560 (0xf0cb bytes) 000067 1789116856.647865 mapwrite 0x013543 thru 0x016d29 (0x37e7 bytes) 000068 1789116856.673345 punch from 0x0120b5 to 0x012ae2 (0xa2e bytes) 000069 1789116856.683497 mapread 0x01240c thru 0x0178a2 (0x5497 bytes) 000070 1789116856.702580 punch from 0x00dbf2 to 0x01cb20 (0xef2f bytes) 000071 1789116856.718675 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24bab): Mirror 2: 0x8b588915 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24bab): Mirror 2: 0x8b588915 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24bab: skipped 000072 1789116857.153867 punch from 0x00e13e to 0x0156c8 (0x758b bytes) 000074 1789116857.160226 mapread 0x003103 thru 0x00e9a8 (0xb8a6 bytes) 000075 1789116857.182354 mapwrite 0x036c3d thru 0x0390ab (0x246f bytes) 000076 1789116857.229296 trunc from 0x0390ac to 0x031d09 (0x73a2 bytes) 000077 1789116857.246990 trunc from 0x031d0a to 0x002ba6 (0x2f163 bytes) 000078 1789116857.264336 trunc from 0x002ba7 to 0x021332 (0x1e78c bytes) 000079 1789116857.280853 mapread 0x00e96a thru 0x00f91d (0xfb4 bytes) 000080 1789116857.303180 fallocate 0x02e256 thru 0x03360e (0x53b9 bytes) 000081 1789116857.309558 read 0x00d29a thru 0x01300d (0x5d74 bytes) 000082 1789116857.313972 write 0x029667 thru 0x0323be (0x8d58 bytes) 000083 1789116857.355116 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000084 1789116857.577671 punch from 0x0199f9 to 0x020d9f (0x73a7 bytes) 000085 1789116857.583319 read 0x0055dc thru 0x0140b7 (0xeadc bytes) 000086 1789116857.603473 mapwrite 0x0064cb thru 0x01285e (0xc394 bytes) 000087 1789116857.632907 fallocate 0x005787 thru 0x00ba9d (0x6317 bytes) 000089 1789116857.640468 read 0x030b78 thru 0x03360e (0x2a97 bytes) 000090 1789116857.656491 trunc from 0x01cfab to 0x0265c5 (0x961b bytes) 000091 1789116857.664388 mapwrite 0x0000e1 thru 0x00efb4 (0xeed4 bytes) 000092 1789116857.693541 read 0x0022ce thru 0x00716a (0x4e9d bytes) 000093 1789116857.694321 punch from 0x0171c8 to 0x026ba0 (0xf9d9 bytes) 000094 1789116857.700303 trunc from 0x006b6c to 0x009128 (0x25bd bytes) 000095 1789116857.705613 write 0x02be2f thru 0x031c59 (0x5e2b bytes) 000096 1789116857.710843 mapread 0x029820 thru 0x03360e (0x9def bytes) 000098 1789116857.715165 fallocate 0x030cf1 thru 0x031fbb (0x12cb bytes) 000099 1789116857.720652 punch from 0x01c15e to 0x01e956 (0x27f9 bytes) 000100 1789116857.725652 trunc from 0x03360f to 0x00bdcd (0x27841 bytes) 000101 1789116857.744342 trunc from 0x005c25 to 0x00bdcd (0x61a9 bytes) 000103 1789116857.749899 write 0x02f5f2 thru 0x033d50 (0x475f bytes) 000104 1789116857.752279 read 0x005cb7 thru 0x00d142 (0x748c bytes) 000105 1789116857.769276 read 0x005e87 thru 0x013662 (0xd7dc bytes) 000106 1789116857.771179 mapread 0x01a23d thru 0x024d6b (0xab2f bytes) 000108 1789116857.774944 write 0x027497 thru 0x035582 (0xe0ec bytes) 000109 1789116857.777495 punch from 0x02a357 to 0x035582 (0xb22c bytes) 000110 1789116857.795290 mapwrite 0x01761a thru 0x01e20b (0x6bf2 bytes) 000111 1789116857.827939 mapread 0x024cf3 thru 0x0303ed (0xb6fb bytes) 000112 1789116857.841476 read 0x01c10f thru 0x027809 (0xb6fb bytes) 000113 1789116857.843279 trunc from 0x01b1e7 to 0x01f366 (0x4180 bytes) 000114 1789116857.850332 fallocate 0x01842f thru 0x01c1d4 (0x3da6 bytes) 000116 1789116857.856358 mapread 0x007794 thru 0x0154c8 (0xdd35 bytes) 000118 1789116857.859932 write 0x0397b0 thru 0x03ffff (0x6850 bytes) 000122 1789116857.871749 read 0x032e63 thru 0x03ffff (0xd19d bytes) 000124 1789116857.888086 mapread 0x001382 thru 0x001cb6 (0x935 bytes) 000125 1789116857.888920 read 0x00fa96 thru 0x01c95b (0xcec6 bytes) 000126 1789116857.889534 mapread 0x01ad23 thru 0x026cd3 (0xbfb1 bytes) 000127 1789116857.890387 mapwrite 0x025a9a thru 0x0308b0 (0xae17 bytes) 000128 1789116857.944605 read 0x0191df thru 0x01a564 (0x1386 bytes) 000129 1789116857.949883 read 0x009b07 thru 0x013e6f (0xa369 bytes) 000130 1789116857.952715 read 0x02eb61 thru 0x0320a4 (0x3544 bytes) 000131 1789116857.959598 punch from 0x001453 to 0x00b608 (0xa1b6 bytes) 000132 1789116857.973991 mapwrite 0x0066d8 thru 0x007d31 (0x165a bytes) 000133 1789116858.041782 fallocate 0x0068c6 thru 0x0076f3 (0xe2e bytes) 000134 1789116858.065377 mapread 0x004e4f thru 0x0068f5 (0x1aa7 bytes) 000135 1789116858.083741 trunc from 0x040000 to 0x02dc5c (0x123a3 bytes) 000136 1789116858.118373 trunc from 0x00f1e5 to 0x00fc7a (0xa96 bytes) 000137 1789116858.133897 mapread 0x0266b5 thru 0x02cbf1 (0x653d bytes) 000138 1789116858.155993 MIRROR_EXTEND to 2 mirrors 000139 1789116859.203917 punch from 0x0212ce to 0x02d9e1 (0xc714 bytes) 000140 1789116859.270986 mapread 0x022ba3 thru 0x02dc5c (0xb0ba bytes) 000141 1789116859.323902 trunc from 0x02dc5d to 0x02298f (0xb2cd bytes) 000142 1789116859.387173 trunc from 0x022990 to 0x00e3e8 (0x145a7 bytes) 000143 1789116859.414222 write 0x035e23 thru 0x039187 (0x3365 bytes) 000144 1789116859.415653 trunc from 0x00e0eb to 0x014ad3 (0x69e9 bytes) 000145 1789116859.420822 trunc from 0x01a16e to 0x020c6e (0x6b01 bytes) 000146 1789116859.425497 fallocate 0x03318e thru 0x03770a (0x457d bytes) 000147 1789116859.430237 mapread 0x028ddf thru 0x03328d (0xa4af bytes) 000148 1789116859.441034 fallocate 0x0173d2 thru 0x0216cf (0xa2fe bytes) 000149 1789116859.445328 mapwrite 0x01dd88 thru 0x02520b (0x7484 bytes) 000150 1789116859.473738 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000151 1789116859.660815 mapread 0x00ef71 thru 0x01a7d1 (0xb861 bytes) 000152 1789116859.731258 punch from 0x00e4d5 to 0x019668 (0xb194 bytes) 000153 1789116859.747435 mapread 0x01f7ac thru 0x0232e1 (0x3b36 bytes) 000155 1789116859.782580 read 0x00412b thru 0x004225 (0xfb bytes) 000156 1789116859.794581 read 0x02d0c6 thru 0x036420 (0x935b bytes) 000157 1789116859.817390 punch from 0x0272de to 0x02f99f (0x86c2 bytes) 000158 1789116859.828270 trunc from 0x039188 to 0x0226db (0x16aac bytes) 000159 1789116859.861374 mapwrite 0x0027c7 thru 0x002a52 (0x28c bytes) 000160 1789116859.915382 mapwrite 0x019a69 thru 0x01f6fc (0x5c94 bytes) 000161 1789116859.967376 trunc from 0x00590c to 0x01310b (0xd800 bytes) 000163 1789116859.978964 mapread 0x0081eb thru 0x00f5dd (0x73f3 bytes) 000165 1789116860.072064 punch from 0x0008db to 0x000974 (0x9a bytes) 000166 1789116860.084314 mapwrite 0x030466 thru 0x03e5db (0xe176 bytes) 000167 1789116860.309634 read 0x015b30 thru 0x016094 (0x565 bytes) 000168 1789116860.313629 mapread 0x01f80b thru 0x02efc9 (0xf7bf bytes) 000169 1789116860.337935 read 0x02b669 thru 0x036d1a (0xb6b2 bytes) 000170 1789116860.338559 read 0x00bda9 thru 0x00d1ef (0x1447 bytes) 000171 1789116860.339047 trunc from 0x02167e to 0x031508 (0xfe8b bytes) 000172 1789116860.353571 fallocate 0x033114 thru 0x0393ca (0x62b7 bytes) 000174 1789116860.360947 write 0x0181e9 thru 0x023ecf (0xbce7 bytes) 000175 1789116860.380371 mapread 0x03be44 thru 0x03e5db (0x2798 bytes) 000176 1789116860.396410 punch from 0x017ab1 to 0x01d2a4 (0x57f4 bytes) 000177 1789116860.422453 punch from 0x03d0d8 to 0x03e5db (0x1504 bytes) 000178 1789116860.451022 mapwrite 0x02be25 thru 0x02c9b7 (0xb93 bytes) 000179 1789116860.486122 trunc from 0x0114c9 to 0x01c081 (0xabb9 bytes) 000180 1789116860.493672 trunc from 0x021b7d to 0x025f13 (0x4397 bytes) 000181 1789116860.504245 mapwrite 0x010892 thru 0x01a5f4 (0x9d63 bytes) 000182 1789116860.598287 punch from 0x028a32 to 0x030ec5 (0x8494 bytes) 000183 1789116860.607203 write 0x038ea2 thru 0x03ffff (0x715e bytes) 000184 1789116860.609145 mapwrite 0x024513 thru 0x0305ea (0xc0d8 bytes) 000186 1789116860.703066 trunc from 0x040000 to 0x02b418 (0x14be7 bytes) 000187 1789116860.733455 write 0x03405d thru 0x0396ac (0x5650 bytes) 000188 1789116860.735201 trunc from 0x02ecbe to 0x0396ac (0xa9ef bytes) 000189 1789116860.772060 trunc from 0x0396ad to 0x033e5c (0x5850 bytes) 000191 1789116860.805115 trunc from 0x010e89 to 0x01997d (0x8af5 bytes) 000192 1789116860.826188 punch from 0x014aac to 0x01cb23 (0x8078 bytes) 000193 1789116860.845375 mapread 0x01670c thru 0x01f633 (0x8f28 bytes) 000194 1789116860.909875 mapwrite 0x0112c8 thru 0x01c332 (0xb06b bytes) 000195 1789116861.006783 mapread 0x015b7b thru 0x01ef19 (0x939f bytes) 000196 1789116861.007691 trunc from 0x033e5d to 0x035716 (0x18ba bytes) 000197 1789116861.036567 mapwrite 0x01cfde thru 0x02634e (0x9371 bytes) 000198 1789116861.081236 punch from 0x025b5a to 0x025e88 (0x32f bytes) 000199 1789116861.085744 mapread 0x023858 thru 0x023879 (0x22 bytes) 000200 1789116861.089109 fallocate 0x01dafc thru 0x024e69 (0x736e bytes) 000202 1789116861.094795 fallocate 0x01b2c7 thru 0x022d47 (0x7a81 bytes) 000203 1789116861.099253 trunc from 0x01fd0d to 0x024edd (0x51d1 bytes) 000204 1789116861.104771 mapwrite 0x00fbb4 thru 0x0180dd (0x852a bytes) 000205 1789116861.127402 punch from 0x01d571 to 0x02bc46 (0xe6d6 bytes) 000206 1789116861.132442 trunc from 0x027370 to 0x035716 (0xe3a7 bytes) 000207 1789116861.142149 mapread 0x02b16d thru 0x0324c5 (0x7359 bytes) 000208 1789116861.214832 write 0x0064bb thru 0x00dc27 (0x776d bytes) 000209 1789116861.216747 trunc from 0x035717 to 0x031dfa (0x391c bytes) 000210 1789116861.249733 mapread 0x02c5e0 thru 0x031dfa (0x581b bytes) 000211 1789116861.250880 mapwrite 0x03e6db thru 0x03ffff (0x1925 bytes) 000212 1789116861.311389 trunc from 0x040000 to 0x016185 (0x29e7a bytes) 000213 1789116861.342694 fallocate 0x023aa4 thru 0x02c5c1 (0x8b1e bytes) 000214 1789116861.361823 trunc from 0x011641 to 0x016185 (0x4b45 bytes) 000215 1789116861.367672 punch from 0x000760 to 0x0092d9 (0x8b7a bytes) 000217 1789116861.378209 trunc from 0x016186 to 0x003e16 (0x1236f bytes) 000218 1789116861.430367 fallocate 0x006139 thru 0x00ce8e (0x6d56 bytes) 000220 1789116861.448028 trunc from 0x003e17 to 0x02ed0d (0x2aef7 bytes) 000221 1789116861.470181 mapread 0x01efb4 thru 0x02dab2 (0xeaff bytes) 000222 1789116861.638760 trunc from 0x0003c5 to 0x0025dd (0x2219 bytes) 000223 1789116861.652194 trunc from 0x02ed0e to 0x0018bf (0x2d44e bytes) 000225 1789116861.688867 mapread 0x0002a0 thru 0x000cfc (0xa5d bytes) 000226 1789116861.697691 trunc from 0x000d66 to 0x0018bf (0xb5a bytes) 000227 1789116861.706152 read 0x0014ec thru 0x0018bf (0x3d4 bytes) 000228 1789116861.716422 trunc from 0x00070e to 0x0018bf (0x11b2 bytes) 000229 1789116861.729497 mapread 0x000de6 thru 0x0018bf (0xada bytes) 000230 1789116861.737455 mapread 0x001126 thru 0x0018bf (0x79a bytes) 000231 1789116861.739250 punch from 0x001400 to 0x0018bf (0x4c0 bytes) 000232 1789116861.744000 fallocate 0x025178 thru 0x032020 (0xcea9 bytes) 000233 1789116861.749257 read 0x0007ee thru 0x0018bf (0x10d2 bytes) 000234 1789116861.757348 mapwrite 0x000b0c thru 0x00e002 (0xd4f7 bytes) 000235 1789116861.809036 read 0x000c17 thru 0x00e002 (0xd3ec bytes) 000236 1789116861.814835 MIRROR_EXTEND to 2 mirrors 000237 1789116862.530464 fallocate 0x00786c thru 0x00c0bb (0x4850 bytes) 000238 1789116862.555639 mapwrite 0x03020a thru 0x03a159 (0x9f50 bytes) 000239 1789116862.605922 fallocate 0x0186cb thru 0x0186e8 (0x1e bytes) 000240 1789116862.610745 mapwrite 0x03ebd0 thru 0x03ffff (0x1430 bytes) 000241 1789116862.639167 trunc from 0x01b926 to 0x02a957 (0xf032 bytes) 000242 1789116862.643487 mapwrite 0x020a33 thru 0x027a89 (0x7057 bytes) 000243 1789116862.667022 mapwrite 0x014195 thru 0x01da1b (0x9887 bytes) 000244 1789116862.691164 read 0x03e254 thru 0x03ffff (0x1dac bytes) 000245 1789116862.692126 write 0x01e3f8 thru 0x01eb51 (0x75a bytes) 000246 1789116862.693462 fallocate 0x002f29 thru 0x003f1e (0xff6 bytes) 000247 1789116862.698423 trunc from 0x02e689 to 0x036888 (0x8200 bytes) 000248 1789116862.707787 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x883be2ce (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3b2b597d Mirror 3: 0x3b2b597d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000249 1789116863.853527 trunc from 0x027a1b to 0x033c81 (0xc267 bytes) 000250 1789116863.915448 punch from 0x0302b6 to 0x03d751 (0xd49c bytes) 000251 1789116863.920882 fallocate 0x02735d thru 0x0273e2 (0x86 bytes) 000252 1789116863.929096 trunc from 0x040000 to 0x0398b1 (0x674e bytes) 000253 1789116863.953822 mapread 0x0356da thru 0x0398b1 (0x41d8 bytes) 000254 1789116863.978104 mapwrite 0x00d73e thru 0x018d9a (0xb65d bytes) 000255 1789116864.042557 trunc from 0x0398b2 to 0x038783 (0x112e bytes) 000256 1789116864.055270 write 0x0280c9 thru 0x036e03 (0xed3b bytes) 000257 1789116864.061782 mapread 0x037883 thru 0x038783 (0xf01 bytes) 000258 1789116864.066994 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38784): Mirror 3: 0x3ff349c6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38784): Mirror 3: 0x3ff349c6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38784: skipped 000259 1789116864.315633 punch from 0x00eeaa to 0x015404 (0x655b bytes) 000260 1789116864.320124 write 0x01c4d7 thru 0x02a25b (0xdd85 bytes) 000262 1789116864.337031 punch from 0x006510 to 0x0136b5 (0xd1a6 bytes) 000263 1789116864.341871 write 0x0365bd thru 0x03d530 (0x6f74 bytes) 000264 1789116864.349939 punch from 0x0234d9 to 0x0331e6 (0xfd0e bytes) 000265 1789116864.370004 mapwrite 0x00b30e thru 0x01383d (0x8530 bytes) 000266 1789116864.408993 fallocate 0x033db4 thru 0x03892b (0x4b78 bytes) 000268 1789116864.414254 write 0x038c7e thru 0x038d44 (0xc7 bytes) 000269 1789116864.419321 MIRROR_RESYNC 2 mirrors 000270 1789116865.107931 trunc from 0x03d531 to 0x03d98b (0x45b bytes) 000271 1789116865.166848 fallocate 0x03f47d thru 0x03ffff (0xb83 bytes) 000272 1789116865.173467 read 0x0135b0 thru 0x017db2 (0x4803 bytes) 000273 1789116865.181846 read 0x001ca5 thru 0x007512 (0x586e bytes) 000274 1789116865.198848 MIRROR_RESYNC 2 mirrors 000275 1789116865.839410 trunc from 0x040000 to 0x01e99b (0x21664 bytes) 000276 1789116865.899148 mapwrite 0x034e8d thru 0x03ffff (0xb173 bytes) 000277 1789116865.957093 write 0x008f8c thru 0x010864 (0x78d9 bytes) 000278 1789116865.973204 mapwrite 0x003cc6 thru 0x00fe52 (0xc18d bytes) 000279 1789116866.009808 mapread 0x038571 thru 0x03ffff (0x7a8f bytes) 000280 1789116866.011446 mapwrite 0x00b5f7 thru 0x014129 (0x8b33 bytes) 000282 1789116866.029446 punch from 0x02539e to 0x02f985 (0xa5e8 bytes) 000283 1789116866.034117 read 0x023a62 thru 0x027916 (0x3eb5 bytes) 000285 1789116866.043440 punch from 0x03c958 to 0x03d593 (0xc3c bytes) 000286 1789116866.048039 trunc from 0x040000 to 0x03bfcc (0x4033 bytes) 000287 1789116866.061074 read 0x021646 thru 0x02a20a (0x8bc5 bytes) 000289 1789116866.071859 punch from 0x0131c4 to 0x0224ee (0xf32b bytes) 000290 1789116866.076937 fallocate 0x016331 thru 0x01c885 (0x6555 bytes) 000291 1789116866.081341 mapread 0x00ee6e thru 0x01db8a (0xed1d bytes) 000292 1789116866.128052 punch from 0x02d940 to 0x03bfcc (0xe68d bytes) 000293 1789116866.133956 trunc from 0x029b0a to 0x037fb8 (0xe4af bytes) 000294 1789116866.139088 trunc from 0x01b7e1 to 0x024f37 (0x9757 bytes) 000295 1789116866.148459 read 0x0111b2 thru 0x0154f6 (0x4345 bytes) 000296 1789116866.149160 mapwrite 0x0038e7 thru 0x004f99 (0x16b3 bytes) 000297 1789116866.163313 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bfcd): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bfcd): Mirror 3: 0x541bd404 (stale) CRC-32 checksum value for chunk [0, 0x3bfcd): Mirror 1: 0xa65d3567 Mirror 4: 0xa65d3567 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bfcd: skipped 000298 1789116866.812192 trunc from 0x03bfcd to 0x02fca7 (0xc325 bytes) 000299 1789116866.950010 read 0x00edab thru 0x01c481 (0xd6d7 bytes) 000300 1789116866.959291 mapread 0x02a0ef thru 0x02fca7 (0x5bb9 bytes) 000301 1789116866.992051 punch from 0x00b7b8 to 0x018f55 (0xd79e bytes) 000302 1789116867.000831 mapwrite 0x01ba63 thru 0x0266ed (0xac8b bytes) 000303 1789116867.040366 write 0x02f5e4 thru 0x030350 (0xd6d bytes) 000305 1789116867.041846 read 0x015319 thru 0x020032 (0xad1a bytes) 000306 1789116867.067399 fallocate 0x02c0bc thru 0x03123e (0x5183 bytes) 000307 1789116867.081305 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x30351): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30351): Mirror 3: 0x844f5cfa (stale) Mirror 4: 0xaaf529f4 (stale) CRC-32 checksum value for chunk [0, 0x30351): Mirror 1: 0xc8a6c28c Mirror 5: 0xc8a6c28c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30351: skipped 000310 1789116868.257162 write 0x01d187 thru 0x028e6d (0xbce7 bytes) 000311 1789116868.351026 trunc from 0x030351 to 0x015e02 (0x1a54e bytes) 000312 1789116868.486022 trunc from 0x00315b to 0x0041ad (0x1053 bytes) 000313 1789116868.493455 trunc from 0x00e17d to 0x010620 (0x24a4 bytes) 000314 1789116868.510312 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x15e03): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15e03): Mirror 3: 0x1afb8dbf (stale) Mirror 4: 0x8c2ce517 (stale) Mirror 5: 0x724444e5 (stale) CRC-32 checksum value for chunk [0, 0x15e03): Mirror 1: 0x93e0837c Mirror 6: 0x93e0837c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15e03: skipped 000315 1789116869.689975 punch from 0x00f1d9 to 0x015e02 (0x6c2a bytes) 000316 1789116869.732012 read 0x014a5b thru 0x015e02 (0x13a8 bytes) 000318 1789116869.738738 trunc from 0x014054 to 0x0157a6 (0x1753 bytes) 000319 1789116869.743370 mapwrite 0x000ea7 thru 0x00a137 (0x9291 bytes) 000320 1789116869.782635 mapread 0x009ad5 thru 0x00e2e5 (0x4811 bytes) 000321 1789116869.783602 read 0x010ef4 thru 0x015e02 (0x4f0f bytes) 000322 1789116869.791084 trunc from 0x01521f to 0x015e02 (0xbe4 bytes) 000323 1789116869.795318 read 0x00bb63 thru 0x00e4ad (0x294b bytes) 000324 1789116869.795922 mapwrite 0x01e068 thru 0x02787f (0x9818 bytes) 000325 1789116869.845327 write 0x0308cd thru 0x035ca3 (0x53d7 bytes) 000326 1789116869.852233 mapwrite 0x01291d thru 0x013c96 (0x137a bytes) 000327 1789116869.891359 punch from 0x00f701 to 0x013b4d (0x444d bytes) 000328 1789116869.903975 mapread 0x0126f5 thru 0x01ebeb (0xc4f7 bytes) 000329 1789116869.937660 read 0x0129e7 thru 0x017975 (0x4f8f bytes) 000330 1789116869.938271 mapread 0x00d6a2 thru 0x0137bb (0x611a bytes) 000331 1789116869.951138 mapread 0x011156 thru 0x016959 (0x5804 bytes) 000332 1789116869.953109 trunc from 0x030cb6 to 0x035ca3 (0x4fee bytes) 000333 1789116869.960382 read 0x02de0d thru 0x035ca3 (0x7e97 bytes) 000334 1789116869.971697 punch from 0x03516f to 0x035ca3 (0xb35 bytes) 000335 1789116869.977652 read 0x031463 thru 0x035ca3 (0x4841 bytes) 000336 1789116869.983283 mapwrite 0x008b55 thru 0x0148bd (0xbd69 bytes) 000337 1789116870.011064 trunc from 0x000b90 to 0x01039a (0xf80b bytes) 000338 1789116870.020000 trunc from 0x00704d to 0x008ff2 (0x1fa6 bytes) 000339 1789116870.038261 fallocate 0x00675d thru 0x00df8f (0x7833 bytes) 000340 1789116870.048593 mapread 0x013a7e thru 0x01aefb (0x747e bytes) 000341 1789116870.053039 mapread 0x00bd36 thru 0x00d123 (0x13ee bytes) 000342 1789116870.083360 trunc from 0x035ca4 to 0x01e0b9 (0x17bea bytes) 000343 1789116870.119680 write 0x035eb6 thru 0x03763f (0x178a bytes) 000344 1789116870.120846 fallocate 0x02e3d3 thru 0x03c3f1 (0xe01f bytes) 000345 1789116870.130143 read 0x011eff thru 0x021999 (0xfa9b bytes) 000346 1789116870.150972 trunc from 0x006994 to 0x0108bb (0x9f28 bytes) 000347 1789116870.159002 fallocate 0x008190 thru 0x016b0b (0xe97c bytes) 000348 1789116870.168376 write 0x0352bf thru 0x039e61 (0x4ba3 bytes) 000349 1789116870.169416 punch from 0x0205e5 to 0x02d874 (0xd290 bytes) 000350 1789116870.192057 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3c3f2): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c3f2): Mirror 3: 0xd77cb31b (stale) Mirror 4: 0x66d23cda (stale) Mirror 5: 0x177767d0 (stale) Mirror 6: 0x227d0075 (stale) CRC-32 checksum value for chunk [0, 0x3c3f2): Mirror 1: 0xc823f684 Mirror 7: 0xc823f684 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c3f2: skipped 000351 1789116871.508471 trunc from 0x03c3f2 to 0x030cf2 (0xb6ff bytes) 000353 1789116871.576193 trunc from 0x030cf3 to 0x01b0f6 (0x15bfc bytes) 000354 1789116871.598645 punch from 0x000363 to 0x00c20a (0xbea8 bytes) 000356 1789116871.603867 punch from 0x0148b7 to 0x01b0f6 (0x6840 bytes) 000357 1789116871.611619 trunc from 0x011b0a to 0x01b0f6 (0x95ed bytes) 000358 1789116871.629279 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b0f7): Mirror 4: 0xeb911cfb (stale) Mirror 5: 0xeaa70d21 (stale) Mirror 6: 0x41fc7bc8 (stale) Mirror 7: 0x767b47ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b0f7): Mirror 4: 0xeb911cfb (stale) Mirror 5: 0xeaa70d21 (stale) Mirror 6: 0x41fc7bc8 (stale) Mirror 7: 0x767b47ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b0f7: skipped 000359 1789116872.024089 trunc from 0x00d2fb to 0x01b0f6 (0xddfc bytes) 000360 1789116872.035841 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1b0f7): Mirror 5: 0xeaa70d21 (stale) Mirror 6: 0x41fc7bc8 (stale) Mirror 7: 0x767b47ac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b0f7): Mirror 5: 0xeaa70d21 (stale) Mirror 6: 0x41fc7bc8 (stale) Mirror 7: 0x767b47ac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b0f7: skipped 000361 1789116872.505177 punch from 0x000a03 to 0x000b40 (0x13e bytes) 000362 1789116872.511343 trunc from 0x01b0f7 to 0x034d79 (0x19c83 bytes) 000363 1789116872.533418 fallocate 0x0004ae thru 0x005407 (0x4f5a bytes) 000364 1789116872.540115 trunc from 0x034d7a to 0x009b2f (0x2b24a bytes) 000365 1789116872.561325 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x9b30): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9b30): Mirror 5: 0x6663ca0a (stale) Mirror 6: 0xb667d1bb (stale) Mirror 7: 0xe14d9ec (stale) CRC-32 checksum value for chunk [0, 0x9b30): Mirror 1: 0xd50142c4 Mirror 8: 0xd50142c4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9b30: skipped 000366 1789116874.020243 write 0x02a37c thru 0x0391c0 (0xee45 bytes) 000367 1789116874.103696 mapread 0x016912 thru 0x01a3f3 (0x3ae2 bytes) 000369 1789116874.134268 mapwrite 0x001219 thru 0x00dfd3 (0xcdbb bytes) 000370 1789116874.176828 mapwrite 0x002903 thru 0x00df05 (0xb603 bytes) 000371 1789116874.221323 trunc from 0x0122e1 to 0x01d144 (0xae64 bytes) 000372 1789116874.226982 trunc from 0x01ba7c to 0x026b3f (0xb0c4 bytes) 000373 1789116874.233293 mapwrite 0x015554 thru 0x018e3d (0x38ea bytes) 000374 1789116874.265450 MIRROR_RESYNC 5 mirrors 000375 1789116875.392949 read 0x00abf2 thru 0x00b2a1 (0x6b0 bytes) 000376 1789116875.402434 trunc from 0x0391c1 to 0x01ea48 (0x1a778 bytes) 000377 1789116875.449654 mapwrite 0x006d4c thru 0x00894d (0x1c02 bytes) 000378 1789116875.481041 trunc from 0x01ea49 to 0x03aff7 (0x1c5af bytes) 000379 1789116875.505144 fallocate 0x035a9c thru 0x03bfcb (0x6530 bytes) 000380 1789116875.516557 read 0x019b29 thru 0x0268df (0xcdb7 bytes) 000381 1789116875.535783 fallocate 0x02941d thru 0x02b6b5 (0x2299 bytes) 000382 1789116875.542399 write 0x033b40 thru 0x03cac7 (0x8f88 bytes) 000383 1789116875.543807 trunc from 0x03cac8 to 0x02abec (0x11edb bytes) 000384 1789116875.576193 mapwrite 0x0140bc thru 0x01c4f8 (0x843d bytes) 000385 1789116875.592911 fallocate 0x039d17 thru 0x03ffff (0x62e9 bytes) 000386 1789116875.597007 punch from 0x00d1b3 to 0x015e88 (0x8cd6 bytes) 000387 1789116875.601202 trunc from 0x03da42 to 0x03ffff (0x25be bytes) 000388 1789116875.603864 fallocate 0x021710 thru 0x0234fd (0x1dee bytes) 000389 1789116875.606839 mapwrite 0x035b7e thru 0x039910 (0x3d93 bytes) 000390 1789116875.633837 trunc from 0x040000 to 0x02b3f4 (0x14c0b bytes) 000391 1789116875.641471 write 0x0359ab thru 0x03ffff (0xa655 bytes) 000392 1789116875.642814 punch from 0x004fd8 to 0x00cf8a (0x7fb3 bytes) 000393 1789116875.647013 trunc from 0x025dd0 to 0x02bfca (0x61fb bytes) 000394 1789116875.651213 trunc from 0x040000 to 0x02a2e3 (0x15d1c bytes) 000395 1789116875.660281 punch from 0x020134 to 0x02a2e3 (0xa1b0 bytes) 000396 1789116875.663151 punch from 0x023b73 to 0x02a2e3 (0x6771 bytes) 000397 1789116875.671184 trunc from 0x019a29 to 0x0261a6 (0xc77e bytes) 000398 1789116875.679113 trunc from 0x020767 to 0x023d5d (0x35f7 bytes) 000399 1789116875.684484 read 0x00ea10 thru 0x00ffd7 (0x15c8 bytes) 000400 1789116875.714736 punch from 0x008d4a to 0x0178f7 (0xebae bytes) 000401 1789116875.724197 write 0x00962f thru 0x00d171 (0x3b43 bytes) 000402 1789116875.757901 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a2e4): Mirror 5: 0x5dea68f2 (stale) Mirror 6: 0x5dea68f2 (stale) Mirror 8: 0x5dea68f2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a2e4): Mirror 5: 0x5dea68f2 (stale) Mirror 6: 0x5dea68f2 (stale) Mirror 8: 0x5dea68f2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a2e4: skipped 000403 1789116876.254255 trunc from 0x01a824 to 0x02842a (0xdc07 bytes) 000404 1789116876.262439 fallocate 0x03e329 thru 0x03ffff (0x1cd7 bytes) 000405 1789116876.267296 trunc from 0x02a2e4 to 0x0151db (0x15108 bytes) 000406 1789116876.284140 trunc from 0x00de57 to 0x012301 (0x44ab bytes) 000407 1789116876.288943 mapread 0x008a0b thru 0x012e7e (0xa474 bytes) 000408 1789116876.309264 punch from 0x00f829 to 0x013793 (0x3f6b bytes) 000409 1789116876.314011 mapwrite 0x02eb13 thru 0x032ee7 (0x43d5 bytes) 000410 1789116876.350133 write 0x01a96c thru 0x023f6b (0x9600 bytes) 000411 1789116876.362006 mapread 0x01a3ce thru 0x029580 (0xf1b3 bytes) 000412 1789116876.368936 write 0x020665 thru 0x023339 (0x2cd5 bytes) 000413 1789116876.370199 read 0x005500 thru 0x00712e (0x1c2f bytes) 000414 1789116876.374709 MIRROR_RESYNC 4 mirrors 000416 1789116877.683150 punch from 0x0093b4 to 0x00980f (0x45c bytes) 000417 1789116877.720792 trunc from 0x032ee8 to 0x001f93 (0x30f54 bytes) 000418 1789116877.742572 write 0x017cca thru 0x01dfd7 (0x630e bytes) 000419 1789116877.745270 fallocate 0x03eda3 thru 0x03ffff (0x125d bytes) 000420 1789116877.751782 mapread 0x027782 thru 0x02a66e (0x2eed bytes) 000421 1789116877.782220 trunc from 0x0293f9 to 0x030345 (0x6f4d bytes) 000422 1789116877.792224 mapwrite 0x00ccbd thru 0x016c89 (0x9fcd bytes) 000423 1789116877.848876 write 0x0177ee thru 0x0253f5 (0xdc08 bytes) 000424 1789116877.851166 punch from 0x00e4ce to 0x00f3ea (0xf1d bytes) 000425 1789116877.863584 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x99981eac (stale) Mirror 8: 0x99981eac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x99981eac (stale) Mirror 8: 0x99981eac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000426 1789116878.357553 mapread 0x003ba3 thru 0x00bc79 (0x80d7 bytes) 000427 1789116878.408629 MIRROR_RESYNC 3 mirrors 000428 1789116879.186879 write 0x010e0b thru 0x01b6d9 (0xa8cf bytes) 000429 1789116879.226623 MIRROR_RESYNC 3 mirrors 000430 1789116879.912914 fallocate 0x03a69e thru 0x03ffff (0x5962 bytes) 000431 1789116879.962071 MIRROR_RESYNC 3 mirrors 000432 1789116880.977025 write 0x0384ee thru 0x03ffff (0x7b12 bytes) 000433 1789116881.015122 mapwrite 0x009fd7 thru 0x016f93 (0xcfbd bytes) 000434 1789116881.066632 mapread 0x03a508 thru 0x03ffff (0x5af8 bytes) 000435 1789116881.067461 trunc from 0x001cda to 0x0096a8 (0x79cf bytes) 000436 1789116881.077039 mapread 0x00ac30 thru 0x00c2a5 (0x1676 bytes) 000439 1789116881.079876 fallocate 0x011d6b thru 0x0196bb (0x7951 bytes) 000440 1789116881.084841 read 0x012959 thru 0x01d575 (0xac1d bytes) 000441 1789116881.089033 mapread 0x01ac24 thru 0x0283e7 (0xd7c4 bytes) 000442 1789116881.110105 punch from 0x02b35e to 0x03275a (0x73fd bytes) 000443 1789116881.114107 trunc from 0x040000 to 0x0379d7 (0x8628 bytes) 000444 1789116881.136996 fallocate 0x03ca98 thru 0x03ffff (0x3568 bytes) 000445 1789116881.152492 trunc from 0x0379d8 to 0x009a8f (0x2df48 bytes) 000446 1789116881.172070 read 0x003b7f thru 0x009a8f (0x5f11 bytes) 000447 1789116881.185793 trunc from 0x001e10 to 0x007473 (0x5664 bytes) 000448 1789116881.202230 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9a90): Mirror 8: 0xc5a85a0a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9a90): Mirror 8: 0xc5a85a0a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9a90: skipped 000449 1789116881.430962 trunc from 0x0014a3 to 0x009a8f (0x85ed bytes) 000450 1789116881.436400 trunc from 0x0038e5 to 0x009a8f (0x61ab bytes) 000451 1789116881.441246 mapwrite 0x02a1b1 thru 0x02d939 (0x3789 bytes) 000452 1789116881.489230 mapread 0x02c148 thru 0x02d939 (0x17f2 bytes) 000453 1789116881.490772 fallocate 0x01b2d0 thru 0x02a23e (0xef6f bytes) 000454 1789116881.495753 read 0x000ee3 thru 0x00d9dd (0xcafb bytes) 000455 1789116881.503198 write 0x017ea0 thru 0x020460 (0x85c1 bytes) 000456 1789116881.510719 punch from 0x00975b to 0x0147d7 (0xb07d bytes) 000457 1789116881.519043 MIRROR_RESYNC 2 mirrors 000458 1789116882.295363 MIRROR_RESYNC 2 mirrors 000459 1789116882.618028 mapwrite 0x020331 thru 0x024710 (0x43e0 bytes) 000460 1789116882.689706 mapread 0x014760 thru 0x014d88 (0x629 bytes) 000461 1789116882.698022 trunc from 0x02d93a to 0x00ce63 (0x20ad6 bytes) 000462 1789116882.762895 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) 000463 1789116882.945847 mapread 0x0006fc thru 0x00ac92 (0xa597 bytes) 000464 1789116882.979415 trunc from 0x00ce64 to 0x027bc2 (0x1ad5f bytes) 000465 1789116883.016563 read 0x0154f2 thru 0x022c9d (0xd7ac bytes) 000466 1789116883.029666 mapwrite 0x0268b0 thru 0x0331f8 (0xc949 bytes) 000467 1789116883.174151 mapread 0x002d66 thru 0x004db6 (0x2051 bytes) 000468 1789116883.174906 punch from 0x003af3 to 0x004bf6 (0x1104 bytes) 000469 1789116883.179110 read 0x02edb6 thru 0x030513 (0x175e bytes) 000470 1789116883.179819 fallocate 0x0085f5 thru 0x00e007 (0x5a13 bytes) 000471 1789116883.191557 trunc from 0x01c64c to 0x02344f (0x6e04 bytes) 000472 1789116883.212294 trunc from 0x02f058 to 0x031ed4 (0x2e7d bytes) 000474 1789116883.240794 MIRROR_EXTEND to 2 mirrors 000475 1789116884.251296 write 0x020d3f thru 0x0237a5 (0x2a67 bytes) 000476 1789116884.310150 trunc from 0x0331f9 to 0x002b67 (0x30691 bytes) 000477 1789116884.354571 trunc from 0x002b68 to 0x033a68 (0x30f01 bytes) 000478 1789116884.368757 fallocate 0x00b2e3 thru 0x011cae (0x69cc bytes) 000479 1789116884.373737 fallocate 0x02afe6 thru 0x032689 (0x76a4 bytes) 000480 1789116884.378431 read 0x00c3bf thru 0x019f5a (0xdb9c bytes) 000481 1789116884.393289 trunc from 0x033a69 to 0x025b3f (0xdf29 bytes) 000482 1789116884.411842 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x25b40): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25b40): Mirror 2: 0x63c8cec0 (stale) CRC-32 checksum value for chunk [0, 0x25b40): Mirror 1: 0x63c8cec0 Mirror 3: 0x63c8cec0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25b40: skipped 000483 1789116885.414670 trunc from 0x025b40 to 0x01e19b (0x79a4 bytes) 000484 1789116885.603126 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1e19c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e19c): Mirror 2: 0xab4e644d (stale) Mirror 3: 0xab4e644d (stale) CRC-32 checksum value for chunk [0, 0x1e19c): Mirror 1: 0xab4e644d Mirror 4: 0xab4e644d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e19c: skipped 000485 1789116886.475778 mapwrite 0x035a12 thru 0x03a87e (0x4e6d bytes) 000486 1789116886.576725 mapwrite 0x038521 thru 0x03e962 (0x6442 bytes) 000488 1789116886.625476 write 0x0012d0 thru 0x00802f (0x6d60 bytes) 000489 1789116886.638332 fallocate 0x022431 thru 0x02b112 (0x8ce2 bytes) 000490 1789116886.643056 fallocate 0x02709e thru 0x0367ce (0xf731 bytes) 000491 1789116886.647891 read 0x0187b2 thru 0x01ada3 (0x25f2 bytes) 000492 1789116886.658577 MIRROR_RESYNC 4 mirrors 000493 1789116887.607396 MIRROR_RESYNC 4 mirrors 000494 1789116887.990011 punch from 0x00aae8 to 0x012d79 (0x8292 bytes) 000495 1789116888.031145 mapwrite 0x0319fa thru 0x037117 (0x571e bytes) 000496 1789116888.078288 trunc from 0x03e963 to 0x006d69 (0x37bf9 bytes) 000497 1789116888.100099 trunc from 0x006d6a to 0x039d7b (0x33012 bytes) 000498 1789116888.121845 punch from 0x020baa to 0x02cebb (0xc312 bytes) 000499 1789116888.130811 punch from 0x01a021 to 0x022ffb (0x8fdb bytes) 000500 1789116888.140293 write 0x02d70c thru 0x038462 (0xad57 bytes) 000501 1789116888.161680 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39d7c): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39d7c): Mirror 2: 0x4e64707c (stale) Mirror 3: 0x4e64707c (stale) Mirror 4: 0x4e64707c (stale) CRC-32 checksum value for chunk [0, 0x39d7c): Mirror 1: 0x2ace4ad6 Mirror 5: 0x2ace4ad6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39d7c: skipped 000503 1789116888.966779 write 0x031205 thru 0x03edc3 (0xdbbf bytes) 000504 1789116889.007790 fallocate 0x01c4a4 thru 0x02721e (0xad7b bytes) 000505 1789116889.011490 trunc from 0x03edc4 to 0x0122af (0x2cb14 bytes) 000506 1789116889.035763 mapread 0x006d1a thru 0x0122af (0xb596 bytes) 000507 1789116889.059984 fallocate 0x01c188 thru 0x022c24 (0x6a9d bytes) 000508 1789116889.069231 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x22c25): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22c25): Mirror 2: 0x31691bbd (stale) Mirror 3: 0x31691bbd (stale) Mirror 4: 0x31691bbd (stale) Mirror 5: 0xe14a32b0 (stale) CRC-32 checksum value for chunk [0, 0x22c25): Mirror 1: 0xe14a32b0 Mirror 6: 0xe14a32b0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22c25: skipped 000509 1789116889.746984 read 0x0147bd thru 0x017397 (0x2bdb bytes) 000512 1789116889.754389 write 0x02550b thru 0x02c497 (0x6f8d bytes) 000514 1789116889.805866 punch from 0x02aad1 to 0x02c497 (0x19c7 bytes) 000515 1789116889.822240 mapread 0x002694 thru 0x0040db (0x1a48 bytes) 000516 1789116889.861217 write 0x037bb6 thru 0x03aaca (0x2f15 bytes) 000517 1789116889.871705 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3aacb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3aacb): Mirror 2: 0x5b2dac58 (stale) Mirror 3: 0x5b2dac58 (stale) Mirror 4: 0x5b2dac58 (stale) Mirror 5: 0x91e318ea (stale) Mirror 6: 0x4bc4a0a8 (stale) CRC-32 checksum value for chunk [0, 0x3aacb): Mirror 1: 0x59618c32 Mirror 7: 0x59618c32 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3aacb: skipped 000518 1789116891.182415 read 0x025163 thru 0x033b70 (0xea0e bytes) 000519 1789116891.192635 trunc from 0x0102b9 to 0x01c147 (0xbe8f bytes) 000520 1789116891.264236 mapwrite 0x03f767 thru 0x03ffff (0x899 bytes) 000521 1789116891.318125 read 0x01bcfb thru 0x02b5ed (0xf8f3 bytes) 000522 1789116891.331218 write 0x034440 thru 0x035632 (0x11f3 bytes) 000523 1789116891.333863 mapwrite 0x026629 thru 0x029df5 (0x37cd bytes) 000524 1789116891.372246 trunc from 0x040000 to 0x037596 (0x8a69 bytes) 000525 1789116891.391391 read 0x0076d7 thru 0x0079cf (0x2f9 bytes) 000526 1789116891.392210 trunc from 0x037597 to 0x02a0e0 (0xd4b6 bytes) 000527 1789116891.424689 trunc from 0x0188d7 to 0x01d2a4 (0x49ce bytes) 000528 1789116891.439950 trunc from 0x02a0e1 to 0x00a7ff (0x1f8e1 bytes) 000529 1789116891.465950 trunc from 0x00a800 to 0x002b02 (0x7cfd bytes) 000530 1789116891.488542 trunc from 0x0008be to 0x002b02 (0x2245 bytes) 000531 1789116891.501041 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2b03): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b03): Mirror 2: 0x8a9c351b (stale) Mirror 3: 0x8a9c351b (stale) Mirror 4: 0x8a9c351b (stale) Mirror 5: 0x8a9c351b (stale) Mirror 6: 0x8a9c351b (stale) Mirror 7: 0x8a9c351b (stale) CRC-32 checksum value for chunk [0, 0x2b03): Mirror 1: 0x40eb443 Mirror 8: 0x40eb443 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b03: skipped 000532 1789116892.181762 trunc from 0x002b03 to 0x02a5a2 (0x27aa0 bytes) 000533 1789116892.252843 write 0x02df26 thru 0x03cecb (0xefa6 bytes) 000534 1789116892.263363 MIRROR_RESYNC 8 mirrors 000536 1789116894.360392 MIRROR_EXTEND to 9 mirrors 000537 1789116895.455108 mapread 0x037592 thru 0x03cecb (0x593a bytes) 000538 1789116895.489480 trunc from 0x03cecc to 0x02daf2 (0xf3d9 bytes) 000539 1789116895.544079 fallocate 0x001bde thru 0x003f70 (0x2393 bytes) 000540 1789116895.547203 mapwrite 0x00ada6 thru 0x014c38 (0x9e93 bytes) 000541 1789116895.566942 mapread 0x001643 thru 0x008120 (0x6ade bytes) 000542 1789116895.570249 fallocate 0x03e35d thru 0x03ffff (0x1ca3 bytes) 000543 1789116895.572468 mapread 0x02c08e thru 0x02daf2 (0x1a65 bytes) 000545 1789116895.578498 trunc from 0x01182f to 0x0141a4 (0x2976 bytes) 000546 1789116895.581495 fallocate 0x03d6b3 thru 0x03ffff (0x294d bytes) 000547 1789116895.584470 punch from 0x016847 to 0x01f7d1 (0x8f8b bytes) 000548 1789116895.589647 punch from 0x00b018 to 0x00b754 (0x73d bytes) 000549 1789116895.596541 punch from 0x01bf24 to 0x0271a6 (0xb283 bytes) 000550 1789116895.600431 read 0x01192b thru 0x01fce3 (0xe3b9 bytes) 000551 1789116895.620414 mapwrite 0x01bdc5 thru 0x01f2f2 (0x352e bytes) 000552 1789116895.643652 mapread 0x01ac82 thru 0x01c8c2 (0x1c41 bytes) 000553 1789116895.644772 punch from 0x00ed66 to 0x01d8db (0xeb76 bytes) 000554 1789116895.657932 read 0x01e20b thru 0x026585 (0x837b bytes) 000555 1789116895.667631 trunc from 0x026f81 to 0x02a5d9 (0x3659 bytes) 000556 1789116895.682769 MIRROR_RESYNC 9 mirrors 000557 1789116898.447055 trunc from 0x02b4be to 0x02daf2 (0x2635 bytes) 000558 1789116898.478709 trunc from 0x00de6e to 0x00e79d (0x930 bytes) 000559 1789116898.483687 write 0x0346a1 thru 0x0396e4 (0x5044 bytes) 000560 1789116898.486086 mapwrite 0x0230cb thru 0x024c1d (0x1b53 bytes) 000561 1789116898.523223 write 0x013c9c thru 0x01a006 (0x636b bytes) 000562 1789116898.532250 mapwrite 0x023331 thru 0x031adb (0xe7ab bytes) 000563 1789116898.566443 trunc from 0x0396e5 to 0x03be86 (0x27a2 bytes) 000564 1789116898.586224 read 0x03a91e thru 0x03be86 (0x1569 bytes) 000565 1789116898.591912 read 0x0116bc thru 0x0136e4 (0x2029 bytes) 000566 1789116898.610112 write 0x007905 thru 0x01278f (0xae8b bytes) 000567 1789116898.611474 trunc from 0x0352bd to 0x03be86 (0x6bca bytes) 000568 1789116898.621051 mapread 0x039df4 thru 0x03be86 (0x2093 bytes) 000569 1789116898.633076 punch from 0x0095c4 to 0x014da0 (0xb7dd bytes) 000570 1789116898.643404 trunc from 0x0296fe to 0x036f9e (0xd8a1 bytes) 000571 1789116898.647069 write 0x02d9a3 thru 0x031f24 (0x4582 bytes) 000572 1789116898.661447 read 0x00907d thru 0x00a1bf (0x1143 bytes) 000574 1789116898.707282 write 0x000c6d thru 0x00b693 (0xaa27 bytes) 000575 1789116898.712627 punch from 0x00c3f0 to 0x019fbe (0xdbcf bytes) 000576 1789116898.719612 write 0x01abb8 thru 0x027007 (0xc450 bytes) 000577 1789116898.721230 fallocate 0x003ca1 thru 0x004bc7 (0xf27 bytes) 000579 1789116898.735341 fallocate 0x02809a thru 0x034955 (0xc8bc bytes) 000580 1789116898.747263 MIRROR_RESYNC 9 mirrors 000581 1789116900.726898 mapwrite 0x003116 thru 0x012b8a (0xfa75 bytes) 000582 1789116900.795824 read 0x02be2d thru 0x036927 (0xaafb bytes) 000585 1789116900.804039 trunc from 0x030c67 to 0x03be86 (0xb220 bytes) 000586 1789116900.807877 mapread 0x007405 thru 0x016abf (0xf6bb bytes) 000587 1789116900.808823 read 0x03a84f thru 0x03be86 (0x1638 bytes) 000589 1789116900.817001 mapread 0x02e69c thru 0x0391f5 (0xab5a bytes) 000590 1789116900.817781 trunc from 0x034534 to 0x03a52e (0x5ffb bytes) 000591 1789116900.821201 trunc from 0x02924a to 0x03350f (0xa2c6 bytes) 000592 1789116900.824974 punch from 0x031ccc to 0x038f6e (0x72a3 bytes) 000593 1789116900.828205 mapread 0x01041c thru 0x01cf57 (0xcb3c bytes) 000594 1789116900.829222 punch from 0x00f48b to 0x01f13a (0xfcb0 bytes) 000595 1789116900.832866 trunc from 0x00b75a to 0x0111ea (0x5a91 bytes) 000596 1789116900.836675 read 0x02e58f thru 0x03ad80 (0xc7f2 bytes) 000597 1789116900.849381 trunc from 0x03be87 to 0x001ea2 (0x39fe4 bytes) 000599 1789116900.864850 punch from 0x000d56 to 0x001ea2 (0x114d bytes) 000600 1789116900.869348 mapwrite 0x01c780 thru 0x024f69 (0x87ea bytes) 000601 1789116900.934089 trunc from 0x013d5e to 0x01c02a (0x82cd bytes) 000602 1789116900.938833 mapread 0x00d1df thru 0x011c35 (0x4a57 bytes) 000603 1789116900.955913 trunc from 0x024f6a to 0x0006f0 (0x24879 bytes) 000604 1789116900.966165 write 0x012836 thru 0x019082 (0x684d bytes) 000605 1789116900.967871 mapread 0x004138 thru 0x00de6c (0x9d35 bytes) 000606 1789116900.998072 punch from 0x007296 to 0x016d89 (0xfaf4 bytes) 000607 1789116901.010084 fallocate 0x02d2d0 thru 0x03c63f (0xf370 bytes) 000608 1789116901.014433 trunc from 0x0086ca to 0x00b184 (0x2abb bytes) 000610 1789116901.019009 trunc from 0x019083 to 0x001c87 (0x173fb bytes) 000611 1789116901.046450 MIRROR_RESYNC 9 mirrors 000612 1789116902.697564 MIRROR_EXTEND to 10 mirrors 000613 1789116903.858051 trunc from 0x000516 to 0x001c87 (0x1772 bytes) 000615 1789116903.916656 mapread 0x0015e2 thru 0x001c87 (0x6a6 bytes) 000616 1789116903.923212 write 0x011103 thru 0x016197 (0x5095 bytes) 000617 1789116903.930484 read 0x0039a6 thru 0x013661 (0xfcbc bytes) 000618 1789116903.937934 trunc from 0x016198 to 0x02c304 (0x1616d bytes) 000619 1789116903.960077 fallocate 0x016937 thru 0x016d1c (0x3e6 bytes) 000620 1789116903.966228 trunc from 0x02c305 to 0x0398e5 (0xd5e1 bytes) 000621 1789116904.005660 mapwrite 0x0013a0 thru 0x002c04 (0x1865 bytes) 000622 1789116904.057588 MIRROR_RESYNC 10 mirrors 000623 1789116907.393021 trunc from 0x0398e6 to 0x023356 (0x1658f bytes) 000625 1789116907.552227 write 0x024df6 thru 0x02ebfa (0x9e05 bytes) 000626 1789116907.567273 punch from 0x01843b to 0x01e419 (0x5fdf bytes) 000627 1789116907.583557 punch from 0x021017 to 0x02ea3a (0xda24 bytes) 000628 1789116907.636244 write 0x00d341 thru 0x01cd13 (0xf9d3 bytes) 000629 1789116907.672083 mapwrite 0x028d84 thru 0x02e1f0 (0x546d bytes) 000630 1789116907.833638 read 0x00f6e4 thru 0x016164 (0x6a81 bytes) 000631 1789116907.834304 punch from 0x014841 to 0x01861f (0x3ddf bytes) 000632 1789116907.848261 trunc from 0x01fa53 to 0x02d5a0 (0xdb4e bytes) 000633 1789116907.877783 write 0x004666 thru 0x00f28e (0xac29 bytes) 000634 1789116907.890345 fallocate 0x01c593 thru 0x029308 (0xcd76 bytes) 000635 1789116907.912226 trunc from 0x02ebfb to 0x016017 (0x18be3 bytes) 000636 1789116907.992382 MIRROR_RESYNC 10 mirrors 000637 1789116910.599655 trunc from 0x00ddf2 to 0x01377c (0x598b bytes) 000638 1789116910.659415 trunc from 0x004bf7 to 0x012a1e (0xde28 bytes) 000639 1789116910.665178 mapread 0x004fe6 thru 0x00ce94 (0x7eaf bytes) 000640 1789116910.696567 mapwrite 0x00231a thru 0x0069d0 (0x46b7 bytes) 000641 1789116910.731163 trunc from 0x00694d to 0x01358f (0xcc43 bytes) 000642 1789116910.756167 MIRROR_RESYNC 10 mirrors 000643 1789116913.590659 fallocate 0x018650 thru 0x024c93 (0xc644 bytes) 000644 1789116913.620795 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x16018): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16018): Mirror 2: 0x99eed88f (stale) Mirror 3: 0x99eed88f (stale) Mirror 4: 0x99eed88f (stale) Mirror 5: 0x99eed88f (stale) Mirror 6: 0x99eed88f (stale) Mirror 7: 0x99eed88f (stale) Mirror 8: 0x99eed88f (stale) Mirror 9: 0x99eed88f (stale) Mirror 10: 0x99eed88f (stale) CRC-32 checksum value for chunk [0, 0x16018): Mirror 1: 0x99eed88f Mirror 11: 0x99eed88f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16018: skipped 000645 1789116914.575625 mapwrite 0x03dda7 thru 0x03ffff (0x2259 bytes) 000646 1789116914.699913 fallocate 0x028676 thru 0x030b50 (0x84db bytes) 000647 1789116914.706613 write 0x03d46b thru 0x03ffff (0x2b95 bytes) 000648 1789116914.711394 fallocate 0x000a07 thru 0x00961d (0x8c17 bytes) 000649 1789116914.722219 mapread 0x030405 thru 0x0351a0 (0x4d9c bytes) 000650 1789116914.743634 punch from 0x02fa4f to 0x03dbf9 (0xe1ab bytes) 000651 1789116914.752750 mapwrite 0x03d182 thru 0x03ffff (0x2e7e bytes) 000652 1789116914.771230 punch from 0x003a45 to 0x00fad4 (0xc090 bytes) 000653 1789116914.774646 trunc from 0x040000 to 0x0115ef (0x2ea10 bytes) 000654 1789116914.797686 write 0x038509 thru 0x03ffff (0x7af7 bytes) 000655 1789116914.799362 mapread 0x0272ed thru 0x030fb6 (0x9cca bytes) 000656 1789116914.833174 write 0x03bb00 thru 0x03ffff (0x4500 bytes) 000657 1789116914.834944 read 0x0021b7 thru 0x00a37f (0x81c9 bytes) 000658 1789116914.851403 trunc from 0x03ae2d to 0x03f196 (0x436a bytes) 000659 1789116914.869472 trunc from 0x025517 to 0x02aa8e (0x5578 bytes) 000660 1789116914.875308 trunc from 0x037b84 to 0x03bd3a (0x41b7 bytes) 000661 1789116914.885984 MIRROR_RESYNC 11 mirrors 000662 1789116916.691069 write 0x016156 thru 0x0188a7 (0x2752 bytes) 000663 1789116916.744082 mapwrite 0x007d42 thru 0x008563 (0x822 bytes) 000664 1789116916.776074 mapwrite 0x02778d thru 0x033162 (0xb9d6 bytes) 000665 1789116916.823251 write 0x022779 thru 0x0266c4 (0x3f4c bytes) 000666 1789116916.825322 write 0x0101e9 thru 0x018cf2 (0x8b0a bytes) 000667 1789116916.832440 punch from 0x02813a to 0x02b70b (0x35d2 bytes) 000669 1789116916.845200 trunc from 0x03e4df to 0x03ffff (0x1b21 bytes) 000670 1789116916.860392 read 0x0158a7 thru 0x021550 (0xbcaa bytes) 000671 1789116916.881816 mapwrite 0x00d9cc thru 0x0177de (0x9e13 bytes) 000672 1789116916.939122 mapwrite 0x03e60e thru 0x03ffff (0x19f2 bytes) 000673 1789116916.978652 mapwrite 0x026130 thru 0x0331a7 (0xd078 bytes) 000674 1789116917.017714 punch from 0x023da7 to 0x028c66 (0x4ec0 bytes) 000675 1789116917.025415 trunc from 0x040000 to 0x031936 (0xe6c9 bytes) 000676 1789116917.051335 trunc from 0x030652 to 0x031936 (0x12e5 bytes) 000677 1789116917.063288 trunc from 0x01b239 to 0x024a12 (0x97da bytes) 000678 1789116917.078956 trunc from 0x031937 to 0x038a49 (0x7113 bytes) 000679 1789116917.177592 punch from 0x023696 to 0x02e01f (0xa98a bytes) 000680 1789116917.199459 write 0x02e9e4 thru 0x038920 (0x9f3d bytes) 000681 1789116917.218772 punch from 0x02bde0 to 0x03530b (0x952c bytes) 000682 1789116917.241398 write 0x03171d thru 0x033bb4 (0x2498 bytes) 000683 1789116917.253427 fallocate 0x038bfa thru 0x03ffff (0x7406 bytes) 000684 1789116917.258101 punch from 0x01949c to 0x02198b (0x84f0 bytes) 000685 1789116917.262267 write 0x030e9a thru 0x03964b (0x87b2 bytes) 000687 1789116917.268074 fallocate 0x01b9d7 thru 0x023d94 (0x83be bytes) 000688 1789116917.273234 mapwrite 0x0026df thru 0x005fe2 (0x3904 bytes) 000689 1789116917.310016 mapwrite 0x01268e thru 0x01f8f0 (0xd263 bytes) 000690 1789116917.369008 mapwrite 0x018f9d thru 0x01c5ee (0x3652 bytes) 000691 1789116917.422255 mapread 0x00ef28 thru 0x011653 (0x272c bytes) 000692 1789116917.423239 punch from 0x02af8e to 0x03964b (0xe6be bytes) 000693 1789116917.464866 mapread 0x0189d3 thru 0x027545 (0xeb73 bytes) 000696 1789116917.506182 mapread 0x027b81 thru 0x035315 (0xd795 bytes) 000697 1789116917.572177 read 0x033588 thru 0x03964b (0x60c4 bytes) 000698 1789116917.573257 fallocate 0x02339f thru 0x031c5f (0xe8c1 bytes) 000699 1789116917.590589 mapread 0x01c320 thru 0x029da0 (0xda81 bytes) 000700 1789116917.595190 trunc from 0x03964c to 0x010fe2 (0x28669 bytes) 000701 1789116917.661845 write 0x01136d thru 0x01b67b (0xa30f bytes) 000702 1789116917.667841 trunc from 0x01b67c to 0x00f6f0 (0xbf8b bytes) 000703 1789116917.704890 trunc from 0x00eea7 to 0x00f6f0 (0x84a bytes) 000704 1789116917.714920 write 0x005c04 thru 0x005dfb (0x1f8 bytes) 000705 1789116917.717022 mapwrite 0x03ac49 thru 0x03ffff (0x53b7 bytes) 000706 1789116917.778846 read 0x027a37 thru 0x033ee2 (0xc4ac bytes) 000707 1789116917.780067 trunc from 0x040000 to 0x00e979 (0x31686 bytes) 000708 1789116917.807867 punch from 0x007a7f to 0x009df7 (0x2379 bytes) 000709 1789116917.815494 punch from 0x008e31 to 0x00d709 (0x48d9 bytes) 000710 1789116917.825683 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0xe97a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe97a): Mirror 2: 0xc110719c (stale) Mirror 3: 0xc110719c (stale) Mirror 4: 0xc110719c (stale) Mirror 5: 0xc110719c (stale) Mirror 6: 0xc110719c (stale) Mirror 7: 0xc110719c (stale) Mirror 8: 0xc110719c (stale) Mirror 9: 0xc110719c (stale) Mirror 10: 0xc110719c (stale) Mirror 11: 0xc110719c (stale) CRC-32 checksum value for chunk [0, 0xe97a): Mirror 1: 0xd7fa6cdc Mirror 12: 0xd7fa6cdc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe97a: skipped 000711 1789116919.177817 trunc from 0x00e97a to 0x014880 (0x5f07 bytes) 000712 1789116919.326843 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x14881): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x14881): Mirror 2: 0x990b5369 (stale) Mirror 3: 0x990b5369 (stale) Mirror 4: 0x990b5369 (stale) Mirror 5: 0x990b5369 (stale) Mirror 6: 0x990b5369 (stale) Mirror 7: 0x990b5369 (stale) Mirror 8: 0x990b5369 (stale) Mirror 9: 0x990b5369 (stale) Mirror 10: 0x990b5369 (stale) Mirror 11: 0x990b5369 (stale) Mirror 12: 0x5c46a170 (stale) CRC-32 checksum value for chunk [0, 0x14881): Mirror 1: 0x5c46a170 Mirror 13: 0x5c46a170 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x14881: skipped 000713 1789116921.078158 mapwrite 0x01d631 thru 0x02d45d (0xfe2d bytes) 000715 1789116921.252153 mapread 0x01ac69 thru 0x028654 (0xd9ec bytes) 000716 1789116921.253202 punch from 0x01a47f to 0x023ba7 (0x9729 bytes) 000717 1789116921.280381 MIRROR_RESYNC 13 mirrors 000718 1789116926.089617 mapwrite 0x01632d thru 0x020610 (0xa2e4 bytes) 000719 1789116926.197224 trunc from 0x02d45e to 0x02ecc4 (0x1867 bytes) 000720 1789116926.219062 punch from 0x02c268 to 0x02ecc4 (0x2a5d bytes) 000721 1789116926.227103 MIRROR_RESYNC 13 mirrors 000722 1789116931.512169 mapwrite 0x02bb30 thru 0x03b5f2 (0xfac3 bytes) 000723 1789116931.644631 mapread 0x035044 thru 0x035bf6 (0xbb3 bytes) 000724 1789116931.646972 mapwrite 0x02573d thru 0x032af5 (0xd3b9 bytes) 000725 1789116931.687811 fallocate 0x02ec0e thru 0x038f22 (0xa315 bytes) 000727 1789116931.711917 mapread 0x01556d thru 0x02037c (0xae10 bytes) 000728 1789116931.729984 trunc from 0x020490 to 0x02ab3b (0xa6ac bytes) 000729 1789116931.738871 write 0x00e30f thru 0x010c41 (0x2933 bytes) 000730 1789116931.740233 mapwrite 0x00c298 thru 0x01b923 (0xf68c bytes) 000731 1789116931.880375 mapread 0x0346ac thru 0x03a16d (0x5ac2 bytes) 000732 1789116931.881305 mapwrite 0x03db6e thru 0x03ffff (0x2492 bytes) 000733 1789116931.978651 read 0x019552 thru 0x028f80 (0xfa2f bytes) 000734 1789116931.991131 fallocate 0x03614b thru 0x03ffff (0x9eb5 bytes) 000735 1789116932.007232 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x1e93ee40 (stale) Mirror 3: 0x1e93ee40 (stale) Mirror 4: 0x1e93ee40 (stale) Mirror 5: 0x1e93ee40 (stale) Mirror 6: 0x1e93ee40 (stale) Mirror 7: 0x1e93ee40 (stale) Mirror 8: 0x1e93ee40 (stale) Mirror 9: 0x1e93ee40 (stale) Mirror 10: 0x1e93ee40 (stale) Mirror 11: 0x1e93ee40 (stale) Mirror 12: 0x1e93ee40 (stale) Mirror 13: 0x1e93ee40 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xdc00a500 Mirror 14: 0xdc00a500 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000736 1789116934.069213 MIRROR_RESYNC 14 mirrors 000737 1789116939.969681 punch from 0x0166c7 to 0x01f548 (0x8e82 bytes) 000739 1789116940.022716 read 0x0258c3 thru 0x02b53f (0x5c7d bytes) 000740 1789116940.032660 trunc from 0x040000 to 0x01e4ec (0x21b13 bytes) 000741 1789116940.058536 trunc from 0x01e4ed to 0x005c39 (0x188b3 bytes) 000742 1789116940.082341 write 0x004e63 thru 0x01387b (0xea19 bytes) 000743 1789116940.090015 punch from 0x008518 to 0x00ddab (0x5894 bytes) 000744 1789116940.104093 mapread 0x013357 thru 0x01387b (0x525 bytes) 000746 1789116940.107425 read 0x0109ca thru 0x01387b (0x2eb2 bytes) 000747 1789116940.108991 mapwrite 0x01056c thru 0x018c52 (0x86e7 bytes) 000748 1789116940.157589 MIRROR_RESYNC 14 mirrors 000749 1789116942.658346 write 0x006e59 thru 0x00a1dc (0x3384 bytes) 000750 1789116942.737928 trunc from 0x0156f6 to 0x01741a (0x1d25 bytes) 000751 1789116942.742833 write 0x02154d thru 0x02d1fa (0xbcae bytes) 000752 1789116942.746205 fallocate 0x00fbcc thru 0x01ddd2 (0xe207 bytes) 000753 1789116942.751635 mapwrite 0x007a65 thru 0x00c1c2 (0x475e bytes) 000754 1789116942.799862 trunc from 0x01b8a9 to 0x025c65 (0xa3bd bytes) 000755 1789116942.815423 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x2d1fb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d1fb): Mirror 2: 0x2cc44ba2 (stale) Mirror 3: 0x2cc44ba2 (stale) Mirror 4: 0x2cc44ba2 (stale) Mirror 5: 0x2cc44ba2 (stale) Mirror 6: 0x2cc44ba2 (stale) Mirror 7: 0x2cc44ba2 (stale) Mirror 8: 0x2cc44ba2 (stale) Mirror 9: 0x2cc44ba2 (stale) Mirror 10: 0x2cc44ba2 (stale) Mirror 11: 0x2cc44ba2 (stale) Mirror 12: 0x2cc44ba2 (stale) Mirror 13: 0x2cc44ba2 (stale) Mirror 14: 0x2cc44ba2 (stale) CRC-32 checksum value for chunk [0, 0x2d1fb): Mirror 1: 0x2fca7fcd Mirror 15: 0x2fca7fcd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d1fb: skipped 000756 1789116944.329002 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x2d1fb): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d1fb): Mirror 2: 0x2cc44ba2 (stale) Mirror 3: 0x2cc44ba2 (stale) Mirror 4: 0x2cc44ba2 (stale) Mirror 5: 0x2cc44ba2 (stale) Mirror 6: 0x2cc44ba2 (stale) Mirror 7: 0x2cc44ba2 (stale) Mirror 8: 0x2cc44ba2 (stale) Mirror 9: 0x2cc44ba2 (stale) Mirror 10: 0x2cc44ba2 (stale) Mirror 11: 0x2cc44ba2 (stale) Mirror 12: 0x2cc44ba2 (stale) Mirror 13: 0x2cc44ba2 (stale) Mirror 14: 0x2cc44ba2 (stale) CRC-32 checksum value for chunk [0, 0x2d1fb): Mirror 1: 0x2fca7fcd Mirror 15: 0x2fca7fcd Mirror 16: 0x2fca7fcd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d1fb: skipped 000757 1789116946.263181 mapread 0x029b25 thru 0x02d1fa (0x36d6 bytes) 000758 1789116946.307622 mapread 0x02b47f thru 0x02d1fa (0x1d7c bytes) 000759 1789116946.314606 MIRROR_SPLIT mirror 14 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2d1fb): Mirror 2: 0x2cc44ba2 (stale) Mirror 3: 0x2cc44ba2 (stale) Mirror 4: 0x2cc44ba2 (stale) Mirror 5: 0x2cc44ba2 (stale) Mirror 6: 0x2cc44ba2 (stale) Mirror 7: 0x2cc44ba2 (stale) Mirror 8: 0x2cc44ba2 (stale) Mirror 9: 0x2cc44ba2 (stale) Mirror 10: 0x2cc44ba2 (stale) Mirror 11: 0x2cc44ba2 (stale) Mirror 12: 0x2cc44ba2 (stale) Mirror 13: 0x2cc44ba2 (stale) Mirror 14: 0x2cc44ba2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2d1fb): Mirror 2: 0x2cc44ba2 (stale) Mirror 3: 0x2cc44ba2 (stale) Mirror 4: 0x2cc44ba2 (stale) Mirror 5: 0x2cc44ba2 (stale) Mirror 6: 0x2cc44ba2 (stale) Mirror 7: 0x2cc44ba2 (stale) Mirror 8: 0x2cc44ba2 (stale) Mirror 9: 0x2cc44ba2 (stale) Mirror 10: 0x2cc44ba2 (stale) Mirror 11: 0x2cc44ba2 (stale) Mirror 12: 0x2cc44ba2 (stale) Mirror 13: 0x2cc44ba2 (stale) Mirror 14: 0x2cc44ba2 (stale) CRC-32 checksum value for chunk [0, 0x2d1fb): Mirror 1: 0x2fca7fcd Mirror 16: 0x2fca7fcd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2d1fb: skipped 000760 1789116947.386341 write 0x00d2fa thru 0x0199a3 (0xc6aa bytes) 000761 1789116947.474883 trunc from 0x01a2fa to 0x028529 (0xe230 bytes) 000762 1789116947.489406 trunc from 0x02d1fb to 0x02689e (0x695c bytes) 000764 1789116947.542686 mapwrite 0x03effd thru 0x03ffff (0x1003 bytes) 000765 1789116947.658700 MIRROR_RESYNC 15 mirrors 000766 1789116951.731762 trunc from 0x006eb5 to 0x0163cf (0xf51b bytes) 000767 1789116951.760755 write 0x0129ae thru 0x01b87e (0x8ed1 bytes) 000768 1789116951.769834 punch from 0x027daf to 0x030fdc (0x922e bytes) 000769 1789116951.772308 fallocate 0x0327cc thru 0x03e9c7 (0xc1fc bytes) 000771 1789116951.778121 MIRROR_SPLIT mirror 3 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xcbd215c6 (stale) Mirror 3: 0xcbd215c6 (stale) Mirror 5: 0xcbd215c6 (stale) Mirror 6: 0xcbd215c6 (stale) Mirror 7: 0xcbd215c6 (stale) Mirror 8: 0xcbd215c6 (stale) Mirror 9: 0xcbd215c6 (stale) Mirror 10: 0xcbd215c6 (stale) Mirror 11: 0xcbd215c6 (stale) Mirror 12: 0xcbd215c6 (stale) Mirror 13: 0xcbd215c6 (stale) Mirror 14: 0xcbd215c6 (stale) Mirror 16: 0xcbd215c6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xcbd215c6 (stale) Mirror 3: 0xcbd215c6 (stale) Mirror 5: 0xcbd215c6 (stale) Mirror 6: 0xcbd215c6 (stale) Mirror 7: 0xcbd215c6 (stale) Mirror 8: 0xcbd215c6 (stale) Mirror 9: 0xcbd215c6 (stale) Mirror 10: 0xcbd215c6 (stale) Mirror 11: 0xcbd215c6 (stale) Mirror 12: 0xcbd215c6 (stale) Mirror 13: 0xcbd215c6 (stale) Mirror 14: 0xcbd215c6 (stale) Mirror 16: 0xcbd215c6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000773 1789116952.317358 trunc from 0x040000 to 0x0195b4 (0x26a4b bytes) 000774 1789116952.348543 trunc from 0x0195b5 to 0x0163d0 (0x31e4 bytes) 000775 1789116952.364601 read 0x00fda0 thru 0x0142a3 (0x4504 bytes) 000776 1789116952.377135 mapread 0x002b83 thru 0x009011 (0x648f bytes) 000777 1789116952.396464 write 0x027c63 thru 0x031ca7 (0xa045 bytes) 000778 1789116952.403441 MIRROR_SPLIT mirror 12 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31ca8): Mirror 2: 0xef4ebd34 (stale) Mirror 3: 0xef4ebd34 (stale) Mirror 5: 0xef4ebd34 (stale) Mirror 6: 0xef4ebd34 (stale) Mirror 7: 0xef4ebd34 (stale) Mirror 8: 0xef4ebd34 (stale) Mirror 9: 0xef4ebd34 (stale) Mirror 10: 0xef4ebd34 (stale) Mirror 11: 0xef4ebd34 (stale) Mirror 12: 0xef4ebd34 (stale) Mirror 13: 0xef4ebd34 (stale) Mirror 16: 0xef4ebd34 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31ca8): Mirror 2: 0xef4ebd34 (stale) Mirror 3: 0xef4ebd34 (stale) Mirror 5: 0xef4ebd34 (stale) Mirror 6: 0xef4ebd34 (stale) Mirror 7: 0xef4ebd34 (stale) Mirror 8: 0xef4ebd34 (stale) Mirror 9: 0xef4ebd34 (stale) Mirror 10: 0xef4ebd34 (stale) Mirror 11: 0xef4ebd34 (stale) Mirror 12: 0xef4ebd34 (stale) Mirror 13: 0xef4ebd34 (stale) Mirror 16: 0xef4ebd34 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31ca8: skipped 000779 1789116953.132219 punch from 0x01165a to 0x012f68 (0x190f bytes) 000781 1789116953.136583 trunc from 0x0261a3 to 0x031ca7 (0xbb05 bytes) 000782 1789116953.140753 fallocate 0x007f92 thru 0x00afa6 (0x3015 bytes) 000783 1789116953.146028 MIRROR_RESYNC 13 mirrors 000785 1789116956.115903 MIRROR_RESYNC 13 mirrors 000786 1789116956.795200 MIRROR_EXTEND to 14 mirrors 000787 1789116957.978011 trunc from 0x0310df to 0x031ca7 (0xbc9 bytes) 000788 1789116958.067626 read 0x01ff18 thru 0x0221ed (0x22d6 bytes) 000789 1789116958.078737 read 0x00a4e5 thru 0x018d6b (0xe887 bytes) 000790 1789116958.105559 mapwrite 0x010a4e thru 0x01911b (0x86ce bytes) 000791 1789116958.164267 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x31ca8): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31ca8): Mirror 2: 0xbfc9f0da (stale) Mirror 3: 0xbfc9f0da (stale) Mirror 5: 0xbfc9f0da (stale) Mirror 6: 0xbfc9f0da (stale) Mirror 7: 0xbfc9f0da (stale) Mirror 8: 0xbfc9f0da (stale) Mirror 9: 0xbfc9f0da (stale) Mirror 10: 0xbfc9f0da (stale) Mirror 11: 0xbfc9f0da (stale) Mirror 12: 0xbfc9f0da (stale) Mirror 13: 0xbfc9f0da (stale) Mirror 16: 0xbfc9f0da (stale) Mirror 17: 0xbfc9f0da (stale) CRC-32 checksum value for chunk [0, 0x31ca8): Mirror 1: 0xcfd46991 Mirror 18: 0xcfd46991 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31ca8: skipped 000792 1789116960.543923 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x31ca8): Mirror 3: 0xbfc9f0da (stale) Mirror 5: 0xbfc9f0da (stale) Mirror 6: 0xbfc9f0da (stale) Mirror 7: 0xbfc9f0da (stale) Mirror 8: 0xbfc9f0da (stale) Mirror 9: 0xbfc9f0da (stale) Mirror 10: 0xbfc9f0da (stale) Mirror 11: 0xbfc9f0da (stale) Mirror 12: 0xbfc9f0da (stale) Mirror 13: 0xbfc9f0da (stale) Mirror 16: 0xbfc9f0da (stale) Mirror 17: 0xbfc9f0da (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31ca8): Mirror 3: 0xbfc9f0da (stale) Mirror 5: 0xbfc9f0da (stale) Mirror 6: 0xbfc9f0da (stale) Mirror 7: 0xbfc9f0da (stale) Mirror 8: 0xbfc9f0da (stale) Mirror 9: 0xbfc9f0da (stale) Mirror 10: 0xbfc9f0da (stale) Mirror 11: 0xbfc9f0da (stale) Mirror 12: 0xbfc9f0da (stale) Mirror 13: 0xbfc9f0da (stale) Mirror 16: 0xbfc9f0da (stale) Mirror 17: 0xbfc9f0da (stale) CRC-32 checksum value for chunk [0, 0x31ca8): Mirror 1: 0xcfd46991 Mirror 18: 0xcfd46991 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31ca8: skipped 000793 1789116962.198080 MIRROR_RESYNC 14 mirrors 000794 1789116964.850364 read 0x02190b thru 0x027d92 (0x6488 bytes) 000795 1789116964.856690 mapread 0x02c006 thru 0x031ca7 (0x5ca2 bytes) 000797 1789116964.872374 mapread 0x007964 thru 0x009255 (0x18f2 bytes) 000798 1789116964.887402 MIRROR_EXTEND to 15 mirrors 000799 1789116965.686076 mapwrite 0x026d55 thru 0x0280b5 (0x1361 bytes) 000800 1789116965.772844 trunc from 0x000389 to 0x00ae0f (0xaa87 bytes) 000802 1789116965.779637 mapwrite 0x03c4a0 thru 0x03ffff (0x3b60 bytes) 000803 1789116965.848317 mapread 0x03a705 thru 0x03ffff (0x58fb bytes) 000804 1789116965.856057 MIRROR_SPLIT mirror 1 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe718b392 (stale) Mirror 6: 0xe718b392 (stale) Mirror 7: 0xe718b392 (stale) Mirror 8: 0xe718b392 (stale) Mirror 9: 0xe718b392 (stale) Mirror 10: 0xe718b392 (stale) Mirror 11: 0xe718b392 (stale) Mirror 12: 0xe718b392 (stale) Mirror 13: 0xe718b392 (stale) Mirror 16: 0xe718b392 (stale) Mirror 17: 0xe718b392 (stale) Mirror 18: 0xe718b392 (stale) Mirror 19: 0xe718b392 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xe718b392 (stale) Mirror 6: 0xe718b392 (stale) Mirror 7: 0xe718b392 (stale) Mirror 8: 0xe718b392 (stale) Mirror 9: 0xe718b392 (stale) Mirror 10: 0xe718b392 (stale) Mirror 11: 0xe718b392 (stale) Mirror 12: 0xe718b392 (stale) Mirror 13: 0xe718b392 (stale) Mirror 16: 0xe718b392 (stale) Mirror 17: 0xe718b392 (stale) Mirror 18: 0xe718b392 (stale) Mirror 19: 0xe718b392 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000805 1789116967.042069 read 0x02e892 thru 0x033c58 (0x53c7 bytes) 000806 1789116967.057313 fallocate 0x0267c6 thru 0x02ded8 (0x7713 bytes) 000807 1789116967.064956 mapread 0x0270bc thru 0x031cd7 (0xac1c bytes) 000808 1789116967.093067 trunc from 0x02b2b7 to 0x0327bd (0x7507 bytes) 000809 1789116967.103519 punch from 0x02bbde to 0x02f25b (0x367e bytes) 000810 1789116967.117137 MIRROR_RESYNC 14 mirrors 000811 1789116970.552786 punch from 0x0170d8 to 0x01bfdd (0x4f06 bytes) 000812 1789116970.631831 trunc from 0x040000 to 0x0031bb (0x3ce44 bytes) 000813 1789116970.655401 MIRROR_RESYNC 14 mirrors 000814 1789116972.509694 trunc from 0x0029c7 to 0x0031bb (0x7f5 bytes) 000815 1789116972.571918 punch from 0x002bcd to 0x0031bb (0x5ef bytes) 000816 1789116972.576681 mapread 0x0011ac thru 0x0031bb (0x2010 bytes) 000817 1789116972.594418 write 0x03e94c thru 0x03eef0 (0x5a5 bytes) 000818 1789116972.596930 trunc from 0x0370f7 to 0x03c325 (0x522f bytes) 000820 1789116972.603083 write 0x028f39 thru 0x0332ae (0xa376 bytes) 000821 1789116972.629521 read 0x011268 thru 0x01dd65 (0xcafe bytes) 000822 1789116972.644128 trunc from 0x03eef1 to 0x00aa4b (0x344a5 bytes) 000823 1789116972.692950 trunc from 0x0034fd to 0x00aa4b (0x754f bytes) 000824 1789116972.707968 write 0x00c63b thru 0x015884 (0x924a bytes) 000825 1789116972.712106 punch from 0x00ca71 to 0x010c63 (0x41f3 bytes) 000826 1789116972.740494 mapread 0x0052b0 thru 0x00f82e (0xa57f bytes) 000827 1789116972.778456 trunc from 0x00cfc5 to 0x010adf (0x3b1b bytes) 000828 1789116972.797673 mapwrite 0x02cf68 thru 0x03a202 (0xd29b bytes) 000829 1789116972.883539 write 0x01ec7b thru 0x026e17 (0x819d bytes) 000830 1789116972.891694 read 0x00abab thru 0x018668 (0xdabe bytes) 000832 1789116972.905340 read 0x01e14c thru 0x023964 (0x5819 bytes) 000833 1789116972.907166 punch from 0x002dc8 to 0x0104bf (0xd6f8 bytes) 000834 1789116972.912930 punch from 0x02cda2 to 0x0391b0 (0xc40f bytes) 000835 1789116972.919689 fallocate 0x03252b thru 0x035ef8 (0x39ce bytes) 000836 1789116972.929820 MIRROR_SPLIT mirror 13 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a203): Mirror 5: 0x8abc5ee3 (stale) Mirror 6: 0x8abc5ee3 (stale) Mirror 7: 0x8abc5ee3 (stale) Mirror 8: 0x8abc5ee3 (stale) Mirror 9: 0x8abc5ee3 (stale) Mirror 10: 0x8abc5ee3 (stale) Mirror 11: 0x8abc5ee3 (stale) Mirror 12: 0x8abc5ee3 (stale) Mirror 13: 0x8abc5ee3 (stale) Mirror 16: 0x8abc5ee3 (stale) Mirror 17: 0x8abc5ee3 (stale) Mirror 18: 0x8abc5ee3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a203): Mirror 5: 0x8abc5ee3 (stale) Mirror 6: 0x8abc5ee3 (stale) Mirror 7: 0x8abc5ee3 (stale) Mirror 8: 0x8abc5ee3 (stale) Mirror 9: 0x8abc5ee3 (stale) Mirror 10: 0x8abc5ee3 (stale) Mirror 11: 0x8abc5ee3 (stale) Mirror 12: 0x8abc5ee3 (stale) Mirror 13: 0x8abc5ee3 (stale) Mirror 16: 0x8abc5ee3 (stale) Mirror 17: 0x8abc5ee3 (stale) Mirror 18: 0x8abc5ee3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a203: skipped 000837 1789116973.628836 trunc from 0x01e2c9 to 0x025052 (0x6d8a bytes) 000838 1789116973.641346 MIRROR_SPLIT mirror 2 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a203): Mirror 5: 0x8abc5ee3 (stale) Mirror 7: 0x8abc5ee3 (stale) Mirror 8: 0x8abc5ee3 (stale) Mirror 9: 0x8abc5ee3 (stale) Mirror 10: 0x8abc5ee3 (stale) Mirror 11: 0x8abc5ee3 (stale) Mirror 12: 0x8abc5ee3 (stale) Mirror 13: 0x8abc5ee3 (stale) Mirror 16: 0x8abc5ee3 (stale) Mirror 17: 0x8abc5ee3 (stale) Mirror 18: 0x8abc5ee3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a203): Mirror 5: 0x8abc5ee3 (stale) Mirror 7: 0x8abc5ee3 (stale) Mirror 8: 0x8abc5ee3 (stale) Mirror 9: 0x8abc5ee3 (stale) Mirror 10: 0x8abc5ee3 (stale) Mirror 11: 0x8abc5ee3 (stale) Mirror 12: 0x8abc5ee3 (stale) Mirror 13: 0x8abc5ee3 (stale) Mirror 16: 0x8abc5ee3 (stale) Mirror 17: 0x8abc5ee3 (stale) Mirror 18: 0x8abc5ee3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a203: skipped 000839 1789116974.469921 mapread 0x018d4b thru 0x01e39c (0x5652 bytes) 000840 1789116974.484099 trunc from 0x03a203 to 0x01fa76 (0x1a78c bytes) 000841 1789116974.500244 punch from 0x0119bb to 0x01dca6 (0xc2ec bytes) 000842 1789116974.505780 trunc from 0x002945 to 0x004204 (0x18c0 bytes) 000843 1789116974.513332 trunc from 0x01fa77 to 0x003590 (0x1c4e6 bytes) 000844 1789116974.537614 fallocate 0x031b31 thru 0x03ffff (0xe4cf bytes) 000845 1789116974.543834 fallocate 0x010622 thru 0x0191b3 (0x8b92 bytes) 000846 1789116974.548470 mapwrite 0x01e78e thru 0x02946f (0xace2 bytes) 000847 1789116974.601437 write 0x009349 thru 0x017e42 (0xeafa bytes) 000848 1789116974.611603 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x29470): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29470): Mirror 5: 0x5c28fa03 (stale) Mirror 7: 0x5c28fa03 (stale) Mirror 8: 0x5c28fa03 (stale) Mirror 9: 0x5c28fa03 (stale) Mirror 10: 0x5c28fa03 (stale) Mirror 11: 0x5c28fa03 (stale) Mirror 12: 0x5c28fa03 (stale) Mirror 13: 0x5c28fa03 (stale) Mirror 16: 0x5c28fa03 (stale) Mirror 17: 0x5c28fa03 (stale) Mirror 18: 0x5c28fa03 (stale) CRC-32 checksum value for chunk [0, 0x29470): Mirror 1: 0x59267362 Mirror 19: 0x59267362 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29470: skipped 000849 1789116975.656280 mapread 0x0157a1 thru 0x01a397 (0x4bf7 bytes) 000850 1789116975.688604 MIRROR_RESYNC 13 mirrors 000852 1789116978.176079 punch from 0x005c62 to 0x00fb9c (0x9f3b bytes) 000853 1789116978.219313 fallocate 0x0259ba thru 0x0280c5 (0x270c bytes) 000854 1789116978.224120 write 0x02d740 thru 0x033af6 (0x63b7 bytes) 000855 1789116978.225398 punch from 0x0009f2 to 0x006e6a (0x6479 bytes) 000856 1789116978.230752 trunc from 0x033af7 to 0x02e28f (0x5867 bytes) 000858 1789116978.250114 read 0x02dbcb thru 0x02e28f (0x6c5 bytes) 000859 1789116978.251291 write 0x00f1cb thru 0x01cac5 (0xd8fb bytes) 000860 1789116978.266163 punch from 0x00c35b to 0x012b7a (0x6820 bytes) 000861 1789116978.283310 trunc from 0x01c553 to 0x01c875 (0x323 bytes) 000862 1789116978.291160 fallocate 0x003168 thru 0x006427 (0x32c0 bytes) 000863 1789116978.301967 MIRROR_SPLIT mirror 6 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e290): Mirror 5: 0xf8455ed (stale) Mirror 7: 0xf8455ed (stale) Mirror 8: 0xf8455ed (stale) Mirror 9: 0xf8455ed (stale) Mirror 10: 0xf8455ed (stale) Mirror 12: 0xf8455ed (stale) Mirror 13: 0xf8455ed (stale) Mirror 16: 0xf8455ed (stale) Mirror 17: 0xf8455ed (stale) Mirror 18: 0xf8455ed (stale) Mirror 19: 0xf8455ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e290): Mirror 5: 0xf8455ed (stale) Mirror 7: 0xf8455ed (stale) Mirror 8: 0xf8455ed (stale) Mirror 9: 0xf8455ed (stale) Mirror 10: 0xf8455ed (stale) Mirror 12: 0xf8455ed (stale) Mirror 13: 0xf8455ed (stale) Mirror 16: 0xf8455ed (stale) Mirror 17: 0xf8455ed (stale) Mirror 18: 0xf8455ed (stale) Mirror 19: 0xf8455ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e290: skipped 000864 1789116978.968823 mapread 0x005ad1 thru 0x00b61a (0x5b4a bytes) 000865 1789116978.996418 write 0x00e8a6 thru 0x014488 (0x5be3 bytes) 000866 1789116979.007930 write 0x02c935 thru 0x03b5bc (0xec88 bytes) 000867 1789116979.016596 mapread 0x016c42 thru 0x0219c3 (0xad82 bytes) 000868 1789116979.036708 mapwrite 0x0266d5 thru 0x034f72 (0xe89e bytes) 000869 1789116979.071792 fallocate 0x03af45 thru 0x03dc24 (0x2ce0 bytes) 000870 1789116979.085281 fallocate 0x0226e6 thru 0x030dbb (0xe6d6 bytes) 000871 1789116979.098249 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dc25): Mirror 5: 0x4197a0e6 (stale) Mirror 7: 0x4197a0e6 (stale) Mirror 8: 0x4197a0e6 (stale) Mirror 9: 0x4197a0e6 (stale) Mirror 10: 0x4197a0e6 (stale) Mirror 12: 0x4197a0e6 (stale) Mirror 13: 0x4197a0e6 (stale) Mirror 16: 0x4197a0e6 (stale) Mirror 17: 0x4197a0e6 (stale) Mirror 18: 0x4197a0e6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dc25): Mirror 5: 0x4197a0e6 (stale) Mirror 7: 0x4197a0e6 (stale) Mirror 8: 0x4197a0e6 (stale) Mirror 9: 0x4197a0e6 (stale) Mirror 10: 0x4197a0e6 (stale) Mirror 12: 0x4197a0e6 (stale) Mirror 13: 0x4197a0e6 (stale) Mirror 16: 0x4197a0e6 (stale) Mirror 17: 0x4197a0e6 (stale) Mirror 18: 0x4197a0e6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dc25: skipped 000872 1789116979.859837 trunc from 0x03dc25 to 0x03f439 (0x1815 bytes) 000873 1789116979.895786 trunc from 0x03f43a to 0x02675e (0x18cdb bytes) 000874 1789116979.929083 mapwrite 0x02630d thru 0x033010 (0xcd04 bytes) 000875 1789116980.051718 read 0x00a37f thru 0x00c3cb (0x204d bytes) 000876 1789116980.070334 trunc from 0x0114fb to 0x02046b (0xef71 bytes) 000877 1789116980.076366 mapwrite 0x03efb9 thru 0x03ffff (0x1047 bytes) 000878 1789116980.125138 trunc from 0x040000 to 0x0381cd (0x7e32 bytes) 000879 1789116980.140677 mapread 0x00ee5b thru 0x01623d (0x73e3 bytes) 000880 1789116980.154831 mapread 0x02a5d7 thru 0x037396 (0xcdc0 bytes) 000881 1789116980.161764 fallocate 0x019635 thru 0x01b1bc (0x1b88 bytes) 000882 1789116980.166810 mapread 0x033964 thru 0x0381cd (0x486a bytes) 000883 1789116980.173986 trunc from 0x0381ce to 0x017211 (0x20fbc bytes) 000884 1789116980.194564 write 0x028b08 thru 0x0349bb (0xbeb4 bytes) 000886 1789116980.214130 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x349bc): Mirror 5: 0x23d18600 (stale) Mirror 7: 0x23d18600 (stale) Mirror 8: 0x23d18600 (stale) Mirror 10: 0x23d18600 (stale) Mirror 12: 0x23d18600 (stale) Mirror 13: 0x23d18600 (stale) Mirror 16: 0x23d18600 (stale) Mirror 17: 0x23d18600 (stale) Mirror 18: 0x23d18600 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x349bc): Mirror 5: 0x23d18600 (stale) Mirror 7: 0x23d18600 (stale) Mirror 8: 0x23d18600 (stale) Mirror 10: 0x23d18600 (stale) Mirror 12: 0x23d18600 (stale) Mirror 13: 0x23d18600 (stale) Mirror 16: 0x23d18600 (stale) Mirror 17: 0x23d18600 (stale) Mirror 18: 0x23d18600 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x349bc: skipped 000887 1789116980.814139 mapwrite 0x03f998 thru 0x03ffff (0x668 bytes) 000888 1789116980.854683 read 0x018d7f thru 0x026f96 (0xe218 bytes) 000890 1789116980.876241 trunc from 0x0380b3 to 0x03b4a6 (0x33f4 bytes) 000891 1789116980.883230 mapread 0x018471 thru 0x021298 (0x8e28 bytes) 000892 1789116980.886092 trunc from 0x023388 to 0x0271a7 (0x3e20 bytes) 000894 1789116980.893982 write 0x0281b4 thru 0x02b8a1 (0x36ee bytes) 000895 1789116980.908809 punch from 0x00f141 to 0x01eb95 (0xfa55 bytes) 000896 1789116980.914259 punch from 0x00064a to 0x00d504 (0xcebb bytes) 000897 1789116980.921133 trunc from 0x040000 to 0x00ab27 (0x354d8 bytes) 000898 1789116980.945863 fallocate 0x00467b thru 0x00a843 (0x61c9 bytes) 000899 1789116980.953889 write 0x02bcb2 thru 0x02ea04 (0x2d53 bytes) 000901 1789116980.958754 read 0x01eed7 thru 0x02d9ce (0xeaf8 bytes) 000902 1789116980.973565 fallocate 0x019028 thru 0x0229f0 (0x99c9 bytes) 000903 1789116980.979643 write 0x02c3ff thru 0x0360da (0x9cdc bytes) 000904 1789116980.982766 fallocate 0x0372f9 thru 0x03ffff (0x8d07 bytes) 000905 1789116980.990701 mapread 0x014eb8 thru 0x01938f (0x44d8 bytes) 000907 1789116981.000325 trunc from 0x009d18 to 0x00fa7c (0x5d65 bytes) 000908 1789116981.009651 mapread 0x01f417 thru 0x023f74 (0x4b5e bytes) 000909 1789116981.010567 write 0x02abff thru 0x02f6ed (0x4aef bytes) 000911 1789116981.017243 fallocate 0x03aba6 thru 0x03b8af (0xd0a bytes) 000912 1789116981.028123 read 0x002fe0 thru 0x00bb2f (0x8b50 bytes) 000913 1789116981.043588 MIRROR_RESYNC 10 mirrors 000914 1789116982.770778 read 0x022aae thru 0x02b9c8 (0x8f1b bytes) 000915 1789116982.779007 trunc from 0x00a90b to 0x012534 (0x7c2a bytes) 000916 1789116982.821992 mapread 0x027582 thru 0x02c445 (0x4ec4 bytes) 000917 1789116982.842670 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b8b0): Mirror 5: 0x2b925792 (stale) Mirror 7: 0x2b925792 (stale) Mirror 8: 0x2b925792 (stale) Mirror 12: 0x2b925792 (stale) Mirror 13: 0x2b925792 (stale) Mirror 16: 0x2b925792 (stale) Mirror 17: 0x2b925792 (stale) Mirror 18: 0x2b925792 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b8b0): Mirror 5: 0x2b925792 (stale) Mirror 7: 0x2b925792 (stale) Mirror 8: 0x2b925792 (stale) Mirror 12: 0x2b925792 (stale) Mirror 13: 0x2b925792 (stale) Mirror 16: 0x2b925792 (stale) Mirror 17: 0x2b925792 (stale) Mirror 18: 0x2b925792 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b8b0: skipped 000918 1789116983.352888 trunc from 0x0258c1 to 0x029260 (0x39a0 bytes) 000920 1789116983.358461 punch from 0x014fac to 0x020c58 (0xbcad bytes) 000921 1789116983.363213 punch from 0x0155f5 to 0x0209f2 (0xb3fe bytes) 000922 1789116983.368410 punch from 0x020463 to 0x02441c (0x3fba bytes) 000923 1789116983.372099 trunc from 0x03b8b0 to 0x024d28 (0x16b87 bytes) 000924 1789116983.387234 mapwrite 0x020345 thru 0x02f38a (0xf046 bytes) 000925 1789116983.460725 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f38b): Mirror 5: 0x1364fcec (stale) Mirror 7: 0x1364fcec (stale) Mirror 8: 0x1364fcec (stale) Mirror 13: 0x1364fcec (stale) Mirror 16: 0x1364fcec (stale) Mirror 17: 0x1364fcec (stale) Mirror 18: 0x1364fcec (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f38b): Mirror 5: 0x1364fcec (stale) Mirror 7: 0x1364fcec (stale) Mirror 8: 0x1364fcec (stale) Mirror 13: 0x1364fcec (stale) Mirror 16: 0x1364fcec (stale) Mirror 17: 0x1364fcec (stale) Mirror 18: 0x1364fcec (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f38b: skipped 000926 1789116983.994003 fallocate 0x01b89b thru 0x022fb5 (0x771b bytes) 000927 1789116984.005144 mapread 0x02d9d4 thru 0x02e603 (0xc30 bytes) 000928 1789116984.021199 mapwrite 0x003a6d thru 0x00ce75 (0x9409 bytes) 000929 1789116984.052822 punch from 0x0164a6 to 0x023e4c (0xd9a7 bytes) 000930 1789116984.059195 mapread 0x0181b2 thru 0x01bdb9 (0x3c08 bytes) 000931 1789116984.068250 trunc from 0x004dbc to 0x0101ae (0xb3f3 bytes) 000932 1789116984.074377 trunc from 0x02f38b to 0x000949 (0x2ea41 bytes) 000933 1789116984.100089 fallocate 0x03cd7b thru 0x03de54 (0x10da bytes) 000934 1789116984.112179 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x94a): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x94a): Mirror 5: 0xe112c204 (stale) Mirror 7: 0xe112c204 (stale) Mirror 8: 0xe112c204 (stale) Mirror 13: 0xe112c204 (stale) Mirror 16: 0xe112c204 (stale) Mirror 17: 0xe112c204 (stale) Mirror 18: 0xe112c204 (stale) CRC-32 checksum value for chunk [0, 0x94a): Mirror 1: 0xe112c204 Mirror 19: 0xe112c204 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x94a: skipped 000935 1789116985.005265 mapwrite 0x0353ef thru 0x03998d (0x459f bytes) 000936 1789116985.136926 trunc from 0x03998e to 0x03b8f7 (0x1f6a bytes) 000937 1789116985.157995 trunc from 0x03b8f8 to 0x0078db (0x3401c bytes) 000938 1789116985.176882 write 0x0369b0 thru 0x03f6bc (0x8d0d bytes) 000939 1789116985.179299 trunc from 0x02f327 to 0x031fee (0x2cc8 bytes) 000940 1789116985.186427 fallocate 0x00c87b thru 0x019e32 (0xd5b8 bytes) 000941 1789116985.191842 mapread 0x02d567 thru 0x032415 (0x4eaf bytes) 000942 1789116985.205443 write 0x002ab3 thru 0x00b3bd (0x890b bytes) 000943 1789116985.222187 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f6bd): Mirror 5: 0xc9bafd44 (stale) Mirror 8: 0xc9bafd44 (stale) Mirror 13: 0xc9bafd44 (stale) Mirror 16: 0xc9bafd44 (stale) Mirror 17: 0xc9bafd44 (stale) Mirror 18: 0xc9bafd44 (stale) Mirror 19: 0x26ad4d4d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f6bd): Mirror 5: 0xc9bafd44 (stale) Mirror 8: 0xc9bafd44 (stale) Mirror 13: 0xc9bafd44 (stale) Mirror 16: 0xc9bafd44 (stale) Mirror 17: 0xc9bafd44 (stale) Mirror 18: 0xc9bafd44 (stale) Mirror 19: 0x26ad4d4d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f6bd: skipped 000944 1789116985.856404 mapread 0x0114a1 thru 0x014f4c (0x3aac bytes) 000945 1789116985.878931 mapread 0x02a819 thru 0x02e676 (0x3e5e bytes) 000946 1789116985.890433 punch from 0x0288fa to 0x02d600 (0x4d07 bytes) 000947 1789116985.896687 fallocate 0x03b235 thru 0x03ffff (0x4dcb bytes) 000948 1789116985.902923 read 0x01c34f thru 0x02b8ec (0xf59e bytes) 000949 1789116985.914906 write 0x02ab73 thru 0x032d69 (0x81f7 bytes) 000950 1789116985.932147 MIRROR_RESYNC 8 mirrors 000951 1789116988.500032 fallocate 0x001752 thru 0x009846 (0x80f5 bytes) 000954 1789116988.605087 punch from 0x038820 to 0x03f6bc (0x6e9d bytes) 000955 1789116988.617614 punch from 0x02f36a to 0x03520e (0x5ea5 bytes) 000956 1789116988.624228 read 0x000157 thru 0x00633e (0x61e8 bytes) 000957 1789116988.646560 trunc from 0x03f6bd to 0x01c5f3 (0x230c9 bytes) 000958 1789116988.723207 trunc from 0x017d61 to 0x01c0c4 (0x4364 bytes) 000959 1789116988.733120 fallocate 0x001466 thru 0x0098fd (0x8498 bytes) 000960 1789116988.738624 punch from 0x010f4e to 0x012085 (0x1138 bytes) 000961 1789116988.745209 MIRROR_RESYNC 8 mirrors 000962 1789116990.492255 trunc from 0x01c5f4 to 0x010382 (0xc271 bytes) 000963 1789116990.556642 mapwrite 0x01ab9f thru 0x02315f (0x85c1 bytes) 000964 1789116990.626246 read 0x01ba78 thru 0x01beb2 (0x43b bytes) 000965 1789116990.628154 mapwrite 0x035440 thru 0x03f92e (0xa4ef bytes) 000966 1789116990.688235 punch from 0x013755 to 0x020489 (0xcd35 bytes) 000967 1789116990.699374 MIRROR_SPLIT mirror 2 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x56:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f92f): Mirror 5: 0xc5e6c430 (stale) Mirror 13: 0xc5e6c430 (stale) Mirror 16: 0xc5e6c430 (stale) Mirror 17: 0xc5e6c430 (stale) Mirror 18: 0xc5e6c430 (stale) Mirror 19: 0xc5e6c430 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f92f): Mirror 5: 0xc5e6c430 (stale) Mirror 13: 0xc5e6c430 (stale) Mirror 16: 0xc5e6c430 (stale) Mirror 17: 0xc5e6c430 (stale) Mirror 18: 0xc5e6c430 (stale) Mirror 19: 0xc5e6c430 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f92f: skipped 000969 1789116991.416670 mapwrite 0x010812 thru 0x01f93d (0xf12c bytes) 000971 1789116991.468602 fallocate 0x000558 thru 0x002a80 (0x2529 bytes) 000972 1789116991.473082 mapread 0x033545 thru 0x033e1a (0x8d6 bytes) 000973 1789116991.484200 trunc from 0x00ad1e to 0x01a19c (0xf47f bytes) 000974 1789116991.496807 punch from 0x009114 to 0x00fd2f (0x6c1c bytes) 000975 1789116991.505683 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3f92f): Mirror 5: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f92f): Mirror 5: 0xc5e6c430 (stale) Mirror 13: 0xc5e6c430 (stale) Mirror 16: 0xc5e6c430 (stale) Mirror 17: 0xc5e6c430 (stale) Mirror 18: 0xc5e6c430 (stale) Mirror 19: 0xc5e6c430 (stale) CRC-32 checksum value for chunk [0, 0x3f92f): Mirror 1: 0x4146b9ac Mirror 20: 0x4146b9ac lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f92f: skipped 000976 1789116992.831044 read 0x0060be thru 0x01411d (0xe060 bytes) 000977 1789116992.854376 trunc from 0x03f92f to 0x0027be (0x3d170 bytes) 000978 1789116993.014054 mapread 0x0017b6 thru 0x0027be (0x1009 bytes) 000979 1789116993.028787 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x27bf): Mirror 5: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27bf): Mirror 5: 0xf1c68062 (stale) Mirror 13: 0xf1c68062 (stale) Mirror 16: 0xf1c68062 (stale) Mirror 17: 0xf1c68062 (stale) Mirror 18: 0xf1c68062 (stale) Mirror 19: 0xf1c68062 (stale) Mirror 20: 0xf1c68062 (stale) CRC-32 checksum value for chunk [0, 0x27bf): Mirror 1: 0xf1c68062 Mirror 21: 0xf1c68062 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27bf: skipped 000980 1789116994.277471 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x27bf): Mirror 5: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27bf): Mirror 5: 0xf1c68062 (stale) Mirror 13: 0xf1c68062 (stale) Mirror 16: 0xf1c68062 (stale) Mirror 17: 0xf1c68062 (stale) Mirror 18: 0xf1c68062 (stale) Mirror 19: 0xf1c68062 (stale) Mirror 20: 0xf1c68062 (stale) CRC-32 checksum value for chunk [0, 0x27bf): Mirror 1: 0xf1c68062 Mirror 21: 0xf1c68062 Mirror 22: 0xf1c68062 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27bf: skipped 000981 1789116995.692067 trunc from 0x001914 to 0x0027be (0xeab bytes) 000982 1789116995.731062 punch from 0x0017ec to 0x0027be (0xfd3 bytes) 000983 1789116995.736478 trunc from 0x0027bf to 0x019621 (0x16e63 bytes) 000984 1789116995.780067 punch from 0x012d1e to 0x019621 (0x6904 bytes) 000985 1789116995.794010 trunc from 0x004d2e to 0x00ba51 (0x6d24 bytes) 000986 1789116995.807719 trunc from 0x019622 to 0x021d71 (0x8750 bytes) 000987 1789116995.861116 punch from 0x019887 to 0x021d71 (0x84eb bytes) 000988 1789116995.876986 fallocate 0x01d074 thru 0x027b1e (0xaaab bytes) 000989 1789116995.886158 fallocate 0x0079fa thru 0x011fba (0xa5c1 bytes) 000990 1789116995.908301 mapwrite 0x026a95 thru 0x0282bf (0x182b bytes) 000991 1789116996.014583 punch from 0x01f135 to 0x02092d (0x17f9 bytes) 000992 1789116996.027256 punch from 0x02596d to 0x028244 (0x28d8 bytes) 000993 1789116996.040635 trunc from 0x0154e4 to 0x018636 (0x3153 bytes) 000994 1789116996.048160 punch from 0x024a29 to 0x0282bf (0x3897 bytes) 000995 1789116996.054988 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x282c0): Mirror 5: (stale - read error) Mirror 13: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x282c0): Mirror 5: 0xba7296bb (stale) Mirror 13: 0xba7296bb (stale) Mirror 16: 0xba7296bb (stale) Mirror 17: 0xba7296bb (stale) Mirror 18: 0xba7296bb (stale) Mirror 19: 0xba7296bb (stale) Mirror 20: 0x764ac0be (stale) Mirror 21: 0x6d5fc0f3 (stale) Mirror 22: 0x6d5fc0f3 (stale) CRC-32 checksum value for chunk [0, 0x282c0): Mirror 1: 0x6d5fc0f3 Mirror 23: 0x6d5fc0f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x282c0: skipped 000996 1789116997.443545 trunc from 0x023bb9 to 0x0254a4 (0x18ec bytes) 000997 1789116997.558705 trunc from 0x0282c0 to 0x00f571 (0x18d4e bytes) 000998 1789116997.604843 trunc from 0x00e167 to 0x00f571 (0x140b bytes) 000999 1789116997.620402 fallocate 0x02a2f5 thru 0x02fd8b (0x5a97 bytes) 001000 1789116997.627693 fallocate 0x036fe7 thru 0x03ad29 (0x3d43 bytes) All operations completed A-OK! PASS 70a (157s) == sanity-flr test 71: check mirror extend parent fid ==== 04:56:45 (1789117005) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x93:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (14s) == sanity-flr test 72: check append on FLR file ========== 04:57:00 (1789117020) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.268295 s, 7.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108328 s, 9.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13d:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13c:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13c:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 04:57:12 (1789117032) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 1022240 .. Extending file size to 2830496 .. Extending file size to 6310208 .. Extending file size to 2073088 .. Extending file size to 5236256 .. Extending file size to 8432352 .. Extending file size to 6556672 .. Extending file size to 7002400 .. Extending file size to 8551072 .. Extending file size to 8540416 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8718304 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8597632 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8897824 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Waiting 62958 62959 62960 62963 62966 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) 192.168.201.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (82s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 04:58:35 (1789117115) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1664704 .. Extending file size to 2912800 .. failed Extending file size to 6236928 .. Extending file size to 6965024 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7709696 .. Extending file size to 7093024 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8552544 .. Extending file size to 8631296 .. Extending file size to 8697312 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9076672 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 64530 64531 64532 64533 64536 failed 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) 192.168.201.148@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 04:59:59 (1789117199) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2584 1268720 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2060 1269244 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 23588 3583432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17748 3589272 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 41336 7172704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2584 1268720 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2060 1269244 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 23588 3583432 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17748 3589272 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 41336 7172704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.176368 s, 11.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.161356 s, 13.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xa158de11 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 05:00:19 (1789117219) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.099224 s, 21.1 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 05:00:27 (1789117227) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xa2:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x159:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x159:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15a:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 05:00:36 (1789117236) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.118594 s, 17.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x212:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x211:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x212:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 05:00:46 (1789117246) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0745724 s, 28.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x213:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x214:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x213:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x213:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x214:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x213:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x214:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 05:00:54 (1789117254) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170573 s, 12.3 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x159:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 05:01:27 (1789117287) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103997 s, 10.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x215:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x216:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x215:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x215:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x216:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x215:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x216:0x0] } PASS 204d (54s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 05:02:21 (1789117341) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x217:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x218:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.133492 s, 15.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x217:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x218:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x217:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x217:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x218:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x217:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (53s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 05:03:14 (1789117394) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg148-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg148-server: Pool lustre.test_204f created Adding targets to pool oleg148-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg148-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.135134 s, 15.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x15c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x15a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg148-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg148-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg148-server: Pool lustre.test_204f destroyed PASS 204f (74s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 05:04:28 (1789117468) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 05:04:37 (1789117477) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 05:04:47 (1789117487) oleg148-server: Pool lustre.test_206 created Adding targets to pool oleg148-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg148-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg148-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg148-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg148-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 05:05:18 (1789117518) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806138 s, 13.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060818 s, 17.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xcbcdd38b Mirror 3: 0xcbcdd38b PASS 207 (11s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 05:05:29 (1789117529) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 05:05:33 (1789117533) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 05:05:37 (1789117537) Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.106386 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1d30000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xb80000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xb80000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xb80000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x2200000 Read this (4 bytes): BBBB 192.168.201.148@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg148-client.virtnet /mnt/lustre2 (opts:) PASS 209a (41s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 05:06:18 (1789117578) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0279766 s, 293 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 05:06:27 (1789117587) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129775 s, 8.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xbf lmm_fid: [0x200000405:0xbf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xbf lmm_fid: [0x200000405:0xbf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x165:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611327 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 05:06:29.000000000 -0400 Modify: 2026-09-11 05:06:29.000000000 -0400 Change: 2026-09-11 05:06:30.000000000 -0400 Birth: 2026-09-11 05:06:28.000000000 -0400 PASS 210a (9s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 05:06:36 (1789117596) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185727 s, 56.5 MB/s oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg148-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg148-server: O/280000401/d6/166: File not found by ext2_lookup PASS 210b (28s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 05:07:04 (1789117624) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.227207 s, 180 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.04921 s, 83.2 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 11 05:07 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xc4:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 11 05:07 /mnt/lustre/f211.sanity-flr PASS 211 (12s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 05:07:17 (1789117637) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 05:07:25 (1789117645) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (9s) == sanity-flr test complete, duration 1941 sec =========== 05:07:34 (1789117654) osc.lustre-OST0000-osc-ffff9a76505b9000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 osc.lustre-OST0001-osc-ffff9a76505b9000.stats_compr= write_pages_compressible: 0 write_pages_incompressible: 0 write_chunks_compressible: 0 write_chunks_incompressible: 0 write_bytes_compr: 0 write_bytes_raw: 0 write_bytes_incompr: 0 read_pages_compressed: 0 read_chunks_compressed: 0 read_bytes_compr: 0 read_bytes_raw: 0 obdfilter.lustre-OST0000.stats_compr= snapshot_time 1789117657.432078881 secs.nsecs start_time 1789116177.956040556 secs.nsecs elapsed_time 1479.476038325 secs.nsecs uncompressed_objects 3321 samples [reqs] obdfilter.lustre-OST0001.stats_compr= snapshot_time 1789117657.432139366 secs.nsecs start_time 1789116220.437303015 secs.nsecs elapsed_time 1436.994836351 secs.nsecs uncompressed_objects 3279 samples [reqs] === sanity-flr: start cleanup 05:07:37 (1789117657) === === sanity-flr: finish cleanup 05:07:45 (1789117665) ===