-----============= acceptance-small: sanity-flr ============----- Thu Sep 17 15:34:26 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg632-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 15:34:50 (1789673690) === oleg632-client.virtnet: executing check_config_client /mnt/lustre oleg632-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg632-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff91cc02b4a800.idle_timeout=debug osc.lustre-OST0001-osc-ffff91cc02b4a800.idle_timeout=debug disable quota as required oleg632-server: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 15:35:30 (1789673730) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8242] == sanity-flr test 0a: lfs mirror create with -N option == 15:35:32 (1789673732) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:35:51 (1789673751) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:35:55 (1789673755) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:35:58 (1789673758) PASS 0d (21s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:36:19 (1789673779) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:36:23 (1789673783) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 15:36:27 (1789673787) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff91cc02b4a800.stats=clear osc.lustre-OST0001-osc-ffff91cc02b4a800.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:36:37 (1789673797) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (16s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:36:53 (1789673813) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:37:03 (1789673823) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 15:37:06 (1789673826) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (7s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:37:14 (1789673834) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 15:37:23 (1789673843) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:37:31 (1789673851) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:37:40 (1789673860) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (7s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:37:47 (1789673867) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:37:57 (1789673877) PASS 7 (5s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:38:03 (1789673883) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 1.2586 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0781264 s, 13.4 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:38:11 (1789673891) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.111481 s, 28.2 MB/s PASS 22 (7s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 15:38:18 (1789673898) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:38:28 (1789673908) Stopping /mnt/lustre-ost1 (opts:) on oleg632-server oleg632-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91cc02b4a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91cc02b4a800.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg632-server: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg632-server: oleg632-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg632-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg632-server oleg632-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91cc02b4a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91cc02b4a800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg632-server: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg632-server: oleg632-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg632-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (66s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:39:35 (1789673975) 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.221726 s, 18.9 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg632-server oleg632-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91cc02b4a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91cc02b4a800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 4194304 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg632-server: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg632-server: oleg632-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg632-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:40:16 (1789674016) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg632-server oleg632-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91cc02b4a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91cc02b4a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg632-server: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg632-server: oleg632-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg632-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg632-server oleg632-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91cc02b4a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91cc02b4a800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg632-server: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg632-server: oleg632-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg632-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (75s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:41:32 (1789674092) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 2.64476 s, 19.8 MB/s -rw-r--r-- 1 root root 52428800 Sep 17 15:41 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.120s user 0m0.000s sys 0m0.175s Stopping /mnt/lustre-ost1 (opts:) on oleg632-server oleg632-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff91cc02b4a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff91cc02b4a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.671s user 0m0.001s sys 0m0.170s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg632-server: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg632-server: oleg632-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg632-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg632-server oleg632-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff91cc02b4a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91cc02b4a800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.446s user 0m0.002s sys 0m0.180s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg632-server: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg632-server: oleg632-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg632-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg632-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff91cc02b4a800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff91cc02b4a800.ost_server_uuid in FULL state after 0 sec PASS 33b (97s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:43:09 (1789674189) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:43:12 (1789674192) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:43:16 (1789674196) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 15:43:19 (1789674199) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 15:43:27 (1789674207) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.514959 s, 24.4 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (11s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:43:38 (1789674218) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0256718 s, 40.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036273 s, 28.9 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:43:45 (1789674225) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:43:52 (1789674232) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000753845 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00111509 s, 918 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00161501 s, 634 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00172457 s, 594 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 15:44:02 (1789674242) create test files with size 15751146 .. 5bed823c5616c8aefcb6cae1f74bedb1 - 62b39a0ee41f6e3117625784e513da29 - 4bdfface0e10271f1378148299736d70 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg632-client.virtnet /mnt/lustre (opts:) Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre PASS 37 (21s) == sanity-flr test 38: resync ============================ 15:44:23 (1789674263) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (12s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:44:35 (1789674275) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:44:41 (1789674281) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820131 s, 12.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0212695 s, 49.3 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (8s) == sanity-flr test 41: lfs mirror resync check =========== 15:44:49 (1789674289) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121398 s, 17.3 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.366988 s, 11.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.186109 s, 11.3 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (14s) == sanity-flr test 42: lfs mirror verify ================= 15:45:03 (1789674303) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 15:45:06 (1789674306) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:45:09 (1789674309) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307488 s, 34.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (7s) == sanity-flr test 44a: lfs mirror split check =========== 15:45:16 (1789674316) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.232369 s, 13.5 MB/s PASS 44a (9s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:45:25 (1789674325) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:45:36 (1789674336) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.353414 s, 29.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg632-client.virtnet /mnt/lustre (opts:) Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (9s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:45:45 (1789674345) striped dir -i0 -c2 -H all_char /mnt/lustre/d44e.sanity-flr llite.lustre-ffff91cc2c9cd800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.214346 s, 48.9 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg632-client.virtnet /mnt/lustre (opts:) Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff91cc0939d800.xattr_cache=1 PASS 44e (13s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:45:58 (1789674358) striped dir -i1 -c2 -H crush /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (31s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:46:29 (1789674389) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (33s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:47:02 (1789674422) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 15:47:06 (1789674426) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.201376 s, 15.6 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.151081 s, 20.8 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x7706c664 (nosync,stale) Mirror 2: 0x7706c664 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 355e03b5cd7f5747b1dafb5ed2844b65 - mirror 1 checksum: 03d2bc4e0483760bd5dfa0290cd95171 - mirror 2 checksum: 355e03b5cd7f5747b1dafb5ed2844b65 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-17T15:47:09-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-17T15:47:09-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 355e03b5cd7f5747b1dafb5ed2844b65 - mirror 1 checksum: 03d2bc4e0483760bd5dfa0290cd95171 - mirror 2 checksum: 03d2bc4e0483760bd5dfa0290cd95171 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-17T15:47:09-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-17T15:47:09-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } PASS 48 (9s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 15:47:15 (1789674435) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0250409 s, 41.9 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 310272.. 311295: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (7s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:47:22 (1789674442) Failing mds1 on oleg632-server Stopping /mnt/lustre-mds1 (opts:) on oleg632-server 15:47:33 (1789674453) shut down facet: mds1 facet_host: oleg632-server facet_failover_host: oleg632-server Failover mds1 to oleg632-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg632-server: oleg632-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg632-client: oleg632-server: ssh exited with exit code 1 Started lustre-MDT0000 15:47:56 (1789674476) targets are mounted 15:47:56 (1789674476) facet_failover done oleg632-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0244408 s, 42.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:48:12 (1789674492) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.055981 s, 366 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0194234 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (8s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:48:20 (1789674500) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.045107 s, 454 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.00634247 s, 3.2 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (8s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:48:28 (1789674508) striped dir -i0 -c2 -H all_char /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00559204 s, 2.9 MB/s PASS 50c (6s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:48:35 (1789674515) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0546505 s, 375 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (6s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:48:41 (1789674521) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00679362 s, 603 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0259105 s, 158 kB/s PASS 60a (6s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:48:48 (1789674528) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x18:0x0]: Invalid argument (22) PASS 60b (17s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:49:06 (1789674546) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (26s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:49:32 (1789674572) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x19:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (23s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:49:55 (1789674595) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (24s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:50:19 (1789674619) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (6s) == sanity-flr test 70: mirror create and split race ====== 15:50:25 (1789674625) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70.sanity-flr mirror create pid 57559 mirror split pid 57561 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57559 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57561 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 15:51:32 (1789674692) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 4694 skipping zero length punch hole skipping zero size read skipping zero length zero range skipping zero length punch hole skipping zero length zero range 000007 1789674694.212244 mapwrite 0x0190bb thru 0x024f14 (0xbe5a bytes) 000009 1789674694.270074 punch from 0x016eaf to 0x017a68 (0xbba bytes) 000010 1789674694.273881 read 0x007992 thru 0x00bf55 (0x45c4 bytes) truncating to largest ever: 0x3a784 000011 1789674694.281633 trunc from 0x024f15 to 0x03a783 (0x1586f bytes) 000012 1789674694.298916 write 0x03450e thru 0x03ffff (0xbaf2 bytes) 000013 1789674694.307827 mapread 0x01715b thru 0x025111 (0xdfb7 bytes) 000014 1789674694.339387 read 0x03aa78 thru 0x03f8e2 (0x4e6b bytes) 000016 1789674694.340768 mapread 0x000465 thru 0x00d2e3 (0xce7f bytes) 000017 1789674694.348608 fallocate 0x01cb7e thru 0x02b57b (0xe9fe bytes) 000018 1789674694.352583 read 0x01caa7 thru 0x0289ef (0xbf49 bytes) 000019 1789674694.353421 trunc from 0x01ec9f to 0x02cf7c (0xe2de bytes) 000020 1789674694.357876 mapwrite 0x039236 thru 0x03dc67 (0x4a32 bytes) 000021 1789674694.374861 write 0x01ae86 thru 0x0255ba (0xa735 bytes) 000022 1789674694.379641 punch from 0x00cd30 to 0x0134d3 (0x67a4 bytes) 000023 1789674694.383724 trunc from 0x040000 to 0x026987 (0x19678 bytes) 000024 1789674694.401337 write 0x0174a9 thru 0x0266c0 (0xf218 bytes) 000025 1789674694.418875 mapwrite 0x0089c8 thru 0x0183f1 (0xfa2a bytes) 000027 1789674694.508145 mapread 0x015919 thru 0x01d3af (0x7a97 bytes) 000028 1789674694.511050 punch from 0x00e8eb to 0x01951c (0xac32 bytes) 000029 1789674694.533408 punch from 0x008462 to 0x0133f5 (0xaf94 bytes) 000030 1789674694.554182 trunc from 0x026988 to 0x026ac3 (0x13c bytes) 000031 1789674694.569131 mapwrite 0x032a4f thru 0x03ffff (0xd5b1 bytes) 000032 1789674694.658122 read 0x039928 thru 0x03ffff (0x66d8 bytes) 000034 1789674694.659593 write 0x004af5 thru 0x00f8e3 (0xadef bytes) 000035 1789674694.682165 trunc from 0x040000 to 0x0074ad (0x38b52 bytes) 000036 1789674694.713429 punch from 0x00675c to 0x0074ad (0xd52 bytes) 000037 1789674694.747599 read 0x004f61 thru 0x0074ad (0x254d bytes) 000038 1789674694.753991 punch from 0x003e80 to 0x0074ad (0x362e bytes) 000039 1789674694.763606 fallocate 0x02e8de thru 0x032850 (0x3f73 bytes) 000040 1789674694.771636 write 0x03788b thru 0x037e33 (0x5a9 bytes) 000043 1789674694.775262 mapread 0x004477 thru 0x00db36 (0x96c0 bytes) 000045 1789674694.811214 trunc from 0x037e34 to 0x01c5d3 (0x1b860 bytes) 000047 1789674694.833114 punch from 0x00c2e5 to 0x01141d (0x5139 bytes) 000048 1789674694.845442 MIRROR_EXTEND to 2 mirrors 000049 1789674695.560041 write 0x023107 thru 0x02aeed (0x7de7 bytes) 000050 1789674695.589784 mapwrite 0x03444c thru 0x0394b5 (0x506a bytes) 000051 1789674695.651106 trunc from 0x0394b6 to 0x01300c (0x264a9 bytes) 000053 1789674695.665756 fallocate 0x0029e9 thru 0x003ab2 (0x10ca bytes) 000054 1789674695.672061 fallocate 0x018fcd thru 0x01e062 (0x5096 bytes) 000055 1789674695.680032 write 0x014c37 thru 0x022d83 (0xe14d bytes) 000056 1789674695.687007 read 0x0209c4 thru 0x022d83 (0x23c0 bytes) 000057 1789674695.687873 read 0x00935c thru 0x0121a2 (0x8e47 bytes) 000058 1789674695.701149 fallocate 0x02b855 thru 0x02ba31 (0x1dd bytes) 000059 1789674695.706639 fallocate 0x018506 thru 0x01ee63 (0x695e bytes) 000060 1789674695.712389 punch from 0x008028 to 0x01162c (0x9605 bytes) 000061 1789674695.722125 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2ba32): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ba32): Mirror 2: 0xbc31e250 (stale) CRC-32 checksum value for chunk [0, 0x2ba32): Mirror 1: 0x817704a5 Mirror 3: 0x817704a5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ba32: skipped 000062 1789674696.384109 mapwrite 0x038fad thru 0x03f38e (0x63e2 bytes) 000063 1789674696.574080 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3f38f): Mirror 3: 0x2bc770b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f38f): Mirror 3: 0x2bc770b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f38f: skipped 000064 1789674696.952959 punch from 0x017598 to 0x026db0 (0xf819 bytes) 000065 1789674696.958871 mapwrite 0x038096 thru 0x03ffff (0x7f6a bytes) 000066 1789674697.054786 mapwrite 0x03d955 thru 0x03ffff (0x26ab bytes) 000067 1789674697.077498 punch from 0x00e25e to 0x01b880 (0xd623 bytes) 000068 1789674697.098735 trunc from 0x040000 to 0x01199f (0x2e660 bytes) 000069 1789674697.136403 read 0x00e38c thru 0x01199f (0x3614 bytes) 000071 1789674697.155159 trunc from 0x000813 to 0x00ba17 (0xb205 bytes) 000072 1789674697.172978 mapread 0x00cce3 thru 0x01199f (0x4cbd bytes) 000073 1789674697.204348 read 0x0060af thru 0x00703e (0xf90 bytes) 000074 1789674697.210494 MIRROR_RESYNC 2 mirrors 000075 1789674697.912011 fallocate 0x005785 thru 0x0100de (0xa95a bytes) 000076 1789674697.994958 punch from 0x00759a to 0x00f260 (0x7cc7 bytes) 000077 1789674697.999430 read 0x00f035 thru 0x01199f (0x296b bytes) 000078 1789674698.005088 punch from 0x00b7d3 to 0x0108a8 (0x50d6 bytes) 000079 1789674698.010067 trunc from 0x0119a0 to 0x00b0f7 (0x68a8 bytes) 000080 1789674698.042003 write 0x030132 thru 0x034adc (0x49ab bytes) 000082 1789674698.048965 read 0x021f31 thru 0x02b755 (0x9825 bytes) 000083 1789674698.064300 fallocate 0x01c75d thru 0x027bb6 (0xb45a bytes) 000084 1789674698.068670 write 0x0271b3 thru 0x02bbba (0x4a08 bytes) 000085 1789674698.070423 trunc from 0x01f449 to 0x0250c8 (0x5c80 bytes) 000086 1789674698.073900 trunc from 0x009516 to 0x017831 (0xe31c bytes) 000087 1789674698.081998 trunc from 0x02bc74 to 0x034adc (0x8e69 bytes) 000088 1789674698.113160 trunc from 0x034add to 0x01f4bb (0x15621 bytes) 000089 1789674698.132022 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1f4bc): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f4bc): Mirror 3: 0x27c84e01 (stale) CRC-32 checksum value for chunk [0, 0x1f4bc): Mirror 1: 0x27c84e01 Mirror 4: 0x27c84e01 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f4bc: skipped 000090 1789674698.931068 mapwrite 0x009800 thru 0x014024 (0xa825 bytes) 000091 1789674699.060031 trunc from 0x01cd98 to 0x01f4bb (0x2724 bytes) 000092 1789674699.071272 write 0x038d4f thru 0x03ffff (0x72b1 bytes) 000093 1789674699.082922 trunc from 0x040000 to 0x032592 (0xda6d bytes) 000094 1789674699.151295 fallocate 0x0119f1 thru 0x016928 (0x4f38 bytes) 000096 1789674699.175230 MIRROR_RESYNC 3 mirrors 000097 1789674699.924990 read 0x02c073 thru 0x0318ee (0x587c bytes) 000098 1789674699.935108 read 0x02bfaf thru 0x032592 (0x65e4 bytes) 000099 1789674699.945455 MIRROR_EXTEND to 4 mirrors 000100 1789674700.494586 punch from 0x0173e0 to 0x01e346 (0x6f67 bytes) 000101 1789674700.519609 write 0x01104e thru 0x01db39 (0xcaec bytes) 000102 1789674700.537456 trunc from 0x020f15 to 0x02b582 (0xa66e bytes) 000103 1789674700.541888 read 0x01a0ec thru 0x01ef0a (0x4e1f bytes) 000104 1789674700.547068 read 0x025649 thru 0x032592 (0xcf4a bytes) 000105 1789674700.556978 punch from 0x00ed45 to 0x00f6be (0x97a bytes) 000106 1789674700.562141 mapwrite 0x02acf6 thru 0x034ab2 (0x9dbd bytes) 000107 1789674700.599954 trunc from 0x034536 to 0x034aa7 (0x572 bytes) 000108 1789674700.604446 write 0x0057c4 thru 0x012653 (0xce90 bytes) 000109 1789674700.605823 write 0x03af92 thru 0x03c722 (0x1791 bytes) 000110 1789674700.607571 read 0x003f87 thru 0x005a75 (0x1aef bytes) 000111 1789674700.611911 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c723): Mirror 3: 0xbefb9df6 (stale) Mirror 5: 0xbefb9df6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c723): Mirror 3: 0xbefb9df6 (stale) Mirror 5: 0xbefb9df6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c723: skipped 000112 1789674700.884947 trunc from 0x0259db to 0x02d57a (0x7ba0 bytes) 000113 1789674700.893580 MIRROR_RESYNC 3 mirrors 000115 1789674701.758917 read 0x01aebb thru 0x01ed16 (0x3e5c bytes) 000116 1789674701.768378 trunc from 0x02aa91 to 0x02f98f (0x4eff bytes) 000117 1789674701.823769 write 0x039e54 thru 0x03ffff (0x61ac bytes) 000118 1789674701.833026 fallocate 0x0164b4 thru 0x022ccd (0xc81a bytes) 000120 1789674701.838048 write 0x006710 thru 0x008860 (0x2151 bytes) 000121 1789674701.844775 read 0x039b0a thru 0x03ffff (0x64f6 bytes) 000122 1789674701.846150 read 0x00bd30 thru 0x01b94d (0xfc1e bytes) 000123 1789674701.856075 mapread 0x02af95 thru 0x02c2bc (0x1328 bytes) 000124 1789674701.859174 read 0x0186f0 thru 0x025ed8 (0xd7e9 bytes) 000125 1789674701.860202 mapread 0x0077c0 thru 0x012e23 (0xb664 bytes) 000126 1789674701.862067 punch from 0x014811 to 0x020233 (0xba23 bytes) 000127 1789674701.866838 fallocate 0x01a1b9 thru 0x024513 (0xa35b bytes) 000129 1789674701.881168 read 0x00b990 thru 0x010b60 (0x51d1 bytes) 000130 1789674701.882178 read 0x009972 thru 0x014949 (0xafd8 bytes) 000131 1789674701.891621 mapread 0x018071 thru 0x01f719 (0x76a9 bytes) 000132 1789674701.892976 punch from 0x029aad to 0x03784b (0xdd9f bytes) 000133 1789674701.898254 trunc from 0x03c00d to 0x03ffff (0x3ff3 bytes) 000134 1789674701.917068 trunc from 0x014546 to 0x01a9c5 (0x6480 bytes) 000135 1789674701.921757 mapwrite 0x028f2e thru 0x02ddd1 (0x4ea4 bytes) 000137 1789674701.964179 read 0x00ae24 thru 0x0195fd (0xe7da bytes) 000138 1789674701.975263 mapread 0x039969 thru 0x03f54e (0x5be6 bytes) 000139 1789674701.977030 trunc from 0x040000 to 0x0281e9 (0x17e16 bytes) 000140 1789674702.015038 trunc from 0x0281ea to 0x03009f (0x7eb6 bytes) 000141 1789674702.027450 trunc from 0x0300a0 to 0x023394 (0xcd0b bytes) 000142 1789674702.042236 mapread 0x00fe28 thru 0x0123dc (0x25b5 bytes) 000143 1789674702.043976 write 0x017c0e thru 0x01dc8e (0x6081 bytes) 000144 1789674702.046270 punch from 0x01e9f4 to 0x023394 (0x49a1 bytes) 000145 1789674702.051058 punch from 0x005de3 to 0x013671 (0xd88f bytes) 000146 1789674702.056914 mapwrite 0x01ee04 thru 0x024649 (0x5846 bytes) 000147 1789674702.118964 fallocate 0x02a189 thru 0x02a830 (0x6a8 bytes) 000148 1789674702.124279 write 0x004881 thru 0x00878a (0x3f0a bytes) 000149 1789674702.131137 read 0x02263e thru 0x024649 (0x200c bytes) 000150 1789674702.131830 punch from 0x00c3e5 to 0x00d84f (0x146b bytes) 000151 1789674702.139740 fallocate 0x035b91 thru 0x038038 (0x24a8 bytes) 000152 1789674702.147141 mapread 0x00621b thru 0x011ef6 (0xbcdc bytes) 000153 1789674702.159159 trunc from 0x02464a to 0x038084 (0x13a3b bytes) 000154 1789674702.192904 read 0x0048f3 thru 0x009daa (0x54b8 bytes) 000155 1789674702.194990 write 0x01b7f9 thru 0x027a88 (0xc290 bytes) 000156 1789674702.202012 trunc from 0x035b38 to 0x038084 (0x254d bytes) 000157 1789674702.207920 mapwrite 0x025bf5 thru 0x030055 (0xa461 bytes) 000158 1789674702.243651 mapread 0x002112 thru 0x007285 (0x5174 bytes) 000159 1789674702.248999 MIRROR_RESYNC 3 mirrors 000160 1789674703.067244 fallocate 0x02363a thru 0x02ffe4 (0xc9ab bytes) 000161 1789674703.108155 write 0x02cc8e thru 0x02d531 (0x8a4 bytes) 000162 1789674703.119280 write 0x02c241 thru 0x02c667 (0x427 bytes) 000163 1789674703.120933 fallocate 0x01425f thru 0x019f7c (0x5d1e bytes) 000164 1789674703.127050 mapwrite 0x019767 thru 0x024057 (0xa8f1 bytes) 000165 1789674703.206725 read 0x010e2b thru 0x01d269 (0xc43f bytes) 000167 1789674703.217819 trunc from 0x038085 to 0x033a44 (0x4640 bytes) 000168 1789674703.250624 trunc from 0x0065ff to 0x00bdcf (0x57d1 bytes) 000170 1789674703.255450 read 0x030951 thru 0x033a44 (0x30f4 bytes) 000171 1789674703.265912 punch from 0x00e769 to 0x01d916 (0xf1ae bytes) 000172 1789674703.270089 trunc from 0x03280f to 0x033a44 (0x1236 bytes) 000173 1789674703.274954 trunc from 0x007169 to 0x00f98e (0x8826 bytes) 000174 1789674703.278759 trunc from 0x033a45 to 0x007ac4 (0x2bf80 bytes) 000175 1789674703.289921 fallocate 0x00c4f7 thru 0x01ad44 (0xe84e bytes) 000176 1789674703.299013 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ad45): Mirror 3: 0x9f9326f0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ad45): Mirror 3: 0x9f9326f0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ad45: skipped 000177 1789674703.542934 read 0x009d18 thru 0x00bb2f (0x1e18 bytes) 000178 1789674703.552598 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1ad45): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ad45): Mirror 3: 0x9f9326f0 (stale) CRC-32 checksum value for chunk [0, 0x1ad45): Mirror 1: 0x9c5c1bd8 Mirror 4: 0x9c5c1bd8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ad45: skipped 000179 1789674704.324722 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) fallocating to largest ever: 0x3f0cb 000180 1789674704.604143 fallocate 0x039e6e thru 0x03f0ca (0x525d bytes) 000181 1789674704.655559 write 0x012d2c thru 0x013362 (0x637 bytes) 000182 1789674704.671435 write 0x0073ca thru 0x01523b (0xde72 bytes) 000183 1789674704.681272 write 0x039e83 thru 0x03ffff (0x617d bytes) 000184 1789674704.686350 read 0x030889 thru 0x03b42e (0xaba6 bytes) 000185 1789674704.696285 mapwrite 0x0150ca thru 0x01d904 (0x883b bytes) 000186 1789674704.735180 trunc from 0x040000 to 0x033d55 (0xc2aa bytes) 000187 1789674704.817240 mapwrite 0x02604b thru 0x029826 (0x37dc bytes) 000189 1789674704.868395 fallocate 0x0329ee thru 0x03c641 (0x9c54 bytes) 000190 1789674704.882422 MIRROR_RESYNC 2 mirrors 000191 1789674705.569183 read 0x016c1f thru 0x024570 (0xd952 bytes) 000192 1789674705.579969 mapwrite 0x0312ae thru 0x03f04d (0xdda0 bytes) 000193 1789674705.667114 punch from 0x02323f to 0x0233b6 (0x178 bytes) 000194 1789674705.671172 read 0x021f10 thru 0x02a3fe (0x84ef bytes) 000195 1789674705.680341 trunc from 0x03f04e to 0x03ecb4 (0x399 bytes) 000196 1789674705.700019 punch from 0x02c269 to 0x037f31 (0xbcc9 bytes) 000197 1789674705.708099 MIRROR_RESYNC 2 mirrors 000198 1789674706.350938 read 0x016691 thru 0x0175f5 (0xf65 bytes) 000199 1789674706.362651 read 0x03be38 thru 0x03ecb4 (0x2e7d bytes) 000200 1789674706.371801 fallocate 0x0216c8 thru 0x02ec52 (0xd58b bytes) 000201 1789674706.410867 mapread 0x00b778 thru 0x016fee (0xb877 bytes) 000202 1789674706.464844 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) 000203 1789674706.600631 punch from 0x03c5b3 to 0x03ecb4 (0x2702 bytes) 000204 1789674706.607448 write 0x014292 thru 0x0171a5 (0x2f14 bytes) 000205 1789674706.622442 trunc from 0x03ecb5 to 0x00e8b4 (0x30400 bytes) 000206 1789674706.637774 mapread 0x00a885 thru 0x00e8b4 (0x4030 bytes) 000207 1789674706.662374 fallocate 0x017444 thru 0x01a09e (0x2c5b bytes) 000208 1789674706.667208 mapwrite 0x02dd88 thru 0x035c90 (0x7f09 bytes) 000209 1789674706.710299 mapread 0x008352 thru 0x017c85 (0xf934 bytes) 000210 1789674706.731278 read 0x027941 thru 0x02c3af (0x4a6f bytes) 000212 1789674706.737146 MIRROR_EXTEND to 2 mirrors 000213 1789674707.177819 write 0x02805d thru 0x028ec6 (0xe6a bytes) 000214 1789674707.212125 read 0x01e16f thru 0x020d1a (0x2bac bytes) 000215 1789674707.218109 punch from 0x01fd4c to 0x02ce21 (0xd0d6 bytes) 000217 1789674707.226243 mapwrite 0x00e22a thru 0x017e39 (0x9c10 bytes) 000218 1789674707.255836 trunc from 0x035c91 to 0x0378a6 (0x1c16 bytes) 000219 1789674707.291813 trunc from 0x0378a7 to 0x005fa0 (0x31906 bytes) 000220 1789674707.303922 punch from 0x0007d6 to 0x005fa0 (0x57cb bytes) 000221 1789674707.310570 fallocate 0x007195 thru 0x0136a0 (0xc50c bytes) 000222 1789674707.314903 punch from 0x004c2c to 0x00de0f (0x91e4 bytes) 000223 1789674707.320984 fallocate 0x0252bf thru 0x02e693 (0x93d5 bytes) 000224 1789674707.325872 write 0x0325b4 thru 0x03ffff (0xda4c bytes) 000225 1789674707.328515 mapwrite 0x013aee thru 0x01fad5 (0xbfe8 bytes) 000226 1789674707.363303 write 0x02f3bd thru 0x0366b1 (0x72f5 bytes) 000227 1789674707.368206 mapread 0x00f309 thru 0x017a4f (0x8747 bytes) 000228 1789674707.369221 trunc from 0x00e9e5 to 0x012b49 (0x4165 bytes) 000229 1789674707.373976 mapread 0x00cf7f thru 0x00d793 (0x815 bytes) 000230 1789674707.375467 mapwrite 0x01c088 thru 0x029485 (0xd3fe bytes) 000231 1789674707.398063 fallocate 0x0229a8 thru 0x02f1be (0xc817 bytes) 000232 1789674707.402219 mapwrite 0x02ac02 thru 0x0382b8 (0xd6b7 bytes) 000233 1789674707.430393 trunc from 0x03af85 to 0x03c64e (0x16ca bytes) 000234 1789674707.435892 mapread 0x02ad28 thru 0x035e1b (0xb0f4 bytes) 000235 1789674707.440786 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x7dbb719b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x667ead02 Mirror 3: 0x667ead02 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000236 1789674708.087147 trunc from 0x01d6b7 to 0x01f752 (0x209c bytes) 000237 1789674708.121892 write 0x034dc9 thru 0x036366 (0x159e bytes) 000238 1789674708.143975 read 0x03a907 thru 0x03ffff (0x56f9 bytes) 000239 1789674708.153411 mapwrite 0x00cd82 thru 0x010132 (0x33b1 bytes) 000241 1789674708.226074 mapread 0x026f9e thru 0x02b860 (0x48c3 bytes) 000242 1789674708.243055 read 0x02d96f thru 0x032614 (0x4ca6 bytes) 000244 1789674708.269118 mapwrite 0x033cb9 thru 0x03aa9f (0x6de7 bytes) 000245 1789674708.298748 mapread 0x003601 thru 0x0118e1 (0xe2e1 bytes) 000246 1789674708.300085 write 0x02df5a thru 0x034a39 (0x6ae0 bytes) 000247 1789674708.304949 trunc from 0x024552 to 0x032a12 (0xe4c1 bytes) 000248 1789674708.327675 mapwrite 0x0254f9 thru 0x0342d2 (0xedda bytes) 000249 1789674708.499408 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x667ead02 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x667ead02 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000250 1789674708.922048 mapread 0x00e7df thru 0x01877e (0x9fa0 bytes) 000251 1789674708.971027 MIRROR_RESYNC 2 mirrors 000254 1789674709.901971 trunc from 0x00e812 to 0x015283 (0x6a72 bytes) 000255 1789674709.930985 write 0x033ac8 thru 0x035fa0 (0x24d9 bytes) 000256 1789674709.947878 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9891219c (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe0105ced Mirror 4: 0xe0105ced lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000257 1789674710.921255 fallocate 0x02ee45 thru 0x036354 (0x7510 bytes) 000258 1789674710.958580 mapread 0x019b4e thru 0x01b72f (0x1be2 bytes) 000259 1789674710.989130 mapread 0x01fa7e thru 0x026059 (0x65dc bytes) 000260 1789674711.012491 write 0x008f07 thru 0x014895 (0xb98f bytes) 000261 1789674711.027647 write 0x02902b thru 0x033e5a (0xae30 bytes) 000262 1789674711.034443 mapwrite 0x028ecf thru 0x02b43f (0x2571 bytes) 000263 1789674711.089479 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x9891219c (stale) Mirror 4: 0xe0105ced (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd2c06868 Mirror 5: 0xd2c06868 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000264 1789674712.000245 trunc from 0x001b59 to 0x00e799 (0xcc41 bytes) 000265 1789674712.091201 mapread 0x00b55e thru 0x013bd8 (0x867b bytes) 000266 1789674712.135915 trunc from 0x035246 to 0x038fd3 (0x3d8e bytes) 000267 1789674712.147972 mapread 0x031d9a thru 0x037b8b (0x5df2 bytes) 000268 1789674712.159834 mapread 0x02b58e thru 0x031b16 (0x6589 bytes) 000269 1789674712.160787 trunc from 0x040000 to 0x03a685 (0x597a bytes) 000270 1789674712.238114 mapwrite 0x005011 thru 0x0099ac (0x499c bytes) 000271 1789674712.286986 fallocate 0x01799e thru 0x01c406 (0x4a69 bytes) 000272 1789674712.293166 trunc from 0x022bdc to 0x02d036 (0xa45b bytes) 000273 1789674712.315898 mapread 0x00b789 thru 0x018924 (0xd19c bytes) 000274 1789674712.342106 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a686): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a686): Mirror 3: 0xa5ec819e (stale) Mirror 4: 0xf802cc2f (stale) Mirror 5: 0xe20bced4 (stale) CRC-32 checksum value for chunk [0, 0x3a686): Mirror 1: 0x6cfc3ee9 Mirror 6: 0x6cfc3ee9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a686: skipped 000275 1789674713.656274 trunc from 0x03a686 to 0x009b60 (0x30b25 bytes) 000276 1789674713.705220 trunc from 0x001583 to 0x009b60 (0x85de bytes) 000277 1789674713.710286 read 0x00820f thru 0x009b60 (0x1952 bytes) 000278 1789674713.716133 mapwrite 0x01abf7 thru 0x024196 (0x95a0 bytes) 000279 1789674713.812017 mapwrite 0x014100 thru 0x023633 (0xf534 bytes) 000280 1789674713.877604 write 0x012bca thru 0x01bd73 (0x91aa bytes) 000283 1789674713.896675 MIRROR_RESYNC 5 mirrors 000284 1789674715.398396 read 0x02065c thru 0x023094 (0x2a39 bytes) 000285 1789674715.407987 read 0x012cf5 thru 0x016176 (0x3482 bytes) 000286 1789674715.429059 trunc from 0x024197 to 0x02d1d9 (0x9043 bytes) 000287 1789674715.469790 trunc from 0x017fc5 to 0x026e3e (0xee7a bytes) 000288 1789674715.473986 read 0x02ace3 thru 0x02cb2d (0x1e4b bytes) 000289 1789674715.481924 mapread 0x01d2df thru 0x02443a (0x715c bytes) 000290 1789674715.483017 mapread 0x00352d thru 0x008365 (0x4e39 bytes) 000291 1789674715.484112 trunc from 0x007870 to 0x011642 (0x9dd3 bytes) 000292 1789674715.489478 MIRROR_RESYNC 5 mirrors 000293 1789674716.389990 mapread 0x00e447 thru 0x013d5c (0x5916 bytes) 000294 1789674716.404206 read 0x012cf9 thru 0x02276e (0xfa76 bytes) 000295 1789674716.411799 mapwrite 0x02895e thru 0x02da4f (0x50f2 bytes) 000296 1789674716.478955 trunc from 0x02da50 to 0x016a9d (0x16fb2 bytes) 000297 1789674716.496894 read 0x00fdc1 thru 0x016a9d (0x6cdd bytes) 000298 1789674716.505168 trunc from 0x004f6d to 0x00fe96 (0xaf2a bytes) 000299 1789674716.510184 mapread 0x014b6b thru 0x016a9d (0x1f33 bytes) 000300 1789674716.511272 trunc from 0x004f39 to 0x011521 (0xc5e9 bytes) 000301 1789674716.519247 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x16a9e): Mirror 3: 0xb158149 (stale) Mirror 5: 0xb158149 (stale) Mirror 6: 0xb158149 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16a9e): Mirror 3: 0xb158149 (stale) Mirror 5: 0xb158149 (stale) Mirror 6: 0xb158149 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16a9e: skipped 000302 1789674716.816034 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x16a9e): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16a9e): Mirror 3: 0xb158149 (stale) Mirror 5: 0xb158149 (stale) Mirror 6: 0xb158149 (stale) CRC-32 checksum value for chunk [0, 0x16a9e): Mirror 1: 0xb158149 Mirror 7: 0xb158149 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16a9e: skipped 000303 1789674717.712717 trunc from 0x000c99 to 0x00981d (0x8b85 bytes) 000304 1789674717.789631 trunc from 0x01585b to 0x016a9d (0x1243 bytes) 000305 1789674717.798580 write 0x036aa7 thru 0x03ffff (0x9559 bytes) 000306 1789674717.809542 mapread 0x02ba4d thru 0x03182a (0x5dde bytes) 000308 1789674717.838903 trunc from 0x0227c8 to 0x02bcb7 (0x94f0 bytes) 000309 1789674717.849710 trunc from 0x035c2d to 0x0377de (0x1bb2 bytes) 000310 1789674717.867742 read 0x00a7e4 thru 0x013edb (0x96f8 bytes) 000311 1789674717.887008 write 0x004b4a thru 0x0116a3 (0xcb5a bytes) 000312 1789674717.892258 fallocate 0x00f2e7 thru 0x011d3b (0x2a55 bytes) 000313 1789674717.903036 trunc from 0x02d3fb to 0x0341e7 (0x6ded bytes) 000314 1789674717.907382 trunc from 0x00370b to 0x009f1a (0x6810 bytes) 000315 1789674717.922833 write 0x03ecd6 thru 0x03ffff (0x132a bytes) 000316 1789674717.923722 mapread 0x02198f thru 0x025c7c (0x42ee bytes) 000317 1789674717.941002 mapwrite 0x017e22 thru 0x0198dd (0x1abc bytes) 000318 1789674717.967408 trunc from 0x040000 to 0x027dd7 (0x18228 bytes) 000319 1789674717.995785 fallocate 0x029b79 thru 0x02e0fe (0x4586 bytes) 000320 1789674718.011211 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2e0ff): Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e0ff): Mirror 3: 0x7818caeb (stale) Mirror 5: 0x7818caeb (stale) Mirror 6: 0x7818caeb (stale) Mirror 7: 0x534442b9 (stale) CRC-32 checksum value for chunk [0, 0x2e0ff): Mirror 1: 0x1d7ac40d Mirror 8: 0x1d7ac40d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e0ff: skipped 000321 1789674719.156589 read 0x001d0d thru 0x004bee (0x2ee2 bytes) 000322 1789674719.175233 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e0ff): Mirror 3: 0x7818caeb (stale) Mirror 5: 0x7818caeb (stale) Mirror 7: 0x534442b9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e0ff): Mirror 3: 0x7818caeb (stale) Mirror 5: 0x7818caeb (stale) Mirror 7: 0x534442b9 (stale) CRC-32 checksum value for chunk [0, 0x2e0ff): Mirror 1: 0x1d7ac40d Mirror 8: 0x1d7ac40d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e0ff: skipped 000323 1789674720.021529 mapread 0x011d68 thru 0x014298 (0x2531 bytes) 000324 1789674720.055796 mapwrite 0x0035c8 thru 0x01134d (0xdd86 bytes) 000325 1789674720.271589 trunc from 0x011156 to 0x01b262 (0xa10d bytes) 000326 1789674720.278047 write 0x0294f7 thru 0x02daf4 (0x45fe bytes) 000327 1789674720.291253 punch from 0x00cd81 to 0x01b82f (0xeaaf bytes) 000328 1789674720.299599 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e0ff): Mirror 3: 0x7818caeb (stale) Mirror 7: 0x534442b9 (stale) Mirror 8: 0x1d7ac40d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e0ff): Mirror 3: 0x7818caeb (stale) Mirror 7: 0x534442b9 (stale) Mirror 8: 0x1d7ac40d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e0ff: skipped 000329 1789674720.655197 punch from 0x01dc72 to 0x024ee5 (0x7274 bytes) 000331 1789674720.662078 trunc from 0x010d24 to 0x0167a8 (0x5a85 bytes) 000332 1789674720.665870 mapread 0x02adca thru 0x02e0fe (0x3335 bytes) 000333 1789674720.692903 trunc from 0x021c2a to 0x0252f4 (0x36cb bytes) 000334 1789674720.696518 fallocate 0x0058bb thru 0x014668 (0xedae bytes) 000335 1789674720.700162 trunc from 0x02e0ff to 0x003b61 (0x2a59d bytes) 000336 1789674720.719062 read 0x001649 thru 0x001914 (0x2cc bytes) 000337 1789674720.726903 mapwrite 0x03d5a1 thru 0x03ffff (0x2a5f bytes) 000338 1789674720.764827 trunc from 0x040000 to 0x03485c (0xb7a3 bytes) 000339 1789674720.785177 punch from 0x009016 to 0x0116d5 (0x86c0 bytes) 000340 1789674720.793560 trunc from 0x002279 to 0x00f963 (0xd6eb bytes) 000341 1789674720.798373 trunc from 0x03485d to 0x003173 (0x316e9 bytes) 000342 1789674720.815812 mapread 0x0027c3 thru 0x003173 (0x9b1 bytes) 000343 1789674720.821094 write 0x01a838 thru 0x01f05d (0x4826 bytes) 000345 1789674720.823758 punch from 0x00b691 to 0x014b9f (0x950f bytes) 000346 1789674720.837947 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x1f05e): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f05e): Mirror 3: 0x5e792aa8 (stale) Mirror 7: 0x15ed4cce (stale) Mirror 8: 0x8b7328be (stale) CRC-32 checksum value for chunk [0, 0x1f05e): Mirror 1: 0xad293058 Mirror 9: 0xad293058 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f05e: skipped 000347 1789674721.808029 trunc from 0x00a285 to 0x00a92b (0x6a7 bytes) 000348 1789674721.840135 write 0x039446 thru 0x03bb3d (0x26f8 bytes) 000349 1789674721.845160 trunc from 0x0378d1 to 0x0393db (0x1b0b bytes) 000350 1789674721.851987 punch from 0x008187 to 0x0147d8 (0xc652 bytes) 000351 1789674721.855814 fallocate 0x0331d7 thru 0x033e69 (0xc93 bytes) 000352 1789674721.862468 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bb3e): Mirror 3: 0x58cdd789 (stale) Mirror 7: 0xef188b90 (stale) Mirror 8: 0x5c95679c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb3e): Mirror 3: 0x58cdd789 (stale) Mirror 7: 0xef188b90 (stale) Mirror 8: 0x5c95679c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb3e: skipped 000353 1789674722.188123 fallocate 0x023449 thru 0x032b8e (0xf746 bytes) 000354 1789674722.201912 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bb3e): Mirror 3: 0x58cdd789 (stale) Mirror 8: 0x5c95679c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb3e): Mirror 3: 0x58cdd789 (stale) Mirror 8: 0x5c95679c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb3e: skipped 000355 1789674722.513218 write 0x02dd45 thru 0x0375c7 (0x9883 bytes) 000356 1789674722.531680 write 0x0043f9 thru 0x004864 (0x46c bytes) 000357 1789674722.536270 read 0x00d993 thru 0x00e911 (0xf7f bytes) 000358 1789674722.540755 trunc from 0x03a112 to 0x03bb3d (0x1a2c bytes) 000359 1789674722.544989 trunc from 0x00893b to 0x0135b5 (0xac7b bytes) 000360 1789674722.548684 trunc from 0x0048a3 to 0x0147e9 (0xff47 bytes) 000361 1789674722.561989 mapread 0x013a41 thru 0x01493f (0xeff bytes) 000362 1789674722.576447 write 0x000772 thru 0x00d682 (0xcf11 bytes) 000363 1789674722.594597 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3bb3e): Mirror 3: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb3e): Mirror 3: 0x58cdd789 (stale) Mirror 8: 0x5c95679c (stale) CRC-32 checksum value for chunk [0, 0x3bb3e): Mirror 1: 0x84310f76 Mirror 9: 0x84310f76 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb3e: skipped 000365 1789674723.791575 punch from 0x02dba5 to 0x03bb3d (0xdf99 bytes) 000366 1789674723.854659 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3bb3e): Mirror 3: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb3e): Mirror 3: 0x58cdd789 (stale) Mirror 8: 0x5c95679c (stale) Mirror 9: 0x84310f76 (stale) CRC-32 checksum value for chunk [0, 0x3bb3e): Mirror 1: 0xefd13bc6 Mirror 10: 0xefd13bc6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb3e: skipped 000367 1789674724.747503 mapread 0x021d4e thru 0x02a632 (0x88e5 bytes) 000368 1789674724.782222 trunc from 0x02c112 to 0x03b425 (0xf314 bytes) 000369 1789674724.839225 trunc from 0x00b0ee to 0x017564 (0xc477 bytes) 000370 1789674724.850290 punch from 0x028d64 to 0x03685a (0xdaf7 bytes) 000371 1789674724.861264 write 0x00f227 thru 0x01a592 (0xb36c bytes) 000372 1789674724.885946 punch from 0x00419f to 0x004456 (0x2b8 bytes) 000373 1789674724.892034 mapread 0x02bdc5 thru 0x0302e8 (0x4524 bytes) 000374 1789674724.903399 trunc from 0x01adc7 to 0x024bc6 (0x9e00 bytes) 000375 1789674724.950412 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3bb3e): Mirror 3: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb3e): Mirror 3: 0x58cdd789 (stale) Mirror 8: 0x5c95679c (stale) Mirror 9: 0x84310f76 (stale) Mirror 10: 0xefd13bc6 (stale) CRC-32 checksum value for chunk [0, 0x3bb3e): Mirror 1: 0xf686fd49 Mirror 11: 0xf686fd49 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb3e: skipped 000376 1789674725.890580 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3bb3e): Mirror 3: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bb3e): Mirror 3: 0x58cdd789 (stale) Mirror 8: 0x5c95679c (stale) Mirror 9: 0x84310f76 (stale) Mirror 10: 0xefd13bc6 (stale) CRC-32 checksum value for chunk [0, 0x3bb3e): Mirror 1: 0xf686fd49 Mirror 11: 0xf686fd49 Mirror 12: 0xf686fd49 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bb3e: skipped 000377 1789674726.900833 trunc from 0x0230d4 to 0x02b34c (0x8279 bytes) 000378 1789674726.987270 trunc from 0x03bb3e to 0x017133 (0x24a0a bytes) 000379 1789674727.014138 punch from 0x00d610 to 0x00f365 (0x1d56 bytes) 000380 1789674727.020062 mapwrite 0x00dc03 thru 0x01120e (0x360c bytes) fallocating to largest ever: 0x3f8f4 000381 1789674727.051712 fallocate 0x0322af thru 0x03f8f3 (0xd645 bytes) 000382 1789674727.059847 mapwrite 0x0260ff thru 0x02c444 (0x6346 bytes) 000383 1789674727.111091 write 0x01bc91 thru 0x021669 (0x59d9 bytes) 000384 1789674727.123866 trunc from 0x02c445 to 0x003112 (0x29332 bytes) 000385 1789674727.145524 trunc from 0x0020a8 to 0x0025dc (0x535 bytes) 000386 1789674727.153770 read 0x002c6a thru 0x003112 (0x4a9 bytes) 000387 1789674727.158852 write 0x01da37 thru 0x022e6e (0x5438 bytes) 000388 1789674727.160026 mapread 0x01c004 thru 0x022e6e (0x6e6b bytes) 000389 1789674727.166649 write 0x02af1c thru 0x02f033 (0x4118 bytes) 000390 1789674727.167896 trunc from 0x02f034 to 0x00db54 (0x214df bytes) 000391 1789674727.186764 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xdb55): Mirror 3: 0xeddd5922 (stale) Mirror 9: 0x740c450f (stale) Mirror 10: 0x740c450f (stale) Mirror 11: 0x92b708d7 (stale) Mirror 12: 0x92b708d7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xdb55): Mirror 3: 0xeddd5922 (stale) Mirror 9: 0x740c450f (stale) Mirror 10: 0x740c450f (stale) Mirror 11: 0x92b708d7 (stale) Mirror 12: 0x92b708d7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xdb55: skipped 000392 1789674727.675009 fallocate 0x00fe3f thru 0x018d1d (0x8edf bytes) 000394 1789674727.680967 mapread 0x007ff4 thru 0x00f8f9 (0x7906 bytes) 000395 1789674727.697069 read 0x00109f thru 0x00117d (0xdf bytes) 000396 1789674727.702064 read 0x014c46 thru 0x0153e8 (0x7a3 bytes) 000397 1789674727.709291 read 0x011b0f thru 0x015172 (0x3664 bytes) 000398 1789674727.710268 trunc from 0x00c92f to 0x015838 (0x8f0a bytes) 000399 1789674727.714832 write 0x00903f thru 0x00c389 (0x334b bytes) 000400 1789674727.722264 write 0x00cc74 thru 0x0169fc (0x9d89 bytes) 000401 1789674727.723713 mapwrite 0x0051de thru 0x009250 (0x4073 bytes) 000402 1789674727.763829 mapwrite 0x011bd7 thru 0x0210cb (0xf4f5 bytes) 000403 1789674727.806907 trunc from 0x0210cc to 0x0248fe (0x3833 bytes) 000404 1789674727.825801 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x248ff): Mirror 3: 0xe13447ca (stale) Mirror 9: 0xec5742b4 (stale) Mirror 11: 0x318b9de8 (stale) Mirror 12: 0x318b9de8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x248ff): Mirror 3: 0xe13447ca (stale) Mirror 9: 0xec5742b4 (stale) Mirror 11: 0x318b9de8 (stale) Mirror 12: 0x318b9de8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x248ff: skipped 000406 1789674728.294235 MIRROR_RESYNC 5 mirrors 000408 1789674729.453088 MIRROR_EXTEND to 6 mirrors 000410 1789674730.086803 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) 000412 1789674730.513082 MIRROR_RESYNC 5 mirrors 000413 1789674731.088724 punch from 0x000699 to 0x00720c (0x6b74 bytes) 000414 1789674731.133118 punch from 0x021c53 to 0x0248fe (0x2cac bytes) 000415 1789674731.143695 fallocate 0x027ed8 thru 0x028e88 (0xfb1 bytes) 000416 1789674731.156768 read 0x00cccd thru 0x018856 (0xbb8a bytes) 000417 1789674731.166938 mapwrite 0x012ac4 thru 0x01d331 (0xa86e bytes) 000418 1789674731.208360 mapwrite 0x028ed6 thru 0x032025 (0x9150 bytes) 000419 1789674731.295176 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32026): Mirror 3: 0xb7731e0a (stale) Mirror 12: 0xb7731e0a (stale) Mirror 13: 0xb7731e0a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32026): Mirror 3: 0xb7731e0a (stale) Mirror 12: 0xb7731e0a (stale) Mirror 13: 0xb7731e0a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32026: skipped 000420 1789674731.766997 trunc from 0x032026 to 0x01c1e2 (0x15e43 bytes) 000421 1789674731.793649 punch from 0x00fd80 to 0x0160bd (0x633e bytes) 000422 1789674731.800615 fallocate 0x005866 thru 0x0094d0 (0x3c6b bytes) 000423 1789674731.805830 write 0x01d86c thru 0x020d97 (0x352c bytes) 000424 1789674731.812763 punch from 0x018356 to 0x01efa1 (0x6c4c bytes) 000426 1789674731.859200 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x20d98): Mirror 3: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20d98): Mirror 3: 0x643fbf72 (stale) Mirror 12: 0x643fbf72 (stale) Mirror 13: 0x643fbf72 (stale) CRC-32 checksum value for chunk [0, 0x20d98): Mirror 1: 0x9da25939 Mirror 14: 0x9da25939 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20d98: skipped 000427 1789674732.858084 trunc from 0x020d98 to 0x019b53 (0x7244 bytes) 000428 1789674732.908444 trunc from 0x007cb0 to 0x00f7d8 (0x7b29 bytes) 000429 1789674732.913280 mapread 0x004de4 thru 0x0118d9 (0xcaf6 bytes) 000430 1789674732.948032 read 0x016dea thru 0x019b53 (0x2d6a bytes) 000431 1789674732.948980 mapread 0x011c5e thru 0x019b53 (0x7ef6 bytes) 000432 1789674732.954855 MIRROR_RESYNC 5 mirrors 000433 1789674734.409229 trunc from 0x00272f to 0x0035b9 (0xe8b bytes) 000434 1789674734.456218 mapread 0x00b80c thru 0x00d28e (0x1a83 bytes) 000435 1789674734.485818 read 0x001fb0 thru 0x003f5a (0x1fab bytes) 000437 1789674734.496142 trunc from 0x015925 to 0x019b53 (0x422f bytes) 000438 1789674734.506792 trunc from 0x00b449 to 0x00efd4 (0x3b8c bytes) 000439 1789674734.512981 read 0x003879 thru 0x004b65 (0x12ed bytes) 000440 1789674734.538260 trunc from 0x019b54 to 0x02ce7c (0x13329 bytes) 000441 1789674734.579784 fallocate 0x0229cd thru 0x031b6a (0xf19e bytes) 000442 1789674734.594787 write 0x019dde thru 0x01c4a1 (0x26c4 bytes) 000443 1789674734.606002 trunc from 0x02ce7d to 0x02a134 (0x2d48 bytes) 000444 1789674734.637971 trunc from 0x02a135 to 0x03869e (0xe56a bytes) 000445 1789674734.696348 trunc from 0x0200cf to 0x020f7b (0xead bytes) 000446 1789674734.704846 write 0x010ca7 thru 0x01fb5b (0xeeb5 bytes) 000447 1789674734.711689 punch from 0x0320a1 to 0x03869e (0x65fe bytes) 000448 1789674734.717689 read 0x002d4f thru 0x00d078 (0xa32a bytes) 000449 1789674734.718679 punch from 0x032bb5 to 0x03869e (0x5aea bytes) 000450 1789674734.725993 mapread 0x00c889 thru 0x016c26 (0xa39e bytes) 000451 1789674734.727978 mapread 0x0285cd thru 0x035e84 (0xd8b8 bytes) 000452 1789674734.747138 read 0x01e361 thru 0x02470d (0x63ad bytes) 000453 1789674734.747880 read 0x035a9d thru 0x03869e (0x2c02 bytes) 000454 1789674734.751915 mapread 0x014bea thru 0x0213bf (0xc7d6 bytes) 000455 1789674734.756610 trunc from 0x00e968 to 0x016f2b (0x85c4 bytes) 000456 1789674734.785801 MIRROR_RESYNC 5 mirrors 000457 1789674735.967804 write 0x02d26a thru 0x03c518 (0xf2af bytes) 000458 1789674736.014114 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c519): Mirror 3: 0x95da3824 (stale) Mirror 12: 0x95da3824 (stale) Mirror 13: 0x95da3824 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c519): Mirror 3: 0x95da3824 (stale) Mirror 12: 0x95da3824 (stale) Mirror 13: 0x95da3824 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c519: skipped 000459 1789674736.413909 MIRROR_RESYNC 4 mirrors 000460 1789674737.453619 MIRROR_RESYNC 4 mirrors 000461 1789674737.919757 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) 000462 1789674738.330347 write 0x03f766 thru 0x03ffff (0x89a bytes) 000464 1789674738.390696 read 0x00bb07 thru 0x011d17 (0x6211 bytes) 000465 1789674738.415660 mapwrite 0x03798f thru 0x039f5e (0x25d0 bytes) 000467 1789674738.489550 read 0x0003fc thru 0x00afd8 (0xabdd bytes) 000468 1789674738.517278 read 0x02feae thru 0x037f80 (0x80d3 bytes) 000469 1789674738.518876 mapwrite 0x02d7a9 thru 0x038a48 (0xb2a0 bytes) 000470 1789674738.555622 write 0x034a8b thru 0x03ffff (0xb575 bytes) 000471 1789674738.557259 write 0x02c3a1 thru 0x0303ec (0x404c bytes) 000472 1789674738.562538 mapwrite 0x036945 thru 0x03d584 (0x6c40 bytes) 000473 1789674738.605651 write 0x00538c thru 0x00f8e5 (0xa55a bytes) 000474 1789674738.608780 trunc from 0x040000 to 0x015786 (0x2a879 bytes) 000475 1789674738.660801 mapread 0x003a83 thru 0x005907 (0x1e85 bytes) 000476 1789674738.677078 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x15787): Mirror 13: 0x7b0e5bf5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15787): Mirror 13: 0x7b0e5bf5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15787: skipped 000477 1789674738.957774 read 0x013611 thru 0x015786 (0x2176 bytes) 000478 1789674738.963365 read 0x00ae39 thru 0x013c4c (0x8e14 bytes) 000479 1789674738.972266 write 0x037a69 thru 0x03ffff (0x8597 bytes) 000481 1789674738.978794 mapwrite 0x013863 thru 0x013d99 (0x537 bytes) 000482 1789674738.993148 mapread 0x038562 thru 0x03ffff (0x7a9e bytes) 000483 1789674738.994486 read 0x021845 thru 0x024602 (0x2dbe bytes) 000484 1789674739.010885 write 0x026905 thru 0x02819f (0x189b bytes) 000486 1789674739.016839 MIRROR_RESYNC 2 mirrors 000487 1789674739.763858 punch from 0x01ab78 to 0x028948 (0xddd1 bytes) 000488 1789674739.791927 read 0x0194f7 thru 0x023371 (0x9e7b bytes) 000489 1789674739.800173 mapwrite 0x01a99e thru 0x01ded0 (0x3533 bytes) 000490 1789674739.827548 write 0x02b2d6 thru 0x02b7ee (0x519 bytes) 000491 1789674739.844109 trunc from 0x037cce to 0x03ffff (0x8332 bytes) 000492 1789674739.857938 trunc from 0x039c79 to 0x03ffff (0x6387 bytes) 000493 1789674739.864002 fallocate 0x014423 thru 0x01eb68 (0xa746 bytes) 000494 1789674739.876444 punch from 0x00112d to 0x006922 (0x57f6 bytes) 000495 1789674739.893055 mapread 0x00b3e2 thru 0x013ce6 (0x8905 bytes) 000496 1789674739.945253 mapwrite 0x023c25 thru 0x02b8df (0x7cbb bytes) 000497 1789674739.997479 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) 000498 1789674740.182784 mapread 0x01fb0a thru 0x029bd0 (0xa0c7 bytes) 000499 1789674740.209209 mapread 0x00b4f4 thru 0x018b51 (0xd65e bytes) 000500 1789674740.232420 read 0x011f47 thru 0x01283c (0x8f6 bytes) 000501 1789674740.232952 mapwrite 0x027dcc thru 0x029792 (0x19c7 bytes) 000502 1789674740.247078 read 0x036a0d thru 0x03ffff (0x95f3 bytes) 000503 1789674740.260052 mapread 0x02f6dd thru 0x02feb2 (0x7d6 bytes) 000504 1789674740.261736 fallocate 0x02d95d thru 0x037cac (0xa350 bytes) 000505 1789674740.266796 fallocate 0x014993 thru 0x0202d2 (0xb940 bytes) 000506 1789674740.271841 mapwrite 0x008867 thru 0x00d5d4 (0x4d6e bytes) 000507 1789674740.290204 mapwrite 0x0108c3 thru 0x0165f2 (0x5d30 bytes) 000508 1789674740.321500 MIRROR_EXTEND to 2 mirrors 000509 1789674740.979988 mapread 0x01faff thru 0x02597f (0x5e81 bytes) 000510 1789674741.012959 trunc from 0x027d42 to 0x028c6a (0xf29 bytes) 000511 1789674741.057199 punch from 0x02d381 to 0x031677 (0x42f7 bytes) fallocating to largest ever: 0x40000 000512 1789674741.062957 fallocate 0x03f1a6 thru 0x03ffff (0xe5a bytes) 000513 1789674741.070062 fallocate 0x03f1e2 thru 0x03ffff (0xe1e bytes) 000514 1789674741.077259 trunc from 0x02dffe to 0x02ea23 (0xa26 bytes) 000515 1789674741.081882 fallocate 0x004da0 thru 0x00668a (0x18eb bytes) 000516 1789674741.089766 read 0x03e4a8 thru 0x03ffff (0x1b58 bytes) 000517 1789674741.097171 trunc from 0x040000 to 0x00d122 (0x32edd bytes) 000518 1789674741.157868 MIRROR_RESYNC 2 mirrors 000519 1789674741.639697 fallocate 0x003925 thru 0x0053dd (0x1ab9 bytes) 000521 1789674741.704731 write 0x03c4cd thru 0x03ee30 (0x2964 bytes) 000522 1789674741.706829 mapread 0x019a70 thru 0x022970 (0x8f01 bytes) 000523 1789674741.721764 trunc from 0x032fc9 to 0x03ee30 (0xbe68 bytes) 000524 1789674741.735588 MIRROR_RESYNC 2 mirrors 000525 1789674742.355209 mapread 0x03aacd thru 0x03ee30 (0x4364 bytes) 000526 1789674742.369759 write 0x00f4a9 thru 0x01c763 (0xd2bb bytes) 000527 1789674742.396963 trunc from 0x03ee31 to 0x012677 (0x2c7b9 bytes) 000528 1789674742.413179 trunc from 0x012678 to 0x012ac0 (0x449 bytes) 000530 1789674742.431732 trunc from 0x01205f to 0x012ac0 (0xa62 bytes) 000531 1789674742.443570 fallocate 0x02ac11 thru 0x036042 (0xb432 bytes) 000532 1789674742.448418 write 0x018d6d thru 0x0255d6 (0xc86a bytes) 000533 1789674742.452681 trunc from 0x00a806 to 0x0177f0 (0xcfeb bytes) 000534 1789674742.457812 trunc from 0x0245ce to 0x024caa (0x6dd bytes) 000535 1789674742.472482 trunc from 0x0255d7 to 0x00b887 (0x19d4f bytes) 000536 1789674742.486702 mapread 0x003d69 thru 0x00b887 (0x7b1f bytes) 000537 1789674742.495713 punch from 0x002426 to 0x006354 (0x3f2f bytes) 000538 1789674742.500861 mapread 0x0002b7 thru 0x00b887 (0xb5d1 bytes) 000539 1789674742.512022 mapwrite 0x017c66 thru 0x026ba7 (0xef42 bytes) 000540 1789674742.558561 write 0x03ac8f thru 0x03ffff (0x5371 bytes) 000541 1789674742.561071 fallocate 0x03b593 thru 0x03ffff (0x4a6d bytes) 000542 1789674742.570030 read 0x02048e thru 0x028168 (0x7cdb bytes) 000543 1789674742.574601 mapwrite 0x030ec9 thru 0x03f48a (0xe5c2 bytes) 000544 1789674742.604230 fallocate 0x0372f4 thru 0x03ffff (0x8d0c bytes) 000545 1789674742.607951 write 0x00a6f3 thru 0x00f6c0 (0x4fce bytes) 000546 1789674742.613258 trunc from 0x040000 to 0x02bf1b (0x140e4 bytes) 000547 1789674742.625631 write 0x02ba9f thru 0x02cd2f (0x1291 bytes) 000548 1789674742.626616 punch from 0x024dfc to 0x027c78 (0x2e7d bytes) 000549 1789674742.631832 fallocate 0x03862b thru 0x03ffff (0x79d5 bytes) 000550 1789674742.639114 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x6fae9e56 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xaa675344 Mirror 3: 0xaa675344 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000551 1789674743.549907 punch from 0x032445 to 0x0370fe (0x4cba bytes) 000552 1789674743.607740 write 0x0125be thru 0x019b20 (0x7563 bytes) 000553 1789674743.628742 read 0x00ad7a thru 0x00efd6 (0x425d bytes) 000554 1789674743.646158 mapwrite 0x000429 thru 0x00bb7c (0xb754 bytes) 000555 1789674743.729265 read 0x01982e thru 0x023bd9 (0xa3ac bytes) 000556 1789674743.751012 punch from 0x01de73 to 0x01f387 (0x1515 bytes) 000557 1789674743.764091 write 0x03ae8c thru 0x03ed13 (0x3e88 bytes) 000558 1789674743.767070 mapwrite 0x007991 thru 0x0164ca (0xeb3a bytes) 000559 1789674743.832113 read 0x01a99c thru 0x0291c6 (0xe82b bytes) 000560 1789674743.847291 trunc from 0x040000 to 0x00ce2d (0x331d2 bytes) 000561 1789674743.878749 fallocate 0x00813f thru 0x00d472 (0x5334 bytes) 000562 1789674743.894846 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0xd473): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd473): Mirror 2: 0xe6f714eb (stale) Mirror 3: 0x1ad7b1b6 (stale) CRC-32 checksum value for chunk [0, 0xd473): Mirror 1: 0xf38666f2 Mirror 4: 0xf38666f2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd473: skipped 000563 1789674744.790657 trunc from 0x00d473 to 0x039ce4 (0x2c872 bytes) 000564 1789674744.860662 trunc from 0x039ce5 to 0x00ffad (0x29d37 bytes) 000565 1789674744.903797 punch from 0x00e0b8 to 0x00ffad (0x1ef6 bytes) 000566 1789674744.908368 write 0x0340a8 thru 0x03a20b (0x6164 bytes) 000567 1789674744.912942 fallocate 0x011723 thru 0x017a83 (0x6361 bytes) 000568 1789674744.917740 read 0x00fc38 thru 0x0102f8 (0x6c1 bytes) 000569 1789674744.930494 MIRROR_RESYNC 4 mirrors 000570 1789674746.185379 write 0x013c78 thru 0x01b396 (0x771f bytes) 000571 1789674746.270846 trunc from 0x02a08a to 0x0389e5 (0xe95c bytes) 000572 1789674746.276834 trunc from 0x02be50 to 0x02d917 (0x1ac8 bytes) 000573 1789674746.297683 write 0x01630f thru 0x023d19 (0xda0b bytes) 000574 1789674746.322078 trunc from 0x0168fc to 0x019c85 (0x338a bytes) 000575 1789674746.369777 write 0x03c441 thru 0x03ffff (0x3bbf bytes) 000576 1789674746.371040 fallocate 0x00f5c1 thru 0x01dd33 (0xe773 bytes) 000577 1789674746.382567 read 0x00d4ad thru 0x01a479 (0xcfcd bytes) 000578 1789674746.468710 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xa580194a (stale) Mirror 3: 0xa580194a (stale) Mirror 4: 0xa580194a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5f91110d Mirror 5: 0x5f91110d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000579 1789674747.687966 trunc from 0x007687 to 0x016b9e (0xf518 bytes) 000580 1789674747.736350 read 0x02167b thru 0x026d0e (0x5694 bytes) 000581 1789674747.754805 trunc from 0x010a63 to 0x01a406 (0x99a4 bytes) 000582 1789674747.763101 trunc from 0x040000 to 0x02f0fe (0x10f01 bytes) 000583 1789674747.799910 punch from 0x0221b0 to 0x02f0fe (0xcf4f bytes) 000584 1789674747.814427 read 0x02a405 thru 0x02f0fe (0x4cfa bytes) 000585 1789674747.834019 fallocate 0x00dbad thru 0x00e791 (0xbe5 bytes) 000586 1789674747.845826 fallocate 0x036e0a thru 0x03ffff (0x91f6 bytes) 000587 1789674747.852780 read 0x03fbfc thru 0x03ffff (0x404 bytes) 000588 1789674747.867920 fallocate 0x03b8d7 thru 0x03ffff (0x4729 bytes) 000589 1789674747.872860 trunc from 0x040000 to 0x003128 (0x3ced7 bytes) 000590 1789674747.890127 punch from 0x0025ce to 0x003128 (0xb5b bytes) 000591 1789674747.895760 trunc from 0x001ec3 to 0x003128 (0x1266 bytes) 000592 1789674747.902547 trunc from 0x003129 to 0x029021 (0x25ef9 bytes) 000593 1789674747.920725 trunc from 0x029022 to 0x00122d (0x27df4 bytes) 000594 1789674747.944699 punch from 0x000ffc to 0x00122d (0x232 bytes) 000596 1789674747.958123 fallocate 0x01fd1e thru 0x02c9d3 (0xccb6 bytes) 000598 1789674747.974076 MIRROR_RESYNC 5 mirrors 000599 1789674748.855079 trunc from 0x01a819 to 0x01c6d3 (0x1ebb bytes) 000600 1789674748.887941 fallocate 0x026ecf thru 0x02b086 (0x41b8 bytes) 000601 1789674748.896397 read 0x00f5d2 thru 0x012d78 (0x37a7 bytes) 000602 1789674748.903723 fallocate 0x03b51c thru 0x03ffff (0x4ae4 bytes) 000603 1789674748.909664 trunc from 0x02c9d4 to 0x0172a9 (0x1572a bytes) 000604 1789674748.938907 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x172aa): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x172aa): Mirror 2: 0xd0add87 (stale) Mirror 3: 0xd0add87 (stale) Mirror 4: 0xd0add87 (stale) Mirror 5: 0xd0add87 (stale) CRC-32 checksum value for chunk [0, 0x172aa): Mirror 1: 0xd0add87 Mirror 6: 0xd0add87 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x172aa: skipped 000605 1789674750.002560 write 0x0300f2 thru 0x03ba37 (0xb946 bytes) 000606 1789674750.053921 write 0x03dda9 thru 0x03ec82 (0xeda bytes) 000607 1789674750.055963 fallocate 0x0281d6 thru 0x036923 (0xe74e bytes) 000608 1789674750.061816 read 0x02d52c thru 0x03b4c9 (0xdf9e bytes) 000609 1789674750.066272 write 0x0186b3 thru 0x0210bc (0x8a0a bytes) 000610 1789674750.076729 mapread 0x03ca78 thru 0x03df57 (0x14e0 bytes) 000612 1789674750.083788 fallocate 0x028b59 thru 0x0298f5 (0xd9d bytes) 000613 1789674750.105632 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3ec83): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ec83): Mirror 2: 0x90c33d2 (stale) Mirror 3: 0x90c33d2 (stale) Mirror 4: 0x90c33d2 (stale) Mirror 5: 0x90c33d2 (stale) Mirror 6: 0x90c33d2 (stale) CRC-32 checksum value for chunk [0, 0x3ec83): Mirror 1: 0x2348bdc9 Mirror 7: 0x2348bdc9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ec83: skipped 000614 1789674751.411447 fallocate 0x036f82 thru 0x03ffff (0x907e bytes) 000615 1789674751.444566 mapread 0x01b40c thru 0x0213d3 (0x5fc8 bytes) 000616 1789674751.461604 trunc from 0x018dbd to 0x01fd16 (0x6f5a bytes) 000617 1789674751.466569 read 0x033d1c thru 0x03ffff (0xc2e4 bytes) 000618 1789674751.480243 punch from 0x031e15 to 0x0396c6 (0x78b2 bytes) 000619 1789674751.486275 trunc from 0x03f516 to 0x03ffff (0xaea bytes) 000620 1789674751.490842 fallocate 0x0271ec thru 0x035bab (0xe9c0 bytes) 000621 1789674751.498574 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xea921fd6 (stale) Mirror 3: 0xea921fd6 (stale) Mirror 4: 0xea921fd6 (stale) Mirror 5: 0xea921fd6 (stale) Mirror 6: 0xea921fd6 (stale) Mirror 7: 0xa287e140 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc9c31873 Mirror 8: 0xc9c31873 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000623 1789674752.755295 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xea921fd6 (stale) Mirror 3: 0xea921fd6 (stale) Mirror 4: 0xea921fd6 (stale) Mirror 5: 0xea921fd6 (stale) Mirror 6: 0xea921fd6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xea921fd6 (stale) Mirror 3: 0xea921fd6 (stale) Mirror 4: 0xea921fd6 (stale) Mirror 5: 0xea921fd6 (stale) Mirror 6: 0xea921fd6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc9c31873 Mirror 8: 0xc9c31873 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000624 1789674753.624777 trunc from 0x040000 to 0x0293b4 (0x16c4b bytes) 000625 1789674753.678660 punch from 0x026c91 to 0x0293b4 (0x2724 bytes) 000626 1789674753.682379 trunc from 0x025119 to 0x0293b4 (0x429c bytes) 000627 1789674753.686908 fallocate 0x021890 thru 0x02c09a (0xa80b bytes) 000628 1789674753.691255 trunc from 0x020282 to 0x027440 (0x71bf bytes) 000629 1789674753.695880 write 0x032549 thru 0x03ffff (0xdab7 bytes) 000630 1789674753.697922 mapread 0x00a385 thru 0x011d20 (0x799c bytes) 000631 1789674753.716022 MIRROR_RESYNC 7 mirrors 000632 1789674755.453050 trunc from 0x040000 to 0x00c51c (0x33ae3 bytes) 000633 1789674755.492023 punch from 0x0019fd to 0x0087a1 (0x6da5 bytes) 000634 1789674755.496848 punch from 0x004864 to 0x00c51c (0x7cb9 bytes) 000636 1789674755.501604 trunc from 0x00c51d to 0x032607 (0x260eb bytes) 000637 1789674755.521064 punch from 0x016ee2 to 0x01e93d (0x7a5c bytes) 000638 1789674755.525608 mapread 0x013fdb thru 0x0178a8 (0x38ce bytes) 000639 1789674755.542283 punch from 0x00737f to 0x00e9dd (0x765f bytes) 000640 1789674755.546687 trunc from 0x030f3d to 0x032607 (0x16cb bytes) 000641 1789674755.551072 fallocate 0x01bdfb thru 0x0242ba (0x84c0 bytes) 000642 1789674755.555544 write 0x03a0ac thru 0x03ffff (0x5f54 bytes) 000643 1789674755.557219 write 0x00d489 thru 0x017fce (0xab46 bytes) 000644 1789674755.564395 read 0x016697 thru 0x01a345 (0x3caf bytes) 000645 1789674755.565188 fallocate 0x03b851 thru 0x03ffff (0x47af bytes) 000646 1789674755.569911 mapread 0x0353b7 thru 0x03875a (0x33a4 bytes) 000647 1789674755.578694 trunc from 0x03b1db to 0x03ffff (0x4e25 bytes) 000648 1789674755.591737 trunc from 0x0004f9 to 0x00cd4b (0xc853 bytes) 000649 1789674755.596160 trunc from 0x040000 to 0x0385f3 (0x7a0c bytes) 000650 1789674755.617886 punch from 0x032c1c to 0x0385f3 (0x59d8 bytes) 000651 1789674755.622914 write 0x020d2e thru 0x02e0aa (0xd37d bytes) 000652 1789674755.629897 write 0x021c0a thru 0x024348 (0x273f bytes) 000653 1789674755.631046 trunc from 0x029866 to 0x02d7f9 (0x3f94 bytes) 000654 1789674755.650360 read 0x001cef thru 0x009e16 (0x8128 bytes) 000655 1789674755.673142 trunc from 0x030f0d to 0x0385f3 (0x76e7 bytes) 000656 1789674755.677982 punch from 0x0100d4 to 0x0108e5 (0x812 bytes) 000657 1789674755.689030 write 0x00f1b1 thru 0x016c52 (0x7aa2 bytes) 000658 1789674755.692168 trunc from 0x0385f4 to 0x00171a (0x36ed9 bytes) 000659 1789674755.727283 trunc from 0x00171b to 0x020d70 (0x1f656 bytes) 000660 1789674755.747351 punch from 0x011d7f to 0x0122d2 (0x554 bytes) 000661 1789674755.754042 fallocate 0x01079d thru 0x01a981 (0xa1e5 bytes) 000662 1789674755.768463 write 0x008872 thru 0x016f7a (0xe709 bytes) 000663 1789674755.781957 mapread 0x01d951 thru 0x020d70 (0x3420 bytes) 000664 1789674755.814710 mapread 0x01c9ea thru 0x020a1a (0x4031 bytes) 000666 1789674755.827067 trunc from 0x020d71 to 0x01a7aa (0x65c6 bytes) 000667 1789674755.845490 mapread 0x004eda thru 0x00ffa0 (0xb0c7 bytes) 000669 1789674755.864642 mapread 0x007a43 thru 0x00e63f (0x6bfd bytes) 000670 1789674755.865774 punch from 0x0050c6 to 0x00c478 (0x73b3 bytes) 000672 1789674755.900869 trunc from 0x01a7ab to 0x01b632 (0xe88 bytes) 000673 1789674755.928597 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x1b633): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1b633): Mirror 2: 0x2f82cab8 (stale) Mirror 3: 0x2f82cab8 (stale) Mirror 4: 0x2f82cab8 (stale) Mirror 5: 0x2f82cab8 (stale) Mirror 6: 0x2f82cab8 (stale) Mirror 8: 0x2f82cab8 (stale) CRC-32 checksum value for chunk [0, 0x1b633): Mirror 1: 0xb6296221 Mirror 9: 0xb6296221 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1b633: skipped 000674 1789674756.706288 mapread 0x009feb thru 0x018a11 (0xea27 bytes) 000675 1789674756.746409 trunc from 0x01b633 to 0x01e885 (0x3253 bytes) 000676 1789674756.793045 MIRROR_RESYNC 8 mirrors 000677 1789674758.046842 fallocate 0x02d1af thru 0x032c27 (0x5a79 bytes) 000678 1789674758.074970 trunc from 0x032c28 to 0x0234cb (0xf75c bytes) 000679 1789674758.090032 mapwrite 0x0004fd thru 0x00d2b6 (0xcdba bytes) 000680 1789674758.123649 trunc from 0x0234cc to 0x017251 (0xc27a bytes) 000681 1789674758.135095 trunc from 0x017252 to 0x0272c0 (0x1006f bytes) 000682 1789674758.153094 MIRROR_RESYNC 8 mirrors 000683 1789674759.670788 trunc from 0x0272c1 to 0x02a7a6 (0x34e6 bytes) 000684 1789674759.783992 mapread 0x010ba6 thru 0x01664b (0x5aa6 bytes) 000685 1789674759.848735 mapwrite 0x017ec8 thru 0x01a275 (0x23ae bytes) 000686 1789674759.918150 fallocate 0x03f02f thru 0x03ffff (0xfd1 bytes) 000687 1789674759.927191 trunc from 0x02a7a7 to 0x01d990 (0xce16 bytes) 000689 1789674759.982720 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d991): Mirror 2: 0xdb9f1f58 (stale) Mirror 3: 0xdb9f1f58 (stale) Mirror 4: 0xdb9f1f58 (stale) Mirror 6: 0xdb9f1f58 (stale) Mirror 8: 0xdb9f1f58 (stale) Mirror 9: 0xdb9f1f58 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d991): Mirror 2: 0xdb9f1f58 (stale) Mirror 3: 0xdb9f1f58 (stale) Mirror 4: 0xdb9f1f58 (stale) Mirror 6: 0xdb9f1f58 (stale) Mirror 8: 0xdb9f1f58 (stale) Mirror 9: 0xdb9f1f58 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d991: skipped 000690 1789674760.833421 trunc from 0x017f29 to 0x01d990 (0x5a68 bytes) 000691 1789674760.843048 write 0x030157 thru 0x037704 (0x75ae bytes) 000692 1789674760.848836 trunc from 0x017d0f to 0x01cc64 (0x4f56 bytes) 000693 1789674760.856756 trunc from 0x037705 to 0x003909 (0x33dfb bytes) 000694 1789674760.878858 fallocate 0x01d9a4 thru 0x025fb4 (0x8611 bytes) 000695 1789674760.882932 read 0x00a56c thru 0x00c2fa (0x1d8f bytes) 000696 1789674760.892237 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x25fb5): Mirror 3: 0xd3c7cfc5 (stale) Mirror 4: 0xd3c7cfc5 (stale) Mirror 6: 0xd3c7cfc5 (stale) Mirror 8: 0xd3c7cfc5 (stale) Mirror 9: 0xd3c7cfc5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25fb5): Mirror 3: 0xd3c7cfc5 (stale) Mirror 4: 0xd3c7cfc5 (stale) Mirror 6: 0xd3c7cfc5 (stale) Mirror 8: 0xd3c7cfc5 (stale) Mirror 9: 0xd3c7cfc5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25fb5: skipped 000697 1789674761.347792 mapwrite 0x02b783 thru 0x036619 (0xae97 bytes) 000698 1789674761.448898 trunc from 0x03661a to 0x00fdd8 (0x26841 bytes) 000699 1789674761.497741 mapwrite 0x030397 thru 0x03af22 (0xab8c bytes) 000700 1789674761.554986 trunc from 0x03af23 to 0x00a007 (0x30f1b bytes) 000701 1789674761.575045 read 0x009f8c thru 0x00a007 (0x7c bytes) 000702 1789674761.587516 write 0x013d47 thru 0x01b8d6 (0x7b90 bytes) 000703 1789674761.590049 trunc from 0x01b8d7 to 0x0290d4 (0xd7fe bytes) 000704 1789674761.622083 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x290d5): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x290d5): Mirror 3: 0x66064988 (stale) Mirror 4: 0x66064988 (stale) Mirror 6: 0x66064988 (stale) Mirror 8: 0x66064988 (stale) Mirror 9: 0x66064988 (stale) CRC-32 checksum value for chunk [0, 0x290d5): Mirror 1: 0x1506a9f8 Mirror 10: 0x1506a9f8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x290d5: skipped 000705 1789674763.077272 mapwrite 0x004bf0 thru 0x008d80 (0x4191 bytes) 000706 1789674763.265066 fallocate 0x0322d3 thru 0x03808e (0x5dbc bytes) 000707 1789674763.281915 trunc from 0x03808f to 0x0317e6 (0x68a8 bytes) 000708 1789674763.324062 punch from 0x01d3f0 to 0x020146 (0x2d57 bytes) 000709 1789674763.329411 mapwrite 0x038918 thru 0x03ffff (0x76e8 bytes) 000712 1789674763.450920 trunc from 0x040000 to 0x038a63 (0x759c bytes) 000713 1789674763.489351 mapread 0x00a1fa thru 0x00fbcb (0x59d2 bytes) 000714 1789674763.492239 mapwrite 0x019106 thru 0x027d5d (0xec58 bytes) 000715 1789674763.590555 write 0x002a2c thru 0x0069ef (0x3fc4 bytes) 000716 1789674763.593018 read 0x037204 thru 0x037d76 (0xb73 bytes) 000717 1789674763.596255 read 0x01683e thru 0x0232a4 (0xca67 bytes) 000718 1789674763.609080 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x38a64): Mirror 3: 0x46c0b9ca (stale) Mirror 6: 0x46c0b9ca (stale) Mirror 8: 0x46c0b9ca (stale) Mirror 9: 0x46c0b9ca (stale) Mirror 10: 0xd870dc7b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a64): Mirror 3: 0x46c0b9ca (stale) Mirror 6: 0x46c0b9ca (stale) Mirror 8: 0x46c0b9ca (stale) Mirror 9: 0x46c0b9ca (stale) Mirror 10: 0xd870dc7b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a64: skipped 000719 1789674764.071891 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38a64): Mirror 3: (stale - read error) Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38a64): Mirror 3: 0x46c0b9ca (stale) Mirror 6: 0x46c0b9ca (stale) Mirror 8: 0x46c0b9ca (stale) Mirror 9: 0x46c0b9ca (stale) Mirror 10: 0xd870dc7b (stale) CRC-32 checksum value for chunk [0, 0x38a64): Mirror 1: 0xf333e647 Mirror 11: 0xf333e647 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38a64: skipped 000720 1789674765.189400 write 0x02561c thru 0x028ac7 (0x34ac bytes) 000721 1789674765.283406 mapread 0x01018b thru 0x0106ac (0x522 bytes) 000722 1789674765.291659 punch from 0x01796d to 0x01e1ee (0x6882 bytes) 000723 1789674765.297564 mapwrite 0x01e22f thru 0x029bca (0xb99c bytes) 000724 1789674765.394924 trunc from 0x01cf71 to 0x026148 (0x91d8 bytes) 000725 1789674765.399968 mapread 0x0200cb thru 0x029f6e (0x9ea4 bytes) 000726 1789674765.411832 write 0x0329a7 thru 0x036529 (0x3b83 bytes) 000727 1789674765.426338 mapread 0x010608 thru 0x02003e (0xfa37 bytes) 000728 1789674765.441088 write 0x016e1e thru 0x022997 (0xbb7a bytes) 000730 1789674765.444926 trunc from 0x038a64 to 0x02e897 (0xa1cc bytes) 000731 1789674765.473850 punch from 0x013f1c to 0x016469 (0x254e bytes) 000732 1789674765.494120 fallocate 0x01c853 thru 0x0246cf (0x7e7d bytes) 000733 1789674765.501044 fallocate 0x01ab5f thru 0x023473 (0x8915 bytes) 000734 1789674765.507116 fallocate 0x004f54 thru 0x00925f (0x430c bytes) 000735 1789674765.516345 read 0x00734f thru 0x009a38 (0x26ea bytes) 000736 1789674765.522336 mapwrite 0x026fdf thru 0x02d3c8 (0x63ea bytes) 000737 1789674765.552605 write 0x014f9a thru 0x021570 (0xc5d7 bytes) 000738 1789674765.560065 mapwrite 0x009cd2 thru 0x018b37 (0xee66 bytes) 000739 1789674765.612634 trunc from 0x0000c3 to 0x00c6e5 (0xc623 bytes) 000740 1789674765.618077 write 0x028a08 thru 0x038393 (0xf98c bytes) 000741 1789674765.626252 trunc from 0x038394 to 0x027bbc (0x107d7 bytes) 000742 1789674765.654656 read 0x001850 thru 0x003521 (0x1cd2 bytes) 000743 1789674765.669760 mapread 0x00f6d4 thru 0x015b84 (0x64b1 bytes) 000744 1789674765.671073 fallocate 0x00c630 thru 0x012d9e (0x676f bytes) 000745 1789674765.692142 write 0x03cbfc thru 0x03dd5e (0x1163 bytes) 000746 1789674765.693959 mapread 0x0113e5 thru 0x01da75 (0xc691 bytes) 000747 1789674765.696694 mapread 0x002a77 thru 0x00519f (0x2729 bytes) 000748 1789674765.698647 punch from 0x03cd91 to 0x03dd5e (0xfce bytes) 000749 1789674765.737788 mapread 0x01d537 thru 0x02463e (0x7108 bytes) 000751 1789674765.742964 read 0x0159ef thru 0x024837 (0xee49 bytes) 000752 1789674765.743700 trunc from 0x002f84 to 0x010e91 (0xdf0e bytes) 000753 1789674765.772562 write 0x027e00 thru 0x02f4f9 (0x76fa bytes) 000754 1789674765.828453 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3dd5f): Mirror 6: 0xa0bcd682 (stale) Mirror 8: 0xa0bcd682 (stale) Mirror 9: 0xa0bcd682 (stale) Mirror 10: 0xcb1e192e (stale) Mirror 11: 0xe25fe24b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd5f): Mirror 6: 0xa0bcd682 (stale) Mirror 8: 0xa0bcd682 (stale) Mirror 9: 0xa0bcd682 (stale) Mirror 10: 0xcb1e192e (stale) Mirror 11: 0xe25fe24b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd5f: skipped 000755 1789674766.377478 punch from 0x0052d3 to 0x005e0d (0xb3b bytes) 000756 1789674766.387250 mapread 0x03d869 thru 0x03dd5e (0x4f6 bytes) 000757 1789674766.395787 punch from 0x013d3c to 0x0225a5 (0xe86a bytes) 000758 1789674766.401376 trunc from 0x03dd5f to 0x0188ff (0x2545f bytes) 000759 1789674766.432006 mapwrite 0x002e66 thru 0x006012 (0x31ad bytes) 000760 1789674766.458508 fallocate 0x017413 thru 0x025281 (0xde6f bytes) 000761 1789674766.463600 mapwrite 0x032813 thru 0x038e61 (0x664f bytes) 000762 1789674766.506271 trunc from 0x010307 to 0x013a0e (0x3708 bytes) 000763 1789674766.511766 trunc from 0x017344 to 0x01cf20 (0x5bdd bytes) 000764 1789674766.521949 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x38e62): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x38e62): Mirror 6: 0x422673e5 (stale) Mirror 8: 0x422673e5 (stale) Mirror 9: 0x422673e5 (stale) Mirror 10: 0x4b93282f (stale) Mirror 11: 0x8cb0eb2c (stale) CRC-32 checksum value for chunk [0, 0x38e62): Mirror 1: 0xad4aec94 Mirror 12: 0xad4aec94 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x38e62: skipped 000765 1789674767.563920 trunc from 0x038e62 to 0x013002 (0x25e5f bytes) 000767 1789674767.629516 write 0x015126 thru 0x02303a (0xdf15 bytes) 000768 1789674767.639742 MIRROR_RESYNC 7 mirrors 000771 1789674770.118168 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) 000772 1789674770.578135 write 0x006caa thru 0x010dcf (0xa126 bytes) 000773 1789674770.648506 read 0x010b01 thru 0x019652 (0x8b52 bytes) 000775 1789674770.676224 trunc from 0x02303b to 0x020d58 (0x22e2 bytes) 000776 1789674770.707116 trunc from 0x020d59 to 0x016788 (0xa5d0 bytes) 000777 1789674770.737807 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x16789): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x16789): Mirror 6: 0x8bf3a79a (stale) Mirror 8: 0x8bf3a79a (stale) Mirror 9: 0x8bf3a79a (stale) Mirror 10: 0x8bf3a79a (stale) Mirror 12: 0x8bf3a79a (stale) CRC-32 checksum value for chunk [0, 0x16789): Mirror 1: 0x391418d9 Mirror 13: 0x391418d9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x16789: skipped 000779 1789674771.886574 fallocate 0x0212b8 thru 0x02f0c1 (0xde0a bytes) 000780 1789674771.950009 write 0x033b85 thru 0x03ffff (0xc47b bytes) 000781 1789674771.955003 write 0x002ed4 thru 0x00708d (0x41ba bytes) 000782 1789674771.975019 mapwrite 0x015071 thru 0x01ecf9 (0x9c89 bytes) 000783 1789674772.042212 punch from 0x03d4c2 to 0x03ffff (0x2b3e bytes) 000784 1789674772.065877 read 0x00afd5 thru 0x013baf (0x8bdb bytes) 000785 1789674772.079129 write 0x01f3c6 thru 0x024b21 (0x575c bytes) 000786 1789674772.085633 write 0x00cfd2 thru 0x00f649 (0x2678 bytes) 000787 1789674772.087649 fallocate 0x014022 thru 0x021dff (0xddde bytes) 000788 1789674772.092638 trunc from 0x040000 to 0x031ccd (0xe332 bytes) 000789 1789674772.113329 trunc from 0x031cce to 0x004a47 (0x2d286 bytes) 000790 1789674772.127153 punch from 0x0010e9 to 0x004a47 (0x395f bytes) 000791 1789674772.133172 trunc from 0x004a48 to 0x024463 (0x1fa1c bytes) 000792 1789674772.155688 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x24464): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24464): Mirror 6: 0x314b13ca (stale) Mirror 8: 0x314b13ca (stale) Mirror 9: 0x314b13ca (stale) Mirror 10: 0x314b13ca (stale) Mirror 12: 0x314b13ca (stale) Mirror 13: 0xb2f5b1a3 (stale) CRC-32 checksum value for chunk [0, 0x24464): Mirror 1: 0xf7a5ba6e Mirror 14: 0xf7a5ba6e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24464: skipped 000793 1789674772.927918 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x24464): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24464): Mirror 6: 0x314b13ca (stale) Mirror 8: 0x314b13ca (stale) Mirror 9: 0x314b13ca (stale) Mirror 10: 0x314b13ca (stale) Mirror 12: 0x314b13ca (stale) Mirror 13: 0xb2f5b1a3 (stale) CRC-32 checksum value for chunk [0, 0x24464): Mirror 1: 0xf7a5ba6e Mirror 14: 0xf7a5ba6e Mirror 15: 0xf7a5ba6e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24464: skipped 000794 1789674773.977124 punch from 0x01eb80 to 0x02396d (0x4dee bytes) 000795 1789674774.026251 punch from 0x01766f to 0x01a628 (0x2fba bytes) 000796 1789674774.032633 trunc from 0x024464 to 0x01f5ca (0x4e99 bytes) 000797 1789674774.095757 MIRROR_RESYNC 9 mirrors 000799 1789674775.841788 write 0x002ad0 thru 0x005a77 (0x2fa8 bytes) 000800 1789674775.879109 write 0x00cf40 thru 0x00e8b2 (0x1973 bytes) 000801 1789674775.888759 read 0x01799c thru 0x01f5ca (0x7c2f bytes) 000803 1789674775.898704 write 0x017c94 thru 0x01850a (0x877 bytes) 000804 1789674775.899908 fallocate 0x037cb3 thru 0x03c66b (0x49b9 bytes) 000805 1789674775.904417 fallocate 0x00c0a1 thru 0x0134af (0x740f bytes) 000806 1789674775.911893 read 0x015f81 thru 0x018a6a (0x2aea bytes) 000808 1789674775.930922 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3c66c): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c66c): Mirror 6: 0xb25a1d98 (stale) Mirror 8: 0xb25a1d98 (stale) Mirror 9: 0xb25a1d98 (stale) Mirror 10: 0xb25a1d98 (stale) Mirror 12: 0xb25a1d98 (stale) Mirror 13: 0xb25a1d98 (stale) Mirror 14: 0xb25a1d98 (stale) Mirror 15: 0xb25a1d98 (stale) CRC-32 checksum value for chunk [0, 0x3c66c): Mirror 1: 0xd1308dd6 Mirror 16: 0xd1308dd6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c66c: skipped 000809 1789674776.963785 punch from 0x0070f0 to 0x00d73f (0x6650 bytes) 000810 1789674777.023545 read 0x00bb3e thru 0x01704e (0xb511 bytes) 000811 1789674777.037453 fallocate 0x02bf01 thru 0x0352a3 (0x93a3 bytes) 000812 1789674777.041690 mapwrite 0x023d5c thru 0x0292dd (0x5582 bytes) 000813 1789674777.088874 fallocate 0x021a93 thru 0x027dd9 (0x6347 bytes) 000814 1789674777.095898 punch from 0x0316e6 to 0x03c66b (0xaf86 bytes) 000815 1789674777.104886 write 0x01022e thru 0x01f7c9 (0xf59c bytes) 000816 1789674777.113498 trunc from 0x003959 to 0x00c352 (0x89fa bytes) 000817 1789674777.120517 write 0x025f5d thru 0x03050b (0xa5af bytes) 000818 1789674777.131752 trunc from 0x00ba2e to 0x017fa5 (0xc578 bytes) 000820 1789674777.174495 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3c66c): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c66c): Mirror 6: 0xb25a1d98 (stale) Mirror 8: 0xb25a1d98 (stale) Mirror 9: 0xb25a1d98 (stale) Mirror 10: 0xb25a1d98 (stale) Mirror 12: 0xb25a1d98 (stale) Mirror 13: 0xb25a1d98 (stale) Mirror 14: 0xb25a1d98 (stale) Mirror 15: 0xb25a1d98 (stale) Mirror 16: 0xd1308dd6 (stale) CRC-32 checksum value for chunk [0, 0x3c66c): Mirror 1: 0xbd159061 Mirror 17: 0xbd159061 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c66c: skipped 000821 1789674778.346621 MIRROR_RESYNC 11 mirrors 000822 1789674781.238603 fallocate 0x031df8 thru 0x033649 (0x1852 bytes) 000824 1789674781.310912 read 0x02babf thru 0x03944b (0xd98d bytes) 000825 1789674781.320805 trunc from 0x03c66c to 0x01ced3 (0x1f798 bytes) 000826 1789674781.370874 mapwrite 0x01caeb thru 0x0267c8 (0x9cde bytes) 000827 1789674781.416793 mapwrite 0x00132f thru 0x00af65 (0x9c37 bytes) 000828 1789674781.447497 mapwrite 0x039b8f thru 0x03ffff (0x6471 bytes) 000829 1789674781.484517 trunc from 0x01a74c to 0x01ecd4 (0x4589 bytes) 000831 1789674781.494061 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8d500732 (stale) Mirror 8: 0x8d500732 (stale) Mirror 9: 0x8d500732 (stale) Mirror 10: 0x8d500732 (stale) Mirror 12: 0x8d500732 (stale) Mirror 13: 0x8d500732 (stale) Mirror 14: 0x8d500732 (stale) Mirror 15: 0x8d500732 (stale) Mirror 16: 0x8d500732 (stale) Mirror 17: 0x8d500732 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa7c618ee Mirror 18: 0xa7c618ee lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000832 1789674782.721524 read 0x03491f thru 0x039a3e (0x5120 bytes) 000833 1789674782.737876 mapread 0x03f0d9 thru 0x03ffff (0xf27 bytes) 000834 1789674782.756417 punch from 0x0352e9 to 0x03cfba (0x7cd2 bytes) 000835 1789674782.803544 write 0x026261 thru 0x029590 (0x3330 bytes) 000836 1789674782.821077 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8d500732 (stale) Mirror 8: 0x8d500732 (stale) Mirror 9: 0x8d500732 (stale) Mirror 10: 0x8d500732 (stale) Mirror 12: 0x8d500732 (stale) Mirror 13: 0x8d500732 (stale) Mirror 14: 0x8d500732 (stale) Mirror 15: 0x8d500732 (stale) Mirror 16: 0x8d500732 (stale) Mirror 17: 0x8d500732 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x8d500732 (stale) Mirror 8: 0x8d500732 (stale) Mirror 9: 0x8d500732 (stale) Mirror 10: 0x8d500732 (stale) Mirror 12: 0x8d500732 (stale) Mirror 13: 0x8d500732 (stale) Mirror 14: 0x8d500732 (stale) Mirror 15: 0x8d500732 (stale) Mirror 16: 0x8d500732 (stale) Mirror 17: 0x8d500732 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000837 1789674783.505130 fallocate 0x00f53b thru 0x016921 (0x73e7 bytes) 000838 1789674783.510913 trunc from 0x039fce to 0x03ef0a (0x4f3d bytes) 000840 1789674783.517920 punch from 0x03850e to 0x03fc41 (0x7734 bytes) 000841 1789674783.524982 trunc from 0x040000 to 0x00b6cd (0x34932 bytes) 000842 1789674783.561010 read 0x001674 thru 0x00b6cd (0xa05a bytes) 000843 1789674783.572599 fallocate 0x03101d thru 0x036d6e (0x5d52 bytes) 000844 1789674783.577482 punch from 0x007f8b to 0x00b6cd (0x3743 bytes) 000845 1789674783.581964 mapwrite 0x02da8c thru 0x034bc0 (0x7135 bytes) 000846 1789674783.623773 write 0x00db44 thru 0x00ea72 (0xf2f bytes) 000847 1789674783.631756 mapwrite 0x00d560 thru 0x00f741 (0x21e2 bytes) 000848 1789674783.655714 fallocate 0x01582f thru 0x022971 (0xd143 bytes) 000849 1789674783.659407 trunc from 0x034bc1 to 0x03e3b5 (0x97f5 bytes) 000850 1789674783.686748 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3e3b6): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e3b6): Mirror 6: 0xb99054d8 (stale) Mirror 8: 0xb99054d8 (stale) Mirror 9: 0xb99054d8 (stale) Mirror 10: 0xb99054d8 (stale) Mirror 12: 0xb99054d8 (stale) Mirror 13: 0xb99054d8 (stale) Mirror 14: 0xb99054d8 (stale) Mirror 15: 0xb99054d8 (stale) Mirror 16: 0xb99054d8 (stale) Mirror 17: 0xb99054d8 (stale) CRC-32 checksum value for chunk [0, 0x3e3b6): Mirror 1: 0xeb2a3794 Mirror 18: 0xeb2a3794 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e3b6: skipped 000853 1789674784.847381 trunc from 0x03e3b6 to 0x008b50 (0x35865 bytes) 000854 1789674784.900446 trunc from 0x006055 to 0x007891 (0x183d bytes) 000855 1789674784.905521 trunc from 0x004026 to 0x008b50 (0x4b2b bytes) 000856 1789674784.909820 trunc from 0x003203 to 0x008b50 (0x594e bytes) 000858 1789674784.914813 punch from 0x001a6d to 0x00453d (0x2ad1 bytes) 000859 1789674784.919050 write 0x0162e4 thru 0x01f9e5 (0x9702 bytes) 000860 1789674784.920905 trunc from 0x01f9e6 to 0x01d3d9 (0x260c bytes) 000861 1789674784.938573 read 0x000abf thru 0x0045ff (0x3b41 bytes) 000862 1789674784.947740 MIRROR_SPLIT mirror 8 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1d3da): Mirror 6: 0x24119e6b (stale) Mirror 8: 0x24119e6b (stale) Mirror 9: 0x24119e6b (stale) Mirror 10: 0x24119e6b (stale) Mirror 12: 0x24119e6b (stale) Mirror 13: 0x24119e6b (stale) Mirror 14: 0x24119e6b (stale) Mirror 16: 0x24119e6b (stale) Mirror 17: 0x24119e6b (stale) Mirror 18: 0x17a8c0f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1d3da): Mirror 6: 0x24119e6b (stale) Mirror 8: 0x24119e6b (stale) Mirror 9: 0x24119e6b (stale) Mirror 10: 0x24119e6b (stale) Mirror 12: 0x24119e6b (stale) Mirror 13: 0x24119e6b (stale) Mirror 14: 0x24119e6b (stale) Mirror 16: 0x24119e6b (stale) Mirror 17: 0x24119e6b (stale) Mirror 18: 0x17a8c0f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1d3da: skipped 000863 1789674785.403824 punch from 0x00368d to 0x00ea11 (0xb385 bytes) 000864 1789674785.409766 fallocate 0x00232b thru 0x011779 (0xf44f bytes) 000867 1789674785.416243 mapread 0x0030bb thru 0x009321 (0x6267 bytes) 000868 1789674785.439738 mapwrite 0x01a0c3 thru 0x027140 (0xd07e bytes) 000869 1789674785.496256 trunc from 0x027141 to 0x01478c (0x129b4 bytes) 000870 1789674785.513909 MIRROR_RESYNC 11 mirrors 000871 1789674787.103733 mapread 0x0049f8 thru 0x006b46 (0x214f bytes) 000872 1789674787.127831 write 0x0343a1 thru 0x03cb47 (0x87a7 bytes) 000873 1789674787.178244 punch from 0x00d1ee to 0x00f308 (0x211b bytes) 000874 1789674787.202723 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3cb48): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb48): Mirror 6: 0x871cf572 (stale) Mirror 8: 0x871cf572 (stale) Mirror 9: 0x871cf572 (stale) Mirror 10: 0x871cf572 (stale) Mirror 12: 0x871cf572 (stale) Mirror 13: 0x871cf572 (stale) Mirror 14: 0x871cf572 (stale) Mirror 16: 0x871cf572 (stale) Mirror 17: 0x871cf572 (stale) Mirror 18: 0x871cf572 (stale) CRC-32 checksum value for chunk [0, 0x3cb48): Mirror 1: 0x8068e159 Mirror 19: 0x8068e159 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb48: skipped 000875 1789674788.178966 mapwrite 0x0252f7 thru 0x030ac7 (0xb7d1 bytes) 000876 1789674788.359222 write 0x0150df thru 0x01758b (0x24ad bytes) 000877 1789674788.374706 read 0x018657 thru 0x0243c7 (0xbd71 bytes) 000878 1789674788.389765 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cb48): Mirror 6: 0x871cf572 (stale) Mirror 8: 0x871cf572 (stale) Mirror 9: 0x871cf572 (stale) Mirror 10: 0x871cf572 (stale) Mirror 12: 0x871cf572 (stale) Mirror 13: 0x871cf572 (stale) Mirror 14: 0x871cf572 (stale) Mirror 16: 0x871cf572 (stale) Mirror 17: 0x871cf572 (stale) Mirror 18: 0x871cf572 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb48): Mirror 6: 0x871cf572 (stale) Mirror 8: 0x871cf572 (stale) Mirror 9: 0x871cf572 (stale) Mirror 10: 0x871cf572 (stale) Mirror 12: 0x871cf572 (stale) Mirror 13: 0x871cf572 (stale) Mirror 14: 0x871cf572 (stale) Mirror 16: 0x871cf572 (stale) Mirror 17: 0x871cf572 (stale) Mirror 18: 0x871cf572 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb48: skipped 000880 1789674788.868977 write 0x005088 thru 0x006673 (0x15ec bytes) 000881 1789674788.880582 write 0x015982 thru 0x016f22 (0x15a1 bytes) 000882 1789674788.887885 mapread 0x03a2de thru 0x03cb47 (0x286a bytes) 000883 1789674788.902375 punch from 0x02e1cb to 0x039053 (0xae89 bytes) 000884 1789674788.911149 MIRROR_SPLIT mirror 8 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cb48): Mirror 6: 0x871cf572 (stale) Mirror 8: 0x871cf572 (stale) Mirror 9: 0x871cf572 (stale) Mirror 10: 0x871cf572 (stale) Mirror 12: 0x871cf572 (stale) Mirror 13: 0x871cf572 (stale) Mirror 14: 0x871cf572 (stale) Mirror 17: 0x871cf572 (stale) Mirror 18: 0x871cf572 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb48): Mirror 6: 0x871cf572 (stale) Mirror 8: 0x871cf572 (stale) Mirror 9: 0x871cf572 (stale) Mirror 10: 0x871cf572 (stale) Mirror 12: 0x871cf572 (stale) Mirror 13: 0x871cf572 (stale) Mirror 14: 0x871cf572 (stale) Mirror 17: 0x871cf572 (stale) Mirror 18: 0x871cf572 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb48: skipped 000885 1789674789.468219 mapwrite 0x00ccac thru 0x015a00 (0x8d55 bytes) 000886 1789674789.520801 mapread 0x008f90 thru 0x010619 (0x768a bytes) 000887 1789674789.522794 read 0x00b830 thru 0x00f14f (0x3920 bytes) 000888 1789674789.531652 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3cb48): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cb48): Mirror 6: 0x871cf572 (stale) Mirror 8: 0x871cf572 (stale) Mirror 9: 0x871cf572 (stale) Mirror 10: 0x871cf572 (stale) Mirror 12: 0x871cf572 (stale) Mirror 13: 0x871cf572 (stale) Mirror 14: 0x871cf572 (stale) Mirror 17: 0x871cf572 (stale) Mirror 18: 0x871cf572 (stale) CRC-32 checksum value for chunk [0, 0x3cb48): Mirror 1: 0x44d5b0a9 Mirror 19: 0x44d5b0a9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cb48: skipped 000890 1789674790.565412 write 0x00b629 thru 0x0196a4 (0xe07c bytes) 000891 1789674790.626497 read 0x00b561 thru 0x018c28 (0xd6c8 bytes) 000892 1789674790.632702 mapread 0x023e8e thru 0x02fb67 (0xbcda bytes) 000894 1789674790.706240 trunc from 0x03cb48 to 0x0314db (0xb66c bytes) 000895 1789674790.750556 punch from 0x022c2f to 0x02b3cb (0x879d bytes) 000896 1789674790.755142 read 0x022d31 thru 0x024da1 (0x2071 bytes) 000897 1789674790.782592 fallocate 0x00a6a6 thru 0x00c2f7 (0x1c52 bytes) 000899 1789674790.790659 punch from 0x00818d to 0x012147 (0x9fbb bytes) 000900 1789674790.807545 mapread 0x0252a1 thru 0x02a41e (0x517e bytes) 000901 1789674790.808781 read 0x00d2fd thru 0x017542 (0xa246 bytes) 000902 1789674790.819039 punch from 0x00ef6c to 0x01d7a8 (0xe83d bytes) 000903 1789674790.823984 write 0x011083 thru 0x013ec8 (0x2e46 bytes) 000904 1789674790.833472 trunc from 0x0274b1 to 0x0314db (0xa02b bytes) 000906 1789674790.839642 trunc from 0x00ebc4 to 0x019ec2 (0xb2ff bytes) 000907 1789674790.850631 trunc from 0x029b08 to 0x02b46c (0x1965 bytes) 000908 1789674790.870508 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x314dc): Mirror 8: 0xac7d12cc (stale) Mirror 9: 0xac7d12cc (stale) Mirror 10: 0xac7d12cc (stale) Mirror 12: 0xac7d12cc (stale) Mirror 13: 0xac7d12cc (stale) Mirror 14: 0xac7d12cc (stale) Mirror 17: 0xac7d12cc (stale) Mirror 18: 0xac7d12cc (stale) Mirror 19: 0x24abf99e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x314dc): Mirror 8: 0xac7d12cc (stale) Mirror 9: 0xac7d12cc (stale) Mirror 10: 0xac7d12cc (stale) Mirror 12: 0xac7d12cc (stale) Mirror 13: 0xac7d12cc (stale) Mirror 14: 0xac7d12cc (stale) Mirror 17: 0xac7d12cc (stale) Mirror 18: 0xac7d12cc (stale) Mirror 19: 0x24abf99e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x314dc: skipped 000909 1789674791.546560 mapread 0x02bf2b thru 0x0314db (0x55b1 bytes) 000910 1789674791.571142 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x314dc): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x314dc): Mirror 8: 0xac7d12cc (stale) Mirror 9: 0xac7d12cc (stale) Mirror 10: 0xac7d12cc (stale) Mirror 12: 0xac7d12cc (stale) Mirror 13: 0xac7d12cc (stale) Mirror 14: 0xac7d12cc (stale) Mirror 17: 0xac7d12cc (stale) Mirror 18: 0xac7d12cc (stale) Mirror 19: 0x24abf99e (stale) CRC-32 checksum value for chunk [0, 0x314dc): Mirror 1: 0xb61fffe Mirror 20: 0xb61fffe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x314dc: skipped 000911 1789674792.551299 trunc from 0x02153b to 0x022700 (0x11c6 bytes) 000912 1789674792.587147 write 0x02f67e thru 0x02f71e (0xa1 bytes) 000913 1789674792.594140 fallocate 0x0168b9 thru 0x01d0f1 (0x6839 bytes) 000914 1789674792.601995 mapread 0x00a014 thru 0x00f33f (0x532c bytes) 000915 1789674792.620900 mapwrite 0x01ee83 thru 0x02e553 (0xf6d1 bytes) 000916 1789674792.654580 fallocate 0x02f5c7 thru 0x03c23f (0xcc79 bytes) 000917 1789674792.658650 trunc from 0x03c240 to 0x030011 (0xc22e bytes) 000918 1789674792.674904 trunc from 0x030012 to 0x0202e2 (0xfd2f bytes) 000919 1789674792.689566 mapread 0x00005e thru 0x0055de (0x5581 bytes) 000920 1789674792.698427 mapwrite 0x019d96 thru 0x0214e8 (0x7753 bytes) 000921 1789674792.728933 punch from 0x01030b to 0x010364 (0x5a bytes) 000922 1789674792.740833 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x214e9): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x214e9): Mirror 8: 0x18d8f8e7 (stale) Mirror 9: 0x18d8f8e7 (stale) Mirror 10: 0x18d8f8e7 (stale) Mirror 12: 0x18d8f8e7 (stale) Mirror 13: 0x18d8f8e7 (stale) Mirror 14: 0x18d8f8e7 (stale) Mirror 17: 0x18d8f8e7 (stale) Mirror 18: 0x18d8f8e7 (stale) Mirror 19: 0xbc117508 (stale) Mirror 20: 0x18f800ca (stale) CRC-32 checksum value for chunk [0, 0x214e9): Mirror 1: 0x16d14ebc Mirror 21: 0x16d14ebc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x214e9: skipped 000923 1789674793.759627 MIRROR_SPLIT mirror 10 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x214e9): Mirror 8: 0x18d8f8e7 (stale) Mirror 9: 0x18d8f8e7 (stale) Mirror 10: 0x18d8f8e7 (stale) Mirror 12: 0x18d8f8e7 (stale) Mirror 13: 0x18d8f8e7 (stale) Mirror 14: 0x18d8f8e7 (stale) Mirror 17: 0x18d8f8e7 (stale) Mirror 18: 0x18d8f8e7 (stale) Mirror 19: 0xbc117508 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x214e9): Mirror 8: 0x18d8f8e7 (stale) Mirror 9: 0x18d8f8e7 (stale) Mirror 10: 0x18d8f8e7 (stale) Mirror 12: 0x18d8f8e7 (stale) Mirror 13: 0x18d8f8e7 (stale) Mirror 14: 0x18d8f8e7 (stale) Mirror 17: 0x18d8f8e7 (stale) Mirror 18: 0x18d8f8e7 (stale) Mirror 19: 0xbc117508 (stale) CRC-32 checksum value for chunk [0, 0x214e9): Mirror 1: 0x16d14ebc Mirror 21: 0x16d14ebc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x214e9: skipped 000924 1789674794.771233 MIRROR_RESYNC 11 mirrors 000925 1789674796.463130 trunc from 0x0214e9 to 0x01064f (0x10e99 bytes) 000926 1789674796.509212 read 0x002712 thru 0x00ce6f (0xa75e bytes) 000927 1789674796.517941 punch from 0x001307 to 0x002901 (0x15fb bytes) 000928 1789674796.522468 mapwrite 0x0047c2 thru 0x0069f6 (0x2235 bytes) 000929 1789674796.532675 trunc from 0x007018 to 0x00df7c (0x6f65 bytes) 000930 1789674796.536975 mapwrite 0x001b18 thru 0x011407 (0xf8f0 bytes) 000931 1789674796.593516 mapwrite 0x01a624 thru 0x01fc67 (0x5644 bytes) 000932 1789674796.647298 write 0x02a237 thru 0x02ce1a (0x2be4 bytes) 000933 1789674796.648445 mapread 0x009f96 thru 0x012033 (0x809e bytes) 000934 1789674796.655198 write 0x01643a thru 0x017b0a (0x16d1 bytes) 000935 1789674796.656380 fallocate 0x0125bd thru 0x01690c (0x4350 bytes) 000936 1789674796.660156 read 0x0101ed thru 0x014d23 (0x4b37 bytes) 000937 1789674796.661566 mapread 0x003e5d thru 0x00b6a7 (0x784b bytes) 000938 1789674796.662576 mapwrite 0x031b76 thru 0x034e7d (0x3308 bytes) 000939 1789674796.711691 trunc from 0x0037bb to 0x005c2e (0x2474 bytes) 000940 1789674796.716191 write 0x02fcfa thru 0x03adaf (0xb0b6 bytes) 000941 1789674796.718575 write 0x005be2 thru 0x008b62 (0x2f81 bytes) 000942 1789674796.724501 mapread 0x00a46b thru 0x019192 (0xed28 bytes) 000943 1789674796.726199 trunc from 0x03adb0 to 0x02f0aa (0xbd05 bytes) 000944 1789674796.738521 read 0x00fe22 thru 0x01b7b0 (0xb98f bytes) 000945 1789674796.739532 read 0x023c06 thru 0x02c584 (0x897f bytes) 000946 1789674796.740716 write 0x03a94b thru 0x03ffff (0x56b5 bytes) 000947 1789674796.742552 write 0x00ff73 thru 0x0104e2 (0x570 bytes) 000948 1789674796.743905 trunc from 0x029c79 to 0x02c3f0 (0x2778 bytes) 000949 1789674796.748196 mapread 0x024a45 thru 0x032c9c (0xe258 bytes) 000950 1789674796.756927 trunc from 0x01b06b to 0x01c6d3 (0x1669 bytes) 000951 1789674796.760807 read 0x02d234 thru 0x032405 (0x51d2 bytes) 000952 1789674796.761988 fallocate 0x0170a3 thru 0x022776 (0xb6d4 bytes) 000953 1789674796.766824 read 0x00a7d3 thru 0x017622 (0xce50 bytes) 000954 1789674796.767968 mapread 0x00c248 thru 0x00c6be (0x477 bytes) 000955 1789674796.769193 read 0x01ce86 thru 0x02b7df (0xe95a bytes) 000957 1789674796.773465 fallocate 0x02b422 thru 0x0365a8 (0xb187 bytes) 000958 1789674796.778002 fallocate 0x00aec6 thru 0x013e2b (0x8f66 bytes) 000961 1789674796.785197 punch from 0x037351 to 0x03d735 (0x63e5 bytes) 000962 1789674796.795435 punch from 0x013173 to 0x01c3d7 (0x9265 bytes) 000964 1789674796.800253 trunc from 0x002867 to 0x00de85 (0xb61f bytes) 000965 1789674796.804748 mapread 0x02dcf2 thru 0x0348a5 (0x6bb4 bytes) 000967 1789674796.807341 punch from 0x033c36 to 0x03be34 (0x81ff bytes) 000969 1789674796.812157 mapwrite 0x031a7c thru 0x03ba8b (0xa010 bytes) 000970 1789674796.861904 trunc from 0x0367ba to 0x03ee24 (0x866b bytes) 000972 1789674796.866938 read 0x00520a thru 0x00a984 (0x577b bytes) 000973 1789674796.875481 mapwrite 0x00729d thru 0x0159d4 (0xe738 bytes) 000974 1789674796.891902 trunc from 0x02eef8 to 0x030f65 (0x206e bytes) 000975 1789674796.895778 trunc from 0x040000 to 0x01e851 (0x217ae bytes) 000976 1789674796.910707 punch from 0x013a0c to 0x0177e0 (0x3dd5 bytes) 000977 1789674796.921285 read 0x009f85 thru 0x0198ef (0xf96b bytes) 000979 1789674796.929374 fallocate 0x00bf8a thru 0x0165bb (0xa632 bytes) 000981 1789674796.934987 trunc from 0x002407 to 0x00dab1 (0xb6ab bytes) 000982 1789674796.940012 trunc from 0x00e32d to 0x01a8b6 (0xc58a bytes) 000983 1789674796.949700 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1dc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e852): Mirror 8: 0x550862d (stale) Mirror 9: 0x550862d (stale) Mirror 10: 0x550862d (stale) Mirror 12: 0x550862d (stale) Mirror 13: 0x550862d (stale) Mirror 14: 0x550862d (stale) Mirror 18: 0x550862d (stale) Mirror 19: 0x550862d (stale) Mirror 21: 0x550862d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e852): Mirror 8: 0x550862d (stale) Mirror 9: 0x550862d (stale) Mirror 10: 0x550862d (stale) Mirror 12: 0x550862d (stale) Mirror 13: 0x550862d (stale) Mirror 14: 0x550862d (stale) Mirror 18: 0x550862d (stale) Mirror 19: 0x550862d (stale) Mirror 21: 0x550862d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e852: skipped 000984 1789674797.785525 punch from 0x003f1a to 0x01263c (0xe723 bytes) 000985 1789674797.791943 write 0x01e0c2 thru 0x02bdcd (0xdd0c bytes) 000986 1789674797.803504 mapwrite 0x02979d thru 0x035047 (0xb8ab bytes) 000987 1789674797.863197 MIRROR_RESYNC 10 mirrors 000988 1789674799.749587 punch from 0x00257c to 0x00e521 (0xbfa6 bytes) 000989 1789674799.779290 read 0x0245a5 thru 0x03252e (0xdf8a bytes) 000990 1789674799.786564 fallocate 0x03dd3c thru 0x03ffff (0x22c4 bytes) 000991 1789674799.792905 punch from 0x0074fb to 0x0098c4 (0x23ca bytes) 000992 1789674799.796985 read 0x00448a thru 0x00ddf5 (0x996c bytes) 000993 1789674799.806373 mapwrite 0x013503 thru 0x01838b (0x4e89 bytes) 000994 1789674799.820532 trunc from 0x03efa3 to 0x03ffff (0x105d bytes) 000995 1789674799.825360 mapwrite 0x02c189 thru 0x031e3c (0x5cb4 bytes) 000996 1789674799.844575 punch from 0x01e7af to 0x02c968 (0xe1ba bytes) 000997 1789674799.856469 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 8: 0x2ff12962 (stale) Mirror 9: 0x2ff12962 (stale) Mirror 10: 0x2ff12962 (stale) Mirror 12: 0x2ff12962 (stale) Mirror 13: 0x2ff12962 (stale) Mirror 14: 0x2ff12962 (stale) Mirror 18: 0x2ff12962 (stale) Mirror 19: 0x2ff12962 (stale) Mirror 21: 0x2ff12962 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb43c5e33 Mirror 22: 0xb43c5e33 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000998 1789674801.007528 trunc from 0x0215f7 to 0x02e125 (0xcb2f bytes) 000999 1789674801.063916 punch from 0x000adb to 0x00c719 (0xbc3f bytes) 001000 1789674801.072252 fallocate 0x00520a thru 0x00a16b (0x4f62 bytes) All operations completed A-OK! PASS 70a (114s) == sanity-flr test 71: check mirror extend parent fid ==== 15:53:26 (1789674806) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x22b:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (8s) == sanity-flr test 72: check append on FLR file ========== 15:53:34 (1789674814) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.190196 s, 11.0 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119369 s, 8.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x113:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (7s) == sanity-flr test 200a: stress test ===================== 15:53:41 (1789674821) Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre2 Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 8194176 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 2288416 .. failed Extending file size to 3715296 .. Extending file size to 5294816 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6770976 .. Extending file size to 7694112 .. Extending file size to 8327008 .. Extending file size to 8496608 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 9045952 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8397408 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9265888 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61609 61611 61612 61614 61616 done 192.168.206.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg632-client.virtnet /mnt/lustre2 (opts:) 192.168.206.132@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg632-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:55:06 (1789674906) Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre2 Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3008512 .. Extending file size to 5300928 .. failed Extending file size to 3671360 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4900896 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8710816 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6056544 .. Extending file size to 6807968 .. Extending file size to 8372992 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9180224 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8993056 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 63377 63378 63379 63380 63381 192.168.206.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg632-client.virtnet /mnt/lustre2 (opts:) 192.168.206.132@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg632-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (76s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:56:23 (1789674983) Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2312 1268992 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2312 1268992 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 25668 3581352 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17872 3589148 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43540 7170500 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2312 1268992 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2312 1268992 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 25668 3581352 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 17872 3589148 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43540 7170500 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.156419 s, 13.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.217948 s, 9.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x2ec71c03 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg632-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:56:43 (1789675003) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.116755 s, 18.0 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 15:56:51 (1789675011) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x5b:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12d:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x12a:0x0] } PASS 203 (6s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:56:57 (1789675017) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123775 s, 16.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a6:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2a5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2a6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:57:05 (1789675025) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0731033 s, 28.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a7:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a8:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2a7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2a8:0x0] } PASS 204b (7s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:57:12 (1789675032) striped dir -i0 -c2 -H all_char /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113552 s, 18.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12b:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12c:0x0] } PASS 204c (32s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:57:44 (1789675064) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg632-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086324 s, 12.1 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12d:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } PASS 204d (49s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:58:33 (1789675113) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg632-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.170995 s, 12.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x2a9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (47s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:59:20 (1789675160) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg632-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg632-server: Pool lustre.test_204f created Adding targets to pool oleg632-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg632-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.109164 s, 19.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x12f:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x131:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg632-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg632-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg632-server: Pool lustre.test_204f destroyed PASS 204f (69s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 16:00:30 (1789675230) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 16:00:37 (1789675237) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 16:00:44 (1789675244) oleg632-server: Pool lustre.test_206 created Adding targets to pool oleg632-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg632-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg632-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg632-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg632-server: Pool lustre.test_206 destroyed PASS 206 (25s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 16:01:09 (1789675269) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127536 s, 8.2 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.030648 s, 34.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xf4954aa4 Mirror 3: 0xf4954aa4 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 16:01:20 (1789675280) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 16:01:23 (1789675283) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (3s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 16:01:26 (1789675286) Starting client: oleg632-client.virtnet: -o user_xattr,flock 192.168.206.132@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0777875 s, 0.1 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1bf0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xcb0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xcb0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xcb0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1250000 Read this (4 bytes): BBBB 192.168.206.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg632-client.virtnet /mnt/lustre2 (opts:) PASS 209a (36s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 16:02:02 (1789675322) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0113915 s, 719 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 16:02:08 (1789675328) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0459061 s, 22.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7a lmm_fid: [0x200000405:0x7a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x7a lmm_fid: [0x200000405:0x7a:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13c:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611258 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-17 16:02:10.000000000 -0400 Modify: 2026-09-17 16:02:10.000000000 -0400 Change: 2026-09-17 16:02:11.000000000 -0400 Birth: 2026-09-17 16:02:10.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 16:02:16 (1789675336) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326792 s, 32.1 MB/s oleg632-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg632-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg632-server: O/280000401/d25/139: File not found by ext2_lookup PASS 210b (19s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 16:02:35 (1789675355) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.229162 s, 179 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0253543 s, 162 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 17 16:02 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x7f:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 17 16:02 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 16:02:43 (1789675363) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 16:02:50 (1789675370) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (8s) == sanity-flr test complete, duration 1709 sec =========== 16:02:58 (1789675378) === sanity-flr: start cleanup 16:03:00 (1789675380) === === sanity-flr: finish cleanup 16:03:08 (1789675388) ===