-----============= acceptance-small: sanity-flr ============----- Tue Sep 29 23:53:04 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky use SANITY_FLR_ONLY='' SANITY_FLR_EXCEPT='' oleg419-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 23:53:24 (1790740404) === oleg419-client.virtnet: executing check_config_client /mnt/lustre oleg419-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg419-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9fb14b058000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fb14b058000.idle_timeout=debug disable quota as required oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 23:54:10 (1790740450) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8195] == sanity-flr test 0a: lfs mirror create with -N option == 23:54:12 (1790740452) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:54:32 (1790740472) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:54:36 (1790740476) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:54:40 (1790740480) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:55:09 (1790740509) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:55:13 (1790740513) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (3s) == sanity-flr test 0g: lfs mirror create flags support === 23:55:17 (1790740517) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9fb14b058000.stats=clear osc.lustre-OST0001-osc-ffff9fb14b058000.stats=clear PASS 0g (10s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:55:26 (1790740526) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--ec-map] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:55:37 (1790740537) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:55:47 (1790740547) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 23:55:51 (1790740551) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:55:59 (1790740559) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 23:56:10 (1790740570) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:56:17 (1790740577) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:56:26 (1790740586) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:56:33 (1790740593) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:56:44 (1790740604) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:56:52 (1790740612) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.665477 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124782 s, 8.4 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:57:02 (1790740622) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0896151 s, 23.4 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 23:57:11 (1790740631) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:57:22 (1790740642) Stopping /mnt/lustre-ost1 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fb14b058000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fb14b058000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg419-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9fb14b058000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fb14b058000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg419-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (80s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:58:42 (1790740722) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.699901 s, 15.0 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fb14b058000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fb14b058000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg419-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (50s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:59:32 (1790740772) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fb14b058000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fb14b058000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg419-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9fb14b058000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fb14b058000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg419-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (91s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 00:01:03 (1790740863) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.71127 s, 14.1 MB/s -rw-r--r-- 1 root root 52428800 Sep 30 00:01 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.155s user 0m0.001s sys 0m0.195s Stopping /mnt/lustre-ost1 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9fb14b058000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9fb14b058000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.104s user 0m0.000s sys 0m0.212s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg419-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg419-server oleg419-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9fb14b058000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fb14b058000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.381s user 0m0.002s sys 0m0.182s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg419-server: oleg419-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg419-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg419-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fb14b058000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9fb14b058000.ost_server_uuid in FULL state after 0 sec PASS 33b (105s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 00:02:48 (1790740968) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 00:02:51 (1790740971) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 00:02:56 (1790740976) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 00:02:59 (1790740979) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 00:03:07 (1790740987) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.625625 s, 20.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 00:03:21 (1790741001) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579406 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564366 s, 18.6 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 00:03:29 (1790741009) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 00:03:38 (1790741018) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0012096 s, 847 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00134222 s, 763 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00141582 s, 723 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00105442 s, 971 kB/s flv=10 olv=10 PASS 36d (10s) == sanity-flr test 37: mirror I/O API verification ======= 00:03:49 (1790741029) create test files with size 15757822 .. b0f92a3f58fb0c56debd9c0800030489 - cc9310f2809ce079d101897952990004 - 59735b9eabf57ec0066d574303f0aaed - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg419-client.virtnet /mnt/lustre (opts:) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 00:04:15 (1790741055) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (18s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 00:04:33 (1790741073) PASS 39 (10s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 00:04:43 (1790741083) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0747575 s, 14.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xca:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0360577 s, 29.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xa9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xaa:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xcc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 00:04:54 (1790741094) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.127179 s, 16.5 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.626305 s, 6.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.17142 s, 12.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 00:05:14 (1790741114) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 00:05:20 (1790741120) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 00:05:25 (1790741125) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674784 s, 15.5 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 00:05:35 (1790741135) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.3506 s, 9.0 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 00:05:47 (1790741147) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd7:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xd7:0x0] } split mirror_id 2 id 131073 error lfs split: file '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' has 1 component, cannot split make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 00:05:59 (1790741159) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.445986 s, 23.5 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x5:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg419-client.virtnet /mnt/lustre (opts:) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 00:06:12 (1790741172) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9fb1435e4800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.401868 s, 26.1 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg419-client.virtnet /mnt/lustre (opts:) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9fb14a86f000.xattr_cache=1 PASS 44e (19s) == sanity-flr test 44f: lfs mirror split does not leak the victim file descriptor ========================================================== 00:06:31 (1790741191) error lfs split: create victim file failed: File exists PASS 44f (21s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 00:06:53 (1790741213) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (44s) == sanity-flr test 46: Verify setstripe --copy option ==== 00:07:37 (1790741257) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 00:08:18 (1790741298) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 00:08:22 (1790741302) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.29235 s, 10.8 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.237507 s, 13.2 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xb12fbb6f (nosync,stale) Mirror 2: 0xb12fbb6f (nosync,stale) ** verify mirror 2 doesn't change original checksum: 1c4cf74b3a74eb540f53b378c123e074 - mirror 1 checksum: e1a67f234d911c7678c5b2f951f0b7fc - mirror 2 checksum: 1c4cf74b3a74eb540f53b378c123e074 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-30T00:08:26-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xeb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-30T00:08:26-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 1c4cf74b3a74eb540f53b378c123e074 - mirror 1 checksum: e1a67f234d911c7678c5b2f951f0b7fc - mirror 2 checksum: e1a67f234d911c7678c5b2f951f0b7fc - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-30T00:08:26-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xeb:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-30T00:08:26-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x5e lmm_fid: [0x200000405:0x5e:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xec:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 00:08:34 (1790741314) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0272043 s, 38.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 00:08:41 (1790741321) Failing mds1 on oleg419-server Stopping /mnt/lustre-mds1 (opts:) on oleg419-server 00:08:49 (1790741329) shut down facet: mds1 facet_host: oleg419-server facet_failover_host: oleg419-server Failover mds1 to oleg419-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 00:09:26 (1790741366) targets are mounted 00:09:26 (1790741366) facet_failover done oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x8:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400276 s, 26.2 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x240000404 lmm_object_id: 0x8 lmm_fid: [0x240000404:0x8:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1c:0x0] } PASS 50A (66s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 00:09:47 (1790741387) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0715841 s, 286 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0450785 s, 454 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (14s) == sanity-flr test 50b: mirror rsync handles sparseness == 00:10:02 (1790741402) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.10172 s, 201 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0144333 s, 1.4 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (13s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 00:10:15 (1790741415) striped dir -i0 -c2 -H crush /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0103675 s, 1.6 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 00:10:24 (1790741424) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0395336 s, 518 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (10s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 00:10:34 (1790741434) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00852034 s, 481 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0254645 s, 161 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 00:10:43 (1790741443) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x70:0x0]: Invalid argument (22) PASS 60b (28s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 00:11:11 (1790741471) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 00:11:43 (1790741503) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 00:12:09 (1790741529) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 00:12:34 (1790741554) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 00:12:42 (1790741562) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 58282 mirror split pid 58283 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 58282 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5095: 58283 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 00:13:51 (1790741631) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 1633 skipping zero length punch hole skipping zero size read skipping zero length zero range skipping zero length zero range 000005 1790741634.229496 mapwrite 0x03e21d thru 0x03ffff (0x1de3 bytes) 000006 1790741634.297930 punch from 0x03642d to 0x03ffff (0x9bd3 bytes) 000007 1790741634.318793 punch from 0x00924e to 0x010863 (0x7616 bytes) 000008 1790741634.330078 mapwrite 0x026606 thru 0x035bde (0xf5d9 bytes) 000009 1790741634.417032 read 0x01e68f thru 0x020fb6 (0x2928 bytes) truncating to largest ever: 0x2a6c 000010 1790741634.422958 trunc from 0x040000 to 0x002a6b (0x3d594 bytes) 000011 1790741634.456979 trunc from 0x0022b7 to 0x002a6b (0x7b5 bytes) truncating to largest ever: 0x3e9b0 000012 1790741634.465892 trunc from 0x002a6c to 0x03e9af (0x3bf44 bytes) 000013 1790741634.517709 mapread 0x023b4e thru 0x025ba6 (0x2059 bytes) 000014 1790741634.560295 trunc from 0x03e9b0 to 0x025ed7 (0x18ad8 bytes) fallocating to largest ever: 0x3f4e6 000015 1790741634.582627 fallocate 0x03da2e thru 0x03f4e5 (0x1ab8 bytes) 000016 1790741634.595311 write 0x00e192 thru 0x01c021 (0xde90 bytes) 000017 1790741634.602236 trunc from 0x009e3d to 0x00bb8c (0x1d50 bytes) 000018 1790741634.616483 read 0x01be0e thru 0x023a5f (0x7c52 bytes) 000021 1790741634.621638 trunc from 0x01b9b9 to 0x024281 (0x88c9 bytes) 000022 1790741634.647925 trunc from 0x023434 to 0x025e9b (0x2a68 bytes) 000023 1790741634.656154 read 0x0015b1 thru 0x00e8fa (0xd34a bytes) 000024 1790741634.697584 read 0x00ecdb thru 0x0127bc (0x3ae2 bytes) 000025 1790741634.698476 read 0x00bdc2 thru 0x017a70 (0xbcaf bytes) 000026 1790741634.699002 trunc from 0x00b704 to 0x015cc4 (0xa5c1 bytes) 000027 1790741634.708771 mapread 0x01a348 thru 0x025ed7 (0xbb90 bytes) 000028 1790741634.717675 write 0x03a05e thru 0x03f120 (0x50c3 bytes) 000029 1790741634.719671 trunc from 0x022ea7 to 0x0310c5 (0xe21f bytes) 000030 1790741634.733915 read 0x02c8fc thru 0x033261 (0x6966 bytes) 000031 1790741634.754690 trunc from 0x0247cb to 0x0335a1 (0xedd7 bytes) 000032 1790741634.761062 write 0x02c8c4 thru 0x02d4b7 (0xbf4 bytes) 000033 1790741634.781814 fallocate 0x02c146 thru 0x03211f (0x5fda bytes) 000034 1790741634.800253 fallocate 0x003d70 thru 0x010f85 (0xd216 bytes) 000035 1790741634.818479 trunc from 0x03f121 to 0x01a3ba (0x24d66 bytes) 000037 1790741634.834925 punch from 0x011677 to 0x01a3ba (0x8d44 bytes) 000038 1790741634.841611 write 0x0025f2 thru 0x00d45d (0xae6c bytes) 000039 1790741634.846883 punch from 0x0101fe to 0x0187f9 (0x85fc bytes) 000041 1790741634.859070 fallocate 0x018b87 thru 0x01b332 (0x27ac bytes) 000042 1790741634.871290 write 0x037cee thru 0x03e98b (0x6c9e bytes) 000043 1790741634.877660 trunc from 0x03e98c to 0x0342d1 (0xa6ba bytes) 000044 1790741634.903023 mapread 0x00e651 thru 0x0149a7 (0x6357 bytes) 000045 1790741634.939297 mapwrite 0x0005e1 thru 0x00a66c (0xa08c bytes) fallocating to largest ever: 0x3f9cb 000046 1790741634.960642 fallocate 0x034e69 thru 0x03f9ca (0xab62 bytes) 000047 1790741634.966040 read 0x00544d thru 0x00c016 (0x6bca bytes) 000048 1790741634.967003 fallocate 0x0294fa thru 0x034840 (0xb347 bytes) 000049 1790741634.974116 fallocate 0x016760 thru 0x01fe0a (0x96ab bytes) 000050 1790741634.979424 write 0x03927a thru 0x03ffff (0x6d86 bytes) 000051 1790741634.986365 MIRROR_EXTEND to 2 mirrors 000052 1790741635.868600 trunc from 0x022b63 to 0x0323f1 (0xf88f bytes) 000053 1790741635.907291 fallocate 0x02cbb9 thru 0x02e340 (0x1788 bytes) 000054 1790741635.912315 mapwrite 0x028a5d thru 0x032c36 (0xa1da bytes) 000055 1790741635.950862 punch from 0x02a81b to 0x033ff1 (0x97d7 bytes) 000056 1790741635.957089 trunc from 0x040000 to 0x020871 (0x1f78e bytes) 000057 1790741635.989279 trunc from 0x020872 to 0x016a16 (0x9e5b bytes) 000058 1790741636.005469 read 0x002b7a thru 0x012036 (0xf4bd bytes) 000059 1790741636.019562 mapwrite 0x026478 thru 0x032dbe (0xc947 bytes) 000060 1790741636.075792 mapread 0x0235b3 thru 0x028adb (0x5529 bytes) 000061 1790741636.076892 read 0x002c94 thru 0x0042c4 (0x1631 bytes) 000062 1790741636.078704 write 0x00e4e7 thru 0x01bcd0 (0xd7ea bytes) 000063 1790741636.087402 trunc from 0x032dbf to 0x0026c4 (0x306fa bytes) 000064 1790741636.105435 punch from 0x000d41 to 0x0026c4 (0x1984 bytes) 000065 1790741636.110971 fallocate 0x0233ba thru 0x031f9b (0xebe2 bytes) 000066 1790741636.115792 mapwrite 0x01e534 thru 0x027e1c (0x98e9 bytes) 000067 1790741636.143384 mapwrite 0x00a842 thru 0x014a66 (0xa225 bytes) 000068 1790741636.188696 trunc from 0x031f9c to 0x00bcee (0x262ad bytes) 000069 1790741636.206701 mapread 0x00b5c2 thru 0x00bcee (0x72d bytes) 000070 1790741636.208499 mapread 0x000962 thru 0x000d75 (0x414 bytes) 000071 1790741636.215045 write 0x03bb32 thru 0x03ffff (0x44ce bytes) 000072 1790741636.217336 trunc from 0x02ac6b to 0x03a28b (0xf621 bytes) 000073 1790741636.226504 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) 000074 1790741636.386869 trunc from 0x032397 to 0x03d665 (0xb2cf bytes) 000075 1790741636.397592 punch from 0x036ae2 to 0x03ffff (0x951e bytes) 000076 1790741636.417760 write 0x005a95 thru 0x00a1f3 (0x475f bytes) 000077 1790741636.434391 mapread 0x01a567 thru 0x023d1a (0x97b4 bytes) 000078 1790741636.464052 trunc from 0x040000 to 0x019601 (0x269fe bytes) 000080 1790741636.503831 read 0x001687 thru 0x00d10d (0xba87 bytes) 000081 1790741636.514825 read 0x0067a5 thru 0x00a680 (0x3edc bytes) 000082 1790741636.515703 mapwrite 0x01b97c thru 0x02284f (0x6ed4 bytes) 000083 1790741636.582314 write 0x02edf5 thru 0x034d23 (0x5f2f bytes) 000084 1790741636.583781 punch from 0x027cbb to 0x032274 (0xa5ba bytes) 000085 1790741636.599435 mapwrite 0x00e153 thru 0x01a159 (0xc007 bytes) 000086 1790741636.635805 fallocate 0x02b137 thru 0x032c0f (0x7ad9 bytes) 000087 1790741636.642010 trunc from 0x034d24 to 0x002fd7 (0x31d4c bytes) 000088 1790741636.674098 read 0x000044 thru 0x001dd1 (0x1d8e bytes) 000089 1790741636.675527 mapwrite 0x03ec2d thru 0x03ffff (0x13d3 bytes) zero_range to largest ever: 0x40000 000090 1790741636.731438 trunc from 0x03c73b to 0x03ffff (0x38c5 bytes) 000091 1790741636.740523 fallocate 0x02bf89 thru 0x039b0e (0xdb86 bytes) 000092 1790741636.747976 trunc from 0x040000 to 0x01714c (0x28eb3 bytes) 000094 1790741636.791498 MIRROR_EXTEND to 2 mirrors 000095 1790741637.756682 punch from 0x001f50 to 0x005d0a (0x3dbb bytes) 000096 1790741637.797259 write 0x03b8cc thru 0x03ffff (0x4734 bytes) 000098 1790741637.812847 read 0x004257 thru 0x012352 (0xe0fc bytes) 000099 1790741637.833715 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) 000100 1790741638.046481 fallocate 0x037cc0 thru 0x0381bf (0x500 bytes) 000101 1790741638.061443 trunc from 0x026406 to 0x032d2c (0xc927 bytes) 000102 1790741638.065815 trunc from 0x040000 to 0x032c6b (0xd394 bytes) 000103 1790741638.096264 trunc from 0x032c6c to 0x027018 (0xbc53 bytes) 000104 1790741638.126410 mapwrite 0x02b631 thru 0x03afd3 (0xf9a3 bytes) 000107 1790741638.205578 write 0x00c75a thru 0x01a059 (0xd900 bytes) 000108 1790741638.217918 trunc from 0x021706 to 0x02bf27 (0xa822 bytes) 000109 1790741638.239771 MIRROR_EXTEND to 2 mirrors 000110 1790741639.144575 trunc from 0x0350eb to 0x038090 (0x2fa6 bytes) 000111 1790741639.219056 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) 000112 1790741639.466031 mapread 0x01b7ec thru 0x01da46 (0x225b bytes) 000113 1790741639.496904 mapread 0x0213d2 thru 0x02692c (0x555b bytes) 000114 1790741639.518307 punch from 0x00f592 to 0x011525 (0x1f94 bytes) 000115 1790741639.524395 trunc from 0x03a919 to 0x03afd3 (0x6bb bytes) 000116 1790741639.531365 read 0x037888 thru 0x03afd3 (0x374c bytes) 000117 1790741639.538422 fallocate 0x016b59 thru 0x01bc47 (0x50ef bytes) 000119 1790741639.549396 MIRROR_EXTEND to 2 mirrors 000120 1790741640.574062 trunc from 0x005c2c to 0x00ec2b (0x9000 bytes) 000121 1790741640.612329 mapwrite 0x026da1 thru 0x0305fe (0x985e bytes) 000122 1790741640.668202 write 0x01ccb1 thru 0x01dc25 (0xf75 bytes) 000123 1790741640.701161 punch from 0x0171ed to 0x0206d4 (0x94e8 bytes) 000124 1790741640.735567 punch from 0x00aeef to 0x01152d (0x663f bytes) 000125 1790741640.745152 trunc from 0x03afd4 to 0x013779 (0x2785a bytes) 000126 1790741640.768635 mapwrite 0x004bf7 thru 0x006365 (0x176f bytes) 000127 1790741640.805916 mapread 0x0001bd thru 0x001959 (0x179d bytes) 000128 1790741640.824113 fallocate 0x02d149 thru 0x038705 (0xb5bd bytes) 000129 1790741640.834439 mapread 0x006fdb thru 0x0104d9 (0x94ff bytes) 000130 1790741640.835759 mapread 0x0126c1 thru 0x0130e8 (0xa28 bytes) 000131 1790741640.852127 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1377a): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1377a): Mirror 2: 0x3c9f0b4 (stale) CRC-32 checksum value for chunk [0, 0x1377a): Mirror 1: 0x326a452e Mirror 3: 0x326a452e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1377a: skipped 000132 1790741641.972849 fallocate 0x008e75 thru 0x0148fd (0xba89 bytes) 000133 1790741642.024229 read 0x00f72b thru 0x0148fd (0x51d3 bytes) 000134 1790741642.049598 MIRROR_RESYNC 3 mirrors 000135 1790741643.464766 mapread 0x003b6b thru 0x0043dc (0x872 bytes) 000136 1790741643.477462 trunc from 0x001e5c to 0x005e74 (0x4019 bytes) 000137 1790741643.573276 mapwrite 0x034270 thru 0x03db6c (0x98fd bytes) 000138 1790741643.631470 trunc from 0x03db6d to 0x013b7b (0x29ff1 bytes) 000139 1790741643.655638 read 0x0057e4 thru 0x007ecc (0x26e9 bytes) 000140 1790741643.671656 read 0x011f71 thru 0x013b7b (0x1c0b bytes) 000141 1790741643.687522 mapwrite 0x02dec9 thru 0x036a88 (0x8bc0 bytes) 000142 1790741643.811703 read 0x028975 thru 0x036a88 (0xe114 bytes) 000143 1790741643.814688 read 0x01c323 thru 0x022409 (0x60e7 bytes) 000144 1790741643.823320 mapread 0x00602a thru 0x00fa10 (0x99e7 bytes) 000145 1790741643.825782 mapwrite 0x03d326 thru 0x03fc57 (0x2932 bytes) 000146 1790741643.911723 trunc from 0x03fc58 to 0x02b0f5 (0x14b62 bytes) 000147 1790741643.933714 mapwrite 0x015fd3 thru 0x01dc6f (0x7c9d bytes) 000148 1790741643.961726 mapread 0x01d030 thru 0x02a1d0 (0xd1a1 bytes) 000150 1790741643.971563 read 0x01c1c8 thru 0x025059 (0x8e92 bytes) 000151 1790741643.973870 mapwrite 0x02db4e thru 0x03da52 (0xff05 bytes) 000152 1790741644.054148 trunc from 0x011cef to 0x01a673 (0x8985 bytes) 000153 1790741644.070186 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3da53): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3da53): Mirror 2: 0x1397fbd4 (stale) Mirror 3: 0x1397fbd4 (stale) CRC-32 checksum value for chunk [0, 0x3da53): Mirror 1: 0xc54c2d3d Mirror 4: 0xc54c2d3d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3da53: skipped 000154 1790741645.621695 fallocate 0x01b08d thru 0x022c7e (0x7bf2 bytes) 000155 1790741645.676100 mapwrite 0x0330b0 thru 0x03d8c9 (0xa81a bytes) 000156 1790741645.754997 trunc from 0x005563 to 0x0136eb (0xe189 bytes) 000157 1790741645.764946 mapwrite 0x036205 thru 0x03ffff (0x9dfb bytes) 000158 1790741645.906071 read 0x0240e6 thru 0x030643 (0xc55e bytes) 000159 1790741645.930799 trunc from 0x040000 to 0x0038bb (0x3c744 bytes) 000160 1790741645.961655 trunc from 0x0038bc to 0x0344c9 (0x30c0e bytes) 000161 1790741645.982693 read 0x02de73 thru 0x0344c9 (0x6657 bytes) 000162 1790741645.999024 punch from 0x0297c4 to 0x02c04c (0x2889 bytes) 000163 1790741646.004343 mapread 0x0030d0 thru 0x012600 (0xf531 bytes) 000164 1790741646.009525 fallocate 0x0384ab thru 0x03ffff (0x7b55 bytes) 000165 1790741646.020395 punch from 0x02cafc to 0x0312b6 (0x47bb bytes) 000166 1790741646.028817 write 0x01680f thru 0x01e226 (0x7a18 bytes) 000167 1790741646.030525 read 0x019a5f thru 0x01f033 (0x55d5 bytes) 000169 1790741646.037906 write 0x0075d9 thru 0x00c3bf (0x4de7 bytes) 000170 1790741646.044475 MIRROR_RESYNC 4 mirrors 000171 1790741647.068573 mapread 0x00e76e thru 0x01bed4 (0xd767 bytes) 000172 1790741647.095413 trunc from 0x03ec1e to 0x03ffff (0x13e2 bytes) 000173 1790741647.130501 write 0x02c4ca thru 0x035230 (0x8d67 bytes) 000174 1790741647.142436 write 0x029f7d thru 0x039ac2 (0xfb46 bytes) 000175 1790741647.152446 mapwrite 0x002085 thru 0x00bb92 (0x9b0e bytes) 000176 1790741647.229205 trunc from 0x010133 to 0x010cc6 (0xb94 bytes) 000177 1790741647.246529 fallocate 0x01dcee thru 0x0262eb (0x85fe bytes) 000178 1790741647.258924 trunc from 0x040000 to 0x0269de (0x19621 bytes) 000179 1790741647.330962 trunc from 0x0269df to 0x00a123 (0x1c8bb bytes) 000180 1790741647.364280 mapread 0x003fab thru 0x00a123 (0x6179 bytes) 000182 1790741647.367605 trunc from 0x00a124 to 0x02cd95 (0x22c72 bytes) 000183 1790741647.401266 mapread 0x01897c thru 0x025f07 (0xd58c bytes) 000184 1790741647.422001 mapwrite 0x011ea9 thru 0x018122 (0x627a bytes) 000185 1790741647.449105 fallocate 0x02c481 thru 0x03b35d (0xeedd bytes) 000187 1790741647.456752 trunc from 0x02a0f4 to 0x02cd95 (0x2ca2 bytes) 000189 1790741647.470656 mapread 0x02a3d0 thru 0x02cd95 (0x29c6 bytes) 000191 1790741647.495275 mapread 0x0001f6 thru 0x010131 (0xff3c bytes) 000192 1790741647.534212 trunc from 0x010ac1 to 0x018b8f (0x80cf bytes) 000193 1790741647.553398 mapread 0x00786f thru 0x008e58 (0x15ea bytes) 000194 1790741647.562052 MIRROR_RESYNC 4 mirrors 000195 1790741648.989720 trunc from 0x0017c6 to 0x005daa (0x45e5 bytes) 000196 1790741649.036847 fallocate 0x020e3f thru 0x0221d7 (0x1399 bytes) 000199 1790741649.045705 mapread 0x00dd56 thru 0x012793 (0x4a3e bytes) 000200 1790741649.085254 mapread 0x00e229 thru 0x0182dd (0xa0b5 bytes) 000201 1790741649.096372 trunc from 0x0030a6 to 0x00c5c7 (0x9522 bytes) 000202 1790741649.102235 trunc from 0x02844c to 0x0286e8 (0x29d bytes) 000203 1790741649.109903 trunc from 0x003a8c to 0x009953 (0x5ec8 bytes) 000204 1790741649.114458 read 0x015c2f thru 0x015deb (0x1bd bytes) 000205 1790741649.115977 punch from 0x0223b7 to 0x02c346 (0x9f90 bytes) 000206 1790741649.122155 read 0x003cdf thru 0x00e5aa (0xa8cc bytes) 000207 1790741649.136879 trunc from 0x02cd96 to 0x017a68 (0x1532d bytes) 000208 1790741649.170035 write 0x00c0ba thru 0x00e160 (0x20a7 bytes) 000209 1790741649.180083 fallocate 0x01bdec thru 0x02043f (0x4654 bytes) 000210 1790741649.190114 read 0x0166e1 thru 0x017a68 (0x1388 bytes) 000211 1790741649.190680 fallocate 0x0099e7 thru 0x017dc2 (0xe3dc bytes) 000213 1790741649.202391 mapwrite 0x0267ca thru 0x02a90e (0x4145 bytes) 000214 1790741649.270584 trunc from 0x004bd9 to 0x00dd09 (0x9131 bytes) 000215 1790741649.280060 fallocate 0x00078a thru 0x000d40 (0x5b7 bytes) 000217 1790741649.292772 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a90f): Mirror 2: 0xaaf58ea9 (stale) Mirror 4: 0xaaf58ea9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a90f): Mirror 2: 0xaaf58ea9 (stale) Mirror 4: 0xaaf58ea9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a90f: skipped 000218 1790741649.736382 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2a90f): Mirror 2: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a90f): Mirror 2: 0xaaf58ea9 (stale) Mirror 4: 0xaaf58ea9 (stale) CRC-32 checksum value for chunk [0, 0x2a90f): Mirror 1: 0xd006b5a0 Mirror 5: 0xd006b5a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a90f: skipped 000219 1790741650.844487 mapread 0x024c4b thru 0x0293db (0x4791 bytes) 000220 1790741650.912459 fallocate 0x037a91 thru 0x0391b2 (0x1722 bytes) 000221 1790741650.984475 mapwrite 0x017ce5 thru 0x024fb1 (0xd2cd bytes) 000222 1790741651.046820 fallocate 0x021a28 thru 0x02456e (0x2b47 bytes) 000223 1790741651.075949 MIRROR_RESYNC 4 mirrors 000224 1790741652.744342 write 0x036df9 thru 0x03a27b (0x3483 bytes) 000225 1790741652.804487 MIRROR_RESYNC 4 mirrors 000226 1790741654.878874 fallocate 0x024c78 thru 0x031370 (0xc6f9 bytes) 000227 1790741654.935719 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3a27c): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a27c): Mirror 2: 0x571ce4d2 (stale) Mirror 4: 0x571ce4d2 (stale) Mirror 5: 0x571ce4d2 (stale) CRC-32 checksum value for chunk [0, 0x3a27c): Mirror 1: 0x571ce4d2 Mirror 6: 0x571ce4d2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a27c: skipped 000228 1790741656.259995 mapwrite 0x0216e5 thru 0x0222c6 (0xbe2 bytes) 000229 1790741656.344159 read 0x0284f2 thru 0x02a69c (0x21ab bytes) 000230 1790741656.346081 mapwrite 0x01cda0 thru 0x0295b3 (0xc814 bytes) 000231 1790741656.398841 mapread 0x0387a5 thru 0x03a27b (0x1ad7 bytes) 000232 1790741656.409360 punch from 0x03a0b4 to 0x03a27b (0x1c8 bytes) 000233 1790741656.417372 trunc from 0x03a27c to 0x012c04 (0x27677 bytes) 000235 1790741656.452492 trunc from 0x012c05 to 0x0233a8 (0x107a4 bytes) 000236 1790741656.472225 trunc from 0x0233a9 to 0x029016 (0x5c6e bytes) 000237 1790741656.492989 write 0x032948 thru 0x037da6 (0x545f bytes) 000238 1790741656.495120 mapread 0x028499 thru 0x030f80 (0x8ae8 bytes) 000239 1790741656.503478 punch from 0x020874 to 0x02be9f (0xb62c bytes) 000240 1790741656.507951 read 0x0320f7 thru 0x036b1d (0x4a27 bytes) 000241 1790741656.508654 trunc from 0x037da7 to 0x00f5b2 (0x287f4 bytes) 000242 1790741656.520947 read 0x002293 thru 0x009b9d (0x790b bytes) 000243 1790741656.532365 mapread 0x0026ab thru 0x009111 (0x6a67 bytes) 000244 1790741656.537461 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xf5b3): Mirror 2: 0x311649c2 (stale) Mirror 5: 0x311649c2 (stale) Mirror 6: 0x311649c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf5b3): Mirror 2: 0x311649c2 (stale) Mirror 5: 0x311649c2 (stale) Mirror 6: 0x311649c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf5b3: skipped 000245 1790741656.820177 read 0x00e166 thru 0x00f5b2 (0x144d bytes) 000246 1790741656.826085 fallocate 0x034bde thru 0x03ffff (0xb422 bytes) 000248 1790741656.834778 mapwrite 0x01c814 thru 0x020089 (0x3876 bytes) 000249 1790741656.912964 fallocate 0x017bf4 thru 0x0211e6 (0x95f3 bytes) 000250 1790741656.921696 trunc from 0x0211e7 to 0x0068c5 (0x1a921 bytes) 000251 1790741656.939112 trunc from 0x0068c6 to 0x00b198 (0x48d3 bytes) 000252 1790741656.951065 mapwrite 0x00aaec thru 0x014496 (0x99ab bytes) 000253 1790741656.997705 write 0x001288 thru 0x004db4 (0x3b2d bytes) 000254 1790741657.003005 punch from 0x00433f to 0x01128d (0xcf4f bytes) 000255 1790741657.015968 MIRROR_RESYNC 4 mirrors 000256 1790741657.879664 write 0x03ab4d thru 0x03f76b (0x4c1f bytes) 000257 1790741657.935397 write 0x012a01 thru 0x01bcac (0x92ac bytes) 000258 1790741657.965439 read 0x03ac1c thru 0x03f76b (0x4b50 bytes) 000259 1790741657.966222 fallocate 0x0327f8 thru 0x036629 (0x3e32 bytes) 000260 1790741657.989688 punch from 0x0094dc to 0x009a35 (0x55a bytes) 000261 1790741658.000886 trunc from 0x017ccd to 0x01fdbb (0x80ef bytes) 000262 1790741658.038489 MIRROR_RESYNC 4 mirrors 000263 1790741659.694632 read 0x007434 thru 0x00daf2 (0x66bf bytes) 000264 1790741659.721172 fallocate 0x02ab4c thru 0x039fe4 (0xf499 bytes) 000265 1790741659.805068 mapread 0x0215c4 thru 0x023243 (0x1c80 bytes) 000266 1790741659.849977 trunc from 0x016996 to 0x0206a3 (0x9d0e bytes) 000267 1790741659.856374 read 0x0132b5 thru 0x01ef4d (0xbc99 bytes) 000268 1790741659.874123 write 0x029f3a thru 0x030b01 (0x6bc8 bytes) 000269 1790741659.884895 trunc from 0x03f76c to 0x033aad (0xbcbe bytes) 000270 1790741659.945749 punch from 0x024ce1 to 0x02a1a4 (0x54c4 bytes) 000271 1790741659.984119 fallocate 0x033164 thru 0x03ffff (0xce9c bytes) 000272 1790741659.998055 mapwrite 0x02802a thru 0x036965 (0xe93c bytes) 000273 1790741660.127241 write 0x00dbc6 thru 0x019571 (0xb9ac bytes) 000274 1790741660.128811 mapwrite 0x003fbd thru 0x00a4a0 (0x64e4 bytes) 000275 1790741660.169667 punch from 0x02bc37 to 0x039c99 (0xe063 bytes) 000276 1790741660.182526 trunc from 0x040000 to 0x0291de (0x16e21 bytes) 000277 1790741660.231760 read 0x011402 thru 0x01644f (0x504e bytes) 000278 1790741660.232461 write 0x023222 thru 0x0239cd (0x7ac bytes) 000279 1790741660.233440 trunc from 0x0073ff to 0x016870 (0xf472 bytes) 000281 1790741660.253684 punch from 0x00c037 to 0x0179aa (0xb974 bytes) 000283 1790741660.266045 punch from 0x021913 to 0x0291de (0x78cc bytes) 000284 1790741660.301472 trunc from 0x0291df to 0x03d420 (0x14242 bytes) 000285 1790741660.354415 trunc from 0x03d421 to 0x01fc4c (0x1d7d4 bytes) 000286 1790741660.405158 MIRROR_RESYNC 4 mirrors 000287 1790741661.802264 MIRROR_RESYNC 4 mirrors 000288 1790741662.356365 mapread 0x0002e1 thru 0x00d59a (0xd2ba bytes) 000289 1790741662.386924 mapwrite 0x03e048 thru 0x03ffff (0x1fb8 bytes) 000290 1790741662.470026 punch from 0x00344c to 0x012963 (0xf518 bytes) 000291 1790741662.479841 trunc from 0x035a11 to 0x03996e (0x3f5e bytes) 000292 1790741662.487071 trunc from 0x040000 to 0x018432 (0x27bcd bytes) 000293 1790741662.514795 trunc from 0x018433 to 0x0335cb (0x1b199 bytes) 000294 1790741662.547115 trunc from 0x0335cc to 0x01eaf6 (0x14ad5 bytes) 000295 1790741662.588860 punch from 0x0143e3 to 0x01eaf6 (0xa714 bytes) 000296 1790741662.607305 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1eaf7): Mirror 5: 0xfe4a822c (stale) Mirror 6: 0xfe4a822c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1eaf7): Mirror 5: 0xfe4a822c (stale) Mirror 6: 0xfe4a822c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1eaf7: skipped 000297 1790741663.204101 fallocate 0x023504 thru 0x02acaa (0x77a7 bytes) 000298 1790741663.217899 punch from 0x01b4d6 to 0x01d0d4 (0x1bff bytes) 000299 1790741663.224046 trunc from 0x0130f8 to 0x0142f9 (0x1202 bytes) 000300 1790741663.242838 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2acab): Mirror 5: 0xa6e7b03a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2acab): Mirror 5: 0xa6e7b03a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2acab: skipped 000301 1790741663.747430 punch from 0x021771 to 0x0222ba (0xb4a bytes) 000302 1790741663.772359 MIRROR_RESYNC 2 mirrors 000303 1790741664.618322 mapread 0x002346 thru 0x00a910 (0x85cb bytes) 000304 1790741664.648249 mapwrite 0x01a36e thru 0x029632 (0xf2c5 bytes) 000305 1790741664.775216 trunc from 0x00bac8 to 0x00c2fb (0x834 bytes) 000306 1790741664.788593 MIRROR_RESYNC 2 mirrors 000307 1790741665.451356 punch from 0x0202ce to 0x02acaa (0xa9dd bytes) 000308 1790741665.492478 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) 000310 1790741665.627168 punch from 0x012af4 to 0x0206f6 (0xdc03 bytes) 000311 1790741665.633369 trunc from 0x01a1ca to 0x01f048 (0x4e7f bytes) 000313 1790741665.643308 read 0x02980c thru 0x02acaa (0x149f bytes) 000315 1790741665.653931 mapwrite 0x02d65e thru 0x02d8f8 (0x29b bytes) 000316 1790741665.692077 trunc from 0x00210a to 0x00d8c6 (0xb7bd bytes) 000319 1790741665.702313 punch from 0x0043c5 to 0x010990 (0xc5cc bytes) 000320 1790741665.708222 mapwrite 0x019a17 thru 0x02754b (0xdb35 bytes) 000322 1790741665.751780 read 0x009e53 thru 0x011f34 (0x80e2 bytes) 000323 1790741665.771472 mapread 0x021d09 thru 0x02d8f8 (0xbbf0 bytes) 000324 1790741665.776588 trunc from 0x02d8f9 to 0x0133fd (0x1a4fb bytes) 000327 1790741665.815794 trunc from 0x0133fe to 0x001707 (0x11cf6 bytes) 000329 1790741665.862083 punch from 0x001384 to 0x001707 (0x384 bytes) 000330 1790741665.901092 punch from 0x000b67 to 0x001707 (0xba1 bytes) 000331 1790741665.914134 fallocate 0x036c02 thru 0x03ffff (0x93fe bytes) 000332 1790741665.937066 trunc from 0x040000 to 0x038d98 (0x7267 bytes) 000333 1790741665.971154 write 0x01c226 thru 0x02bebf (0xfc9a bytes) 000334 1790741666.003039 mapwrite 0x007221 thru 0x00e398 (0x7178 bytes) 000335 1790741666.095055 read 0x015755 thru 0x017e51 (0x26fd bytes) 000336 1790741666.106806 read 0x023b1f thru 0x025772 (0x1c54 bytes) 000337 1790741666.107367 mapwrite 0x02f5f8 thru 0x035a46 (0x644f bytes) 000338 1790741666.150296 trunc from 0x02bdbf to 0x03770b (0xb94d bytes) 000339 1790741666.157233 punch from 0x0023a2 to 0x005d64 (0x39c3 bytes) 000340 1790741666.177026 write 0x02fa7e thru 0x037120 (0x76a3 bytes) 000342 1790741666.205184 trunc from 0x01d5d8 to 0x02d018 (0xfa41 bytes) 000343 1790741666.228259 fallocate 0x01aa50 thru 0x0262a4 (0xb855 bytes) 000345 1790741666.251949 mapwrite 0x012499 thru 0x01a94f (0x84b7 bytes) 000347 1790741666.304998 write 0x0175c2 thru 0x0202b6 (0x8cf5 bytes) 000349 1790741666.313328 mapwrite 0x020463 thru 0x0226b4 (0x2252 bytes) 000350 1790741666.358752 write 0x0100eb thru 0x01c09e (0xbfb4 bytes) 000351 1790741666.364299 trunc from 0x038d99 to 0x00e02d (0x2ad6b bytes) 000352 1790741666.402851 fallocate 0x03d77b thru 0x03ffff (0x2885 bytes) 000353 1790741666.413967 trunc from 0x00e02e to 0x0253ee (0x173c1 bytes) 000354 1790741666.433218 mapread 0x00bc8a thru 0x00e69f (0x2a16 bytes) 000355 1790741666.434188 write 0x03cae6 thru 0x03ffff (0x351a bytes) 000356 1790741666.436358 trunc from 0x012712 to 0x01ee1c (0xc70b bytes) 000357 1790741666.440261 trunc from 0x020231 to 0x02d130 (0xcf00 bytes) 000359 1790741666.445204 fallocate 0x015f24 thru 0x0165ea (0x6c7 bytes) 000360 1790741666.449922 punch from 0x02fd0c to 0x03864e (0x8943 bytes) 000361 1790741666.454242 trunc from 0x00731a to 0x009757 (0x243e bytes) 000362 1790741666.459185 read 0x01585b thru 0x01ef01 (0x96a7 bytes) 000364 1790741666.470198 write 0x02c660 thru 0x03a8ff (0xe2a0 bytes) 000365 1790741666.473563 punch from 0x00dbee to 0x016c77 (0x908a bytes) 000366 1790741666.479007 trunc from 0x012c19 to 0x022089 (0xf471 bytes) 000367 1790741666.483616 mapwrite 0x03df68 thru 0x03fca7 (0x1d40 bytes) 000369 1790741666.511061 fallocate 0x007c7f thru 0x015708 (0xda8a bytes) 000370 1790741666.515406 fallocate 0x024f1c thru 0x0320b8 (0xd19d bytes) 000371 1790741666.520670 mapread 0x01c766 thru 0x01e3e3 (0x1c7e bytes) 000373 1790741666.535848 mapwrite 0x0233c9 thru 0x02d765 (0xa39d bytes) 000374 1790741666.556380 trunc from 0x040000 to 0x01a634 (0x259cb bytes) 000375 1790741666.571657 mapread 0x016fc4 thru 0x01a634 (0x3671 bytes) 000376 1790741666.593203 read 0x01714e thru 0x0198d2 (0x2785 bytes) 000377 1790741666.593925 trunc from 0x01a635 to 0x01a3db (0x259 bytes) 000379 1790741666.607816 mapwrite 0x03112c thru 0x035876 (0x474b bytes) 000380 1790741666.650200 trunc from 0x0127c3 to 0x021a31 (0xf26f bytes) 000381 1790741666.654227 fallocate 0x02a426 thru 0x033dc2 (0x999d bytes) 000382 1790741666.658941 trunc from 0x035877 to 0x001218 (0x3465e bytes) 000383 1790741666.673649 mapwrite 0x005433 thru 0x00775a (0x2328 bytes) 000384 1790741666.711259 mapread 0x0030b5 thru 0x00775a (0x46a6 bytes) 000385 1790741666.721931 read 0x002ef9 thru 0x006b4b (0x3c53 bytes) 000386 1790741666.737399 trunc from 0x007501 to 0x00775a (0x25a bytes) 000387 1790741666.745188 mapwrite 0x0301bc thru 0x03535f (0x51a4 bytes) 000388 1790741666.796410 trunc from 0x02fe83 to 0x03535f (0x54dd bytes) 000389 1790741666.802083 punch from 0x00892f to 0x0187e6 (0xfeb8 bytes) 000392 1790741666.814124 trunc from 0x035360 to 0x019c4c (0x1b713 bytes) 000393 1790741666.842797 write 0x01156c thru 0x017409 (0x5e9e bytes) 000394 1790741666.862872 MIRROR_EXTEND to 2 mirrors 000395 1790741667.655930 punch from 0x016886 to 0x019c4c (0x33c7 bytes) 000396 1790741667.688857 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) 000397 1790741667.896413 trunc from 0x007d83 to 0x0169de (0xec5c bytes) 000398 1790741667.907224 write 0x004f56 thru 0x010a07 (0xbab2 bytes) 000399 1790741667.927939 trunc from 0x019c4d to 0x023d01 (0xa0b5 bytes) 000401 1790741667.953273 mapwrite 0x030f72 thru 0x035cfb (0x4d8a bytes) 000402 1790741668.071111 trunc from 0x02b0ab to 0x03237f (0x72d5 bytes) 000403 1790741668.093039 trunc from 0x001cce to 0x00bcd0 (0xa003 bytes) 000404 1790741668.102372 mapread 0x00c6d9 thru 0x011b43 (0x546b bytes) 000405 1790741668.120190 punch from 0x004ae4 to 0x005fea (0x1507 bytes) 000406 1790741668.127169 write 0x01fcd0 thru 0x023c38 (0x3f69 bytes) 000407 1790741668.151075 mapread 0x01f6f6 thru 0x028c16 (0x9521 bytes) 000408 1790741668.152496 fallocate 0x007393 thru 0x010bbe (0x982c bytes) 000409 1790741668.159338 mapread 0x006c82 thru 0x016743 (0xfac2 bytes) 000410 1790741668.178092 trunc from 0x035cfc to 0x0219b7 (0x14344 bytes) 000411 1790741668.208984 mapwrite 0x029e5b thru 0x02f5d4 (0x577a bytes) 000412 1790741668.274975 trunc from 0x02f5d5 to 0x02c783 (0x2e51 bytes) 000413 1790741668.303876 mapread 0x00318c thru 0x00d23b (0xa0b0 bytes) 000415 1790741668.310357 fallocate 0x01f524 thru 0x025eda (0x69b7 bytes) 000416 1790741668.321139 trunc from 0x00ec1f to 0x00f088 (0x46a bytes) 000418 1790741668.331933 fallocate 0x01c511 thru 0x01d030 (0xb20 bytes) 000419 1790741668.337174 mapread 0x01f2c5 thru 0x02a2ec (0xb028 bytes) 000420 1790741668.338907 punch from 0x015601 to 0x01d315 (0x7d15 bytes) 000421 1790741668.343633 mapwrite 0x03a551 thru 0x03ffff (0x5aaf bytes) 000422 1790741668.396871 trunc from 0x018839 to 0x018922 (0xea bytes) 000423 1790741668.402297 write 0x02a4ad thru 0x02a59e (0xf2 bytes) 000424 1790741668.403271 mapread 0x01de16 thru 0x02c133 (0xe31e bytes) 000425 1790741668.412918 punch from 0x03c256 to 0x03ffff (0x3daa bytes) 000426 1790741668.417369 trunc from 0x007932 to 0x01597c (0xe04b bytes) 000427 1790741668.422872 write 0x026f87 thru 0x02c0a1 (0x511b bytes) 000428 1790741668.424221 mapwrite 0x023a8e thru 0x0294d3 (0x5a46 bytes) 000429 1790741668.439315 trunc from 0x040000 to 0x005526 (0x3aad9 bytes) 000430 1790741668.453773 punch from 0x002378 to 0x005526 (0x31af bytes) 000431 1790741668.462684 fallocate 0x021c07 thru 0x023f75 (0x236f bytes) 000432 1790741668.469118 mapwrite 0x02eccf thru 0x03ac55 (0xbf87 bytes) 000433 1790741668.523861 trunc from 0x00685b to 0x0119a4 (0xb14a bytes) 000434 1790741668.529215 fallocate 0x01c3af thru 0x0282ac (0xbefe bytes) 000435 1790741668.534330 trunc from 0x0357d6 to 0x03ac55 (0x5480 bytes) 000436 1790741668.539801 write 0x00cac4 thru 0x014332 (0x786f bytes) 000437 1790741668.550308 read 0x01662a thru 0x0192a5 (0x2c7c bytes) 000438 1790741668.557712 trunc from 0x022c1f to 0x028efd (0x62df bytes) 000439 1790741668.563155 trunc from 0x03ac56 to 0x03ef80 (0x432b bytes) 000440 1790741668.578298 trunc from 0x03ef81 to 0x006984 (0x385fc bytes) 000441 1790741668.606449 mapread 0x000adb thru 0x006984 (0x5eaa bytes) 000442 1790741668.621356 trunc from 0x006985 to 0x03ad73 (0x343ef bytes) 000443 1790741668.640298 punch from 0x01d43a to 0x027557 (0xa11e bytes) 000444 1790741668.652988 fallocate 0x00922d thru 0x00cf53 (0x3d27 bytes) 000445 1790741668.660182 mapread 0x02cf71 thru 0x03251f (0x55af bytes) 000446 1790741668.675050 punch from 0x02a484 to 0x037350 (0xcecd bytes) 000447 1790741668.680147 trunc from 0x03ad74 to 0x02e1f3 (0xcb80 bytes) 000448 1790741668.693596 mapwrite 0x0228d2 thru 0x029856 (0x6f85 bytes) 000449 1790741668.745390 trunc from 0x02e1f4 to 0x00c550 (0x21ca3 bytes) 000450 1790741668.779134 fallocate 0x01a6b9 thru 0x01ebba (0x4502 bytes) 000451 1790741668.787413 trunc from 0x002440 to 0x00c550 (0xa111 bytes) 000452 1790741668.797952 fallocate 0x02631e thru 0x02f607 (0x92ea bytes) 000453 1790741668.812271 trunc from 0x02f608 to 0x037372 (0x7d6b bytes) 000455 1790741668.856170 punch from 0x02fdc9 to 0x031a05 (0x1c3d bytes) 000456 1790741668.862918 punch from 0x02a5a4 to 0x035ac7 (0xb524 bytes) 000458 1790741668.873452 mapread 0x0163f3 thru 0x01ce85 (0x6a93 bytes) 000459 1790741668.911556 trunc from 0x02ee0f to 0x031b1c (0x2d0e bytes) 000460 1790741668.920783 read 0x03207e thru 0x034ba9 (0x2b2c bytes) 000461 1790741668.936681 trunc from 0x037373 to 0x00bd6d (0x2b605 bytes) 000463 1790741668.998519 MIRROR_EXTEND to 2 mirrors 000464 1790741669.777513 fallocate 0x038616 thru 0x03ffff (0x79ea bytes) 000465 1790741669.806888 punch from 0x005fd0 to 0x00b3e6 (0x5417 bytes) 000466 1790741669.813085 read 0x003835 thru 0x007810 (0x3fdc bytes) 000467 1790741669.830178 mapread 0x00ad2e thru 0x00bd6d (0x1040 bytes) 000468 1790741669.848762 write 0x02272e thru 0x02a187 (0x7a5a bytes) 000469 1790741669.858457 mapread 0x017081 thru 0x01fd5b (0x8cdb bytes) 000470 1790741669.868751 write 0x0334a8 thru 0x03ffff (0xcb58 bytes) 000471 1790741669.871191 fallocate 0x01a991 thru 0x021fdf (0x764f bytes) 000472 1790741669.883061 MIRROR_RESYNC 2 mirrors 000473 1790741670.613336 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) 000474 1790741670.779036 mapwrite 0x013536 thru 0x01b71b (0x81e6 bytes) 000475 1790741670.896311 write 0x030048 thru 0x03449f (0x4458 bytes) 000476 1790741670.917574 fallocate 0x018777 thru 0x022c33 (0xa4bd bytes) 000477 1790741670.925276 fallocate 0x00d134 thru 0x01b855 (0xe722 bytes) 000478 1790741670.931926 mapread 0x01baaf thru 0x028fd8 (0xd52a bytes) 000479 1790741670.984291 mapread 0x0073ce thru 0x0149da (0xd60d bytes) 000480 1790741671.037992 trunc from 0x040000 to 0x03e71f (0x18e0 bytes) 000481 1790741671.067258 trunc from 0x03e720 to 0x005e0c (0x38913 bytes) 000482 1790741671.136309 read 0x00041c thru 0x004009 (0x3bee bytes) 000483 1790741671.158330 trunc from 0x005e0d to 0x01bab0 (0x15ca4 bytes) 000484 1790741671.190318 write 0x01cfda thru 0x0291a6 (0xc1cd bytes) 000485 1790741671.205956 trunc from 0x000d79 to 0x0040cb (0x3353 bytes) 000486 1790741671.219214 trunc from 0x004e51 to 0x013f54 (0xf104 bytes) 000487 1790741671.237254 punch from 0x00ae52 to 0x00c0d3 (0x1282 bytes) 000488 1790741671.257403 fallocate 0x013207 thru 0x01fcdc (0xcad6 bytes) 000490 1790741671.281104 MIRROR_EXTEND to 2 mirrors 000491 1790741672.341663 mapread 0x021160 thru 0x0291a6 (0x8047 bytes) 000492 1790741672.446243 mapread 0x01b447 thru 0x0291a6 (0xdd60 bytes) 000493 1790741672.487674 punch from 0x021224 to 0x0291a6 (0x7f83 bytes) 000494 1790741672.545170 write 0x02a974 thru 0x02ae31 (0x4be bytes) 000495 1790741672.560375 mapread 0x027c35 thru 0x02ae31 (0x31fd bytes) 000496 1790741672.584177 trunc from 0x02ae32 to 0x025122 (0x5d0f bytes) 000497 1790741672.679114 fallocate 0x013f94 thru 0x01bc8c (0x7cf9 bytes) 000498 1790741672.692492 read 0x0163fc thru 0x0211ff (0xae04 bytes) 000499 1790741672.721770 fallocate 0x029453 thru 0x02eeda (0x5a88 bytes) 000500 1790741672.735776 punch from 0x018fc7 to 0x01abe7 (0x1c21 bytes) 000501 1790741672.741688 read 0x0017a8 thru 0x0020e9 (0x942 bytes) 000502 1790741672.756066 read 0x019d21 thru 0x01bb30 (0x1e10 bytes) 000503 1790741672.758109 trunc from 0x025123 to 0x02dbba (0x8a98 bytes) 000504 1790741672.798854 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) 000507 1790741673.002425 mapread 0x0094dc thru 0x012b5c (0x9681 bytes) 000508 1790741673.038026 read 0x00b903 thru 0x016ee9 (0xb5e7 bytes) 000509 1790741673.051102 punch from 0x00398f to 0x009101 (0x5773 bytes) 000511 1790741673.072705 mapread 0x025d1f thru 0x02dbba (0x7e9c bytes) 000512 1790741673.130416 read 0x010ab8 thru 0x018690 (0x7bd9 bytes) 000513 1790741673.147829 write 0x00c411 thru 0x0138b6 (0x74a6 bytes) 000514 1790741673.160175 read 0x02222b thru 0x02dbba (0xb990 bytes) 000516 1790741673.166094 trunc from 0x02dbbb to 0x00413e (0x29a7c bytes) 000517 1790741673.205161 trunc from 0x0027b7 to 0x00413e (0x1988 bytes) 000518 1790741673.218021 write 0x025751 thru 0x030602 (0xaeb2 bytes) 000520 1790741673.225935 trunc from 0x0157ba to 0x023be7 (0xe42e bytes) 000522 1790741673.239464 MIRROR_EXTEND to 2 mirrors 000524 1790741674.270837 write 0x00f30c thru 0x013817 (0x450c bytes) 000525 1790741674.349116 write 0x0202c0 thru 0x02aba0 (0xa8e1 bytes) 000526 1790741674.390795 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x30603): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30603): Mirror 2: 0x58f29899 (stale) CRC-32 checksum value for chunk [0, 0x30603): Mirror 1: 0x4a25aeda Mirror 3: 0x4a25aeda lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30603: skipped 000527 1790741675.908357 mapwrite 0x02058d thru 0x02e24a (0xdcbe bytes) 000528 1790741676.062688 mapwrite 0x017682 thru 0x01ff53 (0x88d2 bytes) 000529 1790741676.137999 trunc from 0x030603 to 0x03fa61 (0xf45f bytes) 000530 1790741676.168066 fallocate 0x036cb0 thru 0x03ffff (0x9350 bytes) 000531 1790741676.175012 fallocate 0x0278a6 thru 0x032d28 (0xb483 bytes) 000532 1790741676.184312 read 0x02fe54 thru 0x033ef4 (0x40a1 bytes) 000534 1790741676.193885 trunc from 0x031bcd to 0x03c9d8 (0xae0c bytes) 000535 1790741676.204300 punch from 0x017773 to 0x020df8 (0x9686 bytes) 000536 1790741676.210031 mapread 0x01ecaf thru 0x027827 (0x8b79 bytes) 000537 1790741676.220216 fallocate 0x0340eb thru 0x038bab (0x4ac1 bytes) 000538 1790741676.224864 read 0x0329bc thru 0x03b693 (0x8cd8 bytes) 000539 1790741676.236743 trunc from 0x040000 to 0x008678 (0x37987 bytes) 000540 1790741676.260101 read 0x0008b7 thru 0x008678 (0x7dc2 bytes) 000541 1790741676.261439 punch from 0x003a4e to 0x008678 (0x4c2b bytes) 000542 1790741676.267635 mapwrite 0x0347f3 thru 0x03b28c (0x6a9a bytes) 000543 1790741676.349228 punch from 0x02657f to 0x0327df (0xc261 bytes) 000544 1790741676.365219 fallocate 0x00cbf7 thru 0x01599d (0x8da7 bytes) 000545 1790741676.373003 mapread 0x0252a6 thru 0x0274ca (0x2225 bytes) 000546 1790741676.401823 fallocate 0x0389dd thru 0x03ffff (0x7623 bytes) 000547 1790741676.418239 fallocate 0x030d71 thru 0x034b41 (0x3dd1 bytes) 000548 1790741676.423808 read 0x02e8a9 thru 0x03b28c (0xc9e4 bytes) 000549 1790741676.433881 mapread 0x015a7e thru 0x01af0d (0x5490 bytes) 000550 1790741676.434965 trunc from 0x031a40 to 0x03b28c (0x984d bytes) 000551 1790741676.447319 trunc from 0x010a58 to 0x019f2d (0x94d6 bytes) 000552 1790741676.463789 mapwrite 0x0139aa thru 0x0142fd (0x954 bytes) 000553 1790741676.496957 read 0x01217f thru 0x01dffd (0xbe7f bytes) 000554 1790741676.510802 trunc from 0x03b28d to 0x0037a8 (0x37ae4 bytes) 000555 1790741676.543118 mapread 0x002f89 thru 0x0037a8 (0x820 bytes) 000556 1790741676.548156 mapwrite 0x025df8 thru 0x030507 (0xa710 bytes) 000557 1790741676.719940 write 0x020d3d thru 0x02e415 (0xd6d9 bytes) 000558 1790741676.731038 fallocate 0x02a32e thru 0x03a132 (0xfe05 bytes) 000559 1790741676.743132 mapwrite 0x00103f thru 0x010241 (0xf203 bytes) 000560 1790741676.923666 mapwrite 0x038870 thru 0x03ffff (0x7790 bytes) 000561 1790741676.987618 write 0x02d1c1 thru 0x03b7ad (0xe5ed bytes) 000562 1790741676.992141 trunc from 0x00522c to 0x008389 (0x315e bytes) 000563 1790741677.000451 read 0x027e50 thru 0x031a8e (0x9c3f bytes) 000564 1790741677.001915 read 0x019fa0 thru 0x027267 (0xd2c8 bytes) 000566 1790741677.019086 MIRROR_RESYNC 3 mirrors 000568 1790741678.812899 mapwrite 0x035625 thru 0x03ea27 (0x9403 bytes) 000570 1790741679.000359 MIRROR_RESYNC 3 mirrors 000571 1790741680.231659 fallocate 0x03a5f2 thru 0x03ffff (0x5a0e bytes) 000572 1790741680.315972 mapwrite 0x00c812 thru 0x018661 (0xbe50 bytes) 000573 1790741680.410030 trunc from 0x00488b to 0x0089cd (0x4143 bytes) 000574 1790741680.416365 trunc from 0x040000 to 0x037e8d (0x8172 bytes) 000575 1790741680.438932 trunc from 0x037e8e to 0x030191 (0x7cfc bytes) 000576 1790741680.458958 read 0x0247db thru 0x02e452 (0x9c78 bytes) 000577 1790741680.481045 trunc from 0x030192 to 0x03c792 (0xc601 bytes) 000578 1790741680.519105 trunc from 0x02c245 to 0x03863c (0xc3f8 bytes) 000579 1790741680.530891 fallocate 0x017294 thru 0x01ba7e (0x47eb bytes) 000580 1790741680.551355 trunc from 0x03c793 to 0x027b46 (0x14c4c bytes) 000581 1790741680.583997 mapwrite 0x00d15a thru 0x01ab50 (0xd9f7 bytes) 000582 1790741680.646795 mapread 0x00fc66 thru 0x012b6e (0x2f09 bytes) 000583 1790741680.648006 fallocate 0x0176cc thru 0x01ffac (0x88e1 bytes) 000584 1790741680.672461 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27b47): Mirror 2: 0x42dfb21e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27b47): Mirror 2: 0x42dfb21e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27b47: skipped 000585 1790741681.230357 mapwrite 0x00c5e1 thru 0x00d7d3 (0x11f3 bytes) 000586 1790741681.293595 trunc from 0x027b47 to 0x015433 (0x12713 bytes) 000587 1790741681.347892 mapwrite 0x03cacd thru 0x03ffff (0x3533 bytes) 000588 1790741681.426014 write 0x03476c thru 0x036a00 (0x2295 bytes) 000589 1790741681.441961 punch from 0x01fac9 to 0x028a9d (0x8fd5 bytes) 000591 1790741681.451245 write 0x02f115 thru 0x037865 (0x8751 bytes) 000592 1790741681.465298 trunc from 0x040000 to 0x02cfe0 (0x1301f bytes) 000593 1790741681.521497 mapwrite 0x0397c5 thru 0x03ecd1 (0x550d bytes) 000594 1790741681.693599 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ecd2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ecd2): Mirror 2: 0x1c6b2eb3 (stale) CRC-32 checksum value for chunk [0, 0x3ecd2): Mirror 1: 0x37b6fe21 Mirror 3: 0x37b6fe21 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ecd2: skipped 000595 1790741683.064343 trunc from 0x03ecd2 to 0x00cde2 (0x31eef bytes) 000596 1790741683.119047 fallocate 0x03bcab thru 0x03ffff (0x4355 bytes) 000597 1790741683.129935 fallocate 0x01ccdf thru 0x0287bf (0xbae1 bytes) 000598 1790741683.143086 mapread 0x01bdc4 thru 0x01d676 (0x18b3 bytes) 000599 1790741683.177029 trunc from 0x0203fd to 0x021171 (0xd75 bytes) 000600 1790741683.182689 mapwrite 0x00bd83 thru 0x00f430 (0x36ae bytes) 000601 1790741683.222070 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xc81cef20 (stale) Mirror 3: 0xa5e29dd3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xb549ab97 Mirror 4: 0xb549ab97 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000602 1790741684.503883 trunc from 0x00192c to 0x01119c (0xf871 bytes) 000603 1790741684.557713 punch from 0x011df9 to 0x015b97 (0x3d9f bytes) 000604 1790741684.562915 write 0x01ccb9 thru 0x01e38c (0x16d4 bytes) 000605 1790741684.576913 mapwrite 0x022fa2 thru 0x0315a3 (0xe602 bytes) 000606 1790741684.624709 write 0x026e09 thru 0x030aa4 (0x9c9c bytes) 000607 1790741684.627228 mapread 0x03389c thru 0x03456b (0xcd0 bytes) 000608 1790741684.628897 trunc from 0x040000 to 0x0387dc (0x7823 bytes) 000609 1790741684.695058 read 0x02d53a thru 0x0312a0 (0x3d67 bytes) 000610 1790741684.695716 trunc from 0x01dd12 to 0x0215c9 (0x38b8 bytes) 000611 1790741684.703036 mapread 0x038752 thru 0x0387dc (0x8b bytes) 000612 1790741684.709993 mapread 0x0022a3 thru 0x011eec (0xfc4a bytes) 000613 1790741684.726980 mapread 0x0097ff thru 0x00b06e (0x1870 bytes) 000614 1790741684.729507 mapwrite 0x01d2a0 thru 0x01df9a (0xcfb bytes) 000615 1790741684.750133 fallocate 0x021744 thru 0x029846 (0x8103 bytes) 000616 1790741684.755044 fallocate 0x039483 thru 0x03ffff (0x6b7d bytes) 000617 1790741684.760861 mapwrite 0x00fb07 thru 0x010e42 (0x133c bytes) 000618 1790741684.772279 mapwrite 0x0243f6 thru 0x032028 (0xdc33 bytes) 000619 1790741684.790264 read 0x01923a thru 0x01fece (0x6c95 bytes) 000620 1790741684.790922 mapwrite 0x020dcc thru 0x02dd60 (0xcf95 bytes) 000621 1790741684.809829 trunc from 0x0387dd to 0x03650d (0x22cf bytes) 000622 1790741684.829908 punch from 0x01c212 to 0x0246c1 (0x84b0 bytes) 000623 1790741684.838432 read 0x02f1c4 thru 0x034bc8 (0x5a05 bytes) 000624 1790741684.839409 fallocate 0x02cea2 thru 0x03b67d (0xe7dc bytes) 000625 1790741684.846841 trunc from 0x022822 to 0x030491 (0xdc70 bytes) 000626 1790741684.851949 mapwrite 0x003140 thru 0x00f320 (0xc1e1 bytes) 000627 1790741684.883941 trunc from 0x03650e to 0x01fe3f (0x166ce bytes) 000628 1790741684.904901 mapwrite 0x0106b1 thru 0x01ed17 (0xe667 bytes) 000629 1790741684.951316 write 0x02c372 thru 0x039938 (0xd5c7 bytes) 000630 1790741684.954064 write 0x012a03 thru 0x016a40 (0x403e bytes) 000633 1790741684.957467 mapread 0x00ba73 thru 0x00f4c8 (0x3a56 bytes) 000634 1790741684.959748 mapwrite 0x027be2 thru 0x0374f8 (0xf917 bytes) 000635 1790741685.006869 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39939): Mirror 2: 0x67cecade (stale) Mirror 3: 0xf8a7004a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39939): Mirror 2: 0x67cecade (stale) Mirror 3: 0xf8a7004a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39939: skipped 000636 1790741685.417741 fallocate 0x010696 thru 0x01e32f (0xdc9a bytes) 000637 1790741685.428788 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39939): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39939): Mirror 2: 0x67cecade (stale) Mirror 3: 0xf8a7004a (stale) CRC-32 checksum value for chunk [0, 0x39939): Mirror 1: 0xfd56b6a4 Mirror 4: 0xfd56b6a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39939: skipped 000639 1790741686.807676 mapread 0x00592a thru 0x00daeb (0x81c2 bytes) 000640 1790741686.856393 write 0x015ce3 thru 0x018991 (0x2caf bytes) 000641 1790741686.936917 punch from 0x009c22 to 0x00e8d1 (0x4cb0 bytes) 000642 1790741686.942408 trunc from 0x039939 to 0x001800 (0x38138 bytes) 000644 1790741686.962143 mapread 0x000497 thru 0x001800 (0x136a bytes) 000645 1790741686.968364 fallocate 0x02e4a3 thru 0x03a277 (0xbdd5 bytes) 000646 1790741686.973400 read 0x000571 thru 0x001800 (0x1290 bytes) 000647 1790741686.973963 read 0x00016c thru 0x001800 (0x1695 bytes) 000648 1790741686.974490 punch from 0x000f43 to 0x001800 (0x8be bytes) 000650 1790741686.990417 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1801): Mirror 2: 0x36cd47e (stale) Mirror 3: 0x36cd47e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1801): Mirror 2: 0x36cd47e (stale) Mirror 3: 0x36cd47e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1801: skipped 000651 1790741687.331970 punch from 0x000037 to 0x001800 (0x17ca bytes) 000653 1790741687.342693 punch from 0x001547 to 0x001800 (0x2ba bytes) 000654 1790741687.351475 trunc from 0x00005f to 0x001800 (0x17a2 bytes) 000655 1790741687.357045 trunc from 0x001801 to 0x0133e8 (0x11be8 bytes) 000656 1790741687.390821 MIRROR_RESYNC 3 mirrors 000657 1790741688.162466 punch from 0x00a224 to 0x00f4b9 (0x5296 bytes) 000658 1790741688.222463 trunc from 0x0133e9 to 0x039b4e (0x26766 bytes) 000659 1790741688.242645 trunc from 0x017ae9 to 0x01ee96 (0x73ae bytes) 000660 1790741688.249102 punch from 0x01c411 to 0x0288ab (0xc49b bytes) 000661 1790741688.254468 punch from 0x0087d8 to 0x013fd2 (0xb7fb bytes) 000662 1790741688.259068 punch from 0x01fedb to 0x02a62b (0xa751 bytes) 000663 1790741688.268750 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39b4f): Mirror 3: 0x3210cb39 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39b4f): Mirror 3: 0x3210cb39 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39b4f: skipped 000664 1790741688.603578 fallocate 0x01d35a thru 0x0241d5 (0x6e7c bytes) 000665 1790741688.610345 mapwrite 0x01e3dd thru 0x026f42 (0x8b66 bytes) 000666 1790741688.681418 trunc from 0x039b4f to 0x02b447 (0xe707 bytes) 000667 1790741688.707589 mapwrite 0x006110 thru 0x00867f (0x2570 bytes) 000668 1790741688.730306 mapread 0x01bfe6 thru 0x025a53 (0x9a6e bytes) 000669 1790741688.731394 fallocate 0x0361db thru 0x037917 (0x173d bytes) 000670 1790741688.737238 mapread 0x01d2ce thru 0x024d0a (0x7a3d bytes) 000671 1790741688.739711 read 0x006d1e thru 0x0072c7 (0x5aa bytes) 000672 1790741688.740788 fallocate 0x03e1c1 thru 0x03ffff (0x1e3f bytes) 000673 1790741688.746987 read 0x006924 thru 0x00dcf3 (0x73d0 bytes) 000675 1790741688.748961 mapread 0x027ec8 thru 0x036c01 (0xed3a bytes) 000676 1790741688.767170 write 0x0161a9 thru 0x02000a (0x9e62 bytes) 000679 1790741688.770741 trunc from 0x040000 to 0x013b54 (0x2c4ab bytes) 000680 1790741688.792469 trunc from 0x013b55 to 0x03947c (0x25928 bytes) 000682 1790741688.813673 mapwrite 0x013ca5 thru 0x016143 (0x249f bytes) 000683 1790741688.849227 mapwrite 0x021e07 thru 0x028b48 (0x6d42 bytes) 000685 1790741688.928274 read 0x019358 thru 0x0290e3 (0xfd8c bytes) 000686 1790741688.944689 trunc from 0x0126f6 to 0x01d236 (0xab41 bytes) 000688 1790741688.950565 write 0x01ef5e thru 0x0231ce (0x4271 bytes) 000689 1790741688.952676 mapwrite 0x02ae24 thru 0x02ef44 (0x4121 bytes) 000691 1790741688.968932 trunc from 0x02344b to 0x02aebb (0x7a71 bytes) 000692 1790741688.975865 punch from 0x01dc45 to 0x028001 (0xa3bd bytes) 000693 1790741688.982453 trunc from 0x015396 to 0x01a676 (0x52e1 bytes) 000694 1790741688.987152 read 0x00ea25 thru 0x01a09e (0xb67a bytes) 000696 1790741688.998465 trunc from 0x0049b1 to 0x005884 (0xed4 bytes) 000697 1790741689.017219 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) 000698 1790741689.201421 mapwrite 0x010383 thru 0x0128e5 (0x2563 bytes) 000700 1790741689.270890 punch from 0x012b8e to 0x021eb1 (0xf324 bytes) 000703 1790741689.284447 read 0x014a79 thru 0x01ac8a (0x6212 bytes) 000704 1790741689.330276 trunc from 0x03947d to 0x034675 (0x4e07 bytes) 000705 1790741689.386654 mapread 0x02e234 thru 0x034675 (0x6442 bytes) 000706 1790741689.410082 fallocate 0x0262ef thru 0x033fe8 (0xdcfa bytes) 000707 1790741689.420194 trunc from 0x034676 to 0x0334e0 (0x1195 bytes) 000708 1790741689.441565 mapread 0x007204 thru 0x014a1a (0xd817 bytes) 000709 1790741689.463880 fallocate 0x03884d thru 0x03ffff (0x77b3 bytes) 000710 1790741689.472210 write 0x033d98 thru 0x039e20 (0x6089 bytes) 000711 1790741689.487025 MIRROR_EXTEND to 2 mirrors 000712 1790741690.797402 write 0x03dbdd thru 0x03ffff (0x2423 bytes) 000714 1790741690.865496 write 0x03b9a8 thru 0x03ffff (0x4658 bytes) 000715 1790741690.883677 fallocate 0x012670 thru 0x01fc22 (0xd5b3 bytes) 000716 1790741690.891494 trunc from 0x019195 to 0x0219aa (0x8816 bytes) 000717 1790741690.902255 trunc from 0x040000 to 0x03e3ea (0x1c15 bytes) 000718 1790741690.941405 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) 000719 1790741691.239715 fallocate 0x00ec86 thru 0x010687 (0x1a02 bytes) 000720 1790741691.254477 mapread 0x01ac5f thru 0x024dce (0xa170 bytes) 000721 1790741691.297128 mapread 0x0066f9 thru 0x00d9a8 (0x72b0 bytes) 000722 1790741691.314081 mapwrite 0x004215 thru 0x0097c4 (0x55b0 bytes) 000724 1790741691.372970 trunc from 0x033adb to 0x03e3ea (0xa910 bytes) 000725 1790741691.396894 MIRROR_EXTEND to 2 mirrors 000726 1790741692.516375 mapread 0x00e96f thru 0x01c281 (0xd913 bytes) 000727 1790741692.563668 mapread 0x01a328 thru 0x02354f (0x9228 bytes) 000728 1790741692.587823 trunc from 0x03e3eb to 0x037c1b (0x67cf bytes) 000729 1790741692.634693 trunc from 0x015679 to 0x023058 (0xd9e0 bytes) 000730 1790741692.642866 read 0x001eb1 thru 0x005985 (0x3ad5 bytes) 000731 1790741692.653291 mapread 0x02cab8 thru 0x0345f4 (0x7b3d bytes) 000732 1790741692.666831 write 0x024962 thru 0x02b914 (0x6fb3 bytes) 000733 1790741692.676437 write 0x01d9c2 thru 0x02b677 (0xdcb6 bytes) 000734 1790741692.689266 trunc from 0x037c1c to 0x00f09b (0x28b80 bytes) 000736 1790741692.737086 trunc from 0x00f09c to 0x01adf3 (0xbd58 bytes) 000738 1790741692.756435 read 0x00fdf8 thru 0x011793 (0x199c bytes) 000739 1790741692.767903 write 0x010a88 thru 0x0198b6 (0x8e2f bytes) 000740 1790741692.770195 punch from 0x00892c to 0x0170a7 (0xe77c bytes) 000741 1790741692.789395 punch from 0x006083 to 0x015a61 (0xf9df bytes) 000742 1790741692.795845 trunc from 0x003976 to 0x00fa71 (0xc0fc bytes) 000743 1790741692.801824 trunc from 0x0152e3 to 0x01adf3 (0x5b11 bytes) 000744 1790741692.807686 mapread 0x00e03e thru 0x00fdbf (0x1d82 bytes) 000745 1790741692.820152 trunc from 0x00f679 to 0x019482 (0x9e0a bytes) 000746 1790741692.827482 read 0x014371 thru 0x01592f (0x15bf bytes) 000747 1790741692.846113 write 0x01b9f2 thru 0x01e3e6 (0x29f5 bytes) 000748 1790741692.848009 trunc from 0x010326 to 0x017e46 (0x7b21 bytes) 000749 1790741692.858629 read 0x003f72 thru 0x012615 (0xe6a4 bytes) 000750 1790741692.876661 read 0x001627 thru 0x0079fb (0x63d5 bytes) 000751 1790741692.881522 read 0x017c91 thru 0x01e3e6 (0x6756 bytes) 000752 1790741692.892029 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x1e3e7): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e3e7): Mirror 2: 0x9b64488a (stale) CRC-32 checksum value for chunk [0, 0x1e3e7): Mirror 1: 0xc202205 Mirror 3: 0xc202205 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e3e7: skipped 000753 1790741694.106279 read 0x00b6a4 thru 0x0102f9 (0x4c56 bytes) 000754 1790741694.117228 write 0x02400f thru 0x029110 (0x5102 bytes) 000755 1790741694.185460 write 0x01ea69 thru 0x02d74a (0xece2 bytes) 000756 1790741694.192806 punch from 0x022fbe to 0x024c21 (0x1c64 bytes) 000757 1790741694.225613 fallocate 0x034e2f thru 0x03e962 (0x9b34 bytes) 000759 1790741694.234833 fallocate 0x024a36 thru 0x028380 (0x394b bytes) 000760 1790741694.244297 fallocate 0x000248 thru 0x00dced (0xdaa6 bytes) 000761 1790741694.250228 punch from 0x0361b2 to 0x0399e9 (0x3838 bytes) 000762 1790741694.257821 read 0x00a72a thru 0x01543f (0xad16 bytes) 000763 1790741694.275834 write 0x036a6a thru 0x038792 (0x1d29 bytes) 000764 1790741694.277570 mapread 0x008cf8 thru 0x00d250 (0x4559 bytes) 000765 1790741694.278438 mapread 0x03005b thru 0x03df0c (0xdeb2 bytes) 000766 1790741694.280216 mapread 0x00d61f thru 0x016bda (0x95bc bytes) 000767 1790741694.281236 write 0x0296b4 thru 0x02ad2e (0x167b bytes) 000768 1790741694.282657 fallocate 0x01b6bd thru 0x029552 (0xde96 bytes) 000769 1790741694.287967 mapread 0x011d5e thru 0x018c85 (0x6f28 bytes) 000770 1790741694.288882 mapwrite 0x00b3a9 thru 0x019943 (0xe59b bytes) 000772 1790741694.318288 fallocate 0x03392a thru 0x03a5b4 (0x6c8b bytes) 000773 1790741694.332479 MIRROR_RESYNC 3 mirrors 000774 1790741696.047875 mapread 0x010e97 thru 0x015752 (0x48bc bytes) 000775 1790741696.080197 trunc from 0x024fa7 to 0x028c9f (0x3cf9 bytes) 000776 1790741696.121132 trunc from 0x021c9c to 0x02a6bb (0x8a20 bytes) 000777 1790741696.132156 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e963): Mirror 3: 0x40aad0db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e963): Mirror 3: 0x40aad0db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e963: skipped 000778 1790741696.463655 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3e963): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e963): Mirror 3: 0x40aad0db (stale) CRC-32 checksum value for chunk [0, 0x3e963): Mirror 1: 0x650d77c0 Mirror 4: 0x650d77c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e963: skipped 000779 1790741697.772314 mapread 0x00ff2d thru 0x01897c (0x8a50 bytes) 000780 1790741697.805650 fallocate 0x0005cc thru 0x01013c (0xfb71 bytes) 000781 1790741697.870073 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e963): Mirror 3: 0x40aad0db (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e963): Mirror 3: 0x40aad0db (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e963: skipped 000783 1790741698.289658 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) 000784 1790741698.426932 trunc from 0x03e963 to 0x00b726 (0x3323c bytes) 000785 1790741698.453895 trunc from 0x00b252 to 0x00b726 (0x4d5 bytes) 000786 1790741698.464646 MIRROR_EXTEND to 2 mirrors 000788 1790741699.120713 mapread 0x004d87 thru 0x008c16 (0x3e90 bytes) 000789 1790741699.164778 fallocate 0x033146 thru 0x03b7c3 (0x867e bytes) 000790 1790741699.220465 trunc from 0x02ca6c to 0x03b7c3 (0xed58 bytes) 000791 1790741699.233805 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b7c4): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b7c4): Mirror 2: 0x276e9778 (stale) CRC-32 checksum value for chunk [0, 0x3b7c4): Mirror 1: 0x276e9778 Mirror 3: 0x276e9778 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b7c4: skipped 000792 1790741700.235099 read 0x0320ee thru 0x03b7c3 (0x96d6 bytes) 000793 1790741700.245209 punch from 0x002ae8 to 0x0067f2 (0x3d0b bytes) 000794 1790741700.286692 punch from 0x014b40 to 0x02253f (0xda00 bytes) 000795 1790741700.297730 mapwrite 0x00018a thru 0x007036 (0x6ead bytes) 000796 1790741700.361240 mapwrite 0x035f6f thru 0x03ffff (0xa091 bytes) 000797 1790741700.459129 read 0x025e04 thru 0x0287d9 (0x29d6 bytes) 000798 1790741700.474839 write 0x02a255 thru 0x02c5ee (0x239a bytes) 000800 1790741700.477397 write 0x039d0b thru 0x03ffff (0x62f5 bytes) 000801 1790741700.479378 mapread 0x026934 thru 0x02ebfa (0x82c7 bytes) 000803 1790741700.481363 read 0x0046d3 thru 0x012ec9 (0xe7f7 bytes) 000804 1790741700.482571 trunc from 0x040000 to 0x022363 (0x1dc9c bytes) 000805 1790741700.500415 mapwrite 0x031516 thru 0x03f90a (0xe3f5 bytes) 000807 1790741700.567500 trunc from 0x01bd3f to 0x02358d (0x784f bytes) 000808 1790741700.579304 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f90b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f90b): Mirror 2: 0xe962be14 (stale) Mirror 3: 0xe962be14 (stale) CRC-32 checksum value for chunk [0, 0x3f90b): Mirror 1: 0xf3ab66d8 Mirror 4: 0xf3ab66d8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f90b: skipped 000810 1790741701.616420 write 0x030190 thru 0x03c929 (0xc79a bytes) 000811 1790741701.703510 punch from 0x0176ff to 0x01cc8e (0x5590 bytes) 000812 1790741701.716245 write 0x01cb11 thru 0x026ea4 (0xa394 bytes) 000813 1790741701.729721 trunc from 0x00df75 to 0x015c7f (0x7d0b bytes) 000814 1790741701.740241 write 0x006b91 thru 0x00eb22 (0x7f92 bytes) 000815 1790741701.761919 fallocate 0x007279 thru 0x0098b4 (0x263c bytes) 000816 1790741701.777617 mapwrite 0x00b4d6 thru 0x0175d9 (0xc104 bytes) 000818 1790741701.876561 read 0x00c102 thru 0x01a743 (0xe642 bytes) 000819 1790741701.890922 punch from 0x01f3a1 to 0x02d607 (0xe267 bytes) 000820 1790741701.907766 punch from 0x01e789 to 0x021a5b (0x32d3 bytes) 000821 1790741701.923373 trunc from 0x03f90b to 0x000441 (0x3f4c9 bytes) 000822 1790741701.968655 trunc from 0x000442 to 0x02e34c (0x2df0b bytes) 000824 1790741701.994017 write 0x01a801 thru 0x028b91 (0xe391 bytes) 000825 1790741702.007704 mapwrite 0x00d29b thru 0x017abf (0xa825 bytes) 000826 1790741702.086714 trunc from 0x025103 to 0x02cce4 (0x7be2 bytes) 000827 1790741702.092332 trunc from 0x029fbf to 0x02a464 (0x4a6 bytes) 000828 1790741702.098047 trunc from 0x0077dd to 0x00a899 (0x30bd bytes) 000829 1790741702.103847 mapread 0x0228d4 thru 0x027dbe (0x54eb bytes) 000830 1790741702.116019 mapread 0x004bdd thru 0x005b91 (0xfb5 bytes) 000831 1790741702.128340 trunc from 0x02e34d to 0x012d73 (0x1b5d9 bytes) 000832 1790741702.171631 read 0x0121da thru 0x012d73 (0xb9a bytes) 000833 1790741702.172242 mapread 0x004ce5 thru 0x00beef (0x720b bytes) 000834 1790741702.173116 mapwrite 0x027095 thru 0x02ae12 (0x3d7e bytes) 000835 1790741702.266150 punch from 0x00fc00 to 0x01aac9 (0xaeca bytes) 000836 1790741702.291261 trunc from 0x02ae13 to 0x0343c4 (0x95b2 bytes) 000838 1790741702.320194 mapread 0x0286bf thru 0x0310aa (0x89ec bytes) 000839 1790741702.354235 read 0x0007cd thru 0x00fd81 (0xf5b5 bytes) 000840 1790741702.387174 punch from 0x00a807 to 0x01290a (0x8104 bytes) 000841 1790741702.392419 mapread 0x012607 thru 0x01b574 (0x8f6e bytes) 000842 1790741702.404359 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x343c5): Mirror 3: 0x6f6ca70e (stale) Mirror 4: 0x5e3ef14f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x343c5): Mirror 3: 0x6f6ca70e (stale) Mirror 4: 0x5e3ef14f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x343c5: skipped 000843 1790741702.869978 read 0x021888 thru 0x023e81 (0x25fa bytes) 000844 1790741702.875475 trunc from 0x0343c5 to 0x01f41a (0x14faa bytes) 000845 1790741702.909487 fallocate 0x035dff thru 0x0363a6 (0x5a8 bytes) 000846 1790741702.926652 mapread 0x00be4f thru 0x01a7a4 (0xe956 bytes) 000847 1790741702.986430 punch from 0x008f09 to 0x017be5 (0xecdd bytes) 000848 1790741702.992146 trunc from 0x01f41b to 0x01623a (0x91e0 bytes) 000849 1790741703.010093 fallocate 0x00c1ed thru 0x01bd16 (0xfb2a bytes) 000850 1790741703.015367 mapwrite 0x03726b thru 0x03ffff (0x8d95 bytes) 000851 1790741703.064339 write 0x00edf3 thru 0x01ed2d (0xff3b bytes) 000852 1790741703.076723 read 0x01e698 thru 0x02aa9d (0xc406 bytes) 000853 1790741703.090429 punch from 0x0097c7 to 0x00ce32 (0x366c bytes) 000854 1790741703.103635 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xe20eea22 (stale) Mirror 4: 0x8d5e07fe (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2c740205 Mirror 5: 0x2c740205 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000855 1790741704.277058 mapwrite 0x0226d0 thru 0x0251ba (0x2aeb bytes) 000856 1790741704.420954 trunc from 0x0116c2 to 0x01314a (0x1a89 bytes) 000857 1790741704.428481 punch from 0x02151c to 0x02de15 (0xc8fa bytes) 000858 1790741704.435072 punch from 0x01fc48 to 0x02899f (0x8d58 bytes) 000859 1790741704.440236 trunc from 0x040000 to 0x02dcaa (0x12355 bytes) 000860 1790741704.498134 MIRROR_RESYNC 4 mirrors 000861 1790741705.894169 punch from 0x00c680 to 0x01b7a3 (0xf124 bytes) 000863 1790741705.947508 trunc from 0x02518d to 0x026ab2 (0x1926 bytes) 000864 1790741705.963938 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dcab): Mirror 3: 0x8b54704f (stale) Mirror 5: 0x8b54704f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dcab): Mirror 3: 0x8b54704f (stale) Mirror 5: 0x8b54704f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dcab: skipped 000865 1790741706.411185 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dcab): Mirror 5: 0x8b54704f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dcab): Mirror 5: 0x8b54704f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dcab: skipped 000866 1790741706.743142 trunc from 0x02dcab to 0x01d285 (0x10a25 bytes) 000867 1790741706.768196 read 0x0086ac thru 0x00a843 (0x2198 bytes) 000868 1790741706.776118 punch from 0x01a4a5 to 0x01d285 (0x2de1 bytes) 000869 1790741706.783171 read 0x018531 thru 0x01d048 (0x4b18 bytes) 000870 1790741706.795972 read 0x011696 thru 0x0153f9 (0x3d64 bytes) 000871 1790741706.796838 fallocate 0x00ecfe thru 0x0107e4 (0x1ae7 bytes) 000872 1790741706.812070 MIRROR_RESYNC 2 mirrors 000873 1790741707.551934 fallocate 0x02d24b thru 0x036df8 (0x9bae bytes) 000874 1790741707.584269 mapwrite 0x03fc63 thru 0x03ffff (0x39d bytes) 000875 1790741707.624120 mapread 0x01b25c thru 0x020d05 (0x5aaa bytes) 000876 1790741707.638850 write 0x027836 thru 0x02b88c (0x4057 bytes) 000877 1790741707.648466 read 0x018d57 thru 0x026c4d (0xdef7 bytes) 000878 1790741707.654173 write 0x03d2b2 thru 0x03ffff (0x2d4e bytes) 000879 1790741707.664924 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) 000880 1790741707.807636 mapread 0x03955d thru 0x03ffff (0x6aa3 bytes) 000881 1790741707.833325 mapwrite 0x028c92 thru 0x02c862 (0x3bd1 bytes) 000882 1790741707.863504 mapread 0x03472e thru 0x03f62a (0xaefd bytes) 000883 1790741707.864502 punch from 0x02a2bb to 0x02c95b (0x26a1 bytes) 000884 1790741707.870964 read 0x01fdb5 thru 0x01ff7d (0x1c9 bytes) 000885 1790741707.875736 trunc from 0x034113 to 0x037cc2 (0x3bb0 bytes) 000886 1790741707.881472 read 0x031529 thru 0x034a66 (0x353e bytes) 000887 1790741707.892470 trunc from 0x02473c to 0x031bad (0xd472 bytes) 000888 1790741707.898454 trunc from 0x03deb9 to 0x03ffff (0x2147 bytes) 000889 1790741707.903122 mapread 0x02bbbb thru 0x0324e5 (0x692b bytes) 000890 1790741707.937394 write 0x029206 thru 0x02de77 (0x4c72 bytes) 000891 1790741707.956916 MIRROR_EXTEND to 2 mirrors 000892 1790741709.042263 trunc from 0x00fce2 to 0x0127d5 (0x2af4 bytes) 000893 1790741709.093025 mapread 0x024eff thru 0x02e6ed (0x97ef bytes) 000894 1790741709.133089 punch from 0x020a0d to 0x0298a0 (0x8e94 bytes) 000895 1790741709.153928 read 0x01bcb3 thru 0x01bdd5 (0x123 bytes) 000896 1790741709.174141 write 0x00149f thru 0x00f3f8 (0xdf5a bytes) 000897 1790741709.226783 trunc from 0x017321 to 0x01dc82 (0x6962 bytes) 000898 1790741709.235401 punch from 0x003405 to 0x0126bb (0xf2b7 bytes) 000899 1790741709.299928 punch from 0x03c677 to 0x03ffff (0x3989 bytes) 000900 1790741709.314081 read 0x00526a thru 0x00f105 (0x9e9c bytes) 000901 1790741709.342265 trunc from 0x040000 to 0x027159 (0x18ea6 bytes) 000902 1790741709.376452 trunc from 0x005b7c to 0x00a571 (0x49f6 bytes) 000903 1790741709.391295 read 0x00987f thru 0x016839 (0xcfbb bytes) 000904 1790741709.406927 read 0x01e15c thru 0x0256c8 (0x756d bytes) 000905 1790741709.422870 read 0x0135f4 thru 0x019f29 (0x6936 bytes) 000906 1790741709.425318 read 0x01471f thru 0x0234db (0xedbd bytes) 000907 1790741709.426892 trunc from 0x02715a to 0x0342e7 (0xd18e bytes) 000908 1790741709.471203 punch from 0x0241ff to 0x02f5a9 (0xb3ab bytes) 000909 1790741709.484916 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x342e8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x342e8): Mirror 2: 0x6a2483bb (stale) CRC-32 checksum value for chunk [0, 0x342e8): Mirror 1: 0xd5d12bbd Mirror 3: 0xd5d12bbd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x342e8: skipped 000910 1790741710.906382 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x342e8): Mirror 2: 0x6a2483bb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x342e8): Mirror 2: 0x6a2483bb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x342e8: skipped 000911 1790741711.408712 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x342e8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x342e8): Mirror 2: 0x6a2483bb (stale) CRC-32 checksum value for chunk [0, 0x342e8): Mirror 1: 0xd5d12bbd Mirror 3: 0xd5d12bbd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x342e8: skipped 000912 1790741712.729928 trunc from 0x0342e8 to 0x028571 (0xbd76 bytes) 000913 1790741712.811110 trunc from 0x028572 to 0x01c655 (0xbf1c bytes) 000915 1790741712.836104 fallocate 0x007b6a thru 0x0147f5 (0xcc8c bytes) 000916 1790741712.845708 punch from 0x000e62 to 0x00c83f (0xb9de bytes) 000917 1790741712.855456 fallocate 0x035f83 thru 0x03634c (0x3ca bytes) 000918 1790741712.863618 read 0x01fa62 thru 0x029a06 (0x9fa5 bytes) 000919 1790741712.876111 trunc from 0x03634d to 0x028f47 (0xd405 bytes) 000921 1790741712.912258 MIRROR_RESYNC 3 mirrors 000922 1790741713.707478 trunc from 0x028f48 to 0x03e045 (0x150fe bytes) 000923 1790741713.780720 punch from 0x010eb0 to 0x016328 (0x5479 bytes) 000924 1790741713.786364 mapwrite 0x00ee4a thru 0x0113a4 (0x255b bytes) 000925 1790741713.821386 trunc from 0x03e046 to 0x01f572 (0x1ead3 bytes) 000926 1790741713.850338 punch from 0x00f405 to 0x01ed03 (0xf8ff bytes) 000927 1790741713.857223 write 0x00c0e8 thru 0x016ebc (0xadd5 bytes) 000928 1790741713.864052 trunc from 0x00f2b0 to 0x017661 (0x83b2 bytes) 000929 1790741713.887397 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1f573): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f573): Mirror 2: 0xfee134e6 (stale) Mirror 3: 0xfee134e6 (stale) CRC-32 checksum value for chunk [0, 0x1f573): Mirror 1: 0xd11cfcc1 Mirror 4: 0xd11cfcc1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f573: skipped 000930 1790741715.272356 write 0x025ee6 thru 0x032699 (0xc7b4 bytes) 000931 1790741715.327693 mapwrite 0x01e425 thru 0x026ddf (0x89bb bytes) 000932 1790741715.459244 write 0x015111 thru 0x01696d (0x185d bytes) 000933 1790741715.483460 trunc from 0x03269a to 0x029bd3 (0x8ac6 bytes) 000934 1790741715.549721 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x29bd4): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29bd4): Mirror 2: 0x944010fd (stale) Mirror 3: 0x944010fd (stale) Mirror 4: 0x53a16271 (stale) CRC-32 checksum value for chunk [0, 0x29bd4): Mirror 1: 0x192fea4c Mirror 5: 0x192fea4c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29bd4: skipped 000935 1790741717.047911 trunc from 0x029bd4 to 0x03ff92 (0x163bf bytes) 000936 1790741717.097923 fallocate 0x0184ad thru 0x026cc2 (0xe816 bytes) 000937 1790741717.103157 trunc from 0x016794 to 0x02453c (0xdda9 bytes) 000938 1790741717.108243 fallocate 0x017d89 thru 0x018bd9 (0xe51 bytes) 000939 1790741717.114321 mapread 0x012da9 thru 0x01d010 (0xa268 bytes) 000941 1790741717.150835 punch from 0x030113 to 0x03886d (0x875b bytes) 000942 1790741717.160708 read 0x01fdee thru 0x02ded7 (0xe0ea bytes) 000943 1790741717.174751 write 0x0239f2 thru 0x02f725 (0xbd34 bytes) 000944 1790741717.176079 write 0x001892 thru 0x00d340 (0xbaaf bytes) 000945 1790741717.194258 fallocate 0x031057 thru 0x039c72 (0x8c1c bytes) 000946 1790741717.213378 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3ff93): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ff93): Mirror 2: 0x1b8e924d (stale) Mirror 3: 0x1b8e924d (stale) Mirror 4: 0x1f094457 (stale) Mirror 5: 0xaa57632e (stale) CRC-32 checksum value for chunk [0, 0x3ff93): Mirror 1: 0x4ab87061 Mirror 6: 0x4ab87061 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ff93: skipped 000947 1790741718.564001 mapread 0x007589 thru 0x00c5bc (0x5034 bytes) 000948 1790741718.608307 trunc from 0x01d95f to 0x0283e7 (0xaa89 bytes) 000949 1790741718.696340 punch from 0x006520 to 0x006c1b (0x6fc bytes) 000950 1790741718.705272 trunc from 0x01e51c to 0x022047 (0x3b2c bytes) 000951 1790741718.710960 trunc from 0x03ff93 to 0x023e59 (0x1c139 bytes) 000952 1790741718.769335 mapread 0x022557 thru 0x023e59 (0x1903 bytes) 000953 1790741718.783294 trunc from 0x010c31 to 0x01b4a8 (0xa878 bytes) 000954 1790741718.788248 punch from 0x01416d to 0x01b5df (0x7473 bytes) 000955 1790741718.792933 punch from 0x004f21 to 0x013aa7 (0xeb87 bytes) 000956 1790741718.797805 trunc from 0x023e5a to 0x002dd6 (0x21083 bytes) 000957 1790741718.816976 mapwrite 0x0192a4 thru 0x01a981 (0x16de bytes) 000958 1790741718.886664 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x1a982): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a982): Mirror 2: 0x7523553c (stale) Mirror 3: 0x7523553c (stale) Mirror 4: 0xaca40e6d (stale) Mirror 5: 0xec27f5a3 (stale) Mirror 6: 0x9db84e13 (stale) CRC-32 checksum value for chunk [0, 0x1a982): Mirror 1: 0x4319a54f Mirror 7: 0x4319a54f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a982: skipped 000959 1790741720.177385 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x163:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1a982): Mirror 2: 0x7523553c (stale) Mirror 4: 0xaca40e6d (stale) Mirror 5: 0xec27f5a3 (stale) Mirror 6: 0x9db84e13 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1a982): Mirror 2: 0x7523553c (stale) Mirror 4: 0xaca40e6d (stale) Mirror 5: 0xec27f5a3 (stale) Mirror 6: 0x9db84e13 (stale) CRC-32 checksum value for chunk [0, 0x1a982): Mirror 1: 0x4319a54f Mirror 7: 0x4319a54f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1a982: skipped 000960 1790741721.358965 write 0x035672 thru 0x03ffff (0xa98e bytes) 000961 1790741721.430143 mapwrite 0x017767 thru 0x01a0c5 (0x295f bytes) 000962 1790741721.521678 write 0x036d42 thru 0x03bccd (0x4f8c bytes) 000963 1790741721.528549 trunc from 0x013b4c to 0x0230fb (0xf5b0 bytes) 000964 1790741721.542983 read 0x0033e8 thru 0x003827 (0x440 bytes) 000966 1790741721.553099 mapread 0x01ae90 thru 0x020992 (0x5b03 bytes) 000967 1790741721.597646 write 0x035ae3 thru 0x03c6c4 (0x6be2 bytes) 000968 1790741721.604126 trunc from 0x03796c to 0x03959f (0x1c34 bytes) 000969 1790741721.618070 punch from 0x016e9b to 0x0247f4 (0xd95a bytes) 000971 1790741721.632830 read 0x03b5dd thru 0x03fc16 (0x463a bytes) 000974 1790741721.662893 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x99cfa8e8 (stale) Mirror 4: 0x8418938d (stale) Mirror 5: 0x3b39a1b7 (stale) Mirror 6: 0x45420159 (stale) Mirror 7: 0x99fdbd51 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa802dff3 Mirror 8: 0xa802dff3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000975 1790741723.090193 read 0x018e52 thru 0x02364d (0xa7fc bytes) 000976 1790741723.113784 mapread 0x0306d0 thru 0x03ffff (0xf930 bytes) 000977 1790741723.167160 read 0x0177c5 thru 0x0209a2 (0x91de bytes) 000978 1790741723.184206 trunc from 0x01d3f6 to 0x01ffc0 (0x2bcb bytes) 000979 1790741723.234288 mapwrite 0x0012d7 thru 0x003a3b (0x2765 bytes) 000980 1790741723.292830 punch from 0x00c544 to 0x00ca41 (0x4fe bytes) 000981 1790741723.314434 write 0x0320ee thru 0x037b35 (0x5a48 bytes) 000982 1790741723.368416 trunc from 0x0180df to 0x01f634 (0x7556 bytes) 000983 1790741723.378952 trunc from 0x016978 to 0x0232c9 (0xc952 bytes) 000984 1790741723.402783 read 0x002bf3 thru 0x00fae6 (0xcef4 bytes) 000985 1790741723.433720 write 0x02d375 thru 0x02f511 (0x219d bytes) 000986 1790741723.440978 mapwrite 0x02213c thru 0x0259b9 (0x387e bytes) 000988 1790741723.508186 trunc from 0x03aaeb to 0x03ffff (0x5515 bytes) 000990 1790741723.530664 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x99cfa8e8 (stale) Mirror 4: 0x8418938d (stale) Mirror 5: 0x3b39a1b7 (stale) Mirror 6: 0x45420159 (stale) Mirror 7: 0x99fdbd51 (stale) Mirror 8: 0xa802dff3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4b73034b Mirror 9: 0x4b73034b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000992 1790741725.192855 mapread 0x00200f thru 0x0083fc (0x63ee bytes) 000993 1790741725.231721 mapwrite 0x0286f9 thru 0x032ff4 (0xa8fc bytes) 000995 1790741725.432088 mapwrite 0x001d67 thru 0x00283f (0xad9 bytes) 000996 1790741725.472292 read 0x028bb8 thru 0x034c52 (0xc09b bytes) 000997 1790741725.473189 fallocate 0x00f9a8 thru 0x0193f8 (0x9a51 bytes) 000999 1790741725.484012 mapwrite 0x0071fc thru 0x015a23 (0xe828 bytes) 001000 1790741725.543882 MIRROR_RESYNC 8 mirrors All operations completed A-OK! PASS 70a (105s) == sanity-flr test 71: check mirror extend parent fid ==== 00:15:36 (1790741736) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x1a2:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (13s) == sanity-flr test 72: check append on FLR file ========== 00:15:50 (1790741750) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.202628 s, 10.3 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133813 s, 7.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 00:16:01 (1790741761) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre2 Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 3476320 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7344352 .. Extending file size to 8266144 .. Extending file size to 7176672 .. Extending file size to 8393216 .. Extending file size to 8297888 .. Extending file size to 8406432 .. Extending file size to 8810656 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8651520 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9019296 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 61981 61983 61984 61986 61987 failed 192.168.204.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg419-client.virtnet /mnt/lustre2 (opts:) 192.168.204.119@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg419-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 00:17:29 (1790741849) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre2 Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8389888 .. Extending file size to 4752288 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7000000 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8804192 .. Extending file size to 8813952 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9155456 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 63588 63589 63591 63592 63595 done 192.168.204.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg419-client.virtnet /mnt/lustre2 (opts:) 192.168.204.119@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg419-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (81s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 00:18:51 (1790741931) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2180 1269124 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2144 1269160 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 18896 3588124 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 24688 3582332 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 43584 7170456 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2180 1269124 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2144 1269160 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 18896 3588124 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 24688 3582332 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 43584 7170456 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.127459 s, 16.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.237741 s, 8.8 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xbdc254ea (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg419-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 00:19:13 (1790741953) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164257 s, 12.8 MB/s PASS 202 (10s) == sanity-flr test 203: mirror file preserve mirror ID === 00:19:23 (1790741963) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xb3:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14c:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14d:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12e:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 00:19:32 (1790741972) striped dir -i0 -c2 -H crush /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.123342 s, 17.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fa:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1fd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1fe:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 00:19:42 (1790741982) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.12538 s, 16.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fc:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ff:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fc:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1ff:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x200:0x0] } PASS 204b (11s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 00:19:53 (1790741993) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.119048 s, 17.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x201:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fd:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x201:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x202:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 00:20:29 (1790742029) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg419-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105093 s, 10.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fe:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x204:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x203:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1fe:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x204:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x203:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1ff:0x0] } PASS 204d (57s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 00:21:26 (1790742086) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg419-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.150239 s, 14.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x14f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x12f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (56s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 00:22:22 (1790742142) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg419-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg419-server: Pool lustre.test_204f created Adding targets to pool oleg419-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg419-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.144403 s, 14.5 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x150:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x130:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg419-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg419-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg419-server: Pool lustre.test_204f destroyed PASS 204f (82s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 00:23:44 (1790742224) PASS 205a (12s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 00:23:57 (1790742237) PASS 205b (11s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 00:24:07 (1790742247) oleg419-server: Pool lustre.test_206 created Adding targets to pool oleg419-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg419-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg419-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg419-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg419-server: Pool lustre.test_206 destroyed PASS 206 (32s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 00:24:39 (1790742279) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780251 s, 13.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0720069 s, 14.6 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xe37fbe91 Mirror 3: 0xe37fbe91 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 00:24:51 (1790742291) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 00:24:56 (1790742296) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 00:25:01 (1790742301) Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.101603 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1ea0000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x2230000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x2230000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x2230000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x6b0000 Read this (4 bytes): BBBB 192.168.204.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg419-client.virtnet /mnt/lustre2 (opts:) PASS 209a (45s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 00:25:45 (1790742345) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.071806 s, 114 kB/s the very first read read with warm cache PASS 209b (10s) == sanity-flr test 210a: handle broken mirrored lovea ==== 00:25:55 (1790742355) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0526675 s, 19.9 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd0 lmm_fid: [0x200000405:0xd0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd0 lmm_fid: [0x200000405:0xd0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x13a:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611344 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-30 00:25:57.000000000 -0400 Modify: 2026-09-30 00:25:57.000000000 -0400 Change: 2026-09-30 00:25:59.000000000 -0400 Birth: 2026-09-30 00:25:58.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 00:26:05 (1790742365) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243947 s, 43.0 MB/s oleg419-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg419-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg419-server: O/280000401/d27/15b: File not found by ext2_lookup PASS 210b (26s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 00:26:31 (1790742391) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.358143 s, 114 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0363814 s, 113 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 30 00:26 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xd5:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 30 00:26 /mnt/lustre/f211.sanity-flr PASS 211 (12s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 00:26:44 (1790742404) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 00:26:54 (1790742414) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 2039 sec =========== 00:27:04 (1790742424) === sanity-flr: start cleanup 00:27:06 (1790742426) === === sanity-flr: finish cleanup 00:27:15 (1790742435) ===