-----============= acceptance-small: sanity-flr ============----- Fri Sep 11 03:09:26 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg233-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 03:09:52 (1789110592) === oleg233-client.virtnet: executing check_config_client /mnt/lustre oleg233-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg233-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8dbf05664800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8dbf05664800.idle_timeout=debug disable quota as required oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 03:10:33 (1789110633) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7654] == sanity-flr test 0a: lfs mirror create with -N option == 03:10:36 (1789110636) PASS 0a (21s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 03:10:57 (1789110657) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 03:11:02 (1789110662) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 03:11:06 (1789110666) PASS 0d (28s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 03:11:34 (1789110694) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (5s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 03:11:39 (1789110699) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 03:11:43 (1789110703) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff8dbf05664800.stats=clear osc.lustre-OST0001-osc-ffff8dbf05664800.stats=clear PASS 0g (11s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 03:11:54 (1789110714) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (19s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 03:12:13 (1789110733) PASS 0j (14s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 03:12:27 (1789110747) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (5s) == sanity-flr test 0m: mirror extend with -N= option ===== 03:12:32 (1789110752) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (12s) == sanity-flr test 1: create components with setstripe options ========================================================== 03:12:44 (1789110764) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 03:12:55 (1789110775) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 03:13:04 (1789110784) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (5s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 03:13:09 (1789110789) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 03:13:18 (1789110798) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 03:13:31 (1789110811) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 03:13:40 (1789110820) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 1.03128 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111047 s, 9.4 MB/s PASS 21 (15s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 03:13:55 (1789110835) 6+0 records in 6+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.102082 s, 61.6 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 03:14:05 (1789110845) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 03:14:19 (1789110859) Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8dbf05664800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8dbf05664800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8dbf05664800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8dbf05664800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg233-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (86s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 03:15:45 (1789110945) 5+0 records in 5+0 records out 5242880 bytes (5.2 MB, 5.0 MiB) copied, 0.462954 s, 11.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8dbf05664800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8dbf05664800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 5242880 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (51s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 03:16:36 (1789110996) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8dbf05664800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8dbf05664800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8dbf05664800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8dbf05664800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg233-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (88s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 03:18:04 (1789111084) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 7.00145 s, 7.5 MB/s -rw-r--r-- 1 root root 52428800 Sep 11 03:18 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.366s user 0m0.000s sys 0m0.181s Stopping /mnt/lustre-ost1 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff8dbf05664800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8dbf05664800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.369s user 0m0.002s sys 0m0.174s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg233-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg233-server oleg233-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff8dbf05664800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8dbf05664800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.457s user 0m0.003s sys 0m0.197s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg233-server: oleg233-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg233-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg233-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8dbf05664800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8dbf05664800.ost_server_uuid in FULL state after 0 sec PASS 33b (102s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 03:19:46 (1789111186) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 03:19:50 (1789111190) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 03:19:54 (1789111194) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 03:19:58 (1789111198) PASS 35 (9s) == sanity-flr test 36a: write to mirrored files ========== 03:20:07 (1789111207) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.697941 s, 18.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (14s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 03:20:21 (1789111221) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0942247 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666825 s, 15.7 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 03:20:29 (1789111229) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 03:20:38 (1789111238) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000981534 s, 1.0 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0032825 s, 312 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00161685 s, 633 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00170729 s, 600 kB/s flv=10 olv=10 PASS 36d (12s) == sanity-flr test 37: mirror I/O API verification ======= 03:20:51 (1789111251) create test files with size 15759043 .. 6702fe96e9712816efc9126bc0f400e2 - 4a9a80c601bf66f2604796ea067ca053 - 8ef5d70b681ab08860da79cbcbbc33b4 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg233-client.virtnet /mnt/lustre (opts:) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 03:21:17 (1789111277) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (19s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 03:21:36 (1789111296) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 03:21:45 (1789111305) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181119 s, 5.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329367 s, 31.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xc9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xcc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (12s) == sanity-flr test 41: lfs mirror resync check =========== 03:21:57 (1789111317) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.267446 s, 7.8 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.457875 s, 9.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.309172 s, 6.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (23s) == sanity-flr test 42: lfs mirror verify ================= 03:22:20 (1789111340) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 03:22:24 (1789111344) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (5s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 03:22:29 (1789111349) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573877 s, 18.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 44a: lfs mirror split check =========== 03:22:38 (1789111358) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (4s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 03:22:42 (1789111362) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd1:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 03:22:54 (1789111374) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.548438 s, 19.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg233-client.virtnet /mnt/lustre (opts:) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (17s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 03:23:11 (1789111391) llite.lustre-ffff8dbf06df4000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.330245 s, 31.8 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg233-client.virtnet /mnt/lustre (opts:) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff8dbf06d28000.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 03:23:29 (1789111409) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (46s) == sanity-flr test 46: Verify setstripe --copy option ==== 03:24:15 (1789111455) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (51s) == sanity-flr test 47: Verify mirror obj alloc =========== 03:25:06 (1789111506) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (3s) == sanity-flr test 48: Verify snapshot mirror ============ 03:25:10 (1789111510) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.41024 s, 7.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.242108 s, 13.0 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xda44b185 (nosync,stale) Mirror 2: 0xda44b185 (nosync,stale) ** verify mirror 2 doesn't change original checksum: d69e86b0da8e6c774c2ce174a798a86c - mirror 1 checksum: 8939f8350a9139a60daf173b5e03676f - mirror 2 checksum: d69e86b0da8e6c774c2ce174a798a86c - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Sep 11 03:25:14 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Fri Sep 11 03:25:14 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: d69e86b0da8e6c774c2ce174a798a86c - mirror 1 checksum: 8939f8350a9139a60daf173b5e03676f - mirror 2 checksum: 8939f8350a9139a60daf173b5e03676f - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Sep 11 03:25:14 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Fri Sep 11 03:25:14 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe9:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 03:25:25 (1789111525) Failing mds1 on oleg233-server Stopping /mnt/lustre-mds1 (opts:) on oleg233-server 03:25:33 (1789111533) shut down facet: mds1 facet_host: oleg233-server facet_failover_host: oleg233-server Failover mds1 to oleg233-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg233-server: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-MDT0000 03:25:57 (1789111557) targets are mounted 03:25:57 (1789111557) facet_failover done oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057821 s, 18.1 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xea:0x0] } PASS 50A (49s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 03:26:15 (1789111575) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (5s) == sanity-flr test 50b: mirror rsync handles sparseness == 03:26:19 (1789111579) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 03:26:23 (1789111583) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0238581 s, 687 kB/s PASS 50c (10s) == sanity-flr test 50d: mirror rsync keep holes ========== 03:26:33 (1789111593) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 03:26:37 (1789111597) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (5s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 03:26:42 (1789111602) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (27s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 03:27:10 (1789111630) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (31s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 03:27:41 (1789111661) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 03:28:06 (1789111686) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 03:28:31 (1789111711) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 03:28:39 (1789111719) mirror create pid 53630 mirror split pid 53631 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5026: 53630 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5026: 53631 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (72s) == sanity-flr test 70a: flr mode fsx test ================ 03:29:51 (1789111791) Seed set to 1794 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0xbcf5 000001 1789111795.006454 trunc from 00000000 to 0x00bcf4 (0xbcf5 bytes) 000004 1789111795.079625 read 0x00415d thru 0x00bcf4 (0x7b98 bytes) 000005 1789111795.094705 mapread 0x006804 thru 0x00bcf4 (0x54f1 bytes) 000007 1789111795.100864 read 0x002efe thru 0x0056c6 (0x27c9 bytes) 000008 1789111795.109163 mapwrite 0x03f619 thru 0x03ffff (0x9e7 bytes) 000009 1789111795.351241 mapread 0x003372 thru 0x00aa2f (0x76be bytes) 000010 1789111795.370817 mapread 0x0004ed thru 0x008007 (0x7b1b bytes) truncating to largest ever: 0x1232b 000011 1789111795.388074 trunc from 0x040000 to 0x01232a (0x2dcd5 bytes) 000012 1789111795.462624 read 0x011eed thru 0x01232a (0x43e bytes) 000013 1789111795.463527 mapwrite 0x011f8d thru 0x01e387 (0xc3fb bytes) 000017 1789111795.934592 mapwrite 0x013eae thru 0x01ac9a (0x6ded bytes) 000018 1789111796.115395 trunc from 0x01e388 to 0x0040ca (0x1a2bd bytes) 000019 1789111796.175707 read 0x0025c9 thru 0x0029dc (0x414 bytes) truncating to largest ever: 0x3fcf3 000020 1789111796.178378 trunc from 0x0040cb to 0x03fcf2 (0x3bc28 bytes) 000023 1789111796.196862 write 0x020b19 thru 0x02c800 (0xbce8 bytes) 000024 1789111796.224316 trunc from 0x03fcf3 to 0x012d19 (0x2cfd9 bytes) 000025 1789111796.256714 mapwrite 0x01a78b thru 0x02a5af (0xfe25 bytes) 000026 1789111796.664651 mapwrite 0x03cb53 thru 0x03ffff (0x34ad bytes) 000029 1789111796.843171 mapwrite 0x035453 thru 0x037c03 (0x27b1 bytes) 000030 1789111796.952727 read 0x02f6ec thru 0x03d33d (0xdc52 bytes) 000032 1789111796.971135 mapwrite 0x0397bb thru 0x03ffff (0x6845 bytes) 000033 1789111797.092404 MIRROR_EXTEND to 2 mirrors 000034 1789111798.727817 read 0x03315c thru 0x03f530 (0xc3d5 bytes) 000035 1789111798.760839 write 0x02ae0e thru 0x034e78 (0xa06b bytes) 000036 1789111798.936480 trunc from 0x040000 to 0x009dae (0x36251 bytes) 000038 1789111799.009586 read 0x009a0c thru 0x009dae (0x3a3 bytes) 000039 1789111799.052801 mapread 0x0010a6 thru 0x008989 (0x78e4 bytes) 000040 1789111799.087719 mapwrite 0x0260f4 thru 0x026221 (0x12e bytes) 000042 1789111799.430289 write 0x03746f thru 0x039394 (0x1f26 bytes) 000043 1789111799.431742 mapread 0x00697e thru 0x00ed71 (0x83f4 bytes) 000044 1789111799.433060 write 0x00d469 thru 0x01a9fb (0xd593 bytes) 000045 1789111799.460421 read 0x00e1b3 thru 0x01064c (0x249a bytes) 000046 1789111799.461223 mapwrite 0x00797e thru 0x016f80 (0xf603 bytes) 000047 1789111799.656604 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x39395): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39395): Mirror 2: 0xae6f0c0 (stale) CRC-32 checksum value for chunk [0, 0x39395): Mirror 1: 0x649f2532 Mirror 3: 0x649f2532 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39395: skipped 000048 1789111801.719644 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39395): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39395): Mirror 2: 0xae6f0c0 (stale) CRC-32 checksum value for chunk [0, 0x39395): Mirror 1: 0x649f2532 Mirror 3: 0x649f2532 Mirror 4: 0x649f2532 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39395: skipped 000053 1789111803.666267 read 0x0313d5 thru 0x039394 (0x7fc0 bytes) 000054 1789111803.686139 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x39395): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39395): Mirror 2: 0xae6f0c0 (stale) CRC-32 checksum value for chunk [0, 0x39395): Mirror 1: 0x649f2532 Mirror 3: 0x649f2532 Mirror 4: 0x649f2532 Mirror 5: 0x649f2532 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39395: skipped 000056 1789111805.434499 trunc from 0x039395 to 0x02a5fc (0xed98 bytes) 000057 1789111805.569004 mapread 0x015023 thru 0x01fadd (0xaabb bytes) 000059 1789111805.601769 mapwrite 0x017e7c thru 0x025021 (0xd1a6 bytes) 000062 1789111805.686781 write 0x00766e thru 0x008bc8 (0x155b bytes) 000063 1789111805.702552 mapwrite 0x006782 thru 0x010c44 (0xa4c3 bytes) 000064 1789111805.763060 MIRROR_RESYNC 5 mirrors 000065 1789111806.954299 read 0x011e09 thru 0x01ec18 (0xce10 bytes) 000066 1789111806.965070 read 0x021b05 thru 0x023133 (0x162f bytes) 000067 1789111806.982548 trunc from 0x02a5fd to 0x03e2d6 (0x13cda bytes) 000068 1789111807.045670 trunc from 0x03e2d7 to 0x034c89 (0x964d bytes) 000071 1789111807.067944 trunc from 0x034c8a to 0x00b79c (0x294ed bytes) 000072 1789111807.090584 write 0x011723 thru 0x01e521 (0xcdff bytes) 000073 1789111807.093453 mapread 0x010543 thru 0x01cc8d (0xc74b bytes) 000074 1789111807.101790 mapwrite 0x02d1c7 thru 0x032804 (0x563e bytes) 000077 1789111807.192532 mapread 0x00feb7 thru 0x0187ec (0x8936 bytes) 000079 1789111807.202189 read 0x00a1f2 thru 0x019034 (0xee43 bytes) 000082 1789111807.216193 mapwrite 0x02d590 thru 0x03c944 (0xf3b5 bytes) 000083 1789111807.363733 trunc from 0x03c945 to 0x01cd2b (0x1fc19 bytes) 000085 1789111807.401202 MIRROR_RESYNC 5 mirrors 000086 1789111808.774088 mapread 0x00fb21 thru 0x01cd2b (0xd20b bytes) 000091 1789111808.819618 mapread 0x019768 thru 0x01cd2b (0x35c4 bytes) 000092 1789111808.822141 mapwrite 0x02dee9 thru 0x037aed (0x9c05 bytes) 000094 1789111809.127988 trunc from 0x037aee to 0x020ef6 (0x16bf7 bytes) 000095 1789111809.167717 mapwrite 0x03e28d thru 0x03ffff (0x1d73 bytes) 000097 1789111809.250318 mapread 0x01029c thru 0x019ea4 (0x9c09 bytes) 000098 1789111809.268730 trunc from 0x040000 to 0x02e49c (0x11b63 bytes) 000099 1789111809.289352 mapwrite 0x0140bf thru 0x01c458 (0x839a bytes) 000101 1789111809.347058 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e49d): Mirror 2: 0xacb42e20 (stale) Mirror 3: 0xacb42e20 (stale) Mirror 5: 0xacb42e20 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e49d): Mirror 2: 0xacb42e20 (stale) Mirror 3: 0xacb42e20 (stale) Mirror 5: 0xacb42e20 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e49d: skipped 000102 1789111809.855186 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2e49d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e49d): Mirror 2: 0xacb42e20 (stale) Mirror 3: 0xacb42e20 (stale) Mirror 5: 0xacb42e20 (stale) CRC-32 checksum value for chunk [0, 0x2e49d): Mirror 1: 0x7cf3484 Mirror 6: 0x7cf3484 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e49d: skipped 000103 1789111811.602142 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2e49d): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e49d): Mirror 2: 0xacb42e20 (stale) Mirror 3: 0xacb42e20 (stale) Mirror 5: 0xacb42e20 (stale) CRC-32 checksum value for chunk [0, 0x2e49d): Mirror 1: 0x7cf3484 Mirror 6: 0x7cf3484 Mirror 7: 0x7cf3484 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e49d: skipped 000104 1789111813.381728 read 0x00951f thru 0x00e296 (0x4d78 bytes) 000105 1789111813.389925 trunc from 0x02e49d to 0x02fbfe (0x1762 bytes) 000106 1789111813.465487 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2fbff): Mirror 2: 0xb065d2e (stale) Mirror 3: 0xb065d2e (stale) Mirror 5: 0xb065d2e (stale) Mirror 7: 0x493f1f6f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fbff): Mirror 2: 0xb065d2e (stale) Mirror 3: 0xb065d2e (stale) Mirror 5: 0xb065d2e (stale) Mirror 7: 0x493f1f6f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fbff: skipped 000107 1789111814.001916 mapread 0x026c26 thru 0x02b9d8 (0x4db3 bytes) 000108 1789111814.044440 mapread 0x01a37a thru 0x026c95 (0xc91c bytes) 000111 1789111814.072350 mapwrite 0x00cb50 thru 0x00d031 (0x4e2 bytes) 000115 1789111814.221410 read 0x016375 thru 0x01fe27 (0x9ab3 bytes) 000117 1789111814.225360 trunc from 0x02fbff to 0x02be70 (0x3d8e bytes) 000118 1789111814.263771 write 0x0070fa thru 0x01686a (0xf771 bytes) 000119 1789111814.276197 mapwrite 0x0188bf thru 0x01c9fc (0x413e bytes) 000120 1789111814.433536 mapwrite 0x015e21 thru 0x016014 (0x1f4 bytes) 000122 1789111814.503305 mapread 0x00e648 thru 0x01a348 (0xbd01 bytes) 000123 1789111814.504609 write 0x002b5f thru 0x005412 (0x28b4 bytes) 000125 1789111814.522578 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2be71): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2be71): Mirror 2: 0xa654d04a (stale) Mirror 3: 0xa654d04a (stale) Mirror 5: 0xa654d04a (stale) Mirror 7: 0xb84e2f41 (stale) CRC-32 checksum value for chunk [0, 0x2be71): Mirror 1: 0x802dc41b Mirror 8: 0x802dc41b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2be71: skipped 000126 1789111816.223217 write 0x00f68b thru 0x01537e (0x5cf4 bytes) 000130 1789111816.432280 MIRROR_RESYNC 6 mirrors 000131 1789111818.486384 read 0x0096ac thru 0x012781 (0x90d6 bytes) 000132 1789111818.514683 write 0x001b4a thru 0x010aa1 (0xef58 bytes) 000133 1789111818.714206 read 0x017170 thru 0x024e79 (0xdd0a bytes) 000135 1789111818.726311 mapread 0x0126a3 thru 0x01c778 (0xa0d6 bytes) 000136 1789111818.727641 write 0x017e53 thru 0x01d084 (0x5232 bytes) 000138 1789111818.731203 read 0x021784 thru 0x021a30 (0x2ad bytes) 000140 1789111818.733190 mapread 0x02b8f6 thru 0x02be70 (0x57b bytes) 000142 1789111818.739392 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2be71): Mirror 3: 0x4cc4a32f (stale) Mirror 5: 0x4cc4a32f (stale) Mirror 7: 0x4cc4a32f (stale) Mirror 8: 0x4cc4a32f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2be71): Mirror 3: 0x4cc4a32f (stale) Mirror 5: 0x4cc4a32f (stale) Mirror 7: 0x4cc4a32f (stale) Mirror 8: 0x4cc4a32f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2be71: skipped 000143 1789111819.389446 write 0x02e21a thru 0x037a4f (0x9836 bytes) 000146 1789111819.411337 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37a50): Mirror 5: 0x2b9f8538 (stale) Mirror 7: 0x2b9f8538 (stale) Mirror 8: 0x2b9f8538 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37a50): Mirror 5: 0x2b9f8538 (stale) Mirror 7: 0x2b9f8538 (stale) Mirror 8: 0x2b9f8538 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37a50: skipped 000148 1789111819.922128 write 0x03d86f thru 0x03fd24 (0x24b6 bytes) 000153 1789111819.933619 mapread 0x028598 thru 0x034638 (0xc0a1 bytes) 000154 1789111819.964722 mapread 0x013d36 thru 0x01fe69 (0xc134 bytes) 000156 1789111819.989808 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3fd25): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd25): Mirror 5: 0x25239d6e (stale) Mirror 7: 0x25239d6e (stale) Mirror 8: 0x25239d6e (stale) CRC-32 checksum value for chunk [0, 0x3fd25): Mirror 1: 0xda5e47f6 Mirror 9: 0xda5e47f6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd25: skipped 000157 1789111821.111705 write 0x03d7f6 thru 0x03ffff (0x280a bytes) 000158 1789111821.218359 read 0x018b0a thru 0x02818e (0xf685 bytes) 000160 1789111821.233790 write 0x0155d1 thru 0x021815 (0xc245 bytes) 000162 1789111821.244571 mapwrite 0x01cdfe thru 0x0292ff (0xc502 bytes) 000163 1789111821.396387 write 0x03ba37 thru 0x03ffff (0x45c9 bytes) 000164 1789111821.449171 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x97f45d1f (stale) Mirror 7: 0x97f45d1f (stale) Mirror 8: 0x97f45d1f (stale) Mirror 9: 0x2b9cf4cf (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3d712de8 Mirror 10: 0x3d712de8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000165 1789111822.747150 trunc from 0x040000 to 0x0324dc (0xdb23 bytes) 000167 1789111822.954846 mapwrite 0x026b94 thru 0x02ed7f (0x81ec bytes) 000168 1789111823.088443 read 0x022ae3 thru 0x027dc5 (0x52e3 bytes) 000169 1789111823.089351 trunc from 0x0324dd to 0x011e76 (0x20666 bytes) 000170 1789111823.135425 read 0x00ca49 thru 0x011e76 (0x542e bytes) 000173 1789111823.151495 trunc from 0x011e77 to 0x02e235 (0x1c3bf bytes) 000174 1789111823.194521 mapread 0x00a0d6 thru 0x016232 (0xc15d bytes) 000175 1789111823.212405 read 0x027f89 thru 0x02e235 (0x62ad bytes) 000176 1789111823.213427 trunc from 0x02e236 to 0x03335e (0x5129 bytes) 000177 1789111823.236847 mapwrite 0x010d8e thru 0x01a16b (0x93de bytes) 000184 1789111823.335848 write 0x0016b2 thru 0x00b933 (0xa282 bytes) 000189 1789111823.360619 mapread 0x032495 thru 0x03335e (0xeca bytes) 000190 1789111823.381581 read 0x00c353 thru 0x017baf (0xb85d bytes) 000191 1789111823.397398 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3335f): Mirror 5: 0x3dfebdfc (stale) Mirror 7: 0x3dfebdfc (stale) Mirror 8: 0x3dfebdfc (stale) Mirror 9: 0x7b9b3946 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3335f): Mirror 5: 0x3dfebdfc (stale) Mirror 7: 0x3dfebdfc (stale) Mirror 8: 0x3dfebdfc (stale) Mirror 9: 0x7b9b3946 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3335f: skipped 000193 1789111824.326303 mapread 0x0290e7 thru 0x03335e (0xa278 bytes) 000194 1789111824.374058 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x3335f): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3335f): Mirror 5: 0x3dfebdfc (stale) Mirror 7: 0x3dfebdfc (stale) Mirror 8: 0x3dfebdfc (stale) Mirror 9: 0x7b9b3946 (stale) CRC-32 checksum value for chunk [0, 0x3335f): Mirror 1: 0x42c74cf7 Mirror 10: 0x42c74cf7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3335f: skipped 000195 1789111825.955178 MIRROR_RESYNC 6 mirrors 000196 1789111827.589981 read 0x0306e7 thru 0x03335e (0x2c78 bytes) 000198 1789111827.604522 write 0x00254e thru 0x011690 (0xf143 bytes) 000201 1789111827.845443 trunc from 0x03335f to 0x02b4ab (0x7eb3 bytes) 000204 1789111827.902411 trunc from 0x02b4ac to 0x028ce1 (0x27ca bytes) 000205 1789111827.954675 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x28ce2): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28ce2): Mirror 5: 0xbbc6e606 (stale) Mirror 7: 0xbbc6e606 (stale) Mirror 8: 0xbbc6e606 (stale) Mirror 9: 0xbbc6e606 (stale) Mirror 10: 0xbbc6e606 (stale) CRC-32 checksum value for chunk [0, 0x28ce2): Mirror 1: 0x28003fe Mirror 11: 0x28003fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28ce2: skipped 000206 1789111829.440495 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x28ce2): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28ce2): Mirror 5: 0xbbc6e606 (stale) Mirror 7: 0xbbc6e606 (stale) Mirror 8: 0xbbc6e606 (stale) Mirror 9: 0xbbc6e606 (stale) Mirror 10: 0xbbc6e606 (stale) CRC-32 checksum value for chunk [0, 0x28ce2): Mirror 1: 0x28003fe Mirror 11: 0x28003fe Mirror 12: 0x28003fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28ce2: skipped 000207 1789111831.075625 mapread 0x023c64 thru 0x028ce1 (0x507e bytes) 000210 1789111831.126309 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x28ce2): Mirror 5: 0xbbc6e606 (stale) Mirror 7: 0xbbc6e606 (stale) Mirror 8: 0xbbc6e606 (stale) Mirror 9: 0xbbc6e606 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x28ce2): Mirror 5: 0xbbc6e606 (stale) Mirror 7: 0xbbc6e606 (stale) Mirror 8: 0xbbc6e606 (stale) Mirror 9: 0xbbc6e606 (stale) CRC-32 checksum value for chunk [0, 0x28ce2): Mirror 1: 0x28003fe Mirror 11: 0x28003fe Mirror 12: 0x28003fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x28ce2: skipped 000214 1789111831.968567 trunc from 0x028ce2 to 0x00d9a0 (0x1b341 bytes) 000216 1789111832.080256 mapread 0x00b1db thru 0x00d9a0 (0x27c6 bytes) 000217 1789111832.099180 mapread 0x006b46 thru 0x00d9a0 (0x6e5b bytes) 000218 1789111832.100621 trunc from 0x00d9a1 to 0x023a76 (0x160d6 bytes) 000222 1789111832.136959 mapread 0x005e05 thru 0x010ae8 (0xace4 bytes) 000223 1789111832.147133 mapread 0x006b04 thru 0x00b672 (0x4b6f bytes) 000225 1789111832.154068 MIRROR_RESYNC 7 mirrors 000226 1789111833.419751 read 0x002553 thru 0x00a433 (0x7ee1 bytes) 000227 1789111833.428717 trunc from 0x023a77 to 0x0097bd (0x1a2b9 bytes) 000228 1789111833.595138 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x97be): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x97be): Mirror 5: 0x4d4e589b (stale) Mirror 7: 0x4d4e589b (stale) Mirror 8: 0x4d4e589b (stale) Mirror 9: 0x4d4e589b (stale) Mirror 11: 0x4d4e589b (stale) Mirror 12: 0x4d4e589b (stale) CRC-32 checksum value for chunk [0, 0x97be): Mirror 1: 0x4d4e589b Mirror 13: 0x4d4e589b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x97be: skipped 000231 1789111834.984197 MIRROR_RESYNC 8 mirrors 000234 1789111835.999207 mapwrite 0x03d030 thru 0x03ffff (0x2fd0 bytes) 000236 1789111836.172824 write 0x02c037 thru 0x035c5f (0x9c29 bytes) 000237 1789111836.189162 mapread 0x02e810 thru 0x032382 (0x3b73 bytes) 000244 1789111836.195487 mapread 0x03fc5f thru 0x03ffff (0x3a1 bytes) 000245 1789111836.197366 mapread 0x00ac6c thru 0x00becf (0x1264 bytes) 000250 1789111836.210754 trunc from 0x040000 to 0x01036b (0x2fc94 bytes) 000251 1789111836.266573 MIRROR_RESYNC 8 mirrors 000252 1789111837.719085 read 0x005afa thru 0x00752c (0x1a33 bytes) 000255 1789111837.726405 mapread 0x00e49b thru 0x01036b (0x1ed1 bytes) 000256 1789111837.748262 MIRROR_EXTEND to 9 mirrors 000257 1789111838.520999 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) 000258 1789111839.102169 write 0x01b2a7 thru 0x01d159 (0x1eb3 bytes) 000260 1789111839.211355 mapwrite 0x032c2c thru 0x03c099 (0x946e bytes) 000261 1789111839.367921 write 0x02405e thru 0x03202d (0xdfd0 bytes) 000263 1789111839.394445 read 0x023f6a thru 0x032b6a (0xec01 bytes) 000265 1789111839.403509 mapread 0x02cc1d thru 0x03be74 (0xf258 bytes) 000267 1789111839.410389 read 0x01eb84 thru 0x02dc45 (0xf0c2 bytes) 000271 1789111839.426202 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3c09a): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c09a): Mirror 5: 0xc91c6329 (stale) Mirror 7: 0xc91c6329 (stale) Mirror 8: 0xc91c6329 (stale) Mirror 9: 0xc91c6329 (stale) Mirror 12: 0xc91c6329 (stale) Mirror 13: 0xc91c6329 (stale) Mirror 14: 0xc91c6329 (stale) CRC-32 checksum value for chunk [0, 0x3c09a): Mirror 1: 0xf6b4a4d4 Mirror 15: 0xf6b4a4d4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c09a: skipped 000272 1789111840.435282 read 0x03259b thru 0x03c099 (0x9aff bytes) 000273 1789111840.453057 trunc from 0x03c09a to 0x009d03 (0x32396 bytes) 000274 1789111840.592710 read 0x007c0b thru 0x009d03 (0x20f9 bytes) 000276 1789111840.612374 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x9d04): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9d04): Mirror 5: 0xb0a20c11 (stale) Mirror 7: 0xb0a20c11 (stale) Mirror 8: 0xb0a20c11 (stale) Mirror 9: 0xb0a20c11 (stale) Mirror 12: 0xb0a20c11 (stale) Mirror 13: 0xb0a20c11 (stale) Mirror 14: 0xb0a20c11 (stale) Mirror 15: 0xb0a20c11 (stale) CRC-32 checksum value for chunk [0, 0x9d04): Mirror 1: 0xb0a20c11 Mirror 16: 0xb0a20c11 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9d04: skipped 000277 1789111842.091182 mapread 0x0012f1 thru 0x009d03 (0x8a13 bytes) 000279 1789111842.237350 trunc from 0x009d04 to 0x0277e8 (0x1dae5 bytes) 000280 1789111842.479425 MIRROR_RESYNC 10 mirrors 000281 1789111844.540048 write 0x03f9b3 thru 0x03ffff (0x64d bytes) 000282 1789111844.604329 read 0x01f64b thru 0x02e161 (0xeb17 bytes) 000283 1789111844.615307 mapwrite 0x01d9f2 thru 0x0242e4 (0x68f3 bytes) 000284 1789111844.679062 write 0x034599 thru 0x035fc4 (0x1a2c bytes) 000285 1789111844.689072 write 0x03437c thru 0x035bbb (0x1840 bytes) 000286 1789111844.690370 trunc from 0x040000 to 0x00e86d (0x31792 bytes) 000289 1789111844.717982 trunc from 0x00e86e to 0x018e26 (0xa5b9 bytes) 000290 1789111844.739544 trunc from 0x018e27 to 0x01596a (0x34bc bytes) 000291 1789111844.760715 trunc from 0x01596b to 0x01ea63 (0x90f9 bytes) 000295 1789111844.783638 mapread 0x005604 thru 0x00fc05 (0xa602 bytes) 000298 1789111844.800056 MIRROR_RESYNC 10 mirrors 000299 1789111846.963474 MIRROR_SPLIT mirror 6 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) 000301 1789111847.547624 write 0x0337c4 thru 0x0370a7 (0x38e4 bytes) 000303 1789111847.674588 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x370a8): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x370a8): Mirror 5: 0xac6c75c3 (stale) Mirror 7: 0xac6c75c3 (stale) Mirror 8: 0xac6c75c3 (stale) Mirror 9: 0xac6c75c3 (stale) Mirror 12: 0xac6c75c3 (stale) Mirror 14: 0xac6c75c3 (stale) Mirror 15: 0xac6c75c3 (stale) Mirror 16: 0xac6c75c3 (stale) CRC-32 checksum value for chunk [0, 0x370a8): Mirror 1: 0x615546ea Mirror 17: 0x615546ea lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x370a8: skipped 000306 1789111848.997605 trunc from 0x0370a8 to 0x02c01a (0xb08d bytes) 000307 1789111849.075410 trunc from 0x02c01b to 0x02aa07 (0x1613 bytes) 000311 1789111849.117510 read 0x01c298 thru 0x02a8a1 (0xe60a bytes) 000312 1789111849.134817 mapread 0x017b02 thru 0x021630 (0x9b2f bytes) 000313 1789111849.163851 mapread 0x00319b thru 0x00f85b (0xc6c1 bytes) 000314 1789111849.175657 write 0x00b0d2 thru 0x0152c4 (0xa1f3 bytes) 000317 1789111849.184609 write 0x03fd1c thru 0x03ffff (0x2e4 bytes) 000319 1789111849.193210 MIRROR_SPLIT mirror 8 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7ce1d577 (stale) Mirror 7: 0x7ce1d577 (stale) Mirror 8: 0x7ce1d577 (stale) Mirror 9: 0x7ce1d577 (stale) Mirror 12: 0x7ce1d577 (stale) Mirror 14: 0x7ce1d577 (stale) Mirror 15: 0x7ce1d577 (stale) Mirror 17: 0xa8c33ef2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x7ce1d577 (stale) Mirror 7: 0x7ce1d577 (stale) Mirror 8: 0x7ce1d577 (stale) Mirror 9: 0x7ce1d577 (stale) Mirror 12: 0x7ce1d577 (stale) Mirror 14: 0x7ce1d577 (stale) Mirror 15: 0x7ce1d577 (stale) Mirror 17: 0xa8c33ef2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000321 1789111849.733639 write 0x0004c1 thru 0x0094fa (0x903a bytes) 000322 1789111849.759511 mapwrite 0x0256fa thru 0x027080 (0x1987 bytes) 000323 1789111849.836803 mapread 0x031f3b thru 0x0395ce (0x7694 bytes) 000325 1789111849.849031 mapwrite 0x0343ff thru 0x03ffff (0xbc01 bytes) 000327 1789111849.916212 write 0x0357d2 thru 0x036bc6 (0x13f5 bytes) 000328 1789111849.918326 trunc from 0x040000 to 0x025175 (0x1ae8a bytes) 000329 1789111849.936126 write 0x038188 thru 0x03a133 (0x1fac bytes) 000331 1789111849.944060 MIRROR_SPLIT mirror 3 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a134): Mirror 5: 0x1facc725 (stale) Mirror 7: 0x1facc725 (stale) Mirror 9: 0x1facc725 (stale) Mirror 12: 0x1facc725 (stale) Mirror 14: 0x1facc725 (stale) Mirror 15: 0x1facc725 (stale) Mirror 17: 0xa3f374ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a134): Mirror 5: 0x1facc725 (stale) Mirror 7: 0x1facc725 (stale) Mirror 9: 0x1facc725 (stale) Mirror 12: 0x1facc725 (stale) Mirror 14: 0x1facc725 (stale) Mirror 15: 0x1facc725 (stale) Mirror 17: 0xa3f374ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a134: skipped 000333 1789111850.315681 read 0x005639 thru 0x00c6c0 (0x7088 bytes) 000334 1789111850.334919 trunc from 0x03a134 to 0x035673 (0x4ac0 bytes) 000335 1789111850.363377 trunc from 0x035674 to 0x030914 (0x4d5f bytes) 000336 1789111850.397335 write 0x029c8d thru 0x02d160 (0x34d4 bytes) 000337 1789111850.423962 MIRROR_RESYNC 8 mirrors 000341 1789111851.674293 mapread 0x009646 thru 0x015117 (0xbad2 bytes) 000342 1789111851.704289 mapwrite 0x02648d thru 0x027b4a (0x16be bytes) 000348 1789111851.958613 write 0x01ecd6 thru 0x01f1c2 (0x4ed bytes) 000349 1789111851.978804 mapwrite 0x016cac thru 0x017948 (0xc9d bytes) 000351 1789111852.124253 mapread 0x01d995 thru 0x024b15 (0x7181 bytes) 000353 1789111852.146282 trunc from 0x030915 to 0x039bc6 (0x92b2 bytes) 000354 1789111852.197515 mapread 0x0050e3 thru 0x005de8 (0xd06 bytes) 000355 1789111852.231468 trunc from 0x039bc7 to 0x02b900 (0xe2c6 bytes) 000357 1789111852.270435 read 0x00b1b8 thru 0x00bb50 (0x999 bytes) 000358 1789111852.271222 trunc from 0x02b901 to 0x03900f (0xd70f bytes) 000359 1789111852.306222 MIRROR_SPLIT mirror 5 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39010): Mirror 5: 0xe278c21c (stale) Mirror 7: 0xe278c21c (stale) Mirror 9: 0xe278c21c (stale) Mirror 12: 0xe278c21c (stale) Mirror 15: 0xe278c21c (stale) Mirror 17: 0xe278c21c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39010): Mirror 5: 0xe278c21c (stale) Mirror 7: 0xe278c21c (stale) Mirror 9: 0xe278c21c (stale) Mirror 12: 0xe278c21c (stale) Mirror 15: 0xe278c21c (stale) Mirror 17: 0xe278c21c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39010: skipped 000361 1789111852.720166 mapwrite 0x0285a6 thru 0x030dec (0x8847 bytes) 000363 1789111852.776676 mapwrite 0x03ff32 thru 0x03ffff (0xce bytes) 000365 1789111852.837357 read 0x022b15 thru 0x0243c6 (0x18b2 bytes) 000366 1789111852.848663 MIRROR_RESYNC 7 mirrors 000368 1789111853.925753 mapread 0x01a532 thru 0x026100 (0xbbcf bytes) 000369 1789111853.955155 mapread 0x011113 thru 0x01f9ec (0xe8da bytes) 000373 1789111853.967214 trunc from 0x040000 to 0x02c26a (0x13d95 bytes) 000374 1789111854.064485 mapwrite 0x025bc1 thru 0x030599 (0xa9d9 bytes) 000375 1789111854.198636 write 0x0283e2 thru 0x0322c5 (0x9ee4 bytes) 000379 1789111854.203623 read 0x002fb5 thru 0x003022 (0x6e bytes) 000381 1789111854.214283 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x322c6): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 12: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x322c6): Mirror 5: 0xebd968c2 (stale) Mirror 7: 0xebd968c2 (stale) Mirror 9: 0xebd968c2 (stale) Mirror 12: 0xebd968c2 (stale) Mirror 15: 0xebd968c2 (stale) Mirror 17: 0xebd968c2 (stale) CRC-32 checksum value for chunk [0, 0x322c6): Mirror 1: 0x69d46f97 Mirror 18: 0x69d46f97 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x322c6: skipped 000382 1789111854.999404 mapread 0x00fc18 thru 0x01eddc (0xf1c5 bytes) 000383 1789111855.062125 mapwrite 0x013e16 thru 0x0192f0 (0x54db bytes) 000384 1789111855.391116 trunc from 0x0322c6 to 0x010b6f (0x21756 bytes) 000385 1789111855.490212 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10b70): Mirror 5: 0x51adb482 (stale) Mirror 7: 0x51adb482 (stale) Mirror 9: 0x51adb482 (stale) Mirror 15: 0x51adb482 (stale) Mirror 17: 0x51adb482 (stale) Mirror 18: 0x51adb482 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10b70): Mirror 5: 0x51adb482 (stale) Mirror 7: 0x51adb482 (stale) Mirror 9: 0x51adb482 (stale) Mirror 15: 0x51adb482 (stale) Mirror 17: 0x51adb482 (stale) Mirror 18: 0x51adb482 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10b70: skipped 000386 1789111856.057132 mapwrite 0x01ab2f thru 0x01b361 (0x833 bytes) 000388 1789111856.137201 MIRROR_RESYNC 7 mirrors 000389 1789111857.329295 mapread 0x003b1a thru 0x009684 (0x5b6b bytes) 000390 1789111857.360316 read 0x0035fb thru 0x012dfc (0xf802 bytes) 000391 1789111857.369464 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) 000392 1789111857.861150 trunc from 0x01b362 to 0x010ff3 (0xa36e bytes) 000393 1789111857.979795 trunc from 0x010ff4 to 0x0143d5 (0x33e2 bytes) 000397 1789111858.001621 mapread 0x011509 thru 0x014312 (0x2e0a bytes) 000398 1789111858.022580 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x143d6): Mirror 7: 0x4488c000 (stale) Mirror 9: 0x4488c000 (stale) Mirror 15: 0x4488c000 (stale) Mirror 17: 0x4488c000 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x143d6): Mirror 7: 0x4488c000 (stale) Mirror 9: 0x4488c000 (stale) Mirror 15: 0x4488c000 (stale) Mirror 17: 0x4488c000 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x143d6: skipped 000399 1789111858.376481 write 0x007f09 thru 0x012cde (0xadd6 bytes) 000403 1789111858.390758 write 0x03cb5f thru 0x03ffff (0x34a1 bytes) 000404 1789111858.393762 trunc from 0x040000 to 0x0135e5 (0x2ca1a bytes) 000407 1789111858.410650 mapread 0x00e727 thru 0x0104a1 (0x1d7b bytes) 000408 1789111858.416729 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x135e6): Mirror 7: (stale - read error) Mirror 9: (stale - read error) Mirror 15: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x135e6): Mirror 7: 0x3add953f (stale) Mirror 9: 0x3add953f (stale) Mirror 15: 0x3add953f (stale) Mirror 17: 0x3add953f (stale) CRC-32 checksum value for chunk [0, 0x135e6): Mirror 1: 0x596d0362 Mirror 18: 0x596d0362 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x135e6: skipped 000410 1789111859.767542 MIRROR_RESYNC 6 mirrors 000411 1789111860.648047 write 0x027bef thru 0x0302aa (0x86bc bytes) 000412 1789111860.761479 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x302ab): Mirror 7: 0xbad80b0d (stale) Mirror 9: 0xbad80b0d (stale) Mirror 15: 0xbad80b0d (stale) Mirror 17: 0xbad80b0d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x302ab): Mirror 7: 0xbad80b0d (stale) Mirror 9: 0xbad80b0d (stale) Mirror 15: 0xbad80b0d (stale) Mirror 17: 0xbad80b0d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x302ab: skipped 000413 1789111861.191423 mapwrite 0x03422d thru 0x03eea2 (0xac76 bytes) 000414 1789111861.291348 trunc from 0x03eea3 to 0x033a30 (0xb472 bytes) 000416 1789111861.324786 mapread 0x013dd8 thru 0x01786b (0x3a94 bytes) 000418 1789111861.338259 mapwrite 0x03c8df thru 0x03ffff (0x3721 bytes) 000419 1789111861.418308 read 0x034ac7 thru 0x03ffff (0xb539 bytes) 000421 1789111861.426196 read 0x0165b6 thru 0x025743 (0xf18e bytes) 000422 1789111861.435452 trunc from 0x040000 to 0x03c1ad (0x3e52 bytes) 000423 1789111861.448157 write 0x010a38 thru 0x013fc3 (0x358c bytes) 000426 1789111861.450090 trunc from 0x03c1ae to 0x00d96a (0x2e843 bytes) 000429 1789111861.462815 read 0x0084e1 thru 0x00d96a (0x548a bytes) 000431 1789111861.464120 mapread 0x00796f thru 0x00a47c (0x2b0e bytes) 000433 1789111861.465431 trunc from 0x00d96b to 0x00a1b5 (0x37b5 bytes) 000434 1789111861.485739 read 0x000015 thru 0x00a1b5 (0xa1a1 bytes) 000435 1789111861.486502 mapread 0x009ef5 thru 0x00a1b5 (0x2c1 bytes) 000438 1789111861.490535 write 0x01fea2 thru 0x022cae (0x2e0d bytes) 000440 1789111861.493946 mapread 0x00f8af thru 0x012ec2 (0x3614 bytes) 000442 1789111861.509001 trunc from 0x022caf to 0x0319d5 (0xed27 bytes) 000443 1789111861.521283 write 0x01a38f thru 0x024838 (0xa4aa bytes) 000444 1789111861.531424 write 0x011864 thru 0x017c38 (0x63d5 bytes) 000445 1789111861.532568 read 0x01404f thru 0x01d850 (0x9802 bytes) 000446 1789111861.533213 mapread 0x027f32 thru 0x02cad2 (0x4ba1 bytes) 000448 1789111861.535159 read 0x006e0d thru 0x00b7d0 (0x49c4 bytes) 000451 1789111861.544630 MIRROR_RESYNC 5 mirrors 000452 1789111862.539046 mapwrite 0x02e006 thru 0x031af0 (0x3aeb bytes) 000453 1789111862.739786 mapwrite 0x03ec1e thru 0x03ffff (0x13e2 bytes) 000455 1789111862.823216 read 0x0308b2 thru 0x03839d (0x7aec bytes) 000456 1789111862.831576 read 0x0311fc thru 0x03f217 (0xe01c bytes) 000457 1789111862.832305 write 0x00124d thru 0x0019ab (0x75f bytes) 000458 1789111862.843545 read 0x032e1d thru 0x03ffff (0xd1e3 bytes) 000460 1789111862.849518 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x8928e7b6 (stale) Mirror 15: 0x8928e7b6 (stale) Mirror 17: 0x8928e7b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x8928e7b6 (stale) Mirror 15: 0x8928e7b6 (stale) Mirror 17: 0x8928e7b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000461 1789111863.281442 write 0x02f2d7 thru 0x03ee70 (0xfb9a bytes) 000464 1789111863.299036 mapread 0x0086d2 thru 0x016f58 (0xe887 bytes) 000465 1789111863.335583 read 0x01cc5c thru 0x027abf (0xae64 bytes) 000466 1789111863.347045 mapread 0x01f26d thru 0x02414c (0x4ee0 bytes) 000467 1789111863.348537 trunc from 0x040000 to 0x02ecaf (0x11350 bytes) 000468 1789111863.372404 write 0x03ff22 thru 0x03ffff (0xde bytes) 000469 1789111863.373481 write 0x01871e thru 0x0197bc (0x109f bytes) 000470 1789111863.376036 read 0x0006fa thru 0x005595 (0x4e9c bytes) 000473 1789111863.387154 trunc from 0x040000 to 0x01b520 (0x24adf bytes) 000474 1789111863.401563 mapwrite 0x03a4c2 thru 0x03b864 (0x13a3 bytes) 000475 1789111863.574787 mapread 0x01d135 thru 0x01fd87 (0x2c53 bytes) 000476 1789111863.611532 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b865): Mirror 9: 0xd16eeadd (stale) Mirror 15: 0xd16eeadd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b865): Mirror 9: 0xd16eeadd (stale) Mirror 15: 0xd16eeadd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b865: skipped 000477 1789111863.988378 mapread 0x011155 thru 0x01dfa0 (0xce4c bytes) 000478 1789111864.023685 MIRROR_RESYNC 3 mirrors 000479 1789111864.706325 read 0x01fe5b thru 0x02afd2 (0xb178 bytes) 000480 1789111864.720102 write 0x016e90 thru 0x019a66 (0x2bd7 bytes) 000482 1789111864.834716 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b865): Mirror 15: 0xf86cbfdf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b865): Mirror 15: 0xf86cbfdf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b865: skipped 000484 1789111865.315505 write 0x02f3f1 thru 0x03ef94 (0xfba4 bytes) 000485 1789111865.324270 write 0x00ee3a thru 0x019ef9 (0xb0c0 bytes) 000486 1789111865.345835 mapwrite 0x028586 thru 0x0379fb (0xf476 bytes) 000491 1789111865.512446 write 0x03b01c thru 0x03ffff (0x4fe4 bytes) 000492 1789111865.517722 mapread 0x00d7b4 thru 0x01474d (0x6f9a bytes) 000495 1789111865.543693 write 0x0190ef thru 0x01c8ef (0x3801 bytes) 000497 1789111865.555373 trunc from 0x040000 to 0x01a324 (0x25cdb bytes) 000498 1789111865.577452 write 0x034483 thru 0x03ffff (0xbb7d bytes) 000499 1789111865.578925 read 0x00dc60 thru 0x0118e2 (0x3c83 bytes) 000500 1789111865.592034 trunc from 0x040000 to 0x027f2a (0x180d5 bytes) 000501 1789111865.610778 read 0x01393f thru 0x020b1d (0xd1df bytes) 000503 1789111865.630560 trunc from 0x027f2b to 0x00850a (0x1fa20 bytes) 000504 1789111865.677320 mapread 0x004caf thru 0x00850a (0x385c bytes) 000505 1789111865.684294 MIRROR_RESYNC 2 mirrors 000506 1789111866.306282 mapread 0x0066d8 thru 0x0071ed (0xb16 bytes) 000507 1789111866.318635 trunc from 0x00850b to 0x003ad6 (0x4a34 bytes) 000511 1789111866.407699 mapread 0x0033e2 thru 0x003ad6 (0x6f5 bytes) 000512 1789111866.419871 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3ad7): Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ad7): Mirror 15: 0x9c870b86 (stale) CRC-32 checksum value for chunk [0, 0x3ad7): Mirror 1: 0x9c870b86 Mirror 16: 0x9c870b86 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ad7: skipped 000513 1789111867.077386 mapread 0x00134c thru 0x003ad6 (0x278b bytes) 000514 1789111867.110539 mapread 0x0007a0 thru 0x003ad6 (0x3337 bytes) 000515 1789111867.140782 trunc from 0x003ad7 to 0x017188 (0x136b2 bytes) 000516 1789111867.343489 trunc from 0x017189 to 0x02f40c (0x18284 bytes) 000520 1789111867.375088 write 0x02bade thru 0x0330ef (0x7612 bytes) 000521 1789111867.385415 trunc from 0x0330f0 to 0x000e29 (0x322c6 bytes) 000522 1789111867.406715 write 0x0313fc thru 0x033bb3 (0x27b8 bytes) 000523 1789111867.407926 mapread 0x015fd2 thru 0x020d13 (0xad42 bytes) 000524 1789111867.436800 write 0x0288f6 thru 0x031ffa (0x9705 bytes) 000526 1789111867.445974 mapwrite 0x023053 thru 0x0276c5 (0x4673 bytes) 000528 1789111867.530661 trunc from 0x033bb4 to 0x02c949 (0x726a bytes) 000529 1789111867.553425 write 0x03ef72 thru 0x03ffff (0x108e bytes) 000532 1789111867.558430 read 0x03d9fb thru 0x03e700 (0xd06 bytes) 000533 1789111867.566064 mapread 0x03bcd3 thru 0x03ffff (0x432d bytes) 000536 1789111867.589535 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 15: 0x4c03bf00 (stale) Mirror 16: 0xfaca0ac6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe39292bd Mirror 17: 0xe39292bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000537 1789111868.457013 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xfaca0ac6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xfaca0ac6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe39292bd Mirror 17: 0xe39292bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000538 1789111869.080384 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0xfaca0ac6 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe39292bd Mirror 17: 0xe39292bd Mirror 18: 0xe39292bd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000540 1789111870.129756 write 0x02a97b thru 0x035a1a (0xb0a0 bytes) 000541 1789111870.218617 read 0x01ded7 thru 0x0257ce (0x78f8 bytes) 000542 1789111870.238553 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xe39292bd (stale) Mirror 18: 0xe39292bd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 17: 0xe39292bd (stale) Mirror 18: 0xe39292bd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000543 1789111870.811526 trunc from 0x040000 to 0x01aaff (0x25500 bytes) 000544 1789111870.863159 read 0x01a765 thru 0x01aaff (0x39b bytes) 000545 1789111870.869664 trunc from 0x01ab00 to 0x018b3e (0x1fc1 bytes) 000548 1789111870.895493 read 0x001392 thru 0x00e6dd (0xd34c bytes) 000554 1789111870.914674 read 0x00f469 thru 0x01407d (0x4c15 bytes) 000555 1789111870.915591 mapwrite 0x009b4c thru 0x01548c (0xb941 bytes) 000556 1789111870.979283 trunc from 0x018b3f to 0x017836 (0x1308 bytes) 000558 1789111871.002086 write 0x03b037 thru 0x03ffff (0x4fc9 bytes) 000559 1789111871.006351 read 0x012ba5 thru 0x01fb92 (0xcfee bytes) 000560 1789111871.016682 write 0x019957 thru 0x01b66a (0x1d14 bytes) 000562 1789111871.018283 write 0x0288cf thru 0x02dc04 (0x5336 bytes) 000564 1789111871.020942 read 0x01335a thru 0x021cf5 (0xe99c bytes) 000566 1789111871.022977 read 0x01ed9f thru 0x01f01a (0x27c bytes) 000567 1789111871.024637 mapread 0x0128b3 thru 0x017d7b (0x54c9 bytes) 000571 1789111871.028720 mapread 0x0027cc thru 0x00fb4a (0xd37f bytes) 000572 1789111871.037181 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xe39292bd (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 18: 0xe39292bd (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000574 1789111871.349323 mapwrite 0x001619 thru 0x00872b (0x7113 bytes) 000575 1789111871.444311 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) 000583 1789111871.609687 mapwrite 0x03fbcb thru 0x03ffff (0x435 bytes) 000585 1789111871.666444 mapread 0x007081 thru 0x009199 (0x2119 bytes) 000586 1789111871.680542 read 0x036885 thru 0x03ffff (0x977b bytes) 000588 1789111871.692610 mapread 0x03e00f thru 0x03ffff (0x1ff1 bytes) 000589 1789111871.701307 MIRROR_EXTEND to 2 mirrors 000590 1789111872.904675 mapwrite 0x025f61 thru 0x02e74d (0x87ed bytes) 000591 1789111873.113418 write 0x004e29 thru 0x00f04e (0xa226 bytes) 000593 1789111873.126125 mapread 0x016e73 thru 0x018722 (0x18b0 bytes) 000595 1789111873.136921 mapread 0x02f2c1 thru 0x0335c2 (0x4302 bytes) 000600 1789111873.149824 trunc from 0x040000 to 0x02cc75 (0x1338a bytes) 000602 1789111873.173226 trunc from 0x02cc76 to 0x0270c7 (0x5bae bytes) 000603 1789111873.195575 MIRROR_RESYNC 2 mirrors 000604 1789111874.137239 read 0x01e00c thru 0x0270c7 (0x90bc bytes) 000605 1789111874.151272 write 0x02555f thru 0x02899f (0x3441 bytes) 000606 1789111874.257945 read 0x00178a thru 0x00a2e1 (0x8b58 bytes) 000608 1789111874.286995 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x289a0): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x289a0): Mirror 2: 0xe8a2723f (stale) CRC-32 checksum value for chunk [0, 0x289a0): Mirror 1: 0x64225543 Mirror 3: 0x64225543 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x289a0: skipped 000611 1789111875.175782 mapwrite 0x00a677 thru 0x00fc1f (0x55a9 bytes) 000612 1789111875.527831 read 0x02615b thru 0x02899f (0x2845 bytes) 000613 1789111875.541514 read 0x0215b7 thru 0x02899f (0x73e9 bytes) 000614 1789111875.558158 MIRROR_RESYNC 3 mirrors 000618 1789111876.681837 mapread 0x016d89 thru 0x02345d (0xc6d5 bytes) 000620 1789111876.715405 write 0x00f549 thru 0x01bcf7 (0xc7af bytes) 000622 1789111876.783735 read 0x0122f1 thru 0x01ddf3 (0xbb03 bytes) 000623 1789111876.793867 read 0x008bf6 thru 0x0133cc (0xa7d7 bytes) 000626 1789111876.803838 read 0x001636 thru 0x00cd81 (0xb74c bytes) 000629 1789111876.817809 mapwrite 0x03dd44 thru 0x03ffff (0x22bc bytes) 000630 1789111876.938945 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x73d1c318 (stale) Mirror 3: 0x73d1c318 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc16a83e8 Mirror 4: 0xc16a83e8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000631 1789111877.966863 read 0x028a3b thru 0x033cd2 (0xb298 bytes) 000632 1789111877.985328 mapwrite 0x0260eb thru 0x031939 (0xb84f bytes) 000633 1789111878.194680 write 0x01f287 thru 0x020390 (0x110a bytes) 000634 1789111878.209875 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x73d1c318 (stale) Mirror 4: 0xc16a83e8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x73d1c318 (stale) Mirror 4: 0xc16a83e8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000635 1789111878.650825 mapread 0x004435 thru 0x00db34 (0x9700 bytes) 000636 1789111878.667040 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc16a83e8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xc16a83e8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000637 1789111878.967449 mapwrite 0x0316bd thru 0x039bb0 (0x84f4 bytes) 000639 1789111879.049559 read 0x00e857 thru 0x01a5f9 (0xbda3 bytes) 000641 1789111879.062274 read 0x017c30 thru 0x018e22 (0x11f3 bytes) 000643 1789111879.063569 mapwrite 0x023c64 thru 0x032d9d (0xf13a bytes) 000644 1789111879.144382 write 0x01bd9c thru 0x025678 (0x98dd bytes) 000645 1789111879.145929 write 0x01babb thru 0x026349 (0xa88f bytes) 000646 1789111879.149436 read 0x00a535 thru 0x00c51f (0x1feb bytes) 000647 1789111879.151028 trunc from 0x040000 to 0x033606 (0xc9f9 bytes) 000649 1789111879.173417 write 0x038ff6 thru 0x03ffff (0x700a bytes) 000651 1789111879.176594 mapwrite 0x01a8ac thru 0x02a7c3 (0xff18 bytes) 000652 1789111879.239553 MIRROR_RESYNC 2 mirrors 000653 1789111879.868874 read 0x034db2 thru 0x03ffff (0xb24e bytes) 000654 1789111879.878755 mapwrite 0x0204dd thru 0x02e948 (0xe46c bytes) 000656 1789111880.093303 trunc from 0x040000 to 0x03a1df (0x5e20 bytes) 000657 1789111880.154733 write 0x00b7d6 thru 0x00f2de (0x3b09 bytes) 000659 1789111880.169712 write 0x03fbe0 thru 0x03ffff (0x420 bytes) 000660 1789111880.171742 mapwrite 0x01b92d thru 0x02860b (0xccdf bytes) 000661 1789111880.244316 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb66bc857 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x353ea3a4 Mirror 5: 0x353ea3a4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000662 1789111881.360549 trunc from 0x040000 to 0x0369b7 (0x9648 bytes) 000664 1789111881.492560 write 0x0162e0 thru 0x021fe0 (0xbd01 bytes) 000665 1789111881.501967 mapread 0x025b5f thru 0x02c343 (0x67e5 bytes) 000667 1789111881.514127 trunc from 0x0369b8 to 0x00bf04 (0x2aab3 bytes) 000668 1789111881.537524 write 0x03cbec thru 0x03ffff (0x3414 bytes) 000669 1789111881.539921 mapread 0x019e09 thru 0x02330e (0x9506 bytes) 000670 1789111881.558404 read 0x014bab thru 0x01e057 (0x94ad bytes) 000671 1789111881.559309 trunc from 0x040000 to 0x0157e6 (0x2a819 bytes) 000672 1789111881.573634 mapwrite 0x01c955 thru 0x02be3c (0xf4e8 bytes) 000675 1789111881.685257 MIRROR_RESYNC 3 mirrors 000676 1789111882.743175 write 0x028e16 thru 0x038027 (0xf212 bytes) 000677 1789111882.888783 trunc from 0x038028 to 0x002de3 (0x35244 bytes) 000678 1789111882.937281 read 0x002cc2 thru 0x002de3 (0x122 bytes) 000679 1789111882.949201 mapread 0x00178a thru 0x002de3 (0x165a bytes) 000680 1789111882.959329 write 0x02e902 thru 0x03788b (0x8f8a bytes) 000681 1789111882.963157 write 0x00f5ea thru 0x01129b (0x1cb2 bytes) 000684 1789111882.988794 write 0x00d438 thru 0x014c0e (0x77d7 bytes) 000686 1789111882.993789 mapread 0x019614 thru 0x025508 (0xbef5 bytes) 000687 1789111883.026810 mapwrite 0x0187fa thru 0x02577b (0xcf82 bytes) 000689 1789111883.298953 mapwrite 0x0248a0 thru 0x02eca3 (0xa404 bytes) 000690 1789111883.464187 mapread 0x01c2ee thru 0x026a77 (0xa78a bytes) 000692 1789111883.466915 mapread 0x00ebcb thru 0x01308b (0x44c1 bytes) 000693 1789111883.468810 write 0x03cc51 thru 0x03ffff (0x33af bytes) 000695 1789111883.471324 write 0x03402e thru 0x036164 (0x2137 bytes) 000696 1789111883.472825 mapread 0x0142a3 thru 0x016eac (0x2c0a bytes) 000697 1789111883.474317 mapread 0x0003a3 thru 0x008ad4 (0x8732 bytes) 000698 1789111883.484206 mapread 0x00fa23 thru 0x01d3ce (0xd9ac bytes) 000699 1789111883.486242 trunc from 0x040000 to 0x02c7cb (0x13834 bytes) 000702 1789111883.512119 mapread 0x01d64f thru 0x02280c (0x51be bytes) 000703 1789111883.518032 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2c7cc): Mirror 5: 0xaeba2e7b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c7cc): Mirror 5: 0xaeba2e7b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c7cc: skipped 000706 1789111883.908119 trunc from 0x02c7cc to 0x0359e0 (0x9215 bytes) 000708 1789111883.933014 read 0x00a33d thru 0x01850f (0xe1d3 bytes) 000709 1789111883.946834 trunc from 0x0359e1 to 0x03bac9 (0x60e9 bytes) 000711 1789111883.969859 mapread 0x0006be thru 0x005f24 (0x5867 bytes) 000714 1789111884.000548 read 0x02f20d thru 0x035b99 (0x698d bytes) 000715 1789111884.017058 mapwrite 0x00fb3f thru 0x01181b (0x1cdd bytes) 000717 1789111884.075305 read 0x02ae20 thru 0x02c80e (0x19ef bytes) 000719 1789111884.078528 mapread 0x0012db thru 0x003607 (0x232d bytes) 000720 1789111884.080843 mapwrite 0x01e2cc thru 0x028013 (0x9d48 bytes) 000723 1789111884.182821 trunc from 0x03baca to 0x031e5c (0x9c6d bytes) 000725 1789111884.221552 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x31e5d): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31e5d): Mirror 5: 0xa195cab9 (stale) CRC-32 checksum value for chunk [0, 0x31e5d): Mirror 1: 0xf06d612b Mirror 6: 0xf06d612b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31e5d: skipped 000726 1789111884.992815 mapread 0x00afe7 thru 0x01231e (0x7338 bytes) 000727 1789111885.039840 MIRROR_RESYNC 3 mirrors 000729 1789111885.738871 trunc from 0x031e5d to 0x01deba (0x13fa2 bytes) 000731 1789111885.886493 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1debb): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1debb): Mirror 5: 0x797f388a (stale) Mirror 6: 0x797f388a (stale) CRC-32 checksum value for chunk [0, 0x1debb): Mirror 1: 0x797f388a Mirror 7: 0x797f388a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1debb: skipped 000732 1789111886.670474 mapwrite 0x02d370 thru 0x03912a (0xbdbb bytes) 000733 1789111886.871816 write 0x02fa74 thru 0x0330be (0x364b bytes) 000734 1789111886.873842 mapwrite 0x01f8de thru 0x02d018 (0xd73b bytes) 000735 1789111886.953390 trunc from 0x03912b to 0x03f99a (0x6870 bytes) 000740 1789111886.983247 mapwrite 0x03f4f6 thru 0x03ffff (0xb0a bytes) 000741 1789111887.067733 read 0x0367db thru 0x03f4e2 (0x8d08 bytes) 000742 1789111887.068818 trunc from 0x040000 to 0x0062e6 (0x39d19 bytes) 000743 1789111887.090812 mapwrite 0x02531a thru 0x02d38e (0x8075 bytes) 000744 1789111887.183693 read 0x00311f thru 0x0058ee (0x27d0 bytes) 000746 1789111887.198721 mapread 0x018285 thru 0x01a955 (0x26d1 bytes) 000748 1789111887.200510 mapwrite 0x0218ef thru 0x02dd8c (0xc49e bytes) 000749 1789111887.285199 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2dd8d): Mirror 5: 0x4f7b9c34 (stale) Mirror 6: 0x4f7b9c34 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd8d): Mirror 5: 0x4f7b9c34 (stale) Mirror 6: 0x4f7b9c34 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd8d: skipped 000750 1789111887.685462 trunc from 0x02dd8d to 0x01c9b0 (0x113dc bytes) 000752 1789111887.710758 mapwrite 0x03f8a5 thru 0x03ffff (0x75b bytes) 000754 1789111887.854769 MIRROR_RESYNC 3 mirrors 000756 1789111888.951434 trunc from 0x040000 to 0x0125bc (0x2da43 bytes) 000759 1789111889.141806 read 0x00c483 thru 0x0125bc (0x613a bytes) 000761 1789111889.151665 trunc from 0x0125bd to 0x028397 (0x15ddb bytes) 000762 1789111889.174189 mapwrite 0x02a261 thru 0x039c90 (0xfa30 bytes) 000764 1789111889.275244 read 0x00f7c6 thru 0x01c2e6 (0xcb21 bytes) 000765 1789111889.288242 mapread 0x02b619 thru 0x03298b (0x7373 bytes) 000767 1789111889.290017 read 0x022e0c thru 0x027ff8 (0x51ed bytes) 000768 1789111889.300749 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x39c91): Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39c91): Mirror 5: 0xc795b64 (stale) Mirror 6: 0xc795b64 (stale) CRC-32 checksum value for chunk [0, 0x39c91): Mirror 1: 0xe36a1965 Mirror 7: 0xe36a1965 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39c91: skipped 000769 1789111890.208502 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x39c91): Mirror 5: 0xc795b64 (stale) Mirror 6: 0xc795b64 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x39c91): Mirror 5: 0xc795b64 (stale) Mirror 6: 0xc795b64 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x39c91: skipped 000771 1789111890.652439 mapread 0x00b45b thru 0x01899f (0xd545 bytes) 000773 1789111890.682741 mapread 0x015656 thru 0x018535 (0x2ee0 bytes) 000774 1789111890.683764 write 0x01158d thru 0x017609 (0x607d bytes) 000778 1789111890.708816 trunc from 0x039c91 to 0x02285b (0x17435 bytes) 000779 1789111890.762517 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2285c): Mirror 6: 0xcb053f33 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2285c): Mirror 6: 0xcb053f33 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2285c: skipped 000781 1789111891.082144 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) 000783 1789111891.221817 mapwrite 0x017bff thru 0x021d85 (0xa187 bytes) 000784 1789111891.333938 write 0x01174c thru 0x017665 (0x5f1a bytes) 000785 1789111891.347821 write 0x019901 thru 0x01996f (0x6f bytes) 000786 1789111891.348993 trunc from 0x02285c to 0x02e6d4 (0xbe79 bytes) 000787 1789111891.375300 mapwrite 0x0091a8 thru 0x0126aa (0x9503 bytes) 000794 1789111891.481623 mapwrite 0x023800 thru 0x02a28d (0x6a8e bytes) 000796 1789111891.552669 mapwrite 0x01235a thru 0x013a7a (0x1721 bytes) 000798 1789111891.616918 read 0x01b790 thru 0x0263fb (0xac6c bytes) 000801 1789111891.626445 trunc from 0x02e6d5 to 0x032e22 (0x474e bytes) 000802 1789111891.642864 read 0x010af6 thru 0x01920b (0x8716 bytes) 000804 1789111891.650696 MIRROR_EXTEND to 2 mirrors 000805 1789111892.494948 mapwrite 0x03f27f thru 0x03ffff (0xd81 bytes) 000807 1789111892.627447 trunc from 0x040000 to 0x010651 (0x2f9ae bytes) 000809 1789111892.652186 write 0x03e5d2 thru 0x03ffff (0x1a2e bytes) 000810 1789111892.654383 trunc from 0x040000 to 0x0127e2 (0x2d81d bytes) 000811 1789111892.671658 write 0x00fc63 thru 0x01bcd7 (0xc075 bytes) 000812 1789111892.677656 mapwrite 0x000b96 thru 0x0096d3 (0x8b3e bytes) 000814 1789111892.737487 trunc from 0x01bcd8 to 0x00ff7e (0xbd59 bytes) 000815 1789111892.755442 mapwrite 0x016479 thru 0x023157 (0xccdf bytes) 000816 1789111892.836070 write 0x008394 thru 0x009e51 (0x1abe bytes) 000817 1789111892.837271 mapread 0x012736 thru 0x01d9f3 (0xb2be bytes) 000818 1789111892.839833 trunc from 0x023158 to 0x0070ee (0x1c069 bytes) 000819 1789111892.855964 mapread 0x003a69 thru 0x00401a (0x5b2 bytes) 000821 1789111892.860757 trunc from 0x0070ef to 0x00bf57 (0x4e69 bytes) 000822 1789111892.884422 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) 000823 1789111893.003399 read 0x0066d5 thru 0x00bf57 (0x5883 bytes) 000824 1789111893.026957 trunc from 0x00bf58 to 0x0293fc (0x1d4a5 bytes) 000828 1789111893.060388 MIRROR_EXTEND to 2 mirrors 000829 1789111893.794605 trunc from 0x0293fd to 0x020763 (0x8c99 bytes) 000834 1789111893.925561 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x20764): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20764): Mirror 2: 0x2b6f8fe2 (stale) CRC-32 checksum value for chunk [0, 0x20764): Mirror 1: 0x2b6f8fe2 Mirror 3: 0x2b6f8fe2 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20764: skipped 000835 1789111895.022141 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) 000837 1789111895.622169 mapwrite 0x02db5e thru 0x03674b (0x8bee bytes) 000840 1789111895.830174 mapwrite 0x00b149 thru 0x013e1d (0x8cd5 bytes) 000842 1789111895.925194 mapwrite 0x03da42 thru 0x03ffff (0x25be bytes) 000846 1789111896.072876 read 0x01edac thru 0x02b5b1 (0xc806 bytes) 000849 1789111896.099261 trunc from 0x040000 to 0x032d3f (0xd2c0 bytes) 000850 1789111896.127872 read 0x019348 thru 0x01d184 (0x3e3d bytes) 000852 1789111896.144624 write 0x006dc1 thru 0x016802 (0xfa42 bytes) 000855 1789111896.150374 write 0x03bdcd thru 0x03ffff (0x4233 bytes) 000858 1789111896.156219 read 0x00fd04 thru 0x01c2e0 (0xc5dd bytes) 000860 1789111896.158434 trunc from 0x040000 to 0x032956 (0xd6a9 bytes) 000861 1789111896.184194 write 0x03e1ec thru 0x03ffff (0x1e14 bytes) 000862 1789111896.193543 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) 000869 1789111896.494767 read 0x028a9a thru 0x032d42 (0xa2a9 bytes) 000872 1789111896.506966 mapwrite 0x02bc72 thru 0x031d04 (0x6093 bytes) 000873 1789111896.590852 write 0x03b2a4 thru 0x03ffff (0x4d5c bytes) 000874 1789111896.597264 mapwrite 0x033d11 thru 0x03eaf5 (0xade5 bytes) 000875 1789111896.658835 mapwrite 0x00a3af thru 0x01a34b (0xff9d bytes) 000876 1789111896.801486 mapread 0x03f1bd thru 0x03f1d1 (0x15 bytes) 000881 1789111896.813807 mapwrite 0x02c858 thru 0x032100 (0x58a9 bytes) 000883 1789111896.881511 trunc from 0x040000 to 0x00f41f (0x30be0 bytes) 000885 1789111896.905654 write 0x02e813 thru 0x030bcf (0x23bd bytes) 000890 1789111896.916842 write 0x029347 thru 0x03283a (0x94f4 bytes) 000891 1789111896.934779 write 0x03b4a6 thru 0x03f133 (0x3c8e bytes) 000894 1789111896.940608 mapwrite 0x037459 thru 0x03ffff (0x8ba7 bytes) 000895 1789111897.080868 MIRROR_EXTEND to 2 mirrors 000896 1789111897.978690 read 0x03bdfa thru 0x03ffff (0x4206 bytes) 000897 1789111898.047554 read 0x03fb2a thru 0x03ffff (0x4d6 bytes) 000898 1789111898.064092 mapwrite 0x0288f2 thru 0x02ad08 (0x2417 bytes) 000900 1789111898.351548 read 0x00bd86 thru 0x01278b (0x6a06 bytes) 000903 1789111898.374521 mapwrite 0x03b7a2 thru 0x03ffff (0x485e bytes) 000905 1789111898.469623 trunc from 0x040000 to 0x01d5e5 (0x22a1a bytes) 000907 1789111898.507698 trunc from 0x01d5e6 to 0x020daf (0x37ca bytes) 000908 1789111898.553791 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x189:0x0]: Invalid argument (22) 000910 1789111898.818998 trunc from 0x020db0 to 0x029dc0 (0x9011 bytes) 000911 1789111898.859752 mapwrite 0x0347b8 thru 0x035c72 (0x14bb bytes) 000913 1789111898.957623 trunc from 0x035c73 to 0x0284d5 (0xd79d bytes) 000914 1789111898.979151 mapwrite 0x0356d6 thru 0x03f31e (0x9c49 bytes) 000915 1789111899.093862 mapwrite 0x019bcf thru 0x01dcd8 (0x410a bytes) 000921 1789111899.214305 mapwrite 0x00522b thru 0x0098aa (0x4680 bytes) 000922 1789111899.303312 trunc from 0x03f31f to 0x01bc1a (0x23704 bytes) 000924 1789111899.334872 read 0x009ca2 thru 0x019983 (0xfce2 bytes) 000925 1789111899.344833 read 0x006ec8 thru 0x0079de (0xb17 bytes) 000927 1789111899.347700 read 0x01aac1 thru 0x01bc1a (0x115a bytes) 000933 1789111899.353074 write 0x02303d thru 0x02f6e0 (0xc6a4 bytes) 000934 1789111899.356480 write 0x009872 thru 0x01443a (0xabc9 bytes) 000935 1789111899.359382 mapwrite 0x02bd4a thru 0x03aee8 (0xf19f bytes) 000936 1789111899.554692 read 0x0152ab thru 0x022652 (0xd3a8 bytes) 000940 1789111899.560830 mapread 0x012265 thru 0x018a6d (0x6809 bytes) 000941 1789111899.562912 trunc from 0x03aee9 to 0x02d8ce (0xd61a bytes) 000943 1789111899.588750 read 0x010527 thru 0x0155d3 (0x50ad bytes) 000947 1789111899.605868 mapwrite 0x03e43b thru 0x03ffff (0x1bc5 bytes) 000951 1789111899.715628 write 0x026fba thru 0x031041 (0xa088 bytes) 000952 1789111899.719222 read 0x02918c thru 0x02c578 (0x33ed bytes) 000953 1789111899.721625 mapwrite 0x00137e thru 0x011133 (0xfdb6 bytes) 000954 1789111899.957530 write 0x016e10 thru 0x025357 (0xe548 bytes) 000955 1789111899.962792 trunc from 0x040000 to 0x002dc8 (0x3d237 bytes) 000957 1789111900.003059 write 0x01f0ce thru 0x027504 (0x8437 bytes) 000962 1789111900.013788 read 0x01d1fc thru 0x027504 (0xa309 bytes) 000963 1789111900.039386 trunc from 0x027505 to 0x018cc2 (0xe842 bytes) 000964 1789111900.083410 read 0x0075d5 thru 0x016f31 (0xf95d bytes) 000967 1789111900.092654 mapread 0x01839e thru 0x018cc2 (0x925 bytes) 000968 1789111900.098282 mapread 0x015e8c thru 0x018cc2 (0x2e37 bytes) 000969 1789111900.099702 read 0x013ce3 thru 0x018cc2 (0x4fe0 bytes) 000971 1789111900.107319 trunc from 0x018cc3 to 0x01575a (0x3568 bytes) 000973 1789111900.138341 read 0x0006f7 thru 0x00467b (0x3f85 bytes) 000974 1789111900.139166 trunc from 0x01575b to 0x0334be (0x1dd64 bytes) 000975 1789111900.161565 mapread 0x011e06 thru 0x0216aa (0xf8a5 bytes) 000977 1789111900.175106 read 0x01666f thru 0x020ca7 (0xa639 bytes) 000978 1789111900.176610 mapwrite 0x0308eb thru 0x03c5cf (0xbce5 bytes) 000981 1789111900.260102 write 0x011e3d thru 0x01d352 (0xb516 bytes) 000986 1789111900.268382 read 0x009a88 thru 0x012d37 (0x92b0 bytes) 000987 1789111900.269468 mapwrite 0x031f85 thru 0x03a56b (0x85e7 bytes) 000989 1789111900.374117 read 0x029640 thru 0x038d6f (0xf730 bytes) 000991 1789111900.377204 write 0x033126 thru 0x03fa34 (0xc90f bytes) 000992 1789111900.386223 mapwrite 0x02ad63 thru 0x0362e9 (0xb587 bytes) 000993 1789111900.523333 mapread 0x031147 thru 0x033123 (0x1fdd bytes) 000994 1789111900.525622 write 0x005738 thru 0x00678b (0x1054 bytes) 000995 1789111900.527255 mapread 0x006fd1 thru 0x00b3f5 (0x4425 bytes) 000997 1789111900.530870 mapread 0x00eef1 thru 0x01be49 (0xcf59 bytes) 000998 1789111900.533349 mapread 0x004837 thru 0x006129 (0x18f3 bytes) 000999 1789111900.538096 write 0x0000ed thru 0x000127 (0x3b bytes) All operations completed A-OK! PASS 70a (115s) == sanity-flr test 71: check mirror extend parent fid ==== 03:31:46 (1789111906) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 03:31:50 (1789111910) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.21158 s, 9.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134775 s, 7.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ad:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ae:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 03:31:58 (1789111918) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 5144256 .. Extending file size to 5410976 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6999968 .. failed Extending file size to 8602240 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7495104 .. Extending file size to 8671232 .. Extending file size to 7796352 .. Extending file size to 8912512 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7866048 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8652288 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 56716 56718 56719 56720 56721 failed 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) 192.168.202.133@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg233-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (88s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 03:33:26 (1789112006) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 3768960 .. Extending file size to 5608384 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 6749632 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7248352 .. Extending file size to 8077344 .. Extending file size to 7518496 .. Extending file size to 8012192 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8537856 .. Extending file size to 8337760 .. Extending file size to 8937344 .. Extending file size to 8827008 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Waiting 57938 57939 57940 57941 57944 done 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) 192.168.202.133@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg233-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (85s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 03:34:52 (1789112092) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2816 2205696 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.194835 s, 10.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.408397 s, 5.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xfe7e2c55 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) PASS 200c (22s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 03:35:15 (1789112115) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.113594 s, 18.5 MB/s PASS 202 (6s) == sanity-flr test 203: mirror file preserve mirror ID === 03:35:21 (1789112121) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1dc:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c7:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c8:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2c3:0x0] } PASS 203 (7s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 03:35:28 (1789112128) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.29818 s, 7.0 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2c9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c4:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c5:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 03:35:36 (1789112136) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.216122 s, 9.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ca:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cb:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 03:35:46 (1789112146) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.317226 s, 6.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 03:36:23 (1789112183) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg233-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166591 s, 6.3 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cd:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ce:0x0] } PASS 204d (64s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 03:37:28 (1789112248) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg233-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.244358 s, 8.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2cf:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (65s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 03:38:33 (1789112313) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg233-server mds-ost sync done. Creating new pool test_204f oleg233-server: Pool lustre.test_204f created Adding targets to pool oleg233-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg233-server: OST lustre-OST0001_UUID added to pool lustre.test_204f Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.358133 s, 5.9 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2d0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg233-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg233-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg233-server: Pool lustre.test_204f destroyed PASS 204f (86s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 03:39:59 (1789112399) PASS 205a (10s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 03:40:09 (1789112409) PASS 205b (10s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 03:40:19 (1789112419) oleg233-server: Pool lustre.test_206 created Adding targets to pool oleg233-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg233-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg233-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg233-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg233-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 03:40:47 (1789112447) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110474 s, 9.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0735712 s, 14.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x66db114c Mirror 3: 0x66db114c PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 03:41:00 (1789112460) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 03:41:05 (1789112465) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (5s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 03:41:09 (1789112469) Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.123668 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1790000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1990000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1990000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1990000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x24a0000 Read this (4 bytes): BBBB 192.168.202.133@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg233-client.virtnet /mnt/lustre2 (opts:) PASS 209a (46s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 03:41:55 (1789112515) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0262836 s, 312 kB/s the very first read read with warm cache PASS 209b (9s) == sanity-flr test 210a: handle broken mirrored lovea ==== 03:42:04 (1789112524) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0445174 s, 23.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x201 lmm_fid: [0x200000404:0x201:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x201 lmm_fid: [0x200000404:0x201:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2da:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834433 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-11 03:42:06.000000000 -0400 Modify: 2026-09-11 03:42:06.000000000 -0400 Change: 2026-09-11 03:42:08.000000000 -0400 Birth: 2026-09-11 03:42:06.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 03:42:14 (1789112534) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 03:42:17 (1789112537) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.892935 s, 45.9 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0654247 s, 62.6 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 11 03:42 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x203:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 11 03:42 /mnt/lustre/f211.sanity-flr PASS 211 (10s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 03:42:28 (1789112548) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 03:42:38 (1789112558) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1999 sec =========== 03:42:48 (1789112568) === sanity-flr: start cleanup 03:42:49 (1789112569) === === sanity-flr: finish cleanup 03:42:57 (1789112577) ===