-----============= acceptance-small: sanity-flr ============----- Sat Sep 5 01:51:55 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg413-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 01:52:17 (1788587537) === oleg413-client.virtnet: executing check_config_client /mnt/lustre oleg413-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg413-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9a7e06a09800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a7e06a09800.idle_timeout=debug disable quota as required oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 01:53:02 (1788587582) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8434] == sanity-flr test 0a: lfs mirror create with -N option == 01:53:05 (1788587585) PASS 0a (19s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 01:53:24 (1788587604) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 01:53:28 (1788587608) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 01:53:32 (1788587612) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 01:54:01 (1788587641) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 01:54:05 (1788587645) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 01:54:09 (1788587649) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9a7e06a09800.stats=clear osc.lustre-OST0001-osc-ffff9a7e06a09800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 01:54:18 (1788587658) striped dir -i0 -c2 -H crush2 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (14s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 01:54:32 (1788587672) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 01:54:41 (1788587681) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 01:54:45 (1788587685) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (11s) == sanity-flr test 1: create components with setstripe options ========================================================== 01:54:57 (1788587697) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (13s) == sanity-flr test 2: create components from existing files ========================================================== 01:55:09 (1788587709) PASS 2 (10s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 01:55:19 (1788587719) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 01:55:28 (1788587728) striped dir -i0 -c2 -H crush2 /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (11s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 01:55:39 (1788587739) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (9s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 01:55:52 (1788587752) PASS 7 (9s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 01:56:01 (1788587761) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 2.4041 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176432 s, 5.9 MB/s PASS 21 (13s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 01:56:14 (1788587774) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.252666 s, 29.1 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 01:56:24 (1788587784) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (12s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 01:56:36 (1788587796) Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a7e06a09800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a7e06a09800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a7e06a09800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a7e06a09800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg413-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (84s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 01:58:00 (1788587880) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 0.562994 s, 20.5 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a7e06a09800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a7e06a09800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 11534336 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (46s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 01:58:46 (1788587926) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a7e06a09800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a7e06a09800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a7e06a09800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a7e06a09800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg413-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (86s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 02:00:12 (1788588012) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 6.57682 s, 8.0 MB/s -rw-r--r-- 1 root root 52428800 Sep 5 02:00 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.185s user 0m0.001s sys 0m0.175s Stopping /mnt/lustre-ost1 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a7e06a09800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a7e06a09800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.319s user 0m0.000s sys 0m0.195s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg413-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg413-server oleg413-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a7e06a09800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a7e06a09800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.337s user 0m0.000s sys 0m0.223s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg413-server: oleg413-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg413-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg413-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a7e06a09800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a7e06a09800.ost_server_uuid in FULL state after 0 sec PASS 33b (101s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 02:01:53 (1788588113) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 02:01:57 (1788588117) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 02:02:01 (1788588121) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 02:02:05 (1788588125) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 02:02:12 (1788588132) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.696037 s, 18.1 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 02:02:24 (1788588144) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415362 s, 25.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0323983 s, 32.4 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 02:02:31 (1788588151) PASS 36c (7s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 02:02:38 (1788588158) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00219575 s, 466 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000863579 s, 1.2 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00218654 s, 468 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00202212 s, 506 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 02:02:49 (1788588169) create test files with size 15752653 .. a7f848bb58da97d5005a83521621d38f - 6316fe480e77097686d0463a79379b8a - a9297802183f4ae73f4814122c66a130 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg413-client.virtnet /mnt/lustre (opts:) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 02:03:16 (1788588196) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (14s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 02:03:30 (1788588210) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 02:03:37 (1788588217) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998853 s, 10.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xba:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbb:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb8:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424188 s, 24.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb9:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xbc:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 02:03:47 (1788588227) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.126045 s, 16.6 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.307168 s, 13.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.189328 s, 11.1 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (18s) == sanity-flr test 42: lfs mirror verify ================= 02:04:05 (1788588245) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 02:04:09 (1788588249) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 02:04:13 (1788588253) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243186 s, 43.1 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 02:04:22 (1788588262) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.419067 s, 7.5 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 02:04:35 (1788588275) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 02:04:46 (1788588286) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.353065 s, 29.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg413-client.virtnet /mnt/lustre (opts:) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (16s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 02:05:02 (1788588302) striped dir -i0 -c2 -H crush2 /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9a7e0ed29000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.451766 s, 23.2 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 16398 flags: 1 Stopping client oleg413-client.virtnet /mnt/lustre (opts:) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 16398 flags: 1 llite.lustre-ffff9a7e1048a800.xattr_cache=1 PASS 44e (22s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 02:05:24 (1788588324) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (49s) == sanity-flr test 46: Verify setstripe --copy option ==== 02:06:13 (1788588373) striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (44s) == sanity-flr test 47: Verify mirror obj alloc =========== 02:06:57 (1788588417) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (5s) == sanity-flr test 48: Verify snapshot mirror ============ 02:07:02 (1788588422) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.36162 s, 8.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.268825 s, 11.7 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xa2340428 (nosync,stale) Mirror 2: 0xa2340428 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 6a4b976a00f734f678dc8fe61a91e0e4 - mirror 1 checksum: d04fa13dbbee9bef7a8b2e6900970c15 - mirror 2 checksum: 6a4b976a00f734f678dc8fe61a91e0e4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T02:07:06-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-05T02:07:06-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6a4b976a00f734f678dc8fe61a91e0e4 - mirror 1 checksum: d04fa13dbbee9bef7a8b2e6900970c15 - mirror 2 checksum: d04fa13dbbee9bef7a8b2e6900970c15 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T02:07:06-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd0:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-05T02:07:06-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } PASS 48 (13s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 02:07:17 (1788588437) Failing mds1 on oleg413-server Stopping /mnt/lustre-mds1 (opts:) on oleg413-server 02:07:26 (1788588446) shut down facet: mds1 facet_host: oleg413-server facet_failover_host: oleg413-server Failover mds1 to oleg413-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg413-server: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg413-client: oleg413-server: ssh exited with exit code 1 Started lustre-MDT0000 02:08:00 (1788588480) targets are mounted 02:08:00 (1788588480) facet_failover done oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xd:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0406117 s, 25.8 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xd lmm_fid: [0x200000405:0xd:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd2:0x0] } PASS 50A (64s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 02:08:21 (1788588501) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 02:08:26 (1788588506) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 02:08:30 (1788588510) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0128375 s, 1.3 MB/s PASS 50c (11s) == sanity-flr test 50d: mirror rsync keep holes ========== 02:08:41 (1788588521) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 02:08:45 (1788588525) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0104642 s, 391 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.179922 s, 22.8 kB/s PASS 60a (12s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 02:08:57 (1788588537) striped dir -i0 -c2 -H crush /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x16:0x0]: Invalid argument (22) PASS 60b (27s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 02:09:24 (1788588564) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 02:09:56 (1788588596) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0xd:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 02:10:21 (1788588621) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 02:10:48 (1788588648) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (10s) == sanity-flr test 70: mirror create and split race ====== 02:10:58 (1788588658) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 56418 mirror split pid 56419 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 56418 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 56419 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 02:12:06 (1788588726) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70a.sanity-flr Seed set to 8729 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero length punch hole truncating to largest ever: 0x2af6 000007 1788588729.271637 trunc from 00000000 to 0x002af5 (0x2af6 bytes) 000008 1788588729.298570 punch from 0x000034 to 0x002af5 (0x2ac2 bytes) truncating to largest ever: 0x3eba4 000009 1788588729.304338 trunc from 0x002af6 to 0x03eba3 (0x3c0ae bytes) 000010 1788588729.324881 trunc from 0x03eba4 to 0x03b841 (0x3362 bytes) 000012 1788588729.354636 mapread 0x01177b thru 0x0213d2 (0xfc58 bytes) 000013 1788588729.395774 write 0x01e261 thru 0x02215e (0x3efe bytes) 000014 1788588729.398540 write 0x019f94 thru 0x02518c (0xb1f9 bytes) 000015 1788588729.415357 read 0x03ae69 thru 0x03b841 (0x9d9 bytes) 000017 1788588729.438637 mapread 0x0264fc thru 0x028e0c (0x2911 bytes) 000019 1788588729.461143 write 0x0129e5 thru 0x01453f (0x1b5b bytes) 000020 1788588729.463626 trunc from 0x03b842 to 0x02c1b7 (0xf68a bytes) 000021 1788588729.499561 write 0x013c64 thru 0x01e51d (0xa8ba bytes) 000026 1788588729.508225 read 0x00374d thru 0x0120a5 (0xe959 bytes) 000029 1788588729.535640 MIRROR_EXTEND to 2 mirrors 000031 1788588730.670276 trunc from 0x02c1b8 to 0x009151 (0x23066 bytes) 000032 1788588730.791548 mapwrite 0x03e5de thru 0x03ffff (0x1a22 bytes) 000033 1788588730.878789 punch from 0x0071a6 to 0x012023 (0xae7e bytes) 000035 1788588730.885733 mapread 0x00f1c7 thru 0x013ac1 (0x48fb bytes) 000037 1788588730.897690 punch from 0x02b8b1 to 0x035a22 (0xa172 bytes) 000038 1788588730.902395 punch from 0x02a4f3 to 0x02ac81 (0x78f bytes) 000039 1788588730.911019 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x5fd6d4ea (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x16e78cd4 Mirror 3: 0x16e78cd4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000040 1788588732.117492 trunc from 0x040000 to 0x0117a7 (0x2e858 bytes) 000041 1788588732.233673 punch from 0x0081c6 to 0x00fb30 (0x796b bytes) 000043 1788588732.246235 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x117a8): Mirror 3: 0x570fb323 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x117a8): Mirror 3: 0x570fb323 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x117a8: skipped 000044 1788588732.558943 punch from 0x0002ca to 0x00a05d (0x9d94 bytes) truncating to largest ever: 0x3fb53 000046 1788588732.565603 trunc from 0x0117a8 to 0x03fb52 (0x2e3ab bytes) 000047 1788588732.586916 write 0x03eb54 thru 0x03ffff (0x14ac bytes) 000050 1788588732.595788 read 0x018bdf thru 0x020c9e (0x80c0 bytes) 000051 1788588732.613113 read 0x019e3b thru 0x01db6f (0x3d35 bytes) 000052 1788588732.615167 trunc from 0x040000 to 0x02150e (0x1eaf1 bytes) 000053 1788588732.639498 mapwrite 0x00eee5 thru 0x01135f (0x247b bytes) 000054 1788588732.745381 read 0x004f0b thru 0x008c13 (0x3d09 bytes) 000056 1788588732.746791 read 0x0074b3 thru 0x007549 (0x97 bytes) 000057 1788588732.748042 write 0x02e891 thru 0x03ab0e (0xc27e bytes) 000058 1788588732.756177 write 0x013e2b thru 0x01bd89 (0x7f5f bytes) 000059 1788588732.758022 mapwrite 0x01078b thru 0x01d7f9 (0xd06f bytes) 000060 1788588732.877423 trunc from 0x03ab0f to 0x00f449 (0x2b6c5 bytes) 000063 1788588732.912287 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) 000068 1788588733.110567 read 0x00f3f3 thru 0x00f449 (0x57 bytes) 000073 1788588733.126633 write 0x0343d1 thru 0x03ffff (0xbc2f bytes) 000075 1788588733.128925 punch from 0x0174f6 to 0x022132 (0xac3d bytes) 000076 1788588733.155183 mapwrite 0x01213f thru 0x013d4d (0x1c0f bytes) 000077 1788588733.217994 write 0x01815c thru 0x0272ed (0xf192 bytes) 000079 1788588733.233017 mapread 0x01d5ec thru 0x01f1d6 (0x1beb bytes) 000080 1788588733.234016 write 0x03fad5 thru 0x03ffff (0x52b bytes) 000081 1788588733.235405 write 0x02cfc8 thru 0x035505 (0x853e bytes) 000082 1788588733.242688 trunc from 0x040000 to 0x0168ef (0x29710 bytes) 000083 1788588733.264628 punch from 0x00552d to 0x01427c (0xed50 bytes) 000084 1788588733.269891 mapwrite 0x024f4a thru 0x0310a9 (0xc160 bytes) 000085 1788588733.379570 write 0x02ecb9 thru 0x039081 (0xa3c9 bytes) 000090 1788588733.386705 trunc from 0x039082 to 0x016320 (0x22d61 bytes) 000092 1788588733.420967 MIRROR_EXTEND to 2 mirrors 000093 1788588734.474332 punch from 0x0085c4 to 0x00b58d (0x2fca bytes) 000094 1788588734.595202 trunc from 0x016321 to 0x009337 (0xcfe9 bytes) 000095 1788588734.621534 punch from 0x005cc8 to 0x009337 (0x3670 bytes) 000097 1788588734.628185 trunc from 0x009338 to 0x0181ec (0xeeb5 bytes) 000099 1788588734.650132 mapwrite 0x025958 thru 0x03324c (0xd8f5 bytes) 000101 1788588734.767248 mapread 0x029028 thru 0x02bb95 (0x2b6e bytes) 000103 1788588734.770785 mapwrite 0x011c5b thru 0x01d897 (0xbc3d bytes) 000104 1788588734.889772 write 0x032d4f thru 0x03ffff (0xd2b1 bytes) 000105 1788588734.895808 mapread 0x02d76f thru 0x039eb6 (0xc748 bytes) 000107 1788588734.902594 mapwrite 0x01f126 thru 0x0260ce (0x6fa9 bytes) 000108 1788588734.975170 mapwrite 0x0307b8 thru 0x03e649 (0xde92 bytes) 000111 1788588735.116187 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6b6777d0 Mirror 3: 0x6b6777d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000112 1788588736.573357 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6b6777d0 Mirror 3: 0x6b6777d0 Mirror 4: 0x6b6777d0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000114 1788588738.371668 write 0x019038 thru 0x023ee2 (0xaeab bytes) 000116 1788588738.493425 MIRROR_RESYNC 4 mirrors 000119 1788588740.025306 punch from 0x00ea74 to 0x01d7f3 (0xed80 bytes) 000121 1788588740.089338 mapwrite 0x0226e8 thru 0x0239a3 (0x12bc bytes) 000122 1788588740.189622 trunc from 0x040000 to 0x029b6d (0x16492 bytes) 000123 1788588740.226209 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29b6e): Mirror 2: 0x6a8201f1 (stale) Mirror 3: 0x6a8201f1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29b6e): Mirror 2: 0x6a8201f1 (stale) Mirror 3: 0x6a8201f1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29b6e: skipped 000125 1788588740.585788 mapread 0x00d08b thru 0x00d493 (0x409 bytes) 000127 1788588740.593155 read 0x01d86c thru 0x025607 (0x7d9c bytes) 000131 1788588740.609487 write 0x028a2c thru 0x03593e (0xcf13 bytes) 000132 1788588740.615873 mapwrite 0x0224b3 thru 0x031f63 (0xfab1 bytes) 000134 1788588740.679784 mapwrite 0x03b394 thru 0x03ffff (0x4c6c bytes) 000135 1788588740.762994 punch from 0x00f7da to 0x013a13 (0x423a bytes) 000136 1788588740.769168 mapread 0x032e67 thru 0x03bb34 (0x8cce bytes) 000137 1788588740.771857 write 0x030eaa thru 0x03cb21 (0xbc78 bytes) 000138 1788588740.774559 punch from 0x00d25c to 0x0166d6 (0x947b bytes) 000139 1788588740.794758 punch from 0x038c3b to 0x03f5fd (0x69c3 bytes) 000140 1788588740.800563 write 0x001a1f thru 0x0051e7 (0x37c9 bytes) 000141 1788588740.813200 mapwrite 0x038d6b thru 0x03ffff (0x7295 bytes) 000142 1788588740.887637 mapread 0x035633 thru 0x03ffff (0xa9cd bytes) 000144 1788588740.890035 mapread 0x035bee thru 0x03cda5 (0x71b8 bytes) 000145 1788588740.891760 punch from 0x01b74a to 0x01f8e6 (0x419d bytes) 000146 1788588740.897759 punch from 0x024d1f to 0x02e245 (0x9527 bytes) 000147 1788588740.909566 mapwrite 0x02d77f thru 0x03340e (0x5c90 bytes) 000148 1788588740.995458 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x93fed485 (stale) Mirror 3: 0x93fed485 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x85ff5427 Mirror 4: 0x85ff5427 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000149 1788588742.197009 mapwrite 0x013ffd thru 0x01f18b (0xb18f bytes) 000150 1788588742.430233 trunc from 0x040000 to 0x01c533 (0x23acc bytes) 000152 1788588742.457033 punch from 0x01435b to 0x01a438 (0x60de bytes) 000155 1788588742.464278 mapread 0x00d692 thru 0x01028f (0x2bfe bytes) 000156 1788588742.467783 write 0x009817 thru 0x01315e (0x9948 bytes) 000158 1788588742.488256 mapwrite 0x03a102 thru 0x03ffff (0x5efe bytes) 000160 1788588742.640309 write 0x03b690 thru 0x03ffff (0x4970 bytes) 000161 1788588742.650475 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x93fed485 (stale) Mirror 3: 0x93fed485 (stale) Mirror 4: 0x85ff5427 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x705e280d Mirror 5: 0x705e280d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000162 1788588744.245441 trunc from 0x040000 to 0x01acdd (0x25322 bytes) 000164 1788588744.456800 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1acde): Mirror 2: 0x761052dd (stale) Mirror 4: 0x9a0f1ff8 (stale) Mirror 5: 0xe6b1340c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1acde): Mirror 2: 0x761052dd (stale) Mirror 4: 0x9a0f1ff8 (stale) Mirror 5: 0xe6b1340c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1acde: skipped 000165 1788588745.098922 mapwrite 0x00b6d3 thru 0x00d177 (0x1aa5 bytes) 000166 1788588745.167947 trunc from 0x01acde to 0x00d945 (0xd398 bytes) 000167 1788588745.186169 trunc from 0x00d946 to 0x02c10f (0x1e7ca bytes) 000168 1788588745.202557 read 0x024d88 thru 0x02c10f (0x7388 bytes) 000169 1788588745.227150 read 0x0196cc thru 0x02708a (0xd9bf bytes) 000170 1788588745.241601 read 0x01a09d thru 0x01add8 (0xd3c bytes) 000172 1788588745.250720 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x2c110): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c110): Mirror 2: 0x9770b734 (stale) Mirror 4: 0x79634fcf (stale) Mirror 5: 0x71d8d4ea (stale) CRC-32 checksum value for chunk [0, 0x2c110): Mirror 1: 0x7c9c389b Mirror 6: 0x7c9c389b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c110: skipped 000173 1788588747.086004 mapread 0x00d577 thru 0x015356 (0x7de0 bytes) 000174 1788588747.134188 mapwrite 0x03cf91 thru 0x03ffff (0x306f bytes) 000175 1788588747.366947 read 0x02494d thru 0x02aed5 (0x6589 bytes) 000176 1788588747.393566 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x93fed485 (stale) Mirror 4: 0x85ff5427 (stale) Mirror 5: 0x705e280d (stale) Mirror 6: 0x5a992d8b (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x67119bb0 Mirror 7: 0x67119bb0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000177 1788588748.403696 write 0x0285e5 thru 0x02d02f (0x4a4b bytes) 000179 1788588748.494030 trunc from 0x040000 to 0x00e27e (0x31d81 bytes) 000182 1788588748.547380 write 0x0099dc thru 0x00b412 (0x1a37 bytes) 000184 1788588748.558581 write 0x03b8b5 thru 0x03ffff (0x474b bytes) 000185 1788588748.565874 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x93fed485 (stale) Mirror 4: 0x85ff5427 (stale) Mirror 5: 0x705e280d (stale) Mirror 6: 0x5a992d8b (stale) Mirror 7: 0x67119bb0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5ebe7951 Mirror 8: 0x5ebe7951 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000187 1788588750.247067 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x85ff5427 (stale) Mirror 5: 0x705e280d (stale) Mirror 6: 0x5a992d8b (stale) Mirror 7: 0x67119bb0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x85ff5427 (stale) Mirror 5: 0x705e280d (stale) Mirror 6: 0x5a992d8b (stale) Mirror 7: 0x67119bb0 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5ebe7951 Mirror 8: 0x5ebe7951 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000189 1788588751.217043 MIRROR_RESYNC 6 mirrors 000191 1788588752.454955 MIRROR_EXTEND to 7 mirrors 000192 1788588753.605336 write 0x03df94 thru 0x03ffff (0x206c bytes) 000193 1788588753.815151 write 0x02e42a thru 0x0304cd (0x20a4 bytes) 000195 1788588753.844304 mapwrite 0x0063ba thru 0x00f435 (0x907c bytes) 000197 1788588754.000759 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5ebe7951 (stale) Mirror 5: 0x5ebe7951 (stale) Mirror 6: 0x5ebe7951 (stale) Mirror 7: 0x5ebe7951 (stale) Mirror 8: 0x5ebe7951 (stale) Mirror 9: 0x5ebe7951 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x291d3dc4 Mirror 10: 0x291d3dc4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000198 1788588755.504000 mapwrite 0x02a1a7 thru 0x031069 (0x6ec3 bytes) 000200 1788588755.753993 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x5ebe7951 (stale) Mirror 5: 0x5ebe7951 (stale) Mirror 6: 0x5ebe7951 (stale) Mirror 7: 0x5ebe7951 (stale) Mirror 8: 0x5ebe7951 (stale) Mirror 9: 0x5ebe7951 (stale) Mirror 10: 0x291d3dc4 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x1c4de0fa Mirror 11: 0x1c4de0fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000202 1788588757.491025 read 0x01df86 thru 0x020901 (0x297c bytes) 000203 1788588757.507450 trunc from 0x040000 to 0x03f04a (0xfb5 bytes) 000204 1788588757.669252 punch from 0x00e0a2 to 0x015074 (0x6fd3 bytes) 000205 1788588757.677006 mapwrite 0x0186fe thru 0x01b16f (0x2a72 bytes) 000206 1788588757.876518 write 0x03532d thru 0x03ffff (0xacd3 bytes) 000208 1788588757.909212 punch from 0x002771 to 0x00b8e8 (0x9178 bytes) 000210 1788588757.948133 MIRROR_RESYNC 9 mirrors 000211 1788588759.931391 MIRROR_EXTEND to 10 mirrors 000212 1788588761.080720 punch from 0x035929 to 0x03ffff (0xa6d7 bytes) 000213 1788588761.180353 write 0x035493 thru 0x03ec36 (0x97a4 bytes) 000214 1788588761.199591 punch from 0x022873 to 0x026fc2 (0x4750 bytes) 000215 1788588761.229140 mapwrite 0x00727e thru 0x01304d (0xbdd0 bytes) 000216 1788588761.374085 trunc from 0x040000 to 0x01e2f9 (0x21d06 bytes) 000217 1788588761.422365 write 0x00f3fa thru 0x014ebc (0x5ac3 bytes) 000218 1788588761.423648 mapread 0x0174e6 thru 0x01e2f9 (0x6e14 bytes) 000220 1788588761.443999 write 0x03d9e6 thru 0x03f7e0 (0x1dfb bytes) 000222 1788588761.451081 mapwrite 0x012c04 thru 0x021409 (0xe806 bytes) 000223 1788588761.628645 mapwrite 0x03a7f3 thru 0x03ffff (0x580d bytes) 000224 1788588761.753455 mapread 0x02e8f0 thru 0x03452b (0x5c3c bytes) 000225 1788588761.769749 punch from 0x0247e0 to 0x02f463 (0xac84 bytes) 000228 1788588761.803157 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x7b18af14 (stale) Mirror 5: 0x7b18af14 (stale) Mirror 6: 0x7b18af14 (stale) Mirror 7: 0x7b18af14 (stale) Mirror 8: 0x7b18af14 (stale) Mirror 9: 0x7b18af14 (stale) Mirror 10: 0x7b18af14 (stale) Mirror 11: 0x7b18af14 (stale) Mirror 12: 0x7b18af14 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x79fa9a7d Mirror 13: 0x79fa9a7d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000229 1788588763.491083 mapread 0x03340c thru 0x03ffff (0xcbf4 bytes) 000230 1788588763.572993 trunc from 0x040000 to 0x001d52 (0x3e2ad bytes) 000231 1788588763.735543 mapread 0x00010f thru 0x001d52 (0x1c44 bytes) 000232 1788588763.744761 write 0x0195d5 thru 0x0243f6 (0xae22 bytes) 000233 1788588763.748789 mapread 0x021d66 thru 0x0243f6 (0x2691 bytes) 000234 1788588763.756720 write 0x018b8b thru 0x0286a4 (0xfb1a bytes) 000235 1788588763.764345 trunc from 0x0286a5 to 0x030742 (0x809e bytes) 000236 1788588763.797702 trunc from 0x030743 to 0x01a7af (0x15f93 bytes) 000238 1788588763.823291 punch from 0x014fea to 0x01a7af (0x57c6 bytes) 000240 1788588763.834594 read 0x00c268 thru 0x00ff2b (0x3cc4 bytes) 000242 1788588763.843759 mapwrite 0x0345e4 thru 0x03a6a6 (0x60c3 bytes) 000243 1788588763.959435 punch from 0x0045c3 to 0x00e258 (0x9c96 bytes) 000244 1788588763.966663 punch from 0x024388 to 0x024beb (0x864 bytes) 000247 1788588763.975437 mapwrite 0x026048 thru 0x026441 (0x3fa bytes) 000248 1788588764.028797 mapwrite 0x020155 thru 0x02e7b6 (0xe662 bytes) 000250 1788588764.089606 punch from 0x02f725 to 0x0322d2 (0x2bae bytes) 000252 1788588764.095163 mapread 0x014978 thru 0x0220db (0xd764 bytes) 000254 1788588764.114439 read 0x0250f8 thru 0x02e739 (0x9642 bytes) 000257 1788588764.117134 punch from 0x02261b to 0x02510c (0x2af2 bytes) 000258 1788588764.124443 mapread 0x02a634 thru 0x02fd2d (0x56fa bytes) 000260 1788588764.164984 MIRROR_RESYNC 11 mirrors 000263 1788588766.753706 MIRROR_SPLIT mirror 1 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) 000266 1788588767.440606 write 0x013dc6 thru 0x019674 (0x58af bytes) 000269 1788588767.591520 write 0x01fdbc thru 0x028442 (0x8687 bytes) 000270 1788588767.607954 write 0x00e66e thru 0x00eba5 (0x538 bytes) 000272 1788588767.616995 trunc from 0x03a6a7 to 0x016c01 (0x23aa5 bytes) 000275 1788588767.650700 punch from 0x00815f to 0x014266 (0xc108 bytes) 000276 1788588767.670884 mapwrite 0x0146c6 thru 0x01aa7b (0x63b6 bytes) 000278 1788588767.860777 write 0x02d80e thru 0x02f497 (0x1c8a bytes) 000279 1788588767.863662 mapread 0x029e3f thru 0x02f497 (0x5659 bytes) 000280 1788588767.871728 mapwrite 0x0036f5 thru 0x0099c2 (0x62ce bytes) 000281 1788588767.960097 MIRROR_RESYNC 10 mirrors 000283 1788588770.057450 mapread 0x00f5ff thru 0x019cc0 (0xa6c2 bytes) 000286 1788588770.110598 read 0x007950 thru 0x00a650 (0x2d01 bytes) 000288 1788588770.128648 write 0x00a444 thru 0x0186ff (0xe2bc bytes) 000290 1788588770.349638 punch from 0x02660f to 0x02cd8e (0x6780 bytes) 000292 1788588770.389186 write 0x03b052 thru 0x03ffff (0x4fae bytes) 000293 1788588770.392234 trunc from 0x040000 to 0x03d1ef (0x2e10 bytes) 000295 1788588770.446768 mapread 0x003b9d thru 0x00b86a (0x7cce bytes) 000296 1788588770.480024 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d1f0): Mirror 5: 0xfe3143bb (stale) Mirror 6: 0xfe3143bb (stale) Mirror 7: 0xfe3143bb (stale) Mirror 8: 0xfe3143bb (stale) Mirror 10: 0xfe3143bb (stale) Mirror 11: 0xfe3143bb (stale) Mirror 12: 0xfe3143bb (stale) Mirror 13: 0xfe3143bb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d1f0): Mirror 5: 0xfe3143bb (stale) Mirror 6: 0xfe3143bb (stale) Mirror 7: 0xfe3143bb (stale) Mirror 8: 0xfe3143bb (stale) Mirror 10: 0xfe3143bb (stale) Mirror 11: 0xfe3143bb (stale) Mirror 12: 0xfe3143bb (stale) Mirror 13: 0xfe3143bb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d1f0: skipped 000298 1788588771.415188 MIRROR_RESYNC 9 mirrors 000299 1788588773.711067 write 0x02e8a1 thru 0x038b90 (0xa2f0 bytes) 000302 1788588773.820599 read 0x000b72 thru 0x008b8f (0x801e bytes) 000303 1788588773.832167 punch from 0x009553 to 0x012649 (0x90f7 bytes) 000304 1788588773.847069 mapwrite 0x02712e thru 0x031503 (0xa3d6 bytes) 000305 1788588773.949087 mapwrite 0x00ee4a thru 0x019c51 (0xae08 bytes) 000307 1788588774.049168 mapread 0x0077af thru 0x00ff2e (0x8780 bytes) 000308 1788588774.051185 read 0x008600 thru 0x011c12 (0x9613 bytes) 000309 1788588774.051993 mapread 0x02447d thru 0x028506 (0x408a bytes) 000310 1788588774.053543 write 0x03dc23 thru 0x03fed9 (0x22b7 bytes) 000311 1788588774.065494 punch from 0x01c65d to 0x02c607 (0xffab bytes) 000313 1788588774.078122 write 0x03f5db thru 0x03ffff (0xa25 bytes) 000314 1788588774.084809 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x777187c2 (stale) Mirror 6: 0x777187c2 (stale) Mirror 7: 0x777187c2 (stale) Mirror 10: 0x777187c2 (stale) Mirror 11: 0x777187c2 (stale) Mirror 12: 0x777187c2 (stale) Mirror 13: 0x777187c2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x777187c2 (stale) Mirror 6: 0x777187c2 (stale) Mirror 7: 0x777187c2 (stale) Mirror 10: 0x777187c2 (stale) Mirror 11: 0x777187c2 (stale) Mirror 12: 0x777187c2 (stale) Mirror 13: 0x777187c2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000315 1788588774.772038 trunc from 0x040000 to 0x0178c3 (0x2873c bytes) 000316 1788588774.803608 read 0x013254 thru 0x0178c3 (0x4670 bytes) 000317 1788588774.823845 trunc from 0x0178c4 to 0x011b29 (0x5d9a bytes) 000318 1788588774.855874 MIRROR_RESYNC 8 mirrors 000319 1788588776.488756 write 0x005f62 thru 0x0088c7 (0x2966 bytes) 000320 1788588776.605580 read 0x00c361 thru 0x011b29 (0x57c9 bytes) 000322 1788588776.639535 trunc from 0x011b2a to 0x0005c0 (0x11569 bytes) 000323 1788588776.675552 punch from 0x000340 to 0x0005c0 (0x281 bytes) 000325 1788588776.687760 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x5c1): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x5c1): Mirror 5: 0xfe39cb45 (stale) Mirror 6: 0xfe39cb45 (stale) Mirror 7: 0xfe39cb45 (stale) Mirror 10: 0xfe39cb45 (stale) Mirror 11: 0xfe39cb45 (stale) Mirror 12: 0xfe39cb45 (stale) Mirror 13: 0xfe39cb45 (stale) CRC-32 checksum value for chunk [0, 0x5c1): Mirror 1: 0xfe39cb45 Mirror 14: 0xfe39cb45 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x5c1: skipped 000329 1788588778.184153 trunc from 0x0005c1 to 0x02b86b (0x2b2ab bytes) 000332 1788588778.393323 mapread 0x020375 thru 0x02a577 (0xa203 bytes) 000333 1788588778.418803 punch from 0x01bdab to 0x0279aa (0xbc00 bytes) 000335 1788588778.426077 write 0x02a7bf thru 0x02c4e0 (0x1d22 bytes) 000337 1788588778.428710 trunc from 0x02c4e1 to 0x00fb4f (0x1c991 bytes) 000340 1788588778.465681 trunc from 0x00fb50 to 0x021307 (0x117b8 bytes) 000341 1788588778.492295 mapwrite 0x009bc5 thru 0x010915 (0x6d51 bytes) 000342 1788588778.696547 mapread 0x01f2c9 thru 0x021307 (0x203f bytes) 000343 1788588778.715048 write 0x01fdf4 thru 0x0249f2 (0x4bff bytes) 000347 1788588778.726714 punch from 0x0093cf to 0x014186 (0xadb8 bytes) 000349 1788588778.749965 read 0x01dc81 thru 0x0249f2 (0x6d72 bytes) 000350 1788588778.755470 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x249f3): Mirror 5: 0xad87f5e3 (stale) Mirror 6: 0xad87f5e3 (stale) Mirror 7: 0xad87f5e3 (stale) Mirror 10: 0xad87f5e3 (stale) Mirror 12: 0xad87f5e3 (stale) Mirror 13: 0xad87f5e3 (stale) Mirror 14: 0x623aeae9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x249f3): Mirror 5: 0xad87f5e3 (stale) Mirror 6: 0xad87f5e3 (stale) Mirror 7: 0xad87f5e3 (stale) Mirror 10: 0xad87f5e3 (stale) Mirror 12: 0xad87f5e3 (stale) Mirror 13: 0xad87f5e3 (stale) Mirror 14: 0x623aeae9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x249f3: skipped 000353 1788588779.284262 MIRROR_RESYNC 8 mirrors 000354 1788588781.309128 trunc from 0x0249f3 to 0x001ce3 (0x22d0f bytes) 000355 1788588781.417425 read 0x000a5e thru 0x001ce3 (0x1286 bytes) 000356 1788588781.422601 mapwrite 0x015c1e thru 0x01c5a9 (0x698c bytes) 000357 1788588781.646683 mapread 0x01806f thru 0x01c5a9 (0x453b bytes) 000358 1788588781.648897 mapwrite 0x013637 thru 0x018170 (0x4b3a bytes) 000359 1788588781.767274 mapwrite 0x03c565 thru 0x03ffff (0x3a9b bytes) 000360 1788588781.996544 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x38a0c123 (stale) Mirror 6: 0x38a0c123 (stale) Mirror 7: 0x38a0c123 (stale) Mirror 10: 0x38a0c123 (stale) Mirror 12: 0x38a0c123 (stale) Mirror 13: 0x38a0c123 (stale) Mirror 14: 0x38a0c123 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xc29ceada Mirror 15: 0xc29ceada lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000361 1788588783.233184 MIRROR_RESYNC 9 mirrors 000362 1788588785.585722 punch from 0x0208c8 to 0x030601 (0xfd3a bytes) 000364 1788588785.797491 mapwrite 0x03d446 thru 0x03ffff (0x2bba bytes) 000365 1788588785.979779 trunc from 0x040000 to 0x00e1e6 (0x31e19 bytes) 000366 1788588786.031411 punch from 0x00c764 to 0x00e1e6 (0x1a83 bytes) 000367 1788588786.048088 trunc from 0x00e1e7 to 0x02737c (0x19196 bytes) 000368 1788588786.072302 punch from 0x00b687 to 0x00bf0e (0x888 bytes) 000369 1788588786.080575 mapread 0x014e1d thru 0x01b29a (0x647e bytes) 000370 1788588786.110265 mapwrite 0x01ecf6 thru 0x0245f4 (0x58ff bytes) 000371 1788588786.267545 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2737d): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2737d): Mirror 5: 0x19f8c097 (stale) Mirror 6: 0x19f8c097 (stale) Mirror 7: 0x19f8c097 (stale) Mirror 10: 0x19f8c097 (stale) Mirror 12: 0x19f8c097 (stale) Mirror 13: 0x19f8c097 (stale) Mirror 14: 0x19f8c097 (stale) Mirror 15: 0x19f8c097 (stale) CRC-32 checksum value for chunk [0, 0x2737d): Mirror 1: 0x2a9212c0 Mirror 16: 0x2a9212c0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2737d: skipped 000372 1788588787.876234 read 0x0078e3 thru 0x014b2f (0xd24d bytes) 000375 1788588787.931765 read 0x0186d5 thru 0x02657d (0xdea9 bytes) 000376 1788588787.975973 write 0x0375d1 thru 0x03e781 (0x71b1 bytes) 000377 1788588788.153195 write 0x0125f4 thru 0x014824 (0x2231 bytes) 000379 1788588788.181801 punch from 0x008519 to 0x00fbdc (0x76c4 bytes) 000381 1788588788.228593 mapread 0x0344f6 thru 0x03e781 (0xa28c bytes) 000383 1788588788.243772 write 0x02fb6a thru 0x03a65f (0xaaf6 bytes) 000384 1788588788.249006 mapread 0x01edc1 thru 0x0274f9 (0x8739 bytes) 000385 1788588788.383804 trunc from 0x03e782 to 0x0139d9 (0x2ada8 bytes) 000387 1788588788.500461 read 0x00b302 thru 0x0139d9 (0x86d8 bytes) 000388 1788588788.548742 punch from 0x012990 to 0x0139d9 (0x104a bytes) 000389 1788588788.555681 mapwrite 0x02cacf thru 0x02f8b1 (0x2de3 bytes) 000391 1788588788.818985 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x2f8b2): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f8b2): Mirror 5: 0xab1ba35f (stale) Mirror 6: 0xab1ba35f (stale) Mirror 7: 0xab1ba35f (stale) Mirror 10: 0xab1ba35f (stale) Mirror 12: 0xab1ba35f (stale) Mirror 13: 0xab1ba35f (stale) Mirror 14: 0xab1ba35f (stale) Mirror 15: 0xab1ba35f (stale) Mirror 16: 0xa64eb1c5 (stale) CRC-32 checksum value for chunk [0, 0x2f8b2): Mirror 1: 0xfd38f85f Mirror 17: 0xfd38f85f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f8b2: skipped 000392 1788588790.611714 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f8b2): Mirror 5: 0xab1ba35f (stale) Mirror 6: 0xab1ba35f (stale) Mirror 7: 0xab1ba35f (stale) Mirror 10: 0xab1ba35f (stale) Mirror 12: 0xab1ba35f (stale) Mirror 14: 0xab1ba35f (stale) Mirror 15: 0xab1ba35f (stale) Mirror 16: 0xa64eb1c5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f8b2): Mirror 5: 0xab1ba35f (stale) Mirror 6: 0xab1ba35f (stale) Mirror 7: 0xab1ba35f (stale) Mirror 10: 0xab1ba35f (stale) Mirror 12: 0xab1ba35f (stale) Mirror 14: 0xab1ba35f (stale) Mirror 15: 0xab1ba35f (stale) Mirror 16: 0xa64eb1c5 (stale) CRC-32 checksum value for chunk [0, 0x2f8b2): Mirror 1: 0xfd38f85f Mirror 17: 0xfd38f85f lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f8b2: skipped 000395 1788588792.422911 punch from 0x001e8b to 0x0084e8 (0x665e bytes) 000397 1788588792.580270 mapwrite 0x009232 thru 0x01152e (0x82fd bytes) 000399 1788588792.833626 mapread 0x0033f7 thru 0x005aac (0x26b6 bytes) 000400 1788588792.854219 mapwrite 0x039de1 thru 0x03ffff (0x621f bytes) 000401 1788588793.049416 mapread 0x002b3b thru 0x00ec45 (0xc10b bytes) 000405 1788588793.058810 mapwrite 0x002d20 thru 0x003ba8 (0xe89 bytes) 000407 1788588793.172058 read 0x033fa1 thru 0x036f23 (0x2f83 bytes) 000410 1788588793.179457 trunc from 0x040000 to 0x03a274 (0x5d8b bytes) 000411 1788588793.222348 mapwrite 0x0099a2 thru 0x01403e (0xa69d bytes) 000412 1788588793.434015 write 0x00ed1e thru 0x017ce5 (0x8fc8 bytes) 000414 1788588793.435862 write 0x0217a4 thru 0x026a17 (0x5274 bytes) 000415 1788588793.452191 mapread 0x009ae5 thru 0x014df4 (0xb310 bytes) 000416 1788588793.453319 mapread 0x02ffe1 thru 0x03909b (0x90bb bytes) 000418 1788588793.462306 mapread 0x024226 thru 0x02c947 (0x8722 bytes) 000419 1788588793.465060 write 0x02e9f8 thru 0x03dd66 (0xf36f bytes) 000420 1788588793.467760 mapread 0x01d623 thru 0x02a60f (0xcfed bytes) 000421 1788588793.479400 punch from 0x002f70 to 0x00d9b4 (0xaa45 bytes) 000422 1788588793.509001 mapread 0x03d47b thru 0x03dd66 (0x8ec bytes) 000426 1788588793.513056 trunc from 0x03dd67 to 0x037db9 (0x5fad bytes) 000427 1788588793.539210 punch from 0x036663 to 0x037db9 (0x1757 bytes) 000428 1788588793.547160 write 0x037a23 thru 0x03fdca (0x83a8 bytes) 000429 1788588793.559357 mapwrite 0x009555 thru 0x00cfe5 (0x3a91 bytes) 000430 1788588793.649983 MIRROR_RESYNC 10 mirrors 000431 1788588796.483183 write 0x036e3a thru 0x03ffff (0x91c6 bytes) 000432 1788588796.574659 write 0x03da4e thru 0x03ffff (0x25b2 bytes) 000433 1788588796.579131 write 0x03c280 thru 0x03cec7 (0xc48 bytes) 000434 1788588796.583475 trunc from 0x040000 to 0x01474e (0x2b8b1 bytes) 000435 1788588796.643218 mapwrite 0x00fae7 thru 0x01a4c4 (0xa9de bytes) 000436 1788588796.842004 mapwrite 0x01610b thru 0x01a472 (0x4368 bytes) 000438 1788588796.905092 write 0x013fe9 thru 0x015ee1 (0x1ef9 bytes) 000439 1788588796.906597 read 0x0007ed thru 0x009890 (0x90a4 bytes) 000440 1788588796.919690 MIRROR_RESYNC 10 mirrors 000441 1788588798.543373 mapwrite 0x00148c thru 0x003d33 (0x28a8 bytes) 000442 1788588798.832751 mapwrite 0x01082d thru 0x01d4f5 (0xccc9 bytes) 000443 1788588798.919607 trunc from 0x01d4f6 to 0x0043dc (0x19119 bytes) 000444 1788588798.951219 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x43dd): Mirror 5: 0x94280550 (stale) Mirror 6: 0x94280550 (stale) Mirror 7: 0x94280550 (stale) Mirror 10: 0x94280550 (stale) Mirror 14: 0x94280550 (stale) Mirror 15: 0x94280550 (stale) Mirror 16: 0x94280550 (stale) Mirror 17: 0x94280550 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x43dd): Mirror 5: 0x94280550 (stale) Mirror 6: 0x94280550 (stale) Mirror 7: 0x94280550 (stale) Mirror 10: 0x94280550 (stale) Mirror 14: 0x94280550 (stale) Mirror 15: 0x94280550 (stale) Mirror 16: 0x94280550 (stale) Mirror 17: 0x94280550 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x43dd: skipped 000445 1788588799.429591 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x43dd): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x43dd): Mirror 5: 0x94280550 (stale) Mirror 6: 0x94280550 (stale) Mirror 7: 0x94280550 (stale) Mirror 10: 0x94280550 (stale) Mirror 14: 0x94280550 (stale) Mirror 15: 0x94280550 (stale) Mirror 16: 0x94280550 (stale) Mirror 17: 0x94280550 (stale) CRC-32 checksum value for chunk [0, 0x43dd): Mirror 1: 0xc5991222 Mirror 18: 0xc5991222 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x43dd: skipped 000446 1788588800.778277 punch from 0x003273 to 0x0043dc (0x116a bytes) 000447 1788588800.878545 mapread 0x0042a8 thru 0x0043dc (0x135 bytes) 000449 1788588800.885620 trunc from 0x0043dd to 0x01eacc (0x1a6f0 bytes) 000450 1788588800.921940 MIRROR_RESYNC 10 mirrors 000451 1788588803.063922 mapwrite 0x0268ba thru 0x029182 (0x28c9 bytes) 000452 1788588803.358187 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x29183): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 10: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29183): Mirror 5: 0x223db382 (stale) Mirror 6: 0x223db382 (stale) Mirror 7: 0x223db382 (stale) Mirror 10: 0x223db382 (stale) Mirror 14: 0x223db382 (stale) Mirror 15: 0x223db382 (stale) Mirror 16: 0x223db382 (stale) Mirror 17: 0x223db382 (stale) Mirror 18: 0x223db382 (stale) CRC-32 checksum value for chunk [0, 0x29183): Mirror 1: 0xf5616095 Mirror 19: 0xf5616095 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29183: skipped 000455 1788588804.789002 read 0x0013db thru 0x007706 (0x632c bytes) 000456 1788588804.812603 mapread 0x00efaf thru 0x01051a (0x156c bytes) 000458 1788588804.884668 mapwrite 0x007cd5 thru 0x01723f (0xf56b bytes) 000459 1788588805.204581 MIRROR_RESYNC 11 mirrors 000461 1788588807.462190 write 0x01d912 thru 0x01e5c5 (0xcb4 bytes) 000462 1788588807.556091 mapwrite 0x00a482 thru 0x018771 (0xe2f0 bytes) 000463 1788588807.698958 mapread 0x002357 thru 0x00e743 (0xc3ed bytes) 000464 1788588807.712329 punch from 0x016a3b to 0x021e4d (0xb413 bytes) 000466 1788588807.720301 mapwrite 0x013ff1 thru 0x01b08b (0x709b bytes) 000467 1788588807.791466 mapwrite 0x0147aa thru 0x0151f2 (0xa49 bytes) 000468 1788588807.838962 mapread 0x0215cc thru 0x028ec9 (0x78fe bytes) 000470 1788588807.849883 mapread 0x028d26 thru 0x029182 (0x45d bytes) 000471 1788588807.851005 mapread 0x011c1d thru 0x01aa91 (0x8e75 bytes) 000476 1788588807.860137 MIRROR_SPLIT mirror 6 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29183): Mirror 5: 0x82d3dc3b (stale) Mirror 6: 0x82d3dc3b (stale) Mirror 7: 0x82d3dc3b (stale) Mirror 10: 0x82d3dc3b (stale) Mirror 14: 0x82d3dc3b (stale) Mirror 16: 0x82d3dc3b (stale) Mirror 17: 0x82d3dc3b (stale) Mirror 18: 0x82d3dc3b (stale) Mirror 19: 0x82d3dc3b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29183): Mirror 5: 0x82d3dc3b (stale) Mirror 6: 0x82d3dc3b (stale) Mirror 7: 0x82d3dc3b (stale) Mirror 10: 0x82d3dc3b (stale) Mirror 14: 0x82d3dc3b (stale) Mirror 16: 0x82d3dc3b (stale) Mirror 17: 0x82d3dc3b (stale) Mirror 18: 0x82d3dc3b (stale) Mirror 19: 0x82d3dc3b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29183: skipped 000480 1788588808.549081 trunc from 0x029183 to 0x015008 (0x1417a bytes) 000481 1788588808.564471 trunc from 0x015009 to 0x003a69 (0x1159f bytes) 000482 1788588808.589794 read 0x00212f thru 0x003a69 (0x193b bytes) 000483 1788588808.602090 punch from 0x00199f to 0x003317 (0x1979 bytes) 000484 1788588808.613483 read 0x0026c2 thru 0x003a69 (0x13a8 bytes) 000486 1788588808.626287 trunc from 0x003a6a to 0x016713 (0x12caa bytes) 000487 1788588808.646919 trunc from 0x016714 to 0x01d9c8 (0x72b5 bytes) 000488 1788588808.665038 write 0x011741 thru 0x01e444 (0xcd04 bytes) 000489 1788588808.681679 MIRROR_SPLIT mirror 4 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1e445): Mirror 5: 0xf7f178b (stale) Mirror 6: 0xf7f178b (stale) Mirror 7: 0xf7f178b (stale) Mirror 14: 0xf7f178b (stale) Mirror 16: 0xf7f178b (stale) Mirror 17: 0xf7f178b (stale) Mirror 18: 0xf7f178b (stale) Mirror 19: 0xf7f178b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e445): Mirror 5: 0xf7f178b (stale) Mirror 6: 0xf7f178b (stale) Mirror 7: 0xf7f178b (stale) Mirror 14: 0xf7f178b (stale) Mirror 16: 0xf7f178b (stale) Mirror 17: 0xf7f178b (stale) Mirror 18: 0xf7f178b (stale) Mirror 19: 0xf7f178b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e445: skipped 000491 1788588809.232180 trunc from 0x01e445 to 0x002c88 (0x1b7bc bytes) 000492 1788588809.258970 punch from 0x0016a0 to 0x002c88 (0x15e9 bytes) 000493 1788588809.268782 mapwrite 0x00cd39 thru 0x012fcd (0x6295 bytes) 000494 1788588809.376283 read 0x006542 thru 0x00ed96 (0x8855 bytes) 000497 1788588809.388256 read 0x00b7a2 thru 0x0128bb (0x711a bytes) 000502 1788588809.391290 write 0x023b7f thru 0x02ce68 (0x92ea bytes) 000503 1788588809.402366 MIRROR_RESYNC 9 mirrors 000504 1788588811.564330 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) 000505 1788588812.174317 trunc from 0x02ce69 to 0x032a9c (0x5c34 bytes) 000506 1788588812.269474 mapwrite 0x03578a thru 0x03ffff (0xa876 bytes) 000507 1788588812.385646 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xd24517d3 (stale) Mirror 7: 0xd24517d3 (stale) Mirror 14: 0xd24517d3 (stale) Mirror 16: 0xd24517d3 (stale) Mirror 17: 0xd24517d3 (stale) Mirror 18: 0xd24517d3 (stale) Mirror 19: 0xd24517d3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd1d8105a Mirror 20: 0xd1d8105a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000508 1788588813.467147 mapwrite 0x005c51 thru 0x00fd6d (0xa11d bytes) 000510 1788588813.829127 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xd24517d3 (stale) Mirror 7: 0xd24517d3 (stale) Mirror 14: 0xd24517d3 (stale) Mirror 16: 0xd24517d3 (stale) Mirror 18: 0xd24517d3 (stale) Mirror 19: 0xd24517d3 (stale) Mirror 20: 0xd1d8105a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xd24517d3 (stale) Mirror 7: 0xd24517d3 (stale) Mirror 14: 0xd24517d3 (stale) Mirror 16: 0xd24517d3 (stale) Mirror 18: 0xd24517d3 (stale) Mirror 19: 0xd24517d3 (stale) Mirror 20: 0xd1d8105a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000513 1788588814.611950 punch from 0x0072d6 to 0x00a0c6 (0x2df1 bytes) 000514 1788588814.626198 mapwrite 0x0154f9 thru 0x01685e (0x1366 bytes) 000515 1788588814.743555 mapread 0x025744 thru 0x03365f (0xdf1c bytes) 000516 1788588814.776983 read 0x011b5c thru 0x0168d6 (0x4d7b bytes) 000517 1788588814.784878 punch from 0x00a3f9 to 0x017c97 (0xd89f bytes) 000518 1788588814.793716 read 0x0315e5 thru 0x038cd8 (0x76f4 bytes) 000520 1788588814.827656 mapread 0x019b5e thru 0x01d657 (0x3afa bytes) 000521 1788588814.828987 mapread 0x015ea5 thru 0x01e2c6 (0x8422 bytes) 000525 1788588814.833930 trunc from 0x040000 to 0x028df7 (0x17208 bytes) 000526 1788588814.890631 read 0x01f181 thru 0x0271a7 (0x8027 bytes) 000527 1788588814.891456 mapread 0x0188f4 thru 0x01df29 (0x5636 bytes) 000528 1788588814.893664 mapread 0x020547 thru 0x028df7 (0x88b1 bytes) 000530 1788588814.896706 trunc from 0x028df8 to 0x01b3cf (0xda28 bytes) 000531 1788588814.928785 punch from 0x0109fd to 0x01b3cf (0xa9d3 bytes) 000532 1788588814.940975 MIRROR_RESYNC 8 mirrors 000534 1788588817.180044 trunc from 0x01b3d0 to 0x0203e8 (0x5019 bytes) 000535 1788588817.346240 write 0x017395 thru 0x01f48e (0x80fa bytes) 000536 1788588817.361231 trunc from 0x0203e9 to 0x01f62f (0xdb9 bytes) 000537 1788588817.394024 punch from 0x00f09d to 0x01e40e (0xf372 bytes) 000538 1788588817.415313 mapwrite 0x03e288 thru 0x03ffff (0x1d78 bytes) 000539 1788588817.582308 mapwrite 0x034939 thru 0x03b48f (0x6b57 bytes) 000540 1788588817.704264 punch from 0x0209e9 to 0x024ccf (0x42e7 bytes) 000541 1788588817.726056 MIRROR_RESYNC 8 mirrors 000542 1788588819.641756 punch from 0x03c1a3 to 0x03ffff (0x3e5d bytes) 000543 1788588819.768627 mapread 0x01632c thru 0x0258db (0xf5b0 bytes) 000545 1788588819.832500 mapwrite 0x0148e1 thru 0x01b5f7 (0x6d17 bytes) 000547 1788588819.914503 punch from 0x02e670 to 0x03bfc6 (0xd957 bytes) 000548 1788588819.925233 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x53c62d02 (stale) Mirror 7: 0x53c62d02 (stale) Mirror 14: 0x53c62d02 (stale) Mirror 16: 0x53c62d02 (stale) Mirror 18: 0x53c62d02 (stale) Mirror 19: 0x53c62d02 (stale) Mirror 20: 0x53c62d02 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x16cff240 Mirror 21: 0x16cff240 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000549 1788588821.360578 mapwrite 0x0138bc thru 0x014021 (0x766 bytes) 000551 1788588821.619621 mapwrite 0x007161 thru 0x013782 (0xc622 bytes) 000552 1788588821.720114 mapread 0x03704c thru 0x03ffff (0x8fb4 bytes) 000553 1788588821.737013 MIRROR_RESYNC 9 mirrors 000555 1788588823.319484 read 0x024c92 thru 0x0343e7 (0xf756 bytes) 000556 1788588823.328601 punch from 0x012a9e to 0x01ed75 (0xc2d8 bytes) 000558 1788588823.416659 punch from 0x008300 to 0x0093c3 (0x10c4 bytes) 000559 1788588823.423277 trunc from 0x040000 to 0x021829 (0x1e7d6 bytes) 000560 1788588823.448707 write 0x038aa1 thru 0x03ffff (0x755f bytes) 000561 1788588823.451087 write 0x027bb0 thru 0x02c266 (0x46b7 bytes) 000562 1788588823.461651 write 0x01f54a thru 0x028343 (0x8dfa bytes) 000563 1788588823.468845 punch from 0x00c852 to 0x017926 (0xb0d5 bytes) 000564 1788588823.486269 punch from 0x01c5c2 to 0x01ef70 (0x29af bytes) 000565 1788588823.491255 punch from 0x034769 to 0x03dc7e (0x9516 bytes) 000567 1788588823.496848 trunc from 0x040000 to 0x012560 (0x2da9f bytes) 000569 1788588823.530792 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12561): Mirror 5: 0x4863709 (stale) Mirror 14: 0x4863709 (stale) Mirror 16: 0x4863709 (stale) Mirror 18: 0x4863709 (stale) Mirror 19: 0x4863709 (stale) Mirror 20: 0x4863709 (stale) Mirror 21: 0x4863709 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12561): Mirror 5: 0x4863709 (stale) Mirror 14: 0x4863709 (stale) Mirror 16: 0x4863709 (stale) Mirror 18: 0x4863709 (stale) Mirror 19: 0x4863709 (stale) Mirror 20: 0x4863709 (stale) Mirror 21: 0x4863709 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12561: skipped 000570 1788588824.212165 mapwrite 0x036c1f thru 0x03ffff (0x93e1 bytes) 000572 1788588824.353192 mapwrite 0x022134 thru 0x022daf (0xc7c bytes) 000573 1788588824.482673 mapread 0x023ccd thru 0x02aade (0x6e12 bytes) 000576 1788588824.487555 write 0x00c462 thru 0x00f016 (0x2bb5 bytes) 000577 1788588824.506514 write 0x0138a7 thru 0x01b484 (0x7bde bytes) 000578 1788588824.519255 mapread 0x03a4c1 thru 0x03ffff (0x5b3f bytes) 000580 1788588824.522652 mapread 0x019333 thru 0x01b2e7 (0x1fb5 bytes) 000581 1788588824.534201 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xc136740e (stale) Mirror 14: 0xc136740e (stale) Mirror 16: 0xc136740e (stale) Mirror 19: 0xc136740e (stale) Mirror 20: 0xc136740e (stale) Mirror 21: 0xc136740e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0xc136740e (stale) Mirror 14: 0xc136740e (stale) Mirror 16: 0xc136740e (stale) Mirror 19: 0xc136740e (stale) Mirror 20: 0xc136740e (stale) Mirror 21: 0xc136740e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000582 1788588825.328915 trunc from 0x040000 to 0x0007f5 (0x3f80a bytes) 000583 1788588825.364079 read 0x0005d7 thru 0x0007f5 (0x21f bytes) 000585 1788588825.378491 mapwrite 0x01d057 thru 0x0232fc (0x62a6 bytes) 000590 1788588825.514224 write 0x00f2b0 thru 0x015bdf (0x6930 bytes) 000591 1788588825.536145 mapwrite 0x027ee1 thru 0x032251 (0xa371 bytes) 000592 1788588825.651201 write 0x00a8bf thru 0x014629 (0x9d6b bytes) 000593 1788588825.668578 punch from 0x002ed3 to 0x006564 (0x3692 bytes) 000594 1788588825.693799 punch from 0x028ba5 to 0x032251 (0x96ad bytes) 000595 1788588825.713267 write 0x00fde8 thru 0x01687b (0x6a94 bytes) 000596 1788588825.724263 read 0x02e16e thru 0x032251 (0x40e4 bytes) 000597 1788588825.755351 trunc from 0x032252 to 0x037c02 (0x59b1 bytes) 000599 1788588825.782915 punch from 0x025cb7 to 0x027657 (0x19a1 bytes) 000601 1788588825.812013 write 0x038907 thru 0x03e80e (0x5f08 bytes) 000602 1788588825.813210 read 0x00d056 thru 0x011101 (0x40ac bytes) 000603 1788588825.824091 trunc from 0x03e80f to 0x000a71 (0x3dd9d bytes) 000604 1788588825.856151 mapread 0x0003e5 thru 0x000a71 (0x68d bytes) 000606 1788588825.868573 read 0x000a57 thru 0x000a71 (0x1b bytes) 000612 1788588825.877228 mapread 0x0003ce thru 0x000a71 (0x6a4 bytes) 000613 1788588825.879468 mapwrite 0x030789 thru 0x03a9e3 (0xa25b bytes) 000614 1788588826.101550 write 0x029c30 thru 0x02ca58 (0x2e29 bytes) 000616 1788588826.112310 write 0x015433 thru 0x0231e0 (0xddae bytes) 000617 1788588826.138048 MIRROR_RESYNC 7 mirrors 000618 1788588827.839074 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) 000619 1788588828.417900 MIRROR_EXTEND to 7 mirrors 000621 1788588829.344587 MIRROR_RESYNC 7 mirrors 000622 1788588830.157736 write 0x00a875 thru 0x0166b8 (0xbe44 bytes) 000623 1788588830.330753 trunc from 0x03a9e4 to 0x00ca21 (0x2dfc2 bytes) 000624 1788588830.365486 mapwrite 0x034ddf thru 0x03d872 (0x8a94 bytes) 000625 1788588830.518810 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3d873): Mirror 5: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d873): Mirror 5: 0x905840d (stale) Mirror 14: 0x905840d (stale) Mirror 16: 0x905840d (stale) Mirror 19: 0x905840d (stale) Mirror 20: 0x905840d (stale) Mirror 21: 0x905840d (stale) CRC-32 checksum value for chunk [0, 0x3d873): Mirror 1: 0xaa4ed5b9 Mirror 22: 0xaa4ed5b9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d873: skipped 000627 1788588831.772691 mapread 0x02761f thru 0x032d89 (0xb76b bytes) 000628 1788588831.839061 mapread 0x02bd72 thru 0x03bca6 (0xff35 bytes) 000629 1788588831.862119 mapread 0x00f8bc thru 0x01374a (0x3e8f bytes) 000630 1788588831.885374 punch from 0x03c74d to 0x03d872 (0x1126 bytes) 000631 1788588831.981095 read 0x0137a7 thru 0x014198 (0x9f2 bytes) 000632 1788588831.988253 trunc from 0x03d873 to 0x00116e (0x3c704 bytes) 000633 1788588832.016905 punch from 0x0006ee to 0x00116e (0xa81 bytes) 000635 1788588832.026083 punch from 0x000ade to 0x00116e (0x691 bytes) 000636 1788588832.037405 mapwrite 0x021ef9 thru 0x0267d9 (0x48e1 bytes) 000638 1788588832.169460 mapwrite 0x02113c thru 0x029fb5 (0x8e7a bytes) 000639 1788588832.366346 punch from 0x027699 to 0x029fb5 (0x291d bytes) 000640 1788588832.372911 write 0x01e291 thru 0x02ba0a (0xd77a bytes) 000641 1788588832.374798 read 0x00cd1a thru 0x013405 (0x66ec bytes) 000643 1788588832.384725 punch from 0x01e906 to 0x029b70 (0xb26b bytes) 000644 1788588832.400541 read 0x014743 thru 0x01f54c (0xae0a bytes) 000645 1788588832.408415 read 0x01b19e thru 0x01b78e (0x5f1 bytes) 000646 1788588832.413234 MIRROR_SPLIT mirror 6 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ba0b): Mirror 5: 0xaf884f9e (stale) Mirror 14: 0xaf884f9e (stale) Mirror 16: 0xaf884f9e (stale) Mirror 19: 0xaf884f9e (stale) Mirror 20: 0xaf884f9e (stale) Mirror 22: 0xb3b47617 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ba0b): Mirror 5: 0xaf884f9e (stale) Mirror 14: 0xaf884f9e (stale) Mirror 16: 0xaf884f9e (stale) Mirror 19: 0xaf884f9e (stale) Mirror 20: 0xaf884f9e (stale) Mirror 22: 0xb3b47617 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ba0b: skipped 000647 1788588833.049326 read 0x009f26 thru 0x00fbd8 (0x5cb3 bytes) 000650 1788588833.065450 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2ba0b): Mirror 5: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ba0b): Mirror 5: 0xaf884f9e (stale) Mirror 14: 0xaf884f9e (stale) Mirror 16: 0xaf884f9e (stale) Mirror 19: 0xaf884f9e (stale) Mirror 20: 0xaf884f9e (stale) Mirror 22: 0xb3b47617 (stale) CRC-32 checksum value for chunk [0, 0x2ba0b): Mirror 1: 0x5ab07936 Mirror 23: 0x5ab07936 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ba0b: skipped 000651 1788588834.224534 mapread 0x024761 thru 0x02ba0a (0x72aa bytes) 000652 1788588834.275671 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2ba0b): Mirror 5: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ba0b): Mirror 5: 0xaf884f9e (stale) Mirror 14: 0xaf884f9e (stale) Mirror 16: 0xaf884f9e (stale) Mirror 19: 0xaf884f9e (stale) Mirror 20: 0xaf884f9e (stale) Mirror 22: 0xb3b47617 (stale) CRC-32 checksum value for chunk [0, 0x2ba0b): Mirror 1: 0x5ab07936 Mirror 23: 0x5ab07936 Mirror 24: 0x5ab07936 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ba0b: skipped 000653 1788588835.737564 punch from 0x001048 to 0x00e059 (0xd012 bytes) 000654 1788588835.866653 mapread 0x003b06 thru 0x00f938 (0xbe33 bytes) 000657 1788588835.913735 read 0x0291e1 thru 0x02ba0a (0x282a bytes) 000658 1788588835.919170 read 0x02095a thru 0x023ab6 (0x315d bytes) 000659 1788588835.929314 punch from 0x003d56 to 0x01054f (0xc7fa bytes) 000660 1788588835.933433 trunc from 0x02ba0b to 0x03a31a (0xe910 bytes) 000661 1788588835.969241 write 0x03194f thru 0x036e34 (0x54e6 bytes) 000662 1788588836.024263 write 0x02ab87 thru 0x031b17 (0x6f91 bytes) 000663 1788588836.027578 trunc from 0x03a31b to 0x031c34 (0x86e6 bytes) 000665 1788588836.096022 mapread 0x00a061 thru 0x00aa08 (0x9a8 bytes) 000666 1788588836.117413 punch from 0x021e26 to 0x0221f2 (0x3cd bytes) 000667 1788588836.143473 mapwrite 0x007c91 thru 0x016e4d (0xf1bd bytes) 000669 1788588836.390056 mapwrite 0x016aca thru 0x01bd93 (0x52ca bytes) 000670 1788588836.503404 write 0x03fed7 thru 0x03ffff (0x129 bytes) 000671 1788588836.504720 trunc from 0x040000 to 0x032604 (0xd9fb bytes) 000672 1788588836.531743 read 0x021d2e thru 0x02cb6f (0xae42 bytes) 000673 1788588836.541869 trunc from 0x032605 to 0x0238f0 (0xed14 bytes) 000674 1788588836.570006 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x238f1): Mirror 5: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x238f1): Mirror 5: 0x59993d33 (stale) Mirror 14: 0x59993d33 (stale) Mirror 16: 0x59993d33 (stale) Mirror 19: 0x59993d33 (stale) Mirror 20: 0x59993d33 (stale) Mirror 22: 0x4756369c (stale) Mirror 23: 0x521f1a90 (stale) Mirror 24: 0x521f1a90 (stale) CRC-32 checksum value for chunk [0, 0x238f1): Mirror 1: 0xc8b61c89 Mirror 25: 0xc8b61c89 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x238f1: skipped 000675 1788588838.472575 read 0x0009ff thru 0x009cb9 (0x92bb bytes) 000676 1788588838.514011 mapread 0x016fe9 thru 0x017a02 (0xa1a bytes) 000677 1788588838.542199 trunc from 0x0238f1 to 0x011f01 (0x119ef bytes) 000679 1788588838.633948 punch from 0x00a67d to 0x00bb50 (0x14d4 bytes) 000680 1788588838.639150 write 0x020951 thru 0x02d674 (0xcd24 bytes) 000681 1788588838.645093 punch from 0x0079f6 to 0x00c599 (0x4ba4 bytes) 000682 1788588838.724418 MIRROR_RESYNC 10 mirrors 000683 1788588841.217253 mapwrite 0x0033ed thru 0x00ff1f (0xcb33 bytes) 000684 1788588841.511247 write 0x01c27d thru 0x029d07 (0xda8b bytes) 000687 1788588841.528208 trunc from 0x02d675 to 0x00f968 (0x1dd0c bytes) 000688 1788588841.560023 punch from 0x0022f9 to 0x00e6da (0xc3e2 bytes) 000689 1788588841.574163 punch from 0x007540 to 0x00f968 (0x8429 bytes) 000692 1788588841.595911 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0xf969): Mirror 5: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xf969): Mirror 5: 0x6750a7aa (stale) Mirror 14: 0x6750a7aa (stale) Mirror 16: 0x6750a7aa (stale) Mirror 19: 0x6750a7aa (stale) Mirror 20: 0x6750a7aa (stale) Mirror 22: 0x6750a7aa (stale) Mirror 23: 0x6750a7aa (stale) Mirror 24: 0x6750a7aa (stale) Mirror 25: 0x6750a7aa (stale) CRC-32 checksum value for chunk [0, 0xf969): Mirror 1: 0x583d3aa7 Mirror 26: 0x583d3aa7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xf969: skipped 000693 1788588843.583276 read 0x00909f thru 0x00e52e (0x5490 bytes) 000695 1788588843.635115 read 0x004cab thru 0x006b27 (0x1e7d bytes) 000696 1788588843.700176 write 0x006785 thru 0x012f8d (0xc809 bytes) 000699 1788588843.879255 MIRROR_SPLIT mirror 4 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x12f8e): Mirror 5: 0xc2c875b5 (stale) Mirror 14: 0xc2c875b5 (stale) Mirror 16: 0xc2c875b5 (stale) Mirror 20: 0xc2c875b5 (stale) Mirror 22: 0xc2c875b5 (stale) Mirror 23: 0xc2c875b5 (stale) Mirror 24: 0xc2c875b5 (stale) Mirror 25: 0xc2c875b5 (stale) Mirror 26: 0x47d851ed (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x12f8e): Mirror 5: 0xc2c875b5 (stale) Mirror 14: 0xc2c875b5 (stale) Mirror 16: 0xc2c875b5 (stale) Mirror 20: 0xc2c875b5 (stale) Mirror 22: 0xc2c875b5 (stale) Mirror 23: 0xc2c875b5 (stale) Mirror 24: 0xc2c875b5 (stale) Mirror 25: 0xc2c875b5 (stale) Mirror 26: 0x47d851ed (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x12f8e: skipped 000700 1788588844.603191 read 0x00ecac thru 0x012f8d (0x42e2 bytes) 000703 1788588844.614543 mapread 0x0091a2 thru 0x00ea16 (0x5875 bytes) 000705 1788588844.640923 punch from 0x011770 to 0x012f8d (0x181e bytes) 000707 1788588844.651314 write 0x00b8a0 thru 0x013edf (0x8640 bytes) 000709 1788588844.654127 punch from 0x00a44e to 0x0124ea (0x809d bytes) 000710 1788588844.688119 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x13ee0): Mirror 14: 0xea40150f (stale) Mirror 16: 0xea40150f (stale) Mirror 20: 0xea40150f (stale) Mirror 22: 0xea40150f (stale) Mirror 23: 0xea40150f (stale) Mirror 24: 0xea40150f (stale) Mirror 25: 0xea40150f (stale) Mirror 26: 0x1df16c58 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13ee0): Mirror 14: 0xea40150f (stale) Mirror 16: 0xea40150f (stale) Mirror 20: 0xea40150f (stale) Mirror 22: 0xea40150f (stale) Mirror 23: 0xea40150f (stale) Mirror 24: 0xea40150f (stale) Mirror 25: 0xea40150f (stale) Mirror 26: 0x1df16c58 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13ee0: skipped 000711 1788588845.580266 mapwrite 0x029bf0 thru 0x02b1f7 (0x1608 bytes) 000712 1788588845.717042 mapread 0x00a74a thru 0x015ab5 (0xb36c bytes) 000713 1788588845.761732 punch from 0x01d1bd to 0x02133b (0x417f bytes) 000714 1788588845.773547 write 0x0204df thru 0x02705d (0x6b7f bytes) 000715 1788588845.783532 mapread 0x02890c thru 0x02b1f7 (0x28ec bytes) 000716 1788588845.787412 write 0x031c96 thru 0x038055 (0x63c0 bytes) 000718 1788588845.795434 trunc from 0x038056 to 0x008733 (0x2f922 bytes) 000719 1788588845.834572 MIRROR_SPLIT mirror 1 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x8734): Mirror 16: 0xa4241cb1 (stale) Mirror 20: 0xa4241cb1 (stale) Mirror 22: 0xa4241cb1 (stale) Mirror 23: 0xa4241cb1 (stale) Mirror 24: 0xa4241cb1 (stale) Mirror 25: 0xa4241cb1 (stale) Mirror 26: 0xa4241cb1 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8734): Mirror 16: 0xa4241cb1 (stale) Mirror 20: 0xa4241cb1 (stale) Mirror 22: 0xa4241cb1 (stale) Mirror 23: 0xa4241cb1 (stale) Mirror 24: 0xa4241cb1 (stale) Mirror 25: 0xa4241cb1 (stale) Mirror 26: 0xa4241cb1 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8734: skipped 000720 1788588846.853639 punch from 0x001b8a to 0x008733 (0x6baa bytes) 000722 1788588846.871679 mapwrite 0x01702b thru 0x01caf9 (0x5acf bytes) 000723 1788588847.170229 trunc from 0x01cafa to 0x037022 (0x1a529 bytes) 000724 1788588847.225165 write 0x036836 thru 0x039f1b (0x36e6 bytes) 000726 1788588847.245387 trunc from 0x039f1c to 0x0253c5 (0x14b56 bytes) 000727 1788588847.308952 MIRROR_RESYNC 8 mirrors 000728 1788588850.681642 punch from 0x0151dd to 0x023dd6 (0xebfa bytes) 000729 1788588850.770628 mapread 0x008437 thru 0x011298 (0x8e62 bytes) 000730 1788588850.791723 mapwrite 0x01006d thru 0x01f079 (0xf00d bytes) 000732 1788588850.936039 write 0x0149e2 thru 0x022689 (0xdca8 bytes) 000733 1788588850.939502 read 0x012e2a thru 0x01b05d (0x8234 bytes) 000734 1788588850.941540 read 0x01a29c thru 0x0253c5 (0xb12a bytes) 000735 1788588850.963303 mapwrite 0x003f2b thru 0x00e5a5 (0xa67b bytes) 000736 1788588851.136186 mapread 0x0181a5 thru 0x01eccd (0x6b29 bytes) 000737 1788588851.137074 write 0x016ac3 thru 0x0260cf (0xf60d bytes) 000738 1788588851.151271 read 0x008d5b thru 0x00fa85 (0x6d2b bytes) 000739 1788588851.159195 mapread 0x017d87 thru 0x01f689 (0x7903 bytes) 000741 1788588851.182602 punch from 0x01faa9 to 0x0260cf (0x6627 bytes) 000742 1788588851.252197 mapwrite 0x009d39 thru 0x0192a8 (0xf570 bytes) 000743 1788588851.372055 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x260d0): Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x260d0): Mirror 16: 0x5f29378e (stale) Mirror 20: 0x5f29378e (stale) Mirror 22: 0x5f29378e (stale) Mirror 23: 0x5f29378e (stale) Mirror 24: 0x5f29378e (stale) Mirror 25: 0x5f29378e (stale) Mirror 26: 0x5f29378e (stale) CRC-32 checksum value for chunk [0, 0x260d0): Mirror 1: 0x7b1506cd Mirror 27: 0x7b1506cd lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x260d0: skipped 000745 1788588853.239909 MIRROR_RESYNC 9 mirrors 000746 1788588855.467541 read 0x00ba88 thru 0x00df08 (0x2481 bytes) 000747 1788588855.479919 mapwrite 0x010ce2 thru 0x01d1c5 (0xc4e4 bytes) 000748 1788588855.760936 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x260d0): Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x260d0): Mirror 16: 0x7b1506cd (stale) Mirror 20: 0x7b1506cd (stale) Mirror 22: 0x7b1506cd (stale) Mirror 23: 0x7b1506cd (stale) Mirror 24: 0x7b1506cd (stale) Mirror 25: 0x7b1506cd (stale) Mirror 26: 0x7b1506cd (stale) Mirror 27: 0x7b1506cd (stale) CRC-32 checksum value for chunk [0, 0x260d0): Mirror 1: 0x5d75e7af Mirror 28: 0x5d75e7af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x260d0: skipped 000751 1788588857.498608 trunc from 0x0260d0 to 0x01023f (0x15e90 bytes) 000752 1788588857.680178 write 0x003684 thru 0x003a73 (0x3f0 bytes) 000753 1788588857.689894 MIRROR_RESYNC 10 mirrors 000754 1788588859.473022 MIRROR_EXTEND to 11 mirrors 000755 1788588860.489543 MIRROR_EXTEND to 12 mirrors 000756 1788588861.993144 trunc from 0x010240 to 0x00d72d (0x2b12 bytes) 000757 1788588862.238672 mapread 0x002b08 thru 0x007db6 (0x52af bytes) 000758 1788588862.284730 MIRROR_RESYNC 12 mirrors 000760 1788588864.735086 punch from 0x0009ed to 0x00c699 (0xbcad bytes) 000761 1788588864.914843 write 0x038373 thru 0x03ffff (0x7c8d bytes) 000762 1788588864.920939 mapwrite 0x00e78a thru 0x01b1f7 (0xca6e bytes) 000764 1788588865.135980 write 0x023428 thru 0x0237a1 (0x37a bytes) 000765 1788588865.160690 mapread 0x00153c thru 0x00a6a7 (0x916c bytes) 000767 1788588865.186652 read 0x03b992 thru 0x03f5ad (0x3c1c bytes) 000768 1788588865.191361 mapwrite 0x017425 thru 0x01b94e (0x452a bytes) 000769 1788588865.281186 trunc from 0x040000 to 0x035463 (0xab9c bytes) 000771 1788588865.314800 trunc from 0x035464 to 0x036c79 (0x1816 bytes) 000773 1788588865.333320 mapwrite 0x034550 thru 0x03ffff (0xbab0 bytes) 000774 1788588865.424293 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x4bf08603 (stale) Mirror 20: 0x4bf08603 (stale) Mirror 22: 0x4bf08603 (stale) Mirror 23: 0x4bf08603 (stale) Mirror 24: 0x4bf08603 (stale) Mirror 25: 0x4bf08603 (stale) Mirror 26: 0x4bf08603 (stale) Mirror 27: 0x4bf08603 (stale) Mirror 28: 0x4bf08603 (stale) Mirror 29: 0x4bf08603 (stale) Mirror 30: 0x4bf08603 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe9cc0525 Mirror 31: 0xe9cc0525 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000777 1788588867.463252 write 0x027f55 thru 0x02d489 (0x5535 bytes) 000779 1788588867.573978 mapwrite 0x011c46 thru 0x01d3dd (0xb798 bytes) 000780 1788588867.779990 read 0x0319a3 thru 0x03ffff (0xe65d bytes) 000781 1788588867.809343 punch from 0x037b45 to 0x03b0c4 (0x3580 bytes) 000783 1788588867.833068 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 20: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x4bf08603 (stale) Mirror 20: 0x4bf08603 (stale) Mirror 22: 0x4bf08603 (stale) Mirror 23: 0x4bf08603 (stale) Mirror 24: 0x4bf08603 (stale) Mirror 25: 0x4bf08603 (stale) Mirror 26: 0x4bf08603 (stale) Mirror 27: 0x4bf08603 (stale) Mirror 28: 0x4bf08603 (stale) Mirror 29: 0x4bf08603 (stale) Mirror 30: 0x4bf08603 (stale) Mirror 31: 0xe9cc0525 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6d620270 Mirror 32: 0x6d620270 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000784 1788588870.016799 MIRROR_SPLIT mirror 2 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x4bf08603 (stale) Mirror 22: 0x4bf08603 (stale) Mirror 23: 0x4bf08603 (stale) Mirror 24: 0x4bf08603 (stale) Mirror 25: 0x4bf08603 (stale) Mirror 26: 0x4bf08603 (stale) Mirror 27: 0x4bf08603 (stale) Mirror 28: 0x4bf08603 (stale) Mirror 29: 0x4bf08603 (stale) Mirror 30: 0x4bf08603 (stale) Mirror 31: 0xe9cc0525 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x4bf08603 (stale) Mirror 22: 0x4bf08603 (stale) Mirror 23: 0x4bf08603 (stale) Mirror 24: 0x4bf08603 (stale) Mirror 25: 0x4bf08603 (stale) Mirror 26: 0x4bf08603 (stale) Mirror 27: 0x4bf08603 (stale) Mirror 28: 0x4bf08603 (stale) Mirror 29: 0x4bf08603 (stale) Mirror 30: 0x4bf08603 (stale) Mirror 31: 0xe9cc0525 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6d620270 Mirror 32: 0x6d620270 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000785 1788588871.680704 write 0x02bda4 thru 0x037b5a (0xbdb7 bytes) 000787 1788588871.858883 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x4bf08603 (stale) Mirror 22: 0x4bf08603 (stale) Mirror 23: 0x4bf08603 (stale) Mirror 24: 0x4bf08603 (stale) Mirror 25: 0x4bf08603 (stale) Mirror 26: 0x4bf08603 (stale) Mirror 27: 0x4bf08603 (stale) Mirror 28: 0x4bf08603 (stale) Mirror 29: 0x4bf08603 (stale) Mirror 30: 0x4bf08603 (stale) Mirror 31: 0xe9cc0525 (stale) Mirror 32: 0x6d620270 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd9df89b1 Mirror 33: 0xd9df89b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000792 1788588873.271380 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 16: 0x4bf08603 (stale) Mirror 22: 0x4bf08603 (stale) Mirror 23: 0x4bf08603 (stale) Mirror 24: 0x4bf08603 (stale) Mirror 25: 0x4bf08603 (stale) Mirror 26: 0x4bf08603 (stale) Mirror 27: 0x4bf08603 (stale) Mirror 28: 0x4bf08603 (stale) Mirror 29: 0x4bf08603 (stale) Mirror 30: 0x4bf08603 (stale) Mirror 31: 0xe9cc0525 (stale) Mirror 32: 0x6d620270 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xd9df89b1 Mirror 33: 0xd9df89b1 Mirror 34: 0xd9df89b1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000793 1788588874.898988 punch from 0x03c903 to 0x03ffff (0x36fd bytes) 000794 1788588875.093553 write 0x01e187 thru 0x0207eb (0x2665 bytes) 000796 1788588875.109593 trunc from 0x040000 to 0x03d990 (0x266f bytes) 000797 1788588875.144165 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3d991): Mirror 16: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d991): Mirror 16: 0x834ae0f9 (stale) Mirror 22: 0x834ae0f9 (stale) Mirror 23: 0x834ae0f9 (stale) Mirror 24: 0x834ae0f9 (stale) Mirror 25: 0x834ae0f9 (stale) Mirror 26: 0x834ae0f9 (stale) Mirror 27: 0x834ae0f9 (stale) Mirror 28: 0x834ae0f9 (stale) Mirror 29: 0x834ae0f9 (stale) Mirror 30: 0x834ae0f9 (stale) Mirror 31: 0xe71c6d70 (stale) Mirror 32: 0x76c64251 (stale) Mirror 33: 0xd17edfe9 (stale) Mirror 34: 0xd17edfe9 (stale) CRC-32 checksum value for chunk [0, 0x3d991): Mirror 1: 0xa2b0aa76 Mirror 35: 0xa2b0aa76 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d991: skipped 000798 1788588876.900728 punch from 0x02641e to 0x0269e5 (0x5c8 bytes) 000800 1788588877.132115 write 0x0278b7 thru 0x027dd5 (0x51f bytes) 000802 1788588877.145901 mapread 0x030aba thru 0x038d14 (0x825b bytes) 000805 1788588877.193097 mapwrite 0x032958 thru 0x03ff95 (0xd63e bytes) 000806 1788588877.342273 write 0x0184c5 thru 0x0270df (0xec1b bytes) 000807 1788588877.354860 punch from 0x0169cd to 0x017a9a (0x10ce bytes) 000808 1788588877.379486 punch from 0x036d0e to 0x03a44c (0x373f bytes) 000810 1788588877.386760 trunc from 0x03ff96 to 0x0110a1 (0x2eef4 bytes) 000811 1788588877.430240 mapwrite 0x00f5f0 thru 0x019a3d (0xa44e bytes) 000813 1788588877.591108 write 0x028c69 thru 0x02ebe1 (0x5f79 bytes) 000814 1788588877.592895 mapread 0x000969 thru 0x00795b (0x6ff3 bytes) 000815 1788588877.604885 mapwrite 0x01648e thru 0x02574a (0xf2bd bytes) 000816 1788588877.680566 read 0x00aa54 thru 0x018f5a (0xe507 bytes) 000817 1788588877.681578 punch from 0x014797 to 0x023ce6 (0xf550 bytes) 000819 1788588877.690912 mapwrite 0x018b05 thru 0x01f7ac (0x6ca8 bytes) 000820 1788588877.804983 mapwrite 0x01a97d thru 0x01d7d1 (0x2e55 bytes) 000825 1788588877.868117 write 0x007a78 thru 0x00eb49 (0x70d2 bytes) 000827 1788588877.873697 mapwrite 0x0089ff thru 0x00f1dd (0x67df bytes) 000828 1788588878.006152 punch from 0x00e852 to 0x01b191 (0xc940 bytes) 000829 1788588878.011882 write 0x017497 thru 0x01e430 (0x6f9a bytes) 000831 1788588878.019917 trunc from 0x02ebe2 to 0x013505 (0x1b6dc bytes) 000832 1788588878.039022 trunc from 0x013506 to 0x006295 (0xd270 bytes) 000833 1788588878.054539 write 0x01b264 thru 0x02a8ac (0xf649 bytes) 000836 1788588878.060628 mapwrite 0x01485f thru 0x0224a3 (0xdc45 bytes) 000837 1788588878.113879 trunc from 0x02a8ad to 0x034c5f (0xa3b3 bytes) 000838 1788588878.132658 mapread 0x013649 thru 0x01b707 (0x80bf bytes) 000840 1788588878.135331 punch from 0x022798 to 0x0296e6 (0x6f4f bytes) 000841 1788588878.148955 write 0x01f8b8 thru 0x0244f6 (0x4c3f bytes) 000843 1788588878.159089 mapread 0x016697 thru 0x01a01a (0x3984 bytes) 000844 1788588878.165182 trunc from 0x034c60 to 0x02d69c (0x75c3 bytes) 000848 1788588878.236552 write 0x032ea2 thru 0x03d752 (0xa8b1 bytes) 000849 1788588878.242048 mapwrite 0x01c672 thru 0x01d10e (0xa9d bytes) 000852 1788588878.346609 punch from 0x00ee9d to 0x01b11a (0xc27e bytes) 000853 1788588878.354634 read 0x016c68 thru 0x025a52 (0xedeb bytes) 000854 1788588878.380040 mapwrite 0x00617a thru 0x008859 (0x26e0 bytes) 000855 1788588878.485981 punch from 0x01b01b to 0x01e65c (0x3642 bytes) 000857 1788588878.505206 trunc from 0x03d753 to 0x00cf60 (0x307f2 bytes) 000858 1788588878.536324 mapwrite 0x032adb thru 0x03cd2a (0xa250 bytes) 000859 1788588878.761750 write 0x032997 thru 0x0331a8 (0x812 bytes) 000860 1788588878.762738 mapread 0x015a58 thru 0x01f6ca (0x9c73 bytes) 000862 1788588878.852846 trunc from 0x03cd2b to 0x001bf4 (0x3b136 bytes) 000863 1788588878.908058 MIRROR_RESYNC 16 mirrors 000864 1788588881.786147 mapwrite 0x018908 thru 0x01c7c0 (0x3eb9 bytes) 000865 1788588882.205931 write 0x02a5a2 thru 0x02ded7 (0x3936 bytes) 000866 1788588882.223987 MIRROR_RESYNC 16 mirrors 000867 1788588887.605479 trunc from 0x02ded8 to 0x027690 (0x6847 bytes) 000870 1788588887.764962 mapread 0x0187a7 thru 0x027690 (0xeeea bytes) 000871 1788588887.859500 mapwrite 0x02d873 thru 0x034f5a (0x76e8 bytes) 000872 1788588888.044967 mapread 0x004762 thru 0x013a63 (0xf302 bytes) 000873 1788588888.104214 mapwrite 0x021f22 thru 0x0284ff (0x65de bytes) 000874 1788588888.172973 trunc from 0x034f5b to 0x039ea6 (0x4f4c bytes) 000875 1788588888.193197 write 0x01f38c thru 0x024d4f (0x59c4 bytes) 000876 1788588888.194316 mapread 0x0019a8 thru 0x00bf27 (0xa580 bytes) 000878 1788588888.199988 punch from 0x036788 to 0x039ea6 (0x371f bytes) 000879 1788588888.227272 mapwrite 0x033c6a thru 0x03a2ad (0x6644 bytes) 000881 1788588888.367218 punch from 0x025d8c to 0x033d7c (0xdff1 bytes) 000882 1788588888.390019 punch from 0x00e2e7 to 0x010ba3 (0x28bd bytes) 000883 1788588888.404477 mapread 0x0098d1 thru 0x0135a3 (0x9cd3 bytes) 000884 1788588888.432800 MIRROR_SPLIT mirror 15 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a2ae): Mirror 16: 0x4ca777c5 (stale) Mirror 22: 0x4ca777c5 (stale) Mirror 23: 0x4ca777c5 (stale) Mirror 24: 0x4ca777c5 (stale) Mirror 25: 0x4ca777c5 (stale) Mirror 26: 0x4ca777c5 (stale) Mirror 27: 0x4ca777c5 (stale) Mirror 28: 0x4ca777c5 (stale) Mirror 29: 0x4ca777c5 (stale) Mirror 30: 0x4ca777c5 (stale) Mirror 31: 0x4ca777c5 (stale) Mirror 32: 0x4ca777c5 (stale) Mirror 33: 0x4ca777c5 (stale) Mirror 34: 0x4ca777c5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a2ae): Mirror 16: 0x4ca777c5 (stale) Mirror 22: 0x4ca777c5 (stale) Mirror 23: 0x4ca777c5 (stale) Mirror 24: 0x4ca777c5 (stale) Mirror 25: 0x4ca777c5 (stale) Mirror 26: 0x4ca777c5 (stale) Mirror 27: 0x4ca777c5 (stale) Mirror 28: 0x4ca777c5 (stale) Mirror 29: 0x4ca777c5 (stale) Mirror 30: 0x4ca777c5 (stale) Mirror 31: 0x4ca777c5 (stale) Mirror 32: 0x4ca777c5 (stale) Mirror 33: 0x4ca777c5 (stale) Mirror 34: 0x4ca777c5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a2ae: skipped 000886 1788588889.935210 MIRROR_RESYNC 15 mirrors 000887 1788588893.426795 punch from 0x015b74 to 0x0211ff (0xb68c bytes) 000888 1788588893.522154 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3a2ae): Mirror 16: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a2ae): Mirror 16: 0x7eccdbe4 (stale) Mirror 22: 0x7eccdbe4 (stale) Mirror 23: 0x7eccdbe4 (stale) Mirror 24: 0x7eccdbe4 (stale) Mirror 25: 0x7eccdbe4 (stale) Mirror 26: 0x7eccdbe4 (stale) Mirror 27: 0x7eccdbe4 (stale) Mirror 28: 0x7eccdbe4 (stale) Mirror 29: 0x7eccdbe4 (stale) Mirror 30: 0x7eccdbe4 (stale) Mirror 31: 0x7eccdbe4 (stale) Mirror 32: 0x7eccdbe4 (stale) Mirror 33: 0x7eccdbe4 (stale) Mirror 34: 0x7eccdbe4 (stale) CRC-32 checksum value for chunk [0, 0x3a2ae): Mirror 1: 0x20956c5d Mirror 35: 0x20956c5d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a2ae: skipped 000889 1788588895.211706 trunc from 0x03a2ae to 0x013ce0 (0x265cd bytes) 000890 1788588895.384796 read 0x00b531 thru 0x00c605 (0x10d5 bytes) 000893 1788588895.405294 write 0x008746 thru 0x011591 (0x8e4c bytes) 000894 1788588895.426433 mapwrite 0x00e85e thru 0x01afcd (0xc770 bytes) 000896 1788588895.602971 write 0x03a242 thru 0x03dda4 (0x3b63 bytes) 000897 1788588895.606554 punch from 0x008c76 to 0x00c023 (0x33ae bytes) 000899 1788588895.641061 write 0x0083ff thru 0x00f48d (0x708f bytes) 000901 1788588895.680546 mapread 0x014d43 thru 0x019c33 (0x4ef1 bytes) 000902 1788588895.684471 mapwrite 0x011429 thru 0x017249 (0x5e21 bytes) 000903 1788588895.804174 punch from 0x036983 to 0x037f40 (0x15be bytes) 000904 1788588895.822072 read 0x025cbe thru 0x02e239 (0x857c bytes) 000906 1788588895.847598 write 0x017cfd thru 0x01ceb8 (0x51bc bytes) 000907 1788588895.850006 mapwrite 0x03de13 thru 0x03ffff (0x21ed bytes) 000908 1788588896.024097 mapread 0x00cd17 thru 0x01762a (0xa914 bytes) 000909 1788588896.030474 punch from 0x033c98 to 0x03ffff (0xc368 bytes) 000910 1788588896.056647 mapwrite 0x03bd4f thru 0x03ffff (0x42b1 bytes) 000911 1788588896.242168 mapread 0x0118e7 thru 0x0212a6 (0xf9c0 bytes) 000912 1788588896.247275 mapwrite 0x0057fc thru 0x00db35 (0x833a bytes) 000914 1788588896.316341 mapread 0x019e88 thru 0x01db7d (0x3cf6 bytes) 000915 1788588896.317536 write 0x03b581 thru 0x03e7e1 (0x3261 bytes) 000918 1788588896.321521 mapread 0x027ecb thru 0x035b69 (0xdc9f bytes) 000919 1788588896.375176 MIRROR_RESYNC 16 mirrors 000920 1788588899.831170 read 0x035b3b thru 0x03f564 (0x9a2a bytes) 000921 1788588899.842695 trunc from 0x040000 to 0x006df7 (0x39208 bytes) 000924 1788588900.042056 MIRROR_RESYNC 16 mirrors 000925 1788588902.807485 mapread 0x001462 thru 0x001568 (0x107 bytes) 000926 1788588902.817196 mapwrite 0x02ca28 thru 0x03a9f9 (0xdfd2 bytes) 000929 1788588903.167162 mapwrite 0x019763 thru 0x01a854 (0x10f2 bytes) 000930 1788588903.279598 mapwrite 0x02c555 thru 0x035a46 (0x94f2 bytes) 000931 1788588903.400707 punch from 0x027eb5 to 0x02c54d (0x4699 bytes) 000933 1788588903.414413 trunc from 0x03a9fa to 0x0253a5 (0x15654 bytes) 000934 1788588903.447332 trunc from 0x0253a6 to 0x02db53 (0x87ae bytes) 000935 1788588903.476226 mapwrite 0x0187ce thru 0x01f2a8 (0x6adb bytes) 000937 1788588903.557617 mapwrite 0x01f7dd thru 0x0252dc (0x5b00 bytes) 000941 1788588903.643885 trunc from 0x02db54 to 0x00ad4e (0x22e05 bytes) 000942 1788588903.670792 read 0x003b11 thru 0x00ad4e (0x723e bytes) 000944 1788588903.693740 write 0x0170c7 thru 0x01d4e3 (0x641d bytes) 000946 1788588903.698342 mapwrite 0x03d9bf thru 0x03ffff (0x2641 bytes) 000947 1788588903.837946 write 0x021018 thru 0x02f286 (0xe26f bytes) 000948 1788588903.851518 mapread 0x02563c thru 0x0278a0 (0x2265 bytes) 000950 1788588903.853551 trunc from 0x040000 to 0x02ccfc (0x13303 bytes) 000951 1788588903.883310 read 0x0003a6 thru 0x0030ba (0x2d15 bytes) 000952 1788588903.894724 trunc from 0x02ccfd to 0x033677 (0x697b bytes) 000953 1788588903.910256 read 0x0234d2 thru 0x02c36b (0x8e9a bytes) 000955 1788588903.911457 mapread 0x01a3f4 thru 0x020172 (0x5d7f bytes) 000957 1788588903.915875 read 0x00f3a3 thru 0x01943c (0xa09a bytes) 000958 1788588903.917579 mapwrite 0x016c08 thru 0x024823 (0xdc1c bytes) 000959 1788588904.001689 mapwrite 0x020069 thru 0x0227a9 (0x2741 bytes) 000961 1788588904.066939 MIRROR_RESYNC 16 mirrors 000963 1788588907.589009 read 0x026b43 thru 0x032b3b (0xbff9 bytes) 000966 1788588907.607171 read 0x01fee1 thru 0x02fb98 (0xfcb8 bytes) 000967 1788588907.626355 mapwrite 0x03a86f thru 0x03ffff (0x5791 bytes) 000968 1788588908.035436 read 0x02a931 thru 0x0385b8 (0xdc88 bytes) 000969 1788588908.068642 read 0x031e2f thru 0x03ba60 (0x9c32 bytes) 000971 1788588908.079642 read 0x011d3d thru 0x01eb3c (0xce00 bytes) 000972 1788588908.086008 write 0x007ac5 thru 0x017135 (0xf671 bytes) 000973 1788588908.093224 write 0x00eb6e thru 0x01446a (0x58fd bytes) 000975 1788588908.126178 MIRROR_SPLIT mirror 1 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x77b81c29 (stale) Mirror 23: 0x77b81c29 (stale) Mirror 24: 0x77b81c29 (stale) Mirror 25: 0x77b81c29 (stale) Mirror 26: 0x77b81c29 (stale) Mirror 27: 0x77b81c29 (stale) Mirror 28: 0x77b81c29 (stale) Mirror 29: 0x77b81c29 (stale) Mirror 30: 0x77b81c29 (stale) Mirror 31: 0x77b81c29 (stale) Mirror 32: 0x77b81c29 (stale) Mirror 33: 0x77b81c29 (stale) Mirror 34: 0x77b81c29 (stale) Mirror 35: 0x77b81c29 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x77b81c29 (stale) Mirror 23: 0x77b81c29 (stale) Mirror 24: 0x77b81c29 (stale) Mirror 25: 0x77b81c29 (stale) Mirror 26: 0x77b81c29 (stale) Mirror 27: 0x77b81c29 (stale) Mirror 28: 0x77b81c29 (stale) Mirror 29: 0x77b81c29 (stale) Mirror 30: 0x77b81c29 (stale) Mirror 31: 0x77b81c29 (stale) Mirror 32: 0x77b81c29 (stale) Mirror 33: 0x77b81c29 (stale) Mirror 34: 0x77b81c29 (stale) Mirror 35: 0x77b81c29 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000976 1788588909.650720 MIRROR_RESYNC 15 mirrors 000977 1788588912.509601 write 0x01d34f thru 0x02b70f (0xe3c1 bytes) 000978 1788588912.625397 mapread 0x0086a3 thru 0x00a1dd (0x1b3b bytes) 000979 1788588912.646421 mapwrite 0x0298da thru 0x029cf0 (0x417 bytes) 000981 1788588912.714115 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) Mirror 34: (stale - read error) Mirror 35: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0x7b5f1d1d (stale) Mirror 23: 0x7b5f1d1d (stale) Mirror 24: 0x7b5f1d1d (stale) Mirror 25: 0x7b5f1d1d (stale) Mirror 26: 0x7b5f1d1d (stale) Mirror 27: 0x7b5f1d1d (stale) Mirror 28: 0x7b5f1d1d (stale) Mirror 29: 0x7b5f1d1d (stale) Mirror 30: 0x7b5f1d1d (stale) Mirror 31: 0x7b5f1d1d (stale) Mirror 32: 0x7b5f1d1d (stale) Mirror 33: 0x7b5f1d1d (stale) Mirror 34: 0x7b5f1d1d (stale) Mirror 35: 0x7b5f1d1d (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xe8437403 Mirror 36: 0xe8437403 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000982 1788588914.405390 trunc from 0x040000 to 0x034d85 (0xb27a bytes) 000985 1788588914.562889 mapwrite 0x004074 thru 0x010eac (0xce39 bytes) 000986 1788588914.671943 read 0x01e07d thru 0x02a6a5 (0xc629 bytes) 000987 1788588914.719189 mapwrite 0x01a6a1 thru 0x01eead (0x480d bytes) 000989 1788588914.835142 read 0x0165f7 thru 0x019a43 (0x344d bytes) 000990 1788588914.836158 write 0x03139b thru 0x036dcb (0x5a31 bytes) 000994 1788588914.873065 mapwrite 0x015461 thru 0x021dec (0xc98c bytes) 000995 1788588914.980647 mapread 0x011559 thru 0x01188c (0x334 bytes) 000997 1788588915.001425 MIRROR_SPLIT mirror 11 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36dcc): Mirror 22: 0x78d87d7a (stale) Mirror 23: 0x78d87d7a (stale) Mirror 24: 0x78d87d7a (stale) Mirror 25: 0x78d87d7a (stale) Mirror 26: 0x78d87d7a (stale) Mirror 27: 0x78d87d7a (stale) Mirror 28: 0x78d87d7a (stale) Mirror 29: 0x78d87d7a (stale) Mirror 30: 0x78d87d7a (stale) Mirror 31: 0x78d87d7a (stale) Mirror 33: 0x78d87d7a (stale) Mirror 34: 0x78d87d7a (stale) Mirror 35: 0x78d87d7a (stale) Mirror 36: 0xcee51f4a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36dcc): Mirror 22: 0x78d87d7a (stale) Mirror 23: 0x78d87d7a (stale) Mirror 24: 0x78d87d7a (stale) Mirror 25: 0x78d87d7a (stale) Mirror 26: 0x78d87d7a (stale) Mirror 27: 0x78d87d7a (stale) Mirror 28: 0x78d87d7a (stale) Mirror 29: 0x78d87d7a (stale) Mirror 30: 0x78d87d7a (stale) Mirror 31: 0x78d87d7a (stale) Mirror 33: 0x78d87d7a (stale) Mirror 34: 0x78d87d7a (stale) Mirror 35: 0x78d87d7a (stale) Mirror 36: 0xcee51f4a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36dcc: skipped 000999 1788588915.967339 mapread 0x028508 thru 0x036dcb (0xe8c4 bytes) 001000 1788588916.010446 MIRROR_SPLIT mirror 9 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36dcc): Mirror 22: 0x78d87d7a (stale) Mirror 23: 0x78d87d7a (stale) Mirror 24: 0x78d87d7a (stale) Mirror 25: 0x78d87d7a (stale) Mirror 26: 0x78d87d7a (stale) Mirror 27: 0x78d87d7a (stale) Mirror 28: 0x78d87d7a (stale) Mirror 29: 0x78d87d7a (stale) Mirror 31: 0x78d87d7a (stale) Mirror 33: 0x78d87d7a (stale) Mirror 34: 0x78d87d7a (stale) Mirror 35: 0x78d87d7a (stale) Mirror 36: 0xcee51f4a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36dcc): Mirror 22: 0x78d87d7a (stale) Mirror 23: 0x78d87d7a (stale) Mirror 24: 0x78d87d7a (stale) Mirror 25: 0x78d87d7a (stale) Mirror 26: 0x78d87d7a (stale) Mirror 27: 0x78d87d7a (stale) Mirror 28: 0x78d87d7a (stale) Mirror 29: 0x78d87d7a (stale) Mirror 31: 0x78d87d7a (stale) Mirror 33: 0x78d87d7a (stale) Mirror 34: 0x78d87d7a (stale) Mirror 35: 0x78d87d7a (stale) Mirror 36: 0xcee51f4a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36dcc: skipped All operations completed A-OK! PASS 70a (197s) == sanity-flr test 71: check mirror extend parent fid ==== 02:15:23 (1788588923) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (5s) == sanity-flr test 72: check append on FLR file ========== 02:15:28 (1788588928) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x140:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.391008 s, 5.4 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x140:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x141:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183843 s, 5.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x140:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x141:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x140:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x141:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x13d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x142:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 02:15:38 (1788588938) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8837216 .. Extending file size to 6942752 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 7063232 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7355840 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8752224 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9224960 .. Waiting 59529 59530 59533 59535 59536 done 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) 192.168.204.113@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (85s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 02:17:03 (1788589023) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5558720 .. Extending file size to 1014080 .. Extending file size to 8099744 .. Extending file size to 8942048 .. failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8581728 .. Extending file size to 9089408 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8800160 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Waiting 61063 61064 61066 61068 61070 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) 192.168.204.113@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (79s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 02:18:23 (1788589103) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3072 2205440 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 2210688 2432 2206208 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 43008 7495680 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3072 2205440 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 2210688 2432 2206208 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3771392 18432 3750912 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 43008 7495680 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.204069 s, 10.3 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.351013 s, 6.0 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x6120eed3 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) PASS 200c (21s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 02:18:46 (1788589126) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0980631 s, 21.4 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 02:18:55 (1788589135) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x9f:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x154:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x155:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x157:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 02:19:04 (1788589144) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.244669 s, 8.6 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x189:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x18a:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x18b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (12s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 02:19:16 (1788589156) striped dir -i0 -c2 -H all_char /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.208124 s, 10.1 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x18a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x18b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x18c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x18a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x18b:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x18c:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x18d:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 02:19:25 (1788589165) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.460742 s, 4.6 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x18c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x18e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x18c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x18e:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x18f:0x0] } PASS 204c (37s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 02:20:02 (1788589202) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg413-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13244 s, 7.9 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x18d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x191:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x190:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x18d:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x191:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x190:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x18e:0x0] } PASS 204d (65s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 02:21:08 (1788589268) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg413-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x156:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.300793 s, 7.0 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x156:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x158:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x156:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x157:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x158:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (69s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 02:22:18 (1788589338) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg413-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg413-server: Pool lustre.test_204f created Adding targets to pool oleg413-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg413-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.31693 s, 6.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x159:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x158:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x159:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg413-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg413-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg413-server: Pool lustre.test_204f destroyed PASS 204f (86s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 02:23:44 (1788589424) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 02:23:53 (1788589433) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 02:24:02 (1788589442) oleg413-server: Pool lustre.test_206 created Adding targets to pool oleg413-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg413-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg413-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg413-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg413-server: Pool lustre.test_206 destroyed PASS 206 (32s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 02:24:34 (1788589474) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100716 s, 10.4 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849497 s, 12.3 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xa0c88a67 Mirror 3: 0xa0c88a67 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 02:24:47 (1788589487) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 02:24:50 (1788589490) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 02:24:54 (1788589494) Starting client: oleg413-client.virtnet: -o user_xattr,flock 192.168.204.113@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.129144 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xb80000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x9d0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x9d0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x9d0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1c50000 Read this (4 bytes): BBBB 192.168.204.113@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg413-client.virtnet /mnt/lustre2 (opts:) PASS 209a (42s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 02:25:36 (1788589536) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0316686 s, 259 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 02:25:44 (1788589544) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743578 s, 14.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xbc lmm_fid: [0x200000405:0xbc:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xbc lmm_fid: [0x200000405:0xbc:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x164:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611324 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-05 02:25:46.000000000 -0400 Modify: 2026-09-05 02:25:46.000000000 -0400 Change: 2026-09-05 02:25:48.000000000 -0400 Birth: 2026-09-05 02:25:46.000000000 -0400 PASS 210a (11s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 02:25:55 (1788589555) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 02:25:59 (1788589559) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.514012 s, 79.7 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0661304 s, 61.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 5 02:26 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xbe:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 5 02:26 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 02:26:08 (1788589568) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 02:26:17 (1788589577) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (13s) == sanity-flr test complete, duration 2073 sec =========== 02:26:30 (1788589590) === sanity-flr: start cleanup 02:26:32 (1788589592) === === sanity-flr: finish cleanup 02:26:42 (1788589602) ===