-----============= acceptance-small: sanity-flr ============----- Thu Jun 11 19:13:00 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg432-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 19:13:20 (1781219600) === oleg432-client.virtnet: executing check_config_client /mnt/lustre oleg432-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg432-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9e5f05077800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9e5f05077800.idle_timeout=debug disable quota as required oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 19:13:53 (1781219633) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7588] == sanity-flr test 0a: lfs mirror create with -N option == 19:13:56 (1781219636) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 19:14:13 (1781219653) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (4s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 19:14:17 (1781219657) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 19:14:21 (1781219661) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 19:14:47 (1781219687) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 19:14:51 (1781219691) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 19:14:54 (1781219694) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9e5f05077800.stats=clear osc.lustre-OST0001-osc-ffff9e5f05077800.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 19:15:03 (1781219703) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--compress|-Z] [--compress-chunk=SIZE] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 19:15:16 (1781219716) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 19:15:26 (1781219726) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 19:15:30 (1781219730) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 19:15:38 (1781219738) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (11s) == sanity-flr test 2: create components from existing files ========================================================== 19:15:49 (1781219749) PASS 2 (9s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 19:15:58 (1781219758) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 19:16:01 (1781219761) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 19:16:09 (1781219769) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 19:16:21 (1781219781) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 19:16:28 (1781219788) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 2.1478 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175896 s, 6.0 MB/s PASS 21 (12s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 19:16:40 (1781219800) 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.518194 s, 32.4 MB/s PASS 22 (9s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 19:16:49 (1781219809) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 19:17:00 (1781219820) Stopping /mnt/lustre-ost1 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e5f05077800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e5f05077800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg432-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e5f05077800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e5f05077800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg432-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (74s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 19:18:14 (1781219894) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.154385 s, 13.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e5f05077800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e5f05077800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 2097152 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg432-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (42s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 19:18:57 (1781219937) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e5f05077800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e5f05077800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg432-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e5f05077800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e5f05077800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg432-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (78s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 19:20:15 (1781220015) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.44223 s, 9.6 MB/s -rw-r--r-- 1 root root 52428800 Jun 11 19:20 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.319s user 0m0.000s sys 0m0.181s Stopping /mnt/lustre-ost1 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9e5f05077800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9e5f05077800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.118s user 0m0.001s sys 0m0.207s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg432-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg432-server oleg432-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9e5f05077800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e5f05077800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.222s user 0m0.002s sys 0m0.191s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg432-server: oleg432-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg432-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg432-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e5f05077800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9e5f05077800.ost_server_uuid in FULL state after 0 sec PASS 33b (97s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 19:21:52 (1781220112) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 19:21:56 (1781220116) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 19:22:00 (1781220120) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 19:22:05 (1781220125) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 19:22:13 (1781220133) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.57562 s, 21.9 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 19:22:26 (1781220146) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377384 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364714 s, 28.8 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 19:22:33 (1781220153) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 19:22:41 (1781220161) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00246757 s, 415 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00111142 s, 921 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00429901 s, 238 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00130523 s, 785 kB/s flv=10 olv=10 PASS 36d (14s) == sanity-flr test 37: mirror I/O API verification ======= 19:22:55 (1781220175) create test files with size 15757200 .. dcbd946d608c6db5a3cc8e010a3c68eb - e445c4be853c5fa61191b059f89bad89 - c5e4747d0cb837091185df91e3e978de - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg432-client.virtnet /mnt/lustre (opts:) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre PASS 37 (28s) == sanity-flr test 38: resync ============================ 19:23:23 (1781220203) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (15s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 19:23:38 (1781220218) PASS 39 (6s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 19:23:44 (1781220224) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0913262 s, 11.5 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbe:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xbf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0387424 s, 27.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xde:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdf:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (9s) == sanity-flr test 41: lfs mirror resync check =========== 19:23:54 (1781220234) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.169739 s, 12.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.31809 s, 13.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.228372 s, 9.2 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 19:24:13 (1781220253) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 19:24:17 (1781220257) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 19:24:20 (1781220260) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0283586 s, 37.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 43c: read prefer uncompressed mirror == 19:24:28 (1781220268) llite.lustre-ffff9e5f04eb4800.enable_compression=1 fail_loc=0x1425 fail_val=-1 PASS 43c (7s) == sanity-flr test 44a: lfs mirror split check =========== 19:24:35 (1781220275) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 19:24:38 (1781220278) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe7:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc8:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc8:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 19:24:50 (1781220290) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.334048 s, 31.4 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x1a:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg432-client.virtnet /mnt/lustre (opts:) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 19:25:04 (1781220304) llite.lustre-ffff9e5f04ea4000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.296311 s, 35.4 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg432-client.virtnet /mnt/lustre (opts:) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9e5f3a0a1800.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 19:25:20 (1781220320) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (47s) == sanity-flr test 46: Verify setstripe --copy option ==== 19:26:07 (1781220367) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 19:26:48 (1781220408) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 19:26:52 (1781220412) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.353993 s, 8.9 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.178752 s, 17.6 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xeb6fd8e7 (nosync,stale) Mirror 2: 0xeb6fd8e7 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 4820159d20380fb030c1bf3e0fd86998 - mirror 1 checksum: 1559e71a4f584be82747fd3b1f6187ee - mirror 2 checksum: 4820159d20380fb030c1bf3e0fd86998 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 19:26:57 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 19:26:57 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 4820159d20380fb030c1bf3e0fd86998 - mirror 1 checksum: 1559e71a4f584be82747fd3b1f6187ee - mirror 2 checksum: 1559e71a4f584be82747fd3b1f6187ee - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 19:26:57 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdd:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 19:26:57 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } PASS 48 (14s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 19:27:07 (1781220427) Failing mds1 on oleg432-server Stopping /mnt/lustre-mds1 (opts:) on oleg432-server 19:27:15 (1781220435) shut down facet: mds1 facet_host: oleg432-server facet_failover_host: oleg432-server Failover mds1 to oleg432-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-MDT0000 19:27:35 (1781220455) targets are mounted 19:27:35 (1781220455) facet_failover done oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0396163 s, 26.5 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 19:27:55 (1781220475) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 19:27:59 (1781220479) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (5s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 19:28:04 (1781220484) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00814641 s, 2.0 MB/s PASS 50c (7s) == sanity-flr test 50d: mirror rsync keep holes ========== 19:28:12 (1781220492) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 19:28:16 (1781220496) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (3s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 19:28:19 (1781220499) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 19:28:44 (1781220524) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 19:29:14 (1781220554) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (26s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 19:29:39 (1781220579) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 19:30:04 (1781220604) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 19:30:13 (1781220613) mirror create pid 54138 mirror split pid 54139 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 54138 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 54139 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (67s) == sanity-flr test 70a: flr mode fsx test ================ 19:31:20 (1781220680) Seed set to 682 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read 000004 1781220682.614713 write 0x01f0f4 thru 0x024fe4 (0x5ef1 bytes) 000008 1781220682.619161 mapread 0x017272 thru 0x01bef4 (0x4c83 bytes) 000010 1781220682.638653 mapwrite 0x015d4a thru 0x020f49 (0xb200 bytes) truncating to largest ever: 0x30c2c 000014 1781220682.734873 trunc from 0x024fe5 to 0x030c2b (0xbc47 bytes) 000017 1781220682.778316 mapwrite 0x00c8b8 thru 0x01c61c (0xfd65 bytes) 000018 1781220682.927687 mapread 0x023136 thru 0x02f92d (0xc7f8 bytes) 000020 1781220682.976308 read 0x024f35 thru 0x0297d2 (0x489e bytes) 000024 1781220682.982324 trunc from 0x030c2c to 0x01058c (0x2069f bytes) 000026 1781220683.013497 trunc from 0x01058d to 0x027ca8 (0x1771c bytes) 000027 1781220683.042278 MIRROR_EXTEND to 2 mirrors 000031 1781220683.808411 write 0x0094eb thru 0x0114f3 (0x8009 bytes) 000032 1781220683.917437 trunc from 0x027ca9 to 0x01c572 (0xb736 bytes) 000035 1781220683.959235 write 0x03755b thru 0x03ffff (0x8aa5 bytes) 000036 1781220683.961784 mapwrite 0x02191c thru 0x030882 (0xef67 bytes) 000037 1781220684.113616 mapwrite 0x028bbd thru 0x030ada (0x7f1e bytes) 000040 1781220684.220909 mapread 0x01eff8 thru 0x01fb67 (0xb70 bytes) 000041 1781220684.224665 mapwrite 0x02188a thru 0x0295ec (0x7d63 bytes) 000042 1781220684.330716 mapread 0x029946 thru 0x02be2c (0x24e7 bytes) 000044 1781220684.333735 mapwrite 0x03c9c0 thru 0x03ffff (0x3640 bytes) 000045 1781220684.394269 write 0x0085d1 thru 0x00d53d (0x4f6d bytes) 000048 1781220684.408779 read 0x02e3eb thru 0x02f47b (0x1091 bytes) 000050 1781220684.410324 mapwrite 0x03e475 thru 0x03ffff (0x1b8b bytes) 000052 1781220684.487806 write 0x011e81 thru 0x01dc72 (0xbdf2 bytes) 000053 1781220684.490114 mapread 0x01d021 thru 0x024ab9 (0x7a99 bytes) 000054 1781220684.499032 mapread 0x01c987 thru 0x02712f (0xa7a9 bytes) 000056 1781220684.508688 write 0x03c860 thru 0x03ffff (0x37a0 bytes) 000061 1781220684.527657 read 0x017175 thru 0x01dbe8 (0x6a74 bytes) 000062 1781220684.528588 read 0x021d64 thru 0x02bfdd (0xa27a bytes) 000063 1781220684.533145 read 0x0386da thru 0x03ffff (0x7926 bytes) 000064 1781220684.535118 write 0x034404 thru 0x03ffff (0xbbfc bytes) 000067 1781220684.574108 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xdc91675a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6652b9c1 Mirror 3: 0x6652b9c1 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000068 1781220685.837801 MIRROR_RESYNC 3 mirrors 000069 1781220686.849520 trunc from 0x040000 to 0x00ce6f (0x33190 bytes) 000070 1781220686.957381 mapwrite 0x035fee thru 0x0360a1 (0xb4 bytes) 000071 1781220687.088045 mapread 0x020203 thru 0x025fe0 (0x5dde bytes) 000072 1781220687.098243 write 0x021472 thru 0x02b988 (0xa517 bytes) 000073 1781220687.104377 trunc from 0x0360a2 to 0x0261d6 (0xfecb bytes) 000074 1781220687.125196 write 0x03f7db thru 0x03ffff (0x825 bytes) 000078 1781220687.129822 write 0x0110e1 thru 0x01810f (0x702f bytes) 000079 1781220687.139943 mapwrite 0x00844d thru 0x014a65 (0xc619 bytes) 000081 1781220687.205186 trunc from 0x040000 to 0x00fce3 (0x3031c bytes) 000083 1781220687.229700 mapwrite 0x00b7a2 thru 0x018b94 (0xd3f3 bytes) 000084 1781220687.339483 mapwrite 0x01951d thru 0x01f9b9 (0x649d bytes) 000085 1781220687.470144 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f9ba): Mirror 3: 0xa0c1f9e5 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f9ba): Mirror 3: 0xa0c1f9e5 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f9ba: skipped 000086 1781220687.782883 mapwrite 0x0390aa thru 0x03f9c2 (0x6919 bytes) 000087 1781220687.870950 write 0x00488b thru 0x0081a5 (0x391b bytes) 000091 1781220687.902595 write 0x009707 thru 0x010fa5 (0x789f bytes) 000093 1781220687.905580 mapread 0x007804 thru 0x00fc52 (0x844f bytes) 000094 1781220687.925225 MIRROR_RESYNC 2 mirrors 000095 1781220688.672493 mapwrite 0x01d813 thru 0x02a992 (0xd180 bytes) 000098 1781220688.883104 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f9c3): Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9c3): Mirror 3: 0x5493334d (stale) CRC-32 checksum value for chunk [0, 0x3f9c3): Mirror 1: 0x769bad4b Mirror 4: 0x769bad4b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9c3: skipped 000101 1781220689.693218 read 0x017562 thru 0x024a98 (0xd537 bytes) 000102 1781220689.703706 mapread 0x016483 thru 0x019864 (0x33e2 bytes) 000103 1781220689.710613 mapwrite 0x0222cb thru 0x0321ec (0xff22 bytes) 000105 1781220689.925601 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3f9c3): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f9c3): Mirror 3: 0x5493334d (stale) Mirror 4: 0x769bad4b (stale) CRC-32 checksum value for chunk [0, 0x3f9c3): Mirror 1: 0x6654805b Mirror 5: 0x6654805b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f9c3: skipped 000106 1781220691.152183 trunc from 0x03f9c3 to 0x01c4f9 (0x234c9 bytes) 000107 1781220691.264766 write 0x00b23a thru 0x010f95 (0x5d5c bytes) 000108 1781220691.290822 write 0x0160fa thru 0x025a77 (0xf97e bytes) 000109 1781220691.318800 mapread 0x01793d thru 0x025493 (0xdb57 bytes) 000110 1781220691.329034 read 0x022332 thru 0x025a77 (0x3746 bytes) 000111 1781220691.329866 read 0x0116ba thru 0x01c6ca (0xb011 bytes) 000113 1781220691.340629 read 0x0097b9 thru 0x015d75 (0xc5bd bytes) 000114 1781220691.368740 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x25a78): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x25a78): Mirror 3: 0xc648b20b (stale) Mirror 4: 0xc10fc720 (stale) Mirror 5: 0x9a0a277d (stale) CRC-32 checksum value for chunk [0, 0x25a78): Mirror 1: 0x93419e09 Mirror 6: 0x93419e09 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x25a78: skipped 000116 1781220692.683006 mapwrite 0x0387ee thru 0x03e479 (0x5c8c bytes) 000117 1781220692.968055 MIRROR_RESYNC 5 mirrors 000118 1781220694.414624 mapread 0x03c521 thru 0x03e479 (0x1f59 bytes) truncating to largest ever: 0x34b0a 000119 1781220694.447600 trunc from 0x03e47a to 0x034b09 (0x9970 bytes) 000120 1781220694.549251 write 0x037d9f thru 0x03ffff (0x8261 bytes) 000121 1781220694.560223 MIRROR_RESYNC 5 mirrors 000122 1781220695.711196 MIRROR_EXTEND to 6 mirrors 000123 1781220696.726010 MIRROR_EXTEND to 7 mirrors truncating to largest ever: 0x374b9 000124 1781220697.758807 trunc from 0x040000 to 0x0374b8 (0x8b47 bytes) 000126 1781220697.887708 write 0x001d97 thru 0x00de5e (0xc0c8 bytes) 000129 1781220697.925665 mapread 0x015f9d thru 0x0203da (0xa43e bytes) 000131 1781220697.956897 mapwrite 0x01c6cf thru 0x01efa3 (0x28d5 bytes) 000132 1781220698.040856 trunc from 0x0374b9 to 0x01d32d (0x1a18b bytes) 000133 1781220698.071101 mapread 0x00ac06 thru 0x00c61b (0x1a16 bytes) truncating to largest ever: 0x3ac8d 000136 1781220698.079524 trunc from 0x01d32e to 0x03ac8c (0x1d95f bytes) 000137 1781220698.101159 read 0x0012ed thru 0x001c29 (0x93d bytes) 000139 1781220698.102725 read 0x02932b thru 0x035c7d (0xc953 bytes) 000141 1781220698.115145 write 0x00bd9e thru 0x0188e3 (0xcb46 bytes) 000142 1781220698.118040 trunc from 0x03ac8d to 0x034abd (0x61cf bytes) 000143 1781220698.143362 mapread 0x007ed5 thru 0x011fdc (0xa108 bytes) truncating to largest ever: 0x3bc84 000145 1781220698.146573 trunc from 0x034abe to 0x03bc83 (0x71c6 bytes) truncating to largest ever: 0x3fd8c 000147 1781220698.175326 trunc from 0x03bc84 to 0x03fd8b (0x4108 bytes) 000149 1781220698.209485 mapread 0x030846 thru 0x03d197 (0xc952 bytes) 000151 1781220698.258942 mapwrite 0x0353f4 thru 0x03ffff (0xac0c bytes) 000152 1781220698.363473 write 0x0001b7 thru 0x00a620 (0xa46a bytes) 000154 1781220698.367482 trunc from 0x040000 to 0x000e51 (0x3f1ae bytes) 000156 1781220698.403883 write 0x004a5e thru 0x00c695 (0x7c38 bytes) 000157 1781220698.411195 write 0x027ef8 thru 0x02e842 (0x694b bytes) 000158 1781220698.418140 trunc from 0x02e843 to 0x03b20f (0xc9cd bytes) 000160 1781220698.462181 trunc from 0x03b210 to 0x033fbb (0x7254 bytes) 000161 1781220698.484931 write 0x0159a7 thru 0x01f139 (0x9793 bytes) 000162 1781220698.495941 mapread 0x01253e thru 0x01e4ba (0xbf7d bytes) 000163 1781220698.513598 mapwrite 0x01e0bc thru 0x02ae95 (0xcdda bytes) 000164 1781220698.630636 MIRROR_SPLIT mirror 1 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x33fbc): Mirror 4: 0x2878669a (stale) Mirror 5: 0x2878669a (stale) Mirror 6: 0x2878669a (stale) Mirror 7: 0x2878669a (stale) Mirror 8: 0x2878669a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33fbc): Mirror 4: 0x2878669a (stale) Mirror 5: 0x2878669a (stale) Mirror 6: 0x2878669a (stale) Mirror 7: 0x2878669a (stale) Mirror 8: 0x2878669a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33fbc: skipped 000166 1781220699.264312 read 0x033941 thru 0x033fbb (0x67b bytes) 000167 1781220699.271788 write 0x016500 thru 0x017c29 (0x172a bytes) 000168 1781220699.284042 write 0x0006c3 thru 0x007ebc (0x77fa bytes) 000169 1781220699.301038 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x33fbc): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33fbc): Mirror 4: 0x2878669a (stale) Mirror 5: 0x2878669a (stale) Mirror 6: 0x2878669a (stale) Mirror 7: 0x2878669a (stale) Mirror 8: 0x2878669a (stale) CRC-32 checksum value for chunk [0, 0x33fbc): Mirror 1: 0xf5d1d9e3 Mirror 9: 0xf5d1d9e3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33fbc: skipped 000170 1781220700.385372 write 0x024cd1 thru 0x030d1d (0xc04d bytes) 000171 1781220700.477003 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x33fbc): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33fbc): Mirror 4: 0x2878669a (stale) Mirror 5: 0x2878669a (stale) Mirror 6: 0x2878669a (stale) Mirror 7: 0x2878669a (stale) Mirror 8: 0x2878669a (stale) Mirror 9: 0xf5d1d9e3 (stale) CRC-32 checksum value for chunk [0, 0x33fbc): Mirror 1: 0xf84f70 Mirror 10: 0xf84f70 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33fbc: skipped 000172 1781220701.737182 mapwrite 0x00e387 thru 0x017b27 (0x97a1 bytes) 000173 1781220701.915151 trunc from 0x033fbc to 0x02cf67 (0x7054 bytes) 000175 1781220701.947089 mapwrite 0x00d24a thru 0x01cbf7 (0xf9ae bytes) 000178 1781220702.012177 mapread 0x00291c thru 0x0094cd (0x6bb2 bytes) 000179 1781220702.054816 mapwrite 0x000cce thru 0x00d248 (0xc57b bytes) 000181 1781220702.185689 write 0x001444 thru 0x005f46 (0x4b03 bytes) 000182 1781220702.190998 trunc from 0x02cf68 to 0x015da3 (0x171c4 bytes) 000183 1781220702.214744 mapwrite 0x013416 thru 0x01ca66 (0x9651 bytes) 000185 1781220702.339866 read 0x01563e thru 0x0177be (0x2181 bytes) 000186 1781220702.346704 MIRROR_SPLIT mirror 1 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1ca67): Mirror 5: 0x54f02769 (stale) Mirror 6: 0x54f02769 (stale) Mirror 7: 0x54f02769 (stale) Mirror 8: 0x54f02769 (stale) Mirror 9: 0x761c4bbb (stale) Mirror 10: 0x761c4bbb (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1ca67): Mirror 5: 0x54f02769 (stale) Mirror 6: 0x54f02769 (stale) Mirror 7: 0x54f02769 (stale) Mirror 8: 0x54f02769 (stale) Mirror 9: 0x761c4bbb (stale) Mirror 10: 0x761c4bbb (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1ca67: skipped 000187 1781220702.835339 mapwrite 0x03115c thru 0x03b6db (0xa580 bytes) 000189 1781220702.982293 mapread 0x011eb1 thru 0x01be2d (0x9f7d bytes) 000190 1781220703.025713 trunc from 0x03b6dc to 0x000c71 (0x3aa6a bytes) 000191 1781220703.059883 mapwrite 0x034a55 thru 0x035594 (0xb40 bytes) 000193 1781220703.132691 mapread 0x01bdc6 thru 0x01c33a (0x575 bytes) 000194 1781220703.145738 mapwrite 0x00e47c thru 0x019e11 (0xb996 bytes) 000196 1781220703.200726 mapwrite 0x03a4fe thru 0x03ef9c (0x4a9f bytes) 000197 1781220703.301148 mapread 0x03a878 thru 0x03ef9c (0x4725 bytes) 000198 1781220703.304187 write 0x037300 thru 0x03ffff (0x8d00 bytes) 000199 1781220703.307326 mapwrite 0x00570a thru 0x01105f (0xb956 bytes) 000201 1781220703.436660 mapwrite 0x02558c thru 0x02e479 (0x8eee bytes) 000202 1781220703.500339 write 0x018af8 thru 0x023dcd (0xb2d6 bytes) 000204 1781220703.503112 write 0x00aff8 thru 0x018fac (0xdfb5 bytes) 000206 1781220703.505786 read 0x001341 thru 0x00e790 (0xd450 bytes) 000208 1781220703.507479 write 0x0373b7 thru 0x03ffff (0x8c49 bytes) 000210 1781220703.509887 write 0x03bbe5 thru 0x03ffff (0x441b bytes) 000214 1781220703.513170 write 0x02abe5 thru 0x02bc5f (0x107b bytes) 000219 1781220703.517748 trunc from 0x040000 to 0x0264cd (0x19b32 bytes) 000221 1781220703.546997 trunc from 0x0264ce to 0x003640 (0x22e8d bytes) 000226 1781220703.584200 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3641): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3641): Mirror 5: 0x749391de (stale) Mirror 6: 0x749391de (stale) Mirror 7: 0x749391de (stale) Mirror 8: 0x749391de (stale) Mirror 9: 0x5bc7c8bd (stale) Mirror 10: 0x5bc7c8bd (stale) CRC-32 checksum value for chunk [0, 0x3641): Mirror 1: 0x9cbd649a Mirror 11: 0x9cbd649a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3641: skipped 000228 1781220704.654485 mapread 0x002653 thru 0x003640 (0xfee bytes) 000229 1781220704.675931 mapread 0x002aec thru 0x003640 (0xb55 bytes) 000230 1781220704.676738 mapwrite 0x0100c1 thru 0x014a77 (0x49b7 bytes) 000232 1781220704.946748 mapwrite 0x02a74f thru 0x03a2b5 (0xfb67 bytes) 000234 1781220705.070717 mapread 0x01f5f1 thru 0x023245 (0x3c55 bytes) 000236 1781220705.083629 write 0x02eb1e thru 0x0364e7 (0x79ca bytes) 000238 1781220705.089960 mapwrite 0x00876d thru 0x00c709 (0x3f9d bytes) 000240 1781220705.145982 read 0x020f92 thru 0x0229fb (0x1a6a bytes) 000242 1781220705.148289 trunc from 0x03a2b6 to 0x035e15 (0x44a0 bytes) 000243 1781220705.169862 trunc from 0x035e16 to 0x01af46 (0x1aecf bytes) 000244 1781220705.193199 mapwrite 0x032657 thru 0x03dfd8 (0xb982 bytes) 000247 1781220705.292444 read 0x025cc3 thru 0x0293a0 (0x36de bytes) 000248 1781220705.304624 mapwrite 0x004799 thru 0x01364b (0xeeb3 bytes) 000251 1781220705.358293 read 0x01b4f3 thru 0x02ab8f (0xf69d bytes) 000252 1781220705.359091 trunc from 0x03dfd9 to 0x002ee2 (0x3b0f6 bytes) 000253 1781220705.379517 read 0x0012a6 thru 0x002ee2 (0x1c3d bytes) 000254 1781220705.388051 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2ee3): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ee3): Mirror 5: 0x3099251 (stale) Mirror 6: 0x3099251 (stale) Mirror 7: 0x3099251 (stale) Mirror 8: 0x3099251 (stale) Mirror 9: 0xe05dbd46 (stale) Mirror 10: 0xe05dbd46 (stale) Mirror 11: 0x66e30223 (stale) CRC-32 checksum value for chunk [0, 0x2ee3): Mirror 1: 0x66e30223 Mirror 12: 0x66e30223 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ee3: skipped 000255 1781220706.496163 mapwrite 0x030de2 thru 0x038e11 (0x8030 bytes) 000258 1781220706.787758 mapread 0x01b123 thru 0x0299c0 (0xe89e bytes) 000259 1781220706.830236 mapread 0x02dd77 thru 0x03732d (0x95b7 bytes) 000261 1781220706.833436 write 0x030258 thru 0x03f1f1 (0xef9a bytes) 000263 1781220706.843370 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x3f1f2): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f1f2): Mirror 5: 0xdafe20b2 (stale) Mirror 6: 0xdafe20b2 (stale) Mirror 7: 0xdafe20b2 (stale) Mirror 8: 0xdafe20b2 (stale) Mirror 9: 0xa30f1798 (stale) Mirror 10: 0xd43db337 (stale) Mirror 11: 0x6915163f (stale) Mirror 12: 0x6915163f (stale) CRC-32 checksum value for chunk [0, 0x3f1f2): Mirror 1: 0x3246f1b Mirror 13: 0x3246f1b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f1f2: skipped 000264 1781220708.220988 mapwrite 0x00c3b2 thru 0x00e4e4 (0x2133 bytes) 000266 1781220708.507351 trunc from 0x03f1f2 to 0x03ccce (0x2523 bytes) 000267 1781220708.555623 write 0x018ec8 thru 0x019c3a (0xd73 bytes) 000269 1781220708.583656 read 0x036927 thru 0x039a6a (0x3144 bytes) 000270 1781220708.593940 mapread 0x01da70 thru 0x02d20f (0xf7a0 bytes) 000271 1781220708.655438 trunc from 0x03cccf to 0x03a7d0 (0x24fe bytes) 000272 1781220708.708011 mapwrite 0x01ba36 thru 0x020202 (0x47cd bytes) 000276 1781220708.811633 mapwrite 0x004bdd thru 0x00b493 (0x68b7 bytes) 000280 1781220708.903832 mapwrite 0x00cc2f thru 0x019861 (0xcc33 bytes) 000282 1781220708.972942 read 0x0368e8 thru 0x03a7d0 (0x3ee9 bytes) 000283 1781220708.986550 mapread 0x008a0f thru 0x012250 (0x9842 bytes) 000287 1781220709.002880 write 0x01c1f9 thru 0x0232fd (0x7105 bytes) 000288 1781220709.012748 mapread 0x026534 thru 0x0304ae (0x9f7b bytes) 000291 1781220709.023486 read 0x03590a thru 0x03a7d0 (0x4ec7 bytes) 000292 1781220709.031371 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3a7d1): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a7d1): Mirror 5: 0x96b7bc8c (stale) Mirror 6: 0x96b7bc8c (stale) Mirror 7: 0x96b7bc8c (stale) Mirror 8: 0x96b7bc8c (stale) Mirror 9: 0x5d4eb4d (stale) Mirror 10: 0xb8a2eba1 (stale) Mirror 11: 0xde41075 (stale) Mirror 12: 0xde41075 (stale) Mirror 13: 0x15989d3 (stale) CRC-32 checksum value for chunk [0, 0x3a7d1): Mirror 1: 0x16abc65 Mirror 14: 0x16abc65 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a7d1: skipped 000294 1781220710.259807 write 0x01fd61 thru 0x027130 (0x73d0 bytes) 000295 1781220710.390514 read 0x02ba1c thru 0x0337d9 (0x7dbe bytes) 000296 1781220710.402485 trunc from 0x03a7d1 to 0x02705d (0x13773 bytes) 000298 1781220710.494014 mapread 0x016eca thru 0x024f52 (0xe089 bytes) 000301 1781220710.534036 mapread 0x001d92 thru 0x004d7d (0x2fec bytes) 000302 1781220710.575624 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2705e): Mirror 5: 0x4b80a236 (stale) Mirror 6: 0x4b80a236 (stale) Mirror 7: 0x4b80a236 (stale) Mirror 8: 0x4b80a236 (stale) Mirror 9: 0xe87f0f93 (stale) Mirror 10: 0x48fac0aa (stale) Mirror 11: 0x9a314a31 (stale) Mirror 12: 0x9a314a31 (stale) Mirror 14: 0x53655c0b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2705e): Mirror 5: 0x4b80a236 (stale) Mirror 6: 0x4b80a236 (stale) Mirror 7: 0x4b80a236 (stale) Mirror 8: 0x4b80a236 (stale) Mirror 9: 0xe87f0f93 (stale) Mirror 10: 0x48fac0aa (stale) Mirror 11: 0x9a314a31 (stale) Mirror 12: 0x9a314a31 (stale) Mirror 14: 0x53655c0b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2705e: skipped 000304 1781220711.445370 read 0x026abb thru 0x02705d (0x5a3 bytes) 000305 1781220711.454531 mapwrite 0x002cb8 thru 0x00738f (0x46d8 bytes) 000310 1781220711.579772 trunc from 0x02705e to 0x00f019 (0x18044 bytes) 000312 1781220711.630339 mapread 0x009de2 thru 0x00e18a (0x43a9 bytes) 000314 1781220711.632240 mapread 0x00a4e1 thru 0x00f019 (0x4b39 bytes) 000315 1781220711.644527 read 0x002234 thru 0x00f019 (0xcde6 bytes) 000316 1781220711.645594 mapread 0x000715 thru 0x0078be (0x71aa bytes) 000317 1781220711.652160 write 0x025c0d thru 0x030e83 (0xb277 bytes) 000320 1781220711.663150 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x30e84): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x30e84): Mirror 5: 0x3d366127 (stale) Mirror 6: 0x3d366127 (stale) Mirror 7: 0x3d366127 (stale) Mirror 8: 0x3d366127 (stale) Mirror 9: 0x98b35196 (stale) Mirror 10: 0x8a489ad9 (stale) Mirror 11: 0x7c85b054 (stale) Mirror 12: 0x7c85b054 (stale) Mirror 14: 0x34f5dec1 (stale) CRC-32 checksum value for chunk [0, 0x30e84): Mirror 1: 0x1bd9786e Mirror 15: 0x1bd9786e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x30e84: skipped 000321 1781220713.663761 write 0x00c3b9 thru 0x018cb7 (0xc8ff bytes) 000322 1781220713.843370 mapwrite 0x025063 thru 0x02c8c0 (0x785e bytes) 000323 1781220713.911648 trunc from 0x030e84 to 0x03837f (0x74fc bytes) 000325 1781220713.954922 MIRROR_RESYNC 11 mirrors 000326 1781220716.209531 mapwrite 0x012e63 thru 0x02107b (0xe219 bytes) 000327 1781220716.553040 write 0x035ba2 thru 0x03e3a3 (0x8802 bytes) 000331 1781220716.602112 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x3e3a4): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e3a4): Mirror 5: 0x94099892 (stale) Mirror 6: 0x94099892 (stale) Mirror 7: 0x94099892 (stale) Mirror 8: 0x94099892 (stale) Mirror 9: 0x94099892 (stale) Mirror 10: 0x94099892 (stale) Mirror 11: 0x94099892 (stale) Mirror 12: 0x94099892 (stale) Mirror 14: 0x94099892 (stale) Mirror 15: 0x94099892 (stale) CRC-32 checksum value for chunk [0, 0x3e3a4): Mirror 1: 0x94901cfa Mirror 16: 0x94901cfa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e3a4: skipped 000332 1781220718.592504 mapread 0x00829c thru 0x00a164 (0x1ec9 bytes) 000333 1781220718.622873 write 0x023f8c thru 0x023fda (0x4f bytes) 000335 1781220718.737519 mapread 0x031675 thru 0x035f34 (0x48c0 bytes) 000336 1781220718.748843 trunc from 0x03e3a4 to 0x02bf02 (0x124a1 bytes) 000337 1781220718.777069 write 0x03f6f2 thru 0x03ffff (0x90e bytes) 000341 1781220718.782301 trunc from 0x040000 to 0x0272ae (0x18d51 bytes) 000342 1781220718.812360 read 0x018561 thru 0x01fe69 (0x7909 bytes) 000347 1781220718.833322 mapread 0x0039b5 thru 0x006312 (0x295e bytes) 000349 1781220718.845291 trunc from 0x0272af to 0x02fb01 (0x8853 bytes) 000350 1781220718.864430 write 0x00877b thru 0x009e25 (0x16ab bytes) 000352 1781220718.866111 mapwrite 0x02e6ad thru 0x03d564 (0xeeb8 bytes) 000355 1781220719.066887 read 0x02e4b2 thru 0x03320b (0x4d5a bytes) 000357 1781220719.075146 write 0x01d66c thru 0x028ff9 (0xb98e bytes) 000358 1781220719.083191 MIRROR_SPLIT mirror 11 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d565): Mirror 5: 0xa6e62be8 (stale) Mirror 6: 0xa6e62be8 (stale) Mirror 7: 0xa6e62be8 (stale) Mirror 8: 0xa6e62be8 (stale) Mirror 9: 0xa6e62be8 (stale) Mirror 10: 0xa6e62be8 (stale) Mirror 11: 0xa6e62be8 (stale) Mirror 12: 0xa6e62be8 (stale) Mirror 14: 0xa6e62be8 (stale) Mirror 15: 0xa6e62be8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d565): Mirror 5: 0xa6e62be8 (stale) Mirror 6: 0xa6e62be8 (stale) Mirror 7: 0xa6e62be8 (stale) Mirror 8: 0xa6e62be8 (stale) Mirror 9: 0xa6e62be8 (stale) Mirror 10: 0xa6e62be8 (stale) Mirror 11: 0xa6e62be8 (stale) Mirror 12: 0xa6e62be8 (stale) Mirror 14: 0xa6e62be8 (stale) Mirror 15: 0xa6e62be8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d565: skipped 000359 1781220720.093475 mapwrite 0x02a2cd thru 0x0358f2 (0xb626 bytes) 000360 1781220720.265567 read 0x01b6a4 thru 0x029001 (0xd95e bytes) 000362 1781220720.280333 trunc from 0x03d565 to 0x02cfe5 (0x1057f bytes) 000363 1781220720.304054 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x2cfe6): Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cfe6): Mirror 5: 0x8359bf86 (stale) Mirror 6: 0x8359bf86 (stale) Mirror 7: 0x8359bf86 (stale) Mirror 8: 0x8359bf86 (stale) Mirror 9: 0x8359bf86 (stale) Mirror 10: 0x8359bf86 (stale) Mirror 11: 0x8359bf86 (stale) Mirror 12: 0x8359bf86 (stale) Mirror 14: 0x8359bf86 (stale) Mirror 15: 0x8359bf86 (stale) CRC-32 checksum value for chunk [0, 0x2cfe6): Mirror 1: 0xcf517f35 Mirror 16: 0xcf517f35 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cfe6: skipped 000364 1781220721.498117 write 0x01583c thru 0x01d7b1 (0x7f76 bytes) 000367 1781220721.622246 mapread 0x029db4 thru 0x02cfe5 (0x3232 bytes) 000368 1781220721.651503 MIRROR_SPLIT mirror 5 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2cfe6): Mirror 5: 0x8359bf86 (stale) Mirror 6: 0x8359bf86 (stale) Mirror 7: 0x8359bf86 (stale) Mirror 8: 0x8359bf86 (stale) Mirror 10: 0x8359bf86 (stale) Mirror 11: 0x8359bf86 (stale) Mirror 12: 0x8359bf86 (stale) Mirror 14: 0x8359bf86 (stale) Mirror 15: 0x8359bf86 (stale) Mirror 16: 0xcf517f35 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cfe6): Mirror 5: 0x8359bf86 (stale) Mirror 6: 0x8359bf86 (stale) Mirror 7: 0x8359bf86 (stale) Mirror 8: 0x8359bf86 (stale) Mirror 10: 0x8359bf86 (stale) Mirror 11: 0x8359bf86 (stale) Mirror 12: 0x8359bf86 (stale) Mirror 14: 0x8359bf86 (stale) Mirror 15: 0x8359bf86 (stale) Mirror 16: 0xcf517f35 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cfe6: skipped 000369 1781220722.485285 trunc from 0x02cfe6 to 0x02459a (0x8a4b bytes) 000370 1781220722.527522 mapread 0x0175be thru 0x021bcc (0xa60f bytes) 000371 1781220722.547270 mapwrite 0x03f2df thru 0x03ffff (0xd21 bytes) 000373 1781220722.637292 mapread 0x02d8ea thru 0x031132 (0x3849 bytes) 000375 1781220722.651806 mapread 0x01a3e2 thru 0x0200c6 (0x5ce5 bytes) 000378 1781220722.655624 write 0x0032c4 thru 0x00fa4b (0xc788 bytes) 000379 1781220722.673901 MIRROR_SPLIT mirror 9 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x9d6222d (stale) Mirror 6: 0x9d6222d (stale) Mirror 7: 0x9d6222d (stale) Mirror 8: 0x9d6222d (stale) Mirror 10: 0x9d6222d (stale) Mirror 11: 0x9d6222d (stale) Mirror 12: 0x9d6222d (stale) Mirror 14: 0x9d6222d (stale) Mirror 16: 0x8425ead4 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x9d6222d (stale) Mirror 6: 0x9d6222d (stale) Mirror 7: 0x9d6222d (stale) Mirror 8: 0x9d6222d (stale) Mirror 10: 0x9d6222d (stale) Mirror 11: 0x9d6222d (stale) Mirror 12: 0x9d6222d (stale) Mirror 14: 0x9d6222d (stale) Mirror 16: 0x8425ead4 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000381 1781220723.426073 MIRROR_RESYNC 10 mirrors 000382 1781220725.644927 read 0x02d822 thru 0x03901e (0xb7fd bytes) 000383 1781220725.655172 read 0x037d96 thru 0x03ffff (0x826a bytes) 000384 1781220725.667293 read 0x03f167 thru 0x03ffff (0xe99 bytes) 000385 1781220725.668001 write 0x036f6a thru 0x03ffff (0x9096 bytes) 000387 1781220725.794009 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x9d2d8f55 (stale) Mirror 7: 0x9d2d8f55 (stale) Mirror 8: 0x9d2d8f55 (stale) Mirror 10: 0x9d2d8f55 (stale) Mirror 11: 0x9d2d8f55 (stale) Mirror 12: 0x9d2d8f55 (stale) Mirror 14: 0x9d2d8f55 (stale) Mirror 16: 0x9d2d8f55 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x9d2d8f55 (stale) Mirror 7: 0x9d2d8f55 (stale) Mirror 8: 0x9d2d8f55 (stale) Mirror 10: 0x9d2d8f55 (stale) Mirror 11: 0x9d2d8f55 (stale) Mirror 12: 0x9d2d8f55 (stale) Mirror 14: 0x9d2d8f55 (stale) Mirror 16: 0x9d2d8f55 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000389 1781220726.444744 mapread 0x022c6a thru 0x024277 (0x160e bytes) 000391 1781220726.465245 trunc from 0x040000 to 0x02a661 (0x1599e bytes) 000392 1781220726.487281 mapread 0x0056c2 thru 0x00a90a (0x5249 bytes) 000393 1781220726.497295 mapwrite 0x031a96 thru 0x03e527 (0xca92 bytes) 000394 1781220726.620371 mapread 0x02a528 thru 0x030be8 (0x66c1 bytes) 000395 1781220726.621790 write 0x030c1e thru 0x032045 (0x1428 bytes) 000398 1781220726.627809 read 0x02e5f1 thru 0x033389 (0x4d99 bytes) 000399 1781220726.629079 write 0x020986 thru 0x02fad1 (0xf14c bytes) 000400 1781220726.631780 read 0x03c25c thru 0x03e527 (0x22cc bytes) 000404 1781220726.640971 read 0x03d87d thru 0x03da73 (0x1f7 bytes) 000405 1781220726.643006 mapread 0x02be1a thru 0x036da9 (0xaf90 bytes) 000406 1781220726.644959 read 0x020adb thru 0x027a78 (0x6f9e bytes) 000410 1781220726.650615 write 0x0003df thru 0x0060b6 (0x5cd8 bytes) 000414 1781220726.670657 mapwrite 0x00efbd thru 0x0108be (0x1902 bytes) 000415 1781220726.727935 write 0x018913 thru 0x02809a (0xf788 bytes) 000416 1781220726.736139 MIRROR_RESYNC 9 mirrors 000418 1781220728.960220 trunc from 0x03e528 to 0x007cd9 (0x3684e bytes) 000424 1781220729.083882 mapread 0x006df8 thru 0x007cd9 (0xee2 bytes) 000425 1781220729.109855 write 0x028069 thru 0x02cea0 (0x4e38 bytes) 000428 1781220729.128786 read 0x01ee12 thru 0x02cea0 (0xe08f bytes) 000429 1781220729.139272 read 0x01727f thru 0x01b208 (0x3f8a bytes) 000430 1781220729.162140 read 0x0053ae thru 0x005cee (0x941 bytes) 000432 1781220729.168739 trunc from 0x02cea1 to 0x023a43 (0x945d bytes) 000434 1781220729.215878 read 0x0061df thru 0x00c8fa (0x671c bytes) 000435 1781220729.226331 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x23a44): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23a44): Mirror 6: 0x4eb848f1 (stale) Mirror 7: 0x4eb848f1 (stale) Mirror 8: 0x4eb848f1 (stale) Mirror 10: 0x4eb848f1 (stale) Mirror 11: 0x4eb848f1 (stale) Mirror 12: 0x4eb848f1 (stale) Mirror 14: 0x4eb848f1 (stale) Mirror 16: 0x4eb848f1 (stale) CRC-32 checksum value for chunk [0, 0x23a44): Mirror 1: 0xe0cfffd4 Mirror 17: 0xe0cfffd4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23a44: skipped 000437 1781220730.642249 mapread 0x00e1ae thru 0x0104a4 (0x22f7 bytes) 000438 1781220730.664139 mapwrite 0x02abc5 thru 0x02b18a (0x5c6 bytes) 000440 1781220730.871640 write 0x00f06b thru 0x01b080 (0xc016 bytes) 000441 1781220730.888667 mapwrite 0x005f70 thru 0x0111c9 (0xb25a bytes) 000442 1781220730.971083 mapwrite 0x037770 thru 0x03ffff (0x8890 bytes) 000443 1781220731.102133 read 0x0245a7 thru 0x02bf65 (0x79bf bytes) 000444 1781220731.102919 mapwrite 0x014dff thru 0x01b7e0 (0x69e2 bytes) 000445 1781220731.168975 trunc from 0x040000 to 0x005de5 (0x3a21a bytes) 000448 1781220731.197047 mapwrite 0x02c7fe thru 0x03a3df (0xdbe2 bytes) 000450 1781220731.367233 MIRROR_RESYNC 10 mirrors 000451 1781220733.347905 mapwrite 0x01524e thru 0x01a2ad (0x5060 bytes) 000453 1781220733.525857 MIRROR_RESYNC 10 mirrors 000454 1781220735.548919 mapwrite 0x00cf14 thru 0x015bbc (0x8ca9 bytes) 000456 1781220735.746960 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x3a3e0): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a3e0): Mirror 6: 0x4d11a939 (stale) Mirror 7: 0x4d11a939 (stale) Mirror 8: 0x4d11a939 (stale) Mirror 10: 0x4d11a939 (stale) Mirror 11: 0x4d11a939 (stale) Mirror 12: 0x4d11a939 (stale) Mirror 14: 0x4d11a939 (stale) Mirror 16: 0x4d11a939 (stale) Mirror 17: 0x4d11a939 (stale) CRC-32 checksum value for chunk [0, 0x3a3e0): Mirror 1: 0x63e4903b Mirror 18: 0x63e4903b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a3e0: skipped 000457 1781220736.959574 trunc from 0x03a3e0 to 0x0396df (0xd00 bytes) 000458 1781220737.080159 read 0x00a2c6 thru 0x0172f8 (0xd033 bytes) 000461 1781220737.092050 mapwrite 0x002330 thru 0x004c68 (0x2939 bytes) 000462 1781220737.201257 read 0x0100b8 thru 0x019488 (0x93d1 bytes) 000464 1781220737.221149 mapread 0x0172ac thru 0x0207c7 (0x951c bytes) 000467 1781220737.225872 mapwrite 0x0095ce thru 0x01453b (0xaf6e bytes) 000468 1781220737.288314 read 0x009396 thru 0x01219d (0x8e08 bytes) 000469 1781220737.289315 read 0x00ee30 thru 0x01796e (0x8b3f bytes) 000470 1781220737.291418 mapread 0x01eab7 thru 0x02a092 (0xb5dc bytes) 000474 1781220737.308371 MIRROR_RESYNC 11 mirrors 000475 1781220739.594293 MIRROR_EXTEND to 12 mirrors 000478 1781220740.910567 mapread 0x000445 thru 0x002df5 (0x29b1 bytes) 000480 1781220740.951006 MIRROR_EXTEND to 13 mirrors 000481 1781220742.275667 read 0x011428 thru 0x0134ec (0x20c5 bytes) 000482 1781220742.287719 write 0x02a39a thru 0x034d7d (0xa9e4 bytes) 000483 1781220742.405358 MIRROR_SPLIT mirror 7 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x396e0): Mirror 6: 0xebccdbd0 (stale) Mirror 7: 0xebccdbd0 (stale) Mirror 8: 0xebccdbd0 (stale) Mirror 10: 0xebccdbd0 (stale) Mirror 11: 0xebccdbd0 (stale) Mirror 12: 0xebccdbd0 (stale) Mirror 16: 0xebccdbd0 (stale) Mirror 17: 0xebccdbd0 (stale) Mirror 18: 0xebccdbd0 (stale) Mirror 19: 0xebccdbd0 (stale) Mirror 20: 0xebccdbd0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x396e0): Mirror 6: 0xebccdbd0 (stale) Mirror 7: 0xebccdbd0 (stale) Mirror 8: 0xebccdbd0 (stale) Mirror 10: 0xebccdbd0 (stale) Mirror 11: 0xebccdbd0 (stale) Mirror 12: 0xebccdbd0 (stale) Mirror 16: 0xebccdbd0 (stale) Mirror 17: 0xebccdbd0 (stale) Mirror 18: 0xebccdbd0 (stale) Mirror 19: 0xebccdbd0 (stale) Mirror 20: 0xebccdbd0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x396e0: skipped 000487 1781220743.256776 mapwrite 0x028811 thru 0x02e7ee (0x5fde bytes) 000489 1781220743.409516 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x396e0): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x396e0): Mirror 6: 0xebccdbd0 (stale) Mirror 7: 0xebccdbd0 (stale) Mirror 8: 0xebccdbd0 (stale) Mirror 10: 0xebccdbd0 (stale) Mirror 11: 0xebccdbd0 (stale) Mirror 12: 0xebccdbd0 (stale) Mirror 16: 0xebccdbd0 (stale) Mirror 17: 0xebccdbd0 (stale) Mirror 18: 0xebccdbd0 (stale) Mirror 19: 0xebccdbd0 (stale) Mirror 20: 0xebccdbd0 (stale) CRC-32 checksum value for chunk [0, 0x396e0): Mirror 1: 0x9708b8fa Mirror 21: 0x9708b8fa lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x396e0: skipped 000491 1781220745.012570 trunc from 0x0396e0 to 0x0148c6 (0x24e19 bytes) 000493 1781220745.163757 mapwrite 0x02ba60 thru 0x02d5c7 (0x1b68 bytes) 000494 1781220745.270860 write 0x02562a thru 0x025c17 (0x5ee bytes) 000495 1781220745.282450 write 0x02eead thru 0x03574c (0x68a0 bytes) 000497 1781220745.285601 mapwrite 0x03b9f3 thru 0x03ffff (0x460d bytes) 000498 1781220745.370666 trunc from 0x040000 to 0x036938 (0x96c7 bytes) 000503 1781220745.393360 read 0x00fcd1 thru 0x0190b8 (0x93e8 bytes) 000504 1781220745.402173 mapwrite 0x00de75 thru 0x00ebee (0xd7a bytes) 000505 1781220745.467077 write 0x024d84 thru 0x02f565 (0xa7e2 bytes) 000507 1781220745.470304 read 0x0221fd thru 0x031886 (0xf68a bytes) 000508 1781220745.471218 trunc from 0x036939 to 0x00f409 (0x2752f bytes) 000509 1781220745.499612 MIRROR_RESYNC 13 mirrors 000510 1781220747.978452 trunc from 0x00f40a to 0x00b2e0 (0x4129 bytes) 000511 1781220748.116357 trunc from 0x00b2e1 to 0x00e0ee (0x2e0e bytes) 000512 1781220748.147579 mapwrite 0x03aa60 thru 0x03ae75 (0x416 bytes) 000513 1781220748.298049 write 0x02f6d4 thru 0x03c6e9 (0xd016 bytes) 000518 1781220748.316458 mapread 0x02c0e1 thru 0x02d4f2 (0x1412 bytes) 000519 1781220748.335959 read 0x039182 thru 0x03c667 (0x34e6 bytes) 000520 1781220748.339372 read 0x02d0ee thru 0x038d39 (0xbc4c bytes) 000521 1781220748.340230 trunc from 0x03c6ea to 0x02b50a (0x111df bytes) 000522 1781220748.398137 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x2b50b): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b50b): Mirror 6: 0xda43ea11 (stale) Mirror 7: 0xda43ea11 (stale) Mirror 8: 0xda43ea11 (stale) Mirror 10: 0xda43ea11 (stale) Mirror 11: 0xda43ea11 (stale) Mirror 12: 0xda43ea11 (stale) Mirror 16: 0xda43ea11 (stale) Mirror 17: 0xda43ea11 (stale) Mirror 18: 0xda43ea11 (stale) Mirror 19: 0xda43ea11 (stale) Mirror 20: 0xda43ea11 (stale) Mirror 21: 0xda43ea11 (stale) CRC-32 checksum value for chunk [0, 0x2b50b): Mirror 1: 0xc1d1d72d Mirror 22: 0xc1d1d72d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b50b: skipped 000523 1781220750.199814 MIRROR_SPLIT mirror 4 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2b50b): Mirror 6: 0xda43ea11 (stale) Mirror 7: 0xda43ea11 (stale) Mirror 8: 0xda43ea11 (stale) Mirror 11: 0xda43ea11 (stale) Mirror 12: 0xda43ea11 (stale) Mirror 16: 0xda43ea11 (stale) Mirror 17: 0xda43ea11 (stale) Mirror 18: 0xda43ea11 (stale) Mirror 19: 0xda43ea11 (stale) Mirror 20: 0xda43ea11 (stale) Mirror 21: 0xda43ea11 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2b50b): Mirror 6: 0xda43ea11 (stale) Mirror 7: 0xda43ea11 (stale) Mirror 8: 0xda43ea11 (stale) Mirror 11: 0xda43ea11 (stale) Mirror 12: 0xda43ea11 (stale) Mirror 16: 0xda43ea11 (stale) Mirror 17: 0xda43ea11 (stale) Mirror 18: 0xda43ea11 (stale) Mirror 19: 0xda43ea11 (stale) Mirror 20: 0xda43ea11 (stale) Mirror 21: 0xda43ea11 (stale) CRC-32 checksum value for chunk [0, 0x2b50b): Mirror 1: 0xc1d1d72d Mirror 22: 0xc1d1d72d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2b50b: skipped 000524 1781220751.541075 trunc from 0x02b50b to 0x02378a (0x7d80 bytes) 000527 1781220751.751562 MIRROR_SPLIT mirror 4 to 12 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2378b): Mirror 6: 0xbd230cb3 (stale) Mirror 7: 0xbd230cb3 (stale) Mirror 8: 0xbd230cb3 (stale) Mirror 12: 0xbd230cb3 (stale) Mirror 16: 0xbd230cb3 (stale) Mirror 17: 0xbd230cb3 (stale) Mirror 18: 0xbd230cb3 (stale) Mirror 19: 0xbd230cb3 (stale) Mirror 20: 0xbd230cb3 (stale) Mirror 21: 0xbd230cb3 (stale) Mirror 22: 0xda11e4a7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2378b): Mirror 6: 0xbd230cb3 (stale) Mirror 7: 0xbd230cb3 (stale) Mirror 8: 0xbd230cb3 (stale) Mirror 12: 0xbd230cb3 (stale) Mirror 16: 0xbd230cb3 (stale) Mirror 17: 0xbd230cb3 (stale) Mirror 18: 0xbd230cb3 (stale) Mirror 19: 0xbd230cb3 (stale) Mirror 20: 0xbd230cb3 (stale) Mirror 21: 0xbd230cb3 (stale) Mirror 22: 0xda11e4a7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2378b: skipped 000528 1781220752.630917 mapread 0x006c43 thru 0x011334 (0xa6f2 bytes) 000530 1781220752.707807 mapread 0x00b685 thru 0x017899 (0xc215 bytes) 000531 1781220752.709191 trunc from 0x02378b to 0x00708b (0x1c6ff bytes) 000534 1781220752.770675 trunc from 0x00708c to 0x023521 (0x1c496 bytes) 000535 1781220752.816023 mapread 0x01fc05 thru 0x023521 (0x391d bytes) 000536 1781220752.848283 mapread 0x017490 thru 0x01c099 (0x4c0a bytes) 000537 1781220752.859705 write 0x01030f thru 0x013c87 (0x3979 bytes) 000540 1781220752.864326 write 0x0114dc thru 0x016a66 (0x558b bytes) 000542 1781220752.870201 mapread 0x007ed7 thru 0x010859 (0x8983 bytes) 000543 1781220752.889815 mapread 0x017ecc thru 0x01e11f (0x6254 bytes) 000544 1781220752.893980 mapread 0x007323 thru 0x015c78 (0xe956 bytes) 000545 1781220752.895733 mapread 0x0002ae thru 0x00e8f7 (0xe64a bytes) 000546 1781220752.917769 read 0x006a93 thru 0x00c24f (0x57bd bytes) 000548 1781220752.919157 write 0x028402 thru 0x02b50f (0x310e bytes) 000550 1781220752.927561 write 0x029364 thru 0x03514c (0xbde9 bytes) 000551 1781220752.934873 write 0x014577 thru 0x022d7d (0xe807 bytes) 000554 1781220752.946751 mapwrite 0x00ba7f thru 0x017278 (0xb7fa bytes) 000555 1781220753.073088 mapread 0x0181df thru 0x027454 (0xf276 bytes) 000557 1781220753.092035 write 0x015b83 thru 0x0211da (0xb658 bytes) 000559 1781220753.095649 trunc from 0x03514d to 0x0069bd (0x2e78f bytes) 000561 1781220753.131903 read 0x005f2a thru 0x0069bd (0xa94 bytes) 000563 1781220753.143613 MIRROR_RESYNC 12 mirrors 000567 1781220755.615020 read 0x0026b2 thru 0x006066 (0x39b5 bytes) 000568 1781220755.630236 trunc from 0x0069be to 0x034b9f (0x2e1e2 bytes) 000569 1781220755.804783 MIRROR_EXTEND to 13 mirrors CRC-32 checksum value for chunk [0, 0x34ba0): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34ba0): Mirror 6: 0xa21d2c54 (stale) Mirror 7: 0xa21d2c54 (stale) Mirror 8: 0xa21d2c54 (stale) Mirror 12: 0xa21d2c54 (stale) Mirror 16: 0xa21d2c54 (stale) Mirror 17: 0xa21d2c54 (stale) Mirror 18: 0xa21d2c54 (stale) Mirror 19: 0xa21d2c54 (stale) Mirror 20: 0xa21d2c54 (stale) Mirror 21: 0xa21d2c54 (stale) Mirror 22: 0xa21d2c54 (stale) CRC-32 checksum value for chunk [0, 0x34ba0): Mirror 1: 0xa21d2c54 Mirror 23: 0xa21d2c54 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34ba0: skipped 000571 1781220757.163677 trunc from 0x034ba0 to 0x01b902 (0x1929d bytes) 000572 1781220757.273053 mapwrite 0x028f18 thru 0x02a7a3 (0x188c bytes) 000573 1781220757.417837 mapread 0x017756 thru 0x021cf9 (0xa5a4 bytes) 000576 1781220757.442566 MIRROR_RESYNC 13 mirrors 000577 1781220759.959528 write 0x00bd75 thru 0x012661 (0x68ed bytes) 000578 1781220760.161978 mapread 0x003c16 thru 0x006213 (0x25fe bytes) 000579 1781220760.180142 mapwrite 0x03b2f6 thru 0x03ffff (0x4d0a bytes) 000580 1781220760.309896 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x245ce649 (stale) Mirror 7: 0x245ce649 (stale) Mirror 8: 0x245ce649 (stale) Mirror 12: 0x245ce649 (stale) Mirror 16: 0x245ce649 (stale) Mirror 17: 0x245ce649 (stale) Mirror 18: 0x245ce649 (stale) Mirror 19: 0x245ce649 (stale) Mirror 20: 0x245ce649 (stale) Mirror 21: 0x245ce649 (stale) Mirror 22: 0x245ce649 (stale) Mirror 23: 0x245ce649 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x17ea05ae Mirror 24: 0x17ea05ae lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000582 1781220761.689829 write 0x02385e thru 0x030954 (0xd0f7 bytes) 000583 1781220761.889701 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x245ce649 (stale) Mirror 7: 0x245ce649 (stale) Mirror 8: 0x245ce649 (stale) Mirror 12: 0x245ce649 (stale) Mirror 16: 0x245ce649 (stale) Mirror 17: 0x245ce649 (stale) Mirror 18: 0x245ce649 (stale) Mirror 19: 0x245ce649 (stale) Mirror 20: 0x245ce649 (stale) Mirror 21: 0x245ce649 (stale) Mirror 22: 0x245ce649 (stale) Mirror 23: 0x245ce649 (stale) Mirror 24: 0x17ea05ae (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x466a6040 Mirror 25: 0x466a6040 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000584 1781220763.492609 MIRROR_RESYNC 15 mirrors 000585 1781220766.150680 trunc from 0x040000 to 0x00834c (0x37cb3 bytes) 000586 1781220766.320825 read 0x006a47 thru 0x00834c (0x1906 bytes) 000589 1781220766.332244 write 0x008022 thru 0x008169 (0x148 bytes) 000590 1781220766.333901 read 0x004136 thru 0x00834c (0x4217 bytes) 000592 1781220766.353544 mapread 0x00425b thru 0x00834c (0x40f2 bytes) 000593 1781220766.375744 MIRROR_RESYNC 15 mirrors 000595 1781220768.821651 mapread 0x0021ed thru 0x00834c (0x6160 bytes) 000596 1781220768.842095 trunc from 0x00834d to 0x000933 (0x7a19 bytes) 000599 1781220768.945721 mapread 0x00015e thru 0x000532 (0x3d5 bytes) 000601 1781220768.953563 mapwrite 0x017906 thru 0x01b45e (0x3b59 bytes) 000602 1781220769.090898 mapwrite 0x039133 thru 0x03e12c (0x4ffa bytes) 000603 1781220769.276824 mapwrite 0x00389d thru 0x004521 (0xc85 bytes) 000604 1781220769.362444 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x3e12d): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) Mirror 21: (stale - read error) Mirror 22: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e12d): Mirror 6: 0xf4129dd8 (stale) Mirror 7: 0xf4129dd8 (stale) Mirror 8: 0xf4129dd8 (stale) Mirror 12: 0xf4129dd8 (stale) Mirror 16: 0xf4129dd8 (stale) Mirror 17: 0xf4129dd8 (stale) Mirror 18: 0xf4129dd8 (stale) Mirror 19: 0xf4129dd8 (stale) Mirror 20: 0xf4129dd8 (stale) Mirror 21: 0xf4129dd8 (stale) Mirror 22: 0xf4129dd8 (stale) Mirror 23: 0xf4129dd8 (stale) Mirror 24: 0xf4129dd8 (stale) Mirror 25: 0xf4129dd8 (stale) CRC-32 checksum value for chunk [0, 0x3e12d): Mirror 1: 0xc4ad6bd6 Mirror 26: 0xc4ad6bd6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e12d: skipped 000605 1781220770.761687 mapread 0x01bbe6 thru 0x02b382 (0xf79d bytes) 000606 1781220770.807989 mapread 0x00fc61 thru 0x015b35 (0x5ed5 bytes) 000607 1781220770.819696 write 0x037cb2 thru 0x03ffff (0x834e bytes) 000609 1781220770.938170 mapread 0x03daaf thru 0x03ffff (0x2551 bytes) 000611 1781220770.945291 mapwrite 0x03fc1b thru 0x03ffff (0x3e5 bytes) 000612 1781220771.015373 MIRROR_SPLIT mirror 11 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x435cb526 (stale) Mirror 7: 0x435cb526 (stale) Mirror 8: 0x435cb526 (stale) Mirror 12: 0x435cb526 (stale) Mirror 16: 0x435cb526 (stale) Mirror 17: 0x435cb526 (stale) Mirror 18: 0x435cb526 (stale) Mirror 19: 0x435cb526 (stale) Mirror 20: 0x435cb526 (stale) Mirror 21: 0x435cb526 (stale) Mirror 23: 0x435cb526 (stale) Mirror 24: 0x435cb526 (stale) Mirror 25: 0x435cb526 (stale) Mirror 26: 0xc0739e6b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x435cb526 (stale) Mirror 7: 0x435cb526 (stale) Mirror 8: 0x435cb526 (stale) Mirror 12: 0x435cb526 (stale) Mirror 16: 0x435cb526 (stale) Mirror 17: 0x435cb526 (stale) Mirror 18: 0x435cb526 (stale) Mirror 19: 0x435cb526 (stale) Mirror 20: 0x435cb526 (stale) Mirror 21: 0x435cb526 (stale) Mirror 23: 0x435cb526 (stale) Mirror 24: 0x435cb526 (stale) Mirror 25: 0x435cb526 (stale) Mirror 26: 0xc0739e6b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000613 1781220772.177039 write 0x005f34 thru 0x0120a4 (0xc171 bytes) 000616 1781220772.197982 mapread 0x0286a0 thru 0x0306cd (0x802e bytes) 000618 1781220772.214847 trunc from 0x040000 to 0x02a83e (0x157c1 bytes) 000619 1781220772.261895 mapwrite 0x010ed0 thru 0x01d5a1 (0xc6d2 bytes) 000620 1781220772.362623 MIRROR_SPLIT mirror 9 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a83f): Mirror 6: 0xf676461 (stale) Mirror 7: 0xf676461 (stale) Mirror 8: 0xf676461 (stale) Mirror 12: 0xf676461 (stale) Mirror 16: 0xf676461 (stale) Mirror 17: 0xf676461 (stale) Mirror 18: 0xf676461 (stale) Mirror 19: 0xf676461 (stale) Mirror 21: 0xf676461 (stale) Mirror 23: 0xf676461 (stale) Mirror 24: 0xf676461 (stale) Mirror 25: 0xf676461 (stale) Mirror 26: 0xd117ee2d (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a83f): Mirror 6: 0xf676461 (stale) Mirror 7: 0xf676461 (stale) Mirror 8: 0xf676461 (stale) Mirror 12: 0xf676461 (stale) Mirror 16: 0xf676461 (stale) Mirror 17: 0xf676461 (stale) Mirror 18: 0xf676461 (stale) Mirror 19: 0xf676461 (stale) Mirror 21: 0xf676461 (stale) Mirror 23: 0xf676461 (stale) Mirror 24: 0xf676461 (stale) Mirror 25: 0xf676461 (stale) Mirror 26: 0xd117ee2d (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a83f: skipped 000622 1781220773.392036 trunc from 0x02a83f to 0x02c812 (0x1fd4 bytes) 000623 1781220773.431796 read 0x01a9a4 thru 0x026f8e (0xc5eb bytes) 000624 1781220773.443933 mapread 0x015687 thru 0x0229ba (0xd334 bytes) 000625 1781220773.470503 mapwrite 0x02b339 thru 0x03b220 (0xfee8 bytes) 000626 1781220773.598695 read 0x00c448 thru 0x012193 (0x5d4c bytes) 000627 1781220773.611988 trunc from 0x03b221 to 0x0311ed (0xa033 bytes) 000628 1781220773.644248 MIRROR_RESYNC 14 mirrors 000630 1781220776.356257 mapread 0x00f1b7 thru 0x01f17c (0xffc6 bytes) 000631 1781220776.420506 mapread 0x002320 thru 0x0058fe (0x35df bytes) 000632 1781220776.442817 trunc from 0x0311ee to 0x022fb5 (0xe238 bytes) 000633 1781220776.574042 mapwrite 0x019ce6 thru 0x0287c3 (0xeade bytes) 000635 1781220776.718414 read 0x0046c1 thru 0x0056ff (0x103f bytes) 000636 1781220776.732500 trunc from 0x0287c4 to 0x0080b3 (0x20710 bytes) 000638 1781220776.767952 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x80b4): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x80b4): Mirror 6: 0xed456e0 (stale) Mirror 7: 0xed456e0 (stale) Mirror 8: 0xed456e0 (stale) Mirror 12: 0xed456e0 (stale) Mirror 16: 0xed456e0 (stale) Mirror 17: 0xed456e0 (stale) Mirror 18: 0xed456e0 (stale) Mirror 19: 0xed456e0 (stale) Mirror 21: 0xed456e0 (stale) Mirror 23: 0xed456e0 (stale) Mirror 24: 0xed456e0 (stale) Mirror 25: 0xed456e0 (stale) Mirror 26: 0xed456e0 (stale) CRC-32 checksum value for chunk [0, 0x80b4): Mirror 1: 0xed456e0 Mirror 27: 0xed456e0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x80b4: skipped 000641 1781220778.151907 write 0x0358e1 thru 0x03ffff (0xa71f bytes) 000642 1781220778.261949 mapread 0x03b88c thru 0x03cc66 (0x13db bytes) 000643 1781220778.263118 write 0x005961 thru 0x008e40 (0x34e0 bytes) 000644 1781220778.276379 write 0x015690 thru 0x021ec2 (0xc833 bytes) 000646 1781220778.291335 read 0x0167a4 thru 0x018517 (0x1d74 bytes) 000647 1781220778.292340 trunc from 0x040000 to 0x000b1b (0x3f4e4 bytes) 000651 1781220778.329628 write 0x00da78 thru 0x00ed07 (0x1290 bytes) 000652 1781220778.331966 read 0x00dcf8 thru 0x00ed07 (0x1010 bytes) 000654 1781220778.335925 read 0x00844c thru 0x00c090 (0x3c45 bytes) 000657 1781220778.349357 write 0x021d3d thru 0x022f93 (0x1257 bytes) 000659 1781220778.352236 read 0x019a82 thru 0x022f93 (0x9512 bytes) 000660 1781220778.363708 mapread 0x00fc6b thru 0x015c85 (0x601b bytes) 000661 1781220778.366014 trunc from 0x022f94 to 0x009566 (0x19a2d bytes) 000662 1781220778.395527 read 0x00764f thru 0x009566 (0x1f18 bytes) 000663 1781220778.396845 write 0x018430 thru 0x01f578 (0x7149 bytes) 000664 1781220778.400967 read 0x011c5d thru 0x0172b7 (0x565b bytes) 000666 1781220778.412549 read 0x01075a thru 0x010c72 (0x519 bytes) 000667 1781220778.422082 MIRROR_SPLIT mirror 13 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1f579): Mirror 6: 0x3d6a8264 (stale) Mirror 7: 0x3d6a8264 (stale) Mirror 8: 0x3d6a8264 (stale) Mirror 12: 0x3d6a8264 (stale) Mirror 16: 0x3d6a8264 (stale) Mirror 17: 0x3d6a8264 (stale) Mirror 18: 0x3d6a8264 (stale) Mirror 19: 0x3d6a8264 (stale) Mirror 21: 0x3d6a8264 (stale) Mirror 23: 0x3d6a8264 (stale) Mirror 24: 0x3d6a8264 (stale) Mirror 25: 0x3d6a8264 (stale) Mirror 27: 0x7a807a74 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f579): Mirror 6: 0x3d6a8264 (stale) Mirror 7: 0x3d6a8264 (stale) Mirror 8: 0x3d6a8264 (stale) Mirror 12: 0x3d6a8264 (stale) Mirror 16: 0x3d6a8264 (stale) Mirror 17: 0x3d6a8264 (stale) Mirror 18: 0x3d6a8264 (stale) Mirror 19: 0x3d6a8264 (stale) Mirror 21: 0x3d6a8264 (stale) Mirror 23: 0x3d6a8264 (stale) Mirror 24: 0x3d6a8264 (stale) Mirror 25: 0x3d6a8264 (stale) Mirror 27: 0x7a807a74 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f579: skipped 000668 1781220779.351656 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x1f579): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1f579): Mirror 6: 0x3d6a8264 (stale) Mirror 7: 0x3d6a8264 (stale) Mirror 8: 0x3d6a8264 (stale) Mirror 12: 0x3d6a8264 (stale) Mirror 16: 0x3d6a8264 (stale) Mirror 17: 0x3d6a8264 (stale) Mirror 18: 0x3d6a8264 (stale) Mirror 19: 0x3d6a8264 (stale) Mirror 21: 0x3d6a8264 (stale) Mirror 23: 0x3d6a8264 (stale) Mirror 24: 0x3d6a8264 (stale) Mirror 25: 0x3d6a8264 (stale) Mirror 27: 0x7a807a74 (stale) CRC-32 checksum value for chunk [0, 0x1f579): Mirror 1: 0x67369182 Mirror 28: 0x67369182 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1f579: skipped 000669 1781220780.652568 trunc from 0x01f579 to 0x016540 (0x9038 bytes) 000671 1781220780.781678 mapwrite 0x01c6ad thru 0x01e97c (0x22d0 bytes) 000673 1781220780.886764 mapread 0x00278d thru 0x0106ff (0xdf73 bytes) 000674 1781220780.904318 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x1e97d): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1e97d): Mirror 6: 0x315bbba5 (stale) Mirror 7: 0x315bbba5 (stale) Mirror 8: 0x315bbba5 (stale) Mirror 12: 0x315bbba5 (stale) Mirror 16: 0x315bbba5 (stale) Mirror 17: 0x315bbba5 (stale) Mirror 18: 0x315bbba5 (stale) Mirror 19: 0x315bbba5 (stale) Mirror 21: 0x315bbba5 (stale) Mirror 23: 0x315bbba5 (stale) Mirror 24: 0x315bbba5 (stale) Mirror 25: 0x315bbba5 (stale) Mirror 27: 0x6ed4d7a9 (stale) Mirror 28: 0x653398c0 (stale) CRC-32 checksum value for chunk [0, 0x1e97d): Mirror 1: 0xb434ac7c Mirror 29: 0xb434ac7c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1e97d: skipped 000675 1781220782.193370 read 0x0083d1 thru 0x00d883 (0x54b3 bytes) 000676 1781220782.207169 read 0x00ffd6 thru 0x01925a (0x9285 bytes) 000679 1781220782.230812 trunc from 0x01e97d to 0x02486b (0x5eef bytes) 000680 1781220782.347678 MIRROR_SPLIT mirror 12 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2486c): Mirror 6: 0x44ad5838 (stale) Mirror 7: 0x44ad5838 (stale) Mirror 8: 0x44ad5838 (stale) Mirror 12: 0x44ad5838 (stale) Mirror 16: 0x44ad5838 (stale) Mirror 17: 0x44ad5838 (stale) Mirror 18: 0x44ad5838 (stale) Mirror 19: 0x44ad5838 (stale) Mirror 21: 0x44ad5838 (stale) Mirror 23: 0x44ad5838 (stale) Mirror 24: 0x44ad5838 (stale) Mirror 27: 0x6446759e (stale) Mirror 28: 0x9d2bfd7a (stale) Mirror 29: 0xb6e296 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2486c): Mirror 6: 0x44ad5838 (stale) Mirror 7: 0x44ad5838 (stale) Mirror 8: 0x44ad5838 (stale) Mirror 12: 0x44ad5838 (stale) Mirror 16: 0x44ad5838 (stale) Mirror 17: 0x44ad5838 (stale) Mirror 18: 0x44ad5838 (stale) Mirror 19: 0x44ad5838 (stale) Mirror 21: 0x44ad5838 (stale) Mirror 23: 0x44ad5838 (stale) Mirror 24: 0x44ad5838 (stale) Mirror 27: 0x6446759e (stale) Mirror 28: 0x9d2bfd7a (stale) Mirror 29: 0xb6e296 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2486c: skipped 000682 1781220783.222937 write 0x01a20e thru 0x023686 (0x9479 bytes) 000685 1781220783.234443 mapwrite 0x0169e8 thru 0x0190e9 (0x2702 bytes) 000687 1781220783.306436 mapread 0x016047 thru 0x01c34f (0x6309 bytes) 000689 1781220783.313441 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x2486c): Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2486c): Mirror 6: 0x44ad5838 (stale) Mirror 7: 0x44ad5838 (stale) Mirror 8: 0x44ad5838 (stale) Mirror 12: 0x44ad5838 (stale) Mirror 16: 0x44ad5838 (stale) Mirror 17: 0x44ad5838 (stale) Mirror 18: 0x44ad5838 (stale) Mirror 19: 0x44ad5838 (stale) Mirror 21: 0x44ad5838 (stale) Mirror 23: 0x44ad5838 (stale) Mirror 24: 0x44ad5838 (stale) Mirror 27: 0x6446759e (stale) Mirror 28: 0x9d2bfd7a (stale) Mirror 29: 0xb6e296 (stale) CRC-32 checksum value for chunk [0, 0x2486c): Mirror 1: 0xdb6764a8 Mirror 30: 0xdb6764a8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2486c: skipped 000690 1781220784.785689 mapread 0x0187bc thru 0x01e31b (0x5b60 bytes) 000691 1781220784.812862 write 0x018108 thru 0x0227fd (0xa6f6 bytes) 000692 1781220784.962017 trunc from 0x02486c to 0x03711b (0x128b0 bytes) 000697 1781220785.037513 MIRROR_SPLIT mirror 2 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3711c): Mirror 6: 0x9c5596ae (stale) Mirror 8: 0x9c5596ae (stale) Mirror 12: 0x9c5596ae (stale) Mirror 16: 0x9c5596ae (stale) Mirror 17: 0x9c5596ae (stale) Mirror 18: 0x9c5596ae (stale) Mirror 19: 0x9c5596ae (stale) Mirror 21: 0x9c5596ae (stale) Mirror 23: 0x9c5596ae (stale) Mirror 24: 0x9c5596ae (stale) Mirror 27: 0x22081508 (stale) Mirror 28: 0x7f73ad09 (stale) Mirror 29: 0xa13e5bda (stale) Mirror 30: 0xec35a23 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3711c): Mirror 6: 0x9c5596ae (stale) Mirror 8: 0x9c5596ae (stale) Mirror 12: 0x9c5596ae (stale) Mirror 16: 0x9c5596ae (stale) Mirror 17: 0x9c5596ae (stale) Mirror 18: 0x9c5596ae (stale) Mirror 19: 0x9c5596ae (stale) Mirror 21: 0x9c5596ae (stale) Mirror 23: 0x9c5596ae (stale) Mirror 24: 0x9c5596ae (stale) Mirror 27: 0x22081508 (stale) Mirror 28: 0x7f73ad09 (stale) Mirror 29: 0xa13e5bda (stale) Mirror 30: 0xec35a23 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3711c: skipped 000698 1781220785.878945 read 0x00324c thru 0x010222 (0xcfd7 bytes) 000700 1781220785.895153 mapwrite 0x011521 thru 0x01bc64 (0xa744 bytes) 000703 1781220785.973310 write 0x031f99 thru 0x03ffff (0xe067 bytes) 000706 1781220785.989370 mapwrite 0x027462 thru 0x02ca4c (0x55eb bytes) 000707 1781220786.048966 mapread 0x03169c thru 0x038693 (0x6ff8 bytes) 000710 1781220786.057824 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x18e0d9e6 (stale) Mirror 8: 0x18e0d9e6 (stale) Mirror 12: 0x18e0d9e6 (stale) Mirror 16: 0x18e0d9e6 (stale) Mirror 17: 0x18e0d9e6 (stale) Mirror 18: 0x18e0d9e6 (stale) Mirror 19: 0x18e0d9e6 (stale) Mirror 21: 0x18e0d9e6 (stale) Mirror 23: 0x18e0d9e6 (stale) Mirror 24: 0x18e0d9e6 (stale) Mirror 27: 0x687df3fa (stale) Mirror 28: 0x1378f5ab (stale) Mirror 29: 0xa04645f7 (stale) Mirror 30: 0x4277a91a (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x5fdd3f54 Mirror 31: 0x5fdd3f54 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000713 1781220787.404525 read 0x00935b thru 0x018e1a (0xfac0 bytes) 000714 1781220787.415174 trunc from 0x040000 to 0x02b240 (0x14dbf bytes) 000715 1781220787.607992 trunc from 0x02b241 to 0x01922a (0x12016 bytes) 000716 1781220787.677312 MIRROR_RESYNC 16 mirrors 000717 1781220790.128126 trunc from 0x01922b to 0x000ceb (0x1853f bytes) 000718 1781220790.288787 read 0x00015b thru 0x000ceb (0xb91 bytes) 000719 1781220790.305067 MIRROR_RESYNC 16 mirrors 000720 1781220792.764483 read 0x00089a thru 0x000ceb (0x452 bytes) 000721 1781220792.777037 write 0x0009ea thru 0x00e048 (0xd65f bytes) 000725 1781220792.944473 write 0x00bf69 thru 0x00de27 (0x1ebf bytes) 000726 1781220792.948663 write 0x007da2 thru 0x0142eb (0xc54a bytes) 000729 1781220792.959140 mapread 0x00445b thru 0x009267 (0x4e0d bytes) 000732 1781220792.963129 write 0x02f613 thru 0x030726 (0x1114 bytes) 000733 1781220792.965799 trunc from 0x030727 to 0x027350 (0x93d6 bytes) 000734 1781220793.004889 trunc from 0x027351 to 0x032313 (0xafc3 bytes) 000735 1781220793.025022 write 0x01963d thru 0x02247f (0x8e43 bytes) 000736 1781220793.041255 MIRROR_SPLIT mirror 6 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32314): Mirror 6: 0x504b7585 (stale) Mirror 8: 0x504b7585 (stale) Mirror 12: 0x504b7585 (stale) Mirror 16: 0x504b7585 (stale) Mirror 17: 0x504b7585 (stale) Mirror 19: 0x504b7585 (stale) Mirror 21: 0x504b7585 (stale) Mirror 23: 0x504b7585 (stale) Mirror 24: 0x504b7585 (stale) Mirror 27: 0x504b7585 (stale) Mirror 28: 0x504b7585 (stale) Mirror 29: 0x504b7585 (stale) Mirror 30: 0x504b7585 (stale) Mirror 31: 0x504b7585 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32314): Mirror 6: 0x504b7585 (stale) Mirror 8: 0x504b7585 (stale) Mirror 12: 0x504b7585 (stale) Mirror 16: 0x504b7585 (stale) Mirror 17: 0x504b7585 (stale) Mirror 19: 0x504b7585 (stale) Mirror 21: 0x504b7585 (stale) Mirror 23: 0x504b7585 (stale) Mirror 24: 0x504b7585 (stale) Mirror 27: 0x504b7585 (stale) Mirror 28: 0x504b7585 (stale) Mirror 29: 0x504b7585 (stale) Mirror 30: 0x504b7585 (stale) Mirror 31: 0x504b7585 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32314: skipped 000737 1781220793.941633 mapwrite 0x009dfb thru 0x016c34 (0xce3a bytes) 000740 1781220794.174237 mapread 0x01f9e8 thru 0x02605a (0x6673 bytes) 000742 1781220794.207455 mapread 0x005b23 thru 0x00eaf4 (0x8fd2 bytes) 000743 1781220794.209012 write 0x001c0d thru 0x00cd52 (0xb146 bytes) 000746 1781220794.244343 read 0x00d8f8 thru 0x0136bf (0x5dc8 bytes) 000748 1781220794.248157 mapwrite 0x0219ac thru 0x03036b (0xe9c0 bytes) 000749 1781220794.400983 MIRROR_SPLIT mirror 6 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x32314): Mirror 6: 0x504b7585 (stale) Mirror 8: 0x504b7585 (stale) Mirror 12: 0x504b7585 (stale) Mirror 16: 0x504b7585 (stale) Mirror 17: 0x504b7585 (stale) Mirror 21: 0x504b7585 (stale) Mirror 23: 0x504b7585 (stale) Mirror 24: 0x504b7585 (stale) Mirror 27: 0x504b7585 (stale) Mirror 28: 0x504b7585 (stale) Mirror 29: 0x504b7585 (stale) Mirror 30: 0x504b7585 (stale) Mirror 31: 0x504b7585 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x32314): Mirror 6: 0x504b7585 (stale) Mirror 8: 0x504b7585 (stale) Mirror 12: 0x504b7585 (stale) Mirror 16: 0x504b7585 (stale) Mirror 17: 0x504b7585 (stale) Mirror 21: 0x504b7585 (stale) Mirror 23: 0x504b7585 (stale) Mirror 24: 0x504b7585 (stale) Mirror 27: 0x504b7585 (stale) Mirror 28: 0x504b7585 (stale) Mirror 29: 0x504b7585 (stale) Mirror 30: 0x504b7585 (stale) Mirror 31: 0x504b7585 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x32314: skipped 000750 1781220795.455170 mapwrite 0x0281da thru 0x037789 (0xf5b0 bytes) 000752 1781220795.565717 trunc from 0x03778a to 0x02eb0c (0x8c7d bytes) 000755 1781220795.590335 trunc from 0x02eb0d to 0x004553 (0x2a5b9 bytes) 000758 1781220795.616739 trunc from 0x004554 to 0x005401 (0xeae bytes) 000760 1781220795.652072 read 0x00136e thru 0x005401 (0x4094 bytes) 000761 1781220795.660964 write 0x004e71 thru 0x009ac6 (0x4c56 bytes) 000762 1781220795.662169 mapread 0x003ca9 thru 0x004b57 (0xeaf bytes) 000763 1781220795.667931 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x9ac7): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9ac7): Mirror 6: 0x5925bde4 (stale) Mirror 8: 0x5925bde4 (stale) Mirror 12: 0x5925bde4 (stale) Mirror 16: 0x5925bde4 (stale) Mirror 17: 0x5925bde4 (stale) Mirror 21: 0x5925bde4 (stale) Mirror 23: 0x5925bde4 (stale) Mirror 24: 0x5925bde4 (stale) Mirror 27: 0x5925bde4 (stale) Mirror 28: 0x5925bde4 (stale) Mirror 29: 0x5925bde4 (stale) Mirror 30: 0x5925bde4 (stale) Mirror 31: 0x5925bde4 (stale) CRC-32 checksum value for chunk [0, 0x9ac7): Mirror 1: 0x58603ec6 Mirror 32: 0x58603ec6 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9ac7: skipped 000764 1781220796.961539 trunc from 0x009ac7 to 0x010997 (0x6ed1 bytes) 000765 1781220797.072264 mapread 0x0083a6 thru 0x010997 (0x85f2 bytes) 000770 1781220797.097559 mapwrite 0x00968b thru 0x00dfe1 (0x4957 bytes) 000771 1781220797.149324 read 0x00a8d0 thru 0x00cfa2 (0x26d3 bytes) 000773 1781220797.150910 mapwrite 0x020ec0 thru 0x024dd9 (0x3f1a bytes) 000774 1781220797.236642 read 0x0189ac thru 0x024dd9 (0xc42e bytes) 000775 1781220797.250885 mapread 0x016a8d thru 0x01c418 (0x598c bytes) 000776 1781220797.252372 mapread 0x01cebd thru 0x024dd9 (0x7f1d bytes) 000777 1781220797.254375 read 0x00c5df thru 0x0120bf (0x5ae1 bytes) 000779 1781220797.256843 trunc from 0x024dda to 0x00f1c7 (0x15c12 bytes) 000781 1781220797.284449 mapread 0x00c75f thru 0x00f1c7 (0x2a69 bytes) 000783 1781220797.295004 MIRROR_RESYNC 15 mirrors 000784 1781220799.645356 mapread 0x0034dd thru 0x009605 (0x6129 bytes) 000786 1781220799.677042 mapread 0x00c0a2 thru 0x00f1c7 (0x3126 bytes) 000789 1781220799.681282 mapwrite 0x03b391 thru 0x03ffff (0x4c6f bytes) 000792 1781220800.016919 MIRROR_EXTEND to 16 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 12: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 24: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x282575f3 (stale) Mirror 8: 0x282575f3 (stale) Mirror 12: 0x282575f3 (stale) Mirror 16: 0x282575f3 (stale) Mirror 17: 0x282575f3 (stale) Mirror 21: 0x282575f3 (stale) Mirror 23: 0x282575f3 (stale) Mirror 24: 0x282575f3 (stale) Mirror 27: 0x282575f3 (stale) Mirror 28: 0x282575f3 (stale) Mirror 29: 0x282575f3 (stale) Mirror 30: 0x282575f3 (stale) Mirror 31: 0x282575f3 (stale) Mirror 32: 0x282575f3 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf848adab Mirror 33: 0xf848adab lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000793 1781220801.627169 mapwrite 0x00d2dc thru 0x00f714 (0x2439 bytes) 000794 1781220801.860888 mapread 0x01190d thru 0x01b757 (0x9e4b bytes) 000795 1781220801.904657 MIRROR_RESYNC 16 mirrors 000796 1781220804.842235 write 0x02170c thru 0x0236e0 (0x1fd5 bytes) 000797 1781220804.949493 MIRROR_RESYNC 16 mirrors 000798 1781220808.211550 mapwrite 0x01e841 thru 0x023b21 (0x52e1 bytes) 000802 1781220808.450170 read 0x0159d3 thru 0x01c2d8 (0x6906 bytes) 000804 1781220808.488646 MIRROR_RESYNC 16 mirrors 000805 1781220811.135831 mapread 0x03774e thru 0x03d021 (0x58d4 bytes) 000808 1781220811.195144 mapwrite 0x003488 thru 0x004681 (0x11fa bytes) 000809 1781220811.474991 mapread 0x03eecc thru 0x03f5e7 (0x71c bytes) 000814 1781220811.517526 MIRROR_SPLIT mirror 3 to 15 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x4811adac (stale) Mirror 8: 0x4811adac (stale) Mirror 16: 0x4811adac (stale) Mirror 17: 0x4811adac (stale) Mirror 21: 0x4811adac (stale) Mirror 23: 0x4811adac (stale) Mirror 24: 0x4811adac (stale) Mirror 27: 0x4811adac (stale) Mirror 28: 0x4811adac (stale) Mirror 29: 0x4811adac (stale) Mirror 30: 0x4811adac (stale) Mirror 31: 0x4811adac (stale) Mirror 32: 0x4811adac (stale) Mirror 33: 0x4811adac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x4811adac (stale) Mirror 8: 0x4811adac (stale) Mirror 16: 0x4811adac (stale) Mirror 17: 0x4811adac (stale) Mirror 21: 0x4811adac (stale) Mirror 23: 0x4811adac (stale) Mirror 24: 0x4811adac (stale) Mirror 27: 0x4811adac (stale) Mirror 28: 0x4811adac (stale) Mirror 29: 0x4811adac (stale) Mirror 30: 0x4811adac (stale) Mirror 31: 0x4811adac (stale) Mirror 32: 0x4811adac (stale) Mirror 33: 0x4811adac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000815 1781220812.487455 mapwrite 0x030cce thru 0x03c2fb (0xb62e bytes) 000817 1781220812.589063 read 0x021d40 thru 0x024785 (0x2a46 bytes) 000818 1781220812.610435 MIRROR_SPLIT mirror 14 to 14 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x4811adac (stale) Mirror 8: 0x4811adac (stale) Mirror 16: 0x4811adac (stale) Mirror 17: 0x4811adac (stale) Mirror 21: 0x4811adac (stale) Mirror 23: 0x4811adac (stale) Mirror 24: 0x4811adac (stale) Mirror 27: 0x4811adac (stale) Mirror 28: 0x4811adac (stale) Mirror 29: 0x4811adac (stale) Mirror 30: 0x4811adac (stale) Mirror 31: 0x4811adac (stale) Mirror 32: 0x4811adac (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x4811adac (stale) Mirror 8: 0x4811adac (stale) Mirror 16: 0x4811adac (stale) Mirror 17: 0x4811adac (stale) Mirror 21: 0x4811adac (stale) Mirror 23: 0x4811adac (stale) Mirror 24: 0x4811adac (stale) Mirror 27: 0x4811adac (stale) Mirror 28: 0x4811adac (stale) Mirror 29: 0x4811adac (stale) Mirror 30: 0x4811adac (stale) Mirror 31: 0x4811adac (stale) Mirror 32: 0x4811adac (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000820 1781220813.588247 trunc from 0x040000 to 0x00eeae (0x31151 bytes) 000823 1781220813.630768 MIRROR_RESYNC 14 mirrors 000824 1781220816.013115 mapwrite 0x02e442 thru 0x032ef2 (0x4ab1 bytes) 000826 1781220816.215842 write 0x0204b6 thru 0x02e1ca (0xdd15 bytes) 000827 1781220816.228357 mapread 0x003320 thru 0x0100f5 (0xcdd6 bytes) 000828 1781220816.240000 trunc from 0x032ef3 to 0x0100b9 (0x22e39 bytes) 000829 1781220816.267457 MIRROR_RESYNC 14 mirrors 000832 1781220818.499271 write 0x0281e0 thru 0x035f13 (0xdd34 bytes) 000833 1781220818.598220 read 0x012d7f thru 0x013352 (0x5d4 bytes) 000834 1781220818.609651 MIRROR_SPLIT mirror 7 to 13 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x161:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35f14): Mirror 6: 0x76d709b3 (stale) Mirror 8: 0x76d709b3 (stale) Mirror 16: 0x76d709b3 (stale) Mirror 17: 0x76d709b3 (stale) Mirror 21: 0x76d709b3 (stale) Mirror 23: 0x76d709b3 (stale) Mirror 27: 0x76d709b3 (stale) Mirror 28: 0x76d709b3 (stale) Mirror 29: 0x76d709b3 (stale) Mirror 30: 0x76d709b3 (stale) Mirror 31: 0x76d709b3 (stale) Mirror 32: 0x76d709b3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35f14): Mirror 6: 0x76d709b3 (stale) Mirror 8: 0x76d709b3 (stale) Mirror 16: 0x76d709b3 (stale) Mirror 17: 0x76d709b3 (stale) Mirror 21: 0x76d709b3 (stale) Mirror 23: 0x76d709b3 (stale) Mirror 27: 0x76d709b3 (stale) Mirror 28: 0x76d709b3 (stale) Mirror 29: 0x76d709b3 (stale) Mirror 30: 0x76d709b3 (stale) Mirror 31: 0x76d709b3 (stale) Mirror 32: 0x76d709b3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35f14: skipped 000835 1781220819.661913 mapwrite 0x038d8e thru 0x03eab2 (0x5d25 bytes) 000836 1781220819.861170 trunc from 0x03eab3 to 0x029a48 (0x1506a bytes) 000837 1781220819.897270 mapwrite 0x011f76 thru 0x0187d2 (0x685d bytes) 000838 1781220819.991303 read 0x01c283 thru 0x026eb1 (0xac2f bytes) 000839 1781220820.008825 MIRROR_RESYNC 13 mirrors 000841 1781220823.066013 read 0x008270 thru 0x0155b4 (0xd345 bytes) 000843 1781220823.085089 mapread 0x02294a thru 0x029a48 (0x70ff bytes) 000844 1781220823.112593 mapread 0x017530 thru 0x01f846 (0x8317 bytes) 000847 1781220823.151371 MIRROR_RESYNC 13 mirrors 000848 1781220824.186593 read 0x0100bb thru 0x010d9c (0xce2 bytes) 000849 1781220824.187064 mapread 0x0166c3 thru 0x0224a7 (0xbde5 bytes) 000851 1781220824.188516 write 0x004db4 thru 0x01185d (0xcaaa bytes) 000852 1781220824.379141 read 0x0172cf thru 0x024dec (0xdb1e bytes) 000853 1781220824.398683 write 0x002aa6 thru 0x0031bb (0x716 bytes) 000855 1781220824.400302 mapread 0x0199e4 thru 0x01d6fa (0x3d17 bytes) 000857 1781220824.414644 mapread 0x01ab59 thru 0x022d33 (0x81db bytes) 000859 1781220824.420606 write 0x01de6d thru 0x0269d7 (0x8b6b bytes) 000862 1781220824.423104 mapread 0x0234c6 thru 0x029a48 (0x6583 bytes) 000864 1781220824.430128 mapwrite 0x019039 thru 0x021f21 (0x8ee9 bytes) 000866 1781220824.533362 read 0x029255 thru 0x029a48 (0x7f4 bytes) 000869 1781220824.541579 write 0x0232e0 thru 0x0268e8 (0x3609 bytes) 000871 1781220824.545231 write 0x0093b1 thru 0x0172df (0xdf2f bytes) 000873 1781220824.548780 write 0x03a999 thru 0x03ffff (0x5667 bytes) 000874 1781220824.550287 write 0x0204bf thru 0x02e256 (0xdd98 bytes) 000875 1781220824.559660 mapread 0x02d04e thru 0x030237 (0x31ea bytes) 000879 1781220824.574010 write 0x01f1d5 thru 0x021d2a (0x2b56 bytes) 000880 1781220824.575863 write 0x00361c thru 0x010dcf (0xd7b4 bytes) 000881 1781220824.577900 write 0x00a71b thru 0x00cdb9 (0x269f bytes) 000884 1781220824.581571 mapwrite 0x020bee thru 0x02125c (0x66f bytes) 000885 1781220824.670705 trunc from 0x040000 to 0x00cc38 (0x333c7 bytes) 000886 1781220824.724122 write 0x01a535 thru 0x02825e (0xdd2a bytes) 000887 1781220824.741242 MIRROR_EXTEND to 14 mirrors CRC-32 checksum value for chunk [0, 0x2825f): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2825f): Mirror 6: 0x53ce8eca (stale) Mirror 8: 0x53ce8eca (stale) Mirror 16: 0x53ce8eca (stale) Mirror 17: 0x53ce8eca (stale) Mirror 21: 0x53ce8eca (stale) Mirror 23: 0x53ce8eca (stale) Mirror 27: 0x53ce8eca (stale) Mirror 28: 0x53ce8eca (stale) Mirror 29: 0x53ce8eca (stale) Mirror 30: 0x53ce8eca (stale) Mirror 31: 0x53ce8eca (stale) Mirror 32: 0x53ce8eca (stale) CRC-32 checksum value for chunk [0, 0x2825f): Mirror 1: 0xf1901589 Mirror 33: 0xf1901589 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2825f: skipped 000888 1781220826.333556 read 0x013c32 thru 0x017baf (0x3f7e bytes) 000889 1781220826.345522 write 0x00a73d thru 0x0172e3 (0xcba7 bytes) 000890 1781220826.535130 mapread 0x02139c thru 0x024a57 (0x36bc bytes) 000891 1781220826.562364 MIRROR_RESYNC 14 mirrors 000892 1781220829.585955 write 0x00cfcc thru 0x01bb83 (0xebb8 bytes) 000893 1781220829.731519 MIRROR_RESYNC 14 mirrors 000895 1781220832.471840 mapwrite 0x0184ca thru 0x01a976 (0x24ad bytes) 000897 1781220832.670507 MIRROR_EXTEND to 15 mirrors CRC-32 checksum value for chunk [0, 0x2825f): Mirror 6: (stale - read error) Mirror 8: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) Mirror 21: (stale - read error) Mirror 23: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) Mirror 31: (stale - read error) Mirror 32: (stale - read error) Mirror 33: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2825f): Mirror 6: 0x7b1483cb (stale) Mirror 8: 0x7b1483cb (stale) Mirror 16: 0x7b1483cb (stale) Mirror 17: 0x7b1483cb (stale) Mirror 21: 0x7b1483cb (stale) Mirror 23: 0x7b1483cb (stale) Mirror 27: 0x7b1483cb (stale) Mirror 28: 0x7b1483cb (stale) Mirror 29: 0x7b1483cb (stale) Mirror 30: 0x7b1483cb (stale) Mirror 31: 0x7b1483cb (stale) Mirror 32: 0x7b1483cb (stale) Mirror 33: 0x7b1483cb (stale) CRC-32 checksum value for chunk [0, 0x2825f): Mirror 1: 0x2a7d6d41 Mirror 34: 0x2a7d6d41 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2825f: skipped 000898 1781220834.125198 write 0x038eb5 thru 0x03e865 (0x59b1 bytes) 000904 1781220834.245053 mapwrite 0x009627 thru 0x00e3a4 (0x4d7e bytes) 000905 1781220834.389139 trunc from 0x03e866 to 0x0157ad (0x290b8 bytes) 000907 1781220834.446915 mapwrite 0x02a7b7 thru 0x02ebe0 (0x442a bytes) 000908 1781220834.556357 write 0x001369 thru 0x009094 (0x7d2c bytes) 000910 1781220834.563467 write 0x02e808 thru 0x03b722 (0xcf1b bytes) 000912 1781220834.569687 mapread 0x035b1b thru 0x037dc0 (0x22a6 bytes) 000913 1781220834.580208 MIRROR_RESYNC 15 mirrors 000914 1781220837.079078 write 0x00fdcd thru 0x01c48d (0xc6c1 bytes) 000916 1781220837.220889 read 0x005828 thru 0x00bdbb (0x6594 bytes) 000919 1781220837.243128 mapwrite 0x034341 thru 0x038e4a (0x4b0a bytes) 000920 1781220837.364138 trunc from 0x03b723 to 0x0385db (0x3147 bytes) 000923 1781220837.389345 trunc from 0x0385dc to 0x016774 (0x21e67 bytes) 000924 1781220837.410545 mapread 0x010ea9 thru 0x016774 (0x58cc bytes) 000927 1781220837.414311 write 0x008109 thru 0x00d667 (0x555f bytes) 000929 1781220837.421211 read 0x0117b1 thru 0x011cee (0x53e bytes) 000930 1781220837.421925 mapwrite 0x039847 thru 0x03eaac (0x5266 bytes) 000931 1781220837.518136 trunc from 0x03eaad to 0x0269a6 (0x18106 bytes) 000934 1781220837.542621 write 0x018548 thru 0x01fe93 (0x794c bytes) 000935 1781220837.546914 trunc from 0x0269a7 to 0x00d87a (0x1912c bytes) 000937 1781220837.584415 mapwrite 0x039f69 thru 0x03ffff (0x6097 bytes) 000940 1781220837.695995 mapwrite 0x03f10a thru 0x03ffff (0xef6 bytes) 000943 1781220837.776245 read 0x035ede thru 0x037504 (0x1627 bytes) 000944 1781220837.790527 write 0x00e5a5 thru 0x01e3fe (0xfe5a bytes) 000945 1781220837.792906 read 0x00e82d thru 0x018eb0 (0xa684 bytes) 000947 1781220837.794480 mapwrite 0x0226b8 thru 0x02770a (0x5053 bytes) 000948 1781220837.845284 mapread 0x039a2e thru 0x03ffff (0x65d2 bytes) 000951 1781220837.849857 trunc from 0x040000 to 0x017842 (0x287bd bytes) 000952 1781220837.874513 mapread 0x00eaf8 thru 0x017842 (0x8d4b bytes) 000953 1781220837.879852 mapwrite 0x03e5c0 thru 0x03ffff (0x1a40 bytes) 000956 1781220837.980133 write 0x03c5fa thru 0x03ffff (0x3a06 bytes) 000957 1781220837.990212 write 0x0056f7 thru 0x00d101 (0x7a0b bytes) 000959 1781220837.996563 write 0x015761 thru 0x020c74 (0xb514 bytes) 000960 1781220837.999076 read 0x02851d thru 0x032832 (0xa316 bytes) 000961 1781220838.000099 trunc from 0x040000 to 0x00fc6f (0x30390 bytes) 000963 1781220838.048510 mapread 0x008606 thru 0x00a91c (0x2317 bytes) 000965 1781220838.050371 trunc from 0x00fc70 to 0x03a680 (0x2aa11 bytes) 000970 1781220838.077179 trunc from 0x03a681 to 0x01af86 (0x1f6fa bytes) 000971 1781220838.100542 read 0x0019dd thru 0x00c074 (0xa698 bytes) 000972 1781220838.103023 mapwrite 0x003b0c thru 0x007d62 (0x4257 bytes) 000975 1781220838.261115 read 0x00db20 thru 0x0143fe (0x68df bytes) 000976 1781220838.288302 write 0x004a8d thru 0x00ed7a (0xa2ee bytes) 000977 1781220838.296311 read 0x007374 thru 0x00e63c (0x72c9 bytes) 000978 1781220838.297109 mapread 0x00cfda thru 0x017b94 (0xabbb bytes) 000980 1781220838.307727 trunc from 0x01af87 to 0x03e603 (0x2367d bytes) 000981 1781220838.354111 write 0x02667e thru 0x02a5b4 (0x3f37 bytes) 000982 1781220838.370156 mapread 0x0399c2 thru 0x03e603 (0x4c42 bytes) 000983 1781220838.371910 write 0x0361a2 thru 0x03efe3 (0x8e42 bytes) 000984 1781220838.388346 MIRROR_RESYNC 15 mirrors 000985 1781220841.420074 mapread 0x023707 thru 0x026915 (0x320f bytes) 000986 1781220841.455724 MIRROR_EXTEND to 16 mirrors 000988 1781220842.935336 mapread 0x0199ad thru 0x01e554 (0x4ba8 bytes) 000989 1781220842.962805 write 0x01956b thru 0x024565 (0xaffb bytes) 000990 1781220843.095379 mapread 0x039479 thru 0x0398a2 (0x42a bytes) 000992 1781220843.112174 write 0x00ef3a thru 0x016201 (0x72c8 bytes) 000993 1781220843.125670 mapread 0x00fdad thru 0x01683a (0x6a8e bytes) 000994 1781220843.127123 mapwrite 0x0242b5 thru 0x024af1 (0x83d bytes) 000995 1781220843.215459 mapwrite 0x006e15 thru 0x00fd38 (0x8f24 bytes) 000996 1781220843.330998 read 0x017a8c thru 0x01d4f5 (0x5a6a bytes) 000999 1781220843.345941 mapread 0x017e73 thru 0x01d24a (0x53d8 bytes) All operations completed A-OK! PASS 70a (168s) == sanity-flr test 71: check mirror extend parent fid ==== 19:34:08 (1781220848) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 19:34:12 (1781220852) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.424442 s, 4.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301025 s, 3.5 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2a9:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2aa:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28e:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28d:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ab:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 19:34:23 (1781220863) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 2390816 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6921664 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 4548064 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7226880 .. Extending file size to 7584928 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7520224 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8222432 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8505472 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8213984 .. Extending file size to 8701632 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8725632 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 9024160 .. Extending file size to 8804704 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8812640 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57178 57179 57180 57183 57184 failed 192.168.204.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre2 (opts:) 192.168.204.132@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (79s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 19:35:42 (1781220942) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 1005536 .. Extending file size to 9022400 .. failed Extending file size to 2727488 .. Extending file size to 8305760 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 8802976 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9058912 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9052096 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58805 58806 58807 58810 58811 done 192.168.204.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre2 (opts:) 192.168.204.132@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (84s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 19:37:06 (1781221026) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3072 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3072 2205440 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0978203 s, 21.4 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.411558 s, 5.1 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0xe33d99de (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.204.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre2 (opts:) PASS 200c (22s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 19:37:29 (1781221049) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.101641 s, 20.6 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 19:37:37 (1781221057) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1ab:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2aa:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ab:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x2c6:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 19:37:45 (1781221065) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.167691 s, 12.5 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ac:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c7:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 19:37:53 (1781221073) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.164811 s, 12.7 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ad:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ae:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2c9:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ca:0x0] } PASS 204b (8s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 19:38:01 (1781221081) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.220585 s, 9.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2af:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cb:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cc:0x0] } PASS 204c (33s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 19:38:34 (1781221114) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg432-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124067 s, 8.5 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b0:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cd:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b1:0x0] } PASS 204d (67s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 19:39:41 (1781221181) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg432-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.365602 s, 5.7 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2cf:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d0:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b2:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (60s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 19:40:41 (1781221241) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg432-server mds-ost sync done. Creating new pool test_204f oleg432-server: Pool lustre.test_204f created Adding targets to pool oleg432-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg432-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.23819 s, 8.8 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x2d1:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2b3:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg432-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg432-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg432-server: Pool lustre.test_204f destroyed PASS 204f (78s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 19:41:59 (1781221319) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 19:42:09 (1781221329) PASS 205b (8s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 19:42:17 (1781221337) oleg432-server: Pool lustre.test_206 created Adding targets to pool oleg432-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg432-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg432-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg432-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg432-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 19:42:44 (1781221364) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123084 s, 8.5 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579638 s, 18.1 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xdf2e622 Mirror 3: 0xdf2e622 PASS 207 (15s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 19:42:58 (1781221378) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (5s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 19:43:03 (1781221383) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 19:43:07 (1781221387) Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.10929 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1480000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1d00000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1d00000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1d00000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xe00000 Read this (4 bytes): BBBB 192.168.204.132@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg432-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 19:43:45 (1781221425) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0222001 s, 369 kB/s the very first read read with warm cache PASS 209b (7s) == sanity-flr test 210a: handle broken mirrored lovea ==== 19:43:53 (1781221433) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504084 s, 20.8 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1d0 lmm_fid: [0x200000404:0x1d0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1d0 lmm_fid: [0x200000404:0x1d0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2bd:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4102 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834384 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-11 19:43:55.000000000 -0400 Modify: 2026-06-11 19:43:55.000000000 -0400 Change: 2026-06-11 19:43:56.000000000 -0400 Birth: 2026-06-11 19:43:55.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 19:44:02 (1781221442) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (5s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 19:44:06 (1781221446) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 1.29571 s, 31.6 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0695505 s, 58.9 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 11 19:44 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1d2:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 11 19:44 /mnt/lustre/f211.sanity-flr PASS 211 (14s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 19:44:20 (1781221460) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 19:44:30 (1781221470) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1899 sec =========== 19:44:41 (1781221481) === sanity-flr: start cleanup 19:44:42 (1781221482) === === sanity-flr: finish cleanup 19:44:50 (1781221490) ===