-----============= acceptance-small: sanity-flr ============----- Thu Sep 10 13:35:12 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg147-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 13:35:35 (1789061735) === oleg147-client.virtnet: executing check_config_client /mnt/lustre oleg147-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg147-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff95040614b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff95040614b000.idle_timeout=debug disable quota as required oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 13:36:21 (1789061781) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8221] == sanity-flr test 0a: lfs mirror create with -N option == 13:36:24 (1789061784) PASS 0a (18s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 13:36:42 (1789061802) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 13:36:46 (1789061806) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (4s) == sanity-flr test 0d: lfs mirror extend with -N option == 13:36:50 (1789061810) PASS 0d (24s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 13:37:14 (1789061834) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 13:37:17 (1789061837) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 13:37:21 (1789061841) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff95040614b000.stats=clear osc.lustre-OST0001-osc-ffff95040614b000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 13:37:29 (1789061849) striped dir -i0 -c2 -H all_char /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 13:37:41 (1789061861) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 13:37:51 (1789061871) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 13:37:55 (1789061875) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 13:38:04 (1789061884) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1e:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 13:38:14 (1789061894) PASS 2 (8s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 13:38:22 (1789061902) PASS 3 (9s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 13:38:32 (1789061912) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 13:38:39 (1789061919) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 13:38:50 (1789061930) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 13:38:58 (1789061938) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 1.56344 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142158 s, 7.4 MB/s PASS 21 (9s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 13:39:08 (1789061948) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 0.913948 s, 17.2 MB/s PASS 22 (12s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 13:39:20 (1789061960) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (13s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 13:39:33 (1789061973) Stopping /mnt/lustre-ost1 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95040614b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95040614b000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg147-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff95040614b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95040614b000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg147-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (82s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 13:40:55 (1789062055) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.708981 s, 14.8 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95040614b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95040614b000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 10485760 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg147-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (57s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 13:41:52 (1789062112) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95040614b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95040614b000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg147-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff95040614b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95040614b000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg147-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (85s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 13:43:18 (1789062198) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.07903 s, 17.0 MB/s -rw-r--r-- 1 root root 52428800 Sep 10 13:43 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m0.771s user 0m0.001s sys 0m0.156s Stopping /mnt/lustre-ost1 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff95040614b000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff95040614b000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m0.920s user 0m0.001s sys 0m0.175s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg147-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg147-server oleg147-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff95040614b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95040614b000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.088s user 0m0.000s sys 0m0.164s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg147-server: oleg147-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg147-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg147-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff95040614b000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff95040614b000.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 13:44:57 (1789062297) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (5s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 13:45:01 (1789062301) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 13:45:05 (1789062305) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 13:45:08 (1789062308) PASS 35 (7s) == sanity-flr test 36a: write to mirrored files ========== 13:45:16 (1789062316) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.699059 s, 18.0 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (12s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 13:45:28 (1789062328) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400759 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337287 s, 31.1 MB/s PASS 36b (8s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 13:45:37 (1789062337) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 13:45:46 (1789062346) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000756455 s, 1.4 MB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000910815 s, 1.1 MB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00134507 s, 761 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000802012 s, 1.3 MB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 13:45:58 (1789062358) create test files with size 15740656 .. ee5c606336748498f56b355cde6be41e - 4232f8fa22b47036f6c365f878baf2b2 - 8c247f5d00c8dcde67d773db808668c4 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg147-client.virtnet /mnt/lustre (opts:) Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre PASS 37 (26s) == sanity-flr test 38: resync ============================ 13:46:24 (1789062384) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 13:46:37 (1789062397) PASS 39 (10s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 13:46:47 (1789062407) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117491 s, 8.9 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x9a:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x9b:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd4:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717974 s, 14.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd5:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xd6:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0x9c:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 13:46:59 (1789062419) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.104461 s, 20.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.312515 s, 13.4 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.151695 s, 13.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 13:47:18 (1789062438) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (5s) == sanity-flr test 43a: mirror pick on write ============= 13:47:24 (1789062444) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (6s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 13:47:30 (1789062450) striped dir -i1 -c2 -H all_char /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317451 s, 33.0 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (10s) == sanity-flr test 44a: lfs mirror split check =========== 13:47:40 (1789062460) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.352852 s, 8.9 MB/s PASS 44a (10s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 13:47:51 (1789062471) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xa5:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xe0:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xe0:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (10s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 13:48:02 (1789062482) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.440514 s, 23.8 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x7:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg147-client.virtnet /mnt/lustre (opts:) Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (14s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 13:48:16 (1789062496) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff950421ac6800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.419736 s, 25.0 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg147-client.virtnet /mnt/lustre (opts:) Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff95040834d000.xattr_cache=1 PASS 44e (16s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 13:48:32 (1789062512) striped dir -i1 -c2 -H crush2 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (54s) == sanity-flr test 46: Verify setstripe --copy option ==== 13:49:26 (1789062566) striped dir -i0 -c2 -H crush /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H all_char /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (42s) == sanity-flr test 47: Verify mirror obj alloc =========== 13:50:08 (1789062608) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 13:50:12 (1789062612) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.391082 s, 8.0 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.179872 s, 17.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x9ea6a22c (nosync,stale) Mirror 2: 0x9ea6a22c (nosync,stale) ** verify mirror 2 doesn't change original checksum: 049216d28ed6ddb6d4dafc845d18c168 - mirror 1 checksum: 0805bb9c51c72aba19c618223fb2d517 - mirror 2 checksum: 049216d28ed6ddb6d4dafc845d18c168 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-10T13:50:18-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-10T13:50:18-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 049216d28ed6ddb6d4dafc845d18c168 - mirror 1 checksum: 0805bb9c51c72aba19c618223fb2d517 - mirror 2 checksum: 0805bb9c51c72aba19c618223fb2d517 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-10T13:50:18-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xaf:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-10T13:50:18-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x9 lmm_fid: [0x200000405:0x9:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb0:0x0] } PASS 48 (18s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 13:50:30 (1789062630) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0352799 s, 29.7 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 310272.. 311295: 1024: 0001: net,eof 1: 0.. 1023: 312320.. 313343: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (9s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 13:50:39 (1789062639) Failing mds1 on oleg147-server Stopping /mnt/lustre-mds1 (opts:) on oleg147-server 13:50:51 (1789062651) shut down facet: mds1 facet_host: oleg147-server facet_failover_host: oleg147-server Failover mds1 to oleg147-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 13:51:25 (1789062685) targets are mounted 13:51:25 (1789062685) facet_failover done oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0xf:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062205 s, 16.9 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xf lmm_fid: [0x200000405:0xf:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xb2:0x0] } PASS 50A (66s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 13:51:46 (1789062706) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0314734 s, 651 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0159656 s, 1.3 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (11s) == sanity-flr test 50b: mirror rsync handles sparseness == 13:51:56 (1789062716) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0709216 s, 289 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0135639 s, 1.5 MB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (12s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 13:52:08 (1789062728) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00998489 s, 1.6 MB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 13:52:17 (1789062737) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0584202 s, 351 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (13s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 13:52:30 (1789062750) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0170005 s, 241 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0322332 s, 127 kB/s PASS 60a (10s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 13:52:40 (1789062760) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x20:0x0]: Invalid argument (22) PASS 60b (25s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 13:53:05 (1789062785) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 13:53:35 (1789062815) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4a:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 13:54:00 (1789062840) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (29s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 13:54:30 (1789062870) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 13:54:38 (1789062878) striped dir -i0 -c2 -H all_char /mnt/lustre/d70.sanity-flr mirror create pid 57588 mirror split pid 57589 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57588 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5116: 57589 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 13:55:46 (1789062946) striped dir -i0 -c2 -H all_char /mnt/lustre/d70a.sanity-flr Seed set to 2948 000001 1789062948.764340 MIRROR_EXTEND to 2 mirrors 000002 1789062949.019632 mapwrite 0x03bc8d thru 0x03bf68 (0x2dc bytes) 000003 1789062949.075832 mapread 0x033af0 thru 0x03bf68 (0x8479 bytes) 000004 1789062949.097502 punch from 0x032f83 to 0x03bf68 (0x8fe6 bytes) 000005 1789062949.109878 write 0x013e17 thru 0x021a5c (0xdc46 bytes) 000007 1789062949.148456 write 0x009b9f thru 0x019b3e (0xffa0 bytes) 000008 1789062949.165266 read 0x000f13 thru 0x00c1c0 (0xb2ae bytes) fallocating to largest ever: 0x189dc 000009 1789062949.187896 fallocate 0x00b42b thru 0x0189db (0xd5b1 bytes) fallocating to largest ever: 0x3522d 000010 1789062949.198281 fallocate 0x034c0b thru 0x03522c (0x622 bytes) 000011 1789062949.206044 fallocate 0x0082af thru 0x014790 (0xc4e2 bytes) 000012 1789062949.216160 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3bf69): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf69): Mirror 2: 0x867cde26 (stale) CRC-32 checksum value for chunk [0, 0x3bf69): Mirror 1: 0x2039138b Mirror 3: 0x2039138b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf69: skipped 000013 1789062950.301983 write 0x01a70d thru 0x01fe52 (0x5746 bytes) 000014 1789062950.343536 mapread 0x036dc4 thru 0x03a3ba (0x35f7 bytes) 000015 1789062950.388237 mapwrite 0x01af90 thru 0x024402 (0x9473 bytes) 000016 1789062950.444460 punch from 0x0214a6 to 0x027b09 (0x6664 bytes) 000017 1789062950.469514 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3bf69): Mirror 2: 0x867cde26 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3bf69): Mirror 2: 0x867cde26 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3bf69: skipped 000018 1789062950.856330 mapwrite 0x0075f5 thru 0x00ee8a (0x7896 bytes) 000019 1789062950.908787 read 0x02e7aa thru 0x0381d3 (0x9a2a bytes) truncating to largest ever: 0x3673f 000020 1789062950.917516 trunc from 0x03bf69 to 0x03673e (0x582a bytes) 000021 1789062950.990453 read 0x02a804 thru 0x032aa9 (0x82a6 bytes) 000022 1789062951.004089 read 0x01f849 thru 0x029b94 (0xa34c bytes) 000023 1789062951.004844 trunc from 0x03673f to 0x013bf8 (0x22b46 bytes) 000024 1789062951.025967 trunc from 0x00eab3 to 0x013bf8 (0x5146 bytes) 000025 1789062951.031547 write 0x00a8e0 thru 0x012fff (0x8720 bytes) 000026 1789062951.033380 read 0x00b5e4 thru 0x013bf8 (0x8615 bytes) 000027 1789062951.043967 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x13bf9): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13bf9): Mirror 2: 0x16968ac2 (stale) CRC-32 checksum value for chunk [0, 0x13bf9): Mirror 1: 0xaf742708 Mirror 3: 0xaf742708 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13bf9: skipped fallocating to largest ever: 0x3c85b 000028 1789062951.826197 fallocate 0x0363ae thru 0x03c85a (0x64ad bytes) 000029 1789062951.873750 trunc from 0x01f676 to 0x029c24 (0xa5af bytes) 000030 1789062951.881615 punch from 0x03a633 to 0x03c85a (0x2228 bytes) 000031 1789062951.895307 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3c85b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c85b): Mirror 2: 0xdc58a46d (stale) Mirror 3: 0xec1cbf31 (stale) CRC-32 checksum value for chunk [0, 0x3c85b): Mirror 1: 0xec1cbf31 Mirror 4: 0xec1cbf31 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c85b: skipped 000032 1789062953.029433 mapread 0x02c622 thru 0x02d172 (0xb51 bytes) 000033 1789062953.083328 punch from 0x01853a to 0x0252d1 (0xcd98 bytes) 000034 1789062953.131173 trunc from 0x008925 to 0x00fd79 (0x7455 bytes) 000035 1789062953.150461 trunc from 0x03c85b to 0x02c8af (0xffab bytes) 000036 1789062953.207392 trunc from 0x01aa58 to 0x01bd06 (0x12af bytes) 000037 1789062953.219046 punch from 0x01beed to 0x020a82 (0x4b96 bytes) 000038 1789062953.225070 trunc from 0x01490c to 0x0214a5 (0xcb9a bytes) 000039 1789062953.242716 punch from 0x00f115 to 0x013c44 (0x4b30 bytes) 000040 1789062953.255518 punch from 0x000a6c to 0x00f78f (0xed24 bytes) 000041 1789062953.262842 mapwrite 0x01761f thru 0x01ac40 (0x3622 bytes) 000042 1789062953.332092 punch from 0x02a808 to 0x02c8af (0x20a8 bytes) 000043 1789062953.339429 punch from 0x00e143 to 0x011e4e (0x3d0c bytes) 000044 1789062953.346733 mapwrite 0x0050b8 thru 0x008a5c (0x39a5 bytes) 000045 1789062953.412741 mapread 0x01d571 thru 0x0210ea (0x3b7a bytes) 000046 1789062953.433739 trunc from 0x02c8b0 to 0x023275 (0x963a bytes) 000047 1789062953.462530 mapread 0x01d136 thru 0x020975 (0x3840 bytes) 000050 1789062953.473515 read 0x017fea thru 0x023275 (0xb28c bytes) 000051 1789062953.475631 read 0x011bcc thru 0x01a8a7 (0x8cdc bytes) 000052 1789062953.487406 trunc from 0x023276 to 0x02da50 (0xa7db bytes) 000053 1789062953.541959 read 0x010946 thru 0x017dcc (0x7487 bytes) fallocating to largest ever: 0x40000 000054 1789062953.547100 fallocate 0x03ba8c thru 0x03ffff (0x4574 bytes) 000055 1789062953.572555 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0xa009fea (stale) Mirror 4: 0xa009fea (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x7cb96e09 Mirror 5: 0x7cb96e09 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000056 1789062954.889601 read 0x007675 thru 0x00bb69 (0x44f5 bytes) 000057 1789062954.923276 trunc from 0x016f80 to 0x01efd0 (0x8051 bytes) 000058 1789062954.996566 punch from 0x015656 to 0x01bf8c (0x6937 bytes) 000059 1789062955.012230 fallocate 0x01c1d5 thru 0x020559 (0x4385 bytes) 000061 1789062955.023378 punch from 0x00d534 to 0x0199db (0xc4a8 bytes) 000062 1789062955.035285 punch from 0x031045 to 0x039d26 (0x8ce2 bytes) 000063 1789062955.041242 punch from 0x01cade to 0x0205f6 (0x3b19 bytes) 000064 1789062955.058175 mapwrite 0x030119 thru 0x035bf0 (0x5ad8 bytes) 000065 1789062955.111612 mapwrite 0x019bde thru 0x01a430 (0x853 bytes) 000066 1789062955.132170 mapwrite 0x038613 thru 0x03ffff (0x79ed bytes) 000067 1789062955.153022 fallocate 0x03dcbf thru 0x03ffff (0x2341 bytes) 000068 1789062955.157400 mapread 0x0167dc thru 0x01bd0b (0x5530 bytes) 000069 1789062955.158611 mapwrite 0x02b2c7 thru 0x036f40 (0xbc7a bytes) 000071 1789062955.189391 read 0x03cb4b thru 0x03ffff (0x34b5 bytes) 000072 1789062955.190273 mapread 0x00b022 thru 0x013f99 (0x8f78 bytes) 000073 1789062955.207139 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0xa009fea (stale) Mirror 4: 0xa009fea (stale) Mirror 5: 0x7cb96e09 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x4fca1799 Mirror 6: 0x4fca1799 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000074 1789062956.344309 trunc from 0x040000 to 0x001e8c (0x3e173 bytes) 000075 1789062956.391465 mapread 0x00167d thru 0x001e8c (0x810 bytes) 000077 1789062956.397710 fallocate 0x02e86b thru 0x03ace8 (0xc47e bytes) 000078 1789062956.404134 mapread 0x013f1d thru 0x0142c8 (0x3ac bytes) 000079 1789062956.417189 trunc from 0x02d34d to 0x034f54 (0x7c08 bytes) 000080 1789062956.423672 mapwrite 0x014a69 thru 0x0205d4 (0xbb6c bytes) 000081 1789062956.452373 fallocate 0x031cff thru 0x03ffff (0xe301 bytes) 000082 1789062956.455482 punch from 0x003b90 to 0x00403d (0x4ae bytes) 000083 1789062956.458548 fallocate 0x006494 thru 0x01141f (0xaf8c bytes) 000084 1789062956.462082 write 0x00ae56 thru 0x018e2a (0xdfd5 bytes) 000085 1789062956.468752 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xe20eea22 (stale) Mirror 3: 0xa009fea (stale) Mirror 4: 0xa009fea (stale) Mirror 5: 0x7cb96e09 (stale) Mirror 6: 0x4fca1799 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x465b40d5 Mirror 7: 0x465b40d5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000086 1789062957.624819 punch from 0x024a2e to 0x03209e (0xd671 bytes) 000087 1789062957.757382 trunc from 0x040000 to 0x005f31 (0x3a0ce bytes) 000088 1789062957.908244 punch from 0x005cde to 0x005f31 (0x254 bytes) 000089 1789062957.918545 write 0x028da5 thru 0x02c77e (0x39da bytes) 000090 1789062957.934958 fallocate 0x02babb thru 0x02f05c (0x35a2 bytes) 000091 1789062957.953342 mapread 0x00c940 thru 0x017799 (0xae5a bytes) 000092 1789062958.059205 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x2c77f): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2c77f): Mirror 2: 0xc78b5ac9 (stale) Mirror 3: 0x35011f53 (stale) Mirror 4: 0x35011f53 (stale) Mirror 5: 0x4f26ef2c (stale) Mirror 6: 0xbac3f6d3 (stale) Mirror 7: 0x202b365b (stale) CRC-32 checksum value for chunk [0, 0x2c77f): Mirror 1: 0xbf7aa03d Mirror 8: 0xbf7aa03d lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2c77f: skipped 000093 1789062960.495160 mapread 0x01bc37 thru 0x02471a (0x8ae4 bytes) 000094 1789062960.531717 mapread 0x02122c thru 0x023094 (0x1e69 bytes) 000095 1789062960.548572 trunc from 0x02c77f to 0x012d57 (0x19a27 bytes) 000096 1789062960.599007 read 0x00a088 thru 0x011aff (0x7a78 bytes) 000097 1789062960.611714 fallocate 0x00a858 thru 0x016583 (0xbd2c bytes) 000098 1789062960.618672 punch from 0x001ccd to 0x005b52 (0x3e86 bytes) 000099 1789062960.623422 fallocate 0x01e6a0 thru 0x027afe (0x945f bytes) 000100 1789062960.629244 mapwrite 0x029822 thru 0x031f79 (0x8758 bytes) 000101 1789062960.705968 mapread 0x004616 thru 0x005c24 (0x160f bytes) 000102 1789062960.735707 trunc from 0x031f7a to 0x027bf2 (0xa387 bytes) 000103 1789062960.781352 fallocate 0x0098ee thru 0x00cf01 (0x3614 bytes) 000104 1789062960.798994 punch from 0x018ec8 to 0x024d42 (0xbe7b bytes) 000105 1789062960.806739 read 0x01a6e0 thru 0x027bf2 (0xd513 bytes) 000106 1789062960.848225 trunc from 0x027bf3 to 0x020873 (0x737f bytes) 000107 1789062960.899533 trunc from 0x020874 to 0x02bbc9 (0xb356 bytes) 000108 1789062960.950995 write 0x029333 thru 0x033d25 (0xa9f3 bytes) 000109 1789062960.976615 trunc from 0x033d26 to 0x00a69a (0x2968b bytes) 000110 1789062961.017402 read 0x00305e thru 0x00a69a (0x763d bytes) 000111 1789062961.023891 write 0x03399c thru 0x03ed70 (0xb3d5 bytes) 000112 1789062961.035623 MIRROR_RESYNC 8 mirrors 000113 1789062962.421871 trunc from 0x03ed71 to 0x00bb7a (0x331f6 bytes) 000114 1789062962.493306 trunc from 0x00a368 to 0x00bb7a (0x1813 bytes) 000115 1789062962.522054 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0xbb7b): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xbb7b): Mirror 2: 0x8cfd978b (stale) Mirror 3: 0x8cfd978b (stale) Mirror 4: 0x8cfd978b (stale) Mirror 5: 0x8cfd978b (stale) Mirror 6: 0x8cfd978b (stale) Mirror 7: 0x8cfd978b (stale) Mirror 8: 0x8cfd978b (stale) CRC-32 checksum value for chunk [0, 0xbb7b): Mirror 1: 0x8cfd978b Mirror 9: 0x8cfd978b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xbb7b: skipped 000116 1789062963.813281 punch from 0x00766e to 0x00bb7a (0x450d bytes) 000117 1789062963.862887 fallocate 0x009fc4 thru 0x00f6c3 (0x5700 bytes) 000118 1789062963.876170 write 0x0163eb thru 0x0201ae (0x9dc4 bytes) 000119 1789062963.883742 punch from 0x006cb3 to 0x00a77b (0x3ac9 bytes) 000120 1789062963.894776 mapwrite 0x02ce04 thru 0x02f08a (0x2287 bytes) 000121 1789062963.961320 mapwrite 0x03aa4d thru 0x03ffff (0x55b3 bytes) 000122 1789062964.011629 write 0x035eca thru 0x03ffff (0xa136 bytes) 000123 1789062964.013771 mapread 0x018170 thru 0x0209cd (0x885e bytes) 000124 1789062964.015530 mapwrite 0x017697 thru 0x017dcb (0x735 bytes) 000125 1789062964.025699 punch from 0x0282bb to 0x0357b3 (0xd4f9 bytes) 000126 1789062964.028587 fallocate 0x032197 thru 0x03e662 (0xc4cc bytes) 000127 1789062964.031231 write 0x02471c thru 0x026729 (0x200e bytes) 000128 1789062964.037394 trunc from 0x040000 to 0x024854 (0x1b7ab bytes) 000129 1789062964.053321 mapread 0x0066ea thru 0x013c0e (0xd525 bytes) 000130 1789062964.111837 fallocate 0x01656a thru 0x01d75a (0x71f1 bytes) 000131 1789062964.118696 trunc from 0x01f878 to 0x0234ce (0x3c57 bytes) 000132 1789062964.127540 write 0x032e60 thru 0x03ffff (0xd1a0 bytes) 000133 1789062964.131295 read 0x011a86 thru 0x01807d (0x65f8 bytes) 000134 1789062964.132371 mapread 0x027fc4 thru 0x035cd6 (0xdd13 bytes) 000135 1789062964.157740 trunc from 0x031030 to 0x03cb20 (0xbaf1 bytes) 000136 1789062964.179056 trunc from 0x03ed62 to 0x03ffff (0x129e bytes) 000137 1789062964.192607 mapwrite 0x03aa45 thru 0x03ffff (0x55bb bytes) 000138 1789062964.260233 MIRROR_RESYNC 9 mirrors 000139 1789062966.454076 write 0x00fdb1 thru 0x01bbf1 (0xbe41 bytes) 000140 1789062966.501421 fallocate 0x020fe1 thru 0x028e5d (0x7e7d bytes) 000141 1789062966.511265 mapwrite 0x01783a thru 0x02559d (0xdd64 bytes) 000142 1789062966.631669 mapwrite 0x02de68 thru 0x03d780 (0xf919 bytes) 000143 1789062966.725788 trunc from 0x040000 to 0x03a2f2 (0x5d0d bytes) 000144 1789062966.773639 mapread 0x00e132 thru 0x01ceef (0xedbe bytes) 000145 1789062966.792774 mapwrite 0x015f8c thru 0x019818 (0x388d bytes) 000146 1789062966.817045 trunc from 0x03a2f3 to 0x02fba1 (0xa751 bytes) 000147 1789062966.841172 trunc from 0x006221 to 0x014fa1 (0xed81 bytes) 000148 1789062966.851657 mapread 0x019a29 thru 0x02015b (0x6733 bytes) 000149 1789062966.857818 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x2fba2): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fba2): Mirror 2: 0x75c79fe7 (stale) Mirror 3: 0x75c79fe7 (stale) Mirror 4: 0x75c79fe7 (stale) Mirror 5: 0x75c79fe7 (stale) Mirror 6: 0x75c79fe7 (stale) Mirror 7: 0x75c79fe7 (stale) Mirror 8: 0x75c79fe7 (stale) Mirror 9: 0x75c79fe7 (stale) CRC-32 checksum value for chunk [0, 0x2fba2): Mirror 1: 0xad40f67c Mirror 10: 0xad40f67c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fba2: skipped 000150 1789062967.920458 read 0x002c8c thru 0x003509 (0x87e bytes) 000151 1789062967.943104 read 0x015ccc thru 0x0224e2 (0xc817 bytes) 000152 1789062967.981783 write 0x00b23b thru 0x01acf4 (0xfaba bytes) 000154 1789062968.079439 read 0x008016 thru 0x0124eb (0xa4d6 bytes) 000155 1789062968.090764 trunc from 0x029889 to 0x029c86 (0x3fe bytes) 000156 1789062968.096233 read 0x002597 thru 0x0037a1 (0x120b bytes) 000157 1789062968.097071 fallocate 0x01de6f thru 0x0248cc (0x6a5e bytes) 000158 1789062968.105847 MIRROR_RESYNC 10 mirrors 000159 1789062970.000310 trunc from 0x0038b5 to 0x011dcf (0xe51b bytes) 000160 1789062970.027325 mapwrite 0x0256f1 thru 0x0337a1 (0xe0b1 bytes) 000161 1789062970.088588 fallocate 0x0146c1 thru 0x016927 (0x2267 bytes) 000162 1789062970.093303 punch from 0x00a0c7 to 0x01268f (0x85c9 bytes) 000163 1789062970.102628 MIRROR_SPLIT mirror 1 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x337a2): Mirror 3: 0x72f239cf (stale) Mirror 4: 0x72f239cf (stale) Mirror 5: 0x72f239cf (stale) Mirror 6: 0x72f239cf (stale) Mirror 7: 0x72f239cf (stale) Mirror 8: 0x72f239cf (stale) Mirror 9: 0x72f239cf (stale) Mirror 10: 0x72f239cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x337a2): Mirror 3: 0x72f239cf (stale) Mirror 4: 0x72f239cf (stale) Mirror 5: 0x72f239cf (stale) Mirror 6: 0x72f239cf (stale) Mirror 7: 0x72f239cf (stale) Mirror 8: 0x72f239cf (stale) Mirror 9: 0x72f239cf (stale) Mirror 10: 0x72f239cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x337a2: skipped 000164 1789062970.682220 mapread 0x0189d3 thru 0x026914 (0xdf42 bytes) 000165 1789062970.724251 write 0x032a9e thru 0x035e79 (0x33dc bytes) 000166 1789062970.747053 mapwrite 0x004dfd thru 0x00fbc8 (0xadcc bytes) 000167 1789062970.780727 trunc from 0x002906 to 0x00caa4 (0xa19f bytes) 000168 1789062970.785044 mapread 0x01ee5c thru 0x02d57e (0xe723 bytes) 000169 1789062970.800534 fallocate 0x00b0bd thru 0x00c54b (0x148f bytes) 000170 1789062970.807828 punch from 0x01dfdb to 0x028fd1 (0xaff7 bytes) 000171 1789062970.814397 trunc from 0x02ccf3 to 0x035e79 (0x9187 bytes) 000172 1789062970.819224 mapread 0x0153d2 thru 0x01c61a (0x7249 bytes) 000173 1789062970.822461 read 0x0035ac thru 0x003e55 (0x8aa bytes) 000174 1789062970.828738 write 0x02253e thru 0x02a8a4 (0x8367 bytes) 000175 1789062970.834578 trunc from 0x02bfcd to 0x030679 (0x46ad bytes) 000176 1789062970.840192 write 0x03bd4d thru 0x03ffff (0x42b3 bytes) 000177 1789062970.841724 write 0x034556 thru 0x034fa9 (0xa54 bytes) 000178 1789062970.848899 read 0x029e0f thru 0x03827e (0xe470 bytes) 000179 1789062970.858627 trunc from 0x040000 to 0x0058f5 (0x3a70a bytes) 000180 1789062970.887584 mapread 0x00530e thru 0x0058f5 (0x5e8 bytes) 000181 1789062970.899531 MIRROR_RESYNC 9 mirrors 000182 1789062972.988146 write 0x020846 thru 0x02edf3 (0xe5ae bytes) 000183 1789062973.067524 trunc from 0x02edf4 to 0x02f579 (0x786 bytes) 000184 1789062973.108583 fallocate 0x00a9db thru 0x017fc1 (0xd5e7 bytes) 000185 1789062973.115184 trunc from 0x02f57a to 0x033514 (0x3f9b bytes) 000186 1789062973.149014 mapread 0x012c1b thru 0x01ded7 (0xb2bd bytes) 000187 1789062973.186326 mapwrite 0x00675c thru 0x015218 (0xeabd bytes) 000188 1789062973.247104 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x33515): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33515): Mirror 3: 0xa381ef41 (stale) Mirror 4: 0xa381ef41 (stale) Mirror 5: 0xa381ef41 (stale) Mirror 6: 0xa381ef41 (stale) Mirror 7: 0xa381ef41 (stale) Mirror 8: 0xa381ef41 (stale) Mirror 9: 0xa381ef41 (stale) Mirror 10: 0xa381ef41 (stale) CRC-32 checksum value for chunk [0, 0x33515): Mirror 1: 0x32eacc0a Mirror 11: 0x32eacc0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33515: skipped 000189 1789062974.682197 MIRROR_EXTEND to 11 mirrors CRC-32 checksum value for chunk [0, 0x33515): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x33515): Mirror 3: 0xa381ef41 (stale) Mirror 4: 0xa381ef41 (stale) Mirror 5: 0xa381ef41 (stale) Mirror 6: 0xa381ef41 (stale) Mirror 7: 0xa381ef41 (stale) Mirror 8: 0xa381ef41 (stale) Mirror 9: 0xa381ef41 (stale) Mirror 10: 0xa381ef41 (stale) CRC-32 checksum value for chunk [0, 0x33515): Mirror 1: 0x32eacc0a Mirror 11: 0x32eacc0a Mirror 12: 0x32eacc0a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x33515: skipped 000190 1789062975.968320 punch from 0x012f8a to 0x016ee0 (0x3f57 bytes) 000191 1789062976.101465 read 0x02819b thru 0x03197d (0x97e3 bytes) 000192 1789062976.116377 trunc from 0x033515 to 0x0240c4 (0xf450 bytes) 000193 1789062976.201488 trunc from 0x0240c5 to 0x027ee7 (0x3e23 bytes) 000194 1789062976.226480 write 0x0274e2 thru 0x037166 (0xfc85 bytes) 000195 1789062976.236860 trunc from 0x037167 to 0x02877f (0xe9e7 bytes) 000196 1789062976.256878 trunc from 0x00ed20 to 0x014c46 (0x5f27 bytes) 000197 1789062976.265072 write 0x0066a9 thru 0x0079fe (0x1356 bytes) 000199 1789062976.283300 read 0x0208ef thru 0x02877f (0x7e91 bytes) 000200 1789062976.296729 fallocate 0x034102 thru 0x039e89 (0x5d88 bytes) 000202 1789062976.307788 trunc from 0x03145a to 0x039241 (0x7de8 bytes) 000203 1789062976.318284 mapread 0x00bdc9 thru 0x00de3c (0x2074 bytes) 000204 1789062976.320324 trunc from 0x0108ff to 0x0166f3 (0x5df5 bytes) 000205 1789062976.327873 write 0x0319f7 thru 0x0338c2 (0x1ecc bytes) 000207 1789062976.364846 write 0x00f4be thru 0x0190b8 (0x9bfb bytes) 000208 1789062976.373334 mapread 0x01c752 thru 0x01dce1 (0x1590 bytes) 000209 1789062976.379677 trunc from 0x039e8a to 0x0241a9 (0x15ce0 bytes) 000210 1789062976.480631 fallocate 0x010407 thru 0x011729 (0x1323 bytes) 000211 1789062976.490342 read 0x003d17 thru 0x00c5e1 (0x88cb bytes) 000212 1789062976.491915 mapwrite 0x01d6b4 thru 0x0242b0 (0x6bfd bytes) 000213 1789062976.527327 punch from 0x002e44 to 0x010fd1 (0xe18e bytes) 000214 1789062976.533402 trunc from 0x010fb8 to 0x01ec60 (0xdca9 bytes) 000216 1789062976.550638 punch from 0x014bb8 to 0x018b45 (0x3f8e bytes) 000217 1789062976.562762 MIRROR_EXTEND to 12 mirrors CRC-32 checksum value for chunk [0, 0x242b1): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 9: (stale - read error) Mirror 10: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x242b1): Mirror 3: 0x79fe2c46 (stale) Mirror 4: 0x79fe2c46 (stale) Mirror 5: 0x79fe2c46 (stale) Mirror 6: 0x79fe2c46 (stale) Mirror 7: 0x79fe2c46 (stale) Mirror 8: 0x79fe2c46 (stale) Mirror 9: 0x79fe2c46 (stale) Mirror 10: 0x79fe2c46 (stale) Mirror 11: 0x2ede39ca (stale) Mirror 12: 0x2ede39ca (stale) CRC-32 checksum value for chunk [0, 0x242b1): Mirror 1: 0x48b10002 Mirror 13: 0x48b10002 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x242b1: skipped 000218 1789062978.085264 MIRROR_RESYNC 12 mirrors 000219 1789062980.628409 read 0x01694d thru 0x0242b0 (0xd964 bytes) 000220 1789062980.645758 write 0x015616 thru 0x017081 (0x1a6c bytes) 000223 1789062980.773206 trunc from 0x008f08 to 0x0180b3 (0xf1ac bytes) 000224 1789062980.809869 mapwrite 0x015efa thru 0x01dec2 (0x7fc9 bytes) 000225 1789062980.909631 read 0x0046e3 thru 0x013f74 (0xf892 bytes) 000226 1789062980.925173 trunc from 0x0242b1 to 0x036bb7 (0x12907 bytes) 000227 1789062980.961884 trunc from 0x036bb8 to 0x00ad8d (0x2be2a bytes) 000228 1789062980.987335 read 0x008a74 thru 0x00ad8d (0x231a bytes) 000229 1789062980.989568 mapwrite 0x0370f8 thru 0x0378c3 (0x7cc bytes) 000231 1789062981.072659 MIRROR_SPLIT mirror 3 to 11 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x378c4): Mirror 3: 0x47bce7af (stale) Mirror 4: 0x47bce7af (stale) Mirror 6: 0x47bce7af (stale) Mirror 7: 0x47bce7af (stale) Mirror 8: 0x47bce7af (stale) Mirror 9: 0x47bce7af (stale) Mirror 10: 0x47bce7af (stale) Mirror 11: 0x47bce7af (stale) Mirror 12: 0x47bce7af (stale) Mirror 13: 0x47bce7af (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x378c4): Mirror 3: 0x47bce7af (stale) Mirror 4: 0x47bce7af (stale) Mirror 6: 0x47bce7af (stale) Mirror 7: 0x47bce7af (stale) Mirror 8: 0x47bce7af (stale) Mirror 9: 0x47bce7af (stale) Mirror 10: 0x47bce7af (stale) Mirror 11: 0x47bce7af (stale) Mirror 12: 0x47bce7af (stale) Mirror 13: 0x47bce7af (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x378c4: skipped 000232 1789062982.132346 read 0x01b88d thru 0x022a9d (0x7211 bytes) 000234 1789062982.143757 trunc from 0x0378c4 to 0x014aaf (0x22e14 bytes) 000235 1789062982.173557 trunc from 0x014ab0 to 0x027dbc (0x1330d bytes) 000236 1789062982.195124 mapwrite 0x008bf8 thru 0x00aa36 (0x1e3f bytes) 000237 1789062982.272881 punch from 0x01ab34 to 0x020513 (0x59e0 bytes) 000239 1789062982.285272 write 0x025714 thru 0x02e173 (0x8a60 bytes) 000240 1789062982.305620 fallocate 0x024dbb thru 0x027a5a (0x2ca0 bytes) 000241 1789062982.347603 MIRROR_SPLIT mirror 7 to 10 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2e174): Mirror 3: 0x9b71934c (stale) Mirror 4: 0x9b71934c (stale) Mirror 6: 0x9b71934c (stale) Mirror 7: 0x9b71934c (stale) Mirror 8: 0x9b71934c (stale) Mirror 9: 0x9b71934c (stale) Mirror 11: 0x9b71934c (stale) Mirror 12: 0x9b71934c (stale) Mirror 13: 0x9b71934c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2e174): Mirror 3: 0x9b71934c (stale) Mirror 4: 0x9b71934c (stale) Mirror 6: 0x9b71934c (stale) Mirror 7: 0x9b71934c (stale) Mirror 8: 0x9b71934c (stale) Mirror 9: 0x9b71934c (stale) Mirror 11: 0x9b71934c (stale) Mirror 12: 0x9b71934c (stale) Mirror 13: 0x9b71934c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2e174: skipped 000242 1789062982.956542 fallocate 0x0063bf thru 0x013c8f (0xd8d1 bytes) 000243 1789062982.962869 fallocate 0x00cdcd thru 0x01ca11 (0xfc45 bytes) 000244 1789062982.966992 fallocate 0x03e339 thru 0x03ffff (0x1cc7 bytes) 000245 1789062982.971703 fallocate 0x006b07 thru 0x007305 (0x7ff bytes) 000246 1789062982.976192 punch from 0x02b4dd to 0x02e173 (0x2c97 bytes) 000247 1789062982.983630 fallocate 0x019d91 thru 0x01f565 (0x57d5 bytes) 000248 1789062982.988509 punch from 0x012ced to 0x019967 (0x6c7b bytes) 000249 1789062982.998434 punch from 0x013dda to 0x0236b3 (0xf8da bytes) 000250 1789062983.004792 punch from 0x020972 to 0x022495 (0x1b24 bytes) 000251 1789062983.010571 write 0x000dc8 thru 0x008f87 (0x81c0 bytes) 000252 1789062983.037022 write 0x0231ac thru 0x02a0b4 (0x6f09 bytes) 000253 1789062983.052072 punch from 0x01c350 to 0x029178 (0xce29 bytes) 000254 1789062983.070415 mapwrite 0x004098 thru 0x0104c0 (0xc429 bytes) 000256 1789062983.137686 read 0x004d82 thru 0x007d8f (0x300e bytes) 000257 1789062983.138395 trunc from 0x02e174 to 0x00f76f (0x1ea04 bytes) 000258 1789062983.161902 fallocate 0x0165a3 thru 0x01fff4 (0x9a52 bytes) 000259 1789062983.171817 mapwrite 0x01cd6d thru 0x028907 (0xbb9b bytes) 000261 1789062983.306978 punch from 0x00b620 to 0x01262a (0x700b bytes) 000262 1789062983.327665 MIRROR_RESYNC 10 mirrors 000263 1789062985.536782 trunc from 0x028908 to 0x023af4 (0x4e13 bytes) 000264 1789062985.587784 punch from 0x00bcbe to 0x00d024 (0x1367 bytes) 000265 1789062985.599805 MIRROR_SPLIT mirror 3 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23af5): Mirror 3: 0x1c21f22c (stale) Mirror 4: 0x1c21f22c (stale) Mirror 7: 0x1c21f22c (stale) Mirror 8: 0x1c21f22c (stale) Mirror 9: 0x1c21f22c (stale) Mirror 11: 0x1c21f22c (stale) Mirror 12: 0x1c21f22c (stale) Mirror 13: 0x1c21f22c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23af5): Mirror 3: 0x1c21f22c (stale) Mirror 4: 0x1c21f22c (stale) Mirror 7: 0x1c21f22c (stale) Mirror 8: 0x1c21f22c (stale) Mirror 9: 0x1c21f22c (stale) Mirror 11: 0x1c21f22c (stale) Mirror 12: 0x1c21f22c (stale) Mirror 13: 0x1c21f22c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23af5: skipped 000266 1789062986.660793 read 0x00cee2 thru 0x0181be (0xb2dd bytes) 000267 1789062986.679145 punch from 0x0215ef to 0x023af4 (0x2506 bytes) 000268 1789062986.687256 punch from 0x023089 to 0x023af4 (0xa6c bytes) 000269 1789062986.710672 trunc from 0x023af5 to 0x030649 (0xcb55 bytes) 000270 1789062986.778617 mapread 0x023b70 thru 0x02d8a5 (0x9d36 bytes) 000271 1789062986.878770 trunc from 0x01faf3 to 0x02a4d1 (0xa9df bytes) 000272 1789062986.902047 write 0x02783a thru 0x0302e2 (0x8aa9 bytes) 000273 1789062986.929715 trunc from 0x03064a to 0x01e99b (0x11cae bytes) 000275 1789062986.964770 trunc from 0x01e99c to 0x000cbf (0x1dcdc bytes) 000276 1789062987.020794 mapwrite 0x008a4a thru 0x0183cd (0xf984 bytes) 000277 1789062987.131920 punch from 0x0094a9 to 0x0109ce (0x7526 bytes) 000278 1789062987.143783 mapwrite 0x00936f thru 0x009802 (0x494 bytes) 000279 1789062987.199498 write 0x02aced thru 0x0392ea (0xe5fe bytes) 000280 1789062987.208271 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x392eb): Mirror 3: 0x312a1447 (stale) Mirror 7: 0x312a1447 (stale) Mirror 8: 0x312a1447 (stale) Mirror 9: 0x312a1447 (stale) Mirror 11: 0x312a1447 (stale) Mirror 12: 0x312a1447 (stale) Mirror 13: 0x312a1447 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x392eb): Mirror 3: 0x312a1447 (stale) Mirror 7: 0x312a1447 (stale) Mirror 8: 0x312a1447 (stale) Mirror 9: 0x312a1447 (stale) Mirror 11: 0x312a1447 (stale) Mirror 12: 0x312a1447 (stale) Mirror 13: 0x312a1447 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x392eb: skipped 000281 1789062988.366817 write 0x000890 thru 0x00d829 (0xcf9a bytes) 000282 1789062988.386471 write 0x0377df thru 0x03e6f5 (0x6f17 bytes) 000283 1789062988.394888 trunc from 0x00ba08 to 0x01b1c1 (0xf7ba bytes) 000284 1789062988.416328 read 0x024ecb thru 0x02bc5c (0x6d92 bytes) 000285 1789062988.435882 MIRROR_RESYNC 8 mirrors 000286 1789062990.545060 trunc from 0x0323fb to 0x03b09e (0x8ca4 bytes) 000287 1789062990.634783 trunc from 0x03e6f6 to 0x03a535 (0x41c0 bytes) 000288 1789062990.693824 read 0x030b60 thru 0x035460 (0x4901 bytes) 000289 1789062990.735703 read 0x030be9 thru 0x038056 (0x746e bytes) 000290 1789062990.771365 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3a536): Mirror 3: 0xd504d5aa (stale) Mirror 7: 0xd504d5aa (stale) Mirror 8: 0xd504d5aa (stale) Mirror 11: 0xd504d5aa (stale) Mirror 12: 0xd504d5aa (stale) Mirror 13: 0xd504d5aa (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a536): Mirror 3: 0xd504d5aa (stale) Mirror 7: 0xd504d5aa (stale) Mirror 8: 0xd504d5aa (stale) Mirror 11: 0xd504d5aa (stale) Mirror 12: 0xd504d5aa (stale) Mirror 13: 0xd504d5aa (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a536: skipped 000291 1789062991.965705 write 0x00e8c6 thru 0x015495 (0x6bd0 bytes) 000292 1789062991.992471 trunc from 0x03a536 to 0x01e558 (0x1bfdd bytes) 000293 1789062992.025743 write 0x03f310 thru 0x03ffff (0xcf0 bytes) 000294 1789062992.027829 read 0x03aec2 thru 0x03ffff (0x513e bytes) 000295 1789062992.047630 mapread 0x0130e7 thru 0x015d0a (0x2c24 bytes) 000296 1789062992.051393 mapwrite 0x02aba5 thru 0x02c236 (0x1692 bytes) 000297 1789062992.181610 mapread 0x013069 thru 0x01e5b2 (0xb54a bytes) 000299 1789062992.196591 fallocate 0x003f98 thru 0x00d644 (0x96ad bytes) 000300 1789062992.205790 punch from 0x0183bf to 0x022da0 (0xa9e2 bytes) 000301 1789062992.216772 mapread 0x012e8e thru 0x01305d (0x1d0 bytes) 000302 1789062992.221856 trunc from 0x040000 to 0x00505c (0x3afa3 bytes) 000303 1789062992.318422 trunc from 0x00505d to 0x0336d7 (0x2e67b bytes) 000304 1789062992.363844 trunc from 0x0336d8 to 0x01deed (0x157ea bytes) 000305 1789062992.431694 punch from 0x002b11 to 0x00df54 (0xb444 bytes) 000306 1789062992.440841 trunc from 0x01deee to 0x024e0b (0x6f1e bytes) 000307 1789062992.491721 write 0x011153 thru 0x0200b0 (0xef5e bytes) 000308 1789062992.523797 read 0x00d0d1 thru 0x013f7b (0x6eab bytes) 000309 1789062992.572728 trunc from 0x024e0c to 0x01e6e1 (0x672a bytes) 000311 1789062992.607743 trunc from 0x01e6e2 to 0x0196eb (0x4ff6 bytes) 000312 1789062992.625749 punch from 0x0002e1 to 0x00982d (0x954d bytes) 000313 1789062992.633680 write 0x02574b thru 0x03072c (0xafe2 bytes) 000314 1789062992.635783 trunc from 0x006934 to 0x00d91b (0x6fe8 bytes) 000315 1789062992.652041 write 0x0221ec thru 0x026c59 (0x4a6e bytes) 000316 1789062992.679222 mapread 0x0089c3 thru 0x0174c8 (0xeb06 bytes) 000317 1789062992.739382 read 0x024289 thru 0x03072c (0xc4a4 bytes) 000318 1789062992.741385 mapwrite 0x036ab5 thru 0x0380ec (0x1638 bytes) 000319 1789062992.813993 punch from 0x00edcf to 0x018d11 (0x9f43 bytes) 000320 1789062992.827279 write 0x03fe97 thru 0x03ffff (0x169 bytes) 000321 1789062992.840090 trunc from 0x0338d7 to 0x03540b (0x1b35 bytes) 000322 1789062992.850155 write 0x008235 thru 0x00bece (0x3c9a bytes) 000323 1789062992.851189 mapwrite 0x03033e thru 0x03c93e (0xc601 bytes) 000324 1789062992.966156 punch from 0x03904a to 0x03da6b (0x4a22 bytes) 000325 1789062992.977365 trunc from 0x03e147 to 0x03ffff (0x1eb9 bytes) 000326 1789062992.988994 MIRROR_RESYNC 7 mirrors 000328 1789062996.223609 read 0x0344b8 thru 0x03807e (0x3bc7 bytes) 000329 1789062996.242831 mapwrite 0x012f3d thru 0x019489 (0x654d bytes) 000330 1789062996.461269 MIRROR_SPLIT mirror 6 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x88d4a989 (stale) Mirror 7: 0x88d4a989 (stale) Mirror 8: 0x88d4a989 (stale) Mirror 11: 0x88d4a989 (stale) Mirror 12: 0x88d4a989 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x88d4a989 (stale) Mirror 7: 0x88d4a989 (stale) Mirror 8: 0x88d4a989 (stale) Mirror 11: 0x88d4a989 (stale) Mirror 12: 0x88d4a989 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000332 1789062997.076106 read 0x0026e1 thru 0x0117d4 (0xf0f4 bytes) 000335 1789062997.125926 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x88d4a989 (stale) Mirror 7: 0x88d4a989 (stale) Mirror 8: 0x88d4a989 (stale) Mirror 11: 0x88d4a989 (stale) Mirror 12: 0x88d4a989 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf47e5e55 Mirror 13: 0xf47e5e55 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000336 1789062999.012035 punch from 0x03fb26 to 0x03ffff (0x4da bytes) 000337 1789062999.069245 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x88d4a989 (stale) Mirror 7: 0x88d4a989 (stale) Mirror 8: 0x88d4a989 (stale) Mirror 11: 0x88d4a989 (stale) Mirror 12: 0x88d4a989 (stale) Mirror 13: 0xf47e5e55 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xf47e5e55 Mirror 14: 0xf47e5e55 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000338 1789063000.176695 trunc from 0x0319da to 0x03d1b0 (0xb7d7 bytes) 000339 1789063000.248830 read 0x035128 thru 0x03ad14 (0x5bed bytes) 000340 1789063000.279661 mapread 0x0167d2 thru 0x0215f6 (0xae25 bytes) 000341 1789063000.342855 write 0x024954 thru 0x02c7eb (0x7e98 bytes) 000342 1789063000.379539 mapwrite 0x03cc7b thru 0x03ffff (0x3385 bytes) 000343 1789063000.442660 mapwrite 0x03ce91 thru 0x03ffff (0x316f bytes) 000344 1789063000.484688 trunc from 0x040000 to 0x02e62f (0x119d0 bytes) 000345 1789063000.539998 trunc from 0x00a748 to 0x01a34b (0xfc04 bytes) 000346 1789063000.557172 trunc from 0x02e630 to 0x026e7d (0x77b2 bytes) 000347 1789063000.607664 read 0x0269c1 thru 0x026e7d (0x4bd bytes) 000348 1789063000.608327 punch from 0x0118dc to 0x01e24f (0xc974 bytes) 000349 1789063000.624017 read 0x00d45c thru 0x00de9a (0xa3f bytes) 000350 1789063000.666353 fallocate 0x0168b2 thru 0x023fe4 (0xd733 bytes) 000351 1789063000.671985 punch from 0x01990f to 0x0231a4 (0x9896 bytes) 000352 1789063000.693257 mapwrite 0x034ca1 thru 0x03876e (0x3ace bytes) 000353 1789063000.780801 mapread 0x011d30 thru 0x01f801 (0xdad2 bytes) 000354 1789063000.884681 fallocate 0x025736 thru 0x026246 (0xb11 bytes) 000355 1789063000.893090 mapwrite 0x0290f1 thru 0x0382b3 (0xf1c3 bytes) 000357 1789063000.976173 trunc from 0x00477d to 0x0083be (0x3c42 bytes) 000358 1789063000.990912 mapread 0x01f7e3 thru 0x02d19f (0xd9bd bytes) 000360 1789063001.046061 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3876f): Mirror 3: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3876f): Mirror 3: 0x8139b712 (stale) Mirror 7: 0x8139b712 (stale) Mirror 8: 0x8139b712 (stale) Mirror 11: 0x8139b712 (stale) Mirror 12: 0x8139b712 (stale) Mirror 13: 0x74820af1 (stale) Mirror 14: 0x74820af1 (stale) CRC-32 checksum value for chunk [0, 0x3876f): Mirror 1: 0x888582fe Mirror 15: 0x888582fe lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3876f: skipped 000361 1789063002.701855 trunc from 0x0024d5 to 0x009d5a (0x7886 bytes) 000362 1789063002.811729 mapwrite 0x03c7ef thru 0x03ffff (0x3811 bytes) 000363 1789063002.966234 fallocate 0x019d9e thru 0x01ab0e (0xd71 bytes) 000366 1789063002.991445 fallocate 0x003a66 thru 0x009392 (0x592d bytes) 000367 1789063003.005673 mapread 0x03e9ff thru 0x03ffff (0x1601 bytes) 000368 1789063003.009633 trunc from 0x040000 to 0x02ba4b (0x145b4 bytes) 000369 1789063003.066503 MIRROR_SPLIT mirror 2 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2ba4c): Mirror 3: 0x63ede5a8 (stale) Mirror 8: 0x63ede5a8 (stale) Mirror 11: 0x63ede5a8 (stale) Mirror 12: 0x63ede5a8 (stale) Mirror 13: 0xd98b8342 (stale) Mirror 14: 0xd98b8342 (stale) Mirror 15: 0x6b60c2a0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ba4c): Mirror 3: 0x63ede5a8 (stale) Mirror 8: 0x63ede5a8 (stale) Mirror 11: 0x63ede5a8 (stale) Mirror 12: 0x63ede5a8 (stale) Mirror 13: 0xd98b8342 (stale) Mirror 14: 0xd98b8342 (stale) Mirror 15: 0x6b60c2a0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ba4c: skipped 000370 1789063003.974519 mapwrite 0x01d066 thru 0x02ca7c (0xfa17 bytes) 000371 1789063004.070511 mapread 0x00b2d7 thru 0x00d7cd (0x24f7 bytes) 000372 1789063004.081623 trunc from 0x004a09 to 0x00e9e4 (0x9fdc bytes) 000373 1789063004.086750 write 0x008093 thru 0x00d7c3 (0x5731 bytes) 000374 1789063004.098100 mapread 0x0276b8 thru 0x029fe5 (0x292e bytes) 000375 1789063004.099259 fallocate 0x025916 thru 0x02ca3b (0x7126 bytes) 000376 1789063004.104927 trunc from 0x02ca7d to 0x03bb6b (0xf0ef bytes) 000377 1789063004.130374 write 0x008edf thru 0x00c931 (0x3a53 bytes) 000378 1789063004.132490 trunc from 0x03bb6c to 0x00d4cd (0x2e69e bytes) 000379 1789063004.161111 trunc from 0x00d4ce to 0x02d934 (0x20467 bytes) 000380 1789063004.188745 mapread 0x003c60 thru 0x00665d (0x29fe bytes) 000381 1789063004.204325 trunc from 0x02d935 to 0x008bb5 (0x24d7f bytes) 000382 1789063004.232902 fallocate 0x022617 thru 0x029d16 (0x7700 bytes) 000383 1789063004.240433 write 0x01383a thru 0x0164bd (0x2c84 bytes) 000385 1789063004.254467 punch from 0x016602 to 0x016f63 (0x962 bytes) 000386 1789063004.274379 write 0x007b3b thru 0x01691c (0xede2 bytes) 000387 1789063004.289502 mapwrite 0x030b7a thru 0x03d80b (0xcc92 bytes) 000388 1789063004.428460 MIRROR_RESYNC 8 mirrors 000389 1789063006.865167 MIRROR_RESYNC 8 mirrors 000390 1789063007.929637 fallocate 0x025025 thru 0x02e1ef (0x91cb bytes) 000391 1789063007.964699 MIRROR_RESYNC 8 mirrors 000392 1789063010.455093 punch from 0x03abda to 0x03d80b (0x2c32 bytes) 000393 1789063010.514211 mapread 0x02d549 thru 0x0367b8 (0x9270 bytes) 000394 1789063010.546512 mapwrite 0x021723 thru 0x027567 (0x5e45 bytes) 000395 1789063010.621675 trunc from 0x03d80c to 0x01d003 (0x20808 bytes) 000396 1789063010.658567 trunc from 0x01d004 to 0x028327 (0xb324 bytes) 000397 1789063010.681763 mapread 0x015f02 thru 0x01bdb6 (0x5eb5 bytes) 000398 1789063010.695925 mapwrite 0x01439d thru 0x0189b7 (0x461b bytes) 000399 1789063010.735787 mapwrite 0x00d227 thru 0x017164 (0x9f3e bytes) 000400 1789063010.814917 trunc from 0x012e29 to 0x0208d9 (0xdab1 bytes) 000401 1789063010.829824 mapread 0x009aca thru 0x00c2a0 (0x27d7 bytes) 000403 1789063010.838346 mapread 0x012bef thru 0x0188f8 (0x5d0a bytes) 000404 1789063010.851121 fallocate 0x0379ff thru 0x03fd6d (0x836f bytes) 000405 1789063010.860293 mapread 0x027dcc thru 0x036a46 (0xec7b bytes) 000406 1789063010.914775 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x3fd6e): Mirror 3: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fd6e): Mirror 3: 0xccfbbb74 (stale) Mirror 8: 0xccfbbb74 (stale) Mirror 11: 0xccfbbb74 (stale) Mirror 12: 0xccfbbb74 (stale) Mirror 13: 0xccfbbb74 (stale) Mirror 14: 0xccfbbb74 (stale) Mirror 15: 0xccfbbb74 (stale) CRC-32 checksum value for chunk [0, 0x3fd6e): Mirror 1: 0xe067d6f9 Mirror 16: 0xe067d6f9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fd6e: skipped 000407 1789063012.511914 trunc from 0x02be59 to 0x03166f (0x5817 bytes) 000408 1789063012.553388 punch from 0x008cb5 to 0x0154d7 (0xc823 bytes) 000409 1789063012.562793 punch from 0x02a97d to 0x039cc0 (0xf344 bytes) 000410 1789063012.568373 trunc from 0x03fd6e to 0x026750 (0x1961d bytes) 000411 1789063012.590408 mapwrite 0x02c019 thru 0x03a95c (0xe944 bytes) 000412 1789063012.677390 mapread 0x010d96 thru 0x018934 (0x7b9f bytes) 000415 1789063012.696748 write 0x01240d thru 0x01c0e5 (0x9cd9 bytes) 000416 1789063012.702783 mapread 0x00fb44 thru 0x01277f (0x2c3c bytes) 000417 1789063012.704016 mapread 0x03a5a4 thru 0x03a95c (0x3b9 bytes) 000418 1789063012.705304 trunc from 0x03a95d to 0x01d22e (0x1d72e bytes) 000419 1789063012.719346 read 0x01a2ed thru 0x01d22e (0x2f42 bytes) 000420 1789063012.723037 mapread 0x0086e2 thru 0x016c8b (0xe5aa bytes) 000421 1789063012.733327 fallocate 0x021df9 thru 0x02f7b4 (0xd9bc bytes) 000422 1789063012.736566 trunc from 0x00e32f to 0x012cff (0x49d1 bytes) 000423 1789063012.747744 mapread 0x008b0c thru 0x00f286 (0x677b bytes) 000425 1789063012.763024 fallocate 0x02fdbb thru 0x03c463 (0xc6a9 bytes) 000426 1789063012.783346 write 0x035374 thru 0x03ffff (0xac8c bytes) 000427 1789063012.807327 write 0x011bcd thru 0x01a914 (0x8d48 bytes) 000428 1789063012.819852 MIRROR_EXTEND to 10 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 12: (stale - read error) Mirror 13: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x7429566 (stale) Mirror 8: 0x7429566 (stale) Mirror 11: 0x7429566 (stale) Mirror 12: 0x7429566 (stale) Mirror 13: 0x7429566 (stale) Mirror 14: 0x7429566 (stale) Mirror 15: 0x7429566 (stale) Mirror 16: 0x8d3579ad (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x3a8c94f0 Mirror 17: 0x3a8c94f0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000429 1789063014.172927 fallocate 0x00c59e thru 0x00eaeb (0x254e bytes) 000430 1789063014.219651 write 0x0340c4 thru 0x035d27 (0x1c64 bytes) 000431 1789063014.233602 trunc from 0x02505f to 0x02aeaa (0x5e4c bytes) 000432 1789063014.242345 MIRROR_RESYNC 10 mirrors 000434 1789063017.207612 read 0x025d86 thru 0x02ad15 (0x4f90 bytes) 000435 1789063017.247130 write 0x019679 thru 0x022923 (0x92ab bytes) 000436 1789063017.349519 trunc from 0x002969 to 0x011c0c (0xf2a4 bytes) 000437 1789063017.358621 mapwrite 0x0171c2 thru 0x024f1b (0xdd5a bytes) 000438 1789063017.414796 mapread 0x02c893 thru 0x034742 (0x7eb0 bytes) 000439 1789063017.436926 punch from 0x035417 to 0x03a34a (0x4f34 bytes) 000440 1789063017.451906 punch from 0x0006cd to 0x008238 (0x7b6c bytes) 000441 1789063017.459782 write 0x001308 thru 0x0045ef (0x32e8 bytes) 000442 1789063017.474023 trunc from 0x024d66 to 0x02ede4 (0xa07f bytes) 000443 1789063017.479491 write 0x00d4c1 thru 0x015449 (0x7f89 bytes) 000444 1789063017.490663 mapread 0x03a49d thru 0x03c89c (0x2400 bytes) 000445 1789063017.503695 write 0x02e677 thru 0x036dd3 (0x875d bytes) 000448 1789063017.515575 fallocate 0x01f1b0 thru 0x022b13 (0x3964 bytes) 000449 1789063017.521406 read 0x028cbf thru 0x031238 (0x857a bytes) 000450 1789063017.532564 mapread 0x024754 thru 0x02c22e (0x7adb bytes) 000451 1789063017.534520 mapread 0x01f444 thru 0x0278d2 (0x848f bytes) 000452 1789063017.535798 trunc from 0x00c121 to 0x012f63 (0x6e43 bytes) 000453 1789063017.567855 mapread 0x026804 thru 0x02af42 (0x473f bytes) 000454 1789063017.571541 read 0x024295 thru 0x02a62b (0x6397 bytes) 000455 1789063017.572744 fallocate 0x01a87a thru 0x022eaf (0x8636 bytes) 000456 1789063017.578269 read 0x0254fa thru 0x02a152 (0x4c59 bytes) 000457 1789063017.579170 mapread 0x03cf82 thru 0x03ffff (0x307e bytes) 000458 1789063017.585481 MIRROR_RESYNC 10 mirrors 000460 1789063020.356109 MIRROR_SPLIT mirror 5 to 9 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) 000461 1789063020.808887 trunc from 0x0350e6 to 0x03aa94 (0x59af bytes) 000462 1789063020.861526 write 0x0049a3 thru 0x00a187 (0x57e5 bytes) 000463 1789063020.884900 fallocate 0x01b6e0 thru 0x029522 (0xde43 bytes) 000464 1789063020.890826 trunc from 0x040000 to 0x033e7f (0xc180 bytes) 000465 1789063020.918689 trunc from 0x018526 to 0x026dc4 (0xe89f bytes) 000467 1789063020.926338 mapread 0x01f275 thru 0x02b588 (0xc314 bytes) 000468 1789063020.964582 trunc from 0x0031bd to 0x0129b5 (0xf7f9 bytes) 000469 1789063020.983105 fallocate 0x019c4c thru 0x01e25f (0x4614 bytes) 000470 1789063020.987726 trunc from 0x033e80 to 0x009aa4 (0x2a3db bytes) 000471 1789063021.003638 fallocate 0x01d820 thru 0x0273c5 (0x9ba6 bytes) 000472 1789063021.010550 mapread 0x004134 thru 0x009aa4 (0x5971 bytes) 000473 1789063021.023259 punch from 0x007d9e to 0x009aa4 (0x1d07 bytes) 000474 1789063021.028506 trunc from 0x005b35 to 0x009aa4 (0x3f70 bytes) 000475 1789063021.034718 write 0x0212a6 thru 0x02955f (0x82ba bytes) 000476 1789063021.039671 fallocate 0x00e797 thru 0x01bb90 (0xd3fa bytes) 000477 1789063021.054051 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29560): Mirror 3: 0x91e2cd98 (stale) Mirror 8: 0x91e2cd98 (stale) Mirror 11: 0x91e2cd98 (stale) Mirror 14: 0x91e2cd98 (stale) Mirror 15: 0x91e2cd98 (stale) Mirror 16: 0x91e2cd98 (stale) Mirror 17: 0x91e2cd98 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29560): Mirror 3: 0x91e2cd98 (stale) Mirror 8: 0x91e2cd98 (stale) Mirror 11: 0x91e2cd98 (stale) Mirror 14: 0x91e2cd98 (stale) Mirror 15: 0x91e2cd98 (stale) Mirror 16: 0x91e2cd98 (stale) Mirror 17: 0x91e2cd98 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29560: skipped 000478 1789063021.702412 fallocate 0x01ab3b thru 0x01b59a (0xa60 bytes) 000479 1789063021.709161 punch from 0x025517 to 0x02955f (0x4049 bytes) 000480 1789063021.717419 punch from 0x01ce34 to 0x022ea7 (0x6074 bytes) 000481 1789063021.724839 mapwrite 0x01b04f thru 0x025df3 (0xada5 bytes) 000482 1789063021.784730 read 0x0248b8 thru 0x02955f (0x4ca8 bytes) 000485 1789063021.794309 punch from 0x0017db to 0x00f1c2 (0xd9e8 bytes) 000486 1789063021.801080 trunc from 0x029560 to 0x00f608 (0x19f57 bytes) 000487 1789063021.834452 punch from 0x0067da to 0x00d3b0 (0x6bd7 bytes) 000488 1789063021.846026 punch from 0x00e083 to 0x00f608 (0x1586 bytes) 000489 1789063021.852091 fallocate 0x00d6ba thru 0x01403a (0x6981 bytes) 000490 1789063021.859731 punch from 0x00d992 to 0x00f608 (0x1c77 bytes) 000491 1789063021.866487 trunc from 0x00e160 to 0x00f608 (0x14a9 bytes) 000492 1789063021.872523 trunc from 0x0038d7 to 0x00f608 (0xbd32 bytes) 000493 1789063021.878210 mapwrite 0x02f517 thru 0x038413 (0x8efd bytes) 000494 1789063022.020141 trunc from 0x026932 to 0x035977 (0xf046 bytes) 000496 1789063022.031991 write 0x01e3ae thru 0x02a980 (0xc5d3 bytes) 000497 1789063022.088916 write 0x020824 thru 0x02f89b (0xf078 bytes) 000498 1789063022.130387 MIRROR_RESYNC 8 mirrors 000499 1789063023.866367 write 0x03ea70 thru 0x03ffff (0x1590 bytes) 000500 1789063023.900858 mapwrite 0x029fca thru 0x034d79 (0xadb0 bytes) 000501 1789063023.955160 mapwrite 0x0158d1 thru 0x02350f (0xdc3f bytes) 000502 1789063023.994477 trunc from 0x021465 to 0x027d71 (0x690d bytes) 000503 1789063023.999462 punch from 0x02ad46 to 0x037155 (0xc410 bytes) 000504 1789063024.009107 trunc from 0x00cd7c to 0x01c11f (0xf3a4 bytes) 000505 1789063024.032189 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 8: (stale - read error) Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 15: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0xf78b0e26 (stale) Mirror 8: 0xf78b0e26 (stale) Mirror 11: 0xf78b0e26 (stale) Mirror 14: 0xf78b0e26 (stale) Mirror 15: 0xf78b0e26 (stale) Mirror 16: 0xf78b0e26 (stale) Mirror 17: 0xf78b0e26 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x6505604a Mirror 18: 0x6505604a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000506 1789063025.334758 MIRROR_RESYNC 9 mirrors 000507 1789063027.993805 write 0x034bf1 thru 0x03f973 (0xad83 bytes) 000509 1789063028.067378 trunc from 0x040000 to 0x02f356 (0x10ca9 bytes) 000510 1789063028.091732 mapwrite 0x02955e thru 0x02df3d (0x49e0 bytes) 000511 1789063028.130054 punch from 0x005854 to 0x012cd9 (0xd486 bytes) 000512 1789063028.135059 read 0x0170e4 thru 0x022e7f (0xbd9c bytes) 000513 1789063028.146578 punch from 0x0125b8 to 0x015a3e (0x3487 bytes) 000514 1789063028.152383 mapread 0x009471 thru 0x010fad (0x7b3d bytes) 000515 1789063028.168484 MIRROR_SPLIT mirror 5 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f357): Mirror 3: 0xa30b42d0 (stale) Mirror 8: 0xa30b42d0 (stale) Mirror 11: 0xa30b42d0 (stale) Mirror 14: 0xa30b42d0 (stale) Mirror 16: 0xa30b42d0 (stale) Mirror 17: 0xa30b42d0 (stale) Mirror 18: 0xa30b42d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f357): Mirror 3: 0xa30b42d0 (stale) Mirror 8: 0xa30b42d0 (stale) Mirror 11: 0xa30b42d0 (stale) Mirror 14: 0xa30b42d0 (stale) Mirror 16: 0xa30b42d0 (stale) Mirror 17: 0xa30b42d0 (stale) Mirror 18: 0xa30b42d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f357: skipped 000516 1789063028.738178 MIRROR_SPLIT mirror 7 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2f357): Mirror 3: 0xa30b42d0 (stale) Mirror 8: 0xa30b42d0 (stale) Mirror 11: 0xa30b42d0 (stale) Mirror 14: 0xa30b42d0 (stale) Mirror 16: 0xa30b42d0 (stale) Mirror 17: 0xa30b42d0 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f357): Mirror 3: 0xa30b42d0 (stale) Mirror 8: 0xa30b42d0 (stale) Mirror 11: 0xa30b42d0 (stale) Mirror 14: 0xa30b42d0 (stale) Mirror 16: 0xa30b42d0 (stale) Mirror 17: 0xa30b42d0 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f357: skipped 000517 1789063029.488721 mapread 0x014c17 thru 0x01589a (0xc84 bytes) 000518 1789063029.512044 write 0x021177 thru 0x02e88b (0xd715 bytes) 000519 1789063029.546329 write 0x012823 thru 0x01e503 (0xbce1 bytes) 000520 1789063029.567213 mapwrite 0x03980a thru 0x03ffff (0x67f6 bytes) 000521 1789063029.738749 MIRROR_RESYNC 7 mirrors 000522 1789063031.889680 MIRROR_RESYNC 7 mirrors 000523 1789063032.745877 trunc from 0x040000 to 0x02a0a9 (0x15f56 bytes) 000524 1789063032.806064 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a0aa): Mirror 3: 0xfe2e4ff (stale) Mirror 11: 0xfe2e4ff (stale) Mirror 14: 0xfe2e4ff (stale) Mirror 16: 0xfe2e4ff (stale) Mirror 17: 0xfe2e4ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a0aa): Mirror 3: 0xfe2e4ff (stale) Mirror 11: 0xfe2e4ff (stale) Mirror 14: 0xfe2e4ff (stale) Mirror 16: 0xfe2e4ff (stale) Mirror 17: 0xfe2e4ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a0aa: skipped 000525 1789063033.663848 punch from 0x015ced to 0x022c2f (0xcf43 bytes) 000526 1789063033.674407 trunc from 0x00d573 to 0x01428a (0x6d18 bytes) 000527 1789063033.700049 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a0aa): Mirror 11: 0xfe2e4ff (stale) Mirror 14: 0xfe2e4ff (stale) Mirror 16: 0xfe2e4ff (stale) Mirror 17: 0xfe2e4ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a0aa): Mirror 11: 0xfe2e4ff (stale) Mirror 14: 0xfe2e4ff (stale) Mirror 16: 0xfe2e4ff (stale) Mirror 17: 0xfe2e4ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a0aa: skipped 000528 1789063034.215163 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x2a0aa): Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 17: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a0aa): Mirror 11: 0xfe2e4ff (stale) Mirror 14: 0xfe2e4ff (stale) Mirror 16: 0xfe2e4ff (stale) Mirror 17: 0xfe2e4ff (stale) CRC-32 checksum value for chunk [0, 0x2a0aa): Mirror 1: 0xb7a0f73e Mirror 18: 0xb7a0f73e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a0aa: skipped 000529 1789063035.049783 write 0x005f28 thru 0x00c74a (0x6823 bytes) 000530 1789063035.105593 trunc from 0x02a0aa to 0x032f2c (0x8e83 bytes) 000531 1789063035.133701 mapwrite 0x00c047 thru 0x01ac1d (0xebd7 bytes) 000532 1789063035.180683 read 0x012ed7 thru 0x01c1fa (0x9324 bytes) 000533 1789063035.181654 punch from 0x0000e7 to 0x008fbc (0x8ed6 bytes) 000534 1789063035.187408 write 0x037f9c thru 0x03fca4 (0x7d09 bytes) 000535 1789063035.189389 mapwrite 0x027233 thru 0x02f2ee (0x80bc bytes) 000537 1789063035.216901 fallocate 0x02e797 thru 0x034bda (0x6444 bytes) 000538 1789063035.222381 write 0x018d86 thru 0x01e9c1 (0x5c3c bytes) 000539 1789063035.223664 fallocate 0x01ec8b thru 0x02d7a4 (0xeb1a bytes) 000540 1789063035.231395 trunc from 0x03fca5 to 0x02767e (0x18626 bytes) 000541 1789063035.250680 trunc from 0x02767f to 0x01effd (0x8681 bytes) 000542 1789063035.267436 fallocate 0x000f35 thru 0x005bb7 (0x4c83 bytes) 000543 1789063035.272361 trunc from 0x01effe to 0x014029 (0xafd4 bytes) 000544 1789063035.290390 read 0x001db4 thru 0x00e611 (0xc85e bytes) 000545 1789063035.301345 trunc from 0x002635 to 0x009e1e (0x77ea bytes) 000546 1789063035.306544 read 0x00de9f thru 0x014029 (0x618b bytes) 000547 1789063035.328435 MIRROR_RESYNC 6 mirrors 000548 1789063037.146882 mapwrite 0x0151e1 thru 0x018956 (0x3776 bytes) 000549 1789063037.258969 read 0x005047 thru 0x00b57b (0x6535 bytes) 000550 1789063037.277578 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18957): Mirror 11: 0x2e241891 (stale) Mirror 14: 0x2e241891 (stale) Mirror 16: 0x2e241891 (stale) Mirror 18: 0x2e241891 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18957): Mirror 11: 0x2e241891 (stale) Mirror 14: 0x2e241891 (stale) Mirror 16: 0x2e241891 (stale) Mirror 18: 0x2e241891 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18957: skipped 000551 1789063037.749770 punch from 0x0046ef to 0x00c149 (0x7a5b bytes) 000552 1789063037.760506 trunc from 0x008dac to 0x015d5c (0xcfb1 bytes) 000553 1789063037.767751 mapwrite 0x024603 thru 0x031925 (0xd323 bytes) 000554 1789063037.844506 write 0x02829c thru 0x02c35e (0x40c3 bytes) 000556 1789063037.846167 trunc from 0x01d0db to 0x020d3b (0x3c61 bytes) 000557 1789063037.858212 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x31926): Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x31926): Mirror 11: 0x2c39dea2 (stale) Mirror 14: 0x2c39dea2 (stale) Mirror 16: 0x2c39dea2 (stale) Mirror 18: 0x2c39dea2 (stale) CRC-32 checksum value for chunk [0, 0x31926): Mirror 1: 0xc08aad4a Mirror 19: 0xc08aad4a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x31926: skipped 000558 1789063038.893272 trunc from 0x008ce2 to 0x018875 (0xfb94 bytes) 000559 1789063038.941711 trunc from 0x031926 to 0x0013b8 (0x3056d bytes) 000560 1789063039.013831 mapread 0x000b60 thru 0x0013b8 (0x859 bytes) 000561 1789063039.076896 fallocate 0x004b00 thru 0x008871 (0x3d72 bytes) 000562 1789063039.106857 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x13b9): Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 16: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x13b9): Mirror 11: 0x1f4e596b (stale) Mirror 14: 0x1f4e596b (stale) Mirror 16: 0x1f4e596b (stale) Mirror 18: 0x1f4e596b (stale) Mirror 19: 0x1f4e596b (stale) CRC-32 checksum value for chunk [0, 0x13b9): Mirror 1: 0x1f4e596b Mirror 20: 0x1f4e596b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x13b9: skipped 000563 1789063040.647744 trunc from 0x0013b9 to 0x0025e5 (0x122d bytes) 000564 1789063040.732563 trunc from 0x0025e6 to 0x039b00 (0x3751b bytes) 000565 1789063040.754374 read 0x00f4fc thru 0x0194ea (0x9fef bytes) 000566 1789063040.761336 trunc from 0x039b01 to 0x01cf5c (0x1cba4 bytes) 000567 1789063040.776742 fallocate 0x01004a thru 0x01dd79 (0xdd30 bytes) 000568 1789063040.781170 mapread 0x007760 thru 0x013c93 (0xc534 bytes) 000569 1789063040.796153 write 0x0043a0 thru 0x008fcb (0x4c2c bytes) 000570 1789063040.797141 read 0x00f4ec thru 0x01dd79 (0xe88e bytes) 000571 1789063040.805834 trunc from 0x004057 to 0x011dc2 (0xdd6c bytes) 000572 1789063040.813478 mapwrite 0x02dfb1 thru 0x034c5d (0x6cad bytes) 000573 1789063040.842842 mapwrite 0x027ae4 thru 0x0315ea (0x9b07 bytes) 000574 1789063040.857530 trunc from 0x034c5e to 0x00a870 (0x2a3ed bytes) 000575 1789063040.868785 read 0x006251 thru 0x00867a (0x242a bytes) 000576 1789063040.874379 trunc from 0x00804d to 0x00a870 (0x2824 bytes) 000578 1789063040.878577 write 0x03fefa thru 0x03ffff (0x106 bytes) 000580 1789063040.880775 trunc from 0x040000 to 0x03e395 (0x1c6a bytes) 000581 1789063040.905675 fallocate 0x0016c0 thru 0x00d8a9 (0xc1ea bytes) 000582 1789063040.914100 read 0x01a2ba thru 0x01abbe (0x905 bytes) 000583 1789063040.932676 punch from 0x0376c2 to 0x03d40f (0x5d4e bytes) 000585 1789063040.950664 MIRROR_SPLIT mirror 3 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e396): Mirror 11: 0x7aa5ac51 (stale) Mirror 14: 0x7aa5ac51 (stale) Mirror 18: 0x7aa5ac51 (stale) Mirror 19: 0x4b3b197f (stale) Mirror 20: 0x1c9ffb14 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e396): Mirror 11: 0x7aa5ac51 (stale) Mirror 14: 0x7aa5ac51 (stale) Mirror 18: 0x7aa5ac51 (stale) Mirror 19: 0x4b3b197f (stale) Mirror 20: 0x1c9ffb14 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e396: skipped 000586 1789063041.634795 fallocate 0x0291c3 thru 0x02a9c5 (0x1803 bytes) 000587 1789063041.646801 read 0x01d6bf thru 0x027f97 (0xa8d9 bytes) 000588 1789063041.665796 punch from 0x0193be to 0x01e78b (0x53ce bytes) 000589 1789063041.690046 mapread 0x00603f thru 0x006b38 (0xafa bytes) 000590 1789063041.708355 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x3e396): Mirror 11: (stale - read error) Mirror 14: (stale - read error) Mirror 18: (stale - read error) Mirror 19: (stale - read error) Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e396): Mirror 11: 0x7aa5ac51 (stale) Mirror 14: 0x7aa5ac51 (stale) Mirror 18: 0x7aa5ac51 (stale) Mirror 19: 0x4b3b197f (stale) Mirror 20: 0x1c9ffb14 (stale) CRC-32 checksum value for chunk [0, 0x3e396): Mirror 1: 0x1c9ffb14 Mirror 21: 0x1c9ffb14 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e396: skipped 000591 1789063042.623360 punch from 0x01b40f to 0x025fe7 (0xabd9 bytes) 000592 1789063042.676858 mapwrite 0x031179 thru 0x035665 (0x44ed bytes) 000593 1789063042.760993 read 0x034b9b thru 0x03e395 (0x97fb bytes) 000594 1789063042.774473 trunc from 0x03e396 to 0x0364cc (0x7ec9 bytes) 000595 1789063042.814029 write 0x0070b5 thru 0x012e31 (0xbd7d bytes) 000596 1789063042.827159 trunc from 0x0364cd to 0x0269af (0xfb1d bytes) 000597 1789063042.853658 mapwrite 0x0071f9 thru 0x011ac3 (0xa8cb bytes) 000598 1789063042.871458 write 0x00a2e1 thru 0x00f110 (0x4e30 bytes) 000599 1789063042.873270 mapwrite 0x024dc9 thru 0x034c16 (0xfe4e bytes) 000600 1789063042.944253 mapread 0x001b61 thru 0x0044b5 (0x2955 bytes) 000601 1789063042.957222 write 0x013b62 thru 0x018ec7 (0x5366 bytes) 000602 1789063042.969297 trunc from 0x034c17 to 0x00c949 (0x282cd bytes) 000603 1789063042.994168 mapwrite 0x017417 thru 0x0182d1 (0xebb bytes) 000604 1789063043.031221 mapread 0x00f39f thru 0x013529 (0x418b bytes) 000605 1789063043.038178 fallocate 0x0032ce thru 0x009c28 (0x695b bytes) 000606 1789063043.046608 read 0x012dca thru 0x0182d1 (0x5508 bytes) 000607 1789063043.048493 trunc from 0x00788b to 0x0088d4 (0x104a bytes) 000608 1789063043.070909 fallocate 0x01d189 thru 0x027a52 (0xa8ca bytes) 000610 1789063043.104051 mapread 0x00d8b0 thru 0x0182d1 (0xaa22 bytes) 000611 1789063043.105108 mapwrite 0x02e491 thru 0x03570b (0x727b bytes) 000612 1789063043.262638 punch from 0x027bc9 to 0x0345a1 (0xc9d9 bytes) 000613 1789063043.286637 write 0x00d92c thru 0x017b27 (0xa1fc bytes) 000614 1789063043.288213 punch from 0x00a42f to 0x00a6c6 (0x298 bytes) 000615 1789063043.299907 mapwrite 0x03a399 thru 0x03b53b (0x11a3 bytes) 000618 1789063043.411919 mapread 0x004bd4 thru 0x00d292 (0x86bf bytes) 000619 1789063043.443001 trunc from 0x03b53c to 0x009f28 (0x31613 bytes) 000620 1789063043.520523 mapread 0x00054c thru 0x009f28 (0x99dd bytes) 000621 1789063043.538214 write 0x0012cb thru 0x005998 (0x46ce bytes) 000622 1789063043.549545 read 0x00425a thru 0x009f28 (0x5ccf bytes) 000623 1789063043.562244 MIRROR_SPLIT mirror 2 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9f29): Mirror 11: 0x80d19afd (stale) Mirror 18: 0x80d19afd (stale) Mirror 19: 0xc866989 (stale) Mirror 20: 0xc866989 (stale) Mirror 21: 0xc866989 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9f29): Mirror 11: 0x80d19afd (stale) Mirror 18: 0x80d19afd (stale) Mirror 19: 0xc866989 (stale) Mirror 20: 0xc866989 (stale) Mirror 21: 0xc866989 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9f29: skipped 000624 1789063044.670858 mapwrite 0x006c1b thru 0x0072f8 (0x6de bytes) 000625 1789063044.759767 fallocate 0x01b9b9 thru 0x02a01f (0xe667 bytes) 000626 1789063044.768622 read 0x008d0c thru 0x009f28 (0x121d bytes) 000627 1789063044.776901 trunc from 0x000c84 to 0x001317 (0x694 bytes) 000628 1789063044.785370 trunc from 0x000c32 to 0x009f28 (0x92f7 bytes) 000629 1789063044.813887 MIRROR_SPLIT mirror 2 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x9f29): Mirror 11: 0x80d19afd (stale) Mirror 19: 0xc866989 (stale) Mirror 20: 0xc866989 (stale) Mirror 21: 0xc866989 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x9f29): Mirror 11: 0x80d19afd (stale) Mirror 19: 0xc866989 (stale) Mirror 20: 0xc866989 (stale) Mirror 21: 0xc866989 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x9f29: skipped 000630 1789063045.562877 mapread 0x009342 thru 0x009f28 (0xbe7 bytes) 000632 1789063045.573968 mapwrite 0x011995 thru 0x011a12 (0x7e bytes) 000633 1789063045.645586 fallocate 0x035130 thru 0x03ffff (0xaed0 bytes) 000634 1789063045.657863 write 0x0288a2 thru 0x029c1f (0x137e bytes) 000635 1789063045.660605 punch from 0x00cf64 to 0x0135a4 (0x6641 bytes) 000636 1789063045.675440 mapwrite 0x004dbb thru 0x007913 (0x2b59 bytes) 000637 1789063045.737208 read 0x022ec7 thru 0x025e3d (0x2f77 bytes) 000638 1789063045.794812 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x29c20): Mirror 11: 0x821008c5 (stale) Mirror 19: 0x820271e6 (stale) Mirror 20: 0x313e848f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29c20): Mirror 11: 0x821008c5 (stale) Mirror 19: 0x820271e6 (stale) Mirror 20: 0x313e848f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29c20: skipped 000639 1789063046.735824 fallocate 0x02da0a thru 0x03c41c (0xea13 bytes) 000641 1789063046.749532 read 0x0218c9 thru 0x022718 (0xe50 bytes) 000642 1789063046.762054 trunc from 0x00502d to 0x00fbef (0xabc3 bytes) 000643 1789063046.778457 trunc from 0x00f2b8 to 0x013b67 (0x48b0 bytes) 000644 1789063046.787864 read 0x0099ea thru 0x00aeed (0x1504 bytes) 000645 1789063046.851032 fallocate 0x02650b thru 0x0298c4 (0x33ba bytes) 000646 1789063046.869385 mapread 0x008499 thru 0x017b09 (0xf671 bytes) 000647 1789063046.870919 fallocate 0x02c657 thru 0x0351e0 (0x8b8a bytes) 000648 1789063046.918385 write 0x01ce69 thru 0x0275ba (0xa752 bytes) 000649 1789063046.943692 fallocate 0x039441 thru 0x03ffff (0x6bbf bytes) 000650 1789063046.967244 mapwrite 0x02c6e8 thru 0x034beb (0x8504 bytes) 000651 1789063047.111292 read 0x03e146 thru 0x03ffff (0x1eba bytes) 000652 1789063047.120301 fallocate 0x037f15 thru 0x03ffff (0x80eb bytes) 000653 1789063047.131152 punch from 0x005bac to 0x006ccb (0x1120 bytes) 000654 1789063047.148179 fallocate 0x013a3e thru 0x0239a9 (0xff6c bytes) 000656 1789063047.180377 trunc from 0x040000 to 0x00bb9a (0x34465 bytes) 000657 1789063047.267677 mapread 0x004620 thru 0x00bb9a (0x757b bytes) 000658 1789063047.293832 punch from 0x001e49 to 0x0042ca (0x2482 bytes) 000659 1789063047.311738 mapwrite 0x02b2b0 thru 0x03b21c (0xff6d bytes) 000660 1789063047.479676 trunc from 0x01035b to 0x01bd5c (0xba02 bytes) 000661 1789063047.526793 MIRROR_RESYNC 4 mirrors 000662 1789063049.517651 punch from 0x00b965 to 0x010d9d (0x5439 bytes) 000663 1789063049.560508 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b21d): Mirror 11: 0xb9b90ce8 (stale) Mirror 20: 0xb9b90ce8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b21d): Mirror 11: 0xb9b90ce8 (stale) Mirror 20: 0xb9b90ce8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b21d: skipped 000664 1789063050.395530 mapread 0x002e33 thru 0x0096b8 (0x6886 bytes) 000665 1789063050.444267 MIRROR_RESYNC 3 mirrors 000666 1789063052.519813 MIRROR_RESYNC 3 mirrors 000667 1789063053.561691 mapwrite 0x02d014 thru 0x033e84 (0x6e71 bytes) 000668 1789063054.396725 fallocate 0x021279 thru 0x02f088 (0xde10 bytes) 000669 1789063054.402331 punch from 0x01ff70 to 0x02669a (0x672b bytes) 000670 1789063054.408339 mapwrite 0x030a0c thru 0x033d0f (0x3304 bytes) 000671 1789063054.482722 fallocate 0x02a6cb thru 0x031090 (0x69c6 bytes) 000673 1789063054.507197 write 0x02e76a thru 0x02f84b (0x10e2 bytes) 000674 1789063054.508153 punch from 0x02aa48 to 0x037447 (0xca00 bytes) 000675 1789063054.531849 trunc from 0x006225 to 0x00fbe7 (0x99c3 bytes) 000676 1789063054.542792 write 0x02b266 thru 0x0330a3 (0x7e3e bytes) 000677 1789063054.570119 read 0x01c6c3 thru 0x02a970 (0xe2ae bytes) 000678 1789063054.606708 trunc from 0x03b21d to 0x03dc50 (0x2a34 bytes) 000679 1789063054.685745 mapwrite 0x024611 thru 0x026e00 (0x27f0 bytes) 000680 1789063054.715570 mapwrite 0x010345 thru 0x01c69d (0xc359 bytes) 000681 1789063054.889225 punch from 0x02fc60 to 0x03dc50 (0xdff1 bytes) 000682 1789063054.897914 write 0x039e4f thru 0x03ffff (0x61b1 bytes) 000685 1789063054.906716 trunc from 0x040000 to 0x027236 (0x18dc9 bytes) 000686 1789063054.980774 MIRROR_RESYNC 3 mirrors 000687 1789063057.361394 punch from 0x00fa24 to 0x017a58 (0x8035 bytes) 000688 1789063057.430140 mapread 0x01dad7 thru 0x027236 (0x9760 bytes) 000689 1789063057.459487 mapwrite 0x038d4a thru 0x03a99d (0x1c54 bytes) 000690 1789063057.547496 read 0x013e61 thru 0x021422 (0xd5c2 bytes) 000691 1789063057.586420 read 0x02ac98 thru 0x02f2a5 (0x460e bytes) 000692 1789063057.627269 trunc from 0x03a99e to 0x00b3ff (0x2f59e bytes) 000694 1789063057.707297 read 0x005288 thru 0x0081e4 (0x2f5d bytes) 000695 1789063057.714404 write 0x00a22f thru 0x0166f7 (0xc4c9 bytes) 000696 1789063057.719849 punch from 0x00c0e1 to 0x015b1e (0x9a3e bytes) 000697 1789063057.791225 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x166f8): Mirror 20: 0x5aec8db3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x166f8): Mirror 20: 0x5aec8db3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x166f8: skipped 000698 1789063058.480522 trunc from 0x012fb8 to 0x0166f7 (0x3740 bytes) 000699 1789063058.488172 fallocate 0x007f98 thru 0x013b5f (0xbbc8 bytes) 000700 1789063058.497497 write 0x031ae4 thru 0x03aa11 (0x8f2e bytes) 000701 1789063058.524394 punch from 0x007300 to 0x0147d2 (0xd4d3 bytes) 000702 1789063058.536956 trunc from 0x02b1be to 0x0326ee (0x7531 bytes) 000703 1789063058.604895 trunc from 0x0381b1 to 0x03aa11 (0x2861 bytes) 000704 1789063058.622125 trunc from 0x03aa12 to 0x038230 (0x27e1 bytes) 000705 1789063058.672180 punch from 0x00302a to 0x003836 (0x80d bytes) 000706 1789063058.685584 mapwrite 0x01662b thru 0x016c8f (0x665 bytes) 000707 1789063058.731439 punch from 0x003a48 to 0x0045cd (0xb86 bytes) 000708 1789063058.740672 mapwrite 0x02d984 thru 0x03cdd1 (0xf44e bytes) 000709 1789063058.944993 trunc from 0x03cdd2 to 0x00a774 (0x3265d bytes) 000710 1789063058.967395 mapwrite 0x03c462 thru 0x03de2c (0x19cb bytes) 000711 1789063059.041225 trunc from 0x0058ac to 0x0109a8 (0xb0fd bytes) 000712 1789063059.059731 mapwrite 0x001a75 thru 0x008f66 (0x74f2 bytes) 000713 1789063059.100242 trunc from 0x03de2d to 0x03c2f1 (0x1b3b bytes) 000714 1789063059.147610 trunc from 0x002acc to 0x010fdb (0xe510 bytes) 000715 1789063059.159173 trunc from 0x03c2f2 to 0x032e03 (0x94ee bytes) 000716 1789063059.178042 mapread 0x027426 thru 0x02aca0 (0x387b bytes) 000717 1789063059.206516 trunc from 0x000a53 to 0x003178 (0x2726 bytes) 000718 1789063059.217627 trunc from 0x032e04 to 0x01abd0 (0x18233 bytes) 000719 1789063059.260235 trunc from 0x017a7a to 0x01abd0 (0x3157 bytes) 000720 1789063059.272259 trunc from 0x01abd1 to 0x00d252 (0xd97e bytes) 000721 1789063059.301012 punch from 0x0034c0 to 0x006386 (0x2ec7 bytes) 000722 1789063059.306822 read 0x009742 thru 0x00a947 (0x1206 bytes) 000723 1789063059.317361 fallocate 0x00dd48 thru 0x00fa0f (0x1cc8 bytes) 000724 1789063059.325767 trunc from 0x00ecd2 to 0x00fa0f (0xd3e bytes) 000725 1789063059.344014 punch from 0x004bc5 to 0x00cf4e (0x838a bytes) 000726 1789063059.349879 trunc from 0x00a72f to 0x00f2a9 (0x4b7b bytes) 000727 1789063059.356349 mapwrite 0x021778 thru 0x028df4 (0x767d bytes) 000728 1789063059.512400 fallocate 0x01b1cc thru 0x026db6 (0xbbeb bytes) 000729 1789063059.529082 mapread 0x00b88a thru 0x00c315 (0xa8c bytes) 000730 1789063059.588406 MIRROR_RESYNC 2 mirrors 000731 1789063060.483550 fallocate 0x032a60 thru 0x032cba (0x25b bytes) 000733 1789063060.592812 trunc from 0x013915 to 0x01dbcd (0xa2b9 bytes) 000734 1789063060.597685 trunc from 0x032cbb to 0x02f912 (0x33a8 bytes) 000736 1789063060.654040 write 0x018520 thru 0x023485 (0xaf66 bytes) 000737 1789063060.691091 write 0x018d7e thru 0x02889f (0xfb22 bytes) 000738 1789063060.704238 trunc from 0x02f913 to 0x02ed1e (0xbf4 bytes) 000739 1789063060.737087 write 0x027a5e thru 0x02fd34 (0x82d7 bytes) 000740 1789063060.746095 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2fd35): Mirror 20: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2fd35): Mirror 20: 0xe9090019 (stale) CRC-32 checksum value for chunk [0, 0x2fd35): Mirror 1: 0xedefdb7a Mirror 21: 0xedefdb7a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2fd35: skipped 000741 1789063061.797833 trunc from 0x02fd35 to 0x029091 (0x6ca3 bytes) 000742 1789063061.852167 trunc from 0x029092 to 0x002f13 (0x2617e bytes) 000743 1789063061.868578 write 0x00d023 thru 0x0172dc (0xa2ba bytes) 000744 1789063061.870473 write 0x024923 thru 0x03270d (0xddeb bytes) 000745 1789063061.873128 mapwrite 0x00e692 thru 0x01428b (0x5bfa bytes) 000746 1789063061.886958 fallocate 0x03e10b thru 0x03ffff (0x1ef5 bytes) 000747 1789063061.893531 mapread 0x033a6f thru 0x0342ba (0x84c bytes) 000750 1789063061.915457 trunc from 0x021a31 to 0x0305e5 (0xebb5 bytes) 000751 1789063061.928532 write 0x009fed thru 0x0118a5 (0x78b9 bytes) 000752 1789063061.936652 trunc from 0x0101aa to 0x01e1f5 (0xe04c bytes) 000753 1789063061.966022 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x3f8f5646 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x3f8f5646 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000754 1789063062.323459 trunc from 0x011b5b to 0x012001 (0x4a7 bytes) 000755 1789063062.329922 mapwrite 0x03c00d thru 0x03ffff (0x3ff3 bytes) 000756 1789063062.385207 trunc from 0x040000 to 0x032ebb (0xd144 bytes) 000757 1789063062.425987 trunc from 0x022774 to 0x0257b3 (0x3040 bytes) 000758 1789063062.435558 fallocate 0x031efd thru 0x03490f (0x2a13 bytes) 000759 1789063062.450381 write 0x039073 thru 0x03ffff (0x6f8d bytes) 000760 1789063062.463274 MIRROR_RESYNC 2 mirrors 000761 1789063063.666754 read 0x004b24 thru 0x00c916 (0x7df3 bytes) 000762 1789063063.692870 mapwrite 0x0163e7 thru 0x0223fb (0xc015 bytes) 000764 1789063063.950115 punch from 0x029ced to 0x03999c (0xfcb0 bytes) 000765 1789063063.966186 mapread 0x008f6d thru 0x014af2 (0xbb86 bytes) 000766 1789063064.022231 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 21: 0x3f2d6227 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0xa6e6322 Mirror 22: 0xa6e6322 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000767 1789063065.620978 write 0x00dc8b thru 0x0109d6 (0x2d4c bytes) 000768 1789063065.666630 trunc from 0x000f3b to 0x001570 (0x636 bytes) 000769 1789063065.671385 fallocate 0x036ce0 thru 0x037024 (0x345 bytes) 000770 1789063065.675772 punch from 0x01f9d4 to 0x023fc4 (0x45f1 bytes) 000771 1789063065.687060 trunc from 0x011df1 to 0x014738 (0x2948 bytes) 000772 1789063065.710421 trunc from 0x040000 to 0x02e863 (0x1179c bytes) 000773 1789063065.748397 mapwrite 0x032047 thru 0x03b67d (0x9637 bytes) 000774 1789063065.825358 read 0x036b28 thru 0x03915d (0x2636 bytes) 000775 1789063065.826604 mapread 0x0079f5 thru 0x00f789 (0x7d95 bytes) 000776 1789063065.855640 mapwrite 0x027bf6 thru 0x033956 (0xbd61 bytes) 000777 1789063065.924885 write 0x00b636 thru 0x010e97 (0x5862 bytes) 000778 1789063065.927038 read 0x01ca34 thru 0x02b059 (0xe626 bytes) 000779 1789063065.952886 fallocate 0x0149c1 thru 0x023084 (0xe6c4 bytes) 000780 1789063065.991796 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b67e): Mirror 22: 0xbaf3f1f8 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b67e): Mirror 22: 0xbaf3f1f8 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b67e: skipped 000781 1789063066.436014 trunc from 0x01df44 to 0x02c0d3 (0xe190 bytes) 000782 1789063066.444835 write 0x00ea4e thru 0x00f802 (0xdb5 bytes) 000783 1789063066.461121 read 0x018dcf thru 0x023fba (0xb1ec bytes) 000784 1789063066.470242 punch from 0x034965 to 0x03b67d (0x6d19 bytes) 000785 1789063066.481072 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b67e): Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b67e): Mirror 22: 0xbaf3f1f8 (stale) CRC-32 checksum value for chunk [0, 0x3b67e): Mirror 1: 0x7e63721a Mirror 23: 0x7e63721a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b67e: skipped 000786 1789063067.612365 fallocate 0x011b6a thru 0x0185dc (0x6a73 bytes) 000787 1789063067.698196 MIRROR_RESYNC 3 mirrors 000788 1789063069.138849 read 0x0089e0 thru 0x00b85f (0x2e80 bytes) 000789 1789063069.155025 punch from 0x02a787 to 0x0322ad (0x7b27 bytes) 000790 1789063069.222009 trunc from 0x008ca7 to 0x016d93 (0xe0ed bytes) 000791 1789063069.231024 mapread 0x001ba5 thru 0x001bf0 (0x4c bytes) 000792 1789063069.256330 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3b67e): Mirror 22: 0x7e63721a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b67e): Mirror 22: 0x7e63721a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b67e: skipped 000794 1789063069.588029 fallocate 0x0193db thru 0x01f2cb (0x5ef1 bytes) 000795 1789063069.595172 trunc from 0x00ff88 to 0x012426 (0x249f bytes) 000796 1789063069.601232 fallocate 0x01ebe0 thru 0x02d8ce (0xecef bytes) 000797 1789063069.607350 punch from 0x020a27 to 0x02556c (0x4b46 bytes) 000798 1789063069.612832 mapread 0x034e8f thru 0x03b67d (0x67ef bytes) 000799 1789063069.654511 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3b67e): Mirror 22: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b67e): Mirror 22: 0x7e63721a (stale) CRC-32 checksum value for chunk [0, 0x3b67e): Mirror 1: 0x223196b8 Mirror 23: 0x223196b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b67e: skipped 000800 1789063070.809086 fallocate 0x0063e1 thru 0x00bb13 (0x5733 bytes) 000801 1789063070.872398 mapwrite 0x02f2f1 thru 0x03658d (0x729d bytes) 000803 1789063070.978350 punch from 0x02c51d to 0x02e655 (0x2139 bytes) 000804 1789063070.998652 punch from 0x028c13 to 0x034927 (0xbd15 bytes) 000805 1789063071.007121 mapwrite 0x008d10 thru 0x0151e9 (0xc4da bytes) 000806 1789063071.075178 read 0x0321f6 thru 0x032d20 (0xb2b bytes) 000807 1789063071.081230 read 0x007eef thru 0x00b1df (0x32f1 bytes) 000808 1789063071.087565 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3b67e): Mirror 22: (stale - read error) Mirror 23: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3b67e): Mirror 22: 0x7e63721a (stale) Mirror 23: 0x223196b8 (stale) CRC-32 checksum value for chunk [0, 0x3b67e): Mirror 1: 0x8b3dc78e Mirror 24: 0x8b3dc78e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3b67e: skipped 000809 1789063072.670096 trunc from 0x00f0a1 to 0x018ddc (0x9d3c bytes) 000810 1789063072.781095 read 0x01adb7 thru 0x01dd58 (0x2fa2 bytes) 000811 1789063072.803807 write 0x012a1a thru 0x018793 (0x5d7a bytes) 000812 1789063072.847328 fallocate 0x004470 thru 0x009b59 (0x56ea bytes) 000813 1789063072.851584 write 0x009a05 thru 0x0176bd (0xdcb9 bytes) 000814 1789063072.880283 trunc from 0x02c9b3 to 0x03099d (0x3feb bytes) 000815 1789063072.900872 MIRROR_RESYNC 4 mirrors 000817 1789063075.387110 mapread 0x02e9df thru 0x03497b (0x5f9d bytes) 000818 1789063075.437794 read 0x010f3e thru 0x0193a1 (0x8464 bytes) 000819 1789063075.459287 punch from 0x01e9b2 to 0x02df9d (0xf5ec bytes) 000820 1789063075.518780 punch from 0x025a62 to 0x029229 (0x37c8 bytes) 000821 1789063075.529574 punch from 0x02f12c to 0x03b67d (0xc552 bytes) 000822 1789063075.550136 MIRROR_RESYNC 4 mirrors 000823 1789063077.229452 trunc from 0x02bf48 to 0x02f965 (0x3a1e bytes) 000824 1789063077.282025 fallocate 0x0281f1 thru 0x0368dd (0xe6ed bytes) 000825 1789063077.294488 trunc from 0x03b67e to 0x02065f (0x1b01e bytes) 000826 1789063077.352670 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x20660): Mirror 22: 0xab97d8ff (stale) Mirror 24: 0xab97d8ff (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x20660): Mirror 22: 0xab97d8ff (stale) Mirror 24: 0xab97d8ff (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x20660: skipped 000827 1789063077.853072 write 0x025b65 thru 0x027bc3 (0x205f bytes) 000828 1789063077.865904 read 0x00dcba thru 0x017d4d (0xa094 bytes) 000829 1789063077.881878 read 0x002b71 thru 0x0069b5 (0x3e45 bytes) 000830 1789063077.903186 fallocate 0x0027d7 thru 0x00b127 (0x8951 bytes) 000831 1789063077.914216 fallocate 0x00015f thru 0x0094ca (0x936c bytes) 000832 1789063077.936879 read 0x02243b thru 0x0240f1 (0x1cb7 bytes) 000833 1789063077.938245 read 0x02098a thru 0x027bc3 (0x723a bytes) 000834 1789063077.939035 fallocate 0x03abf1 thru 0x03ffff (0x540f bytes) 000835 1789063077.979664 fallocate 0x0041cf thru 0x00f47a (0xb2ac bytes) 000837 1789063077.996129 write 0x0281d7 thru 0x02ba87 (0x38b1 bytes) 000838 1789063078.011700 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2ba88): Mirror 22: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2ba88): Mirror 22: 0x50e6b03b (stale) Mirror 24: 0x50e6b03b (stale) CRC-32 checksum value for chunk [0, 0x2ba88): Mirror 1: 0x78aa056 Mirror 25: 0x78aa056 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2ba88: skipped 000839 1789063079.604400 punch from 0x02a1b3 to 0x02ba87 (0x18d5 bytes) 000840 1789063079.665426 punch from 0x011729 to 0x016b04 (0x53dc bytes) 000841 1789063079.671363 write 0x0390c5 thru 0x03cf3f (0x3e7b bytes) 000842 1789063079.679654 read 0x01fc45 thru 0x028867 (0x8c23 bytes) 000843 1789063079.708147 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3cf40): Mirror 22: 0x1cb9e42b (stale) Mirror 24: 0x1cb9e42b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cf40): Mirror 22: 0x1cb9e42b (stale) Mirror 24: 0x1cb9e42b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cf40: skipped 000844 1789063080.359643 mapwrite 0x00b809 thru 0x01b2e7 (0xfadf bytes) 000845 1789063080.472371 punch from 0x003618 to 0x00ceab (0x9894 bytes) 000847 1789063080.481670 write 0x0112dd thru 0x013e19 (0x2b3d bytes) 000848 1789063080.492771 read 0x000b20 thru 0x00f4f3 (0xe9d4 bytes) 000849 1789063080.505572 mapwrite 0x03140a thru 0x031b38 (0x72f bytes) 000850 1789063080.555975 trunc from 0x03cf40 to 0x03a7fd (0x2742 bytes) 000851 1789063080.591313 punch from 0x01eecf to 0x020bce (0x1d00 bytes) 000852 1789063080.606152 mapwrite 0x01b0ba thru 0x01c53b (0x1482 bytes) 000853 1789063080.641288 mapread 0x01534a thru 0x0226b7 (0xd36e bytes) 000854 1789063080.667391 trunc from 0x0234d2 to 0x023c96 (0x7c5 bytes) 000855 1789063080.675588 mapwrite 0x0164ad thru 0x0188bd (0x2411 bytes) 000856 1789063080.721470 mapwrite 0x00a5bf thru 0x019ec6 (0xf908 bytes) 000857 1789063080.794982 fallocate 0x00df2f thru 0x014023 (0x60f5 bytes) 000858 1789063080.807511 read 0x035da4 thru 0x03a05f (0x42bc bytes) 000859 1789063080.839621 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3a7fe): Mirror 22: (stale - read error) Mirror 24: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a7fe): Mirror 22: 0x95aac52b (stale) Mirror 24: 0x95aac52b (stale) CRC-32 checksum value for chunk [0, 0x3a7fe): Mirror 1: 0xfe9e4d53 Mirror 25: 0xfe9e4d53 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a7fe: skipped 000860 1789063082.712406 read 0x02c273 thru 0x02d1db (0xf69 bytes) 000863 1789063082.742137 punch from 0x00471c to 0x011d9d (0xd682 bytes) 000865 1789063082.787565 fallocate 0x03f3ea thru 0x03ffff (0xc16 bytes) 000866 1789063082.793343 mapwrite 0x03d4ac thru 0x03ffff (0x2b54 bytes) 000867 1789063082.855374 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 22: 0xb73dfe99 (stale) Mirror 24: 0xb73dfe99 (stale) Mirror 25: 0x333f0721 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x92a863c3 Mirror 26: 0x92a863c3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000868 1789063084.631262 mapread 0x02df17 thru 0x02fdd9 (0x1ec3 bytes) 000869 1789063084.711554 read 0x00c458 thru 0x01c42a (0xffd3 bytes) 000870 1789063084.745619 mapread 0x035206 thru 0x03c789 (0x7584 bytes) 000871 1789063084.820157 trunc from 0x027d28 to 0x03342e (0xb707 bytes) 000872 1789063084.926209 punch from 0x00272a to 0x010f8d (0xe864 bytes) 000873 1789063084.932819 fallocate 0x01513d thru 0x01e8de (0x97a2 bytes) 000874 1789063084.940654 trunc from 0x023d26 to 0x027e4e (0x4129 bytes) 000875 1789063084.953418 punch from 0x01bbfc to 0x02b057 (0xf45c bytes) 000876 1789063084.963773 write 0x0008a1 thru 0x00ca3d (0xc19d bytes) 000877 1789063085.022557 punch from 0x019193 to 0x0261d8 (0xd046 bytes) 000878 1789063085.029427 read 0x033ec2 thru 0x03ffff (0xc13e bytes) 000879 1789063085.162569 mapread 0x008906 thru 0x017c25 (0xf320 bytes) 000880 1789063085.163964 mapwrite 0x004d69 thru 0x00b2d1 (0x6569 bytes) 000881 1789063085.257409 fallocate 0x021094 thru 0x02bc32 (0xab9f bytes) 000882 1789063085.283742 read 0x023c0c thru 0x0333a3 (0xf798 bytes) 000883 1789063085.285341 write 0x007fe8 thru 0x015d3b (0xdd54 bytes) 000884 1789063085.292609 punch from 0x005228 to 0x00d2c4 (0x809d bytes) 000885 1789063085.326131 mapread 0x0140c7 thru 0x0208fa (0xc834 bytes) 000886 1789063085.336123 write 0x0113c6 thru 0x017db5 (0x69f0 bytes) 000887 1789063085.338773 trunc from 0x040000 to 0x0296c2 (0x1693d bytes) 000888 1789063085.412048 mapwrite 0x02d179 thru 0x02fa06 (0x288e bytes) 000889 1789063085.633002 fallocate 0x022667 thru 0x0251fa (0x2b94 bytes) 000890 1789063085.654614 read 0x02bc03 thru 0x02fa06 (0x3e04 bytes) 000891 1789063085.655650 mapwrite 0x027e2b thru 0x0311ad (0x9383 bytes) 000892 1789063085.767740 fallocate 0x032ca2 thru 0x039388 (0x66e7 bytes) 000893 1789063085.782712 mapread 0x016aee thru 0x02277c (0xbc8f bytes) 000894 1789063085.789222 fallocate 0x008209 thru 0x008f1c (0xd14 bytes) 000895 1789063085.825029 mapread 0x02bcaa thru 0x032549 (0x68a0 bytes) 000896 1789063085.864106 trunc from 0x039389 to 0x017f35 (0x21453 bytes) 000897 1789063085.896858 punch from 0x0080b6 to 0x00f3be (0x7309 bytes) 000898 1789063085.938293 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x17f36): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 25: (stale - read error) Mirror 26: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x17f36): Mirror 22: 0x90f0c214 (stale) Mirror 24: 0x90f0c214 (stale) Mirror 25: 0x71ebde28 (stale) Mirror 26: 0x5afb5cec (stale) CRC-32 checksum value for chunk [0, 0x17f36): Mirror 1: 0xc703a605 Mirror 27: 0xc703a605 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x17f36: skipped 000901 1789063087.429764 fallocate 0x01d7e5 thru 0x027534 (0x9d50 bytes) 000902 1789063087.506437 trunc from 0x01fc57 to 0x027534 (0x78de bytes) 000904 1789063087.518533 trunc from 0x027535 to 0x033a63 (0xc52f bytes) 000906 1789063087.564617 mapwrite 0x018f24 thru 0x01bff8 (0x30d5 bytes) 000907 1789063087.617370 trunc from 0x01cd34 to 0x028550 (0xb81d bytes) 000908 1789063087.625005 trunc from 0x033a64 to 0x03b868 (0x7e05 bytes) 000909 1789063087.648796 punch from 0x007ff1 to 0x00a107 (0x2117 bytes) 000910 1789063087.656000 write 0x02574d thru 0x02b23a (0x5aee bytes) 000911 1789063087.669709 trunc from 0x0368ea to 0x03b271 (0x4988 bytes) 000912 1789063087.674667 punch from 0x027b4b to 0x02cfa4 (0x545a bytes) 000913 1789063087.692054 trunc from 0x03b869 to 0x02af3e (0x1092a bytes) 000914 1789063087.731865 trunc from 0x0044ff to 0x012bf0 (0xe6f2 bytes) 000915 1789063087.746894 trunc from 0x000670 to 0x000bff (0x590 bytes) 000916 1789063087.756583 trunc from 0x02af3f to 0x03bcb2 (0x10d74 bytes) 000917 1789063087.783431 trunc from 0x03bcb3 to 0x027385 (0x1492d bytes) 000918 1789063087.810584 fallocate 0x01c6e0 thru 0x01e5e9 (0x1f0a bytes) 000919 1789063087.818634 trunc from 0x00ea26 to 0x015559 (0x6b34 bytes) 000920 1789063087.834812 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x27386): Mirror 22: 0xa2c6dfa8 (stale) Mirror 24: 0xa2c6dfa8 (stale) Mirror 26: 0xe2a65b61 (stale) Mirror 27: 0xf8edd24e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x27386): Mirror 22: 0xa2c6dfa8 (stale) Mirror 24: 0xa2c6dfa8 (stale) Mirror 26: 0xe2a65b61 (stale) Mirror 27: 0xf8edd24e (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x27386: skipped 000921 1789063088.843649 trunc from 0x01f5b4 to 0x027385 (0x7dd2 bytes) 000922 1789063088.869810 trunc from 0x01cc8d to 0x026793 (0x9b07 bytes) 000923 1789063088.890079 mapwrite 0x01d6da thru 0x029bc1 (0xc4e8 bytes) 000924 1789063089.110809 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x29bc2): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x29bc2): Mirror 22: 0xc2b04a9b (stale) Mirror 24: 0xc2b04a9b (stale) Mirror 26: 0xbbc3447d (stale) Mirror 27: 0x482f7458 (stale) CRC-32 checksum value for chunk [0, 0x29bc2): Mirror 1: 0x31272bca Mirror 28: 0x31272bca lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x29bc2: skipped 000925 1789063090.880312 mapread 0x0028f3 thru 0x009ab3 (0x71c1 bytes) 000926 1789063090.956439 punch from 0x007ae2 to 0x013849 (0xbd68 bytes) 000927 1789063091.067599 punch from 0x018a14 to 0x026fe5 (0xe5d2 bytes) 000928 1789063091.081052 mapwrite 0x005f02 thru 0x013e81 (0xdf80 bytes) 000929 1789063091.185497 read 0x014a05 thru 0x0248f1 (0xfeed bytes) 000930 1789063091.207666 trunc from 0x029bc2 to 0x008ef5 (0x20ccc bytes) 000931 1789063091.282725 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x8ef6): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x8ef6): Mirror 22: 0x3bfdb522 (stale) Mirror 24: 0x3bfdb522 (stale) Mirror 26: 0xfaa03534 (stale) Mirror 27: 0xe8506bc0 (stale) Mirror 28: 0xbb706a6e (stale) CRC-32 checksum value for chunk [0, 0x8ef6): Mirror 1: 0xa03b44cf Mirror 29: 0xa03b44cf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x8ef6: skipped 000932 1789063092.482038 trunc from 0x008ef6 to 0x0032d0 (0x5c25 bytes) 000933 1789063092.559182 fallocate 0x02f386 thru 0x03606c (0x6ce7 bytes) 000934 1789063092.563794 write 0x027773 thru 0x031ebb (0xa749 bytes) 000935 1789063092.566602 fallocate 0x01aae2 thru 0x0215c8 (0x6ae7 bytes) 000936 1789063092.570910 punch from 0x010eb5 to 0x01567f (0x47cb bytes) 000937 1789063092.576036 trunc from 0x031ebc to 0x0061b9 (0x2bd02 bytes) 000940 1789063092.600352 trunc from 0x0061ba to 0x013d27 (0xdb6e bytes) 000941 1789063092.620035 punch from 0x00afb1 to 0x010512 (0x5562 bytes) 000942 1789063092.624610 read 0x006094 thru 0x00d894 (0x7801 bytes) 000943 1789063092.633656 punch from 0x009f16 to 0x013d27 (0x9e12 bytes) 000944 1789063092.638604 mapwrite 0x018878 thru 0x02378b (0xaf14 bytes) 000945 1789063092.687495 trunc from 0x02378c to 0x00757b (0x1c210 bytes) 000946 1789063092.708229 fallocate 0x013060 thru 0x014a2c (0x19cd bytes) 000947 1789063092.717841 MIRROR_RESYNC 7 mirrors 000948 1789063094.361418 mapread 0x0015a3 thru 0x00757b (0x5fd9 bytes) 000949 1789063094.405538 read 0x00747e thru 0x00757b (0xfe bytes) 000950 1789063094.405923 write 0x033b47 thru 0x03cc8e (0x9148 bytes) 000951 1789063094.481451 trunc from 0x012c7d to 0x016b7d (0x3f01 bytes) 000952 1789063094.486906 trunc from 0x01a378 to 0x01c1d4 (0x1e5d bytes) 000953 1789063094.496726 fallocate 0x016d92 thru 0x025dec (0xf05b bytes) 000954 1789063094.520643 MIRROR_EXTEND to 8 mirrors CRC-32 checksum value for chunk [0, 0x3cc8f): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3cc8f): Mirror 22: 0xbadb831f (stale) Mirror 24: 0xbadb831f (stale) Mirror 26: 0xbadb831f (stale) Mirror 27: 0xbadb831f (stale) Mirror 28: 0xbadb831f (stale) Mirror 29: 0xbadb831f (stale) CRC-32 checksum value for chunk [0, 0x3cc8f): Mirror 1: 0x933e521e Mirror 30: 0x933e521e lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3cc8f: skipped 000955 1789063095.975002 write 0x008cec thru 0x00ce65 (0x417a bytes) 000956 1789063096.044884 write 0x02e533 thru 0x03df9f (0xfa6d bytes) 000957 1789063096.060889 mapread 0x0068fa thru 0x006da2 (0x4a9 bytes) 000958 1789063096.069631 punch from 0x011c6c to 0x016b91 (0x4f26 bytes) 000959 1789063096.078871 write 0x023bcb thru 0x02f783 (0xbbb9 bytes) 000960 1789063096.098649 punch from 0x0123be to 0x01615d (0x3da0 bytes) 000961 1789063096.105536 mapwrite 0x0287b3 thru 0x03321a (0xaa68 bytes) 000962 1789063096.151364 write 0x02d030 thru 0x030f33 (0x3f04 bytes) 000963 1789063096.153115 trunc from 0x03dfa0 to 0x00e91b (0x2f684 bytes) 000965 1789063096.185031 punch from 0x00b15e to 0x00b89a (0x73d bytes) 000966 1789063096.193911 write 0x013356 thru 0x01a46d (0x7118 bytes) 000967 1789063096.198962 punch from 0x01482b to 0x01a3da (0x5bb0 bytes) 000968 1789063096.224879 fallocate 0x01c86f thru 0x02aa56 (0xe1e8 bytes) 000969 1789063096.238784 punch from 0x015a59 to 0x01e5ce (0x8b76 bytes) 000970 1789063096.245258 read 0x00eb0c thru 0x016a8f (0x7f84 bytes) 000971 1789063096.259670 trunc from 0x000b72 to 0x00b99d (0xae2c bytes) 000972 1789063096.286542 MIRROR_EXTEND to 9 mirrors CRC-32 checksum value for chunk [0, 0x2aa57): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 27: (stale - read error) Mirror 28: (stale - read error) Mirror 29: (stale - read error) Mirror 30: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aa57): Mirror 22: 0x897c027e (stale) Mirror 24: 0x897c027e (stale) Mirror 26: 0x897c027e (stale) Mirror 27: 0x897c027e (stale) Mirror 28: 0x897c027e (stale) Mirror 29: 0x897c027e (stale) Mirror 30: 0x897c027e (stale) CRC-32 checksum value for chunk [0, 0x2aa57): Mirror 1: 0x68722b85 Mirror 31: 0x68722b85 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aa57: skipped 000973 1789063097.928562 MIRROR_SPLIT mirror 4 to 8 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2aa57): Mirror 22: 0x897c027e (stale) Mirror 24: 0x897c027e (stale) Mirror 26: 0x897c027e (stale) Mirror 28: 0x897c027e (stale) Mirror 29: 0x897c027e (stale) Mirror 30: 0x897c027e (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aa57): Mirror 22: 0x897c027e (stale) Mirror 24: 0x897c027e (stale) Mirror 26: 0x897c027e (stale) Mirror 28: 0x897c027e (stale) Mirror 29: 0x897c027e (stale) Mirror 30: 0x897c027e (stale) CRC-32 checksum value for chunk [0, 0x2aa57): Mirror 1: 0x68722b85 Mirror 31: 0x68722b85 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aa57: skipped 000975 1789063099.014448 fallocate 0x00977a thru 0x012b87 (0x940e bytes) 000976 1789063099.146857 punch from 0x00adc2 to 0x014a2a (0x9c69 bytes) 000977 1789063099.154338 fallocate 0x011556 thru 0x01f065 (0xdb10 bytes) 000978 1789063099.169660 MIRROR_SPLIT mirror 4 to 7 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2aa57): Mirror 22: 0x897c027e (stale) Mirror 24: 0x897c027e (stale) Mirror 26: 0x897c027e (stale) Mirror 29: 0x897c027e (stale) Mirror 30: 0x897c027e (stale) Mirror 31: 0x68722b85 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2aa57): Mirror 22: 0x897c027e (stale) Mirror 24: 0x897c027e (stale) Mirror 26: 0x897c027e (stale) Mirror 29: 0x897c027e (stale) Mirror 30: 0x897c027e (stale) Mirror 31: 0x68722b85 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2aa57: skipped 000979 1789063099.858887 write 0x031b12 thru 0x03e5ff (0xcaee bytes) 000980 1789063099.878220 MIRROR_SPLIT mirror 5 to 6 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x1b:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3e600): Mirror 22: 0x3ac85ba4 (stale) Mirror 24: 0x3ac85ba4 (stale) Mirror 26: 0x3ac85ba4 (stale) Mirror 29: 0x3ac85ba4 (stale) Mirror 31: 0x539cf524 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3e600): Mirror 22: 0x3ac85ba4 (stale) Mirror 24: 0x3ac85ba4 (stale) Mirror 26: 0x3ac85ba4 (stale) Mirror 29: 0x3ac85ba4 (stale) Mirror 31: 0x539cf524 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3e600: skipped 000981 1789063100.748902 trunc from 0x039129 to 0x03e5ff (0x54d7 bytes) 000982 1789063100.774524 mapread 0x02737f thru 0x036b80 (0xf802 bytes) 000983 1789063100.847752 trunc from 0x01661f to 0x019e0d (0x37ef bytes) 000984 1789063100.854013 read 0x0182c4 thru 0x01f7ad (0x74ea bytes) 000985 1789063100.870486 read 0x025f88 thru 0x032ed7 (0xcf50 bytes) 000986 1789063100.871589 punch from 0x02138a to 0x02605e (0x4cd5 bytes) 000987 1789063100.890796 trunc from 0x03e600 to 0x02cb8f (0x11a70 bytes) 000988 1789063100.972675 mapwrite 0x0176a5 thru 0x01e8b5 (0x7211 bytes) 000989 1789063101.038628 read 0x026965 thru 0x02cb8f (0x622b bytes) 000990 1789063101.042509 mapread 0x02376d thru 0x02cb8f (0x9423 bytes) 000991 1789063101.053719 MIRROR_EXTEND to 7 mirrors CRC-32 checksum value for chunk [0, 0x2cb90): Mirror 22: (stale - read error) Mirror 24: (stale - read error) Mirror 26: (stale - read error) Mirror 29: (stale - read error) Mirror 31: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2cb90): Mirror 22: 0xb67a3dfc (stale) Mirror 24: 0xb67a3dfc (stale) Mirror 26: 0xb67a3dfc (stale) Mirror 29: 0xb67a3dfc (stale) Mirror 31: 0x930d1f2a (stale) CRC-32 checksum value for chunk [0, 0x2cb90): Mirror 1: 0x555f403b Mirror 32: 0x555f403b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2cb90: skipped 000992 1789063103.319576 write 0x02b045 thru 0x03a21d (0xf1d9 bytes) 000993 1789063103.420161 punch from 0x00eb56 to 0x019e4c (0xb2f7 bytes) 000994 1789063103.425864 mapwrite 0x01708c thru 0x026997 (0xf90c bytes) 000995 1789063103.529741 punch from 0x0150a2 to 0x01c458 (0x73b7 bytes) 000996 1789063103.544211 fallocate 0x005f1a thru 0x006318 (0x3ff bytes) 000997 1789063103.556490 mapwrite 0x0344d3 thru 0x03e5b2 (0xa0e0 bytes) 000998 1789063103.684434 read 0x0171b9 thru 0x024ea8 (0xdcf0 bytes) 001000 1789063103.708135 write 0x03edde thru 0x03ffff (0x1222 bytes) All operations completed A-OK! PASS 70a (166s) == sanity-flr test 71: check mirror extend parent fid ==== 13:58:32 (1789063112) striped dir -i1 -c2 -H all_char /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x200000405:0x1be:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (13s) == sanity-flr test 72: check append on FLR file ========== 13:58:45 (1789063125) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.217287 s, 9.7 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2cf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0756113 s, 13.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2cf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ed:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ce:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2cf:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ed:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x2ec:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2d0:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (11s) == sanity-flr test 200a: stress test ===================== 13:58:56 (1789063136) Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre2 Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8698880 .. Extending file size to 1298336 .. done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6100640 .. Extending file size to 8291712 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8752128 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9072480 .. failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61436 61437 61440 61442 61443 done 192.168.201.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg147-client.virtnet /mnt/lustre2 (opts:) 192.168.201.147@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg147-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (94s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 14:00:30 (1789063230) Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre2 Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3933696 .. done Extending file size to 3992000 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 7937024 .. Extending file size to 5943936 .. resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8157344 .. Extending file size to 7596480 .. Extending file size to 8292832 .. Extending file size to 9390944 .. Extending file size to 8470304 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8923040 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9037760 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 62618 62619 62622 62623 62626 done 192.168.201.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg147-client.virtnet /mnt/lustre2 (opts:) 192.168.201.147@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg147-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (82s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 14:01:52 (1789063312) Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2412 1268892 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2084 1269220 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 19068 3587952 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 25032 3581988 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 44100 7169940 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2412 1268892 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2084 1269220 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 19068 3587952 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 25032 3581988 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 44100 7169940 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.221336 s, 9.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.2641 s, 7.9 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x925b2865 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.201.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg147-client.virtnet /mnt/lustre2 (opts:) PASS 200c (23s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 14:02:18 (1789063338) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0836262 s, 25.1 MB/s PASS 202 (8s) == sanity-flr test 203: mirror file preserve mirror ID === 14:02:26 (1789063346) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1cd:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x303:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x303:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x304:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e5:0x0] } PASS 203 (9s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 14:02:35 (1789063355) striped dir -i0 -c2 -H all_char /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132244 s, 15.9 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x305:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e6:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e7:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 14:02:45 (1789063365) striped dir -i0 -c2 -H crush /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0713459 s, 29.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x307:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x306:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x307:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e8:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2e9:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 14:02:56 (1789063376) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.121816 s, 17.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x53:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x54:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x53:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x54:0x0] } PASS 204c (39s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 14:03:34 (1789063414) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg147-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157048 s, 6.7 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x308:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ea:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x308:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ea:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x309:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 14:04:34 (1789063474) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg147-server mds-ost sync done. striped dir -i0 -c2 -H crush /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x30b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0882895 s, 23.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x30b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ec:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x30b:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ec:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 14:05:31 (1789063531) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg147-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg147-server: Pool lustre.test_204f created Adding targets to pool oleg147-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg147-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.108633 s, 19.3 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x30c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x30c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2ed:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg147-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg147-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg147-server: Pool lustre.test_204f destroyed PASS 204f (82s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 14:06:53 (1789063613) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 14:07:03 (1789063623) PASS 205b (12s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 14:07:14 (1789063634) oleg147-server: Pool lustre.test_206 created Adding targets to pool oleg147-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg147-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg147-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg147-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg147-server: Pool lustre.test_206 destroyed PASS 206 (34s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 14:07:48 (1789063668) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116652 s, 9.0 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629681 s, 16.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0xba63764d Mirror 3: 0xba63764d PASS 207 (14s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 14:08:03 (1789063683) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 14:08:07 (1789063687) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 14:08:12 (1789063692) Starting client: oleg147-client.virtnet: -o user_xattr,flock 192.168.201.147@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0815955 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x1650000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x7e0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x7e0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x7e0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1a50000 Read this (4 bytes): BBBB 192.168.201.147@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg147-client.virtnet /mnt/lustre2 (opts:) PASS 209a (45s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 14:08:56 (1789063736) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0359705 s, 228 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 14:09:04 (1789063744) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427098 s, 24.6 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1f0 lmm_fid: [0x200000405:0x1f0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x1f0 lmm_fid: [0x200000405:0x1f0:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x2f8:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611632 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-10 14:09:06.000000000 -0400 Modify: 2026-09-10 14:09:06.000000000 -0400 Change: 2026-09-10 14:09:07.000000000 -0400 Birth: 2026-09-10 14:09:06.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 14:09:14 (1789063754) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369062 s, 28.4 MB/s oleg147-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg147-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg147-server: O/280000401/d22/316: File not found by ext2_lookup PASS 210b (27s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 14:09:41 (1789063781) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.292142 s, 140 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0318598 s, 129 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 10 14:09 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x1f5:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 10 14:09 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 14:09:50 (1789063790) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (9s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 14:10:00 (1789063800) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 2096 sec =========== 14:10:10 (1789063810) === sanity-flr: start cleanup 14:10:12 (1789063812) === === sanity-flr: finish cleanup 14:10:21 (1789063821) ===