-----============= acceptance-small: sanity-flr ============----- Thu Sep 3 15:20:18 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg619-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 15:20:37 (1788463237) === oleg619-client.virtnet: executing check_config_client /mnt/lustre oleg619-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg619-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9575c2c5e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9575c2c5e800.idle_timeout=debug disable quota as required oleg619-server: oleg619-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 15:21:13 (1788463273) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7631] == sanity-flr test 0a: lfs mirror create with -N option == 15:21:14 (1788463274) PASS 0a (20s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 15:21:34 (1788463294) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 15:21:37 (1788463297) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 15:21:41 (1788463301) PASS 0d (26s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 15:22:08 (1788463328) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 15:22:11 (1788463331) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 15:22:15 (1788463335) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9575c2c5e800.stats=clear osc.lustre-OST0001-osc-ffff9575c2c5e800.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 15:22:24 (1788463344) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (11s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 15:22:35 (1788463355) PASS 0j (8s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 15:22:43 (1788463363) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (3s) == sanity-flr test 0m: mirror extend with -N= option ===== 15:22:46 (1788463366) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 15:22:54 (1788463374) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 15:23:04 (1788463384) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 15:23:11 (1788463391) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (4s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 15:23:15 (1788463395) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 15:23:23 (1788463403) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (8s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 15:23:34 (1788463414) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 15:23:41 (1788463421) 11+0 records in 11+0 records out 11534336 bytes (12 MB, 11 MiB) copied, 1.81264 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181982 s, 5.8 MB/s PASS 21 (11s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 15:23:52 (1788463432) 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 0.494142 s, 42.4 MB/s PASS 22 (8s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 15:24:00 (1788463440) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (10s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 15:24:10 (1788463450) Stopping /mnt/lustre-ost1 (opts:) on oleg619-server oleg619-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9575c2c5e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9575c2c5e800.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg619-server: oleg619-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg619-server: oleg619-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg619-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg619-server oleg619-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9575c2c5e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9575c2c5e800.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg619-server: oleg619-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg619-server: oleg619-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg619-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (71s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 15:25:22 (1788463522) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.516022 s, 18.3 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg619-server oleg619-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9575c2c5e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9575c2c5e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg619-server: oleg619-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg619-server: oleg619-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg619-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (43s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 15:26:05 (1788463565) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg619-server oleg619-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9575c2c5e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9575c2c5e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg619-server: oleg619-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg619-server: oleg619-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg619-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg619-server oleg619-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9575c2c5e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9575c2c5e800.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg619-server: oleg619-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg619-server: oleg619-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg619-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (82s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 15:27:27 (1788463647) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 5.68541 s, 9.2 MB/s -rw-r--r-- 1 root root 52428800 Sep 3 15:27 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.121s user 0m0.001s sys 0m0.170s Stopping /mnt/lustre-ost1 (opts:) on oleg619-server oleg619-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9575c2c5e800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9575c2c5e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.220s user 0m0.001s sys 0m0.181s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg619-server: oleg619-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg619-server: oleg619-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg619-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg619-server oleg619-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9575c2c5e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9575c2c5e800.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.441s user 0m0.000s sys 0m0.179s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg619-server: oleg619-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg619-server: oleg619-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg619-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg619-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9575c2c5e800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9575c2c5e800.ost_server_uuid in FULL state after 0 sec PASS 33b (98s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 15:29:05 (1788463745) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 15:29:08 (1788463748) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 15:29:12 (1788463752) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (3s) == sanity-flr test 35: allow to write to mirrored files == 15:29:15 (1788463755) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 15:29:23 (1788463763) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 1.0339 s, 12.2 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (15s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 15:29:38 (1788463778) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237075 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457197 s, 22.9 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 15:29:45 (1788463785) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 15:29:53 (1788463793) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.001255 s, 816 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00148536 s, 689 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00146571 s, 699 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00104041 s, 984 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 15:30:04 (1788463804) create test files with size 15745742 .. 1e0cccab953c5afce2b6bac045f62a80 - 76cd2517f5f69f245e36d1646cdd2647 - eadc2016f9aa640ed942cdc46ed594a0 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg619-client.virtnet /mnt/lustre (opts:) Starting client: oleg619-client.virtnet: -o user_xattr,flock 192.168.206.119@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 15:30:27 (1788463827) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 15:30:44 (1788463844) PASS 39 (8s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 15:30:51 (1788463851) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201483 s, 5.2 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc0:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc1:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdb:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440917 s, 23.8 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdc:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xdd:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc2:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 15:31:02 (1788463862) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.189704 s, 11.1 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.39152 s, 10.7 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.355795 s, 5.9 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 15:31:21 (1788463881) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 15:31:25 (1788463885) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (3s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 15:31:28 (1788463888) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587309 s, 17.9 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 15:31:36 (1788463896) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 15:31:39 (1788463899) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xe4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xc9:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 15:31:49 (1788463909) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.293561 s, 35.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg619-client.virtnet /mnt/lustre (opts:) Starting client: oleg619-client.virtnet: -o user_xattr,flock 192.168.206.119@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 15:32:02 (1788463922) llite.lustre-ffff9575f2606000.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.303989 s, 34.5 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg619-client.virtnet /mnt/lustre (opts:) Starting client: oleg619-client.virtnet: -o user_xattr,flock 192.168.206.119@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff9575d0679800.xattr_cache=1 PASS 44e (14s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 15:32:16 (1788463936) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (38s) == sanity-flr test 46: Verify setstripe --copy option ==== 15:32:55 (1788463975) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (39s) == sanity-flr test 47: Verify mirror obj alloc =========== 15:33:34 (1788464014) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 15:33:38 (1788464018) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.24775 s, 12.7 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.137556 s, 22.9 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x72b79b52 (nosync,stale) Mirror 2: 0x72b79b52 (nosync,stale) ** verify mirror 2 doesn't change original checksum: 6880476b039476adbcc61cd5d09b8af4 - mirror 1 checksum: 4a90e6e3d777fbd471d631ec847fed19 - mirror 2 checksum: 6880476b039476adbcc61cd5d09b8af4 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-03T15:33:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-09-03T15:33:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 6880476b039476adbcc61cd5d09b8af4 - mirror 1 checksum: 4a90e6e3d777fbd471d631ec847fed19 - mirror 2 checksum: 4a90e6e3d777fbd471d631ec847fed19 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-03T15:33:42-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xde:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-09-03T15:33:42-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xdf:0x0] } PASS 48 (12s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 15:33:52 (1788464032) Failing mds1 on oleg619-server Stopping /mnt/lustre-mds1 (opts:) on oleg619-server 15:33:59 (1788464039) shut down facet: mds1 facet_host: oleg619-server facet_failover_host: oleg619-server Failover mds1 to oleg619-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg619-server: oleg619-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg619-client: oleg619-server: ssh exited with exit code 1 Started lustre-MDT0000 15:34:23 (1788464063) targets are mounted 15:34:23 (1788464063) facet_failover done oleg619-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0367198 s, 28.6 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe0:0x0] } PASS 50A (48s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 15:34:41 (1788464081) SKIP: sanity-flr test_50a lseek for ZFS is not accurate if obj is not committed SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 15:34:44 (1788464084) SKIP: sanity-flr test_50b lseek for ZFS is not accurate if obj is not committed SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 15:34:48 (1788464088) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0175506 s, 934 kB/s PASS 50c (9s) == sanity-flr test 50d: mirror rsync keep holes ========== 15:34:58 (1788464098) SKIP: sanity-flr test_50d lseek for ZFS is not accurate if obj is not committed SKIP 50d (5s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 15:35:02 (1788464102) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00596638 s, 687 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.136181 s, 30.1 kB/s PASS 60a (9s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 15:35:11 (1788464111) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x22:0x0]: Invalid argument (22) PASS 60b (24s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 15:35:35 (1788464135) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (29s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 15:36:04 (1788464164) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x53:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (27s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 15:36:31 (1788464191) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (27s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 15:36:58 (1788464218) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (11s) == sanity-flr test 70: mirror create and split race ====== 15:37:09 (1788464229) mirror create pid 54331 mirror split pid 54333 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54331 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5114: 54333 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (69s) == sanity-flr test 70a: flr mode fsx test ================ 15:38:19 (1788464299) Seed set to 4300 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x3529f 000001 1788464301.224943 trunc from 00000000 to 0x03529e (0x3529f bytes) 000002 1788464301.241346 punch from 0x02a6e5 to 0x03187b (0x7197 bytes) 000003 1788464301.245349 trunc from 0x03529f to 0x00e67e (0x26c20 bytes) truncating to largest ever: 0x3e02a 000004 1788464301.265084 trunc from 0x00e67f to 0x03e029 (0x2f9ab bytes) 000006 1788464301.282389 write 0x03a8c2 thru 0x03ffff (0x573e bytes) 000007 1788464301.294833 punch from 0x03c69a to 0x03cfa9 (0x910 bytes) 000009 1788464301.322646 read 0x036abe thru 0x03e4a9 (0x79ec bytes) 000010 1788464301.346060 punch from 0x03884e to 0x03ffff (0x77b2 bytes) 000011 1788464301.360243 punch from 0x01b5dc to 0x029810 (0xe235 bytes) 000013 1788464301.369850 write 0x02ceea thru 0x03021a (0x3331 bytes) 000015 1788464301.394562 read 0x0117cc thru 0x01fde3 (0xe618 bytes) 000017 1788464301.415773 mapwrite 0x0362c7 thru 0x03ffff (0x9d39 bytes) 000018 1788464301.509188 MIRROR_EXTEND to 2 mirrors 000019 1788464302.599965 MIRROR_RESYNC 2 mirrors 000020 1788464303.219852 write 0x02f09d thru 0x0337e4 (0x4748 bytes) 000021 1788464303.384306 mapwrite 0x023930 thru 0x02d5d9 (0x9caa bytes) 000023 1788464303.588274 mapread 0x037f6c thru 0x03d07a (0x510f bytes) 000026 1788464303.614654 trunc from 0x040000 to 0x021bff (0x1e400 bytes) 000027 1788464303.662811 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x21c00): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x21c00): Mirror 2: 0x46cd28ff (stale) CRC-32 checksum value for chunk [0, 0x21c00): Mirror 1: 0x46cd28ff Mirror 3: 0x46cd28ff lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x21c00: skipped 000029 1788464305.033989 read 0x010fdd thru 0x011e29 (0xe4d bytes) 000031 1788464305.068133 write 0x00fa0e thru 0x013342 (0x3935 bytes) 000036 1788464305.187864 punch from 0x020ccb to 0x021bff (0xf35 bytes) 000037 1788464305.206304 read 0x003022 thru 0x009ad8 (0x6ab7 bytes) 000039 1788464305.238187 mapread 0x00891f thru 0x018792 (0xfe74 bytes) 000040 1788464305.239254 trunc from 0x021c00 to 0x01b1fc (0x6a03 bytes) 000042 1788464305.273376 mapread 0x0151e6 thru 0x017719 (0x2534 bytes) 000043 1788464305.275966 write 0x03d8e1 thru 0x03ffff (0x271f bytes) 000044 1788464305.277812 read 0x03d3cb thru 0x03ffff (0x2c35 bytes) 000046 1788464305.279192 trunc from 0x040000 to 0x004b59 (0x3b4a6 bytes) 000048 1788464305.305880 read 0x0013f4 thru 0x00432a (0x2f37 bytes) 000049 1788464305.306490 punch from 0x00112e to 0x004b59 (0x3a2c bytes) 000050 1788464305.316005 read 0x0017d4 thru 0x002671 (0xe9e bytes) 000051 1788464305.325407 mapwrite 0x02275c thru 0x030f4b (0xe7f0 bytes) 000052 1788464305.613039 trunc from 0x030f4c to 0x03a32e (0x93e3 bytes) 000053 1788464305.634975 trunc from 0x03a32f to 0x00e9de (0x2b950 bytes) 000054 1788464305.654314 mapwrite 0x035025 thru 0x038829 (0x3805 bytes) 000056 1788464305.851427 mapwrite 0x032b60 thru 0x03a578 (0x7a19 bytes) 000057 1788464306.025549 read 0x03169a thru 0x037f56 (0x68bd bytes) 000058 1788464306.026195 punch from 0x00ad9d to 0x00cb2c (0x1d90 bytes) 000059 1788464306.035743 punch from 0x02154c to 0x02da3a (0xc4ef bytes) 000060 1788464306.041867 punch from 0x02c1d5 to 0x03a578 (0xe3a4 bytes) 000063 1788464306.068492 punch from 0x006a51 to 0x00ca5f (0x600f bytes) 000064 1788464306.078702 trunc from 0x03a579 to 0x022b4d (0x17a2b bytes) 000066 1788464306.129044 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x22b4e): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x22b4e): Mirror 2: 0x8159e4a0 (stale) Mirror 3: 0x8159e4a0 (stale) CRC-32 checksum value for chunk [0, 0x22b4e): Mirror 1: 0x8159e4a0 Mirror 4: 0x8159e4a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x22b4e: skipped 000067 1788464307.643927 MIRROR_RESYNC 4 mirrors 000068 1788464308.841926 read 0x00dc16 thru 0x01d74b (0xfb36 bytes) 000069 1788464308.875951 mapwrite 0x0208b1 thru 0x02cfab (0xc6fb bytes) 000070 1788464309.174245 trunc from 0x02cfac to 0x006641 (0x2696a bytes) 000071 1788464309.203700 mapread 0x000051 thru 0x003fcf (0x3f7f bytes) 000072 1788464309.212609 punch from 0x00525c to 0x006641 (0x13e6 bytes) 000073 1788464309.215762 mapread 0x001a61 thru 0x004664 (0x2c04 bytes) 000075 1788464309.217895 punch from 0x00584b to 0x006641 (0xdf7 bytes) 000076 1788464309.222269 trunc from 0x006642 to 0x00205f (0x45e2 bytes) 000077 1788464309.236441 trunc from 0x002060 to 0x023182 (0x21123 bytes) 000078 1788464309.288046 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x23183): Mirror 3: 0x7be1e8bc (stale) Mirror 4: 0x7be1e8bc (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23183): Mirror 3: 0x7be1e8bc (stale) Mirror 4: 0x7be1e8bc (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23183: skipped 000080 1788464309.647253 mapread 0x01b3ac thru 0x01cd1a (0x196f bytes) 000082 1788464309.658432 read 0x0038a0 thru 0x005ddb (0x253c bytes) 000083 1788464309.664032 write 0x02b378 thru 0x038e0f (0xda98 bytes) 000084 1788464309.665798 write 0x01c5a8 thru 0x022d2f (0x6788 bytes) 000085 1788464309.674130 mapread 0x01399b thru 0x01b0e1 (0x7747 bytes) 000086 1788464309.684997 punch from 0x005e20 to 0x00bc9c (0x5e7d bytes) 000087 1788464309.705097 read 0x02ed52 thru 0x02f52f (0x7de bytes) 000088 1788464309.705755 trunc from 0x038e10 to 0x030c7b (0x8194 bytes) 000093 1788464309.725337 read 0x006ebf thru 0x00951f (0x2661 bytes) 000095 1788464309.735713 write 0x000bc7 thru 0x00f7ee (0xec28 bytes) 000096 1788464309.737123 punch from 0x0129a6 to 0x014f13 (0x256e bytes) 000098 1788464309.754216 trunc from 0x030c7c to 0x027d92 (0x8ee9 bytes) 000099 1788464309.779785 trunc from 0x027d93 to 0x02abb3 (0x2e21 bytes) 000101 1788464309.824921 trunc from 0x02abb4 to 0x00094e (0x2a265 bytes) 000103 1788464309.850169 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x94f): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x94f): Mirror 3: 0x21c44cd4 (stale) Mirror 4: 0x21c44cd4 (stale) CRC-32 checksum value for chunk [0, 0x94f): Mirror 1: 0x21c44cd4 Mirror 5: 0x21c44cd4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x94f: skipped 000104 1788464310.822004 mapread 0x000601 thru 0x00094e (0x34e bytes) 000106 1788464310.922737 trunc from 0x00094f to 0x020541 (0x1fbf3 bytes) 000107 1788464311.147017 read 0x0085bf thru 0x008fd0 (0xa12 bytes) 000108 1788464311.158633 trunc from 0x020542 to 0x00c04e (0x144f3 bytes) 000113 1788464311.238310 mapread 0x00bc8c thru 0x00c04e (0x3c3 bytes) 000115 1788464311.293278 mapwrite 0x030caf thru 0x0314b8 (0x80a bytes) 000118 1788464311.515631 mapread 0x00ab2b thru 0x014320 (0x97f6 bytes) 000119 1788464311.577985 mapwrite 0x01927c thru 0x01e7c5 (0x554a bytes) 000120 1788464311.731431 mapwrite 0x03aff1 thru 0x03ffff (0x500f bytes) 000121 1788464311.895170 punch from 0x030005 to 0x03aaf6 (0xaaf2 bytes) 000122 1788464311.904178 mapwrite 0x00ba83 thru 0x011d07 (0x6285 bytes) 000123 1788464311.962700 trunc from 0x040000 to 0x012b51 (0x2d4ae bytes) 000124 1788464311.984135 trunc from 0x012b52 to 0x023ce5 (0x11194 bytes) 000126 1788464312.005105 mapread 0x01d8c8 thru 0x023ce5 (0x641e bytes) 000127 1788464312.016599 punch from 0x004f8d to 0x00c656 (0x76ca bytes) 000128 1788464312.023029 punch from 0x01da85 to 0x021766 (0x3ce2 bytes) 000130 1788464312.029722 trunc from 0x023ce6 to 0x01d573 (0x6772 bytes) 000131 1788464312.047204 read 0x011401 thru 0x011b15 (0x715 bytes) 000132 1788464312.047973 mapwrite 0x02d9d0 thru 0x0358b1 (0x7ee2 bytes) 000133 1788464312.193011 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x358b2): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x358b2): Mirror 3: 0x6aea8d3e (stale) Mirror 4: 0x6aea8d3e (stale) Mirror 5: 0x6aea8d3e (stale) CRC-32 checksum value for chunk [0, 0x358b2): Mirror 1: 0xd1cddda9 Mirror 6: 0xd1cddda9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x358b2: skipped 000134 1788464313.284887 mapread 0x0354e4 thru 0x0358b1 (0x3ce bytes) 000137 1788464313.365087 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x358b2): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x358b2): Mirror 3: 0x6aea8d3e (stale) Mirror 4: 0x6aea8d3e (stale) Mirror 5: 0x6aea8d3e (stale) CRC-32 checksum value for chunk [0, 0x358b2): Mirror 1: 0xd1cddda9 Mirror 6: 0xd1cddda9 Mirror 7: 0xd1cddda9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x358b2: skipped 000138 1788464314.474729 punch from 0x01b228 to 0x01c9c8 (0x17a1 bytes) 000139 1788464314.576008 mapwrite 0x000be6 thru 0x005f95 (0x53b0 bytes) 000140 1788464314.714192 read 0x01866d thru 0x024a86 (0xc41a bytes) 000142 1788464314.733293 mapwrite 0x024195 thru 0x02d7d1 (0x963d bytes) 000144 1788464314.880074 mapwrite 0x03d95d thru 0x03ffff (0x26a3 bytes) 000145 1788464315.112563 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 6: 0x4956a124 (stale) Mirror 7: 0x4956a124 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 6: 0x4956a124 (stale) Mirror 7: 0x4956a124 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000146 1788464315.866786 write 0x005bb5 thru 0x00b211 (0x565d bytes) 000147 1788464315.909430 punch from 0x032f1f to 0x03c63d (0x971f bytes) 000148 1788464315.946452 write 0x03401c thru 0x03cc65 (0x8c4a bytes) 000149 1788464315.972951 mapwrite 0x022c03 thru 0x0300bf (0xd4bd bytes) 000150 1788464316.115991 MIRROR_SPLIT mirror 3 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 7: 0x4956a124 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xe20eea22 (stale) Mirror 5: 0xe20eea22 (stale) Mirror 7: 0x4956a124 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000151 1788464316.793782 write 0x01d631 thru 0x01f7e1 (0x21b1 bytes) 000153 1788464316.817988 read 0x008fa6 thru 0x013ace (0xab29 bytes) 000154 1788464316.857438 read 0x032295 thru 0x03958b (0x72f7 bytes) 000156 1788464316.879041 write 0x03e6d2 thru 0x03ffff (0x192e bytes) 000157 1788464316.880581 read 0x035b54 thru 0x035b99 (0x46 bytes) 000158 1788464316.881288 punch from 0x03e500 to 0x03ffff (0x1b00 bytes) 000161 1788464316.896607 trunc from 0x040000 to 0x010fe9 (0x2f016 bytes) 000162 1788464316.914974 punch from 0x003428 to 0x0062da (0x2eb3 bytes) 000163 1788464316.920185 trunc from 0x010fea to 0x025084 (0x1409b bytes) 000164 1788464316.939253 read 0x00003a thru 0x007a3c (0x7a03 bytes) 000166 1788464316.953329 read 0x01e097 thru 0x025084 (0x6fee bytes) 000167 1788464316.954310 punch from 0x005e0b to 0x010d6d (0xaf63 bytes) 000168 1788464316.961846 trunc from 0x025085 to 0x0171ac (0xded8 bytes) 000169 1788464316.980572 MIRROR_RESYNC 4 mirrors 000170 1788464319.093208 read 0x000862 thru 0x000e83 (0x622 bytes) 000171 1788464319.103927 mapwrite 0x01c914 thru 0x023720 (0x6e0d bytes) 000173 1788464319.368602 read 0x0215af thru 0x023720 (0x2172 bytes) 000175 1788464319.369646 punch from 0x0157ca to 0x015b82 (0x3b9 bytes) 000178 1788464319.380966 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x23721): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x23721): Mirror 4: 0xcd75e8c7 (stale) Mirror 5: 0xcd75e8c7 (stale) Mirror 7: 0xcd75e8c7 (stale) CRC-32 checksum value for chunk [0, 0x23721): Mirror 1: 0x80c135b8 Mirror 8: 0x80c135b8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x23721: skipped 000179 1788464320.844575 write 0x02a707 thru 0x030da4 (0x669e bytes) 000180 1788464320.957067 write 0x029279 thru 0x034dc1 (0xbb49 bytes) 000181 1788464320.967165 punch from 0x01b263 to 0x0237d0 (0x856e bytes) 000183 1788464321.009315 mapwrite 0x0242d4 thru 0x02ad94 (0x6ac1 bytes) truncating to largest ever: 0x3ebfb 000184 1788464321.156236 trunc from 0x034dc2 to 0x03ebfa (0x9e39 bytes) 000185 1788464321.189798 read 0x03a463 thru 0x03ebfa (0x4798 bytes) 000187 1788464321.206661 mapread 0x03d2bc thru 0x03ebfa (0x193f bytes) 000188 1788464321.207550 trunc from 0x03ebfb to 0x00941b (0x357df bytes) 000189 1788464321.253846 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x941c): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 7: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x941c): Mirror 4: 0x66f9b11a (stale) Mirror 5: 0x66f9b11a (stale) Mirror 7: 0x66f9b11a (stale) Mirror 8: 0x66f9b11a (stale) CRC-32 checksum value for chunk [0, 0x941c): Mirror 1: 0x66f9b11a Mirror 9: 0x66f9b11a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x941c: skipped 000190 1788464322.838867 read 0x007726 thru 0x00941b (0x1cf6 bytes) 000191 1788464322.859359 MIRROR_RESYNC 6 mirrors 000192 1788464323.994699 mapread 0x00388c thru 0x00941b (0x5b90 bytes) 000193 1788464324.017610 mapwrite 0x012649 thru 0x019f6b (0x7923 bytes) 000194 1788464324.242192 MIRROR_SPLIT mirror 4 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x19f6c): Mirror 4: 0x7f897651 (stale) Mirror 5: 0x7f897651 (stale) Mirror 7: 0x7f897651 (stale) Mirror 9: 0x7f897651 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19f6c): Mirror 4: 0x7f897651 (stale) Mirror 5: 0x7f897651 (stale) Mirror 7: 0x7f897651 (stale) Mirror 9: 0x7f897651 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19f6c: skipped 000196 1788464324.618179 trunc from 0x019f6c to 0x02b9ab (0x11a40 bytes) 000197 1788464324.652294 punch from 0x010192 to 0x01afb1 (0xae20 bytes) 000198 1788464324.657582 mapwrite 0x01d013 thru 0x02a116 (0xd104 bytes) 000199 1788464324.733997 read 0x01cba8 thru 0x0290e1 (0xc53a bytes) 000200 1788464324.734678 mapwrite 0x033fdb thru 0x035ab3 (0x1ad9 bytes) 000202 1788464324.816169 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35ab4): Mirror 5: 0x9146220a (stale) Mirror 7: 0x9146220a (stale) Mirror 9: 0x9146220a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35ab4): Mirror 5: 0x9146220a (stale) Mirror 7: 0x9146220a (stale) Mirror 9: 0x9146220a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35ab4: skipped 000203 1788464325.217657 read 0x02e0d8 thru 0x035ab3 (0x79dc bytes) 000205 1788464325.233363 mapwrite 0x03a19f thru 0x03c071 (0x1ed3 bytes) 000206 1788464325.324364 mapwrite 0x033501 thru 0x03e8ba (0xb3ba bytes) 000207 1788464325.430463 mapwrite 0x0054cd thru 0x012ec4 (0xd9f8 bytes) 000208 1788464325.565736 punch from 0x02e10d to 0x038b22 (0xaa16 bytes) 000209 1788464325.574407 mapread 0x011bbc thru 0x013b61 (0x1fa6 bytes) 000210 1788464325.578656 mapwrite 0x009114 thru 0x01431b (0xb208 bytes) 000211 1788464325.665179 mapwrite 0x00e794 thru 0x011e99 (0x3706 bytes) 000212 1788464325.713232 mapwrite 0x00bb41 thru 0x017e3d (0xc2fd bytes) 000213 1788464325.773876 write 0x038bc8 thru 0x039c7d (0x10b6 bytes) 000214 1788464325.782170 punch from 0x025806 to 0x02f81a (0xa015 bytes) 000215 1788464325.793095 trunc from 0x03e8bb to 0x00ba54 (0x32e66 bytes) 000219 1788464325.812541 punch from 0x003f14 to 0x006130 (0x221d bytes) 000220 1788464325.820913 read 0x00a3e4 thru 0x00ba54 (0x1671 bytes) 000222 1788464325.822222 read 0x002ab2 thru 0x00ba54 (0x8fa3 bytes) 000224 1788464325.833024 write 0x01cc8d thru 0x02c410 (0xf784 bytes) 000225 1788464325.834644 read 0x024c68 thru 0x02c410 (0x77a9 bytes) 000226 1788464325.836390 trunc from 0x02c411 to 0x038a10 (0xc600 bytes) 000227 1788464325.863235 mapwrite 0x000cc4 thru 0x00477c (0x3ab9 bytes) 000229 1788464325.922870 mapwrite 0x033911 thru 0x03c05a (0x874a bytes) 000231 1788464326.034734 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c05b): Mirror 5: 0x88c91781 (stale) Mirror 9: 0x88c91781 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c05b): Mirror 5: 0x88c91781 (stale) Mirror 9: 0x88c91781 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c05b: skipped 000233 1788464326.543655 mapwrite 0x0143ef thru 0x01eb28 (0xa73a bytes) 000235 1788464326.704072 trunc from 0x03c05b to 0x03dbf7 (0x1b9d bytes) 000238 1788464326.740154 MIRROR_RESYNC 3 mirrors 000239 1788464327.863228 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) 000240 1788464328.381106 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) 000242 1788464328.569140 punch from 0x003693 to 0x0094ae (0x5e1c bytes) 000243 1788464328.596047 punch from 0x03057a to 0x036f50 (0x69d7 bytes) 000244 1788464328.612901 mapwrite 0x02cc22 thru 0x03b3d5 (0xe7b4 bytes) 000245 1788464328.772791 mapwrite 0x02f756 thru 0x03ca12 (0xd2bd bytes) 000246 1788464328.904188 punch from 0x033e72 to 0x03ce99 (0x9028 bytes) 000247 1788464328.909427 read 0x02190d thru 0x02848a (0x6b7e bytes) 000248 1788464328.936184 punch from 0x03a3aa to 0x03dbf7 (0x384e bytes) 000249 1788464328.943740 read 0x00af95 thru 0x013238 (0x82a4 bytes) 000250 1788464328.956448 trunc from 0x03dbf8 to 0x004cd6 (0x38f21 bytes) 000251 1788464328.975598 mapread 0x000e1e thru 0x004cd6 (0x3eb9 bytes) 000252 1788464328.976627 mapwrite 0x0168c3 thru 0x01e2aa (0x79e8 bytes) 000254 1788464329.153813 mapwrite 0x03a29e thru 0x03c528 (0x228b bytes) 000257 1788464329.345167 punch from 0x00cb16 to 0x018be9 (0xc0d4 bytes) 000258 1788464329.367109 mapwrite 0x01d741 thru 0x0259ce (0x828e bytes) 000259 1788464329.497866 write 0x0178c5 thru 0x01b46c (0x3ba8 bytes) 000260 1788464329.507798 read 0x02dbfc thru 0x031f22 (0x4327 bytes) 000262 1788464329.525061 punch from 0x034853 to 0x03c528 (0x7cd6 bytes) 000263 1788464329.547485 mapread 0x01eec9 thru 0x0294bb (0xa5f3 bytes) 000267 1788464329.551178 write 0x0338ec thru 0x03f4e2 (0xbbf7 bytes) 000271 1788464329.558942 punch from 0x008fb1 to 0x010c87 (0x7cd7 bytes) 000273 1788464329.583172 mapwrite 0x017976 thru 0x01e0ba (0x6745 bytes) 000274 1788464329.656150 punch from 0x00e6e6 to 0x0140da (0x59f5 bytes) 000276 1788464329.662246 trunc from 0x03f4e3 to 0x0090c6 (0x3641c bytes) 000280 1788464329.685639 MIRROR_EXTEND to 2 mirrors 000281 1788464330.481658 MIRROR_RESYNC 2 mirrors 000283 1788464330.782079 mapwrite 0x00074f thru 0x0033cb (0x2c7d bytes) 000284 1788464331.014837 trunc from 0x0090c7 to 0x014f3a (0xbe74 bytes) 000285 1788464331.038971 trunc from 0x014f3b to 0x034a4f (0x1fb15 bytes) 000286 1788464331.074733 read 0x0245ae thru 0x02ae67 (0x68ba bytes) 000288 1788464331.087084 trunc from 0x034a50 to 0x01fd30 (0x14d1f bytes) 000289 1788464331.103841 read 0x0101b0 thru 0x018684 (0x84d5 bytes) 000290 1788464331.114814 punch from 0x003bde to 0x00ab20 (0x6f43 bytes) 000291 1788464331.120722 trunc from 0x01fd31 to 0x02e2a5 (0xe575 bytes) 000292 1788464331.148685 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) 000293 1788464331.316593 punch from 0x012422 to 0x01ba00 (0x95df bytes) 000294 1788464331.321469 write 0x0295ac thru 0x02a013 (0xa68 bytes) 000296 1788464331.333091 punch from 0x01dc65 to 0x0208b9 (0x2c55 bytes) 000297 1788464331.345671 MIRROR_EXTEND to 2 mirrors 000298 1788464332.260901 mapread 0x0074e6 thru 0x014a33 (0xd54e bytes) 000299 1788464332.306434 punch from 0x0138ad to 0x013b45 (0x299 bytes) 000300 1788464332.450285 mapwrite 0x01bafd thru 0x02765c (0xbb60 bytes) 000301 1788464332.519836 write 0x02078a thru 0x02773f (0x6fb6 bytes) 000303 1788464332.521446 mapwrite 0x0333a9 thru 0x0351c4 (0x1e1c bytes) 000308 1788464332.659974 MIRROR_RESYNC 2 mirrors 000309 1788464333.680415 mapwrite 0x008106 thru 0x008765 (0x660 bytes) 000311 1788464333.861258 mapread 0x011f95 thru 0x01398d (0x19f9 bytes) 000315 1788464333.882187 trunc from 0x0351c5 to 0x03cd0c (0x7b48 bytes) 000316 1788464333.924476 mapwrite 0x024874 thru 0x02ebea (0xa377 bytes) 000317 1788464334.045946 write 0x03c8b7 thru 0x03f4e1 (0x2c2b bytes) 000318 1788464334.059792 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3f4e2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3f4e2): Mirror 2: 0xb803450d (stale) CRC-32 checksum value for chunk [0, 0x3f4e2): Mirror 1: 0xfbb2dc12 Mirror 3: 0xfbb2dc12 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3f4e2: skipped 000319 1788464335.419609 mapwrite 0x02c60e thru 0x038c17 (0xc60a bytes) 000320 1788464335.681548 trunc from 0x03f4e2 to 0x02c4a6 (0x1303b bytes) 000324 1788464335.710006 read 0x01730f thru 0x020a82 (0x9774 bytes) 000325 1788464335.733435 mapread 0x00a8d1 thru 0x017818 (0xcf48 bytes) 000327 1788464335.771295 trunc from 0x02c4a7 to 0x02e3d1 (0x1f2b bytes) 000328 1788464335.813261 write 0x002909 thru 0x00717e (0x4876 bytes) 000329 1788464335.821051 mapread 0x024e16 thru 0x027857 (0x2a42 bytes) 000331 1788464335.832545 write 0x02ca77 thru 0x03b3dc (0xe966 bytes) 000332 1788464335.835047 read 0x027327 thru 0x02f6bf (0x8399 bytes) 000334 1788464335.839816 trunc from 0x03b3dd to 0x039cd2 (0x170a bytes) 000335 1788464335.865848 read 0x0381a5 thru 0x039cd2 (0x1b2e bytes) 000338 1788464335.870922 trunc from 0x039cd3 to 0x03ce06 (0x3134 bytes) 000339 1788464335.897802 mapread 0x02506a thru 0x029a59 (0x49f0 bytes) 000340 1788464335.898851 write 0x01569b thru 0x02036d (0xacd3 bytes) 000342 1788464335.904035 punch from 0x01b412 to 0x01e9b7 (0x35a6 bytes) 000343 1788464335.925089 mapread 0x038929 thru 0x038ecd (0x5a5 bytes) 000345 1788464335.927980 mapwrite 0x0020a5 thru 0x004c69 (0x2bc5 bytes) 000347 1788464336.009644 write 0x028d40 thru 0x0387cf (0xfa90 bytes) 000350 1788464336.014001 mapwrite 0x03a1c6 thru 0x03ffff (0x5e3a bytes) 000352 1788464336.114660 read 0x03e0e8 thru 0x03ffff (0x1f18 bytes) 000354 1788464336.115728 write 0x01bcd2 thru 0x026254 (0xa583 bytes) 000356 1788464336.119973 mapwrite 0x0210a5 thru 0x02d9cf (0xc92b bytes) 000357 1788464336.214898 punch from 0x03934f to 0x03ffff (0x6cb1 bytes) 000358 1788464336.221760 write 0x014b46 thru 0x023171 (0xe62c bytes) 000359 1788464336.224834 punch from 0x017b19 to 0x0198f3 (0x1ddb bytes) 000360 1788464336.242172 trunc from 0x040000 to 0x001d35 (0x3e2ca bytes) 000361 1788464336.267630 trunc from 0x001d36 to 0x01dbd8 (0x1bea3 bytes) 000365 1788464336.315407 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1dbd9): Mirror 2: 0x2100ac7 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1dbd9): Mirror 2: 0x2100ac7 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1dbd9: skipped 000369 1788464336.706781 mapwrite 0x0372cf thru 0x03ca19 (0x574b bytes) 000370 1788464336.820410 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) 000371 1788464336.935812 mapread 0x02e29a thru 0x0397b4 (0xb51b bytes) 000372 1788464336.970422 trunc from 0x03ca1a to 0x03ad78 (0x1ca1 bytes) 000373 1788464336.986928 mapwrite 0x00782d thru 0x008508 (0xcdc bytes) 000374 1788464337.046501 mapwrite 0x029b2f thru 0x0391e2 (0xf6b4 bytes) 000376 1788464337.142965 mapread 0x02bb90 thru 0x03441f (0x8890 bytes) 000378 1788464337.154988 write 0x02a7a6 thru 0x02beb3 (0x170e bytes) 000379 1788464337.165012 mapread 0x0396d9 thru 0x03ad78 (0x16a0 bytes) 000381 1788464337.170913 read 0x02e091 thru 0x03ad78 (0xcce8 bytes) 000384 1788464337.177801 punch from 0x031d9c to 0x03a9fd (0x8c62 bytes) 000386 1788464337.197688 write 0x03abe9 thru 0x03ffff (0x5417 bytes) 000387 1788464337.205743 punch from 0x028af6 to 0x029c60 (0x116b bytes) 000388 1788464337.226947 read 0x03854d thru 0x03ffff (0x7ab3 bytes) 000389 1788464337.246937 write 0x0379d6 thru 0x03ffff (0x862a bytes) 000390 1788464337.251158 read 0x004165 thru 0x00eb8f (0xaa2b bytes) 000392 1788464337.253497 trunc from 0x040000 to 0x02bad8 (0x14527 bytes) 000394 1788464337.281805 trunc from 0x02bad9 to 0x0374e9 (0xba11 bytes) 000395 1788464337.298044 mapread 0x0246b5 thru 0x02c0c4 (0x7a10 bytes) 000396 1788464337.312638 trunc from 0x0374ea to 0x013f14 (0x235d5 bytes) 000398 1788464337.328605 punch from 0x00b5ed to 0x013f14 (0x8928 bytes) 000400 1788464337.335645 mapwrite 0x014b71 thru 0x021cc2 (0xd152 bytes) 000401 1788464337.474013 write 0x015b02 thru 0x01f39b (0x989a bytes) 000405 1788464337.479274 read 0x00ae07 thru 0x019861 (0xea5b bytes) 000406 1788464337.490871 punch from 0x0151fa to 0x02120e (0xc015 bytes) 000407 1788464337.507459 mapread 0x00121d thru 0x0098d9 (0x86bd bytes) 000409 1788464337.509790 write 0x00a01c thru 0x00b3a1 (0x1386 bytes) 000410 1788464337.510756 trunc from 0x021cc3 to 0x030ccc (0xf00a bytes) 000412 1788464337.536288 trunc from 0x030ccd to 0x01adaa (0x15f22 bytes) 000413 1788464337.554856 read 0x002f4b thru 0x010368 (0xd41e bytes) 000414 1788464337.556132 trunc from 0x01adab to 0x013c56 (0x7154 bytes) 000416 1788464337.572558 mapwrite 0x032b81 thru 0x03ffff (0xd47f bytes) 000417 1788464337.674831 trunc from 0x040000 to 0x00e47e (0x31b81 bytes) 000419 1788464337.695052 punch from 0x002510 to 0x005404 (0x2ef5 bytes) 000420 1788464337.700941 mapwrite 0x00bd0b thru 0x011666 (0x595c bytes) 000421 1788464337.825813 write 0x0030b8 thru 0x00f451 (0xc39a bytes) 000422 1788464337.831911 mapwrite 0x02bd27 thru 0x02ee04 (0x30de bytes) 000423 1788464337.918325 punch from 0x013836 to 0x01686e (0x3039 bytes) 000424 1788464337.924690 mapread 0x006f46 thru 0x00f1cf (0x828a bytes) 000425 1788464337.925949 trunc from 0x02ee05 to 0x0055ab (0x29859 bytes) 000430 1788464337.950852 trunc from 0x0055ac to 0x035133 (0x2fb88 bytes) 000431 1788464337.969359 punch from 0x02b72e to 0x02f77b (0x404e bytes) 000433 1788464337.975273 mapread 0x033a41 thru 0x035133 (0x16f3 bytes) 000434 1788464337.991123 punch from 0x032e54 to 0x035133 (0x22e0 bytes) 000435 1788464338.001136 MIRROR_EXTEND to 2 mirrors 000436 1788464339.013343 write 0x034bb6 thru 0x03ffff (0xb44a bytes) 000437 1788464339.088659 MIRROR_RESYNC 2 mirrors 000438 1788464340.283745 mapwrite 0x033d5f thru 0x0391c1 (0x5463 bytes) 000439 1788464340.605622 mapwrite 0x00ad30 thru 0x00fe50 (0x5121 bytes) 000440 1788464340.763466 mapwrite 0x01d695 thru 0x02735a (0x9cc6 bytes) 000441 1788464340.905500 mapread 0x031e75 thru 0x03b0a3 (0x922f bytes) 000442 1788464340.908867 write 0x0205f3 thru 0x024202 (0x3c10 bytes) 000443 1788464340.911925 mapwrite 0x0168ed thru 0x025eab (0xf5bf bytes) 000445 1788464341.007218 punch from 0x03c4a8 to 0x03ffff (0x3b58 bytes) 000446 1788464341.012014 punch from 0x01e021 to 0x0210c3 (0x30a3 bytes) 000448 1788464341.018895 trunc from 0x040000 to 0x005667 (0x3a998 bytes) 000450 1788464341.054256 write 0x00259a thru 0x0031d8 (0xc3f bytes) 000451 1788464341.063185 punch from 0x002e00 to 0x005667 (0x2868 bytes) 000452 1788464341.074981 mapread 0x001eac thru 0x005667 (0x37bc bytes) 000453 1788464341.085718 write 0x03b266 thru 0x03ffff (0x4d9a bytes) 000454 1788464341.086866 mapread 0x02700c thru 0x02eccb (0x7cc0 bytes) 000455 1788464341.097407 mapread 0x03ec80 thru 0x03ffff (0x1380 bytes) 000457 1788464341.099023 mapwrite 0x01a24e thru 0x0289ef (0xe7a2 bytes) 000458 1788464341.192231 trunc from 0x040000 to 0x033bc3 (0xc43c bytes) 000460 1788464341.232234 MIRROR_RESYNC 2 mirrors 000461 1788464342.100928 MIRROR_RESYNC 2 mirrors 000463 1788464342.515128 write 0x0360d9 thru 0x03ffff (0x9f27 bytes) 000465 1788464342.637466 mapread 0x012139 thru 0x01277c (0x644 bytes) 000467 1788464342.659378 punch from 0x003f65 to 0x009e11 (0x5ead bytes) 000468 1788464342.693370 punch from 0x016475 to 0x01ea7c (0x8608 bytes) 000470 1788464342.697786 punch from 0x0264cb to 0x032f52 (0xca88 bytes) 000472 1788464342.708558 mapwrite 0x03bcdc thru 0x03ffff (0x4324 bytes) 000473 1788464342.845785 MIRROR_RESYNC 2 mirrors 000475 1788464344.382719 MIRROR_EXTEND to 3 mirrors 000477 1788464345.658399 read 0x036123 thru 0x03ffff (0x9edd bytes) 000478 1788464345.686764 mapwrite 0x038361 thru 0x038742 (0x3e2 bytes) 000479 1788464345.821349 punch from 0x002ee7 to 0x0086a1 (0x57bb bytes) 000480 1788464345.826494 read 0x029993 thru 0x0353ff (0xba6d bytes) 000483 1788464345.836980 read 0x017ae7 thru 0x02548b (0xd9a5 bytes) 000485 1788464345.842723 MIRROR_RESYNC 3 mirrors 000486 1788464347.227208 mapread 0x023a53 thru 0x02610f (0x26bd bytes) 000489 1788464347.246020 trunc from 0x040000 to 0x000a92 (0x3f56d bytes) 000490 1788464347.348501 punch from 0x0005cb to 0x000a92 (0x4c8 bytes) 000491 1788464347.356424 trunc from 0x000a93 to 0x002e86 (0x23f4 bytes) 000495 1788464347.384115 trunc from 0x002e87 to 0x034a55 (0x31bcf bytes) 000496 1788464347.402567 punch from 0x013dc4 to 0x015ac5 (0x1d02 bytes) 000497 1788464347.413261 MIRROR_RESYNC 3 mirrors 000498 1788464348.381106 trunc from 0x034a56 to 0x0273bb (0xd69a bytes) 000499 1788464348.469972 punch from 0x00fdee to 0x00fefe (0x111 bytes) 000500 1788464348.475459 trunc from 0x0273bc to 0x022e21 (0x459a bytes) 000502 1788464348.503859 MIRROR_RESYNC 3 mirrors 000504 1788464349.517102 punch from 0x01c2c8 to 0x01ed6b (0x2aa4 bytes) 000505 1788464349.569864 trunc from 0x022e22 to 0x037827 (0x14a06 bytes) 000506 1788464349.628656 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x37828): Mirror 3: 0xc655f12 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x37828): Mirror 3: 0xc655f12 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x37828: skipped 000510 1788464350.053561 read 0x02d648 thru 0x02fe77 (0x2830 bytes) 000512 1788464350.074465 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) 000513 1788464350.273365 read 0x014c4d thru 0x02306c (0xe420 bytes) 000514 1788464350.287670 write 0x015d7f thru 0x01ab56 (0x4dd8 bytes) 000516 1788464350.290577 write 0x013683 thru 0x0209c0 (0xd33e bytes) 000518 1788464350.297111 trunc from 0x037828 to 0x017b66 (0x1fcc1 bytes) 000521 1788464350.317687 write 0x01b37a thru 0x020025 (0x4cac bytes) 000522 1788464350.319764 write 0x0043f9 thru 0x01268b (0xe293 bytes) 000523 1788464350.329895 trunc from 0x020026 to 0x016412 (0x9c13 bytes) 000526 1788464350.348630 trunc from 0x016413 to 0x00d635 (0x8ddd bytes) 000527 1788464350.370407 trunc from 0x00d636 to 0x02b921 (0x1e2ec bytes) 000528 1788464350.381383 mapread 0x023875 thru 0x02b921 (0x80ad bytes) 000529 1788464350.400887 mapwrite 0x03bfc5 thru 0x03ffff (0x403b bytes) 000530 1788464350.471787 read 0x03046d thru 0x038c2f (0x87c3 bytes) 000531 1788464350.481599 mapwrite 0x012c78 thru 0x01ab73 (0x7efc bytes) 000533 1788464350.537868 read 0x02cea7 thru 0x03a3e9 (0xd543 bytes) 000534 1788464350.539021 trunc from 0x040000 to 0x0339ef (0xc610 bytes) 000535 1788464350.562652 punch from 0x00a76f to 0x00f123 (0x49b5 bytes) 000536 1788464350.567839 trunc from 0x0339f0 to 0x01d95b (0x16094 bytes) 000539 1788464350.595395 trunc from 0x01d95c to 0x009c2f (0x13d2c bytes) 000542 1788464350.618410 mapread 0x006d28 thru 0x007fcb (0x12a4 bytes) 000543 1788464350.621978 read 0x00201e thru 0x0060a6 (0x4089 bytes) 000544 1788464350.623064 mapread 0x005188 thru 0x0062c0 (0x1139 bytes) 000549 1788464350.630702 write 0x0030e3 thru 0x00fd6c (0xcc8a bytes) 000550 1788464350.633661 mapwrite 0x023814 thru 0x02802c (0x4819 bytes) 000552 1788464350.723202 mapread 0x0021df thru 0x011577 (0xf399 bytes) 000553 1788464350.724395 read 0x0070a6 thru 0x00e686 (0x75e1 bytes) 000554 1788464350.725122 read 0x00c42a thru 0x013a87 (0x765e bytes) 000556 1788464350.726286 punch from 0x01bca4 to 0x0253f5 (0x9752 bytes) 000557 1788464350.732718 punch from 0x019f44 to 0x02802c (0xe0e9 bytes) 000558 1788464350.744436 write 0x032453 thru 0x03ca51 (0xa5ff bytes) 000559 1788464350.757218 write 0x01c57c thru 0x020ddd (0x4862 bytes) 000562 1788464350.771690 mapwrite 0x03a7bd thru 0x03ffff (0x5843 bytes) 000569 1788464350.850298 punch from 0x020117 to 0x022746 (0x2630 bytes) 000571 1788464350.867249 mapwrite 0x01cf4d thru 0x0266e2 (0x9796 bytes) 000575 1788464350.955950 read 0x011d46 thru 0x01fa45 (0xdd00 bytes) truncating to largest ever: 0x3efda 000576 1788464350.957690 trunc from 0x040000 to 0x03efd9 (0x1026 bytes) 000577 1788464350.978254 punch from 0x02792d to 0x036cb1 (0xf385 bytes) 000578 1788464351.013281 MIRROR_EXTEND to 2 mirrors 000579 1788464352.102823 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) 000580 1788464352.255461 write 0x00702e thru 0x00b9cd (0x49a0 bytes) 000581 1788464352.282352 mapread 0x0104a3 thru 0x01a2e8 (0x9e46 bytes) 000582 1788464352.302109 punch from 0x01e2e9 to 0x0245ae (0x62c6 bytes) 000584 1788464352.314713 write 0x008f13 thru 0x013421 (0xa50f bytes) 000586 1788464352.319436 mapread 0x017245 thru 0x023324 (0xc0e0 bytes) 000587 1788464352.344618 trunc from 0x03efda to 0x02bf30 (0x130a9 bytes) 000590 1788464352.374947 mapread 0x0253d9 thru 0x02bf30 (0x6b58 bytes) 000591 1788464352.393776 punch from 0x01c489 to 0x01dc86 (0x17fe bytes) 000593 1788464352.410984 mapread 0x020e75 thru 0x02bf30 (0xb0bc bytes) 000594 1788464352.414304 trunc from 0x02bf31 to 0x01dffb (0xdf35 bytes) 000595 1788464352.438230 mapread 0x014b28 thru 0x01dffb (0x94d4 bytes) 000596 1788464352.444103 read 0x009748 thru 0x0169d7 (0xd290 bytes) 000597 1788464352.445013 punch from 0x014cd3 to 0x01dffb (0x9329 bytes) 000599 1788464352.453638 mapwrite 0x020b1f thru 0x024a7c (0x3f5e bytes) 000602 1788464352.524492 trunc from 0x024a7d to 0x029f38 (0x54bc bytes) 000603 1788464352.552386 read 0x013c46 thru 0x020886 (0xcc41 bytes) 000604 1788464352.565097 read 0x01c083 thru 0x01dcf0 (0x1c6e bytes) 000607 1788464352.568007 write 0x033ea6 thru 0x03ffff (0xc15a bytes) 000609 1788464352.572211 punch from 0x009d77 to 0x013d42 (0x9fcc bytes) 000611 1788464352.590196 mapwrite 0x01ee10 thru 0x020457 (0x1648 bytes) 000615 1788464352.647535 write 0x0171a9 thru 0x01af33 (0x3d8b bytes) 000616 1788464352.649553 mapread 0x000c56 thru 0x002a6b (0x1e16 bytes) 000618 1788464352.652037 punch from 0x011e8c to 0x020748 (0xe8bd bytes) 000622 1788464352.663900 punch from 0x02614f to 0x0342ae (0xe160 bytes) 000625 1788464352.670744 mapread 0x01b10a thru 0x021083 (0x5f7a bytes) 000628 1788464352.683192 mapwrite 0x02b783 thru 0x031f68 (0x67e6 bytes) 000629 1788464352.734067 mapread 0x011626 thru 0x01a5a2 (0x8f7d bytes) 000630 1788464352.736014 punch from 0x00e46f to 0x01aec2 (0xca54 bytes) 000632 1788464352.744245 write 0x028180 thru 0x0372fe (0xf17f bytes) 000634 1788464352.747616 mapread 0x014b3f thru 0x022428 (0xd8ea bytes) 000636 1788464352.790634 read 0x03b782 thru 0x03ffff (0x487e bytes) 000637 1788464352.791235 read 0x03cb54 thru 0x03ffff (0x34ac bytes) 000638 1788464352.794589 read 0x016c5a thru 0x01e329 (0x76d0 bytes) 000639 1788464352.796580 mapwrite 0x02243c thru 0x02782c (0x53f1 bytes) 000640 1788464352.888851 mapwrite 0x00ce4e thru 0x016d88 (0x9f3b bytes) 000643 1788464353.012084 write 0x00a30d thru 0x00f036 (0x4d2a bytes) 000644 1788464353.014225 mapread 0x033a64 thru 0x03571a (0x1cb7 bytes) 000645 1788464353.015084 mapwrite 0x003d84 thru 0x008505 (0x4782 bytes) 000647 1788464353.108804 punch from 0x03fa8a to 0x03ffff (0x576 bytes) 000654 1788464353.122413 read 0x0020cf thru 0x01012a (0xe05c bytes) 000658 1788464353.128037 write 0x0138d8 thru 0x01c6bc (0x8de5 bytes) 000659 1788464353.131971 mapwrite 0x03ef4c thru 0x03ffff (0x10b4 bytes) 000661 1788464353.219060 read 0x032e4f thru 0x036e1c (0x3fce bytes) 000662 1788464353.219741 mapwrite 0x0267e5 thru 0x033744 (0xcf60 bytes) 000663 1788464353.296888 write 0x002b4a thru 0x00a3a5 (0x785c bytes) 000665 1788464353.302815 mapwrite 0x010358 thru 0x01fb57 (0xf800 bytes) 000666 1788464353.406649 mapwrite 0x0010f4 thru 0x00972c (0x8639 bytes) 000667 1788464353.476989 write 0x0203e5 thru 0x02b44b (0xb067 bytes) 000668 1788464353.479495 punch from 0x0057aa to 0x0131fe (0xda55 bytes) 000669 1788464353.498496 mapwrite 0x03eef5 thru 0x03ffff (0x110b bytes) 000670 1788464353.568282 punch from 0x021ee6 to 0x028188 (0x62a3 bytes) 000671 1788464353.581934 trunc from 0x040000 to 0x03e8c8 (0x1737 bytes) 000672 1788464353.617890 read 0x00698f thru 0x0147c0 (0xde32 bytes) 000673 1788464353.636103 read 0x023792 thru 0x028620 (0x4e8f bytes) 000675 1788464353.638429 mapread 0x014f06 thru 0x01e47d (0x9578 bytes) 000678 1788464353.649501 mapwrite 0x034ae4 thru 0x037f04 (0x3421 bytes) 000679 1788464353.742675 punch from 0x036212 to 0x03e8c8 (0x86b7 bytes) 000680 1788464353.749077 write 0x032535 thru 0x03ffff (0xdacb bytes) 000682 1788464353.752176 mapwrite 0x0031cd thru 0x005180 (0x1fb4 bytes) 000684 1788464353.788896 mapread 0x03c741 thru 0x03ffff (0x38bf bytes) 000685 1788464353.792166 mapread 0x03e88c thru 0x03ffff (0x1774 bytes) 000688 1788464353.801110 write 0x01a227 thru 0x01dd7b (0x3b55 bytes) 000689 1788464353.805853 mapwrite 0x00f223 thru 0x0130a5 (0x3e83 bytes) 000690 1788464353.911711 read 0x01807a thru 0x025508 (0xd48f bytes) 000691 1788464353.915950 mapread 0x0190ca thru 0x01bc87 (0x2bbe bytes) 000692 1788464353.920459 mapread 0x039655 thru 0x03afd7 (0x1983 bytes) 000693 1788464353.923293 write 0x01e0d7 thru 0x028ce9 (0xac13 bytes) 000695 1788464353.931846 write 0x027afe thru 0x03251b (0xaa1e bytes) 000696 1788464353.939280 mapread 0x02415f thru 0x030787 (0xc629 bytes) 000697 1788464353.943334 mapwrite 0x01d404 thru 0x029cbb (0xc8b8 bytes) 000698 1788464354.028153 mapread 0x00b7ac thru 0x00e20e (0x2a63 bytes) 000699 1788464354.029190 mapread 0x0157b3 thru 0x01befd (0x674b bytes) 000700 1788464354.030374 mapread 0x02440b thru 0x029a8e (0x5684 bytes) 000701 1788464354.031419 mapwrite 0x02ad9a thru 0x032af5 (0x7d5c bytes) 000702 1788464354.111791 MIRROR_EXTEND to 2 mirrors 000703 1788464354.784074 MIRROR_EXTEND to 3 mirrors 000704 1788464355.562333 mapwrite 0x000b75 thru 0x008fb7 (0x8443 bytes) 000705 1788464355.746334 write 0x0180eb thru 0x01a66d (0x2583 bytes) 000706 1788464355.785079 write 0x00514f thru 0x00b068 (0x5f1a bytes) 000707 1788464355.786450 mapread 0x03a665 thru 0x03ffff (0x599b bytes) 000708 1788464355.855359 trunc from 0x040000 to 0x00446f (0x3bb90 bytes) 000709 1788464355.908719 trunc from 0x004470 to 0x013365 (0xeef6 bytes) 000710 1788464355.945594 punch from 0x003915 to 0x00f8cb (0xbfb7 bytes) 000711 1788464355.951542 trunc from 0x013366 to 0x00e77e (0x4be7 bytes) 000712 1788464355.978126 read 0x002020 thru 0x00218b (0x16c bytes) 000714 1788464355.988887 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xe77f): Mirror 2: 0xd75ab75a (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xe77f): Mirror 2: 0xd75ab75a (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xe77f: skipped 000715 1788464356.512829 write 0x01cd4d thru 0x026632 (0x98e6 bytes) 000717 1788464356.528655 MIRROR_RESYNC 2 mirrors 000718 1788464357.855737 punch from 0x010455 to 0x015507 (0x50b3 bytes) 000719 1788464357.908344 mapwrite 0x019596 thru 0x01cfa9 (0x3a14 bytes) 000720 1788464357.964277 trunc from 0x026633 to 0x029bb7 (0x3585 bytes) 000723 1788464357.984450 read 0x027e42 thru 0x029bb7 (0x1d76 bytes) 000724 1788464357.989332 write 0x0325e7 thru 0x033c2e (0x1648 bytes) 000727 1788464357.992538 mapwrite 0x023534 thru 0x0289e2 (0x54af bytes) 000731 1788464358.042002 trunc from 0x033c2f to 0x000460 (0x337ce bytes) 000735 1788464358.057714 write 0x037d82 thru 0x03e959 (0x6bd8 bytes) 000736 1788464358.059455 mapread 0x02e893 thru 0x034518 (0x5c86 bytes) 000737 1788464358.066162 mapwrite 0x02134e thru 0x0239a8 (0x265b bytes) 000738 1788464358.114119 mapwrite 0x02fa01 thru 0x03e435 (0xea35 bytes) 000739 1788464358.158723 trunc from 0x03e95a to 0x01d3bb (0x2159e bytes) 000740 1788464358.179162 trunc from 0x01d3bc to 0x02a976 (0xd5bb bytes) 000741 1788464358.196198 mapwrite 0x03c428 thru 0x03ffff (0x3bd8 bytes) 000742 1788464358.302136 punch from 0x006e71 to 0x00cf17 (0x60a7 bytes) 000743 1788464358.317267 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) 000745 1788464358.498205 write 0x008fcb thru 0x015b15 (0xcb4b bytes) 000748 1788464358.508391 punch from 0x01006f to 0x016a4a (0x69dc bytes) 000749 1788464358.520098 punch from 0x0029e5 to 0x00fcd6 (0xd2f2 bytes) 000751 1788464358.531018 write 0x0336ec thru 0x03e555 (0xae6a bytes) 000752 1788464358.539603 trunc from 0x040000 to 0x00a3eb (0x35c14 bytes) 000753 1788464358.555392 mapread 0x00079f thru 0x007118 (0x697a bytes) 000754 1788464358.573752 mapread 0x006d6e thru 0x00a3eb (0x367e bytes) 000756 1788464358.577768 punch from 0x003b52 to 0x00a3eb (0x689a bytes) 000757 1788464358.582380 punch from 0x0085f3 to 0x00a3eb (0x1df9 bytes) 000758 1788464358.585892 mapread 0x003926 thru 0x00a3eb (0x6ac6 bytes) 000761 1788464358.595649 write 0x03f9b6 thru 0x03ffff (0x64a bytes) 000762 1788464358.597455 mapwrite 0x004224 thru 0x00a954 (0x6731 bytes) 000763 1788464358.632991 mapwrite 0x0052c2 thru 0x00e302 (0x9041 bytes) 000765 1788464358.678315 punch from 0x01065d to 0x01d8b4 (0xd258 bytes) 000766 1788464358.682714 punch from 0x015e0b to 0x019fe2 (0x41d8 bytes) 000767 1788464358.686673 write 0x02d48b thru 0x03c487 (0xeffd bytes) 000768 1788464358.698762 MIRROR_EXTEND to 2 mirrors 000769 1788464360.181983 trunc from 0x040000 to 0x035870 (0xa78f bytes) 000770 1788464360.374727 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35871): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35871): Mirror 2: 0x9afc311b (stale) CRC-32 checksum value for chunk [0, 0x35871): Mirror 1: 0x9afc311b Mirror 3: 0x9afc311b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35871: skipped 000771 1788464361.892164 write 0x021457 thru 0x026918 (0x54c2 bytes) 000773 1788464361.979879 mapwrite 0x012ad7 thru 0x016d47 (0x4271 bytes) 000774 1788464362.081131 read 0x001c84 thru 0x00ca45 (0xadc2 bytes) 000777 1788464362.095797 mapwrite 0x005965 thru 0x0126e2 (0xcd7e bytes) 000778 1788464362.199677 read 0x029743 thru 0x0324aa (0x8d68 bytes) 000779 1788464362.243815 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x35871): Mirror 2: 0x9afc311b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35871): Mirror 2: 0x9afc311b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35871: skipped 000780 1788464362.601101 write 0x0078a0 thru 0x008a45 (0x11a6 bytes) 000781 1788464362.619987 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x35871): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x35871): Mirror 2: 0x9afc311b (stale) CRC-32 checksum value for chunk [0, 0x35871): Mirror 1: 0xaee31ab5 Mirror 3: 0xaee31ab5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x35871: skipped 000782 1788464363.957967 MIRROR_RESYNC 3 mirrors 000783 1788464365.266599 write 0x01121f thru 0x017e4e (0x6c30 bytes) 000785 1788464365.464103 trunc from 0x035871 to 0x03eb7a (0x930a bytes) 000786 1788464365.500173 trunc from 0x03eb7b to 0x03dd64 (0xe16 bytes) 000787 1788464365.560848 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3dd65): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3dd65): Mirror 2: 0xf24d7463 (stale) Mirror 3: 0xf24d7463 (stale) CRC-32 checksum value for chunk [0, 0x3dd65): Mirror 1: 0xc2e08eed Mirror 4: 0xc2e08eed lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3dd65: skipped 000788 1788464367.592917 trunc from 0x03dd65 to 0x00a413 (0x33951 bytes) 000789 1788464367.797275 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xa414): Mirror 3: 0xc7fe53ca (stale) Mirror 4: 0xc7fe53ca (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xa414): Mirror 3: 0xc7fe53ca (stale) Mirror 4: 0xc7fe53ca (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xa414: skipped 000790 1788464368.409715 mapwrite 0x03234e thru 0x038464 (0x6117 bytes) 000791 1788464368.528320 trunc from 0x038465 to 0x01ceba (0x1b5aa bytes) 000792 1788464368.563671 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x1cebb): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1cebb): Mirror 3: 0x64f1b5da (stale) Mirror 4: 0xad7f24 (stale) CRC-32 checksum value for chunk [0, 0x1cebb): Mirror 1: 0xa9642344 Mirror 5: 0xa9642344 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1cebb: skipped 000793 1788464370.315444 write 0x02ab53 thru 0x030c6a (0x6118 bytes) 000794 1788464370.402093 mapwrite 0x00c2dd thru 0x0176aa (0xb3ce bytes) 000797 1788464370.485963 mapwrite 0x01fedb thru 0x024a40 (0x4b66 bytes) 000799 1788464370.608933 trunc from 0x030c6b to 0x020bf5 (0x10075 bytes) 000800 1788464370.636543 write 0x02829a thru 0x02d9c9 (0x5730 bytes) 000801 1788464370.639551 mapread 0x00d55d thru 0x01b87c (0xe320 bytes) 000802 1788464370.644947 mapread 0x01f3b4 thru 0x022e7d (0x3aca bytes) 000804 1788464370.652938 write 0x02b2e8 thru 0x03705f (0xbd78 bytes) 000805 1788464370.655738 mapread 0x02cb63 thru 0x034f1a (0x83b8 bytes) 000808 1788464370.659134 read 0x01e7e3 thru 0x0218ac (0x30ca bytes) 000810 1788464370.660438 trunc from 0x037060 to 0x0387d3 (0x1774 bytes) 000811 1788464370.676849 mapread 0x00e546 thru 0x01d126 (0xebe1 bytes) 000813 1788464370.679494 trunc from 0x0387d4 to 0x006ecc (0x31907 bytes) 000814 1788464370.695333 trunc from 0x006ecd to 0x02e7c2 (0x278f6 bytes) 000815 1788464370.711821 read 0x01450b thru 0x020eff (0xc9f5 bytes) 000817 1788464370.721864 mapread 0x009f55 thru 0x00d65c (0x3708 bytes) 000819 1788464370.725597 punch from 0x006ad6 to 0x0121c7 (0xb6f2 bytes) 000820 1788464370.730614 mapread 0x02e769 thru 0x02e7c2 (0x5a bytes) 000822 1788464370.732139 read 0x029e73 thru 0x02e7c2 (0x4950 bytes) 000823 1788464370.734806 write 0x008922 thru 0x017b16 (0xf1f5 bytes) 000824 1788464370.746631 read 0x007465 thru 0x00d08f (0x5c2b bytes) 000825 1788464370.747893 punch from 0x02108f to 0x027a7e (0x69f0 bytes) 000826 1788464370.764391 mapwrite 0x005d74 thru 0x00e436 (0x86c3 bytes) 000828 1788464370.816359 punch from 0x0228a8 to 0x02e7c2 (0xbf1b bytes) 000829 1788464370.821409 mapread 0x01bd74 thru 0x02abaa (0xee37 bytes) 000830 1788464370.832916 trunc from 0x02e7c3 to 0x03ab79 (0xc3b7 bytes) 000831 1788464370.846935 punch from 0x030dc6 to 0x03ab79 (0x9db4 bytes) 000832 1788464370.851914 mapread 0x0158ef thru 0x02019d (0xa8af bytes) 000834 1788464370.854996 punch from 0x00bcde to 0x00c974 (0xc97 bytes) 000835 1788464370.861842 mapread 0x02c090 thru 0x0362b2 (0xa223 bytes) 000836 1788464370.875149 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3ab7a): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3ab7a): Mirror 3: 0x678d8e12 (stale) Mirror 4: 0x60c06246 (stale) Mirror 5: 0x80aa4b02 (stale) CRC-32 checksum value for chunk [0, 0x3ab7a): Mirror 1: 0x383fb113 Mirror 6: 0x383fb113 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3ab7a: skipped 000838 1788464372.417586 write 0x0373ae thru 0x03ffff (0x8c52 bytes) 000839 1788464372.503763 write 0x034311 thru 0x035a83 (0x1773 bytes) 000840 1788464372.521948 mapwrite 0x03ba8a thru 0x03fd9d (0x4314 bytes) 000842 1788464372.592978 trunc from 0x040000 to 0x0119ae (0x2e651 bytes) 000843 1788464372.617320 read 0x0101a1 thru 0x0119ae (0x180e bytes) 000844 1788464372.631689 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x119af): Mirror 3: (stale - read error) Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x119af): Mirror 3: 0x6e7944ea (stale) Mirror 4: 0x159e792 (stale) Mirror 5: 0x72363782 (stale) Mirror 6: 0x76d3f7bc (stale) CRC-32 checksum value for chunk [0, 0x119af): Mirror 1: 0x76d3f7bc Mirror 7: 0x76d3f7bc lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x119af: skipped 000845 1788464374.093109 mapwrite 0x026fb4 thru 0x02e730 (0x777d bytes) 000847 1788464374.374920 trunc from 0x02e731 to 0x00685b (0x27ed5 bytes) 000848 1788464374.411731 read 0x00194a thru 0x00685b (0x4f12 bytes) 000849 1788464374.443687 mapwrite 0x03d6c1 thru 0x03ffff (0x293f bytes) 000850 1788464374.605775 mapwrite 0x00311d thru 0x008197 (0x507b bytes) 000851 1788464374.724886 read 0x027964 thru 0x02b0ef (0x378c bytes) 000854 1788464374.740845 write 0x029241 thru 0x038d7f (0xfb3f bytes) 000856 1788464374.750263 punch from 0x01032d to 0x01a30e (0x9fe2 bytes) 000860 1788464374.813457 punch from 0x03f7f2 to 0x03ffff (0x80e bytes) 000861 1788464374.820098 trunc from 0x040000 to 0x00c406 (0x33bf9 bytes) 000862 1788464374.856742 MIRROR_SPLIT mirror 3 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc407): Mirror 3: 0x456d29d5 (stale) Mirror 4: 0x456d29d5 (stale) Mirror 6: 0x86f60b96 (stale) Mirror 7: 0x86f60b96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc407): Mirror 3: 0x456d29d5 (stale) Mirror 4: 0x456d29d5 (stale) Mirror 6: 0x86f60b96 (stale) Mirror 7: 0x86f60b96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc407: skipped 000863 1788464375.391084 read 0x0033ec thru 0x00c406 (0x901b bytes) 000865 1788464375.402920 MIRROR_SPLIT mirror 4 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc407): Mirror 3: 0x456d29d5 (stale) Mirror 4: 0x456d29d5 (stale) Mirror 6: 0x86f60b96 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc407): Mirror 3: 0x456d29d5 (stale) Mirror 4: 0x456d29d5 (stale) Mirror 6: 0x86f60b96 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc407: skipped 000868 1788464375.666756 trunc from 0x00c407 to 0x039970 (0x2d56a bytes) 000869 1788464375.684374 trunc from 0x039971 to 0x02944b (0x10525 bytes) 000871 1788464375.696376 punch from 0x0051ab to 0x00c451 (0x72a7 bytes) 000874 1788464375.704728 trunc from 0x02944c to 0x00b9e6 (0x1da65 bytes) 000875 1788464375.730116 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xb9e7): Mirror 4: 0xc55e13c0 (stale) Mirror 6: 0x30358840 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xb9e7): Mirror 4: 0xc55e13c0 (stale) Mirror 6: 0x30358840 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xb9e7: skipped 000877 1788464376.040233 mapread 0x008f4f thru 0x00b9e6 (0x2a98 bytes) 000878 1788464376.072201 trunc from 0x00b9e7 to 0x01b658 (0xfc72 bytes) 000879 1788464376.088990 mapread 0x012645 thru 0x017c16 (0x55d2 bytes) 000880 1788464376.102698 read 0x014d67 thru 0x01ad9a (0x6034 bytes) 000882 1788464376.104073 trunc from 0x01b659 to 0x03809c (0x1ca44 bytes) 000883 1788464376.132461 read 0x0379f3 thru 0x03809c (0x6aa bytes) 000884 1788464376.153826 mapwrite 0x034e0d thru 0x03e82e (0x9a22 bytes) 000887 1788464376.254153 mapwrite 0x00d914 thru 0x0130a8 (0x5795 bytes) 000890 1788464376.317328 write 0x019cc1 thru 0x01a307 (0x647 bytes) 000891 1788464376.332153 MIRROR_RESYNC 3 mirrors 000892 1788464377.751592 mapread 0x01a9cf thru 0x01f489 (0x4abb bytes) 000894 1788464377.782778 trunc from 0x03e82f to 0x036c3c (0x7bf2 bytes) 000895 1788464377.923325 trunc from 0x036c3d to 0x024761 (0x124db bytes) 000896 1788464377.969721 trunc from 0x024762 to 0x00aaa6 (0x19cbb bytes) 000898 1788464378.006134 trunc from 0x00aaa7 to 0x019da6 (0xf300 bytes) 000899 1788464378.024313 trunc from 0x019da7 to 0x0126bd (0x76e9 bytes) 000900 1788464378.052863 mapread 0x00eb46 thru 0x0126bd (0x3b78 bytes) 000901 1788464378.072138 write 0x0164ca thru 0x01dbac (0x76e3 bytes) 000902 1788464378.075155 mapread 0x001935 thru 0x0023dd (0xaa9 bytes) 000903 1788464378.101295 mapwrite 0x039a32 thru 0x03dda9 (0x4378 bytes) 000904 1788464378.224237 trunc from 0x03ddaa to 0x0063eb (0x379be bytes) 000905 1788464378.266809 punch from 0x000803 to 0x004c59 (0x4457 bytes) 000907 1788464378.276253 mapwrite 0x02ef81 thru 0x03410d (0x518d bytes) 000909 1788464378.412197 write 0x03b5f5 thru 0x03ffff (0x4a0b bytes) 000912 1788464378.416861 trunc from 0x040000 to 0x038462 (0x7b9d bytes) 000915 1788464378.441917 mapread 0x01579f thru 0x01fd43 (0xa5a5 bytes) 000917 1788464378.467747 read 0x0221bc thru 0x02f292 (0xd0d7 bytes) 000920 1788464378.471068 write 0x023cb7 thru 0x02f4bb (0xb805 bytes) 000921 1788464378.473662 write 0x01048a thru 0x0121b3 (0x1d2a bytes) 000922 1788464378.475723 mapwrite 0x0397c1 thru 0x03c8e1 (0x3121 bytes) 000924 1788464378.568768 mapwrite 0x02ef01 thru 0x03b121 (0xc221 bytes) 000925 1788464378.666404 mapread 0x0353fc thru 0x03b60a (0x620f bytes) 000926 1788464378.667484 write 0x0068c4 thru 0x00db75 (0x72b2 bytes) 000928 1788464378.685162 trunc from 0x03c8e2 to 0x038eba (0x3a27 bytes) 000931 1788464378.728395 MIRROR_RESYNC 3 mirrors 000932 1788464380.074694 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) 000935 1788464380.511682 read 0x02f068 thru 0x0340b3 (0x504c bytes) 000936 1788464380.520679 trunc from 0x038ebb to 0x019f3e (0x1ef7c bytes) 000937 1788464380.588300 read 0x019529 thru 0x019f3e (0xa16 bytes) 000938 1788464380.601254 MIRROR_RESYNC 2 mirrors 000939 1788464381.405796 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1bc:0x0]: Invalid argument (22) 000940 1788464381.546591 mapread 0x00eb2e thru 0x019f3e (0xb411 bytes) 000941 1788464381.612472 read 0x0176c8 thru 0x019f3e (0x2877 bytes) 000942 1788464381.613650 trunc from 0x019f3f to 0x00bcff (0xe23f bytes) 000943 1788464381.641468 mapread 0x007084 thru 0x00bcff (0x4c7c bytes) 000944 1788464381.649799 mapread 0x009838 thru 0x00a367 (0xb30 bytes) 000945 1788464381.651301 punch from 0x00bb93 to 0x00bcff (0x16d bytes) 000947 1788464381.657071 punch from 0x009332 to 0x00bcff (0x29ce bytes) 000948 1788464381.661714 read 0x0001bb thru 0x00bcff (0xbb45 bytes) 000949 1788464381.669647 mapread 0x00a557 thru 0x00bcff (0x17a9 bytes) 000950 1788464381.671408 mapread 0x001ad9 thru 0x00aaa9 (0x8fd1 bytes) 000951 1788464381.677479 MIRROR_EXTEND to 2 mirrors 000952 1788464382.219472 mapread 0x0096a3 thru 0x00bcff (0x265d bytes) 000955 1788464382.245309 punch from 0x00917c to 0x00bcff (0x2b84 bytes) truncating to largest ever: 0x3f9d8 000956 1788464382.321990 trunc from 0x00bd00 to 0x03f9d7 (0x33cd8 bytes) 000957 1788464382.392344 write 0x019c2b thru 0x0297a8 (0xfb7e bytes) 000959 1788464382.443219 MIRROR_RESYNC 2 mirrors 000960 1788464383.658066 punch from 0x025b5d to 0x02e722 (0x8bc6 bytes) 000961 1788464383.737350 mapwrite 0x01b5a5 thru 0x02a55f (0xefbb bytes) 000963 1788464383.816230 write 0x0153e5 thru 0x018a8e (0x36aa bytes) 000964 1788464383.821621 write 0x002853 thru 0x00353e (0xcec bytes) 000965 1788464383.833358 read 0x031b51 thru 0x037a76 (0x5f26 bytes) 000967 1788464383.862019 MIRROR_RESYNC 2 mirrors 000968 1788464385.018954 MIRROR_RESYNC 2 mirrors 000969 1788464385.443819 write 0x0288cf thru 0x03197f (0x90b1 bytes) 000970 1788464385.580273 read 0x0346dd thru 0x039b1b (0x543f bytes) 000971 1788464385.590025 write 0x031c25 thru 0x03d8b2 (0xbc8e bytes) 000972 1788464385.597214 mapwrite 0x03b210 thru 0x03ffff (0x4df0 bytes) 000973 1788464385.667551 write 0x01a011 thru 0x021e65 (0x7e55 bytes) 000974 1788464385.676366 trunc from 0x040000 to 0x03cfc2 (0x303d bytes) 000975 1788464385.704576 mapread 0x01589b thru 0x0194aa (0x3c10 bytes) 000976 1788464385.720353 mapread 0x031f5d thru 0x037407 (0x54ab bytes) 000977 1788464385.721793 write 0x02ae50 thru 0x02c593 (0x1744 bytes) 000978 1788464385.723262 punch from 0x01aeaa to 0x021683 (0x67da bytes) 000979 1788464385.736767 punch from 0x01bff8 to 0x01e8f7 (0x2900 bytes) 000980 1788464385.741432 mapwrite 0x02c332 thru 0x02ed57 (0x2a26 bytes) 000982 1788464385.788894 mapread 0x002b76 thru 0x007ac0 (0x4f4b bytes) 000983 1788464385.798716 mapwrite 0x0379de thru 0x03ffff (0x8622 bytes) 000984 1788464385.869559 MIRROR_RESYNC 2 mirrors 000987 1788464386.853314 trunc from 0x040000 to 0x0103b2 (0x2fc4d bytes) 000988 1788464387.026834 mapwrite 0x02d95d thru 0x03012c (0x27d0 bytes) 000991 1788464387.211936 punch from 0x0009d6 to 0x00db90 (0xd1bb bytes) 000992 1788464387.225318 punch from 0x02ebae to 0x03012c (0x157f bytes) 000994 1788464387.239480 mapread 0x002c4a thru 0x00c5d4 (0x998b bytes) 000998 1788464387.289166 read 0x000a4d thru 0x00fcbd (0xf271 bytes) 000999 1788464387.300180 read 0x00d68b thru 0x017984 (0xa2fa bytes) 001000 1788464387.334625 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3012d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3012d): Mirror 2: 0x97c74e21 (stale) CRC-32 checksum value for chunk [0, 0x3012d): Mirror 1: 0xeb4e6913 Mirror 3: 0xeb4e6913 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3012d: skipped All operations completed A-OK! PASS 70a (94s) == sanity-flr test 71: check mirror extend parent fid ==== 15:39:53 (1788464393) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (3s) == sanity-flr test 72: check append on FLR file ========== 15:39:57 (1788464397) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.195032 s, 10.8 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176789 s, 5.9 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2eb:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ec:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x308:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x307:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x2ed:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (8s) == sanity-flr test 200a: stress test ===================== 15:40:05 (1788464405) Starting client: oleg619-client.virtnet: -o user_xattr,flock 192.168.206.119@tcp:/lustre /mnt/lustre2 Starting client: oleg619-client.virtnet: -o user_xattr,flock 192.168.206.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 5564992 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 5719776 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 7247680 .. resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 7905984 .. Extending file size to 8406144 .. Extending file size to 7960800 .. Extending file size to 9006880 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8587648 .. Extending file size to 8838688 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9092640 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 57425 57427 57428 57430 57431 done 192.168.206.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg619-client.virtnet /mnt/lustre2 (opts:) 192.168.206.119@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg619-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (80s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 15:41:25 (1788464485) Starting client: oleg619-client.virtnet: -o user_xattr,flock 192.168.206.119@tcp:/lustre /mnt/lustre2 Starting client: oleg619-client.virtnet: -o user_xattr,flock 192.168.206.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 4406208 .. Extending file size to 852864 .. Extending file size to 5770560 .. Extending file size to 7119520 .. Extending file size to 8803040 .. Extending file size to 7391808 .. Extending file size to 8154432 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8726080 .. done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9004960 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58886 58887 58888 58892 58893 done 192.168.206.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg619-client.virtnet /mnt/lustre2 (opts:) 192.168.206.119@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg619-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (87s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 15:42:52 (1788464572) Starting client: oleg619-client.virtnet: -o user_xattr,flock 192.168.206.119@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 16384 3752960 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 23552 3745792 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 39936 7498752 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.132066 s, 15.9 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.4015 s, 5.2 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0x1d15eabc (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.206.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg619-client.virtnet /mnt/lustre2 (opts:) PASS 200c (18s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 15:43:12 (1788464592) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.111461 s, 18.8 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 15:43:20 (1788464600) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1fd:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31e:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x31f:0x0] } - 1: { l_ost_idx: 1, l_fid: [0x280000400:0x302:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 15:43:28 (1788464608) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.216107 s, 9.7 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x320:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x303:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x304:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (8s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 15:43:37 (1788464617) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.473126 s, 4.4 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x321:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x321:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x322:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x305:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x306:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 15:43:47 (1788464627) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.30876 s, 6.8 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x323:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x307:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x308:0x0] } PASS 204c (39s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 15:44:27 (1788464667) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg619-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209881 s, 5.0 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x324:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30a:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x309:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x325:0x0] } PASS 204d (69s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 15:45:35 (1788464735) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg619-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.397814 s, 5.3 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x326:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x327:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30b:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (58s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 15:46:33 (1788464793) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg619-server mds-ost sync done. Creating new pool test_204f oleg619-server: Pool lustre.test_204f created Adding targets to pool oleg619-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg619-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.22699 s, 9.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x328:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x30c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg619-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg619-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg619-server: Pool lustre.test_204f destroyed PASS 204f (82s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 15:47:55 (1788464875) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 15:48:03 (1788464883) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 15:48:12 (1788464892) oleg619-server: Pool lustre.test_206 created Adding targets to pool oleg619-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg619-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Waiting 90s for 'lustre-OST0000_UUID lustre-OST0001_UUID ' Destroy the created pools: test_206 lustre.test_206 oleg619-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg619-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg619-server: Pool lustre.test_206 destroyed PASS 206 (26s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 15:48:38 (1788464918) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0707827 s, 14.8 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0420749 s, 24.9 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x193e23b8 Mirror 3: 0x193e23b8 PASS 207 (10s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 15:48:48 (1788464928) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 15:48:51 (1788464931) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 15:48:55 (1788464935) Starting client: oleg619-client.virtnet: -o user_xattr,flock 192.168.206.119@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.141539 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0xc00000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1ac0000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1ac0000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1ac0000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x10c0000 Read this (4 bytes): BBBB 192.168.206.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg619-client.virtnet /mnt/lustre2 (opts:) PASS 209a (38s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 15:49:33 (1788464973) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0112371 s, 729 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 15:49:39 (1788464979) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573029 s, 18.3 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x222 lmm_fid: [0x200000404:0x222:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x222 lmm_fid: [0x200000404:0x222:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x316:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834466 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-09-03 15:49:41.000000000 -0400 Modify: 2026-09-03 15:49:41.000000000 -0400 Change: 2026-09-03 15:49:43.000000000 -0400 Birth: 2026-09-03 15:49:41.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 15:49:47 (1788464987) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (3s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 15:49:50 (1788464990) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.656252 s, 62.4 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0735431 s, 55.7 kB/s size after second write -rw-r--r-- 1 root root 4096 Sep 3 15:49 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x224:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Sep 3 15:49 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 15:50:00 (1788465000) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (10s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 15:50:09 (1788465009) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:a92bdcf8 and mirror 2:fad40f76: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x17): Mirror 1: 0xa92bdcf8 Mirror 2: 0xfad40f76 Mirror 3: 0xa92bdcf8 verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (11s) == sanity-flr test complete, duration 1801 sec =========== 15:50:20 (1788465020) === sanity-flr: start cleanup 15:50:22 (1788465022) === === sanity-flr: finish cleanup 15:50:30 (1788465030) ===