-----============= acceptance-small: sanity-flr ============----- Thu Jun 11 23:18:26 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg219-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c === sanity-flr: start setup 23:18:46 (1781234326) === oleg219-client.virtnet: executing check_config_client /mnt/lustre oleg219-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg219-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9a1683825000.idle_timeout=debug osc.lustre-OST0001-osc-ffff9a1683825000.idle_timeout=debug disable quota as required oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-flr: finish setup 23:19:23 (1781234363) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8197] == sanity-flr test 0a: lfs mirror create with -N option == 23:19:25 (1781234365) PASS 0a (16s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 23:19:42 (1781234382) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 23:19:44 (1781234384) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 23:19:47 (1781234387) PASS 0d (27s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 23:20:14 (1781234414) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (3s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 23:20:18 (1781234418) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (4s) == sanity-flr test 0g: lfs mirror create flags support === 23:20:22 (1781234422) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff9a1683825000.stats=clear osc.lustre-OST0001-osc-ffff9a1683825000.stats=clear PASS 0g (9s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 23:20:31 (1781234431) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d0h.sanity-flr lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] [--compress|-Z TYPE:LEVEL] [--compress-type=TYPE] [--compress-level=LEVEL] [--compress-chunk=SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--compress|-Z] [--compress-chunk=SIZE] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [--compress|--compr|-Z] [--compress-type|--compr-type] [--compress-level|--compr-level] [--compress-chunk|--compr-chunk] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (12s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 23:20:43 (1781234443) PASS 0j (10s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 23:20:53 (1781234453) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 23:20:57 (1781234457) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (9s) == sanity-flr test 1: create components with setstripe options ========================================================== 23:21:06 (1781234466) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000402:0x1f:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (10s) == sanity-flr test 2: create components from existing files ========================================================== 23:21:16 (1781234476) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 23:21:23 (1781234483) PASS 3 (8s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 23:21:31 (1781234491) striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr striped dir -i0 -c2 -H all_char /mnt/lustre/d4.sanity-flr/td error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 23:21:39 (1781234499) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 23:21:50 (1781234510) PASS 7 (7s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 23:21:58 (1781234518) 15+0 records in 15+0 records out 15728640 bytes (16 MB, 15 MiB) copied, 1.09526 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786018 s, 13.3 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 23:22:07 (1781234527) 17+0 records in 17+0 records out 17825792 bytes (18 MB, 17 MiB) copied, 0.515913 s, 34.6 MB/s PASS 22 (11s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 23:22:18 (1781234538) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0001-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 lod.lustre-MDT0001-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 23:22:29 (1781234549) Stopping /mnt/lustre-ost1 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a1683825000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a1683825000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg219-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a1683825000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a1683825000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg219-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (72s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 23:23:41 (1781234621) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.239773 s, 13.1 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a1683825000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a1683825000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 3145728 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg219-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (48s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 23:24:30 (1781234670) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a1683825000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a1683825000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg219-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a1683825000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a1683825000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg219-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (75s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 23:25:45 (1781234745) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 3.00237 s, 17.5 MB/s -rw-r--r-- 1 root root 52428800 Jun 11 23:25 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.272s user 0m0.001s sys 0m0.189s Stopping /mnt/lustre-ost1 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9a1683825000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff9a1683825000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.297s user 0m0.003s sys 0m0.195s Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg219-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg219-server oleg219-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff9a1683825000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a1683825000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.286s user 0m0.001s sys 0m0.216s Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg219-server: oleg219-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg219-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg219-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9a1683825000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff9a1683825000.ost_server_uuid in FULL state after 0 sec PASS 33b (89s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 23:27:14 (1781234834) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (4s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 23:27:18 (1781234838) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (4s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 23:27:22 (1781234842) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 23:27:26 (1781234846) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 23:27:34 (1781234854) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.620134 s, 20.3 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 23:27:48 (1781234868) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0285697 s, 36.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253395 s, 41.4 MB/s PASS 36b (9s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 23:27:56 (1781234876) PASS 36c (9s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 23:28:05 (1781234885) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=2 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00255206 s, 401 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0011149 s, 918 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00162596 s, 630 kB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00213175 s, 480 kB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 23:28:16 (1781234896) create test files with size 15760517 .. deef5f347f69859541fdbf9cf3ac7cd1 - 6a4d1bc056776065fa0f08466de40354 - 1d7a4f4bb5d829bba5ba6ade90cf7f47 - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg219-client.virtnet /mnt/lustre (opts:) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre PASS 37 (22s) == sanity-flr test 38: resync ============================ 23:28:38 (1781234918) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (13s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 23:28:51 (1781234931) PASS 39 (7s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 23:28:58 (1781234938) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769716 s, 13.6 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xb9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb6:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x7 lmm_fid: 0x200000403:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032106 s, 32.7 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb7:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xb8:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x280000401:0xba:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000403 lmm_object_id: 0x9 lmm_fid: 0x200000403:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (10s) == sanity-flr test 41: lfs mirror resync check =========== 23:29:08 (1781234948) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0793766 s, 26.4 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.319428 s, 13.1 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.237729 s, 8.8 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (19s) == sanity-flr test 42: lfs mirror verify ================= 23:29:27 (1781234967) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (4s) == sanity-flr test 43a: mirror pick on write ============= 23:29:32 (1781234972) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 23:29:36 (1781234976) striped dir -i1 -c2 -H crush2 /mnt/lustre/d43b.sanity-flr ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519439 s, 20.2 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (9s) == sanity-flr test 43c: read prefer uncompressed mirror == 23:29:44 (1781234984) llite.lustre-ffff9a1683b81800.enable_compression=1 striped dir -i1 -c2 -H all_char /mnt/lustre/d43c.sanity-flr fail_loc=0x1425 fail_val=-1 PASS 43c (8s) == sanity-flr test 44a: lfs mirror split check =========== 23:29:52 (1781234992) 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.194263 s, 16.2 MB/s PASS 44a (12s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 23:30:04 (1781235004) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0xc4:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xc3:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (11s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 23:30:17 (1781235017) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.307817 s, 34.1 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000403:0x8:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg219-client.virtnet /mnt/lustre (opts:) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 23:30:31 (1781235031) striped dir -i0 -c2 -H crush /mnt/lustre/d44e.sanity-flr llite.lustre-ffff9a1689306800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.371024 s, 28.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 Stopping client oleg219-client.virtnet /mnt/lustre (opts:) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 mdt.lustre-MDT0001.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 40960 flags: 1 llite.lustre-ffff9a1683ee5800.xattr_cache=1 PASS 44e (17s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 23:30:47 (1781235047) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d45.sanity-flr 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (45s) == sanity-flr test 46: Verify setstripe --copy option ==== 23:31:32 (1781235092) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d46.sanity-flr/dir striped dir -i0 -c2 -H crush2 /mnt/lustre/d46.sanity-flr/dir.copy ** 5. plain dir ** 6. FLR dir PASS 46 (41s) == sanity-flr test 47: Verify mirror obj alloc =========== 23:32:14 (1781235134) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (6s) == sanity-flr test 48: Verify snapshot mirror ============ 23:32:19 (1781235139) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.172479 s, 18.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.153788 s, 20.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0x940c37ca (nosync,stale) Mirror 2: 0x940c37ca (nosync,stale) ** verify mirror 2 doesn't change original checksum: 8c057b6180e87cd4503b59c11c0505f5 - mirror 1 checksum: 92d7ac7d357cdd196d940142717cd005 - mirror 2 checksum: 8c057b6180e87cd4503b59c11c0505f5 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 23:32:21 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: 'Thu Jun 11 23:32:21 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 8c057b6180e87cd4503b59c11c0505f5 - mirror 1 checksum: 92d7ac7d357cdd196d940142717cd005 - mirror 2 checksum: 92d7ac7d357cdd196d940142717cd005 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 23:32:21 2026' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xce:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: 'Thu Jun 11 23:32:21 2026' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0xb lmm_fid: [0x200000405:0xb:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xcf:0x0] } PASS 48 (11s) == sanity-flr test 49a: FIEMAP upon FLR file ============= 23:32:30 (1781235150) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0345003 s, 30.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f49a.sanity-flr is 1048576 (1024 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 312320.. 313343: 1024: 0001: net,eof 1: 0.. 1023: 308224.. 309247: 1024: 0000: last,net,eof /mnt/lustre/f49a.sanity-flr: 2 extents found FIEMAP on /mnt/lustre/f49a.sanity-flr succeeded PASS 49a (8s) == sanity-flr test 50A: mirror split update layout generation ========================================================== 23:32:38 (1781235158) Failing mds1 on oleg219-server Stopping /mnt/lustre-mds1 (opts:) on oleg219-server 23:32:51 (1781235171) shut down facet: mds1 facet_host: oleg219-server facet_failover_host: oleg219-server Failover mds1 to oleg219-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 23:33:16 (1781235196) targets are mounted 23:33:16 (1781235196) facet_failover done oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x11:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0234496 s, 44.7 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x11 lmm_fid: [0x200000405:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xd1:0x0] } PASS 50A (54s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 23:33:32 (1781235212) ** create striped file /mnt/lustre/d50a.sanity-flr/f50a.sanity-flr ** write 1st data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0687581 s, 298 kB/s ** write 2nd data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0189626 s, 1.1 MB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** extend the file with new mirror ** verify mirror #2 sparseness first data offset: 1044480 hole at the end: 3500000 ** copy mirror #2 to mirror #1 mirror copied successfully: 1 ** verify mirror #1 sparseness again first data offset: 1044480 hole at the end: 3500000 ** final consumed blocks: 104 PASS 50a (10s) == sanity-flr test 50b: mirror rsync handles sparseness == 23:33:42 (1781235222) ** create mirrored file /mnt/lustre/d50b.sanity-flr/f50b.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0323645 s, 633 kB/s ** create hole at the file end ** verify sparseness first data offset: 1044480 hole at the end: 3500000 ** resync mirror #2 to mirror #1 ** consumed blocks: 96 ** truncate file down ** write data chunk at 2M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0220145 s, 930 kB/s ** resync mirror #2 to mirror #1 with nosync 1st component ** resync mirror #2 to mirror #1 again ** final consumed blocks: 96 PASS 50b (11s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 23:33:53 (1781235233) striped dir -i0 -c2 -H crush2 /mnt/lustre/d50c.sanity-flr 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.00674064 s, 2.4 MB/s PASS 50c (11s) == sanity-flr test 50d: mirror rsync keep holes ========== 23:34:04 (1781235244) ** create mirrored file /mnt/lustre/d50d.sanity-flr/f50d.sanity-flr ** write data chunk at 1M boundary 20+0 records in 20+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0655303 s, 313 kB/s ** create hole at the file start ** verify sparseness first data offset: 1048576 ** resync mirror #2 ** verify sparseness of mirror #2 first data offset: 1048576 PASS 50d (12s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 23:34:16 (1781235256) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0236209 s, 173 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0387855 s, 106 kB/s PASS 60a (8s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 23:34:24 (1781235264) striped dir -i0 -c2 -H crush2 /mnt/lustre/d60b.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x22:0x0]: Invalid argument (22) PASS 60b (21s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 23:34:45 (1781235285) mdd.lustre-MDT0000.atime_diff=1 mdd.lustre-MDT0001.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 mdd.lustre-MDT0001.atime_diff=60 PASS 61a (32s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 23:35:17 (1781235317) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x4c:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 23:35:42 (1781235342) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 23:36:07 (1781235367) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (8s) == sanity-flr test 70: mirror create and split race ====== 23:36:16 (1781235376) striped dir -i0 -c2 -H crush /mnt/lustre/d70.sanity-flr mirror create pid 57765 mirror split pid 57766 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5016: 57765 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5016: 57766 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (72s) == sanity-flr test 70a: flr mode fsx test ================ 23:37:27 (1781235447) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70a.sanity-flr Seed set to 5449 000001 1781235449.845367 mapwrite 0x03f3b2 thru 0x03ffff (0xc4e bytes) zero_range to largest ever: 0x2d23e 000002 1781235449.903374 trunc from 0x01ff4a to 0x02d23d (0xd2f4 bytes) 000005 1781235449.908602 read 0x00047f thru 0x006827 (0x63a9 bytes) 000006 1781235449.944498 trunc from 0x003ba0 to 0x0040bf (0x520 bytes) 000008 1781235449.957306 mapwrite 0x01df23 thru 0x02ce16 (0xeef4 bytes) 000009 1781235450.073235 read 0x01069b thru 0x01d70a (0xd070 bytes) 000010 1781235450.107599 mapread 0x02f48d thru 0x03bd8c (0xc900 bytes) 000012 1781235450.109537 mapwrite 0x01fcd2 thru 0x0262c0 (0x65ef bytes) 000013 1781235450.167519 fallocate 0x01aec6 thru 0x024e54 (0x9f8f bytes) 000016 1781235450.187475 mapwrite 0x0189be thru 0x0248a9 (0xbeec bytes) 000017 1781235450.236319 mapread 0x008827 thru 0x008c23 (0x3fd bytes) 000018 1781235450.238979 trunc from 0x040000 to 0x01a953 (0x256ac bytes) 000019 1781235450.285498 trunc from 0x01a954 to 0x0292ff (0xe9ac bytes) 000020 1781235450.321723 MIRROR_EXTEND to 2 mirrors 000021 1781235450.847309 trunc from 0x0185a9 to 0x0234cb (0xaf23 bytes) 000022 1781235450.932181 trunc from 0x029300 to 0x015d53 (0x135ac bytes) 000023 1781235451.035933 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x15d54): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15d54): Mirror 2: 0x9a81e091 (stale) CRC-32 checksum value for chunk [0, 0x15d54): Mirror 1: 0x9a81e091 Mirror 3: 0x9a81e091 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15d54: skipped 000024 1781235452.381604 read 0x01525b thru 0x015d53 (0xaf9 bytes) 000025 1781235452.387090 mapwrite 0x004007 thru 0x00f8b2 (0xb8ac bytes) 000026 1781235452.542058 punch from 0x009375 to 0x0147e7 (0xb473 bytes) 000027 1781235452.555812 write 0x00ebc2 thru 0x019576 (0xa9b5 bytes) fallocating to largest ever: 0x30de1 000028 1781235452.589067 fallocate 0x028e8a thru 0x030de0 (0x7f57 bytes) 000029 1781235452.596476 mapwrite 0x029359 thru 0x038b00 (0xf7a8 bytes) 000030 1781235452.682603 MIRROR_RESYNC 3 mirrors 000031 1781235453.824142 trunc from 0x038b01 to 0x010b5b (0x27fa5 bytes) 000032 1781235453.867378 punch from 0x00f628 to 0x010b5b (0x1534 bytes) 000033 1781235453.873171 trunc from 0x00a1c6 to 0x010b5b (0x6996 bytes) 000034 1781235453.878817 mapread 0x0086e3 thru 0x010b5b (0x8479 bytes) 000035 1781235453.903461 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x10b5c): Mirror 2: 0x66834702 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x10b5c): Mirror 2: 0x66834702 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x10b5c: skipped 000036 1781235454.173581 MIRROR_RESYNC 2 mirrors 000037 1781235454.854010 mapread 0x0020be thru 0x0069ca (0x490d bytes) 000038 1781235454.887550 fallocate 0x018d40 thru 0x01cc7a (0x3f3b bytes) 000039 1781235454.952393 read 0x01026b thru 0x01b41d (0xb1b3 bytes) 000040 1781235454.985544 write 0x01e055 thru 0x02d7ab (0xf757 bytes) 000041 1781235454.992863 mapread 0x020474 thru 0x027007 (0x6b94 bytes) 000042 1781235455.000352 mapwrite 0x01c084 thru 0x021ae7 (0x5a64 bytes) 000043 1781235455.046809 trunc from 0x004408 to 0x00bef9 (0x7af2 bytes) 000044 1781235455.052126 punch from 0x014f4b to 0x018978 (0x3a2e bytes) 000045 1781235455.065302 trunc from 0x0216df to 0x02d7ab (0xc0cd bytes) 000046 1781235455.073428 write 0x004a19 thru 0x0134ba (0xeaa2 bytes) 000047 1781235455.094488 mapwrite 0x00d09a thru 0x019f03 (0xce6a bytes) 000048 1781235455.179881 trunc from 0x02d7ac to 0x01548d (0x1831e bytes) 000049 1781235455.210973 punch from 0x00cb11 to 0x01548d (0x897d bytes) 000050 1781235455.219095 mapread 0x0065ab thru 0x00bc81 (0x56d7 bytes) 000051 1781235455.224886 write 0x013afa thru 0x019560 (0x5a67 bytes) 000052 1781235455.240156 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x19561): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19561): Mirror 2: 0xdd0f5342 (stale) CRC-32 checksum value for chunk [0, 0x19561): Mirror 1: 0x89ef8b09 Mirror 3: 0x89ef8b09 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19561: skipped 000053 1781235456.409471 mapwrite 0x00435a thru 0x00b083 (0x6d2a bytes) 000054 1781235456.556071 punch from 0x010ff9 to 0x0188da (0x78e2 bytes) 000055 1781235456.568028 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x19561): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19561): Mirror 2: 0xdd0f5342 (stale) Mirror 3: 0x89ef8b09 (stale) CRC-32 checksum value for chunk [0, 0x19561): Mirror 1: 0x40aff4 Mirror 4: 0x40aff4 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19561: skipped 000056 1781235458.172121 mapwrite 0x00e9f1 thru 0x01df57 (0xf567 bytes) 000057 1781235458.397475 trunc from 0x001fcc to 0x010674 (0xe6a9 bytes) truncating to largest ever: 0x3eabb 000058 1781235458.415982 trunc from 0x01df58 to 0x03eaba (0x20b63 bytes) 000059 1781235458.459714 punch from 0x03b5cb to 0x03eaba (0x34f0 bytes) 000060 1781235458.467058 punch from 0x019559 to 0x021c56 (0x86fe bytes) 000061 1781235458.484649 MIRROR_RESYNC 4 mirrors 000063 1781235460.080888 mapread 0x0016fd thru 0x007a64 (0x6368 bytes) 000064 1781235460.107802 mapread 0x02f868 thru 0x030f5d (0x16f6 bytes) 000065 1781235460.124869 trunc from 0x03eabb to 0x015d52 (0x28d68 bytes) 000066 1781235460.212154 read 0x00d8ba thru 0x015a5b (0x81a2 bytes) 000067 1781235460.225182 fallocate 0x02ce96 thru 0x0399a2 (0xcb0d bytes) 000068 1781235460.231673 read 0x0157fc thru 0x015d52 (0x557 bytes) 000069 1781235460.232859 punch from 0x0038f1 to 0x010104 (0xc814 bytes) 000070 1781235460.249831 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x15d53): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x15d53): Mirror 2: 0x12e6e09a (stale) Mirror 3: 0x12e6e09a (stale) Mirror 4: 0x12e6e09a (stale) CRC-32 checksum value for chunk [0, 0x15d53): Mirror 1: 0x12e6e09a Mirror 5: 0x12e6e09a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x15d53: skipped 000071 1781235461.359894 mapread 0x00a3b1 thru 0x00deae (0x3afe bytes) 000072 1781235461.419965 punch from 0x00787f to 0x012418 (0xab9a bytes) 000073 1781235461.550800 mapread 0x013386 thru 0x015d52 (0x29cd bytes) 000074 1781235461.580933 mapwrite 0x014cac thru 0x0203bb (0xb710 bytes) 000075 1781235461.813943 mapwrite 0x03c9a2 thru 0x03e4e1 (0x1b40 bytes) 000076 1781235461.927858 mapread 0x034e0a thru 0x03e4e1 (0x96d8 bytes) 000077 1781235461.973893 mapwrite 0x006790 thru 0x00bd23 (0x5594 bytes) 000078 1781235462.069776 mapwrite 0x011af6 thru 0x01f012 (0xd51d bytes) 000080 1781235462.130890 mapread 0x005c77 thru 0x00cf64 (0x72ee bytes) 000081 1781235462.132226 write 0x036cbf thru 0x03fb0b (0x8e4d bytes) 000082 1781235462.157646 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3fb0c): Mirror 2: 0xe41097f2 (stale) Mirror 4: 0xe41097f2 (stale) Mirror 5: 0xc4975f93 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fb0c): Mirror 2: 0xe41097f2 (stale) Mirror 4: 0xe41097f2 (stale) Mirror 5: 0xc4975f93 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fb0c: skipped 000083 1781235462.750065 fallocate 0x006717 thru 0x00c5f9 (0x5ee3 bytes) 000084 1781235462.757682 read 0x03faf2 thru 0x03fb0b (0x1a bytes) 000085 1781235462.775621 fallocate 0x010891 thru 0x013dd1 (0x3541 bytes) 000087 1781235462.791802 trunc from 0x03fb0c to 0x0263f1 (0x1971a bytes) 000088 1781235462.829949 trunc from 0x0263f2 to 0x018936 (0xdabb bytes) 000090 1781235462.855914 trunc from 0x018937 to 0x001c79 (0x16cbd bytes) 000091 1781235462.881156 trunc from 0x001c7a to 0x011f56 (0x102dd bytes) 000092 1781235462.900305 write 0x01a597 thru 0x022cff (0x8769 bytes) fallocating to largest ever: 0x40000 000093 1781235462.904317 fallocate 0x03cc12 thru 0x03ffff (0x33ee bytes) 000094 1781235462.911761 write 0x01b981 thru 0x01c22b (0x8ab bytes) 000095 1781235462.913650 trunc from 0x022d00 to 0x02e051 (0xb352 bytes) 000097 1781235462.935668 mapread 0x015ff0 thru 0x019fde (0x3fef bytes) 000098 1781235462.958000 trunc from 0x002a86 to 0x011da0 (0xf31b bytes) 000099 1781235462.963775 mapread 0x029b19 thru 0x02e051 (0x4539 bytes) 000100 1781235462.975325 trunc from 0x00843c to 0x01830a (0xfecf bytes) 000101 1781235462.982347 mapread 0x02762c thru 0x02be05 (0x47da bytes) 000102 1781235462.985452 trunc from 0x024c57 to 0x027f50 (0x32fa bytes) 000104 1781235462.992411 mapwrite 0x03f261 thru 0x03ffff (0xd9f bytes) 000105 1781235463.045095 read 0x037cfb thru 0x03e849 (0x6b4f bytes) 000106 1781235463.090271 punch from 0x0309ea to 0x03e066 (0xd67d bytes) 000107 1781235463.097446 mapwrite 0x01ea98 thru 0x01eb13 (0x7c bytes) 000108 1781235463.113325 mapread 0x0128fe thru 0x01cd5d (0xa460 bytes) 000109 1781235463.115851 write 0x00e7bd thru 0x01c113 (0xd957 bytes) 000110 1781235463.119458 punch from 0x02d705 to 0x033a1b (0x6317 bytes) 000112 1781235463.130080 read 0x007989 thru 0x015524 (0xdb9c bytes) 000113 1781235463.131497 read 0x03a05d thru 0x03ffff (0x5fa3 bytes) 000114 1781235463.156312 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb7a70507 (stale) Mirror 5: 0x9288994c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xb7a70507 (stale) Mirror 5: 0x9288994c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000115 1781235463.628464 punch from 0x02ca27 to 0x030857 (0x3e31 bytes) 000116 1781235463.636241 read 0x00e28c thru 0x01dd57 (0xfacc bytes) 000117 1781235463.655342 trunc from 0x040000 to 0x001683 (0x3e97c bytes) 000118 1781235463.688396 mapwrite 0x03bbbb thru 0x03ffff (0x4445 bytes) 000119 1781235463.787631 mapwrite 0x03555c thru 0x03b975 (0x641a bytes) 000120 1781235463.826242 fallocate 0x003344 thru 0x012f5f (0xfc1c bytes) 000121 1781235463.832486 write 0x011f07 thru 0x012abb (0xbb5 bytes) 000122 1781235463.847629 trunc from 0x040000 to 0x020aa5 (0x1f55a bytes) 000123 1781235463.891832 MIRROR_RESYNC 3 mirrors 000124 1781235464.756861 punch from 0x01ebe9 to 0x020aa5 (0x1ebd bytes) 000125 1781235464.796571 write 0x00ac81 thru 0x017cca (0xd04a bytes) 000127 1781235464.811833 trunc from 0x020aa6 to 0x01159e (0xf507 bytes) 000128 1781235464.833443 mapwrite 0x030194 thru 0x03e02b (0xde98 bytes) 000130 1781235464.896101 read 0x012477 thru 0x01c171 (0x9cfb bytes) 000131 1781235464.914374 fallocate 0x01a1d0 thru 0x01f776 (0x55a7 bytes) 000132 1781235464.926887 write 0x00b0da thru 0x0157b3 (0xa6da bytes) 000133 1781235464.929529 mapwrite 0x0105df thru 0x01d2e9 (0xcd0b bytes) 000135 1781235464.976755 read 0x02fd62 thru 0x037bfa (0x7e99 bytes) 000136 1781235464.978369 trunc from 0x00d1e5 to 0x015df7 (0x8c13 bytes) 000138 1781235464.987857 write 0x020df1 thru 0x021585 (0x795 bytes) 000139 1781235464.990839 read 0x00ee8e thru 0x015e20 (0x6f93 bytes) 000140 1781235465.001852 trunc from 0x03e02c to 0x03246a (0xbbc1 bytes) 000141 1781235465.020568 mapwrite 0x01aa0a thru 0x024497 (0x9a8e bytes) 000143 1781235465.070308 mapwrite 0x02d9ab thru 0x039a35 (0xc08b bytes) 000144 1781235465.183813 mapwrite 0x02748e thru 0x032c24 (0xb797 bytes) 000145 1781235465.222183 mapwrite 0x029a4e thru 0x02ed74 (0x5327 bytes) 000146 1781235465.256309 trunc from 0x039a36 to 0x03d17e (0x3749 bytes) 000147 1781235465.285561 trunc from 0x018e60 to 0x019728 (0x8c9 bytes) 000148 1781235465.304119 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3d17f): Mirror 4: 0xf3074066 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3d17f): Mirror 4: 0xf3074066 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3d17f: skipped 000149 1781235465.695343 write 0x01d96f thru 0x0292fc (0xb98e bytes) 000151 1781235465.710841 trunc from 0x03d17f to 0x03c259 (0xf25 bytes) 000152 1781235465.731127 trunc from 0x016ee8 to 0x018b18 (0x1c31 bytes) 000153 1781235465.739515 trunc from 0x037daf to 0x03c259 (0x44ab bytes) 000154 1781235465.748811 trunc from 0x03c25a to 0x02dd51 (0xe508 bytes) 000155 1781235465.780173 punch from 0x005e13 to 0x00ee32 (0x9020 bytes) 000156 1781235465.785811 mapread 0x011019 thru 0x01c59a (0xb582 bytes) 000157 1781235465.837715 mapwrite 0x014d7b thru 0x01d084 (0x830a bytes) 000159 1781235465.882325 fallocate 0x001000 thru 0x008ec9 (0x7eca bytes) 000160 1781235465.892120 fallocate 0x004c59 thru 0x013f1d (0xf2c5 bytes) 000162 1781235465.912158 trunc from 0x018587 to 0x023c6f (0xb6e9 bytes) 000163 1781235465.930184 read 0x023060 thru 0x02aa7c (0x7a1d bytes) 000164 1781235465.968095 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2dd52): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2dd52): Mirror 4: 0xa13cf38e (stale) CRC-32 checksum value for chunk [0, 0x2dd52): Mirror 1: 0x71344fb8 Mirror 5: 0x71344fb8 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2dd52: skipped 000165 1781235467.692892 trunc from 0x00fba7 to 0x01095b (0xdb5 bytes) 000166 1781235467.755890 trunc from 0x02dd52 to 0x02f953 (0x1c02 bytes) 000167 1781235467.803043 mapwrite 0x000867 thru 0x009f9a (0x9734 bytes) 000168 1781235467.866639 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2f954): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2f954): Mirror 4: 0xab6b17f7 (stale) Mirror 5: 0x1e41c2b0 (stale) CRC-32 checksum value for chunk [0, 0x2f954): Mirror 1: 0x63a48577 Mirror 6: 0x63a48577 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2f954: skipped 000169 1781235469.121620 trunc from 0x02f954 to 0x019281 (0x166d2 bytes) 000170 1781235469.189696 punch from 0x017aa6 to 0x019281 (0x17dc bytes) 000171 1781235469.195180 read 0x018d65 thru 0x019281 (0x51d bytes) 000173 1781235469.206141 mapwrite 0x024b49 thru 0x024da6 (0x25e bytes) 000174 1781235469.271401 punch from 0x01eab0 to 0x02358d (0x4ade bytes) 000175 1781235469.292901 read 0x0220a0 thru 0x024da6 (0x2d07 bytes) 000176 1781235469.335793 fallocate 0x02c813 thru 0x035405 (0x8bf3 bytes) 000177 1781235469.356099 punch from 0x00d3c8 to 0x00d7cd (0x406 bytes) 000178 1781235469.364862 trunc from 0x005661 to 0x005701 (0xa1 bytes) 000179 1781235469.370817 trunc from 0x003dad to 0x00f8fb (0xbb4f bytes) 000180 1781235469.377516 read 0x01fb09 thru 0x024da6 (0x529e bytes) 000182 1781235469.380573 trunc from 0x024da7 to 0x023e4c (0xf5a bytes) 000183 1781235469.396414 mapread 0x02060d thru 0x023e4c (0x3840 bytes) 000184 1781235469.398879 trunc from 0x01bbef to 0x023e4c (0x825e bytes) 000185 1781235469.404865 punch from 0x0159b5 to 0x018356 (0x29a2 bytes) 000186 1781235469.410853 punch from 0x002f7a to 0x00745c (0x44e3 bytes) 000188 1781235469.417775 mapread 0x013521 thru 0x01ac3f (0x771f bytes) 000189 1781235469.430413 trunc from 0x023e4d to 0x02f403 (0xb5b7 bytes) 000191 1781235469.456409 trunc from 0x0222b1 to 0x022637 (0x387 bytes) 000193 1781235469.466980 MIRROR_RESYNC 4 mirrors 000195 1781235470.524397 trunc from 0x02f404 to 0x03973c (0xa339 bytes) 000196 1781235470.635129 mapwrite 0x008d60 thru 0x00a3d8 (0x1679 bytes) 000198 1781235470.730227 read 0x018f55 thru 0x025951 (0xc9fd bytes) 000199 1781235470.742547 fallocate 0x00fdd3 thru 0x01fa5b (0xfc89 bytes) 000200 1781235470.748687 punch from 0x01a59f to 0x026c6a (0xc6cc bytes) 000201 1781235470.757683 punch from 0x011f3b to 0x01d2e7 (0xb3ad bytes) 000202 1781235470.774267 punch from 0x0346fc to 0x037b38 (0x343d bytes) 000203 1781235470.783375 read 0x0238cb thru 0x02e494 (0xabca bytes) 000204 1781235470.813990 write 0x020fe4 thru 0x02500d (0x402a bytes) 000205 1781235470.818520 fallocate 0x0270e7 thru 0x02c808 (0x5722 bytes) 000206 1781235470.826360 mapwrite 0x02ae7b thru 0x02ebf3 (0x3d79 bytes) 000207 1781235470.859693 mapwrite 0x0121ce thru 0x020bae (0xe9e1 bytes) 000208 1781235470.903153 trunc from 0x026a82 to 0x029c56 (0x31d5 bytes) 000209 1781235470.915947 write 0x02d2cf thru 0x02eb39 (0x186b bytes) 000210 1781235470.919182 trunc from 0x03973d to 0x018523 (0x21219 bytes) 000211 1781235470.956721 punch from 0x00fe7e to 0x018523 (0x86a6 bytes) 000212 1781235470.963102 read 0x005833 thru 0x00ae06 (0x55d4 bytes) 000213 1781235470.964350 trunc from 0x0012b3 to 0x00960c (0x835a bytes) 000214 1781235470.975182 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x18524): Mirror 4: 0xe47efbf2 (stale) Mirror 5: 0xe47efbf2 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x18524): Mirror 4: 0xe47efbf2 (stale) Mirror 5: 0xe47efbf2 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x18524: skipped 000215 1781235471.327079 fallocate 0x036394 thru 0x03ffff (0x9c6c bytes) 000216 1781235471.334904 fallocate 0x03f053 thru 0x03ffff (0xfad bytes) 000217 1781235471.340924 trunc from 0x040000 to 0x039952 (0x66ad bytes) 000218 1781235471.365581 write 0x016928 thru 0x01c65e (0x5d37 bytes) 000219 1781235471.375288 punch from 0x01d6bd to 0x02d212 (0xfb56 bytes) 000220 1781235471.380736 trunc from 0x039953 to 0x010af9 (0x28e59 bytes) 000221 1781235471.394476 fallocate 0x029d49 thru 0x02a552 (0x80a bytes) 000222 1781235471.403042 trunc from 0x02a1c3 to 0x02a552 (0x390 bytes) 000223 1781235471.406597 punch from 0x01963f to 0x029393 (0xfd55 bytes) 000224 1781235471.411467 mapwrite 0x03c09d thru 0x03ffff (0x3f63 bytes) 000225 1781235471.450659 punch from 0x00140a to 0x00c713 (0xb30a bytes) 000226 1781235471.456707 read 0x02422f thru 0x02ee5f (0xac31 bytes) 000227 1781235471.466457 punch from 0x011cfc to 0x01ada5 (0x90aa bytes) 000228 1781235471.476593 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6405e8b6 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x6405e8b6 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000229 1781235471.760288 mapread 0x021e4c thru 0x02d6ab (0xb860 bytes) 000230 1781235471.791809 fallocate 0x006ff3 thru 0x00eca3 (0x7cb1 bytes) 000231 1781235471.797075 mapread 0x024d73 thru 0x03010e (0xb39c bytes) 000232 1781235471.798634 fallocate 0x015d84 thru 0x01c72f (0x69ac bytes) 000233 1781235471.803378 fallocate 0x021d6e thru 0x025c28 (0x3ebb bytes) 000234 1781235471.807944 write 0x03600a thru 0x03fe7f (0x9e76 bytes) 000235 1781235471.823897 fallocate 0x03788c thru 0x03ffff (0x8774 bytes) 000236 1781235471.829104 trunc from 0x040000 to 0x03a2ae (0x5d51 bytes) 000237 1781235471.841164 trunc from 0x03a2af to 0x02f9be (0xa8f0 bytes) 000238 1781235471.855086 write 0x031997 thru 0x03a1dc (0x8846 bytes) 000239 1781235471.857083 trunc from 0x02c872 to 0x02fba4 (0x3333 bytes) 000240 1781235471.864515 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3a1dd): Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3a1dd): Mirror 4: 0xbf9b95a2 (stale) CRC-32 checksum value for chunk [0, 0x3a1dd): Mirror 1: 0x251911a0 Mirror 5: 0x251911a0 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3a1dd: skipped 000241 1781235472.847167 trunc from 0x0300ed to 0x03a1dc (0xa0f0 bytes) 000243 1781235472.909964 fallocate 0x007733 thru 0x00dc82 (0x6550 bytes) 000244 1781235472.916550 fallocate 0x004494 thru 0x00f7d1 (0xb33e bytes) 000245 1781235472.922660 trunc from 0x03a1dd to 0x03fc53 (0x5a77 bytes) 000246 1781235472.954772 mapread 0x00049e thru 0x00925b (0x8dbe bytes) 000247 1781235472.987804 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x3fc54): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3fc54): Mirror 4: 0xb51e2ff2 (stale) Mirror 5: 0xe0928333 (stale) CRC-32 checksum value for chunk [0, 0x3fc54): Mirror 1: 0x9a46d907 Mirror 6: 0x9a46d907 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3fc54: skipped 000248 1781235474.103318 MIRROR_RESYNC 4 mirrors 000249 1781235475.553365 trunc from 0x015275 to 0x015d9e (0xb2a bytes) 000251 1781235475.603562 punch from 0x01e9c5 to 0x01f491 (0xacd bytes) 000252 1781235475.617988 MIRROR_RESYNC 4 mirrors 000253 1781235477.009895 punch from 0x01e467 to 0x02102a (0x2bc4 bytes) 000254 1781235477.047962 MIRROR_RESYNC 4 mirrors 000255 1781235478.403294 fallocate 0x02fc3a thru 0x03b926 (0xbced bytes) 000257 1781235478.441297 punch from 0x0361b9 to 0x03fc53 (0x9a9b bytes) 000258 1781235478.446455 trunc from 0x03fc54 to 0x019f93 (0x25cc0 bytes) 000259 1781235478.471877 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x19f94): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19f94): Mirror 4: 0x465df18b (stale) Mirror 5: 0x465df18b (stale) Mirror 6: 0x465df18b (stale) CRC-32 checksum value for chunk [0, 0x19f94): Mirror 1: 0x465df18b Mirror 7: 0x465df18b lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19f94: skipped 000260 1781235479.505854 fallocate 0x00fc16 thru 0x0100a9 (0x494 bytes) 000261 1781235479.585170 write 0x00db58 thru 0x018723 (0xabcc bytes) 000262 1781235479.630001 MIRROR_EXTEND to 6 mirrors CRC-32 checksum value for chunk [0, 0x19f94): Mirror 4: (stale - read error) Mirror 5: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x19f94): Mirror 4: 0x465df18b (stale) Mirror 5: 0x465df18b (stale) Mirror 6: 0x465df18b (stale) Mirror 7: 0x465df18b (stale) CRC-32 checksum value for chunk [0, 0x19f94): Mirror 1: 0xe4292142 Mirror 8: 0xe4292142 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x19f94: skipped 000263 1781235480.949135 read 0x012aa0 thru 0x018ef7 (0x6458 bytes) 000264 1781235480.960017 write 0x03aa0f thru 0x03ffff (0x55f1 bytes) 000265 1781235481.025093 read 0x0113e0 thru 0x017f9a (0x6bbb bytes) 000266 1781235481.043840 MIRROR_SPLIT mirror 5 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x70cc3186 (stale) Mirror 5: 0x70cc3186 (stale) Mirror 6: 0x70cc3186 (stale) Mirror 7: 0x70cc3186 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x70cc3186 (stale) Mirror 5: 0x70cc3186 (stale) Mirror 6: 0x70cc3186 (stale) Mirror 7: 0x70cc3186 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000267 1781235481.643115 read 0x0247d4 thru 0x026f14 (0x2741 bytes) 000268 1781235481.652940 trunc from 0x02ce05 to 0x034cff (0x7efb bytes) 000269 1781235481.659892 punch from 0x019630 to 0x025fbb (0xc98c bytes) 000270 1781235481.681274 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x70cc3186 (stale) Mirror 6: 0x70cc3186 (stale) Mirror 7: 0x70cc3186 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x70cc3186 (stale) Mirror 6: 0x70cc3186 (stale) Mirror 7: 0x70cc3186 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000271 1781235482.262402 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 6: (stale - read error) Mirror 7: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x70cc3186 (stale) Mirror 6: 0x70cc3186 (stale) Mirror 7: 0x70cc3186 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x9c3d28d7 Mirror 8: 0x9c3d28d7 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000272 1781235483.160803 mapwrite 0x0286d4 thru 0x02d345 (0x4c72 bytes) 000273 1781235483.335539 mapwrite 0x008283 thru 0x011af7 (0x9875 bytes) 000274 1781235483.407039 fallocate 0x024bd2 thru 0x026868 (0x1c97 bytes) 000275 1781235483.423431 MIRROR_RESYNC 5 mirrors 000277 1781235484.941181 mapwrite 0x018fdc thru 0x028e30 (0xfe55 bytes) 000278 1781235485.068695 trunc from 0x018f9a to 0x022c6d (0x9cd4 bytes) 000279 1781235485.076852 write 0x016b4b thru 0x01978d (0x2c43 bytes) 000280 1781235485.095281 MIRROR_RESYNC 5 mirrors 000281 1781235486.325276 mapread 0x02a381 thru 0x02afa0 (0xc20 bytes) 000282 1781235486.331735 write 0x0012d3 thru 0x00dcd1 (0xc9ff bytes) 000284 1781235486.392235 punch from 0x006c0a to 0x00731d (0x714 bytes) 000285 1781235486.411712 MIRROR_SPLIT mirror 2 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8f7af81b (stale) Mirror 7: 0x8f7af81b (stale) Mirror 8: 0x8f7af81b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8f7af81b (stale) Mirror 7: 0x8f7af81b (stale) Mirror 8: 0x8f7af81b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000286 1781235486.772896 trunc from 0x016c40 to 0x018f7d (0x233e bytes) 000287 1781235486.780705 fallocate 0x004216 thru 0x013661 (0xf44c bytes) 000288 1781235486.787715 trunc from 0x0351a7 to 0x03cd92 (0x7bec bytes) 000289 1781235486.793816 fallocate 0x035468 thru 0x03ffff (0xab98 bytes) 000290 1781235486.799779 mapread 0x009cdd thru 0x018f5b (0xf27f bytes) 000291 1781235486.853377 fallocate 0x039581 thru 0x03de1a (0x489a bytes) 000292 1781235486.863687 write 0x03733d thru 0x03c3b0 (0x5074 bytes) 000293 1781235486.892338 mapwrite 0x004cc3 thru 0x006d54 (0x2092 bytes) 000294 1781235486.946873 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8f7af81b (stale) Mirror 8: 0x8f7af81b (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8f7af81b (stale) Mirror 8: 0x8f7af81b (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000295 1781235487.182882 MIRROR_RESYNC 3 mirrors 000296 1781235487.854332 mapread 0x03df3b thru 0x03ffff (0x20c5 bytes) 000297 1781235487.871486 mapwrite 0x00f91e thru 0x017b8f (0x8272 bytes) 000298 1781235487.960496 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 8: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0xda35a8b7 (stale) Mirror 8: 0xda35a8b7 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x91a52a0c Mirror 9: 0x91a52a0c lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000300 1781235488.983990 trunc from 0x040000 to 0x03d86f (0x2790 bytes) 000301 1781235489.056714 mapread 0x0366ec thru 0x03d86f (0x7184 bytes) 000302 1781235489.100074 fallocate 0x008466 thru 0x013534 (0xb0cf bytes) 000303 1781235489.108125 trunc from 0x03d870 to 0x000f17 (0x3c958 bytes) 000304 1781235489.153725 trunc from 0x000f18 to 0x01639c (0x15485 bytes) 000306 1781235489.178705 fallocate 0x023958 thru 0x028b78 (0x5221 bytes) 000307 1781235489.199870 MIRROR_SPLIT mirror 2 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x1639d): Mirror 4: 0xb0b13fbc (stale) Mirror 9: 0x26e8cbad (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x1639d): Mirror 4: 0xb0b13fbc (stale) Mirror 9: 0x26e8cbad (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x1639d: skipped 000308 1781235489.639130 write 0x037c5c thru 0x03ffff (0x83a4 bytes) 000309 1781235489.668366 trunc from 0x005b82 to 0x00d015 (0x7494 bytes) 000310 1781235489.687963 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x91a52a0c (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 9: 0x91a52a0c (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000311 1781235490.034064 MIRROR_RESYNC 2 mirrors 000312 1781235490.821041 trunc from 0x040000 to 0x01e04c (0x21fb3 bytes) 000313 1781235490.860617 read 0x00b6ef thru 0x0135cb (0x7edd bytes) 000314 1781235490.874286 trunc from 0x01e04d to 0x031510 (0x134c4 bytes) 000315 1781235490.908245 trunc from 0x031511 to 0x015d10 (0x1b800 bytes) 000316 1781235490.938410 read 0x008a52 thru 0x015d10 (0xd2bf bytes) 000317 1781235490.949553 fallocate 0x035d1f thru 0x03cf99 (0x727b bytes) 000318 1781235490.953592 trunc from 0x03cf9a to 0x024b39 (0x18460 bytes) 000319 1781235490.963622 punch from 0x021b30 to 0x02378c (0x1c5d bytes) 000320 1781235490.969250 MIRROR_RESYNC 2 mirrors 000321 1781235491.634150 mapwrite 0x028191 thru 0x02fe54 (0x7cc4 bytes) 000322 1781235491.739079 read 0x00f78c thru 0x019f49 (0xa7be bytes) 000324 1781235491.746062 mapread 0x00cdd2 thru 0x015eb6 (0x90e5 bytes) 000325 1781235491.767675 punch from 0x00a368 to 0x013e50 (0x9ae9 bytes) 000326 1781235491.777594 trunc from 0x02fe55 to 0x03cefa (0xd0a6 bytes) 000327 1781235491.812161 fallocate 0x021d9c thru 0x02c293 (0xa4f8 bytes) 000328 1781235491.820721 mapread 0x023a8b thru 0x0248fc (0xe72 bytes) 000329 1781235491.842489 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) 000330 1781235492.110894 fallocate 0x008bbc thru 0x00ca1f (0x3e64 bytes) 000331 1781235492.119186 write 0x02eb05 thru 0x034879 (0x5d75 bytes) 000332 1781235492.151088 trunc from 0x03cefb to 0x0272ff (0x15bfb bytes) 000333 1781235492.179158 mapread 0x00d32b thru 0x017b1d (0xa7f3 bytes) 000335 1781235492.214950 fallocate 0x0380eb thru 0x039c9c (0x1bb2 bytes) 000336 1781235492.221030 MIRROR_EXTEND to 2 mirrors 000337 1781235492.853596 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) 000338 1781235492.916729 trunc from 0x039c9d to 0x00a759 (0x2f543 bytes) 000340 1781235492.950129 mapwrite 0x0012a9 thru 0x006923 (0x567b bytes) 000342 1781235492.999530 write 0x02dbbb thru 0x036c23 (0x9069 bytes) 000343 1781235493.005653 mapread 0x00878d thru 0x009a44 (0x12b8 bytes) 000344 1781235493.007704 read 0x0144ed thru 0x01b412 (0x6f26 bytes) 000345 1781235493.022727 fallocate 0x03120d thru 0x032589 (0x137d bytes) 000346 1781235493.032917 punch from 0x035321 to 0x036c23 (0x1903 bytes) 000348 1781235493.050493 fallocate 0x03e354 thru 0x03ffff (0x1cac bytes) 000350 1781235493.058028 fallocate 0x00d492 thru 0x010a60 (0x35cf bytes) 000351 1781235493.064188 write 0x010315 thru 0x01725f (0x6f4b bytes) 000352 1781235493.076924 punch from 0x033455 to 0x03931a (0x5ec6 bytes) 000354 1781235493.083390 read 0x017d2b thru 0x01c3d5 (0x46ab bytes) 000355 1781235493.094897 read 0x03f637 thru 0x03ffff (0x9c9 bytes) 000358 1781235493.097997 mapread 0x012622 thru 0x01b513 (0x8ef2 bytes) 000360 1781235493.100782 mapwrite 0x01cd6a thru 0x029739 (0xc9d0 bytes) 000361 1781235493.120812 trunc from 0x040000 to 0x02eca9 (0x11356 bytes) 000362 1781235493.134545 write 0x005a81 thru 0x0130a0 (0xd620 bytes) 000363 1781235493.136930 mapread 0x000023 thru 0x00fa65 (0xfa43 bytes) 000364 1781235493.139032 mapread 0x027815 thru 0x02eca9 (0x7495 bytes) 000365 1781235493.140715 trunc from 0x015f86 to 0x01c34a (0x63c5 bytes) 000366 1781235493.152747 write 0x010a74 thru 0x01671d (0x5caa bytes) 000367 1781235493.167734 punch from 0x006949 to 0x00788d (0xf45 bytes) 000369 1781235493.196703 mapwrite 0x0221bc thru 0x030572 (0xe3b7 bytes) 000370 1781235493.290701 write 0x0197f8 thru 0x026845 (0xd04e bytes) 000371 1781235493.292291 trunc from 0x030573 to 0x0114a1 (0x1f0d1 bytes) 000373 1781235493.335911 fallocate 0x025460 thru 0x0339f0 (0xe591 bytes) 000374 1781235493.358803 punch from 0x00242d to 0x008a80 (0x6654 bytes) 000375 1781235493.365717 mapread 0x0084f3 thru 0x01056c (0x807a bytes) 000376 1781235493.370138 fallocate 0x025cc3 thru 0x026ead (0x11eb bytes) 000377 1781235493.373812 trunc from 0x010b67 to 0x0114a1 (0x93b bytes) 000378 1781235493.377456 punch from 0x00cb3e to 0x0114a1 (0x4964 bytes) 000379 1781235493.383116 trunc from 0x0114a2 to 0x004bc2 (0xc8df bytes) 000380 1781235493.403347 fallocate 0x01a15c thru 0x01feed (0x5d92 bytes) 000381 1781235493.407591 fallocate 0x00d046 thru 0x0131d0 (0x618b bytes) 000382 1781235493.411862 mapread 0x008890 thru 0x00ec66 (0x63d7 bytes) 000385 1781235493.449133 trunc from 0x0029ba to 0x00716b (0x47b2 bytes) 000386 1781235493.454654 read 0x0130d0 thru 0x01d4fc (0xa42d bytes) 000388 1781235493.464014 mapread 0x0198c2 thru 0x01feed (0x662c bytes) 000389 1781235493.464989 punch from 0x019460 to 0x01e3a7 (0x4f48 bytes) 000390 1781235493.468140 fallocate 0x0261b6 thru 0x02c5f5 (0x6440 bytes) 000391 1781235493.471878 mapwrite 0x0188d7 thru 0x019769 (0xe93 bytes) 000392 1781235493.494275 trunc from 0x025d23 to 0x02c5f5 (0x68d3 bytes) 000393 1781235493.498089 read 0x0098eb thru 0x0104f1 (0x6c07 bytes) 000394 1781235493.498767 mapwrite 0x039ea6 thru 0x03b32b (0x1486 bytes) 000395 1781235493.552694 mapwrite 0x021efc thru 0x02c8c3 (0xa9c8 bytes) 000396 1781235493.582065 punch from 0x01e435 to 0x01ead8 (0x6a4 bytes) 000397 1781235493.585727 mapwrite 0x01c0f9 thru 0x01efbf (0x2ec7 bytes) 000398 1781235493.600048 punch from 0x00dcca to 0x015eba (0x81f1 bytes) 000399 1781235493.604792 write 0x03f9b3 thru 0x03ffff (0x64d bytes) 000400 1781235493.605787 trunc from 0x040000 to 0x0016f8 (0x3e907 bytes) 000401 1781235493.618640 fallocate 0x02f92f thru 0x038443 (0x8b15 bytes) 000404 1781235493.625654 read 0x003c2b thru 0x0099de (0x5db4 bytes) 000405 1781235493.637500 read 0x0270a0 thru 0x036c65 (0xfbc6 bytes) 000406 1781235493.638434 mapread 0x00e28c thru 0x016d7f (0x8af4 bytes) 000407 1781235493.639810 mapwrite 0x022180 thru 0x028c57 (0x6ad8 bytes) 000408 1781235493.658942 write 0x03b037 thru 0x03c7c4 (0x178e bytes) 000409 1781235493.665439 fallocate 0x01d8f0 thru 0x02cd9f (0xf4b0 bytes) 000410 1781235493.670076 fallocate 0x008aa5 thru 0x00ca55 (0x3fb1 bytes) 000411 1781235493.677817 mapwrite 0x02b91e thru 0x02c75f (0xe42 bytes) 000412 1781235493.691643 trunc from 0x03c7c5 to 0x026268 (0x1655c bytes) 000413 1781235493.709375 trunc from 0x022588 to 0x026268 (0x3ce1 bytes) 000415 1781235493.714849 trunc from 0x026269 to 0x03d768 (0x17500 bytes) 000416 1781235493.724187 trunc from 0x03d769 to 0x02e88b (0xeedd bytes) 000417 1781235493.737311 write 0x01d850 thru 0x02a142 (0xc8f3 bytes) 000419 1781235493.742172 read 0x00ba65 thru 0x01993f (0xdedb bytes) 000420 1781235493.742904 punch from 0x00e198 to 0x011be1 (0x3a4a bytes) 000421 1781235493.746362 read 0x004ad3 thru 0x004bac (0xda bytes) 000422 1781235493.746854 trunc from 0x02e88c to 0x02741c (0x746f bytes) 000423 1781235493.762581 mapwrite 0x007c9e thru 0x00dce5 (0x6048 bytes) 000424 1781235493.777747 fallocate 0x025f1f thru 0x02b2b5 (0x5397 bytes) 000426 1781235493.783176 fallocate 0x001170 thru 0x00667f (0x5510 bytes) 000427 1781235493.788124 trunc from 0x0213b6 to 0x02741c (0x6067 bytes) 000428 1781235493.792501 fallocate 0x0339d2 thru 0x03ffff (0xc62e bytes) 000429 1781235493.796041 fallocate 0x01cc90 thru 0x01f17b (0x24ec bytes) 000430 1781235493.800074 punch from 0x006da2 to 0x012edf (0xc13e bytes) 000431 1781235493.803988 trunc from 0x0248e1 to 0x0250d1 (0x7f1 bytes) 000432 1781235493.807621 read 0x009f2c thru 0x00ca2f (0x2b04 bytes) 000433 1781235493.835067 trunc from 0x02b6b9 to 0x034a7e (0x93c6 bytes) 000435 1781235493.841139 mapwrite 0x020462 thru 0x02944d (0x8fec bytes) 000436 1781235493.866794 MIRROR_EXTEND to 2 mirrors 000438 1781235494.582345 trunc from 0x040000 to 0x023755 (0x1c8aa bytes) 000439 1781235494.681753 write 0x014349 thru 0x01b8cb (0x7583 bytes) 000440 1781235494.711638 trunc from 0x023756 to 0x006ee1 (0x1c874 bytes) 000441 1781235494.760850 punch from 0x001aef to 0x006875 (0x4d87 bytes) 000442 1781235494.787063 fallocate 0x008877 thru 0x00f4ef (0x6c79 bytes) 000443 1781235494.803054 read 0x00468c thru 0x00d4a6 (0x8e1b bytes) 000444 1781235494.833050 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) 000445 1781235495.012630 write 0x00c434 thru 0x00f0f9 (0x2cc6 bytes) 000447 1781235495.034715 read 0x000db6 thru 0x00be79 (0xb0c4 bytes) 000448 1781235495.047855 punch from 0x007be3 to 0x00f4ef (0x790d bytes) 000449 1781235495.062772 mapwrite 0x03c906 thru 0x03ffff (0x36fa bytes) 000452 1781235495.114017 mapread 0x005f2e thru 0x00615a (0x22d bytes) 000453 1781235495.116186 mapread 0x000df1 thru 0x0052d8 (0x44e8 bytes) 000455 1781235495.119458 trunc from 0x036af4 to 0x03c18c (0x5699 bytes) 000456 1781235495.126087 read 0x03a470 thru 0x03cfc4 (0x2b55 bytes) 000457 1781235495.144868 mapread 0x0332fc thru 0x0373d8 (0x40dd bytes) 000459 1781235495.147516 fallocate 0x00c879 thru 0x00fc30 (0x33b8 bytes) 000460 1781235495.157122 read 0x00630c thru 0x013b82 (0xd877 bytes) 000462 1781235495.158749 punch from 0x00021e to 0x004afc (0x48df bytes) 000465 1781235495.168137 mapread 0x03207f thru 0x03b9a0 (0x9922 bytes) 000466 1781235495.171872 mapread 0x015ada thru 0x023b50 (0xe077 bytes) 000467 1781235495.174021 punch from 0x0028b8 to 0x00cfa4 (0xa6ed bytes) 000468 1781235495.182566 read 0x0298b5 thru 0x02da33 (0x417f bytes) 000469 1781235495.190723 write 0x025125 thru 0x02d02e (0x7f0a bytes) 000470 1781235495.198660 mapwrite 0x0088aa thru 0x00915e (0x8b5 bytes) 000471 1781235495.242035 trunc from 0x040000 to 0x025bab (0x1a454 bytes) 000472 1781235495.278996 trunc from 0x025bac to 0x02adf7 (0x524c bytes) 000474 1781235495.320965 punch from 0x016758 to 0x0223ce (0xbc77 bytes) 000475 1781235495.342742 MIRROR_EXTEND to 2 mirrors 000476 1781235496.415663 fallocate 0x024f6d thru 0x02bf1c (0x6fb0 bytes) 000478 1781235496.483124 punch from 0x02712c to 0x02bf1c (0x4df1 bytes) 000479 1781235496.493458 write 0x03c688 thru 0x03ee87 (0x2800 bytes) 000481 1781235496.503633 mapwrite 0x004429 thru 0x008283 (0x3e5b bytes) 000482 1781235496.581481 fallocate 0x039f4e thru 0x03ffff (0x60b2 bytes) 000483 1781235496.589632 trunc from 0x02b85f to 0x03a313 (0xeab5 bytes) 000484 1781235496.602841 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) 000486 1781235496.755530 mapwrite 0x025689 thru 0x02cb9e (0x7516 bytes) 000487 1781235496.795656 write 0x015d22 thru 0x022bb8 (0xce97 bytes) 000488 1781235496.810858 write 0x01edaf thru 0x02df52 (0xf1a4 bytes) 000489 1781235496.812945 trunc from 0x00266b to 0x00a16f (0x7b05 bytes) 000490 1781235496.817817 mapwrite 0x02e2ea thru 0x032d4d (0x4a64 bytes) 000492 1781235496.847369 read 0x0096ac thru 0x012cd9 (0x962e bytes) 000493 1781235496.854746 fallocate 0x0221b7 thru 0x03139a (0xf1e4 bytes) 000494 1781235496.859542 fallocate 0x007d81 thru 0x0141b1 (0xc431 bytes) 000496 1781235496.865479 mapwrite 0x01742d thru 0x0179e6 (0x5ba bytes) 000498 1781235496.879102 read 0x011c39 thru 0x016020 (0x43e8 bytes) 000499 1781235496.880695 write 0x02849c thru 0x030bfc (0x8761 bytes) 000500 1781235496.882135 punch from 0x020472 to 0x026ad0 (0x665f bytes) 000502 1781235496.889669 trunc from 0x040000 to 0x02f1f6 (0x10e09 bytes) 000503 1781235496.904911 mapwrite 0x00f0aa thru 0x016c45 (0x7b9c bytes) 000505 1781235496.925014 write 0x039f46 thru 0x03b2fd (0x13b8 bytes) 000506 1781235496.926129 mapread 0x036278 thru 0x03b2fd (0x5086 bytes) 000507 1781235496.936612 punch from 0x019c04 to 0x01ff3a (0x6337 bytes) 000509 1781235496.944628 mapread 0x01c5ef thru 0x0252cc (0x8cde bytes) 000511 1781235496.963328 write 0x03b017 thru 0x03e150 (0x313a bytes) 000512 1781235496.964486 mapwrite 0x00df0f thru 0x01a96b (0xca5d bytes) 000513 1781235496.993980 trunc from 0x03e151 to 0x008193 (0x35fbd bytes) 000514 1781235497.011107 trunc from 0x00329f to 0x008193 (0x4ef5 bytes) 000515 1781235497.020016 trunc from 0x008194 to 0x0308bf (0x2872c bytes) 000517 1781235497.038450 trunc from 0x0308c0 to 0x024d49 (0xbb76 bytes) 000518 1781235497.056908 trunc from 0x024d4a to 0x004261 (0x20ae8 bytes) 000519 1781235497.072458 write 0x00b226 thru 0x011dba (0x6b95 bytes) 000520 1781235497.074898 trunc from 0x00e2b7 to 0x011dba (0x3b04 bytes) 000521 1781235497.093606 mapread 0x00ea61 thru 0x011dba (0x335a bytes) 000522 1781235497.111944 write 0x0127f5 thru 0x01fc34 (0xd440 bytes) 000523 1781235497.114069 trunc from 0x005262 to 0x0117b9 (0xc558 bytes) 000525 1781235497.120179 write 0x038a96 thru 0x03ffff (0x756a bytes) 000526 1781235497.122515 read 0x0261ed thru 0x02dd7d (0x7b91 bytes) 000527 1781235497.139600 mapread 0x01587c thru 0x01c17a (0x68ff bytes) 000528 1781235497.140912 trunc from 0x01ed84 to 0x02e0fb (0xf378 bytes) 000529 1781235497.157674 trunc from 0x017686 to 0x027616 (0xff91 bytes) 000530 1781235497.162831 trunc from 0x01fb5f to 0x02a294 (0xa736 bytes) 000531 1781235497.167602 punch from 0x02986b to 0x038547 (0xecdd bytes) 000532 1781235497.172908 mapread 0x031813 thru 0x03e8a1 (0xd08f bytes) 000533 1781235497.207722 punch from 0x006e86 to 0x00fca1 (0x8e1c bytes) 000535 1781235497.213175 trunc from 0x040000 to 0x01a2f9 (0x25d06 bytes) 000536 1781235497.226026 punch from 0x004466 to 0x00b7ef (0x738a bytes) 000537 1781235497.230880 mapwrite 0x01affd thru 0x028c9f (0xdca3 bytes) 000538 1781235497.327115 trunc from 0x028ca0 to 0x000d57 (0x27f48 bytes) 000539 1781235497.348695 trunc from 0x0003a3 to 0x000d57 (0x9b5 bytes) 000541 1781235497.355325 write 0x02da69 thru 0x03973e (0xbcd6 bytes) 000542 1781235497.358990 trunc from 0x03973f to 0x02770a (0x12034 bytes) 000543 1781235497.377416 read 0x000e00 thru 0x007448 (0x6649 bytes) 000544 1781235497.403244 mapwrite 0x02d483 thru 0x035f63 (0x8ae1 bytes) 000545 1781235497.447427 mapwrite 0x00228d thru 0x003a38 (0x17ac bytes) 000547 1781235497.459993 fallocate 0x00bbfe thru 0x0179a0 (0xbda3 bytes) 000549 1781235497.465995 fallocate 0x0224a1 thru 0x031c86 (0xf7e6 bytes) 000550 1781235497.478773 mapwrite 0x003509 thru 0x0112c7 (0xddbf bytes) 000551 1781235497.504823 punch from 0x0197c4 to 0x0228b9 (0x90f6 bytes) 000552 1781235497.510555 trunc from 0x025a46 to 0x0269c3 (0xf7e bytes) 000554 1781235497.516043 mapread 0x0282a4 thru 0x02a90b (0x2668 bytes) 000555 1781235497.517173 fallocate 0x005b74 thru 0x00e3d9 (0x8866 bytes) 000556 1781235497.521714 trunc from 0x009dcb to 0x00c17a (0x23b0 bytes) 000557 1781235497.526921 read 0x002b4b thru 0x00555f (0x2a15 bytes) 000558 1781235497.527772 read 0x006276 thru 0x00a8d4 (0x465f bytes) 000559 1781235497.540921 trunc from 0x035f64 to 0x03f8d5 (0x9972 bytes) 000560 1781235497.556562 trunc from 0x01f79e to 0x02aa50 (0xb2b3 bytes) 000561 1781235497.562492 mapread 0x001678 thru 0x004fce (0x3957 bytes) 000562 1781235497.564059 write 0x009f04 thru 0x017cc8 (0xddc5 bytes) 000563 1781235497.566458 punch from 0x02228d to 0x023180 (0xef4 bytes) 000564 1781235497.570924 mapwrite 0x03f55c thru 0x03ffff (0xaa4 bytes) 000565 1781235497.609859 trunc from 0x040000 to 0x008192 (0x37e6d bytes) 000566 1781235497.627833 fallocate 0x020d39 thru 0x0212e0 (0x5a8 bytes) 000567 1781235497.638563 trunc from 0x001eac to 0x01054a (0xe69f bytes) 000568 1781235497.644882 mapwrite 0x02a9d0 thru 0x0360f3 (0xb724 bytes) 000569 1781235497.736790 fallocate 0x007ccd thru 0x01148d (0x97c1 bytes) 000570 1781235497.742053 punch from 0x010214 to 0x011e87 (0x1c74 bytes) 000571 1781235497.746517 trunc from 0x02e226 to 0x0356ae (0x7489 bytes) 000572 1781235497.755942 mapwrite 0x02e07d thru 0x02f96a (0x18ee bytes) 000577 1781235497.781746 trunc from 0x0360f4 to 0x03bef8 (0x5e05 bytes) 000580 1781235497.804371 trunc from 0x02ce72 to 0x0366ca (0x9859 bytes) 000581 1781235497.811083 mapread 0x00d641 thru 0x01405b (0x6a1b bytes) 000583 1781235497.848705 read 0x029485 thru 0x02d1d3 (0x3d4f bytes) 000584 1781235497.859131 punch from 0x037f9a to 0x03bef8 (0x3f5f bytes) 000585 1781235497.864480 fallocate 0x014bc1 thru 0x0202ff (0xb73f bytes) 000586 1781235497.871033 fallocate 0x03c6b6 thru 0x03f1b0 (0x2afb bytes) 000587 1781235497.880177 MIRROR_EXTEND to 2 mirrors 000588 1781235499.013872 mapwrite 0x011ca9 thru 0x02102f (0xf387 bytes) 000591 1781235499.155489 fallocate 0x00a343 thru 0x00d491 (0x314f bytes) 000592 1781235499.165006 write 0x00a36b thru 0x013c87 (0x991d bytes) 000593 1781235499.195700 fallocate 0x017940 thru 0x022df0 (0xb4b1 bytes) 000594 1781235499.204151 punch from 0x010b74 to 0x01eeba (0xe347 bytes) 000595 1781235499.219723 read 0x03b995 thru 0x03bef8 (0x564 bytes) 000596 1781235499.225243 trunc from 0x03bef9 to 0x03a564 (0x1994 bytes) 000597 1781235499.243305 mapwrite 0x03336c thru 0x03ac09 (0x789e bytes) 000598 1781235499.304650 fallocate 0x0199e4 thru 0x01f331 (0x594e bytes) 000599 1781235499.311049 mapwrite 0x027e47 thru 0x028d00 (0xeba bytes) 000600 1781235499.343174 trunc from 0x000eee to 0x008aa6 (0x7bb9 bytes) 000601 1781235499.349227 mapread 0x00c782 thru 0x01219a (0x5a19 bytes) 000602 1781235499.362460 trunc from 0x03ac0a to 0x00e345 (0x2c8c4 bytes) 000603 1781235499.388128 read 0x007983 thru 0x00e345 (0x69c3 bytes) 000605 1781235499.399875 write 0x027a6e thru 0x03745e (0xf9f1 bytes) 000606 1781235499.402615 mapread 0x007886 thru 0x016209 (0xe984 bytes) 000607 1781235499.415870 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x3745f): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3745f): Mirror 2: 0x32f45dd7 (stale) CRC-32 checksum value for chunk [0, 0x3745f): Mirror 1: 0x93171f07 Mirror 3: 0x93171f07 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3745f: skipped 000608 1781235500.105842 trunc from 0x02679b to 0x02743e (0xca4 bytes) 000609 1781235500.142868 trunc from 0x017e52 to 0x025e19 (0xdfc8 bytes) 000610 1781235500.171958 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3745f): Mirror 3: 0x93171f07 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3745f): Mirror 3: 0x93171f07 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3745f: skipped 000611 1781235500.434509 mapread 0x0026af thru 0x008bb2 (0x6504 bytes) 000612 1781235500.452233 trunc from 0x03745f to 0x039cba (0x285c bytes) 000613 1781235500.490351 trunc from 0x01b3e8 to 0x0254b2 (0xa0cb bytes) 000614 1781235500.495504 read 0x0128b5 thru 0x01ba91 (0x91dd bytes) 000615 1781235500.509652 punch from 0x00a966 to 0x013308 (0x89a3 bytes) 000616 1781235500.514977 punch from 0x01a7cd to 0x022ba2 (0x83d6 bytes) 000617 1781235500.519997 read 0x033660 thru 0x039cba (0x665b bytes) 000618 1781235500.532274 write 0x02858c thru 0x02b2a5 (0x2d1a bytes) 000619 1781235500.533599 mapread 0x00e361 thru 0x01264c (0x42ec bytes) 000621 1781235500.536681 mapread 0x037711 thru 0x037a11 (0x301 bytes) 000622 1781235500.537752 punch from 0x0135cb to 0x019360 (0x5d96 bytes) 000623 1781235500.542783 mapwrite 0x01ffcb thru 0x0214e3 (0x1519 bytes) 000625 1781235500.559683 mapwrite 0x02f730 thru 0x037436 (0x7d07 bytes) 000626 1781235500.577456 mapread 0x035f30 thru 0x039cba (0x3d8b bytes) 000627 1781235500.578868 read 0x037e64 thru 0x038b71 (0xd0e bytes) 000629 1781235500.581038 trunc from 0x01486f to 0x0241ed (0xf97f bytes) 000630 1781235500.585887 write 0x0043da thru 0x011f1b (0xdb42 bytes) 000631 1781235500.591762 trunc from 0x02577e to 0x029c50 (0x44d3 bytes) 000632 1781235500.600952 MIRROR_RESYNC 2 mirrors 000633 1781235501.184839 mapwrite 0x012a3e thru 0x0130bc (0x67f bytes) 000634 1781235501.261678 mapread 0x036561 thru 0x039cba (0x375a bytes) 000635 1781235501.272283 write 0x0172f0 thru 0x0253a5 (0xe0b6 bytes) 000636 1781235501.277976 trunc from 0x039cbb to 0x032685 (0x7635 bytes) 000637 1781235501.320131 trunc from 0x029219 to 0x02d9cd (0x47b5 bytes) 000639 1781235501.329579 mapread 0x02a80a thru 0x032685 (0x7e7c bytes) 000640 1781235501.339439 mapread 0x022de8 thru 0x02d6f1 (0xa90a bytes) 000641 1781235501.340750 trunc from 0x032686 to 0x027297 (0xb3ee bytes) 000642 1781235501.356089 trunc from 0x0246f2 to 0x027297 (0x2ba6 bytes) 000643 1781235501.376860 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) 000644 1781235501.508886 trunc from 0x01d059 to 0x027297 (0xa23f bytes) 000646 1781235501.515549 mapwrite 0x022bd4 thru 0x02a8c4 (0x7cf1 bytes) 000647 1781235501.579529 fallocate 0x02dfa2 thru 0x034cd9 (0x6d38 bytes) 000648 1781235501.584533 mapwrite 0x023769 thru 0x0298e3 (0x617b bytes) 000649 1781235501.604275 mapwrite 0x01a17a thru 0x0246ca (0xa551 bytes) 000650 1781235501.628813 trunc from 0x0243c8 to 0x02a8c4 (0x64fd bytes) 000651 1781235501.634010 write 0x00de2f thru 0x01d334 (0xf506 bytes) 000652 1781235501.643114 read 0x00ecb0 thru 0x0174c4 (0x8815 bytes) 000653 1781235501.643923 read 0x011ced thru 0x0191e8 (0x74fc bytes) 000654 1781235501.645486 trunc from 0x01ac1e to 0x01bcf2 (0x10d5 bytes) 000655 1781235501.659665 fallocate 0x0349ae thru 0x03ffff (0xb652 bytes) 000656 1781235501.664510 trunc from 0x040000 to 0x01c6e7 (0x23918 bytes) 000657 1781235501.680560 mapread 0x019fa4 thru 0x01bc03 (0x1c60 bytes) 000658 1781235501.692516 trunc from 0x009c42 to 0x017826 (0xdbe5 bytes) 000659 1781235501.697920 fallocate 0x034990 thru 0x037bbe (0x322f bytes) 000660 1781235501.706852 read 0x034590 thru 0x0349d6 (0x447 bytes) 000661 1781235501.719712 trunc from 0x02cf5a to 0x037bbe (0xac65 bytes) 000662 1781235501.724545 punch from 0x028ae9 to 0x02b6d9 (0x2bf1 bytes) 000664 1781235501.730757 mapwrite 0x0328d9 thru 0x039c4c (0x7374 bytes) 000666 1781235501.805666 punch from 0x017445 to 0x0270c5 (0xfc81 bytes) 000667 1781235501.812776 write 0x0041f1 thru 0x00e752 (0xa562 bytes) 000668 1781235501.815065 mapwrite 0x034f18 thru 0x03ffff (0xb0e8 bytes) 000669 1781235501.877750 trunc from 0x0104ab to 0x01a66e (0xa1c4 bytes) 000670 1781235501.883258 write 0x0021c3 thru 0x00a5b7 (0x83f5 bytes) 000671 1781235501.884866 fallocate 0x02a8f0 thru 0x030673 (0x5d84 bytes) 000672 1781235501.890363 read 0x03d8d8 thru 0x03ffff (0x2728 bytes) 000673 1781235501.891452 punch from 0x02976f to 0x03566a (0xbefc bytes) 000674 1781235501.896927 trunc from 0x02b3b7 to 0x036e05 (0xba4f bytes) 000676 1781235501.903814 punch from 0x006ad7 to 0x008fbe (0x24e8 bytes) 000677 1781235501.917079 trunc from 0x040000 to 0x0066bd (0x39942 bytes) 000678 1781235501.935064 mapwrite 0x00da5f thru 0x01049f (0x2a41 bytes) 000680 1781235501.977579 mapwrite 0x035a84 thru 0x03d97f (0x7efc bytes) 000681 1781235502.019715 punch from 0x0315e7 to 0x037bd5 (0x65ef bytes) 000682 1781235502.026486 mapwrite 0x03ce6f thru 0x03ffff (0x3191 bytes) 000683 1781235502.059049 trunc from 0x040000 to 0x0087be (0x37841 bytes) 000684 1781235502.075117 punch from 0x0013ed to 0x00234a (0xf5e bytes) 000685 1781235502.080505 write 0x01ba30 thru 0x02423d (0x880e bytes) 000686 1781235502.084657 mapwrite 0x00e0d0 thru 0x017835 (0x9766 bytes) 000687 1781235502.156421 punch from 0x01e90c to 0x02423d (0x5932 bytes) 000688 1781235502.162742 write 0x03b6bf thru 0x03ec44 (0x3586 bytes) 000689 1781235502.164847 punch from 0x02bf77 to 0x0385eb (0xc675 bytes) 000690 1781235502.169365 mapwrite 0x03d40f thru 0x03ffff (0x2bf1 bytes) 000691 1781235502.209562 trunc from 0x040000 to 0x01091d (0x2f6e2 bytes) 000692 1781235502.224850 trunc from 0x01091e to 0x02928b (0x1896e bytes) 000693 1781235502.249817 write 0x0161bd thru 0x017064 (0xea8 bytes) 000694 1781235502.260100 fallocate 0x010f3c thru 0x01cbed (0xbcb2 bytes) 000695 1781235502.264908 trunc from 0x01cb38 to 0x01e1fc (0x16c5 bytes) 000696 1781235502.273956 trunc from 0x02928c to 0x01ab6f (0xe71c bytes) 000697 1781235502.288112 mapwrite 0x0205e3 thru 0x02992b (0x9349 bytes) 000698 1781235502.376028 trunc from 0x024aa7 to 0x028db3 (0x430d bytes) 000699 1781235502.382076 read 0x0208ac thru 0x02992b (0x9080 bytes) 000701 1781235502.402486 fallocate 0x0258e6 thru 0x034c1e (0xf339 bytes) 000702 1781235502.410101 trunc from 0x02992c to 0x020a39 (0x8ef2 bytes) 000703 1781235502.429860 mapwrite 0x03bf29 thru 0x03e4b8 (0x2590 bytes) 000704 1781235502.480313 trunc from 0x0092de to 0x013481 (0xa1a4 bytes) 000705 1781235502.486922 mapwrite 0x0318c7 thru 0x03c2d9 (0xaa13 bytes) 000707 1781235502.509962 mapread 0x030c2c thru 0x03d841 (0xcc16 bytes) 000708 1781235502.511396 punch from 0x01b8c8 to 0x01ec54 (0x338d bytes) 000709 1781235502.516445 fallocate 0x012b81 thru 0x021b4e (0xefce bytes) 000710 1781235502.521649 trunc from 0x017eea to 0x01be96 (0x3fad bytes) 000711 1781235502.529858 trunc from 0x022439 to 0x0306be (0xe286 bytes) 000712 1781235502.535591 write 0x03927a thru 0x03ffff (0x6d86 bytes) 000713 1781235502.539965 punch from 0x00d6b2 to 0x010dac (0x36fb bytes) 000715 1781235502.545918 write 0x020a09 thru 0x021e0d (0x1405 bytes) 000716 1781235502.551012 trunc from 0x040000 to 0x0165a9 (0x29a56 bytes) 000717 1781235502.573787 mapwrite 0x034691 thru 0x03d3c0 (0x8d30 bytes) 000718 1781235502.664484 punch from 0x020ca5 to 0x0263f7 (0x5753 bytes) 000719 1781235502.669596 trunc from 0x014c16 to 0x018349 (0x3734 bytes) 000720 1781235502.675004 punch from 0x032546 to 0x03d3c0 (0xae7b bytes) 000721 1781235502.682992 mapwrite 0x02b458 thru 0x03700d (0xbbb6 bytes) 000722 1781235502.767937 write 0x03d4fc thru 0x03ffa8 (0x2aad bytes) 000723 1781235502.773978 fallocate 0x00b556 thru 0x010432 (0x4edd bytes) 000724 1781235502.779782 write 0x03aa37 thru 0x03e7d4 (0x3d9e bytes) 000726 1781235502.791026 read 0x01f4e4 thru 0x02cff2 (0xdb0f bytes) 000727 1781235502.815962 mapwrite 0x012a90 thru 0x01920b (0x677c bytes) 000729 1781235502.835712 punch from 0x007a21 to 0x0111a8 (0x9788 bytes) 000730 1781235502.840835 mapread 0x027be0 thru 0x037120 (0xf541 bytes) 000731 1781235502.843002 punch from 0x026ed0 to 0x032a40 (0xbb71 bytes) 000732 1781235502.848765 mapread 0x03bbf4 thru 0x03fa42 (0x3e4f bytes) 000733 1781235502.849836 write 0x01dec1 thru 0x02c2cc (0xe40c bytes) 000734 1781235502.856353 trunc from 0x02af5e to 0x030a60 (0x5b03 bytes) 000735 1781235502.870649 trunc from 0x03ffa9 to 0x01ed9e (0x2120a bytes) 000736 1781235502.889988 mapread 0x01ea71 thru 0x01ed9e (0x32e bytes) 000737 1781235502.891239 read 0x01d7d0 thru 0x01ed9e (0x15cf bytes) 000738 1781235502.892972 fallocate 0x002dff thru 0x00e3e6 (0xb5e8 bytes) 000739 1781235502.899132 trunc from 0x0142a5 to 0x01ed9e (0xaafa bytes) 000740 1781235502.906374 mapwrite 0x00c77d thru 0x014ac2 (0x8346 bytes) 000742 1781235502.937476 read 0x00c44b thru 0x015558 (0x910e bytes) 000743 1781235502.938184 trunc from 0x0053b0 to 0x00ce80 (0x7ad1 bytes) 000744 1781235502.944898 trunc from 0x01ed9f to 0x011b17 (0xd287 bytes) 000746 1781235502.967713 read 0x0060f6 thru 0x0077e3 (0x16ee bytes) 000747 1781235502.977742 trunc from 0x00aaf4 to 0x011b17 (0x7024 bytes) 000748 1781235502.983677 mapwrite 0x0251eb thru 0x02d7de (0x85f4 bytes) 000749 1781235503.033075 mapread 0x0216b9 thru 0x02d7de (0xc126 bytes) 000750 1781235503.035730 fallocate 0x02221d thru 0x02857b (0x635f bytes) 000751 1781235503.042064 trunc from 0x02d7df to 0x03e5e0 (0x10e02 bytes) 000752 1781235503.063542 mapwrite 0x028caf thru 0x0325b2 (0x9904 bytes) 000753 1781235503.131659 mapwrite 0x03cc2b thru 0x03e32e (0x1704 bytes) 000754 1781235503.153454 mapwrite 0x02a9a2 thru 0x03747c (0xcadb bytes) 000756 1781235503.180626 mapread 0x00ba32 thru 0x00dc8d (0x225c bytes) 000757 1781235503.198031 fallocate 0x03bd8e thru 0x03ffff (0x4272 bytes) 000758 1781235503.204780 read 0x006ace thru 0x009c4e (0x3181 bytes) 000759 1781235503.207844 fallocate 0x0065fc thru 0x012cb9 (0xc6be bytes) 000760 1781235503.213886 trunc from 0x03b802 to 0x03ffff (0x47fe bytes) 000761 1781235503.218832 mapread 0x00141c thru 0x009edd (0x8ac2 bytes) 000762 1781235503.221762 trunc from 0x01ff71 to 0x02e105 (0xe195 bytes) 000763 1781235503.229566 mapread 0x036022 thru 0x03bc10 (0x5bef bytes) 000764 1781235503.242547 write 0x0151d5 thru 0x01a642 (0x546e bytes) 000765 1781235503.244731 trunc from 0x007261 to 0x008c47 (0x19e7 bytes) 000766 1781235503.253806 read 0x025fde thru 0x0308ff (0xa922 bytes) 000768 1781235503.255773 mapread 0x011f0d thru 0x016af7 (0x4beb bytes) 000769 1781235503.257268 mapread 0x0317e0 thru 0x03bf71 (0xa792 bytes) 000770 1781235503.259657 trunc from 0x040000 to 0x03cd93 (0x326c bytes) 000771 1781235503.275015 write 0x03bc5c thru 0x03ffff (0x43a4 bytes) 000772 1781235503.277397 write 0x034780 thru 0x03f695 (0xaf16 bytes) 000773 1781235503.280628 read 0x013220 thru 0x01f4e6 (0xc2c7 bytes) 000775 1781235503.282978 trunc from 0x040000 to 0x037267 (0x8d98 bytes) 000778 1781235503.300701 mapwrite 0x03464c thru 0x03eab7 (0xa46c bytes) 000779 1781235503.352525 punch from 0x00b6d9 to 0x011694 (0x5fbc bytes) 000780 1781235503.359474 trunc from 0x03eab8 to 0x025d2e (0x18d89 bytes) 000781 1781235503.375850 mapread 0x01525d thru 0x01b35e (0x6102 bytes) 000782 1781235503.377994 trunc from 0x025d2f to 0x01006b (0x15cc3 bytes) 000783 1781235503.398622 trunc from 0x00c80d to 0x01006b (0x385f bytes) 000784 1781235503.403866 trunc from 0x009ec9 to 0x01006b (0x61a3 bytes) 000785 1781235503.409746 trunc from 0x00507f to 0x00728f (0x2211 bytes) 000786 1781235503.417095 write 0x01eb49 thru 0x023a96 (0x4f4e bytes) 000787 1781235503.418478 mapwrite 0x0155b1 thru 0x015a01 (0x451 bytes) 000788 1781235503.444060 mapread 0x0146d4 thru 0x015c36 (0x1563 bytes) 000789 1781235503.450851 trunc from 0x00cc7e to 0x01b0e6 (0xe469 bytes) 000790 1781235503.458709 write 0x029b0e thru 0x02bca0 (0x2193 bytes) 000792 1781235503.462060 mapread 0x00f5df thru 0x015410 (0x5e32 bytes) 000793 1781235503.497692 trunc from 0x02013e to 0x026a6f (0x6932 bytes) 000794 1781235503.502811 mapread 0x01105d thru 0x0167c7 (0x576b bytes) 000795 1781235503.511599 fallocate 0x01d882 thru 0x023352 (0x5ad1 bytes) 000796 1781235503.521801 MIRROR_EXTEND to 2 mirrors 000797 1781235504.454614 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) 000798 1781235504.619615 trunc from 0x0135f1 to 0x014cbf (0x16cf bytes) 000799 1781235504.637615 trunc from 0x0001d4 to 0x001f37 (0x1d64 bytes) 000800 1781235504.643063 fallocate 0x03ae98 thru 0x03ffff (0x5168 bytes) 000801 1781235504.648965 mapwrite 0x01790e thru 0x01b74a (0x3e3d bytes) 000802 1781235504.688594 trunc from 0x02bca1 to 0x02bb4b (0x155 bytes) 000803 1781235504.716890 mapread 0x0165fd thru 0x01e0ca (0x7ace bytes) 000805 1781235504.722646 fallocate 0x00fdb9 thru 0x016527 (0x676f bytes) 000806 1781235504.728535 trunc from 0x01b861 to 0x023c61 (0x8401 bytes) 000807 1781235504.735736 trunc from 0x02bb4c to 0x02f769 (0x3c1e bytes) 000808 1781235504.762010 punch from 0x00f01b to 0x01b086 (0xc06c bytes) 000809 1781235504.769686 punch from 0x02cf74 to 0x02f769 (0x27f6 bytes) 000810 1781235504.775537 mapread 0x02c142 thru 0x02f769 (0x3628 bytes) 000811 1781235504.786932 write 0x01d617 thru 0x01ef36 (0x1920 bytes) 000812 1781235504.807663 MIRROR_EXTEND to 2 mirrors 000813 1781235505.569743 punch from 0x01a54f to 0x020972 (0x6424 bytes) 000814 1781235505.608091 read 0x0105f4 thru 0x01aa86 (0xa493 bytes) 000815 1781235505.625503 fallocate 0x03a16d thru 0x03c01f (0x1eb3 bytes) 000816 1781235505.635241 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) 000817 1781235505.794488 read 0x0372b5 thru 0x03a2bc (0x3008 bytes) 000818 1781235505.800391 mapwrite 0x02f537 thru 0x0359d1 (0x649b bytes) 000819 1781235505.869079 fallocate 0x034f34 thru 0x03ffff (0xb0cc bytes) 000820 1781235505.887260 mapwrite 0x02e623 thru 0x02f69e (0x107c bytes) 000821 1781235505.927478 fallocate 0x03f473 thru 0x03ffff (0xb8d bytes) 000822 1781235505.935167 mapwrite 0x01acbc thru 0x022881 (0x7bc6 bytes) 000824 1781235505.972081 fallocate 0x028766 thru 0x033f0f (0xb7aa bytes) 000825 1781235505.978886 trunc from 0x03c020 to 0x02ae3c (0x111e3 bytes) 000826 1781235506.024585 fallocate 0x0221cb thru 0x02ad44 (0x8b7a bytes) 000828 1781235506.044837 fallocate 0x03b294 thru 0x03ffff (0x4d6c bytes) 000829 1781235506.050531 mapread 0x03c001 thru 0x03ffff (0x3fff bytes) 000830 1781235506.067814 trunc from 0x0156cc to 0x018321 (0x2c56 bytes) 000831 1781235506.086894 trunc from 0x011413 to 0x019e4d (0x8a3b bytes) 000833 1781235506.108604 write 0x01bbb8 thru 0x01c67a (0xac3 bytes) 000834 1781235506.109850 fallocate 0x039bf1 thru 0x03ffff (0x640f bytes) 000835 1781235506.121471 trunc from 0x03d5bc to 0x03ffff (0x2a44 bytes) 000837 1781235506.134463 trunc from 0x02552c to 0x028222 (0x2cf7 bytes) 000838 1781235506.147443 write 0x01cf6f thru 0x027664 (0xa6f6 bytes) 000839 1781235506.161595 trunc from 0x01b326 to 0x028e40 (0xdb1b bytes) 000840 1781235506.190949 trunc from 0x02a9a8 to 0x037418 (0xca71 bytes) 000841 1781235506.206367 mapread 0x020ffd thru 0x024c40 (0x3c44 bytes) 000842 1781235506.226693 MIRROR_EXTEND to 2 mirrors 000844 1781235507.470067 mapread 0x032bb6 thru 0x033e3d (0x1288 bytes) 000845 1781235507.481925 write 0x03770b thru 0x03ffff (0x88f5 bytes) 000846 1781235507.527589 trunc from 0x03424d to 0x03ffff (0xbdb3 bytes) 000847 1781235507.543081 mapwrite 0x03a24c thru 0x03ffff (0x5db4 bytes) 000848 1781235507.581881 fallocate 0x00a5ef thru 0x0158ad (0xb2bf bytes) 000850 1781235507.591479 read 0x0273f0 thru 0x034230 (0xce41 bytes) 000851 1781235507.609727 write 0x028952 thru 0x02efec (0x669b bytes) 000852 1781235507.618939 punch from 0x00f11e to 0x01b006 (0xbee9 bytes) 000853 1781235507.627867 read 0x01e85f thru 0x025d28 (0x74ca bytes) 000854 1781235507.657038 mapwrite 0x032f6d thru 0x03ffff (0xd093 bytes) 000855 1781235507.716292 MIRROR_RESYNC 2 mirrors 000856 1781235508.477588 punch from 0x01bc8c to 0x02582a (0x9b9f bytes) 000859 1781235508.528596 mapwrite 0x027149 thru 0x035fad (0xee65 bytes) 000860 1781235508.607162 MIRROR_RESYNC 2 mirrors 000861 1781235509.500904 trunc from 0x040000 to 0x015c36 (0x2a3c9 bytes) 000862 1781235509.534909 punch from 0x0005fa to 0x00c9f1 (0xc3f8 bytes) 000863 1781235509.550690 MIRROR_RESYNC 2 mirrors 000864 1781235510.066973 trunc from 0x004e9a to 0x00a6bd (0x5824 bytes) 000865 1781235510.109992 trunc from 0x015c37 to 0x005aa5 (0x10191 bytes) 000866 1781235510.134372 punch from 0x000229 to 0x004a39 (0x4811 bytes) 000867 1781235510.141690 mapread 0x0041b1 thru 0x005aa5 (0x18f5 bytes) 000868 1781235510.153020 fallocate 0x015637 thru 0x0157df (0x1a9 bytes) 000869 1781235510.159121 trunc from 0x0157e0 to 0x03fb83 (0x2a3a4 bytes) 000870 1781235510.178186 mapwrite 0x03f19a thru 0x03ffff (0xe66 bytes) 000871 1781235510.228998 fallocate 0x03635f thru 0x037813 (0x14b5 bytes) 000872 1781235510.237705 write 0x02c257 thru 0x034d5d (0x8b07 bytes) 000873 1781235510.252435 write 0x02ac29 thru 0x03789b (0xcc73 bytes) 000874 1781235510.258651 read 0x017444 thru 0x019798 (0x2355 bytes) 000876 1781235510.264553 trunc from 0x008aff to 0x00c80c (0x3d0e bytes) 000877 1781235510.271318 read 0x03ead3 thru 0x03ffff (0x152d bytes) 000878 1781235510.272009 write 0x027d88 thru 0x02b300 (0x3579 bytes) 000880 1781235510.275611 mapwrite 0x01d6fc thru 0x027bd0 (0xa4d5 bytes) 000881 1781235510.344146 trunc from 0x040000 to 0x026537 (0x19ac8 bytes) 000882 1781235510.371312 trunc from 0x026538 to 0x02a827 (0x42f0 bytes) 000883 1781235510.397674 trunc from 0x02a828 to 0x014bda (0x15c4d bytes) 000885 1781235510.417873 read 0x005997 thru 0x00999b (0x4005 bytes) 000887 1781235510.419223 write 0x0227be thru 0x0258fb (0x313e bytes) 000888 1781235510.426708 fallocate 0x01f387 thru 0x02f035 (0xfcaf bytes) 000889 1781235510.434774 trunc from 0x002951 to 0x00f5bb (0xcc6b bytes) 000890 1781235510.448767 trunc from 0x0258fc to 0x0027fb (0x23100 bytes) 000891 1781235510.474668 mapwrite 0x02cd09 thru 0x03b5b4 (0xe8ac bytes) 000892 1781235510.565465 write 0x02ef8a thru 0x03c134 (0xd1ab bytes) 000893 1781235510.567070 trunc from 0x0078fe to 0x010bd8 (0x92db bytes) 000894 1781235510.575129 punch from 0x02f313 to 0x03be07 (0xcaf5 bytes) 000895 1781235510.611785 trunc from 0x00dfdd to 0x01d6d7 (0xf6fb bytes) 000896 1781235510.625587 write 0x01fddf thru 0x0242b1 (0x44d3 bytes) 000897 1781235510.628910 fallocate 0x014e8d thru 0x019206 (0x437a bytes) 000898 1781235510.638136 mapwrite 0x01e3db thru 0x024d0c (0x6932 bytes) 000899 1781235510.667907 trunc from 0x03c135 to 0x03fcd4 (0x3ba0 bytes) 000900 1781235510.688018 fallocate 0x013929 thru 0x01850b (0x4be3 bytes) 000901 1781235510.697002 mapwrite 0x02427c thru 0x0249d0 (0x755 bytes) 000902 1781235510.718722 mapwrite 0x013c17 thru 0x018ebc (0x52a6 bytes) 000903 1781235510.797923 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) 000904 1781235510.914453 trunc from 0x03fcd5 to 0x02ae1b (0x14eb9 bytes) 000906 1781235510.933639 punch from 0x014ee0 to 0x020f18 (0xc039 bytes) 000907 1781235510.937640 trunc from 0x02ae1c to 0x011368 (0x19ab3 bytes) 000908 1781235510.954488 mapread 0x00f9ee thru 0x011368 (0x197b bytes) 000909 1781235510.970418 trunc from 0x000fa0 to 0x010bce (0xfc2f bytes) 000910 1781235510.977107 MIRROR_EXTEND to 2 mirrors 000912 1781235511.675833 punch from 0x0098ad to 0x00e5bc (0x4d10 bytes) 000913 1781235511.748758 trunc from 0x011369 to 0x01d445 (0xc0dd bytes) 000914 1781235511.787310 trunc from 0x009f9b to 0x00a193 (0x1f9 bytes) 000915 1781235511.794579 mapread 0x00af1e thru 0x00d346 (0x2429 bytes) 000916 1781235511.824243 fallocate 0x03357f thru 0x03a3a4 (0x6e26 bytes) 000917 1781235511.831448 trunc from 0x01d446 to 0x018d41 (0x4704 bytes) 000918 1781235511.855155 fallocate 0x015724 thru 0x01e903 (0x91e0 bytes) 000919 1781235511.862465 fallocate 0x01a2c4 thru 0x02341c (0x9159 bytes) 000920 1781235511.874009 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2341d): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2341d): Mirror 2: 0x53d4c092 (stale) CRC-32 checksum value for chunk [0, 0x2341d): Mirror 1: 0x53d4c092 Mirror 3: 0x53d4c092 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2341d: skipped 000921 1781235512.906369 trunc from 0x02341d to 0x0252df (0x1ec3 bytes) 000922 1781235512.996444 punch from 0x008e97 to 0x011a92 (0x8bfc bytes) 000923 1781235513.005882 trunc from 0x0212bc to 0x0218b9 (0x5fe bytes) 000924 1781235513.011573 fallocate 0x0285a4 thru 0x02bf6f (0x39cc bytes) 000925 1781235513.017747 trunc from 0x013353 to 0x018581 (0x522f bytes) 000926 1781235513.026908 trunc from 0x0252e0 to 0x01c668 (0x8c77 bytes) 000927 1781235513.065930 mapwrite 0x03232a thru 0x033462 (0x1139 bytes) 000928 1781235513.186326 write 0x000fc9 thru 0x009b10 (0x8b48 bytes) 000929 1781235513.203873 trunc from 0x03287b to 0x033462 (0xbe8 bytes) 000931 1781235513.216849 write 0x02d4e0 thru 0x03b07e (0xdb9f bytes) 000932 1781235513.236789 fallocate 0x027cdb thru 0x033660 (0xb986 bytes) 000933 1781235513.249897 trunc from 0x037512 to 0x03b07e (0x3b6d bytes) 000934 1781235513.291726 trunc from 0x03b07f to 0x03fc78 (0x4bfa bytes) 000935 1781235513.314431 trunc from 0x03fc79 to 0x031903 (0xe375 bytes) 000936 1781235513.336640 fallocate 0x02e1ad thru 0x035787 (0x75db bytes) 000937 1781235513.343605 write 0x03b224 thru 0x03ffff (0x4ddc bytes) 000938 1781235513.346609 write 0x01a429 thru 0x02437a (0x9f52 bytes) 000939 1781235513.362917 punch from 0x01ce12 to 0x0257f6 (0x89e5 bytes) 000941 1781235513.386591 trunc from 0x040000 to 0x01eedd (0x21122 bytes) 000942 1781235513.405785 read 0x01a6a5 thru 0x01da9e (0x33fa bytes) 000943 1781235513.413191 fallocate 0x018605 thru 0x01a482 (0x1e7e bytes) 000944 1781235513.418803 read 0x009b79 thru 0x0159c4 (0xbe4c bytes) 000945 1781235513.440135 punch from 0x01b57c to 0x01eedd (0x3962 bytes) 000946 1781235513.449551 trunc from 0x007b70 to 0x00ef94 (0x7425 bytes) 000947 1781235513.458873 mapread 0x01a176 thru 0x01eedd (0x4d68 bytes) 000949 1781235513.490993 write 0x01cfca thru 0x01ff9b (0x2fd2 bytes) 000950 1781235513.496901 read 0x013539 thru 0x01728f (0x3d57 bytes) 000951 1781235513.500691 write 0x014773 thru 0x016230 (0x1abe bytes) 000952 1781235513.510548 trunc from 0x01ff9c to 0x02d0f4 (0xd159 bytes) 000953 1781235513.560558 mapread 0x016108 thru 0x01ab46 (0x4a3f bytes) 000954 1781235513.567400 trunc from 0x02d0f5 to 0x03213d (0x5049 bytes) 000955 1781235513.630831 MIRROR_RESYNC 3 mirrors 000956 1781235514.661662 write 0x0040c6 thru 0x01052e (0xc469 bytes) 000957 1781235514.720826 mapwrite 0x017f91 thru 0x019491 (0x1501 bytes) 000958 1781235514.753836 trunc from 0x03213e to 0x024619 (0xdb24 bytes) 000959 1781235514.781889 punch from 0x01b2ca to 0x021745 (0x647c bytes) 000961 1781235514.794938 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x2461a): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2461a): Mirror 2: 0x43e17c12 (stale) Mirror 3: 0x43e17c12 (stale) CRC-32 checksum value for chunk [0, 0x2461a): Mirror 1: 0xe7c76f2a Mirror 4: 0xe7c76f2a lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2461a: skipped 000962 1781235515.707637 punch from 0x01c44f to 0x01f126 (0x2cd8 bytes) 000963 1781235515.748973 punch from 0x00ba5d to 0x00d8de (0x1e82 bytes) 000964 1781235515.757727 mapread 0x002c64 thru 0x002d5a (0xf7 bytes) 000965 1781235515.764677 mapread 0x000cd3 thru 0x00e33e (0xd66c bytes) 000966 1781235515.785076 write 0x01e5cc thru 0x020fad (0x29e2 bytes) 000967 1781235515.804392 write 0x00b8f1 thru 0x011e28 (0x6538 bytes) 000968 1781235515.810742 trunc from 0x002748 to 0x00dbf4 (0xb4ad bytes) 000969 1781235515.827360 read 0x01b049 thru 0x0201a9 (0x5161 bytes) 000970 1781235515.833731 MIRROR_SPLIT mirror 3 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x240000404:0x153:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2461a): Mirror 2: 0x43e17c12 (stale) Mirror 3: 0x43e17c12 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2461a): Mirror 2: 0x43e17c12 (stale) Mirror 3: 0x43e17c12 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2461a: skipped 000971 1781235516.353188 punch from 0x001fd1 to 0x00a6c6 (0x86f6 bytes) 000972 1781235516.365960 trunc from 0x02461a to 0x00a9b7 (0x19c62 bytes) 000973 1781235516.403474 read 0x008168 thru 0x00a9b7 (0x2850 bytes) 000974 1781235516.416300 MIRROR_RESYNC 3 mirrors 000975 1781235517.135804 MIRROR_EXTEND to 4 mirrors 000976 1781235517.871821 punch from 0x003838 to 0x00a9b7 (0x7180 bytes) 000977 1781235517.944601 mapwrite 0x016373 thru 0x01a7af (0x443d bytes) 000978 1781235518.086042 trunc from 0x01a7b0 to 0x011130 (0x967f bytes) 000979 1781235518.129707 trunc from 0x011131 to 0x0197b9 (0x8689 bytes) 000980 1781235518.148183 trunc from 0x00126d to 0x005d22 (0x4ab6 bytes) 000982 1781235518.168053 trunc from 0x011c06 to 0x0183b4 (0x67af bytes) 000983 1781235518.173570 trunc from 0x0197ba to 0x03271b (0x18f62 bytes) 000984 1781235518.227500 MIRROR_RESYNC 4 mirrors 000985 1781235519.389244 mapwrite 0x02fe47 thru 0x03f000 (0xf1ba bytes) 000987 1781235519.560783 read 0x02abb7 thru 0x033c4a (0x9094 bytes) 000988 1781235519.565580 write 0x015f5f thru 0x02348d (0xd52f bytes) 000989 1781235519.584573 trunc from 0x03f001 to 0x01269e (0x2c962 bytes) 000990 1781235519.610463 trunc from 0x012622 to 0x01269e (0x7d bytes) 000991 1781235519.618667 read 0x00e1bd thru 0x01269e (0x44e2 bytes) 000992 1781235519.636422 trunc from 0x01269f to 0x003a27 (0xec77 bytes) 000993 1781235519.667002 MIRROR_RESYNC 4 mirrors 000994 1781235520.404499 trunc from 0x00055b to 0x003a27 (0x34cd bytes) 000995 1781235520.449291 punch from 0x000a30 to 0x003a27 (0x2ff8 bytes) 000996 1781235520.456951 mapwrite 0x02ba70 thru 0x035a7e (0xa00f bytes) 000997 1781235520.522308 read 0x022c3d thru 0x025d7b (0x313f bytes) 000998 1781235520.530634 fallocate 0x032be3 thru 0x03d6e4 (0xab02 bytes) 000999 1781235520.535713 trunc from 0x01ff2f to 0x02504a (0x511c bytes) 001000 1781235520.541411 trunc from 0x0076e4 to 0x01767b (0xff98 bytes) All operations completed A-OK! PASS 70a (80s) == sanity-flr test 71: check mirror extend parent fid ==== 23:38:47 (1781235527) striped dir -i1 -c2 -H crush2 /mnt/lustre/d71.sanity-flr ** lfs fid2path /mnt/lustre 0x240000404:0x187:0x0 /mnt/lustre/d71.sanity-flr/f71.sanity-flr PASS 71 (14s) == sanity-flr test 72: check append on FLR file ========== 23:39:01 (1781235541) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.208424 s, 10.1 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102479 s, 10.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x114:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x115:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf8:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0xf7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x116:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 23:39:11 (1781235551) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre2 Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 Extending file size to 1846880 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6252576 .. done Extending file size to 4363872 .. lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9008576 .. Extending file size to 8752416 .. Extending file size to 9238144 .. done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed Extending file size to 9190816 .. resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..done lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 61380 61382 61383 61384 61387 done 192.168.202.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg219-client.virtnet /mnt/lustre2 (opts:) 192.168.202.119@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg219-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (84s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 23:40:35 (1781235635) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre2 Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 459520 .. Extending file size to 3561056 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 6291968 .. failed Extending file size to 3966624 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8158080 .. Extending file size to 7442080 .. failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..Extending file size to 8113984 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8326144 .. failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8429472 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed Extending file size to 8444608 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 8568064 .. done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8818336 .. resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Waiting 62852 62853 62854 62855 62856 192.168.202.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg219-client.virtnet /mnt/lustre2 (opts:) 192.168.202.119@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg219-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (78s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 23:41:53 (1781235713) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2316 1268988 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2108 1269196 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 23956 3583064 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 18524 3588496 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 42480 7171560 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2316 1268988 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 2108 1269196 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 23956 3583064 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 18524 3588496 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 42480 7171560 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0703315 s, 29.8 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.27722 s, 7.6 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [2] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 1: 0x8b6c8d2e (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 2: skipped 192.168.202.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg219-client.virtnet /mnt/lustre2 (opts:) PASS 200c (19s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 23:42:14 (1781235734) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.052546 s, 39.9 MB/s PASS 202 (7s) == sanity-flr test 203: mirror file preserve mirror ID === 23:42:21 (1781235741) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x64:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x114:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x115:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x280000401:0x131:0x0] } PASS 203 (10s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 23:42:31 (1781235751) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204a.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.120289 s, 17.4 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d4:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d5:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d6:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (9s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 23:42:40 (1781235760) striped dir -i0 -c2 -H crush2 /mnt/lustre/d204b.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0905837 s, 23.2 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x116:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x117:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x132:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x133:0x0] } PASS 204b (10s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 23:42:50 (1781235770) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204c.sanity-flr 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.107662 s, 19.5 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d5:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d7:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d8:0x0] } PASS 204c (35s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 23:43:25 (1781235805) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg219-server mds-ost sync done. striped dir -i0 -c2 -H all_char /mnt/lustre/d204d.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991829 s, 10.6 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d6:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1da:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000400:0x1d9:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000400:0x1d7:0x0] } PASS 204d (56s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 23:44:22 (1781235862) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg219-server mds-ost sync done. striped dir -i0 -c2 -H crush2 /mnt/lustre/d204e.sanity-flr /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0928615 s, 22.6 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x134:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x135:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x118:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (54s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 23:45:16 (1781235916) Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg219-server mds-ost sync done. striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d204f.sanity-flr Creating new pool test_204f oleg219-server: Pool lustre.test_204f created Adding targets to pool oleg219-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg219-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.166271 s, 12.6 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x280000401:0x136:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x119:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg219-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg219-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg219-server: Pool lustre.test_204f destroyed PASS 204f (88s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 23:46:44 (1781236004) PASS 205a (9s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 23:46:53 (1781236013) PASS 205b (7s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 23:47:01 (1781236021) oleg219-server: Pool lustre.test_206 created Adding targets to pool oleg219-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg219-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg219-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg219-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg219-server: Pool lustre.test_206 destroyed PASS 206 (30s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 23:47:30 (1781236050) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098097 s, 10.7 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0824991 s, 12.7 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x708e3526 Mirror 3: 0x708e3526 PASS 207 (13s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 23:47:43 (1781236063) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (4s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 23:47:47 (1781236067) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 23:47:51 (1781236071) Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre2 osd-ldiskfs.lustre-OST0000.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=0 osd-ldiskfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.0850244 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x2220000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0x1750000 Read this (4 bytes): AAAA PAUSING Buffer address : 0x1750000 Read this (4 bytes): BBBB PAUSING Buffer address : 0x1750000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0xb70000 Read this (4 bytes): BBBB 192.168.202.119@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg219-client.virtnet /mnt/lustre2 (opts:) PASS 209a (43s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 23:48:34 (1781236114) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0281463 s, 291 kB/s the very first read read with warm cache PASS 209b (8s) == sanity-flr test 210a: handle broken mirrored lovea ==== 23:48:42 (1781236122) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412652 s, 25.4 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000405 lmm_object_id: 0x83 lmm_fid: [0x200000405:0x83:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x2c0000401:0x124:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 4096 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205339611267 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-11 23:48:44.000000000 -0400 Modify: 2026-06-11 23:48:44.000000000 -0400 Change: 2026-06-11 23:48:45.000000000 -0400 Birth: 2026-06-11 23:48:43.000000000 -0400 PASS 210a (8s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 23:48:50 (1781236130) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0381845 s, 27.5 MB/s oleg219-server: debugfs 1.47.3-wc2 (11-Nov-2025) fail_loc=0x1428 Waiting for MDT destroys to complete oleg219-server: debugfs 1.47.3-wc2 (11-Nov-2025) oleg219-server: O/280000401/d0/140: File not found by ext2_lookup PASS 210b (22s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 23:49:12 (1781236152) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.290372 s, 141 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0302415 s, 135 kB/s size after second write -rw-r--r-- 1 root root 4096 Jun 11 23:49 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000405:0x88:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jun 11 23:49 /mnt/lustre/f211.sanity-flr PASS 211 (8s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 23:49:20 (1781236160) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (8s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 23:49:28 (1781236168) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1871 sec =========== 23:49:38 (1781236178) === sanity-flr: start cleanup 23:49:40 (1781236180) === === sanity-flr: finish cleanup 23:49:48 (1781236188) ===