-----============= acceptance-small: sanity-flr ============----- Tue Jul 14 12:10:19 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg258-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanity-flr.*ex': No such file or directory excepting tests: 6 201 44c 49a === sanity-flr: start setup 12:10:39 (1784045439) === oleg258-client.virtnet: executing check_config_client /mnt/lustre oleg258-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg258-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff89d959d07000.idle_timeout=debug osc.lustre-OST0001-osc-ffff89d959d07000.idle_timeout=debug disable quota as required oleg258-server: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity-flr: finish setup 12:11:12 (1784045472) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7615] == sanity-flr test 0a: lfs mirror create with -N option == 12:11:15 (1784045475) PASS 0a (14s) == sanity-flr test 0b: lfs mirror create plain layout mirrors ========================================================== 12:11:29 (1784045489) SKIP: sanity-flr test_0b need >= 4 OSTs SKIP 0b (3s) == sanity-flr test 0c: lfs mirror create composite layout mirrors ========================================================== 12:11:32 (1784045492) SKIP: sanity-flr test_0c need >= 4 OSTs SKIP 0c (3s) == sanity-flr test 0d: lfs mirror extend with -N option == 12:11:36 (1784045496) PASS 0d (29s) == sanity-flr test 0e: lfs mirror extend plain layout mirrors ========================================================== 12:12:04 (1784045524) SKIP: sanity-flr test_0e need >= 4 OSTs SKIP 0e (4s) == sanity-flr test 0f: lfs mirror extend composite layout mirrors ========================================================== 12:12:08 (1784045528) SKIP: sanity-flr test_0f need >= 4 OSTs SKIP 0f (5s) == sanity-flr test 0g: lfs mirror create flags support === 12:12:13 (1784045533) error: lfs create: --flags must be specified with --mirror-count|-N option Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: component flag 'foo' not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) lfs create: unsupported mirror flags: stale lfs create: unsupported mirror flags: prefer,init lfs create: inverted flags are not supported Create a mirrored file. usage: lfs mirror create --mirror-count|-N[MIRROR_COUNT] [SETSTRIPE_OPTIONS] ... FILENAME|DIRECTORY ... lfs commands used to manage files with mirrored components: lfs mirror create - create a mirrored file or directory lfs mirror extend - add mirror(s) to an existing file lfs mirror split - split a mirror from an existing mirrored file lfs mirror resync - resynchronize out-of-sync mirrored file(s) lfs mirror read - read a mirror content of a mirrored file lfs mirror write - write to a mirror of a mirrored file lfs mirror copy - copy a mirror to other mirror(s) of a file lfs mirror verify - verify mirrored file(s) osc.lustre-OST0000-osc-ffff89d959d07000.stats=clear osc.lustre-OST0001-osc-ffff89d959d07000.stats=clear PASS 0g (8s) == sanity-flr test 0h: set, clear and test flags for FLR files ========================================================== 12:12:21 (1784045541) lfs setstripe: component flag 'foo' not supported Create a file with specified striping/composite layout, or set the default layout on an existing directory. Usage: setstripe [--component-add|--component-del|--delete|-d] [--comp-set --comp-id|-I COMP_ID|--comp-flags=COMP_FLAGS] [--component-end|-E END_OFFSET] [--copy=SOURCE_LAYOUT_FILE]|--yaml|-y YAML_TEMPLATE_FILE] [--ec|--ec-stripe-count DATA+PARITY] [--ec-expert DATA+PARITY] [--extension-size|--ext-size|-z EXT_SIZE] [--help|-h] [--foreign=FOREIGN_TYPE --xattr|-x LAYOUT] [--layout|-L PATTERN] [--mode FILE_MODE] [--mirror-count|-N[MIRROR_COUNT]] [--ost|-o OST_INDEX[,OST_INDEX,...]] [--overstripe-count|-C STRIPE_COUNT] [--pool|-p POOL_NAME] [--stripe-count|-c STRIPE_COUNT] [--stripe-index|-i START_OST_IDX] [--stripe-size|-S STRIPE_SIZE] FILENAME|DIRECTORY lfs: component flag 'foo' not supported error: getstripe bad component flags 'prefer,foo'. List the layout pattern for a given file or files in a directory or recursively for all files in a directory tree. Usage: getstripe [--ost|-O OST_NAME] [--quiet|-q] [--verbose|-v] [--stripe-count|-c] [--stripe-index|-i] [--fid|-F] [--pool|-p] [--stripe-size|-S] [--directory|-d] [--mdt-index|-m] [--recursive|-r] [--raw|-R] [--layout|-L] [--generation|-g] [--yaml|-y] [--help|-h] [--hex-idx] [--component-id|-I[=COMP_ID]] [--component-flags[=COMP_FLAGS]] [--component-count] [--extension-size|--ext-size|-z] [--component-start[=[+-]START_OFFSET]] [--component-end|-E[[+-]END_OFFSET]] [[!] --mirror-index=[+-]MIRROR_INDEX | [!] --mirror-id=[+-]MIRROR_ID] [--mirror-count|-N] [--no-follow] FILENAME|DIRECTORY lfs setstripe: cannot clear 'stale' flags from component. Please use lfs-mirror-resync(1) instead PASS 0h (13s) == sanity-flr test 0j: test lfs mirror read/write commands ========================================================== 12:12:34 (1784045554) PASS 0j (9s) == sanity-flr test 0k: mirroring a file in directory with default FLR layout ========================================================== 12:12:43 (1784045563) SKIP: sanity-flr test_0k need >= 3 OSTs SKIP 0k (4s) == sanity-flr test 0m: mirror extend with -N= option ===== 12:12:47 (1784045567) warning: '/mnt/lustre/d0m.sanity-flr/f0m.sanity-flr' already has 2 mirrors, no new mirrors will be created PASS 0m (8s) == sanity-flr test 1: create components with setstripe options ========================================================== 12:12:55 (1784045575) lfs mirror mirror: cannot get UNLOCK lease, ext 4 for [0x200000401:0x28:0x0]: Numerical result out of range (34) error: lfs extend: /mnt/lustre/f1.sanity-flr: cannot merge layout: Numerical result out of range PASS 1 (9s) == sanity-flr test 2: create components from existing files ========================================================== 12:13:04 (1784045584) PASS 2 (7s) == sanity-flr test 3: create components from files located on different MDTs ========================================================== 12:13:11 (1784045591) SKIP: sanity-flr test_3 need >= 2 MDTs SKIP 3 (3s) == sanity-flr test 4: Make sure mirror attributes can be inhertied from directory ========================================================== 12:13:14 (1784045594) error: lfs extend: /mnt/lustre/d4.sanity-flr: cannot get the last init comp: Is a directory PASS 4 (8s) == sanity-flr test 5: Make sure init size work for mirrored layout ========================================================== 12:13:22 (1784045602) /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK /mnt/lustre/f5.sanity-flr has type file OK /mnt/lustre/f5.sanity-flr has size 12345 OK PASS 5 (7s) SKIP: sanity-flr test_6a skipping ALWAYS excluded test 6a (base 6) SKIP: sanity-flr test_6b skipping ALWAYS excluded test 6b (base 6) == sanity-flr test 7: Create mirror with DoM component === 12:13:33 (1784045613) PASS 7 (8s) == sanity-flr test 21: glimpse should report accurate i_blocks ========================================================== 12:13:41 (1784045621) 7+0 records in 7+0 records out 7340032 bytes (7.3 MB, 7.0 MiB) copied, 0.811746 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0956616 s, 11.0 MB/s PASS 21 (10s) == sanity-flr test 22: no glimpse to OSTs for READ_ONLY files ========================================================== 12:13:51 (1784045631) 13+0 records in 13+0 records out 13631488 bytes (14 MB, 13 MiB) copied, 0.397662 s, 34.3 MB/s PASS 22 (10s) == sanity-flr test 28: create file with 256 mirrors when mirror_count_max=256 ========================================================== 12:14:01 (1784045641) lod.lustre-MDT0000-mdtlov.mirror_count_max=256 lod.lustre-MDT0000-mdtlov.mirror_count_max=16 PASS 28 (11s) == sanity-flr test 31: make sure glimpse request can be retried ========================================================== 12:14:12 (1784045652) Stopping /mnt/lustre-ost1 (opts:) on oleg258-server oleg258-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89d959d07000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89d959d07000.ost_server_uuid in IDLE state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg258-server: oleg258-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg258-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg258-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg258-server oleg258-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89d959d07000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89d959d07000.ost_server_uuid in DISCONN state after 0 sec Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg258-server: oleg258-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg258-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 31 (142s) == sanity-flr test 32: data should be mirrored to newly created mirror ========================================================== 12:16:35 (1784045795) 9+0 records in 9+0 records out 9437184 bytes (9.4 MB, 9.0 MiB) copied, 0.620816 s, 15.2 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg258-server oleg258-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89d959d07000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89d959d07000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f32.sanity-flr has type file OK /mnt/lustre/f32.sanity-flr has size 9437184 OK reading file from the 2nd mirror and verify checksum Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg258-server: oleg258-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg258-server: *.lustre-OST0000.recovery_status status: COMPLETE PASS 32 (45s) == sanity-flr test 33a: read can choose available mirror to read ========================================================== 12:17:19 (1784045839) error: lfs extend: /mnt/lustre/f33a.sanity-flr: file busy or contents don't match: Invalid argument /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data can be provided by any ost)... Stopping /mnt/lustre-ost1 (opts:) on oleg258-server oleg258-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89d959d07000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89d959d07000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg258-server: oleg258-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg258-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg258-server oleg258-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89d959d07000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89d959d07000.ost_server_uuid in DISCONN state after 0 sec /mnt/lustre/f33a.sanity-flr has type file OK /mnt/lustre/f33a.sanity-flr has size 500 OK reading file (data should be provided by ost1)... Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg258-server: oleg258-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg258-server: *.lustre-OST0001.recovery_status status: COMPLETE PASS 33a (90s) == sanity-flr test 33b: avoid reading from unhealthy mirror ========================================================== 12:18:49 (1784045929) 25+0 records in 25+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 7.2338 s, 7.2 MB/s -rw-r--r-- 1 root root 52428800 Jul 14 12:18 /mnt/lustre/f33b.sanity-flr reading file (data can be provided by any ost)... real 0m1.520s user 0m0.001s sys 0m0.149s Stopping /mnt/lustre-ost1 (opts:) on oleg258-server oleg258-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff89d959d07000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff89d959d07000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost2)... real 0m1.414s user 0m0.000s sys 0m0.174s Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg258-server: oleg258-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg258-server: *.lustre-OST0000.recovery_status status: COMPLETE Stopping /mnt/lustre-ost2 (opts:) on oleg258-server oleg258-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0001-osc-ffff89d959d07000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89d959d07000.ost_server_uuid in DISCONN state after 0 sec reading file (data should be provided by ost1)... real 0m1.379s user 0m0.001s sys 0m0.156s Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg258-server: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-OST0001 affected facets: ost2 oleg258-server: oleg258-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 oleg258-server: *.lustre-OST0001.recovery_status status: COMPLETE oleg258-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89d959d07000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff89d959d07000.ost_server_uuid in FULL state after 0 sec PASS 33b (110s) == sanity-flr test 33c: keep reading among unhealthy mirrors ========================================================== 12:20:39 (1784046039) SKIP: sanity-flr test_33c need >= 3 OSTs SKIP 33c (3s) == sanity-flr test 34a: read mirrored file with multiple stripes ========================================================== 12:20:43 (1784046043) SKIP: sanity-flr test_34a need >= 4 OSTs SKIP 34a (3s) == sanity-flr test 34b: read mirrored file with multiple components ========================================================== 12:20:46 (1784046046) SKIP: sanity-flr test_34b need >= 4 OSTs SKIP 34b (4s) == sanity-flr test 35: allow to write to mirrored files == 12:20:50 (1784046050) PASS 35 (8s) == sanity-flr test 36a: write to mirrored files ========== 12:20:59 (1784046059) 12+0 records in 12+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.558144 s, 22.5 MB/s fail_val=10 fail_loc=0x1A01 fail_loc=0 PASS 36a (13s) == sanity-flr test 36b: write should not hung when prefered mirror is stale ========================================================== 12:21:12 (1784046072) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0536984 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693545 s, 15.1 MB/s PASS 36b (7s) == sanity-flr test 36c: change prefer mirror during write shouldn't hung ========================================================== 12:21:19 (1784046079) PASS 36c (8s) == sanity-flr test 36d: write/punch FLR file update OST layout version ========================================================== 12:21:28 (1784046088) ** create /mnt/lustre/f36d.sanity-flr ** mirror extend /mnt/lustre/f36d.sanity-flr (1/2) flv=2 olv=0 ** mirror extend /mnt/lustre/f36d.sanity-flr (2/2) flv=3 olv=0 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00212541 s, 482 kB/s flv=4 olv=4 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00119645 s, 856 kB/s flv=4 olv=4 ** resync /mnt/lustre/f36d.sanity-flr flv=6 olv=6 ** truncate /mnt/lustre/f36d.sanity-flr (1/2) flv=7 olv=7 ** truncate /mnt/lustre/f36d.sanity-flr (2/2) flv=7 olv=7 ** resync /mnt/lustre/f36d.sanity-flr flv=9 olv=9 ** write /mnt/lustre/f36d.sanity-flr (1/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000806061 s, 1.3 MB/s flv=10 olv=10 ** write /mnt/lustre/f36d.sanity-flr (2/2) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.000768837 s, 1.3 MB/s flv=10 olv=10 PASS 36d (11s) == sanity-flr test 37: mirror I/O API verification ======= 12:21:38 (1784046098) create test files with size 15730074 .. 363f973bb9160f6f793a03fc00ac7add - 1eb5b168d9c824620ab975ca32145b03 - 80d9ffb5127288811ed38b848373270a - 3 Verifying mirror read .. Verifying mirror write .. Verifying mirror copy .. mirror copied successfully: 2 3 Stopping client oleg258-client.virtnet /mnt/lustre (opts:) Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre PASS 37 (27s) == sanity-flr test 38: resync ============================ 12:22:05 (1784046125) 4 mirror IDs: 1 2 3 4 mirror_io: 391: hit by error injection mirror_io: 391: hit by error injection PASS 38 (16s) == sanity-flr test 39: check FLR+PFL (a.k.a. PFLR) creation ========================================================== 12:22:22 (1784046142) PASS 39 (9s) == sanity-flr test 40: PFLR rdonly state instantiation check ========================================================== 12:22:30 (1784046150) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129319 s, 8.1 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc8:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xc9:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x7 lmm_fid: 0x200000402:0x7:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233194 s, 45.0 MB/s composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 760 lcm_flags: wp lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 7 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcb:0x0 - lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init,prefer lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - l_ost_idx: 0 l_fid: 0x240000400:0xcc:0x0 - lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: prefer lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x280000400:0xca:0x0 - lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 - lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x9 lmm_fid: 0x200000402:0x9:0x0 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 40 (11s) == sanity-flr test 41: lfs mirror resync check =========== 12:22:41 (1784046161) **create two FLR files /mnt/lustre/f41.sanity-flr /mnt/lustre/f41.sanity-flr-1 **verify files be RDONLY 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.129493 s, 16.2 MB/s 4+0 records in 4+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.343695 s, 12.2 MB/s **verify files be WRITE_PENDING **verify files have stale component **full resync **verify /mnt/lustre/f41.sanity-flr-1 data consistency in all mirrors **verify files be RDONLY **verify files do not contain stale component **write /mnt/lustre/f41.sanity-flr-1 for partial resync test 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.191323 s, 11.0 MB/s **only resync mirror 2 (without force, should fail) **only resync mirror 2 (with --force-resync) lfs: using target mirror 2 as source (no other non-stale mirrors found). lfs: using mirror 2 as source for force resync. lfs: marked 4 components as stale for force resync. lfs: proceeding with resync of 4 components. **verify mirrors are now consistent after force resync **resync mirror 2 again (with --force-resync) lfs: using mirror 1 as source for force resync. lfs: marked 2 components as stale for force resync. lfs: proceeding with resync of 2 components. **verify should now succeed after resync **verify /mnt/lustre/f41.sanity-flr-1 contains stale component **full resync /mnt/lustre/f41.sanity-flr-1 **full resync /mnt/lustre/f41.sanity-flr-1 again **verify /mnt/lustre/f41.sanity-flr-1 does not contain stale component PASS 41 (20s) == sanity-flr test 42: lfs mirror verify ================= 12:23:01 (1784046181) SKIP: sanity-flr test_42 need >= 4 OSTs SKIP 42 (3s) == sanity-flr test 43a: mirror pick on write ============= 12:23:05 (1784046185) SKIP: sanity-flr test_43a needs >= 3 OSTs SKIP 43a (4s) == sanity-flr test 43b: allow writing to multiple preferred mirror file ========================================================== 12:23:08 (1784046188) ** write to /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0273488 s, 38.3 MB/s ** resync /mnt/lustre/d43b.sanity-flr/f43b.sanity-flr PASS 43b (8s) == sanity-flr test 44a: lfs mirror split check =========== 12:23:16 (1784046196) SKIP: sanity-flr test_44a needs >= 2 MDTs SKIP 44a (3s) == sanity-flr test 44b: mirror split does not create stale file ========================================================== 12:23:20 (1784046200) /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 2 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0xd3:0x0] } lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd1:0x0] } split mirror_id 1 id 65537 /mnt/lustre/d44b.sanity-flr/f44b.sanity-flr lcm_layout_gen: 5 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xd1:0x0] } split mirror_id 2 id 131073 lfs split: '/mnt/lustre/d44b.sanity-flr/f44b.sanity-flr' is not a FLR file. Cannot resync file make sure there's no stale comp in the file PASS 44b (9s) SKIP: sanity-flr test_44c skipping ALWAYS excluded test 44c == sanity-flr test 44d: lfs mirror split does not break size ========================================================== 12:23:31 (1784046211) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.353014 s, 29.7 MB/s ** before mirror ops, file size=10485760 ** after mirror extend, file size=10485760 lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000402:0x18:0x0]: Invalid argument (22) ** after mirror split, file size=10485760 Stopping client oleg258-client.virtnet /mnt/lustre (opts:) Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre ** after mirror split & remount, file size=10485760 PASS 44d (13s) == sanity-flr test 44e: basic FLR SOM tests + disable SOM ========================================================== 12:23:44 (1784046224) llite.lustre-ffff89d9454dd800.xattr_cache=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.357525 s, 29.3 MB/s ** before mirror resync, file size=10485760 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 Stopping client oleg258-client.virtnet /mnt/lustre (opts:) Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre mdt.lustre-MDT0000.enable_strict_som=0 file: /mnt/lustre/d44e.sanity-flr/f44e.sanity-flr size: 10485760 blocks: 2 flags: 1 llite.lustre-ffff89d959d68800.xattr_cache=1 PASS 44e (15s) == sanity-flr test 45: Verify setstripe/getstripe with YAML with FLR file ========================================================== 12:23:59 (1784046239) 1. FLR file: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 1. FLR file: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 1. FLR file: compare 2. FLR file with flags: getstripe --yaml /mnt/lustre/d45.sanity-flr/f45.sanity-flr 2. FLR file with flags: setstripe --yaml=/mnt/lustre/d45.sanity-flr/template /mnt/lustre/d45.sanity-flr/f45.sanity-flr.copy 2. FLR file with flags: compare PASS 45 (40s) == sanity-flr test 46: Verify setstripe --copy option ==== 12:24:40 (1784046280) ** 1. PFL file ** 2. plain file ** 3. FLR file ** 4. PFL dir ** 5. plain dir ** 6. FLR dir PASS 46 (42s) == sanity-flr test 47: Verify mirror obj alloc =========== 12:25:22 (1784046322) SKIP: sanity-flr test_47 needs >= 3 OSTs SKIP 47 (4s) == sanity-flr test 48: Verify snapshot mirror ============ 12:25:27 (1784046327) ** create 2 mirrors FLR file /mnt/lustre/f48.sanity-flr ** write it 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.238242 s, 13.2 MB/s ** resync the file ** snapshot mirror 2 ** write it again 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.128293 s, 24.5 MB/s ** resync it again CRC-32 checksum value for chunk [0, 0x300000): Mirror 2: 0xff9740ea (nosync,stale) Mirror 2: 0xff9740ea (nosync,stale) ** verify mirror 2 doesn't change original checksum: 5a2eaf07ceb70c7ad032932d2c3ce3df - mirror 1 checksum: bbcae66641f20f9e776d7a942d8f3ae7 - mirror 2 checksum: 5a2eaf07ceb70c7ad032932d2c3ce3df - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: wp lcm_layout_gen: 10 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-14T12:25:31-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale,nosync lcme_timestamp: '2026-07-14T12:25:31-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } ** resync mirror 2 ** verify mirror 2 resync-ed original checksum: 5a2eaf07ceb70c7ad032932d2c3ce3df - mirror 1 checksum: bbcae66641f20f9e776d7a942d8f3ae7 - mirror 2 checksum: bbcae66641f20f9e776d7a942d8f3ae7 - ** mirror 2 stripe info /mnt/lustre/f48.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 448 lcm_flags: ro lcm_layout_gen: 12 lcm_mirror_count: 2 lcm_entry_count: 4 components: - lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-14T12:25:31-0400' lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lcme_offset: 336 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe6:0x0] } - lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,nosync lcme_timestamp: '2026-07-14T12:25:31-0400' lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lcme_offset: 392 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x11 lmm_fid: [0x200000404:0x11:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe7:0x0] } PASS 48 (11s) SKIP: sanity-flr test_49a skipping ALWAYS excluded test 49a == sanity-flr test 50A: mirror split update layout generation ========================================================== 12:25:40 (1784046340) Failing mds1 on oleg258-server Stopping /mnt/lustre-mds1 (opts:) on oleg258-server 12:25:45 (1784046345) shut down facet: mds1 facet_host: oleg258-server facet_failover_host: oleg258-server Failover mds1 to oleg258-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg258-server: oleg258-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg258-client: oleg258-server: ssh exited with exit code 1 Started lustre-MDT0000 12:26:09 (1784046369) targets are mounted 12:26:09 (1784046369) facet_failover done oleg258-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x15:0x0]: Invalid argument (22) /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr layout generation from 1 to 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467119 s, 22.4 MB/s /mnt/lustre/d50A.sanity-flr/f50A.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 136 lcm_flags: 0 lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 1 components: - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 80 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x15 lmm_fid: [0x200000404:0x15:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0xe8:0x0] } PASS 50A (51s) == sanity-flr test 50a: mirror extend/copy preserves sparseness ========================================================== 12:26:30 (1784046390) SKIP: sanity-flr test_50a OST does not support SEEK_HOLE SKIP 50a (4s) == sanity-flr test 50b: mirror rsync handles sparseness == 12:26:34 (1784046394) SKIP: sanity-flr test_50b OST does not support SEEK_HOLE SKIP 50b (4s) == sanity-flr test 50c: punch_hole/mmap_write stale other mirrors ========================================================== 12:26:38 (1784046398) 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0187249 s, 875 kB/s PASS 50c (11s) == sanity-flr test 50d: mirror rsync keep holes ========== 12:26:49 (1784046409) SKIP: sanity-flr test_50d OST does not support SEEK_HOLE SKIP 50d (4s) == sanity-flr test 60a: mirror extend sets correct size on sparse file ========================================================== 12:26:54 (1784046414) SKIP: sanity-flr test_60a OST does not support SEEK_HOLE SKIP 60a (5s) == sanity-flr test 60b: mirror merge/split cancel client's in-memory layout gen ========================================================== 12:26:58 (1784046418) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1e:0x0]: Invalid argument (22) PASS 60b (20s) == sanity-flr test 61a: mirror extend and migrate preserve timestamps ========================================================== 12:27:19 (1784046439) mdd.lustre-MDT0000.atime_diff=1 create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr create /mnt/lustre/d61a.sanity-flr/f61a.sanity-flr-2 sleep 5 seconds, then cat f61a.sanity-flr XXX sleep 5 seconds, then re-write f61a.sanity-flr sleep 5 seconds mirror merge f61a.sanity-flr-2 to f61a.sanity-flr and test timestamps mirror extend f61a.sanity-flr and test timestamps migrate f61a.sanity-flr and test timestamps normal user migrate f61a.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [migrate] [-n] [/mnt/lustre/d61a.sanity-flr/f61a.sanity-flr] mdd.lustre-MDT0000.atime_diff=60 PASS 61a (30s) == sanity-flr test 61b: mirror extend and split preserve timestamps ========================================================== 12:27:49 (1784046469) create /mnt/lustre/d61b.sanity-flr/f61b.sanity-flr sleep 5 seconds, then cat f61b.sanity-flr XXX sleep 5 seconds, then re-write f61b.sanity-flr sleep 5 seconds, then test timestamps mirror extend f61b.sanity-flr and test timestamps mirror split f61b.sanity-flr and test timestamps lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x4f:0x0]: Invalid argument (22) normal user mirror extend f61b.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 0 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [extend] [-N] [-c1] [-i1] [/mnt/lustre/d61b.sanity-flr/f61b.sanity-flr] PASS 61b (25s) == sanity-flr test 61c: mirror resync preserves timestamps ========================================================== 12:28:15 (1784046495) create /mnt/lustre/d61c.sanity-flr/f61c.sanity-flr sleep 5 seconds, then cat f61c.sanity-flr XXX sleep 5 seconds, then mirror extend f61c.sanity-flr and write it sleep 5 seconds, then resync f61c.sanity-flr and test timestamps normal user resync f61c.sanity-flr and test timestamps running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [mirror] [resync] [/mnt/lustre/d61c.sanity-flr/f61c.sanity-flr] PASS 61c (25s) == sanity-flr test 62: read/write with unknown type of mirror ========================================================== 12:28:40 (1784046520) create mirror file with unknown magic fail_loc=0x1426 fail_val=2 create mirror file with unknown pattern fail_loc=0x1427 fail_val=1 PASS 62 (9s) == sanity-flr test 70: mirror create and split race ====== 12:28:49 (1784046529) mirror create pid 53959 mirror split pid 53961 mirror create and split race for 60 seconds, should not crash /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 53959 Killed while true; do rm -f $tf; $LFS mirror create -N -E 1M -c -1 -E eof -N $tf; echo xxxx > $tf; done (wd: ~) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5041: 53961 Killed while true; do $LFS mirror split -d --mirror-id=1 $tf &> /dev/null; done (wd: ~) PASS 70 (68s) == sanity-flr test 70a: flr mode fsx test ================ 12:29:57 (1784046597) Seed set to 6599 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000003 1784046599.985988 mapwrite 0x021116 thru 0x02cc18 (0xbb03 bytes) 000004 1784046600.151824 mapread 0x01671b thru 0x02224e (0xbb34 bytes) truncating to largest ever: 0x24887 000005 1784046600.186778 trunc from 0x02cc19 to 0x024886 (0x8392 bytes) 000009 1784046600.221767 read 0x022af1 thru 0x023680 (0xb90 bytes) 000011 1784046600.223114 trunc from 0x024887 to 0x00a280 (0x1a606 bytes) 000017 1784046600.251282 mapwrite 0x03e2f2 thru 0x03ffff (0x1d0e bytes) 000019 1784046600.406537 write 0x00faf1 thru 0x01c79b (0xccab bytes) 000020 1784046600.429483 trunc from 0x040000 to 0x01ac70 (0x2538f bytes) 000022 1784046600.474558 read 0x0067d7 thru 0x00a669 (0x3e93 bytes) 000023 1784046600.491261 read 0x0189e1 thru 0x01ac70 (0x2290 bytes) 000026 1784046600.498030 MIRROR_EXTEND to 2 mirrors 000028 1784046601.166411 write 0x0184f5 thru 0x01d5db (0x50e7 bytes) 000030 1784046601.274574 read 0x01a4a0 thru 0x01d5db (0x313c bytes) 000034 1784046601.278531 write 0x003dad thru 0x006f1f (0x3173 bytes) 000036 1784046601.290842 write 0x00cfd7 thru 0x00d747 (0x771 bytes) 000037 1784046601.304831 MIRROR_RESYNC 2 mirrors 000038 1784046602.091203 mapread 0x015b79 thru 0x01d5db (0x7a63 bytes) 000040 1784046602.130191 read 0x0129c3 thru 0x016912 (0x3f50 bytes) 000041 1784046602.137001 mapread 0x016b2c thru 0x01ccc8 (0x619d bytes) 000043 1784046602.139220 read 0x0133d0 thru 0x01d5db (0xa20c bytes) 000046 1784046602.146336 mapread 0x008325 thru 0x0155b5 (0xd291 bytes) 000048 1784046602.165866 read 0x007429 thru 0x016a96 (0xf66e bytes) 000049 1784046602.166393 mapread 0x0069c0 thru 0x01466e (0xdcaf bytes) 000050 1784046602.168439 trunc from 0x01d5dc to 0x0175a1 (0x603a bytes) 000051 1784046602.360776 read 0x00257b thru 0x003835 (0x12bb bytes) 000053 1784046602.375910 mapread 0x006547 thru 0x010f38 (0xa9f2 bytes) 000055 1784046602.380905 read 0x00d061 thru 0x0175a1 (0xa541 bytes) 000056 1784046602.382871 write 0x02d92d thru 0x034069 (0x673d bytes) 000059 1784046602.388882 mapread 0x004c6e thru 0x007561 (0x28f4 bytes) 000060 1784046602.391131 read 0x012d4a thru 0x021ec0 (0xf177 bytes) 000066 1784046602.420180 mapread 0x005e10 thru 0x0063c7 (0x5b8 bytes) 000069 1784046602.422582 write 0x034c04 thru 0x035975 (0xd72 bytes) 000070 1784046602.425400 read 0x02fca8 thru 0x035975 (0x5cce bytes) 000072 1784046602.432914 MIRROR_RESYNC 2 mirrors 000073 1784046603.606772 read 0x00805b thru 0x010306 (0x82ac bytes) 000074 1784046603.623654 read 0x022d6f thru 0x02e3fb (0xb68d bytes) 000075 1784046603.642181 mapwrite 0x01547f thru 0x01b84e (0x63d0 bytes) 000076 1784046604.085469 trunc from 0x035976 to 0x019176 (0x1c7ff bytes) 000077 1784046604.154121 mapwrite 0x000ddf thru 0x00cf58 (0xc17a bytes) 000079 1784046604.538508 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) truncating to largest ever: 0x3d59e 000080 1784046604.886609 trunc from 0x019177 to 0x03d59d (0x24427 bytes) 000085 1784046604.924531 read 0x02367a thru 0x028599 (0x4f20 bytes) 000089 1784046604.939284 mapread 0x023a72 thru 0x02fa33 (0xbfc2 bytes) 000090 1784046604.974727 trunc from 0x03d59e to 0x00b47f (0x3211e bytes) 000091 1784046605.002268 write 0x00260f thru 0x011688 (0xf07a bytes) 000095 1784046605.015630 mapwrite 0x00252c thru 0x005735 (0x320a bytes) 000096 1784046605.092001 mapread 0x00701e thru 0x011688 (0xa66b bytes) 000097 1784046605.098519 read 0x000cdb thru 0x00e977 (0xdc9d bytes) 000102 1784046605.122809 read 0x00e12b thru 0x011688 (0x355e bytes) 000103 1784046605.123535 mapwrite 0x010fb1 thru 0x012e96 (0x1ee6 bytes) 000106 1784046605.232555 read 0x0080e1 thru 0x00fd7d (0x7c9d bytes) 000109 1784046605.248720 mapread 0x001462 thru 0x00816a (0x6d09 bytes) 000110 1784046605.252948 mapread 0x00b477 thru 0x012e96 (0x7a20 bytes) 000111 1784046605.255547 trunc from 0x012e97 to 0x0026d4 (0x107c2 bytes) 000112 1784046605.314932 trunc from 0x0026d5 to 0x02e90b (0x2c237 bytes) 000113 1784046605.359727 write 0x0348e3 thru 0x03b1f6 (0x6914 bytes) 000115 1784046605.368865 trunc from 0x03b1f7 to 0x032742 (0x8ab4 bytes) 000117 1784046605.409314 mapwrite 0x03cf7d thru 0x03ffff (0x3083 bytes) 000119 1784046605.501680 trunc from 0x040000 to 0x02b7cf (0x14830 bytes) 000121 1784046605.526295 mapread 0x01df59 thru 0x01dfd7 (0x7f bytes) 000122 1784046605.527904 mapread 0x00cb96 thru 0x00f92b (0x2d96 bytes) 000124 1784046605.530910 write 0x0222aa thru 0x02e1c6 (0xbf1d bytes) 000125 1784046605.533344 write 0x039d6f thru 0x03d772 (0x3a04 bytes) 000127 1784046605.536837 mapread 0x037ef1 thru 0x038140 (0x250 bytes) 000129 1784046605.549340 read 0x031afb thru 0x0352a8 (0x37ae bytes) 000130 1784046605.559560 mapread 0x0087d0 thru 0x010706 (0x7f37 bytes) 000131 1784046605.561136 write 0x03065e thru 0x03fcb5 (0xf658 bytes) 000132 1784046605.565599 trunc from 0x03fcb6 to 0x010494 (0x2f821 bytes) 000133 1784046605.586024 read 0x006fc6 thru 0x00d262 (0x629d bytes) 000134 1784046605.587909 trunc from 0x010495 to 0x025fad (0x15b19 bytes) 000135 1784046605.608561 read 0x00cf31 thru 0x016f9c (0xa06c bytes) 000136 1784046605.622596 mapread 0x017b9f thru 0x025d51 (0xe1b3 bytes) 000139 1784046605.625750 mapwrite 0x00a0b5 thru 0x00bf36 (0x1e82 bytes) 000144 1784046605.677723 trunc from 0x025fae to 0x00afb5 (0x1aff8 bytes) 000145 1784046605.716066 mapread 0x00acf0 thru 0x00afb5 (0x2c6 bytes) 000146 1784046605.717146 read 0x003243 thru 0x00afb5 (0x7d73 bytes) 000148 1784046605.726989 write 0x013f8b thru 0x01f758 (0xb7ce bytes) 000151 1784046605.744123 MIRROR_EXTEND to 2 mirrors 000152 1784046606.722820 trunc from 0x01f759 to 0x01b3f1 (0x4367 bytes) 000153 1784046606.875070 mapwrite 0x01bd4f thru 0x01f452 (0x3704 bytes) 000154 1784046606.972961 mapread 0x0190ee thru 0x01f452 (0x6365 bytes) 000156 1784046606.975733 write 0x00f5bc thru 0x01e592 (0xefd7 bytes) 000158 1784046606.983549 mapwrite 0x03dd2a thru 0x03ffff (0x22d6 bytes) 000161 1784046607.085473 MIRROR_RESYNC 2 mirrors 000162 1784046607.766496 trunc from 0x040000 to 0x029176 (0x16e89 bytes) 000163 1784046607.882696 write 0x03ee16 thru 0x03ffff (0x11ea bytes) 000165 1784046607.887482 mapwrite 0x027917 thru 0x036e2c (0xf516 bytes) 000166 1784046608.031293 read 0x027c96 thru 0x02a9ef (0x2d5a bytes) 000167 1784046608.032752 read 0x00208f thru 0x0056e8 (0x365a bytes) 000168 1784046608.066964 mapwrite 0x01cbe2 thru 0x026a2e (0x9e4d bytes) 000169 1784046608.121914 mapwrite 0x028e87 thru 0x037387 (0xe501 bytes) 000170 1784046608.178580 write 0x036b92 thru 0x03f1c8 (0x8637 bytes) 000173 1784046608.183665 read 0x01c641 thru 0x021d05 (0x56c5 bytes) 000174 1784046608.189178 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) 000175 1784046608.410879 read 0x02f1dd thru 0x038a88 (0x98ac bytes) 000176 1784046608.426716 read 0x01a28e thru 0x02391c (0x968f bytes) 000179 1784046608.487675 mapwrite 0x02d4bd thru 0x02e69d (0x11e1 bytes) 000180 1784046608.553261 write 0x000bf2 thru 0x0058b8 (0x4cc7 bytes) 000181 1784046608.555517 mapread 0x030c3f thru 0x03b4e0 (0xa8a2 bytes) 000183 1784046608.559874 read 0x0290a5 thru 0x029ad5 (0xa31 bytes) 000185 1784046608.562709 write 0x013a4e thru 0x01aef7 (0x74aa bytes) 000187 1784046608.567575 mapwrite 0x018fb4 thru 0x020ed8 (0x7f25 bytes) 000188 1784046608.650858 MIRROR_EXTEND to 2 mirrors 000189 1784046609.521248 mapread 0x039192 thru 0x03ffff (0x6e6e bytes) 000190 1784046609.574316 read 0x00daa8 thru 0x0191b7 (0xb710 bytes) 000191 1784046609.606711 mapread 0x005c3c thru 0x010b9d (0xaf62 bytes) 000192 1784046609.691435 read 0x021bf7 thru 0x026b11 (0x4f1b bytes) 000196 1784046609.822990 mapwrite 0x0196a6 thru 0x02625f (0xcbba bytes) 000197 1784046610.390504 MIRROR_RESYNC 2 mirrors 000199 1784046611.267497 mapwrite 0x03b4b6 thru 0x03ffff (0x4b4a bytes) 000200 1784046611.574977 mapwrite 0x02bb41 thru 0x0376d2 (0xbb92 bytes) 000204 1784046611.752175 read 0x006096 thru 0x0078d3 (0x183e bytes) 000205 1784046611.778786 read 0x034542 thru 0x03da56 (0x9515 bytes) 000208 1784046611.824153 mapwrite 0x00b258 thru 0x017a07 (0xc7b0 bytes) 000209 1784046611.987843 mapwrite 0x03d072 thru 0x03ffff (0x2f8e bytes) 000211 1784046612.067831 trunc from 0x040000 to 0x0239fd (0x1c602 bytes) 000212 1784046612.129472 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) 000213 1784046612.343701 mapwrite 0x019127 thru 0x01b94a (0x2824 bytes) 000214 1784046612.455016 read 0x0232f8 thru 0x0239fd (0x706 bytes) 000215 1784046612.458901 write 0x00ebd8 thru 0x01ad25 (0xc14e bytes) 000217 1784046612.478581 write 0x024790 thru 0x02f53b (0xadac bytes) 000218 1784046612.485636 mapwrite 0x037587 thru 0x03973c (0x21b6 bytes) 000219 1784046612.615620 read 0x022b85 thru 0x02ad61 (0x81dd bytes) 000220 1784046612.616600 write 0x0147e8 thru 0x01a34c (0x5b65 bytes) 000221 1784046612.619255 trunc from 0x03973d to 0x006fa0 (0x3279c bytes) 000222 1784046612.647267 mapwrite 0x016891 thru 0x022391 (0xbb01 bytes) 000223 1784046612.753532 mapread 0x0104f9 thru 0x013fda (0x3ae2 bytes) 000224 1784046612.755717 mapwrite 0x031b55 thru 0x03def5 (0xc3a1 bytes) 000225 1784046612.866876 mapread 0x0141cd thru 0x0239c0 (0xf7f4 bytes) 000227 1784046612.878918 read 0x00f39b thru 0x01b383 (0xbfe9 bytes) 000229 1784046612.882631 mapwrite 0x024c09 thru 0x03103f (0xc437 bytes) 000231 1784046612.969680 write 0x03d3d3 thru 0x03ffff (0x2c2d bytes) 000232 1784046612.970862 mapread 0x0018e6 thru 0x00f413 (0xdb2e bytes) 000233 1784046612.974529 mapwrite 0x0315ae thru 0x03d4a8 (0xbefb bytes) 000235 1784046613.062852 MIRROR_EXTEND to 2 mirrors 000236 1784046613.823857 write 0x03bef3 thru 0x03c631 (0x73f bytes) 000238 1784046613.875504 write 0x01ed02 thru 0x023d8b (0x508a bytes) 000239 1784046613.888688 mapwrite 0x01a21e thru 0x02062a (0x640d bytes) 000240 1784046613.972146 trunc from 0x040000 to 0x0231b6 (0x1ce49 bytes) 000243 1784046614.005783 trunc from 0x0231b7 to 0x02ad0e (0x7b58 bytes) 000244 1784046614.029864 trunc from 0x02ad0f to 0x02be6c (0x115e bytes) 000246 1784046614.056616 MIRROR_RESYNC 2 mirrors 000248 1784046614.981112 MIRROR_RESYNC 2 mirrors 000250 1784046615.435899 trunc from 0x02be6d to 0x02abeb (0x1281 bytes) 000251 1784046615.536180 mapwrite 0x02d400 thru 0x038c4d (0xb84e bytes) 000252 1784046615.655450 mapwrite 0x0315da thru 0x03f405 (0xde2c bytes) 000253 1784046615.724971 write 0x02e0b8 thru 0x0375ce (0x9517 bytes) 000254 1784046615.725945 read 0x014b7d thru 0x01edb4 (0xa238 bytes) 000255 1784046615.733407 mapwrite 0x00a2c8 thru 0x015a8e (0xb7c7 bytes) 000256 1784046615.817611 trunc from 0x03f406 to 0x01c16c (0x23299 bytes) 000257 1784046615.843020 write 0x010643 thru 0x01aed4 (0xa892 bytes) 000258 1784046615.845837 mapread 0x005c51 thru 0x00e036 (0x83e6 bytes) 000259 1784046615.848041 write 0x010115 thru 0x011c9d (0x1b89 bytes) 000260 1784046615.852494 mapread 0x01a7e9 thru 0x01c16c (0x1984 bytes) 000261 1784046615.854731 write 0x004474 thru 0x0115c5 (0xd152 bytes) 000262 1784046615.862677 MIRROR_RESYNC 2 mirrors 000263 1784046616.781323 write 0x015aaf thru 0x01db82 (0x80d4 bytes) 000265 1784046616.846879 trunc from 0x01db83 to 0x011559 (0xc629 bytes) 000267 1784046616.870153 MIRROR_RESYNC 2 mirrors 000268 1784046617.569449 mapread 0x00c8a5 thru 0x011559 (0x4cb5 bytes) 000270 1784046617.600962 MIRROR_RESYNC 2 mirrors 000271 1784046617.901920 read 0x00b3d9 thru 0x011559 (0x6181 bytes) 000274 1784046617.902733 write 0x004e00 thru 0x00d07b (0x827c bytes) 000275 1784046617.994594 trunc from 0x01155a to 0x0120bb (0xb62 bytes) 000276 1784046618.019889 mapwrite 0x029099 thru 0x030371 (0x72d9 bytes) 000277 1784046618.224203 trunc from 0x030372 to 0x022bc2 (0xd7af bytes) 000278 1784046618.258968 read 0x01eb64 thru 0x0221b8 (0x3655 bytes) 000279 1784046618.272399 read 0x01fd88 thru 0x022bc2 (0x2e3b bytes) 000280 1784046618.273090 read 0x01e6bc thru 0x022bc2 (0x4507 bytes) 000281 1784046618.273754 read 0x00138e thru 0x010cbd (0xf930 bytes) 000284 1784046618.279564 mapread 0x002662 thru 0x002f4b (0x8ea bytes) 000286 1784046618.284604 read 0x00f8b6 thru 0x01b642 (0xbd8d bytes) 000288 1784046618.286469 write 0x01f866 thru 0x02cab9 (0xd254 bytes) 000290 1784046618.290516 mapwrite 0x0349cc thru 0x036b20 (0x2155 bytes) 000291 1784046618.404789 read 0x02193b thru 0x030054 (0xe71a bytes) 000293 1784046618.412737 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x36b21): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36b21): Mirror 2: 0x46abe7e3 (stale) CRC-32 checksum value for chunk [0, 0x36b21): Mirror 1: 0xc26dc657 Mirror 3: 0xc26dc657 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36b21: skipped 000296 1784046619.256289 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x36b21): Mirror 2: 0x46abe7e3 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x36b21): Mirror 2: 0x46abe7e3 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x36b21: skipped 000297 1784046619.570114 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) 000298 1784046619.715422 mapread 0x01ff49 thru 0x02146c (0x1524 bytes) 000299 1784046619.737497 trunc from 0x036b21 to 0x002c05 (0x33f1b bytes) 000300 1784046619.797572 mapread 0x000feb thru 0x002c05 (0x1c1b bytes) 000303 1784046619.805549 read 0x00154c thru 0x002976 (0x142b bytes) 000306 1784046619.809651 mapread 0x001f35 thru 0x002c05 (0xcd1 bytes) 000307 1784046619.811504 trunc from 0x002c06 to 0x020f76 (0x1e371 bytes) 000308 1784046619.832981 mapread 0x01ea42 thru 0x020f76 (0x2535 bytes) 000311 1784046619.845002 mapread 0x018bc8 thru 0x019454 (0x88d bytes) 000314 1784046619.848767 write 0x02e117 thru 0x0360ca (0x7fb4 bytes) 000315 1784046619.850575 mapread 0x0230ce thru 0x028e04 (0x5d37 bytes) 000316 1784046619.858870 trunc from 0x0360cb to 0x017d42 (0x1e388 bytes) 000318 1784046619.890421 write 0x03643f thru 0x03c334 (0x5ef6 bytes) 000321 1784046619.892340 trunc from 0x03c335 to 0x0211f5 (0x1b13f bytes) 000323 1784046619.911990 trunc from 0x0211f6 to 0x02cf87 (0xbd92 bytes) 000324 1784046619.933731 write 0x037188 thru 0x03ffff (0x8e78 bytes) 000326 1784046619.936544 write 0x0042e5 thru 0x004509 (0x225 bytes) 000327 1784046619.943543 mapwrite 0x01df4c thru 0x021444 (0x34f9 bytes) 000332 1784046620.020340 trunc from 0x040000 to 0x03a155 (0x5eaa bytes) 000333 1784046620.048998 read 0x00b16f thru 0x010e55 (0x5ce7 bytes) 000335 1784046620.063813 trunc from 0x03a156 to 0x03c534 (0x23df bytes) 000336 1784046620.075645 trunc from 0x03c535 to 0x0291e5 (0x1334f bytes) 000337 1784046620.091200 read 0x015c2f thru 0x01be6e (0x6240 bytes) 000338 1784046620.092681 trunc from 0x0291e6 to 0x0174b7 (0x11d2e bytes) 000339 1784046620.113475 write 0x019fe3 thru 0x028406 (0xe424 bytes) 000340 1784046620.114985 mapread 0x01d8e5 thru 0x027cd2 (0xa3ee bytes) 000342 1784046620.118792 trunc from 0x028407 to 0x011114 (0x172f2 bytes) 000343 1784046620.152900 mapread 0x0026e4 thru 0x011114 (0xea31 bytes) 000344 1784046620.159969 mapread 0x008378 thru 0x011114 (0x8d9d bytes) 000345 1784046620.163899 mapread 0x00ac16 thru 0x00fb8d (0x4f78 bytes) 000346 1784046620.176117 mapread 0x004710 thru 0x010ee4 (0xc7d5 bytes) 000348 1784046620.182345 trunc from 0x011115 to 0x02ad5a (0x19c46 bytes) 000349 1784046620.219196 mapwrite 0x039023 thru 0x03be9a (0x2e78 bytes) 000351 1784046620.430415 trunc from 0x03be9b to 0x017ebb (0x23fdf bytes) 000352 1784046620.484861 write 0x00a5f1 thru 0x014599 (0x9fa9 bytes) 000355 1784046620.491223 write 0x0070c7 thru 0x0076a6 (0x5e0 bytes) 000356 1784046620.495183 write 0x036d5d thru 0x036e61 (0x105 bytes) 000363 1784046620.503422 mapwrite 0x026feb thru 0x030c99 (0x9caf bytes) 000364 1784046620.654884 mapwrite 0x0326e9 thru 0x03e840 (0xc158 bytes) 000365 1784046620.814873 write 0x0352f9 thru 0x03ffff (0xad07 bytes) 000366 1784046620.816507 trunc from 0x040000 to 0x00ee93 (0x3116c bytes) 000368 1784046620.831668 mapwrite 0x033bd4 thru 0x03ebba (0xafe7 bytes) 000369 1784046620.967959 MIRROR_EXTEND to 2 mirrors 000370 1784046621.772684 trunc from 0x03ebbb to 0x028ba8 (0x16012 bytes) 000371 1784046621.908931 read 0x028b02 thru 0x028ba8 (0xa7 bytes) 000372 1784046621.923686 write 0x00bdca thru 0x014989 (0x8bc0 bytes) 000374 1784046621.949264 write 0x03475a thru 0x03acdf (0x6586 bytes) 000375 1784046621.952793 mapwrite 0x03d472 thru 0x03ffff (0x2b8e bytes) 000377 1784046622.085734 mapread 0x00b87d thru 0x012f5a (0x76de bytes) 000378 1784046622.088053 read 0x03620e thru 0x03ffff (0x9df2 bytes) 000379 1784046622.088906 trunc from 0x040000 to 0x02306f (0x1cf90 bytes) 000380 1784046622.119146 mapread 0x00cf93 thru 0x016a88 (0x9af6 bytes) 000381 1784046622.129597 mapread 0x0192e1 thru 0x02306f (0x9d8f bytes) 000385 1784046622.136239 mapwrite 0x02450b thru 0x02fa37 (0xb52d bytes) 000387 1784046622.250250 mapread 0x02ad14 thru 0x02ad84 (0x71 bytes) 000388 1784046622.251433 trunc from 0x02fa38 to 0x0015e7 (0x2e450 bytes) 000391 1784046622.281867 mapwrite 0x025511 thru 0x0348f7 (0xf3e7 bytes) 000394 1784046622.445474 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) 000395 1784046622.577950 read 0x01fbdc thru 0x02517f (0x55a4 bytes) 000396 1784046622.590024 mapwrite 0x008ff1 thru 0x015b67 (0xcb77 bytes) 000398 1784046622.674513 mapread 0x008818 thru 0x01418e (0xb977 bytes) 000399 1784046622.675752 write 0x029e14 thru 0x033dff (0x9fec bytes) 000401 1784046622.686608 mapread 0x00d027 thru 0x00e0de (0x10b8 bytes) 000402 1784046622.687634 mapread 0x00c861 thru 0x014f3d (0x86dd bytes) truncating to largest ever: 0x3e519 000403 1784046622.690314 trunc from 0x0348f8 to 0x03e518 (0x9c21 bytes) 000407 1784046622.715152 mapread 0x02ec45 thru 0x03ca8e (0xde4a bytes) 000409 1784046622.724144 mapread 0x03bf50 thru 0x03e518 (0x25c9 bytes) 000410 1784046622.730486 write 0x00f647 thru 0x014095 (0x4a4f bytes) 000413 1784046622.737520 write 0x03b287 thru 0x03ffff (0x4d79 bytes) 000414 1784046622.739532 mapwrite 0x02a681 thru 0x02e519 (0x3e99 bytes) 000417 1784046622.840077 trunc from 0x040000 to 0x015339 (0x2acc6 bytes) 000421 1784046622.875540 MIRROR_EXTEND to 2 mirrors 000422 1784046623.679666 mapread 0x00c86a thru 0x00ff7c (0x3713 bytes) 000423 1784046623.729307 read 0x00a61e thru 0x00c5f9 (0x1fdc bytes) 000425 1784046623.761277 trunc from 0x01533a to 0x03399a (0x1e661 bytes) 000426 1784046623.847927 trunc from 0x03399b to 0x008615 (0x2b385 bytes) 000427 1784046623.876872 trunc from 0x008616 to 0x039bd2 (0x315bd bytes) 000429 1784046623.900688 write 0x00137e thru 0x00b7e3 (0xa466 bytes) 000431 1784046623.913656 read 0x00f031 thru 0x01e886 (0xf856 bytes) 000433 1784046623.950757 mapwrite 0x0213a2 thru 0x02c8b9 (0xb518 bytes) 000437 1784046624.108018 mapwrite 0x0010e7 thru 0x007176 (0x6090 bytes) 000438 1784046624.158760 read 0x026d28 thru 0x02d376 (0x664f bytes) 000441 1784046624.160296 mapwrite 0x008c3c thru 0x00caa3 (0x3e68 bytes) 000442 1784046624.257282 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) 000445 1784046624.448495 mapwrite 0x03b415 thru 0x03ffff (0x4beb bytes) 000447 1784046624.565145 read 0x02fa41 thru 0x035f81 (0x6541 bytes) 000448 1784046624.585425 mapread 0x01e3a2 thru 0x02c88f (0xe4ee bytes) 000450 1784046624.663627 mapwrite 0x02f83b thru 0x031b2d (0x22f3 bytes) 000452 1784046624.725433 mapread 0x00a97b thru 0x01521a (0xa8a0 bytes) 000454 1784046624.751149 mapread 0x0052a6 thru 0x006a07 (0x1762 bytes) 000456 1784046624.754417 read 0x00e44a thru 0x016ae1 (0x8698 bytes) 000457 1784046624.760158 mapwrite 0x007c43 thru 0x00d6c1 (0x5a7f bytes) 000458 1784046624.805498 write 0x02b2a2 thru 0x036ce6 (0xba45 bytes) 000459 1784046624.808645 trunc from 0x040000 to 0x00efd6 (0x31029 bytes) 000460 1784046624.833524 trunc from 0x00efd7 to 0x012f5e (0x3f88 bytes) 000464 1784046624.855096 mapread 0x0029af thru 0x0090a7 (0x66f9 bytes) 000465 1784046624.867150 trunc from 0x012f5f to 0x00f650 (0x390e bytes) 000468 1784046624.894944 mapwrite 0x002e78 thru 0x008114 (0x529d bytes) 000469 1784046624.950048 mapwrite 0x0353a6 thru 0x03ffff (0xac5a bytes) 000470 1784046625.045218 read 0x009ff4 thru 0x011b9a (0x7ba7 bytes) 000473 1784046625.061495 mapwrite 0x013597 thru 0x01af2a (0x7994 bytes) 000474 1784046625.143033 mapwrite 0x00f46a thru 0x018111 (0x8ca8 bytes) 000483 1784046625.204559 read 0x026a00 thru 0x02d1a2 (0x67a3 bytes) 000485 1784046625.207072 mapread 0x028ed5 thru 0x030d3b (0x7e67 bytes) 000487 1784046625.210179 mapwrite 0x006f7d thru 0x01092f (0x99b3 bytes) 000488 1784046625.277636 mapread 0x03fad0 thru 0x03ffff (0x530 bytes) 000492 1784046625.282188 write 0x015343 thru 0x01715b (0x1e19 bytes) 000493 1784046625.283707 write 0x02b0cf thru 0x02ddab (0x2cdd bytes) 000495 1784046625.286494 read 0x0010a1 thru 0x005248 (0x41a8 bytes) 000496 1784046625.290766 write 0x02e40a thru 0x034ad6 (0x66cd bytes) 000497 1784046625.292712 trunc from 0x040000 to 0x01e503 (0x21afc bytes) 000498 1784046625.314613 write 0x004dd9 thru 0x00e8a2 (0x9aca bytes) 000503 1784046625.320918 trunc from 0x01e504 to 0x009260 (0x152a3 bytes) 000505 1784046625.347902 read 0x007902 thru 0x009260 (0x195f bytes) 000507 1784046625.350089 mapread 0x0076d8 thru 0x009260 (0x1b89 bytes) 000508 1784046625.351612 mapread 0x004564 thru 0x0078ce (0x336b bytes) 000511 1784046625.354857 mapwrite 0x02b323 thru 0x0308f6 (0x55d4 bytes) 000514 1784046625.449216 write 0x02a754 thru 0x039745 (0xeff2 bytes) 000516 1784046625.453472 mapread 0x0073dd thru 0x01601b (0xec3f bytes) 000519 1784046625.457882 mapwrite 0x0337d1 thru 0x039713 (0x5f43 bytes) 000520 1784046625.520007 mapwrite 0x0171c9 thru 0x02003d (0x8e75 bytes) 000522 1784046625.575984 trunc from 0x039746 to 0x029c83 (0xfac2 bytes) 000524 1784046625.592650 mapread 0x00550a thru 0x00f9fd (0xa4f4 bytes) 000526 1784046625.594932 mapread 0x02945c thru 0x029c83 (0x828 bytes) 000529 1784046625.598859 read 0x0109e1 thru 0x018ff6 (0x8616 bytes) 000530 1784046625.599789 mapwrite 0x01b7b9 thru 0x02aa6a (0xf2b2 bytes) 000536 1784046625.679539 mapwrite 0x02ddf5 thru 0x03d235 (0xf441 bytes) 000539 1784046625.763114 read 0x037f39 thru 0x03d235 (0x52fd bytes) 000540 1784046625.764449 mapwrite 0x029fb6 thru 0x02aa48 (0xa93 bytes) 000541 1784046625.800691 mapread 0x025998 thru 0x0322fb (0xc964 bytes) 000542 1784046625.803186 mapwrite 0x024f79 thru 0x0261c8 (0x1250 bytes) 000543 1784046625.853588 trunc from 0x03d236 to 0x01bd27 (0x2150e bytes) 000548 1784046625.875739 mapread 0x01448f thru 0x01bd27 (0x7899 bytes) 000549 1784046625.878095 mapwrite 0x0233cb thru 0x02c2e2 (0x8f18 bytes) 000550 1784046625.957937 mapread 0x002562 thru 0x00760c (0x50ab bytes) 000551 1784046625.959172 trunc from 0x02c2e3 to 0x02a8aa (0x1a38 bytes) 000552 1784046625.976599 write 0x02803b thru 0x02d25c (0x5222 bytes) 000558 1784046625.986206 write 0x039d6e thru 0x03ffff (0x6292 bytes) 000561 1784046625.988719 read 0x03b43c thru 0x03ffff (0x4bc4 bytes) 000565 1784046625.992607 write 0x01022c thru 0x017b7f (0x7954 bytes) 000566 1784046625.994231 write 0x03b2b9 thru 0x03e74d (0x3495 bytes) 000567 1784046625.997415 mapread 0x015963 thru 0x015f1a (0x5b8 bytes) 000568 1784046625.998758 trunc from 0x040000 to 0x004f0a (0x3b0f5 bytes) 000573 1784046626.021627 write 0x01bffa thru 0x0219f6 (0x59fd bytes) 000577 1784046626.026935 mapread 0x0000b7 thru 0x0061e0 (0x612a bytes) 000578 1784046626.035139 trunc from 0x0219f7 to 0x0050f6 (0x1c900 bytes) 000579 1784046626.062199 trunc from 0x0050f7 to 0x006989 (0x1893 bytes) 000580 1784046626.083841 trunc from 0x00698a to 0x03954b (0x32bc2 bytes) 000581 1784046626.105076 mapwrite 0x0319d8 thru 0x03ffff (0xe628 bytes) 000584 1784046626.198889 MIRROR_EXTEND to 2 mirrors 000585 1784046626.779810 mapwrite 0x02348f thru 0x02857c (0x50ee bytes) 000587 1784046626.998570 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x44a21abd (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x54f0cc63 Mirror 3: 0x54f0cc63 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000589 1784046627.830823 mapwrite 0x028da8 thru 0x02dbb0 (0x4e09 bytes) 000592 1784046628.007482 write 0x03eb0f thru 0x03ffff (0x14f1 bytes) 000594 1784046628.015044 mapwrite 0x02f9b3 thru 0x03b1e4 (0xb832 bytes) 000597 1784046628.079278 trunc from 0x040000 to 0x02e773 (0x1188c bytes) 000603 1784046628.113082 read 0x019e2d thru 0x01e79c (0x4970 bytes) 000604 1784046628.123567 mapwrite 0x030d75 thru 0x032dc4 (0x2050 bytes) 000607 1784046628.184738 mapwrite 0x0372b4 thru 0x038639 (0x1386 bytes) 000608 1784046628.248432 trunc from 0x03863a to 0x006a9d (0x31b9c bytes) 000611 1784046628.275740 write 0x03bdc8 thru 0x03ffff (0x4238 bytes) 000612 1784046628.278144 mapwrite 0x0277ce thru 0x02b161 (0x3994 bytes) 000613 1784046628.342759 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x44a21abd (stale) Mirror 3: 0x54f0cc63 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x40e86af5 Mirror 4: 0x40e86af5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000614 1784046629.284744 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x54f0cc63 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x54f0cc63 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x40e86af5 Mirror 4: 0x40e86af5 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000615 1784046630.120189 write 0x01b343 thru 0x01fc67 (0x4925 bytes) 000616 1784046630.298058 mapread 0x00e385 thru 0x0158a0 (0x751c bytes) 000618 1784046630.332287 write 0x03a6cf thru 0x03bd85 (0x16b7 bytes) 000619 1784046630.352057 mapread 0x034797 thru 0x0377f5 (0x305f bytes) 000621 1784046630.361797 write 0x01ccf8 thru 0x02cba0 (0xfea9 bytes) 000622 1784046630.370568 trunc from 0x040000 to 0x0070d4 (0x38f2b bytes) 000625 1784046630.434188 trunc from 0x0070d5 to 0x03295e (0x2b88a bytes) 000626 1784046630.450973 mapread 0x02e357 thru 0x03295e (0x4608 bytes) 000627 1784046630.469146 mapwrite 0x03d882 thru 0x03ffff (0x277e bytes) 000628 1784046630.576940 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x54f0cc63 (stale) Mirror 4: 0x40e86af5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x742f1653 Mirror 5: 0x742f1653 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000629 1784046632.023547 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x54f0cc63 (stale) Mirror 4: 0x40e86af5 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x742f1653 Mirror 5: 0x742f1653 Mirror 6: 0x742f1653 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000630 1784046633.015671 mapread 0x0242ab thru 0x026b64 (0x28ba bytes) 000633 1784046633.052037 mapwrite 0x034278 thru 0x03e216 (0x9f9f bytes) 000634 1784046633.392215 read 0x0286d9 thru 0x028a97 (0x3bf bytes) 000635 1784046633.419337 read 0x01358a thru 0x01f5e0 (0xc057 bytes) 000639 1784046633.453159 read 0x026f94 thru 0x02e687 (0x76f4 bytes) 000643 1784046633.459251 mapwrite 0x01cdd8 thru 0x02136c (0x4595 bytes) 000644 1784046633.578869 trunc from 0x040000 to 0x024c43 (0x1b3bc bytes) 000645 1784046633.621261 read 0x005887 thru 0x011642 (0xbdbc bytes) 000648 1784046633.623197 mapread 0x0200d2 thru 0x024c43 (0x4b72 bytes) 000650 1784046633.639848 MIRROR_RESYNC 5 mirrors 000655 1784046634.602991 MIRROR_EXTEND to 6 mirrors 000656 1784046635.464771 mapwrite 0x00b599 thru 0x00bd10 (0x778 bytes) 000658 1784046635.738482 MIRROR_SPLIT mirror 1 to 5 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x24c44): Mirror 4: 0x8fdc4f1f (stale) Mirror 5: 0x8fdc4f1f (stale) Mirror 6: 0x8fdc4f1f (stale) Mirror 7: 0x8fdc4f1f (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x24c44): Mirror 4: 0x8fdc4f1f (stale) Mirror 5: 0x8fdc4f1f (stale) Mirror 6: 0x8fdc4f1f (stale) Mirror 7: 0x8fdc4f1f (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x24c44: skipped 000659 1784046636.277557 read 0x007e22 thru 0x013424 (0xb603 bytes) 000660 1784046636.289100 trunc from 0x024c44 to 0x03051e (0xb8db bytes) 000662 1784046636.309558 mapread 0x016201 thru 0x025f0a (0xfd0a bytes) 000666 1784046636.340185 mapwrite 0x0223e5 thru 0x02c3f9 (0xa015 bytes) 000667 1784046636.401888 mapwrite 0x039ded thru 0x03ffff (0x6213 bytes) 000668 1784046636.509401 read 0x0359de thru 0x03ab4e (0x5171 bytes) 000670 1784046636.512684 write 0x014b54 thru 0x0220ef (0xd59c bytes) 000671 1784046636.516164 mapread 0x02d70c thru 0x0346a8 (0x6f9d bytes) 000672 1784046636.530569 mapwrite 0x027a83 thru 0x035916 (0xde94 bytes) 000673 1784046636.594101 trunc from 0x040000 to 0x025f92 (0x1a06d bytes) 000676 1784046636.618609 write 0x02ab1d thru 0x02d592 (0x2a76 bytes) 000677 1784046636.620790 trunc from 0x02d593 to 0x00c064 (0x2152e bytes) 000680 1784046636.648559 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0xc065): Mirror 5: 0xe5bd6644 (stale) Mirror 6: 0xe5bd6644 (stale) Mirror 7: 0xe5bd6644 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xc065): Mirror 5: 0xe5bd6644 (stale) Mirror 6: 0xe5bd6644 (stale) Mirror 7: 0xe5bd6644 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xc065: skipped 000682 1784046637.021021 write 0x00abd2 thru 0x00edb6 (0x41e5 bytes) 000686 1784046637.034634 write 0x00fa10 thru 0x01ebf4 (0xf1e5 bytes) 000687 1784046637.038186 mapwrite 0x01851e thru 0x019f9f (0x1a82 bytes) 000690 1784046637.108642 write 0x012d61 thru 0x0163c8 (0x3668 bytes) 000691 1784046637.111065 read 0x00d4fa thru 0x012d1f (0x5826 bytes) truncating to largest ever: 0x3fae0 000693 1784046637.114119 trunc from 0x01ebf5 to 0x03fadf (0x20eeb bytes) 000694 1784046637.139780 mapwrite 0x0241e5 thru 0x025660 (0x147c bytes) 000696 1784046637.237757 mapread 0x01f2ec thru 0x020e7f (0x1b94 bytes) 000697 1784046637.267544 write 0x03f086 thru 0x03ffff (0xf7a bytes) 000699 1784046637.276537 mapwrite 0x0120e4 thru 0x0162e5 (0x4202 bytes) 000700 1784046637.380689 write 0x0253d2 thru 0x02da38 (0x8667 bytes) 000701 1784046637.387198 write 0x02fd16 thru 0x03655f (0x684a bytes) 000702 1784046637.398930 mapread 0x02371a thru 0x02a6ee (0x6fd5 bytes) 000704 1784046637.400859 mapwrite 0x01420d thru 0x01ac90 (0x6a84 bytes) 000705 1784046637.474562 mapwrite 0x03fa1f thru 0x03ffff (0x5e1 bytes) 000706 1784046637.522194 write 0x0073f7 thru 0x00943c (0x2046 bytes) 000708 1784046637.535592 mapwrite 0x00011a thru 0x00597c (0x5863 bytes) 000709 1784046637.608108 write 0x026356 thru 0x029da3 (0x3a4e bytes) 000710 1784046637.610339 trunc from 0x040000 to 0x02d4b5 (0x12b4a bytes) 000712 1784046637.638652 write 0x02e0b6 thru 0x03d316 (0xf261 bytes) 000714 1784046637.642986 mapread 0x016af3 thru 0x01a0cb (0x35d9 bytes) 000717 1784046637.646620 mapread 0x00e424 thru 0x00fd5d (0x193a bytes) 000718 1784046637.649046 write 0x02f073 thru 0x03561f (0x65ad bytes) 000719 1784046637.652092 write 0x02bb3e thru 0x037f59 (0xc41c bytes) 000720 1784046637.653925 mapread 0x01cc80 thru 0x020aef (0x3e70 bytes) 000721 1784046637.656807 mapread 0x02715b thru 0x0368fb (0xf7a1 bytes) 000723 1784046637.660637 trunc from 0x03d317 to 0x030ca3 (0xc673 bytes) 000726 1784046637.687471 mapwrite 0x0211da thru 0x022d90 (0x1bb7 bytes) 000728 1784046637.770647 write 0x025136 thru 0x03404d (0xef18 bytes) 000729 1784046637.777642 mapread 0x013fd4 thru 0x018828 (0x4855 bytes) 000730 1784046637.781451 mapread 0x02abe6 thru 0x033258 (0x8673 bytes) 000732 1784046637.785552 mapwrite 0x019ad1 thru 0x01baf0 (0x2020 bytes) 000734 1784046637.849139 write 0x03e7f3 thru 0x03ffff (0x180d bytes) 000736 1784046637.859250 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1bc1137 (stale) Mirror 7: 0x1bc1137 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1bc1137 (stale) Mirror 7: 0x1bc1137 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000737 1784046638.291614 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1bc1137 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 6: 0x1bc1137 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000738 1784046638.556840 mapwrite 0x012136 thru 0x01865a (0x6525 bytes) 000739 1784046638.618095 read 0x013b58 thru 0x01f72f (0xbbd8 bytes) 000740 1784046638.627187 mapwrite 0x002baf thru 0x00579c (0x2bee bytes) 000743 1784046638.686154 mapread 0x016e3c thru 0x023c2b (0xcdf0 bytes) 000744 1784046638.700077 mapread 0x00f60e thru 0x014b91 (0x5584 bytes) 000747 1784046638.703463 trunc from 0x040000 to 0x02b744 (0x148bb bytes) 000749 1784046638.726914 write 0x02ca12 thru 0x03757a (0xab69 bytes) 000750 1784046638.730081 mapread 0x018753 thru 0x0187db (0x89 bytes) 000751 1784046638.732924 write 0x03c7c4 thru 0x03ffff (0x383c bytes) 000752 1784046638.735528 trunc from 0x040000 to 0x036519 (0x9ae6 bytes) 000754 1784046638.762124 read 0x0272ff thru 0x02a2ee (0x2ff0 bytes) 000755 1784046638.777085 mapread 0x0227cf thru 0x030743 (0xdf75 bytes) 000757 1784046638.784683 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) 000758 1784046638.931269 trunc from 0x03651a to 0x019bc0 (0x1c959 bytes) 000759 1784046638.955004 mapwrite 0x014ade thru 0x01611a (0x163d bytes) 000761 1784046639.027982 mapwrite 0x01c715 thru 0x01dc3e (0x152a bytes) 000762 1784046639.089030 mapwrite 0x026b78 thru 0x032980 (0xbe09 bytes) 000763 1784046639.174388 read 0x0043b5 thru 0x008e87 (0x4ad3 bytes) 000764 1784046639.184157 mapread 0x01f0ea thru 0x0218b0 (0x27c7 bytes) 000768 1784046639.187946 trunc from 0x032981 to 0x02ea30 (0x3f50 bytes) 000770 1784046639.210187 mapwrite 0x03d3cf thru 0x03ffff (0x2c31 bytes) 000771 1784046639.271193 mapwrite 0x03b0ab thru 0x03ffff (0x4f55 bytes) 000774 1784046639.319217 trunc from 0x040000 to 0x02a17f (0x15e80 bytes) 000775 1784046639.340160 mapread 0x005925 thru 0x006925 (0x1001 bytes) 000777 1784046639.343135 mapwrite 0x03253d thru 0x039c26 (0x76ea bytes) 000779 1784046639.423482 write 0x002246 thru 0x006620 (0x43db bytes) 000780 1784046639.430312 mapread 0x00113e thru 0x005107 (0x3fca bytes) 000781 1784046639.441639 mapread 0x00dc90 thru 0x013304 (0x5675 bytes) 000788 1784046639.447088 trunc from 0x039c27 to 0x03f9f4 (0x5dce bytes) 000790 1784046639.475597 MIRROR_EXTEND to 2 mirrors 000792 1784046640.126686 mapwrite 0x025ac5 thru 0x02f83c (0x9d78 bytes) 000795 1784046640.396496 write 0x02bd3b thru 0x03b424 (0xf6ea bytes) 000798 1784046640.411645 read 0x038f6e thru 0x03f9f4 (0x6a87 bytes) 000799 1784046640.414484 write 0x00efe2 thru 0x01af1d (0xbf3c bytes) 000802 1784046640.435246 write 0x027ed9 thru 0x02dc50 (0x5d78 bytes) 000803 1784046640.439132 mapread 0x039bbe thru 0x03f9f4 (0x5e37 bytes) 000804 1784046640.440972 write 0x03753b thru 0x03ffff (0x8ac5 bytes) 000805 1784046640.443182 mapread 0x034c7b thru 0x0376b7 (0x2a3d bytes) 000807 1784046640.445777 trunc from 0x040000 to 0x0110fb (0x2ef04 bytes) 000808 1784046640.497563 trunc from 0x0110fc to 0x02a4c1 (0x193c6 bytes) 000809 1784046640.529178 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x2a4c2): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a4c2): Mirror 2: 0x815b7984 (stale) CRC-32 checksum value for chunk [0, 0x2a4c2): Mirror 1: 0xf8d58623 Mirror 3: 0xf8d58623 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a4c2: skipped 000810 1784046641.333554 write 0x01783d thru 0x0214e5 (0x9ca9 bytes) 000811 1784046641.437181 read 0x00865c thru 0x00f23c (0x6be1 bytes) 000812 1784046641.452324 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x2a4c2): Mirror 2: 0x815b7984 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x2a4c2): Mirror 2: 0x815b7984 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x2a4c2: skipped 000814 1784046641.826546 mapwrite 0x02d65a thru 0x02fcf1 (0x2698 bytes) 000816 1784046642.018230 write 0x034bb0 thru 0x03ffff (0xb450 bytes) 000817 1784046642.023440 write 0x035669 thru 0x03cdb6 (0x774e bytes) 000820 1784046642.035274 mapwrite 0x00c0a0 thru 0x00e1bb (0x211c bytes) 000821 1784046642.131196 mapread 0x028583 thru 0x032c06 (0xa684 bytes) 000823 1784046642.152663 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0xb08de191 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x127f17cf Mirror 3: 0x127f17cf lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000825 1784046643.178703 read 0x01f0c2 thru 0x02b69c (0xc5db bytes) 000827 1784046643.196018 write 0x0004cf thru 0x004c90 (0x47c2 bytes) 000828 1784046643.298138 mapread 0x02ef66 thru 0x032f94 (0x402f bytes) 000831 1784046643.329199 MIRROR_SPLIT mirror 1 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x127f17cf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 3: 0x127f17cf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000835 1784046643.587116 write 0x00fead thru 0x010ba4 (0xcf8 bytes) 000836 1784046643.599776 read 0x02cb8a thru 0x038d7f (0xc1f6 bytes) 000837 1784046643.616041 MIRROR_SPLIT mirror 1 to 1 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) 000839 1784046643.754039 write 0x0043f4 thru 0x013348 (0xef55 bytes) 000840 1784046643.766951 mapwrite 0x013eaf thru 0x01e93e (0xaa90 bytes) 000841 1784046643.858913 write 0x02649c thru 0x02ddf7 (0x795c bytes) 000842 1784046643.871114 read 0x01cf79 thru 0x01f4cf (0x2557 bytes) 000843 1784046643.877773 mapwrite 0x00cf96 thru 0x019c02 (0xcc6d bytes) 000844 1784046643.929438 write 0x025799 thru 0x02903a (0x38a2 bytes) 000846 1784046643.936355 mapread 0x0148ea thru 0x018714 (0x3e2b bytes) 000847 1784046643.938089 write 0x0246fe thru 0x0283c1 (0x3cc4 bytes) 000848 1784046643.943836 trunc from 0x040000 to 0x00dd9d (0x32262 bytes) 000852 1784046643.992894 trunc from 0x00dd9e to 0x012694 (0x48f7 bytes) 000853 1784046644.008943 mapwrite 0x025276 thru 0x02fa7f (0xa80a bytes) 000854 1784046644.116496 mapwrite 0x006680 thru 0x00961f (0x2fa0 bytes) 000855 1784046644.148284 MIRROR_EXTEND to 2 mirrors 000862 1784046645.034197 mapwrite 0x02d0ba thru 0x03021a (0x3161 bytes) 000863 1784046645.274296 write 0x012af4 thru 0x017815 (0x4d22 bytes) 000864 1784046645.289536 write 0x03e7e2 thru 0x03ffff (0x181e bytes) 000865 1784046645.296155 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4f86b994 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x51c167f3 Mirror 3: 0x51c167f3 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000866 1784046646.311500 trunc from 0x040000 to 0x034a72 (0xb58d bytes) 000867 1784046646.484565 MIRROR_SPLIT mirror 2 to 2 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x34a73): Mirror 2: 0xcad4ec25 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x34a73): Mirror 2: 0xcad4ec25 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x34a73: skipped 000869 1784046646.851287 trunc from 0x034a73 to 0x00d8a7 (0x271cb bytes) 000871 1784046646.876186 read 0x002e09 thru 0x00330c (0x504 bytes) 000873 1784046646.902634 mapread 0x00658a thru 0x0085e0 (0x2057 bytes) 000874 1784046646.944860 read 0x00649f thru 0x00d8a7 (0x7409 bytes) 000878 1784046646.952319 read 0x008ab9 thru 0x00d8a7 (0x4def bytes) 000879 1784046646.958463 MIRROR_EXTEND to 3 mirrors CRC-32 checksum value for chunk [0, 0xd8a8): Mirror 2: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0xd8a8): Mirror 2: 0x6ab6e5cb (stale) CRC-32 checksum value for chunk [0, 0xd8a8): Mirror 1: 0x6ab6e5cb Mirror 3: 0x6ab6e5cb lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0xd8a8: skipped 000880 1784046647.915708 mapread 0x0020dc thru 0x00aa21 (0x8946 bytes) 000881 1784046647.971951 mapwrite 0x02316b thru 0x02efc4 (0xbe5a bytes) 000884 1784046648.155488 write 0x02aa3e thru 0x02f900 (0x4ec3 bytes) 000885 1784046648.157858 trunc from 0x02f901 to 0x008d29 (0x26bd7 bytes) 000886 1784046648.183255 mapwrite 0x032012 thru 0x03ffff (0xdfee bytes) 000888 1784046648.313013 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: (stale - read error) Mirror 3: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 2: 0x4f86b994 (stale) Mirror 3: 0x7b6c45ef (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x701fe643 Mirror 4: 0x701fe643 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000889 1784046649.264452 read 0x02cac5 thru 0x0319c6 (0x4f02 bytes) 000890 1784046649.278637 write 0x0155cf thru 0x021732 (0xc164 bytes) 000891 1784046649.340054 mapread 0x03e5bb thru 0x03f67e (0x10c4 bytes) 000892 1784046649.356121 mapwrite 0x02667f thru 0x027d84 (0x1706 bytes) 000893 1784046649.418288 mapwrite 0x02c92c thru 0x03bf50 (0xf625 bytes) 000897 1784046649.495147 trunc from 0x040000 to 0x03d72c (0x28d3 bytes) 000900 1784046649.539940 write 0x011f64 thru 0x01cc74 (0xad11 bytes) 000903 1784046649.547344 mapread 0x0062eb thru 0x0149b2 (0xe6c8 bytes) 000904 1784046649.562273 write 0x03f42b thru 0x03ffff (0xbd5 bytes) 000907 1784046649.565034 read 0x006e22 thru 0x01549a (0xe679 bytes) 000908 1784046649.565948 mapwrite 0x03b9ce thru 0x03ffff (0x4632 bytes) truncating to largest ever: 0x3faef 000909 1784046649.618546 trunc from 0x040000 to 0x03faee (0x511 bytes) 000910 1784046649.647615 read 0x039630 thru 0x03faee (0x64bf bytes) 000912 1784046649.648882 write 0x0222bd thru 0x028730 (0x6474 bytes) 000913 1784046649.660540 MIRROR_EXTEND to 5 mirrors CRC-32 checksum value for chunk [0, 0x3faef): Mirror 2: (stale - read error) Mirror 3: (stale - read error) Mirror 4: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3faef): Mirror 2: 0xfa2d0598 (stale) Mirror 3: 0x9ac642fa (stale) Mirror 4: 0x7b926e49 (stale) CRC-32 checksum value for chunk [0, 0x3faef): Mirror 1: 0x46abdce9 Mirror 5: 0x46abdce9 lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3faef: skipped 000914 1784046650.920491 mapread 0x03ece3 thru 0x03faee (0xe0c bytes) 000916 1784046650.939140 mapread 0x02af38 thru 0x03520e (0xa2d7 bytes) 000918 1784046650.960825 read 0x0379f6 thru 0x03faee (0x80f9 bytes) 000920 1784046650.967660 write 0x01ba28 thru 0x028c05 (0xd1de bytes) 000921 1784046651.079603 mapread 0x03ceda thru 0x03faee (0x2c15 bytes) 000924 1784046651.122987 write 0x0230de thru 0x031325 (0xe248 bytes) 000925 1784046651.137758 write 0x014b9e thru 0x01dca6 (0x9109 bytes) 000926 1784046651.154556 mapread 0x01cec1 thru 0x01fc8f (0x2dcf bytes) 000928 1784046651.171887 MIRROR_SPLIT mirror 1 to 4 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3faef): Mirror 3: 0x9ac642fa (stale) Mirror 4: 0x7b926e49 (stale) Mirror 5: 0x46abdce9 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3faef): Mirror 3: 0x9ac642fa (stale) Mirror 4: 0x7b926e49 (stale) Mirror 5: 0x46abdce9 (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3faef: skipped 000929 1784046651.873413 trunc from 0x03faef to 0x027cd1 (0x17e1d bytes) 000930 1784046651.898660 mapread 0x024021 thru 0x027cd1 (0x3cb1 bytes) 000931 1784046651.922055 read 0x01b320 thru 0x027cd1 (0xc9b2 bytes) 000932 1784046651.932011 trunc from 0x027cd2 to 0x03deb8 (0x161e7 bytes) 000939 1784046651.958901 write 0x025d83 thru 0x0342b4 (0xe532 bytes) 000941 1784046651.972182 write 0x0335e4 thru 0x0365b3 (0x2fd0 bytes) 000943 1784046651.981092 mapwrite 0x03e608 thru 0x03ffff (0x19f8 bytes) 000944 1784046652.078637 write 0x0122e0 thru 0x01359b (0x12bc bytes) 000947 1784046652.093705 mapwrite 0x03b2fb thru 0x03dc58 (0x295e bytes) 000948 1784046652.141243 trunc from 0x040000 to 0x014a99 (0x2b566 bytes) 000950 1784046652.163945 write 0x017597 thru 0x021bae (0xa618 bytes) 000952 1784046652.165648 mapwrite 0x031def thru 0x03c78c (0xa99e bytes) 000953 1784046652.251625 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x3c78d): Mirror 4: 0xfdac7c2c (stale) Mirror 5: 0x80c833bf (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x3c78d): Mirror 4: 0xfdac7c2c (stale) Mirror 5: 0x80c833bf (stale) lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0, 0x100000) is only valid in mirror 1: skipped lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x3c78d: skipped 000954 1784046652.603872 MIRROR_RESYNC 3 mirrors 000955 1784046653.741086 mapread 0x0196a9 thru 0x0230ae (0x9a06 bytes) 000956 1784046653.778042 read 0x004320 thru 0x00663c (0x231d bytes) 000958 1784046653.801050 MIRROR_RESYNC 3 mirrors truncating to largest ever: 0x3feb7 000959 1784046654.331123 trunc from 0x03c78d to 0x03feb6 (0x372a bytes) 000961 1784046654.439915 read 0x001fd6 thru 0x00f9e9 (0xda14 bytes) 000963 1784046654.463491 write 0x01b9ff thru 0x01f353 (0x3955 bytes) 000964 1784046654.477690 read 0x014d64 thru 0x02449d (0xf73a bytes) 000965 1784046654.492947 read 0x025737 thru 0x02fa02 (0xa2cc bytes) 000967 1784046654.495349 mapwrite 0x03b907 thru 0x03ffff (0x46f9 bytes) 000970 1784046654.608829 read 0x025991 thru 0x02add3 (0x5443 bytes) 000971 1784046654.612661 mapread 0x007587 thru 0x00a452 (0x2ecc bytes) 000972 1784046654.614344 mapwrite 0x03ae9c thru 0x03d93e (0x2aa3 bytes) 000973 1784046654.678662 read 0x0000cc thru 0x00cc42 (0xcb77 bytes) 000974 1784046654.680258 mapread 0x005e57 thru 0x00c4f3 (0x669d bytes) 000977 1784046654.688255 read 0x00d4d7 thru 0x017f5d (0xaa87 bytes) 000978 1784046654.697345 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: (stale - read error) Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 4: 0x8823a330 (stale) Mirror 5: 0x8823a330 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2ba100af Mirror 6: 0x2ba100af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000980 1784046656.048122 read 0x02bf42 thru 0x03148b (0x554a bytes) 000982 1784046656.077058 read 0x02cb38 thru 0x034223 (0x76ec bytes) 000983 1784046656.095935 read 0x021716 thru 0x02988c (0x8177 bytes) 000985 1784046656.116798 MIRROR_SPLIT mirror 1 to 3 mirrors lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x14c:0x0]: Invalid argument (22) CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8823a330 (stale) mirror op 1: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8823a330 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2ba100af Mirror 6: 0x2ba100af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000986 1784046656.968067 MIRROR_EXTEND to 4 mirrors CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: (stale - read error) mirror op 0: lfs mirror verify /mnt/lustre/d70a.sanity-flr/f70a.sanity-flr: 116 CRC-32 checksum value for chunk [0, 0x40000): Mirror 5: 0x8823a330 (stale) CRC-32 checksum value for chunk [0, 0x40000): Mirror 1: 0x2ba100af Mirror 6: 0x2ba100af Mirror 7: 0x2ba100af lfs verify: '/mnt/lustre/d70a.sanity-flr/f70a.sanity-flr' chunk [0x100000, 0xffffffffffffffff) exceeds file size 0x40000: skipped 000987 1784046658.118709 read 0x017ebe thru 0x01b1f2 (0x3335 bytes) 000988 1784046658.128979 mapread 0x00edf2 thru 0x01818b (0x939a bytes) 000990 1784046658.176751 MIRROR_RESYNC 4 mirrors 000992 1784046659.295144 read 0x000a6e thru 0x00eff0 (0xe583 bytes) 000993 1784046659.326860 mapread 0x00ad88 thru 0x01a316 (0xf58f bytes) 000994 1784046659.375249 mapwrite 0x000187 thru 0x00b0a7 (0xaf21 bytes) 000995 1784046659.757071 read 0x02088b thru 0x02a6cb (0x9e41 bytes) 000996 1784046659.784405 mapread 0x007d36 thru 0x00a131 (0x23fc bytes) 000997 1784046659.786212 read 0x021255 thru 0x027bbf (0x696b bytes) All operations completed A-OK! PASS 70a (68s) == sanity-flr test 71: check mirror extend parent fid ==== 12:31:05 (1784046665) SKIP: sanity-flr test_71 ldiskfs only test SKIP 71 (4s) == sanity-flr test 72: check append on FLR file ========== 12:31:09 (1784046669) /mnt/lustre/f72.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x262:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.402599 s, 5.2 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 11 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x262:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288853 s, 3.6 MB/s /mnt/lustre/f72.sanity-flr lcm_layout_gen: 13 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x263:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x262:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/f72.sanity-flr lcm_layout_gen: 15 lcm_mirror_count: 2 lcm_entry_count: 7 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x25f:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 3145728 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x263:0x0] } lcme_id: 65540 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 3145728 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 2097152 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x262:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 2097152 lcme_extent.e_end: 4194304 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x260:0x0] } lcme_id: 131079 lcme_mirror_id: 2 lcme_flags: 0 lcme_extent.e_start: 4194304 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 72 (10s) == sanity-flr test 200a: stress test ===================== 12:31:19 (1784046679) Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre2 Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre3 fail_loc=0x1A03 fail_loc=0x1A03 resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 3143328 .. Extending file size to 3654752 .. failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 6016256 .. Extending file size to 8920704 .. Extending file size to 5461184 .. Extending file size to 7699936 .. Extending file size to 7737600 .. Extending file size to 8060576 .. done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 8751264 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Extending file size to 9122336 .. failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200a.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..failed lock to resync file /mnt/lustre3/f200a.sanity-flr with 'mirror_io resync -e resync_start' ..Waiting 56818 56819 56820 56821 56824 failed 192.168.202.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg258-client.virtnet /mnt/lustre2 (opts:) 192.168.202.158@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg258-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200a (83s) == sanity-flr test 200b: racing IO, mirror extend and resync ========================================================== 12:32:42 (1784046762) Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre2 Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre3 fail_loc=0x1423 lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 3181696 .. Extending file size to 5474016 .. Extending file size to 8139936 .. Extending file size to 4429504 .. Extending file size to 8726080 .. Extending file size to 8492800 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e resync_start' ..failed Extending file size to 8944352 .. lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Extending file size to 9260864 .. done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..done resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..failed resync file /mnt/lustre3/f200b.sanity-flr with 'mirror_io resync -e delay_before_copy -d 1' ..failed lock to resync file /mnt/lustre3/f200b.sanity-flr with '/home/green/git/lustre-release/lustre/utils/lfs mirror resync' ..Waiting 58342 58343 58344 58346 58347 done 192.168.202.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg258-client.virtnet /mnt/lustre2 (opts:) 192.168.202.158@tcp:/lustre /mnt/lustre3 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg258-client.virtnet /mnt/lustre3 (opts:) 3 PASS 200b (80s) == sanity-flr test 200c: layout change racing with open: LOVEA changes ========================================================== 12:34:02 (1784046842) Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 24576 3744768 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 17408 3751936 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 41984 7496704 1% /mnt/lustre2 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.11336 s, 18.5 MB/s fail_loc=0x80000175 fail_val=10 mirror extend mirror extend done fail_loc=0 fail_val=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.472133 s, 4.4 MB/s dd completed Chunks to be verified in /mnt/lustre/f200c.sanity-flr: [0, 0xffffffffffffffff) [1] 1 CRC-32 checksum value for chunk [0, 0x200000): Mirror 2: 0xf1334c17 (stale) lfs verify: '/mnt/lustre/f200c.sanity-flr' chunk [0, 0xffffffffffffffff) is only valid in mirror 1: skipped 192.168.202.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg258-client.virtnet /mnt/lustre2 (opts:) PASS 200c (20s) SKIP: sanity-flr test_201 skipping ALWAYS excluded test 201 == sanity-flr test 202: lfs setstripe --add-component wide striping ========================================================== 12:34:24 (1784046864) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0715228 s, 29.3 MB/s PASS 202 (9s) == sanity-flr test 203: mirror file preserve mirror ID === 12:34:33 (1784046873) lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x188:0x0]: Invalid argument (22) /mnt/lustre/f203.sanity-flr lcm_layout_gen: 3 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x279:0x0] } /mnt/lustre/f203.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 2 lcme_id: 131074 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x279:0x0] } lcme_id: 196609 lcme_mirror_id: 3 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27a:0x0] } - 1: { l_ost_idx: 0, l_fid: [0x240000400:0x275:0x0] } PASS 203 (8s) == sanity-flr test 204a: FLR write/stale/resync tests with self-extending mirror ========================================================== 12:34:41 (1784046881) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.290779 s, 7.2 MB/s /mnt/lustre/d204a.sanity-flr/f204a.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27b:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x276:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x277:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 PASS 204a (10s) == sanity-flr test 204b: FLR write/stale/resync tests with self-extending primary ========================================================== 12:34:51 (1784046891) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.359643 s, 5.8 MB/s /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x278:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204b.sanity-flr/f204b.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27c:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: 68157440 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27d:0x0] } lcme_id: 65539 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 68157440 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x278:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x279:0x0] } PASS 204b (9s) == sanity-flr test 204c: FLR write/stale/resync test with component removal ========================================================== 12:35:00 (1784046900) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.33888 s, 6.2 MB/s /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 7 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204c.sanity-flr/f204c.sanity-flr lcm_layout_gen: 9 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 lmm_stripe_count: 1 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27a:0x0] } lcme_id: 131078 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27b:0x0] } PASS 204c (36s) == sanity-flr test 204d: FLR write/stale/resync sel test with repeated comp ========================================================== 12:35:37 (1784046937) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg258-server mds-ost sync done. 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142574 s, 7.4 MB/s /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: stale lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 /mnt/lustre/d204d.sanity-flr/f204d.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x27f:0x0] } lcme_id: 65541 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27d:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 10485760 lmm_stripe_count: 1 lmm_stripe_size: 2097152 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27c:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 10485760 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x280:0x0] } PASS 204d (59s) == sanity-flr test 204e: FLR write/stale/resync sel test with repeated comp ========================================================== 12:36:36 (1784046996) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg258-server mds-ost sync done. /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 4 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27f:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 67108864 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.364078 s, 5.8 MB/s /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x281:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 /mnt/lustre/d204e.sanity-flr/f204e.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 5 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27e:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 67108864 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x27f:0x0] } lcme_id: 131077 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 67108864 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x281:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 PASS 204e (57s) == sanity-flr test 204f: FLR write/stale/resync sel w/forced extension ========================================================== 12:37:33 (1784047053) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg258-server mds-ost sync done. Creating new pool test_204f oleg258-server: Pool lustre.test_204f created Adding targets to pool oleg258-server: OST lustre-OST0000_UUID added to pool lustre.test_204f oleg258-server: OST lustre-OST0001_UUID added to pool lustre.test_204f 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.228071 s, 9.2 MB/s /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 6 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x280:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init,stale lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x282:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f /mnt/lustre/d204f.sanity-flr/f204f.sanity-flr lcm_layout_gen: 8 lcm_mirror_count: 2 lcm_entry_count: 4 lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 104857600 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 lmm_objects: - 0: { l_ost_idx: 0, l_fid: [0x240000400:0x280:0x0] } lcme_id: 65538 lcme_mirror_id: 1 lcme_flags: 0 lcme_extent.e_start: 104857600 lcme_extent.e_end: EOF lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lcme_id: 131075 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 134217728 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_pool: test_204f lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x282:0x0] } lcme_id: 131076 lcme_mirror_id: 2 lcme_flags: extension lcme_extent.e_start: 134217728 lcme_extent.e_end: EOF lmm_stripe_count: 0 lmm_extension_size: 67108864 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_pool: test_204f Destroy the created pools: test_204f lustre.test_204f oleg258-server: OST lustre-OST0000_UUID removed from pool lustre.test_204f oleg258-server: OST lustre-OST0001_UUID removed from pool lustre.test_204f oleg258-server: Pool lustre.test_204f destroyed PASS 204f (83s) == sanity-flr test 205a: lfs mirror extend to set prefer flag ========================================================== 12:38:56 (1784047136) PASS 205a (8s) == sanity-flr test 205b: lfs mirror extend to set nocompr flag ========================================================== 12:39:04 (1784047144) PASS 205b (9s) == sanity-flr test 206: lfs setstripe -pool .. --comp-flags=.. ========================================================== 12:39:14 (1784047154) oleg258-server: Pool lustre.test_206 created Adding targets to pool oleg258-server: OST lustre-OST0000_UUID added to pool lustre.test_206 oleg258-server: OST lustre-OST0001_UUID added to pool lustre.test_206 Destroy the created pools: test_206 lustre.test_206 oleg258-server: OST lustre-OST0000_UUID removed from pool lustre.test_206 oleg258-server: OST lustre-OST0001_UUID removed from pool lustre.test_206 oleg258-server: Pool lustre.test_206 destroyed PASS 206 (27s) == sanity-flr test 207: create another replica with existing out-of-sync one ========================================================== 12:39:41 (1784047181) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743217 s, 14.1 MB/s 2 mirror IDs: 1 2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400257 s, 26.2 MB/s 2 mirror IDs: 1 2 2 mirror IDs: 1 2 3 mirror IDs: 1 2 3 CRC-32 checksum value for chunk [0, 0x100000): Mirror 2: (stale - read error) CRC-32 checksum value for chunk [0, 0x100000): Mirror 1: 0x131d7792 Mirror 3: 0x131d7792 PASS 207 (12s) == sanity-flr test 208a: mirror selection to prefer non-rotational devices for reads ========================================================== 12:39:53 (1784047193) SKIP: sanity-flr test_208a needs >= 4 OSTs SKIP 208a (3s) == sanity-flr test 208b: mirror selection to prefer non-rotational devices for writes ========================================================== 12:39:56 (1784047196) SKIP: sanity-flr test_208b needs >= 4 OSTs SKIP 208b (4s) == sanity-flr test 209a: skip fast reads after layout invalidation ========================================================== 12:40:00 (1784047200) Starting client: oleg258-client.virtnet: -o user_xattr,flock 192.168.202.158@tcp:/lustre /mnt/lustre2 osd-zfs.lustre-OST0000.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=0 osd-zfs.lustre-OST0001.nonrotational=1 replicated file created open(O_RDONLY) and first read from OST first read complete 4+0 records in 4+0 records out 4 bytes copied, 0.110471 s, 0.0 kB/s BBBB written which made replica on ost1 stale fast read from pagecache in the original process read via /mnt/lustre2 new open(2) Buffer address : 0x900000 Read this (4 bytes): BBBB fast read from pagecache after 5s in the original process Buffer address : 0xb60000 Read this (4 bytes): AAAA PAUSING Buffer address : 0xb60000 Read this (4 bytes): BBBB PAUSING Buffer address : 0xb60000 Read this (4 bytes): BBBB read via new open(2) Buffer address : 0x1ee0000 Read this (4 bytes): BBBB 192.168.202.158@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg258-client.virtnet /mnt/lustre2 (opts:) PASS 209a (35s) == sanity-flr test 209b: pagecache can be used after LL cancellation ========================================================== 12:40:36 (1784047236) 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.029968 s, 273 kB/s the very first read read with warm cache PASS 209b (6s) == sanity-flr test 210a: handle broken mirrored lovea ==== 12:40:43 (1784047243) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869344 s, 12.1 MB/s fail_loc=0x1428 /mnt/lustre/f210a.sanity-flr composite_header: lcm_magic: 0x0BD60BD0 lcm_size: 240 lcm_flags: ro lcm_layout_gen: 1 lcm_mirror_count: 2 lcm_entry_count: 2 components: - lcme_id: 65537 lcme_mirror_id: 1 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 128 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1ad lmm_fid: [0x200000404:0x1ad:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 lmm_objects: - 0: { l_ost_idx: -1, l_fid: [0:0x0:0x0] } - lcme_id: 131073 lcme_mirror_id: 2 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: EOF lcme_offset: 184 lcme_size: 56 sub_layout: lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000404 lmm_object_id: 0x1ad lmm_fid: [0x200000404:0x1ad:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - 0: { l_ost_idx: 1, l_fid: [0x280000400:0x28d:0x0] } File: /mnt/lustre/f210a.sanity-flr Size: 1048576 Blocks: 2052 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205322834349 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-07-14 12:40:45.000000000 -0400 Modify: 2026-07-14 12:40:45.000000000 -0400 Change: 2026-07-14 12:40:47.000000000 -0400 Birth: 2026-07-14 12:40:45.000000000 -0400 PASS 210a (10s) == sanity-flr test 210b: handle broken mirrored lovea (unlink) ========================================================== 12:40:53 (1784047253) SKIP: sanity-flr test_210b ZFS file number is not accurate SKIP 210b (4s) == sanity-flr test 211: mirror delete should not cause bad size ========================================================== 12:40:57 (1784047257) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.397561 s, 103 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0357571 s, 115 kB/s size after second write -rw-r--r-- 1 root root 4096 Jul 14 12:41 /mnt/lustre/f211.sanity-flr lfs mirror mirror: cannot get UNLOCK lease, ext 8 for [0x200000404:0x1af:0x0]: Invalid argument (22) size after mirror delete -rw-r--r-- 1 root root 4096 Jul 14 12:41 /mnt/lustre/f211.sanity-flr PASS 211 (9s) == sanity-flr test 212a: Testing lfs setstripe --comp-set --comp-flags=stale --mirror-id ========================================================== 12:41:06 (1784047266) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212a.sanity-flr' CRC-32 checksum value for chunk [0, 0xe): Mirror 2: (stale - read error) PASS 212a (7s) == sanity-flr test 212b: Testing lfs mirror verify --stale option ========================================================== 12:41:13 (1784047273) CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xb093617a Mirror 3: 0xb093617a Writing different data to mirror 2 only === Verify inconsistent mirrors === lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a verify correctly failed with inconsistent mirrors lfs verify: chunk [0, 0xffffffffffffffff) has different checksum value on mirror 1:b093617a and mirror 2:ff8c0066: Structure needs cleaning CRC-32 checksum value for chunk [0, 0x1d): Mirror 1: 0xb093617a Mirror 2: 0xff8c0066 Mirror 3: 0xb093617a Marking mirror 2 as stale (has_prefwr=no) Set flags on 1 component(s) of mirror 2 in '/mnt/lustre/f212b.sanity-flr' Found 1 stale components === Resync and final verification === verify --stale test completed successfully PASS 212b (10s) == sanity-flr test complete, duration 1862 sec =========== 12:41:23 (1784047283) === sanity-flr: start cleanup 12:41:25 (1784047285) === === sanity-flr: finish cleanup 12:41:32 (1784047292) ===