-----============= acceptance-small: sanity ============----- Sun Jun 22 07:19:32 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg110-server: /home/green/git/lustre-release/lustre/tests/except/sanity.0f.ex oleg110-server: /home/green/git/lustre-release/lustre/tests/except/sanity.ex - see client > v2_15_63-134-gdacc4b6d38 for LU-15963, go 312 - need mds1 <= 2.14.55-100-g8a84c7f9c7 for LU-14927, skip 0f - need mds1 < v2_14_55-100-g8a84c7f9c7 for LU-14927, skip 0f excepting tests: 42a 42c 42b 118c 118d 407 119i 411b 130b 130c 130d 130e 130f 130g 312 skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 51b === sanity: start setup 07:19:46 (1750591186) === oleg110-client.virtnet: executing check_config_client /mnt/lustre oleg110-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg110-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff880137177800.idle_timeout=debug osc.lustre-OST0001-osc-ffff880137177800.idle_timeout=debug disable quota as required oleg110-server: oleg110-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanity: finish setup 07:20:03 (1750591203) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7295] preparing for tests involving mounts mke2fs 1.46.2.wc5 (26-Mar-2022) debug=all == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 07:20:07 (1750591207) SKIP: sanity test_60a missing subtest run-llog.sh SKIP 60a (3s) == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 07:20:10 (1750591210) PASS 60b (3s) == sanity test 60c: unlink file when mds full ============ 07:20:14 (1750591214) create 5000 files - open/close 1070 (time 1750591225.94 total 10.00 last 107.00) - open/close 2126 (time 1750591235.95 total 20.01 last 105.52) - open/close 3026 (time 1750591245.96 total 30.02 last 89.90) - open/close 3871 (time 1750591255.97 total 40.02 last 84.45) - open/close 4839 (time 1750591265.97 total 50.03 last 96.75) total: 5000 open/close in 51.47 seconds: 97.14 ops/second fail_loc=0x80000137 - unlinked 0 (time 1750591268 ; total 0 ; last 0) total: 5000 unlinks in 36 seconds: 138.888885 unlinks/second fail_loc=0 PASS 60c (93s) == sanity test 60d: test printk console message masking == 07:21:47 (1750591307) printk=0 emerg PASS 60d (3s) == sanity test 60e: no space while new llog is being created ========================================================== 07:21:50 (1750591310) fail_loc=0x15b PASS 60e (3s) == sanity test 60f: change debug_path works ============== 07:21:53 (1750591313) debug_path=/tmp/f60f.sanity fail_loc=0x8000050e ls: cannot access /tmp/f60f.sanity*: No such file or directory 0 /tmp/f60f.sanity.1750591314.13842 debug_path=/tmp/lustre-log PASS 60f (4s) == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 07:21:57 (1750591317) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4794: 14660 Killed ( local index=0; while true; do $LFS setdirstripe -i $(($index % $MDSCOUNT)) -c $MDSCOUNT $DIR/$tdir/subdir$index 2> /dev/null; mkdir $DIR/$tdir/subdir$index 2> /dev/null; rmdir $DIR/$tdir/subdir$index 2> /dev/null; index=$((index + 1)); done ) Started LFSCK on the device lustre-MDT0000: scrub namespace /mnt/lustre/d60g.sanity: subdir105 subdir118 subdir13 subdir143 subdir163 subdir177 subdir188 subdir201 subdir229 subdir250 subdir26 subdir278 subdir285 subdir291 subdir33 subdir330 subdir337 subdir352 subdir358 subdir365 subdir373 subdir39 subdir394 subdir406 subdir420 subdir446 subdir46 subdir465 subdir493 subdir500 subdir507 subdir53 subdir532 subdir539 subdir563 subdir593 subdir606 subdir612 subdir634 subdir654 subdir661 subdir668 subdir67 subdir681 subdir695 subdir91 /mnt/lustre/d60g.sanity/subdir105: /mnt/lustre/d60g.sanity/subdir118: /mnt/lustre/d60g.sanity/subdir13: /mnt/lustre/d60g.sanity/subdir143: /mnt/lustre/d60g.sanity/subdir163: /mnt/lustre/d60g.sanity/subdir177: /mnt/lustre/d60g.sanity/subdir188: /mnt/lustre/d60g.sanity/subdir201: /mnt/lustre/d60g.sanity/subdir229: /mnt/lustre/d60g.sanity/subdir250: /mnt/lustre/d60g.sanity/subdir26: /mnt/lustre/d60g.sanity/subdir278: /mnt/lustre/d60g.sanity/subdir285: /mnt/lustre/d60g.sanity/subdir291: /mnt/lustre/d60g.sanity/subdir33: /mnt/lustre/d60g.sanity/subdir330: /mnt/lustre/d60g.sanity/subdir337: /mnt/lustre/d60g.sanity/subdir352: /mnt/lustre/d60g.sanity/subdir358: /mnt/lustre/d60g.sanity/subdir365: /mnt/lustre/d60g.sanity/subdir373: /mnt/lustre/d60g.sanity/subdir39: /mnt/lustre/d60g.sanity/subdir394: /mnt/lustre/d60g.sanity/subdir406: /mnt/lustre/d60g.sanity/subdir420: /mnt/lustre/d60g.sanity/subdir446: /mnt/lustre/d60g.sanity/subdir46: /mnt/lustre/d60g.sanity/subdir465: /mnt/lustre/d60g.sanity/subdir493: /mnt/lustre/d60g.sanity/subdir500: /mnt/lustre/d60g.sanity/subdir507: /mnt/lustre/d60g.sanity/subdir53: /mnt/lustre/d60g.sanity/subdir532: /mnt/lustre/d60g.sanity/subdir539: /mnt/lustre/d60g.sanity/subdir563: /mnt/lustre/d60g.sanity/subdir593: /mnt/lustre/d60g.sanity/subdir606: /mnt/lustre/d60g.sanity/subdir612: /mnt/lustre/d60g.sanity/subdir634: /mnt/lustre/d60g.sanity/subdir654: /mnt/lustre/d60g.sanity/subdir661: /mnt/lustre/d60g.sanity/subdir668: /mnt/lustre/d60g.sanity/subdir67: /mnt/lustre/d60g.sanity/subdir681: /mnt/lustre/d60g.sanity/subdir695: /mnt/lustre/d60g.sanity/subdir91: PASS 60g (54s) == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 07:22:51 (1750591371) SKIP: sanity test_60h Need at least 2 MDTs SKIP 60h (2s) SKIP: sanity test_60i skipping SLOW test 60i == sanity test 60j: llog_reader reports corruptions ====== 07:22:53 (1750591373) SKIP: sanity test_60j ldiskfs only test SKIP 60j (2s) == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 07:22:55 (1750591375) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00730153 s, 561 kB/s PASS 61a (3s) == sanity test 61b: mmap() of unstriped file is successful ========================================================== 07:22:58 (1750591378) PASS 61b (3s) == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 07:23:01 (1750591381) /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 20461 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 20480 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 20487 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 20495 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 20503 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 20511 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 20519 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 20527 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 20534 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10610: 20542 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 19456 3727360 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 22528 7493632 1% /mnt/lustre pass grant check: client:492437504 server:492437504 PASS 63a (64s) == sanity test 63b: async write errors should be returned to fsync ============================================================= 07:24:05 (1750591445) debug=-1 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00825225 s, 496 kB/s fail_loc=0x80000406 fsync: Input/output error checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre pass grant check: client:492437504 server:492437504 PASS 63b (6s) == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 07:24:11 (1750591451) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre osc.lustre-OST0000-osc-ffff880137177800.cur_grant_bytes=5111808 osc.lustre-OST0000-osc-ffff880137177800.cur_lost_grant_bytes=1703936 osc.lustre-OST0001-osc-ffff880137177800.cur_grant_bytes=482213888 osc.lustre-OST0001-osc-ffff880137177800.cur_lost_grant_bytes=3407872 checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre pass grant check: client:492437504 server:492437504 PASS 64a (4s) SKIP: sanity test_64b skipping SLOW test 64b == sanity test 64c: verify grant shrink ================== 07:24:15 (1750591455) osc.lustre-OST0000-osc-ffff880137177800.cur_grant_bytes=0 checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre pass grant check: client:490733568 server:490733568 PASS 64c (4s) == sanity test 64d: check grant limit exceed ============= 07:24:19 (1750591459) 3 5111808 while dd ... pid 23660 ... tgt:lustre-OST0000-osc-ffff880137177800 cur_grant: 1703936 ?>? max_cur_granted: 493748224 tgt:lustre-OST0000-osc-ffff880137177800 cur_grant: 143130624 ?>? max_cur_granted: 493748224 tgt:lustre-OST0000-osc-ffff880137177800 cur_grant: 274333696 ?>? max_cur_granted: 493748224 tgt:lustre-OST0000-osc-ffff880137177800 cur_grant: 407240704 ?>? max_cur_granted: 493748224 tgt:lustre-OST0000-osc-ffff880137177800 cur_grant: 486932480 ?>? max_cur_granted: 493748224 tgt:lustre-OST0000-osc-ffff880137177800 cur_grant: 489029632 ?>? max_cur_granted: 493748224 tgt:lustre-OST0000-osc-ffff880137177800 cur_grant: 492044288 ?>? max_cur_granted: 493748224 tgt:lustre-OST0000-osc-ffff880137177800 cur_grant: 489029632 ?>? max_cur_granted: 493748224 tgt:lustre-OST0000-osc-ffff880137177800 cur_grant: 490340352 ?>? max_cur_granted: 493748224 tgt:lustre-OST0000-osc-ffff880137177800 cur_grant: 490340352 ?>? max_cur_granted: 493748224 tgt:lustre-OST0000-osc-ffff880137177800 cur_grant: 489160704 ?>? max_cur_granted: 493748224 tgt:lustre-OST0000-osc-ffff880137177800 cur_grant: 489029632 ?>? max_cur_granted: 493748224 tgt:lustre-OST0000-osc-ffff880137177800 cur_grant: 490995712 ?>? max_cur_granted: 493748224 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 12.5651 s, 83.5 MB/s /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10735: kill: (23660) - No such process checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 1027072 2742272 28% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 1030144 6508544 14% /mnt/lustre pass grant check: client:976355328 server:976355328 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 64d (36s) == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 07:24:55 (1750591495) debug=+cache Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre fail_loc=0x725 1+0 records in 1+0 records out 290816 bytes (291 kB) copied, 0.202453 s, 1.4 MB/s fail_loc=0 Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre fail_loc=0x725 fail_loc=0 PASS 64e (6s) == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 07:25:01 (1750591501) debug=+cache Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre 1+0 records in 1+0 records out 368640 bytes (369 kB) copied, 0.0661036 s, 5.6 MB/s Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre llite.lustre-ffff88012ddcf000.hybrid_io=0 fail_loc=0x50a fail_val=3 1+0 records in 1+0 records out 368640 bytes (369 kB) copied, 0.0255415 s, 14.4 MB/s fail_loc=0 fail_val=0 PASS 64f (4s) == sanity test 64g: grant shrink on MDT ================== 07:25:06 (1750591506) 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.0109982 s, 11.9 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.00932799 s, 14.1 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.0128424 s, 10.2 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.0148027 s, 8.9 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.0477069 s, 2.7 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.00928049 s, 14.1 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.00923184 s, 14.2 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.0161461 s, 8.1 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.0104183 s, 12.6 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB) copied, 0.0563879 s, 2.3 MB/s 0 grants, 0 pages mdc.lustre-MDT0000-mdc-ffff88012ddcf000.grant_shrink_interval=1200 PASS 64g (5s) == sanity test 64h: grant shrink on read ================= 07:25:11 (1750591511) osc.lustre-OST0000-osc-ffff88012ddcf000.grant_shrink=1 osc.lustre-OST0000-osc-ffff88012ddcf000.grant_shrink_interval=10 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 1.09374 s, 9.6 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0147993 s, 277 kB/s PASS 64h (13s) == sanity test 64i: shrink on reconnect ================== 07:25:24 (1750591524) 64+0 records in 64+0 records out 67108864 bytes (67 MB) copied, 3.71299 s, 18.1 MB/s fail_loc=0x80000513 fail_val=17 osc.lustre-OST0000-osc-ffff88012ddcf000.cur_grant_bytes=65601536B Failing ost1 on oleg110-server Stopping /mnt/lustre-ost1 (opts:) on oleg110-server 07:25:31 (1750591531) shut down facet: ost1 facet_host: oleg110-server facet_failover_host: oleg110-server Failover ost1 to oleg110-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug all all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 07:25:45 (1750591545) targets are mounted 07:25:45 (1750591545) facet_failover done oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.347943 s, 24.1 MB/s PASS 64i (29s) == sanity test 64j: check grants on re-done rpc ========== 07:25:53 (1750591553) fail_loc=0x80000256 checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3968 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3764224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7530496 1% /mnt/lustre pass grant check: client:22544384 server:22544384 PASS 64j (5s) == sanity test 65a: directory with no stripe info ======== 07:25:58 (1750591558) default stripe 1, ost count 2 PASS 65a (3s) == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 07:26:01 (1750591561) dir stripe 1, default stripe 1, ost count 2 PASS 65b (4s) == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 07:26:05 (1750591565) dir stripe 1, default stripe 1, ost count 2 PASS 65c (3s) == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 07:26:08 (1750591568) dir stripe 0, default stripe 1, ost count 2 PASS 65d (3s) == sanity test 65e: directory setstripe defaults ========= 07:26:11 (1750591571) (Default) /mnt/lustre/d65e.sanity default stripe 1, ost count 2 PASS 65e (3s) == sanity test 65f: dir setstripe permission (should return error) ============================================================= 07:26:14 (1750591574) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [/mnt/lustre/d65f.sanityf] lfs setstripe: setstripe error for '/mnt/lustre/d65f.sanityf': Operation not permitted PASS 65f (2s) == sanity test 65g: directory setstripe -d =============== 07:26:16 (1750591576) (Default) /mnt/lustre/d65g.sanity PASS 65g (3s) == sanity test 65h: directory stripe info inherit ============================================================================== 07:26:19 (1750591579) PASS 65h (3s) == sanity test 65i: various tests to set root directory striping ========================================================== 07:26:22 (1750591582) /mnt/lustre stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65e.sanity stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65h.sanity stripe_count: 1 stripe_size: 8388608 pattern: raid0 stripe_offset: 0 /mnt/lustre/d65g.sanity stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f64j.sanity lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2513 0x9d1 0x240000400 /mnt/lustre/d65a.sanity stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f63b.sanity lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2505 0x9c9 0x280000400 /mnt/lustre/d65c.sanity stripe_count: 1 stripe_size: 16777216 pattern: raid0 stripe_offset: 1 /mnt/lustre/d60f.sanity stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f61 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2504 0x9c8 0x240000400 /mnt/lustre/d65b.sanity stripe_count: 1 stripe_size: 8388608 pattern: raid0 stripe_offset: 0 /mnt/lustre/f61b.sanity has no stripe info /mnt/lustre/d65f.sanityf stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65d.sanity stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre/f60b.sanity lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x280000400 /mnt/lustre lmm_fid: [0x200000007:0x1:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65e.sanity lmm_fid: [0x200000406:0x26:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65h.sanity lmm_fid: [0x200000406:0x2c:0x0] stripe_count: 1 stripe_size: 8388608 pattern: raid0 stripe_offset: 0 /mnt/lustre/d65g.sanity lmm_fid: [0x200000406:0x2a:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f64j.sanity lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000406 lmm_object_id: 0x1c lmm_fid: [0x200000406:0x1c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2513 0x9d1 0x240000400 /mnt/lustre/d65a.sanity lmm_fid: [0x200000406:0x1d:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f63b.sanity lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x191b lmm_fid: [0x200000401:0x191b:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2505 0x9c9 0x280000400 /mnt/lustre/d65c.sanity lmm_fid: [0x200000406:0x21:0x0] stripe_count: 1 stripe_size: 16777216 pattern: raid0 stripe_offset: 1 /mnt/lustre/d60f.sanity lmm_fid: [0x200000401:0x138d:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f61 lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x190f lmm_fid: [0x200000401:0x190f:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2504 0x9c8 0x240000400 /mnt/lustre/d65b.sanity lmm_fid: [0x200000406:0x1f:0x0] stripe_count: 1 stripe_size: 8388608 pattern: raid0 stripe_offset: 0 /mnt/lustre/f61b.sanity has no stripe info /mnt/lustre/d65f.sanityf lmm_fid: [0x200000406:0x29:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65d.sanity lmm_fid: [0x200000406:0x23:0x0] stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre/f60b.sanity lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000401 lmm_object_id: 0x3 lmm_fid: [0x200000401:0x3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x280000400 PASS 65i (3s) == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 07:26:25 (1750591585) PASS 65j (3s) == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 07:26:29 (1750591589) Check OST status: lustre-OST0000-osc-MDT0000 is active lustre-OST0001-osc-MDT0000 is active total: 1000 open/close in 6.56 seconds: 152.52 ops/second Deactivate: lustre-OST0000-osc-MDT0000 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/d65k.sanity/0 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 1 -c 1 /mnt/lustre/d65k.sanity/1 - unlinked 0 (time 1750591602 ; total 0 ; last 0) total: 1000 unlinks in 4 seconds: 250.000000 unlinks/second lustre-OST0000-osc-MDT0000 is Activate oleg110-server: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg110-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec total: 1000 open/close in 6.57 seconds: 152.13 ops/second Deactivate: lustre-OST0001-osc-MDT0000 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/d65k.sanity/0 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 1 -c 1 /mnt/lustre/d65k.sanity/1 - unlinked 0 (time 1750591620 ; total 0 ; last 0) total: 1000 unlinks in 3 seconds: 333.333344 unlinks/second lustre-OST0001-osc-MDT0000 is Activate oleg110-server: oleg110-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg110-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec PASS 65k (40s) == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 07:27:09 (1750591629) PASS 65l (3s) == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 07:27:12 (1750591632) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-c] [2] [/mnt/lustre] lfs setstripe: setstripe error for '/mnt/lustre': Operation not permitted PASS 65m (2s) == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 07:27:14 (1750591634) Creating new pool test_65n oleg110-server: Pool lustre.test_65n created Adding targets to pool oleg110-server: OST lustre-OST0000_UUID added to pool lustre.test_65n oleg110-server: OST lustre-OST0001_UUID added to pool lustre.test_65n /home/green/git/lustre-release/lustre/utils/lfs getstripe -d /mnt/lustre/d65n.sanity-4 stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65n /home/green/git/lustre-release/lustre/utils/lfs getstripe -d /mnt/lustre stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65n SKIP: sanity test_65n needs >= 2 MDTs Destroy the created pools: test_65n lustre.test_65n oleg110-server: OST lustre-OST0000_UUID removed from pool lustre.test_65n oleg110-server: OST lustre-OST0001_UUID removed from pool lustre.test_65n oleg110-server: Pool lustre.test_65n destroyed SKIP 65n (14s) == sanity test 65o: pool inheritance for mdt component === 07:27:28 (1750591648) Creating new pool test_65o oleg110-server: Pool lustre.test_65o created Adding targets to pool oleg110-server: OST lustre-OST0000_UUID added to pool lustre.test_65o oleg110-server: OST lustre-OST0001_UUID added to pool lustre.test_65o /mnt/lustre/d65o.sanity lcm_layout_gen: 0 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 stripe_count: 0 stripe_size: 1048576 pattern: mdt stripe_offset: -1 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65o /mnt/lustre/d65o.sanity/dir2 lcm_layout_gen: 0 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 stripe_count: 0 stripe_size: 1048576 pattern: mdt stripe_offset: -1 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65o lcm_layout_gen: 0 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 0 lcme_extent.e_end: EOF stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65o Destroy the created pools: test_65o lustre.test_65o oleg110-server: OST lustre-OST0000_UUID removed from pool lustre.test_65o oleg110-server: OST lustre-OST0001_UUID removed from pool lustre.test_65o oleg110-server: Pool lustre.test_65o destroyed PASS 65o (15s) == sanity test 65p: setstripe with yaml file and huge number ========================================================== 07:27:43 (1750591663) PASS 65p (3s) == sanity test 65q: setstripe with >=8E offset should fail ========================================================== 07:27:46 (1750591666) lfs setstripe: lfs setstripe: cannot set default composite layout for '/mnt/lustre/d65q.sanity/src_dir': Invalid argument (22) PASS 65q (3s) == sanity test 65r: prevent all-zero offsets ============= 07:27:49 (1750591669) PASS 65r (3s) == sanity test 66: update inode blocks count on client ========================================================================= 07:27:52 (1750591672) 8+0 records in 8+0 records out 8192 bytes (8.2 kB) copied, 0.0154712 s, 529 kB/s PASS 66 (5s) == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 07:27:57 (1750591677) directio on /mnt/lustre/f69.sanity.2 for 1x4194304 bytes PASS fail_loc=0x217 directio on /mnt/lustre/f69.sanity for 2x4194304 bytes Write error No such file or directory (rc = -1, len = 8388608) fail_loc=0 directio on /mnt/lustre/f69.sanity for 2x4194304 bytes PASS directio on /mnt/lustre/f69.sanity for 1x4194304 bytes PASS fail_loc=0x217 directio on /mnt/lustre/f69.sanity for 1x4194304 bytes Read error: No such file or directory rc = -1 fail_loc=0 PASS 69 (5s) == sanity test 70a: verify health_check, health_write don't explode (on OST) ========================================================== 07:28:02 (1750591682) enable_health_write=off enable_health_write=0 enable_health_write=on enable_health_write=1 enable_health_write=0 PASS 70a (5s) SKIP: sanity test_71 skipping SLOW test 71 == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 07:28:08 (1750591688) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7295] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/f72a.sanity] [bs=512] [count=1] 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00878615 s, 58.3 kB/s PASS 72a (3s) == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 07:28:11 (1750591691) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7295] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/f72b.sanity-fg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/f72b.sanity-fu] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fu': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/f72b.sanity-dg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-dg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/f72b.sanity-du] chmod: changing permissions of '/mnt/lustre/f72b.sanity-du': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [2777] [/mnt/lustre/f72b.sanity-fg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [2777] [/mnt/lustre/f72b.sanity-fu] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fu': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [2777] [/mnt/lustre/f72b.sanity-dg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-dg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [2777] [/mnt/lustre/f72b.sanity-du] chmod: changing permissions of '/mnt/lustre/f72b.sanity-du': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [4777] [/mnt/lustre/f72b.sanity-fg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [4777] [/mnt/lustre/f72b.sanity-fu] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fu': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [4777] [/mnt/lustre/f72b.sanity-dg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-dg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [4777] [/mnt/lustre/f72b.sanity-du] chmod: changing permissions of '/mnt/lustre/f72b.sanity-du': Operation not permitted PASS 72b (4s) == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 07:28:15 (1750591695) multiop /mnt/lustre/d73-1/f73-1 vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7295 fail_loc=0x80000129 fail_loc=0 /mnt/lustre/d73-1/f73-1 has type file OK /mnt/lustre/d73-1/f73-2 has type file OK /mnt/lustre/d73-2/f73-3 has type file OK PASS 73 (28s) == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 07:28:43 (1750591723) fail_loc=0x8000030e /mnt/lustre/f74a fail_loc=0 PASS 74a (3s) == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 07:28:46 (1750591726) fail_loc=0x8000030e fail_loc=0 PASS 74b (2s) == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 07:28:48 (1750591728) fail_loc=0x319 touch: cannot touch '/mnt/lustre/f74c.sanity': No such file or directory fail_loc=0 PASS 74c (3s) == sanity test 76a: confirm clients recycle inodes properly ============================================================== 07:28:51 (1750591731) before slab objects: 45 created: 512, after slab objects: 45 PASS 76a (35s) == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 07:29:26 (1750591766) slab objects before: 45, after: 45 PASS 76b (23s) == sanity test 77a: normal checksum read/write operation ========================================================== 07:29:49 (1750591789) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.0923327 s, 90.9 MB/s 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.32415 s, 25.9 MB/s PASS 77a (3s) == sanity test 77b: checksum error on client write, read ========================================================== 07:29:52 (1750591792) fail_loc=0x80000409 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.327425 s, 25.6 MB/s fail_loc=0 set checksum type to crc32, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to adler, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to crc32c, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to t10ip512, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to t10ip4K, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to t10crc512, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to t10crc4K, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to crc32c, rc = 0 PASS 77b (19s) == sanity test 77c: checksum error on client read with debug ========================================================== 07:30:11 (1750591811) debug_path=/tmp/lustre-log 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.395869 s, 21.2 MB/s osc.lustre-OST0000-osc-ffff88012ddcf000.checksum_dump=1 osc.lustre-OST0001-osc-ffff88012ddcf000.checksum_dump=1 obdfilter.lustre-OST0000.checksum_dump=1 obdfilter.lustre-OST0001.checksum_dump=1 fail_loc=0x80000408 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 1.58355 s, 5.3 MB/s fail_loc=0 osc.lustre-OST0000-osc-ffff88012ddcf000.checksum_dump=0 osc.lustre-OST0001-osc-ffff88012ddcf000.checksum_dump=0 obdfilter.lustre-OST0000.checksum_dump=0 obdfilter.lustre-OST0001.checksum_dump=0 PASS 77c (14s) == sanity test 77d: checksum error on OST direct write, read ========================================================== 07:30:25 (1750591825) fail_loc=0x80000409 directio on /mnt/lustre/f77d.sanity for 8x1048576 bytes PASS fail_loc=0 fail_loc=0x80000408 directio on /mnt/lustre/f77d.sanity for 8x1048576 bytes PASS fail_loc=0 PASS 77d (5s) == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 07:30:30 (1750591830) set checksum type to crc32, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to adler, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to crc32c, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to t10ip512, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to t10ip4K, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to t10crc512, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to t10crc4K, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to crc32c, rc = 0 PASS 77f (406s) == sanity test 77g: checksum error on OST write, read ==== 07:37:16 (1750592236) fail_loc=0x8000021a 8+0 records in 8+0 records out 8388608 bytes (8.4 MB) copied, 0.489807 s, 17.1 MB/s fail_loc=0 fail_loc=0x8000021b fail_loc=0 PASS 77g (9s) == sanity test 77k: enable/disable checksum correctly ==== 07:37:25 (1750592245) remount client, checksum should be 0 Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre Waiting 90s for '1' remount client, checksum should be 1 Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre remount client with option checksum, checksum should be 1 192.168.201.110@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Starting client: oleg110-client.virtnet: -o user_xattr,flock,checksum 192.168.201.110@tcp:/lustre /mnt/lustre remount client with option nochecksum, checksum should be 0 192.168.201.110@tcp:/lustre /mnt/lustre lustre rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Starting client: oleg110-client.virtnet: -o user_xattr,flock,nochecksum 192.168.201.110@tcp:/lustre /mnt/lustre Stopping client oleg110-client.virtnet /mnt/lustre (opts:) Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre Waiting 90s for '0' Updated after 2s: want '0' got '0' PASS 77k (10s) == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 07:37:35 (1750592255) osc.lustre-OST0000-osc-ffff8800a985c800.idle_timeout=10 osc.lustre-OST0001-osc-ffff8800a985c800.idle_timeout=10 error: set_param: setting /sys/fs/lustre/osc/lustre-OST0000-osc-ffff8800a985c800/checksum_type=invalid: Invalid argument error: set_param: setting /sys/fs/lustre/osc/lustre-OST0001-osc-ffff8800a985c800/checksum_type=invalid: Invalid argument error: set_param: setting 'osc/*osc-[^mM]*/checksum_type'='invalid': Invalid argument set checksum type to invalid, rc = 22 set checksum type to crc32, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=600000->3900s oleg110-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid in IDLE state after 2 sec oleg110-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid in FULL state after 0 sec set checksum type to adler, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg110-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid in IDLE state after 9 sec oleg110-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid in FULL state after 0 sec set checksum type to crc32c, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg110-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid in IDLE state after 9 sec oleg110-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid in FULL state after 0 sec set checksum type to t10ip512, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg110-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid in IDLE state after 9 sec oleg110-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid in FULL state after 0 sec set checksum type to t10ip4K, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg110-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid in IDLE state after 8 sec oleg110-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid in FULL state after 0 sec set checksum type to t10crc512, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg110-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid in IDLE state after 9 sec oleg110-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid in FULL state after 0 sec set checksum type to t10crc4K, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg110-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid in IDLE state after 9 sec oleg110-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8800a985c800.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0000-osc-ffff8800a985c800.idle_timeout=20 osc.lustre-OST0001-osc-ffff8800a985c800.idle_timeout=20 set checksum type to crc32c, rc = 0 PASS 77l (104s) == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 07:39:19 (1750592359) checksum_speed= adler32: 1071 crc32: 1343 crc32c: 5987 PASS 77m (4s) == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 07:39:23 (1750592363) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0101201 s, 405 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.0022147 s, 1.8 MB/s SKIP: sanity test_77n f77n.sanity blocks not contiguous around hole SKIP 77n (2s) == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 07:39:25 (1750592365) obdfilter.lustre-*.checksum_type: crc32 adler [crc32c] t10ip512 t10ip4K t10crc512 t10crc4K crc32 adler [crc32c] t10ip512 t10ip4K t10crc512 t10crc4K mdt.lustre-*.checksum_type: crc32 adler [crc32c] t10ip512 t10ip4K t10crc512 t10crc4K PASS 77o (6s) == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 07:39:31 (1750592371) MemFree: 3441, Max file size: 600000 MemTotal: 3730 Mem to use for directio: 1737 Smallest OST: 3766272 File size: 32 directIO rdwr round 1 of 1 directio on /mnt/lustre/f78.sanity for 32x1048576 bytes PASS PASS 78 (5s) == sanity test 79: df report consistency check =========== 07:39:36 (1750592376) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4480 2204160 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre 192.168.201.110@tcp:/lustre 7542784 6144 7532544 1% /mnt/lustre llite.lustre-ffff8800a985c800.maxbytes=18446744073709551615 lmv.lustre-clilmv-ffff8800a985c800.maxbytes=18446744073709551615 lov.lustre-clilov-ffff8800a985c800.maxbytes=18446744073709551615 mdc.lustre-MDT0000-mdc-ffff8800a985c800.maxbytes=18446744073709551615 osc.lustre-OST0000-osc-ffff8800a985c800.maxbytes=18446744073709551615 osc.lustre-OST0001-osc-ffff8800a985c800.maxbytes=18446744073709551615 llite.lustre-ffff8800a985c800.namelen_max=255 lmv.lustre-clilmv-ffff8800a985c800.namelen_max=256 lov.lustre-clilov-ffff8800a985c800.namelen_max=255 mdc.lustre-MDT0000-mdc-ffff8800a985c800.namelen_max=256 osc.lustre-OST0000-osc-ffff8800a985c800.namelen_max=255 osc.lustre-OST0001-osc-ffff8800a985c800.namelen_max=255 lod.lustre-MDT0000-mdtlov.maxbytes=18446744073709551615 lov.lustre-MDT0000-mdtlov.maxbytes=18446744073709551615 obdfilter.lustre-OST0000.maxbytes=18446744073709551615 obdfilter.lustre-OST0001.maxbytes=18446744073709551615 osc.lustre-OST0000-osc-MDT0000.maxbytes=18446744073709551615 osc.lustre-OST0001-osc-MDT0000.maxbytes=18446744073709551615 osd-zfs.lustre-MDT0000.maxbytes=18446744073709551615 osd-zfs.lustre-OST0000.maxbytes=18446744073709551615 osd-zfs.lustre-OST0001.maxbytes=18446744073709551615 osp.lustre-OST0000-osc-MDT0000.maxbytes=18446744073709551615 osp.lustre-OST0001-osc-MDT0000.maxbytes=18446744073709551615 lod.lustre-MDT0000-mdtlov.namelen_max=256 lov.lustre-MDT0000-mdtlov.namelen_max=256 obdfilter.lustre-OST0000.namelen_max=255 obdfilter.lustre-OST0001.namelen_max=255 osc.lustre-OST0000-osc-MDT0000.namelen_max=255 osc.lustre-OST0001-osc-MDT0000.namelen_max=255 osd-zfs.lustre-MDT0000.namelen_max=256 osd-zfs.lustre-OST0000.namelen_max=256 osd-zfs.lustre-OST0001.namelen_max=256 osp.lustre-OST0000-osc-MDT0000.namelen_max=255 osp.lustre-OST0001-osc-MDT0000.namelen_max=255 lod.lustre-MDT0000-mdtlov.maxbytes=18446744073709551615 lov.lustre-MDT0000-mdtlov.maxbytes=18446744073709551615 obdfilter.lustre-OST0000.maxbytes=18446744073709551615 obdfilter.lustre-OST0001.maxbytes=18446744073709551615 osc.lustre-OST0000-osc-MDT0000.maxbytes=18446744073709551615 osc.lustre-OST0001-osc-MDT0000.maxbytes=18446744073709551615 osd-zfs.lustre-MDT0000.maxbytes=18446744073709551615 osd-zfs.lustre-OST0000.maxbytes=18446744073709551615 osd-zfs.lustre-OST0001.maxbytes=18446744073709551615 osp.lustre-OST0000-osc-MDT0000.maxbytes=18446744073709551615 osp.lustre-OST0001-osc-MDT0000.maxbytes=18446744073709551615 lod.lustre-MDT0000-mdtlov.namelen_max=256 lov.lustre-MDT0000-mdtlov.namelen_max=256 obdfilter.lustre-OST0000.namelen_max=255 obdfilter.lustre-OST0001.namelen_max=255 osc.lustre-OST0000-osc-MDT0000.namelen_max=255 osc.lustre-OST0001-osc-MDT0000.namelen_max=255 osd-zfs.lustre-MDT0000.namelen_max=256 osd-zfs.lustre-OST0000.namelen_max=256 osd-zfs.lustre-OST0001.namelen_max=256 osp.lustre-OST0000-osc-MDT0000.namelen_max=255 osp.lustre-OST0001-osc-MDT0000.namelen_max=255 PASS 79 (19s) == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 07:39:55 (1750592395) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB) copied, 0.0543851 s, 19.3 MB/s PASS 80 (5s) == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 07:40:00 (1750592400) fail_loc=0x80000228 PASS 81a (4s) == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 07:40:04 (1750592404) fail_loc=0x228 write: No space left on device PASS 81b (3s) == sanity test 99: cvs strange file/directory operations ========================================================== 07:40:07 (1750592407) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cvs] [-d] [/mnt/lustre/d99.sanity.cvsroot] [init] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cvs] [-d] [/mnt/lustre/d99.sanity.cvsroot] [import] [-m] [nomesg] [d99.sanity.reposname] [vtag] [rtag] cvs import: cannot open /root/.cvsignore: Permission denied N d99.sanity.reposname/README N d99.sanity.reposname/network N d99.sanity.reposname/netconsole N d99.sanity.reposname/functions No conflicts created by this import running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cvs] [-d] [/mnt/lustre/d99.sanity.cvsroot] [co] [d99.sanity.reposname] cvs checkout: cannot open /root/.cvsignore: Permission denied cvs checkout: Updating d99.sanity.reposname U d99.sanity.reposname/README U d99.sanity.reposname/functions U d99.sanity.reposname/netconsole U d99.sanity.reposname/network running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [foo99] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cvs] [add] [-m] [addmsg] [foo99] cvs add: scheduling file `foo99' for addition cvs add: use 'cvs commit' to add this file permanently running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cvs] [update] cvs update: cannot open /root/.cvsignore: Permission denied cvs update: Updating . cvs [update aborted]: cannot open directory /mnt/lustre/d99.sanity.cvsroot/d99.sanity.reposname: Bad address running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cvs] [commit] [-m] [nomsg] [foo99] RCS file: /mnt/lustre/d99.sanity.cvsroot/d99.sanity.reposname/foo99,v done Checking in foo99; /mnt/lustre/d99.sanity.cvsroot/d99.sanity.reposname/foo99,v <-- foo99 initial revision: 1.1 done PASS 99 (15s) == sanity test 100: check local port using privileged port ========================================================== 07:40:22 (1750592422) PASS 100 (3s) == sanity test 101a: check read-ahead for random reads === 07:40:25 (1750592425) nreads: 10000 file size: 96MB 69.355167s, 9.44933MB/s osc.lustre-OST0000-osc-ffff8800a985c800.rpc_stats= snapshot_time: 1750592502.529854584 secs.nsecs start_time: 1750592426.645327810 secs.nsecs elapsed_time: 75.884526774 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 DIO RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 0 0 0 | 1 0 0 2: 0 0 0 | 1 0 1 4: 3 0 0 | 0 0 1 8: 1 0 0 | 1 0 2 16: 5001 99 100 | 0 0 2 32: 0 0 100 | 0 0 2 64: 0 0 100 | 0 0 2 128: 0 0 100 | 0 0 2 256: 0 0 100 | 100 97 100 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 5002 99 99 | 99 96 96 2: 3 0 100 | 4 3 100 read write offset rpcs % cum % | rpcs % cum % 0: 3 0 0 | 4 3 3 1: 0 0 0 | 0 0 3 2: 0 0 0 | 0 0 3 4: 0 0 0 | 0 0 3 8: 0 0 0 | 0 0 3 16: 4 0 0 | 0 0 3 32: 6 0 0 | 0 0 3 64: 16 0 0 | 0 0 3 128: 26 0 1 | 0 0 3 256: 44 0 1 | 2 1 5 512: 107 2 4 | 2 1 7 1024: 204 4 8 | 4 3 11 2048: 427 8 16 | 8 7 19 4096: 853 17 33 | 16 15 34 8192: 1670 33 67 | 32 31 66 16384: 1645 32 100 | 35 33 100 osc.lustre-OST0001-osc-ffff8800a985c800.rpc_stats= snapshot_time: 1750592502.529921080 secs.nsecs start_time: 1750592426.645349805 secs.nsecs elapsed_time: 75.884571275 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 DIO RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 0 0 0 | 3 100 100 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 0 0 0 | 2 66 66 2: 0 0 0 | 1 33 100 read write offset rpcs % cum % | rpcs % cum % 0: 0 0 0 | 3 100 100 llite.lustre-ffff8800a985c800.read_ahead_stats= snapshot_time 1750592502.535818252 secs.nsecs start_time 1750592426.651356355 secs.nsecs elapsed_time 75.884461897 secs.nsecs hits 75000 samples [pages] misses 5005 samples [pages] readpage_not_consecutive 9991 samples [pages] zero_size_window 75000 samples [pages] failed_to_fast_read 5005 samples [pages] readahead_pages 5005 samples [pages] 3 15 75000 PASS 101a (80s)