-----============= acceptance-small: sanity ============----- Tue Apr 1 13:15:20 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 225 255 256 400a 42a 42c 42b 118c 118d 407 56x 56xa 56xb 411b skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 === sanity: start setup 13:15:26 (1743527726) === oleg337-client.virtnet: executing check_config_client /mnt/lustre oleg337-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg337-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b6714000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6714000.idle_timeout=debug disable quota as required oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity: finish setup 13:15:37 (1743527737) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f7551] preparing for tests involving mounts mke2fs 1.46.2.wc5 (26-Mar-2022) debug=all == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 13:15:38 (1743527738) SKIP: sanity test_60a missing subtest run-llog.sh SKIP 60a (2s) == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 13:15:40 (1743527740) PASS 60b (2s) == sanity test 60c: unlink file when mds full ============ 13:15:42 (1743527742) create 5000 files - open/close 1417 (time 1743527754.24 total 10.00 last 141.64) - open/close 2884 (time 1743527764.25 total 20.01 last 146.66) - open/close 4613 (time 1743527774.25 total 30.01 last 172.87) total: 5000 open/close in 32.32 seconds: 154.71 ops/second fail_loc=0x80000137 - unlinked 0 (time 1743527777 ; total 0 ; last 0) total: 5000 unlinks in 17 seconds: 294.117645 unlinks/second fail_loc=0 PASS 60c (53s) == sanity test 60d: test printk console message masking == 13:16:35 (1743527795) printk=0 emerg PASS 60d (2s) == sanity test 60e: no space while new llog is being created ========================================================== 13:16:37 (1743527797) fail_loc=0x15b PASS 60e (3s) == sanity test 60f: change debug_path works ============== 13:16:40 (1743527800) debug_path=/tmp/f60f.sanity fail_loc=0x8000050e striped dir -i0 -c2 -H crush2 /mnt/lustre/d60f.sanity ls: cannot access /tmp/f60f.sanity*: No such file or directory 0 /tmp/f60f.sanity.1743527801.13311 debug_path=/tmp/lustre-log PASS 60f (2s) == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 13:16:43 (1743527803) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d60g.sanity /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4758: 14138 Killed ( local index=0; while true; do $LFS setdirstripe -i $(($index % $MDSCOUNT)) -c $MDSCOUNT $DIR/$tdir/subdir$index 2> /dev/null; mkdir $DIR/$tdir/subdir$index 2> /dev/null; rmdir $DIR/$tdir/subdir$index 2> /dev/null; index=$((index + 1)); done ) Started LFSCK on the device lustre-MDT0000: scrub namespace /mnt/lustre/d60g.sanity: subdir121 subdir125 subdir13 subdir192 subdir242 subdir251 /mnt/lustre/d60g.sanity/subdir121: ls: closing directory /mnt/lustre/d60g.sanity/subdir121: No such file or directory /mnt/lustre/d60g.sanity/subdir125: /mnt/lustre/d60g.sanity/subdir13: /mnt/lustre/d60g.sanity/subdir192: /mnt/lustre/d60g.sanity/subdir242: /mnt/lustre/d60g.sanity/subdir251: PASS 60g (38s) == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 13:17:21 (1743527841) fail_loc=0x80000188 /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9658: /mnt/lustre/d60h.sanity-0x80000188/2: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9658: /mnt/lustre/d60h.sanity-0x80000188/3: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9658: /mnt/lustre/d60h.sanity-0x80000188/4: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9658: /mnt/lustre/d60h.sanity-0x80000188/7: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9658: /mnt/lustre/d60h.sanity-0x80000188/8: No such device lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000400:0xe8:0x0] 0 [0:0x0:0x0] lmv_stripe_count: 3 lmv_stripe_offset: 1 lmv_hash_type: crush,migrating,fixed mdtidx FID[seq:oid:ver] 1 [0x240000400:0xe6:0x0] 0 [0x200000400:0xe8:0x0] 0 [0:0x0:0x0] /mnt/lustre/d60h.sanity-0x80000188 ~ ~ fail_loc=0x80000189 /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9658: /mnt/lustre/d60h.sanity-0x80000189/2: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9658: /mnt/lustre/d60h.sanity-0x80000189/3: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9658: /mnt/lustre/d60h.sanity-0x80000189/4: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9658: /mnt/lustre/d60h.sanity-0x80000189/7: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9658: /mnt/lustre/d60h.sanity-0x80000189/8: No such device lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000400:0xe9:0x0] 0 [0:0x0:0x0] lmv_stripe_count: 3 lmv_stripe_offset: 1 lmv_hash_type: crush,migrating,fixed mdtidx FID[seq:oid:ver] 1 [0x240000400:0xe7:0x0] 0 [0x200000400:0xe9:0x0] 0 [0:0x0:0x0] /mnt/lustre/d60h.sanity-0x80000189 ~ ~ PASS 60h (5s) SKIP: sanity test_60i skipping SLOW test 60i == sanity test 60j: llog_reader reports corruptions ====== 13:17:26 (1743527846) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0001: clear the changelog for cl1 of all records total: 100 open/close in 0.66 seconds: 150.61 ops/second - unlinked 0 (time 1743527851 ; total 0 ; last 0) total: 100 unlinks in 0 seconds: inf unlinks/second oleg337-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg337-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps SKIP: sanity test_60j path oi.1/0x1:0xc:0x0 is not in 'O/1/d/' format lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 SKIP 60j (8s) == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 13:17:34 (1743527854) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00784403 s, 522 kB/s PASS 61a (3s) == sanity test 61b: mmap() of unstriped file is successful ========================================================== 13:17:37 (1743527857) PASS 61b (2s) == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 13:17:40 (1743527860) /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9792: 19847 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9792: 19855 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9792: 19863 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9792: 19870 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9792: 19878 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9792: 19886 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9792: 19894 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9792: 19902 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9792: 19910 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9792: 19917 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4004 1283684 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3072 1284616 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 46592 3560212 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 48172 7165652 1% /mnt/lustre pass grant check: client:582598656 server:582598656 PASS 63a (63s) == sanity test 63b: async write errors should be returned to fsync ============================================================= 13:18:42 (1743527922) debug=-1 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00484879 s, 845 kB/s fail_loc=0x80000406 fsync: Input/output error checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4004 1283684 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3072 1284616 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605268 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210708 1% /mnt/lustre pass grant check: client:582598656 server:582598656 PASS 63b (5s) == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 13:18:47 (1743527927) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4004 1283684 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3072 1284616 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605268 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210708 1% /mnt/lustre osc.lustre-OST0000-osc-ffff8800b6714000.cur_lost_grant_bytes=28672 osc.lustre-OST0001-osc-ffff8800b6714000.cur_lost_grant_bytes=3940352 osc.lustre-OST0000-osc-ffff8800b6714000.cur_grant_bytes=92467200 osc.lustre-OST0001-osc-ffff8800b6714000.cur_grant_bytes=486162432 checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4004 1283684 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3072 1284616 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605268 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210708 1% /mnt/lustre pass grant check: client:582598656 server:582598656 PASS 64a (1s) SKIP: sanity test_64b skipping SLOW test 64b == sanity test 64c: verify grant shrink ================== 13:18:49 (1743527929) osc.lustre-OST0000-osc-ffff8800b6714000.cur_grant_bytes=0 checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4004 1283684 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3072 1284616 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1580 3605440 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605268 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3116 7210708 1% /mnt/lustre pass grant check: client:582569984 server:582569984 PASS 64c (2s) == sanity test 64d: check grant limit exceed ============= 13:18:51 (1743527931) 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 6.64869 s, 158 MB/s /home/green/git/lustre-release/lustre/tests/sanity.sh: line 9921: kill: (23052) - No such process checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4004 1283684 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3072 1284616 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1025584 2581436 29% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1536 3605268 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 1027120 6186704 15% /mnt/lustre pass grant check: client:979984384 server:979984384 Waiting for MDT destroys to complete PASS 64d (20s) == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 13:19:11 (1743527951) debug=+cache Stopping client oleg337-client.virtnet /mnt/lustre (opts:)