-----============= acceptance-small: sanityn ============----- Thu Oct 23 08:04:50 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg138-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 08:05:10 (1761221110) === Starting client oleg138-client.virtnet: -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Mount client oleg138-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.138@tcp:/lustre /mnt/lustre2 Started clients oleg138-client.virtnet: 192.168.201.138@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg138-client.virtnet: executing check_config_client /mnt/lustre oleg138-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg138-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff92d0c84b1800.idle_timeout=debug osc.lustre-OST0000-osc-ffff92d0cab1e000.idle_timeout=debug osc.lustre-OST0001-osc-ffff92d0c84b1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff92d0cab1e000.idle_timeout=debug disable quota as required oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanityn: finish setup 08:05:48 (1761221148) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038237 s, 27.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8200] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 08:05:51 (1761221151) PASS 0 (9s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 08:05:59 (1761221159) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 08:06:06 (1761221166) -rw-r--r-- 1 root root 0 Oct 23 08:06 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (6s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 08:06:12 (1761221172) -rw-r--r-- 1 root root 0 Oct 23 08:06 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (7s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 08:06:19 (1761221179) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 08:06:25 (1761221185) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (5s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 08:06:30 (1761221190) total 0 -rwxrwxrwx 1 root root 0 Oct 23 08:06 f2a -rwxrwxrwx 1 root root 0 Oct 23 08:06 f2b total 0 -rwxrwxrwx 1 root root 0 Oct 23 08:06 f2a -rwxrwxrwx 1 root root 0 Oct 23 08:06 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0139844 s, 36.6 kB/s PASS 2e (6s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 08:06:37 (1761221197) /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has type file OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has perms 0777 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by user #500 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by group #500 OK Can't lstat /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn: No such file or directory PASS 2f (8s) == sanityn test 2g: check blocks update on sync write ==== 08:06:45 (1761221205) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.406205 s, 5.2 MB/s /mnt/lustre/f2g.sanityn has 4096 blocks /mnt/lustre2/f2g.sanityn has 4096 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 08:06:53 (1761221213) PASS 3 (8s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 08:07:00 (1761221220) PASS 4 (9s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 08:07:10 (1761221230) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (8s) == sanityn test 6: remove of open file on other node ============================================================================ 08:07:18 (1761221238) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 08:07:25 (1761221245) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 08:07:31 (1761221251) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (5s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 08:07:37 (1761221257) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 08:07:44 (1761221264) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374784 s, 28.0 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.104191 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (6s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 08:07:50 (1761221270) 1+0 records in 1+0 records out 1 byte copied, 0.0047037 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0150024 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0135856 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0199635 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0103509 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0172479 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0107208 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0112676 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0184874 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0302087 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00856126 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0268677 s, 0.0 kB/s PASS 10a (7s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 08:07:57 (1761221277) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0135895 s, 226 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0382739 s, 107 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000253423 s, 12.1 MB/s PASS 10b (6s) == sanityn test 11: execution of file opened for write should return error ============================================================== 08:08:03 (1761221283) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d11 multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8200 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (7s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 08:08:10 (1761221290) warning: '-runtime' deprecated, use '-t runtime' instead - link 364 (time 1761221302.43 total 10.02 last 36.34) - link 697 (time 1761221312.43 total 20.02 last 33.30) - link 1053 (time 1761221322.44 total 30.02 last 35.58) - link 1442 (time 1761221332.44 total 40.03 last 38.87) - link 1826 (time 1761221342.46 total 50.05 last 38.34) - link 2246 (time 1761221352.47 total 60.06 last 41.96) - link 2595 (time 1761221362.48 total 70.06 last 34.88) - link 2920 (time 1761221372.50 total 80.09 last 32.41) - link 3248 (time 1761221382.52 total 90.11 last 32.73) total: 3584 link in 100.02 seconds: 35.83 ops/second using seed 739821980 running for 100 seconds - stat 10000 (time 1761221302 ; total 9 ; last 9) - stat 20000 (time 1761221312 ; total 19 ; last 10) - stat 30000 (time 1761221321 ; total 28 ; last 9) - stat 40000 (time 1761221330 ; total 37 ; last 9) - stat 50000 (time 1761221337 ; total 44 ; last 7) - stat 57698 (time 1761221348 ; total 55 ; last 11) - stat 60000 (time 1761221353 ; total 60 ; last 5) - stat 64733 (time 1761221364 ; total 71 ; last 11) - stat 69187 (time 1761221375 ; total 82 ; last 11) - stat 70000 (time 1761221377 ; total 84 ; last 2) - stat 74808 (time 1761221388 ; total 95 ; last 11) total: 77104 stats in 100 seconds: 771.039978 stats/second - unlinked 0 (time 1761221343 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3584) error: No such file or directory total: 3583 unlinks in 87 seconds: 41.183907 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (22683) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (22686) - No such process PASS 12 (146s) == sanityn test 13: test directory page revocation ======= 08:10:36 (1761221436) striped dir -i1 -c2 -H crush2 /mnt/lustre/d13 f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 08:10:45 (1761221445) striped dir -i0 -c2 -H crush /mnt/lustre/d14aa.sanityn multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.8200 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 23898 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 08:10:52 (1761221452) striped dir -i0 -c2 -H crush /mnt/lustre/d14ab.sanityn open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 24489 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 08:10:58 (1761221458) striped dir -i0 -c2 -H crush2 /mnt/lustre/d14b.sanityn truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 25079 Terminated $DIR1/$tdir/sleep 60 PASS 14b (5s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 08:11:03 (1761221463) striped dir -i0 -c2 -H all_char /mnt/lustre/d14c.sanityn cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 25673 Terminated $DIR1/$tdir/sleep 60 PASS 14c (6s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 08:11:09 (1761221469) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d14d.sanityn chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 26268 Terminated $DIR1/$tdir/sleep 60 PASS 14d (7s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 08:11:16 (1761221476) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky Waiting for MDT destroys to complete 0 STRIPECOUNT=2 ORIGFREE=7207348 MAXFREE=2097152000 0 lustre-OST0000 avl=0 grnt=2483 diff=-2483 limit=46331 FULL lustre-OST0001 avl=5300 grnt=5535 diff=-235 limit=46331 FULL ERROR: dd blocks 7192077 != file size 7192096 LOG LOG2 file /tmp/oos2.log:dd: error writing '/mnt/lustre/oosfile': No space left on device /tmp/oos2.log:3604006+0 records in /tmp/oos2.log:3604005+0 records out /tmp/oos2.log:3690501120 bytes (3.7 GB, 3.4 GiB) copied, 677.312 s, 5.4 MB/s /tmp/oos2.log2:dd: error writing '/mnt/lustre2/oosfile2': No space left on device /tmp/oos2.log2:3588073+0 records in /tmp/oos2.log2:3588072+0 records out /tmp/oos2.log2:3674185728 bytes (3.7 GB, 3.4 GiB) copied, 674.327 s, 5.4 MB/s Waiting for MDT destroys to complete Waiting for MDT destroys to complete PASS 15 (729s) == sanityn test 16a: 2500 iterations of dual-mount fsx === 08:23:25 (1761222205) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.185537 s, 45.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.182176 s, 46.0 MB/s Chance of close/open is 1 in 50 Seed set to 2207 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn skipping zero length zero range fallocating to largest ever: 0x1fa601 truncating to largest ever: 0x30a67f fallocating to largest ever: 0x91ee65 truncating to largest ever: 0x928454 fallocating to largest ever: 0x94e93e truncating to largest ever: 0x95c223 truncating to largest ever: 0x9c7689 000100[0] 1761222212.424616 fallocate 0x0565d8 thru 0x0601c9 (0x9bf2 bytes) fallocating to largest ever: 0x9e13cf truncating to largest ever: 0x9ed91e 000300[0] 1761222219.923706 fallocate 0x99fab4 thru 0x9a4f48 (0x5495 bytes) 000400[1] 1761222223.887286 fallocate 0x8d80dd thru 0x8dae0c (0x2d30 bytes) truncating to largest ever: 0x9f92bb 000500[0] 1761222227.585003 write 0x48a104 thru 0x490a20 (0x691d bytes) truncating to largest ever: 0x9fbb00 000600[0] 1761222231.406236 write 0x4b1363 thru 0x4b261b (0x12b9 bytes) truncating to largest ever: 0x9fc215 000700[1] 1761222235.065978 fallocate 0x7ecc4f thru 0x7f65aa (0x995c bytes) 000800[1] 1761222239.238107 fallocate 0x417db8 thru 0x421cd8 (0x9f21 bytes) 000900[0] 1761222243.218773 punch from 0x1f78d1 to 0x20603b (0xe76b bytes) 001000[0] 1761222246.306735 punch from 0x92a2ad to 0x92b8f2 (0x1646 bytes) 001100[1] 1761222250.324593 trunc from 0x809eb7 to 0x3e772f (0x422787 bytes) 001200[0] 1761222253.948732 mapread 0x02c3e8 thru 0x037eec (0xbb05 bytes) 001300[1] 1761222257.680179 trunc from 0x4565dc to 0x60c935 (0x1b635a bytes) 001400[1] 1761222261.320358 mapwrite 0x94fc7c thru 0x950cb4 (0x1039 bytes) 001500[1] 1761222263.608830 mapread 0x2c61cd thru 0x2d3806 (0xd63a bytes) 001600[1] 1761222266.418856 mapread 0x2917f5 thru 0x29f503 (0xdd0f bytes) 001700[0] 1761222269.929928 mapread 0x8c911a thru 0x8cb37d (0x2264 bytes) 001900[0] 1761222275.944884 trunc from 0x4e3564 to 0x422c55 (0xc090e bytes) 002300[0] 1761222289.753393 mapread 0x76f1ab thru 0x7700cd (0xf23 bytes) 002500[0] 1761222296.539502 mapwrite 0x9d38c6 thru 0x9d5fc1 (0x26fc bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 2296 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn truncating to largest ever: 0x7fc06f fallocating to largest ever: 0x93cafc fallocating to largest ever: 0x9d4548 fallocating to largest ever: 0x9e8bb9 000200[0] 1761222302.991501 read 0x247000 thru 0x2563c1 (0xf3c2 bytes) 000300[0] 1761222305.458864 punch from 0x3ed016 to 0x3f1365 (0x4350 bytes) truncating to largest ever: 0x9f7c5f 000400[1] 1761222308.490625 read_OD 0x3f1000 thru 0x3f5fff (0x5000 bytes) fallocating to largest ever: 0xa00000 000700[0] 1761222317.059847 read_OD 0x0d9000 thru 0x0dcfff (0x4000 bytes) 000900[1] 1761222322.257839 fallocate 0x3160ff thru 0x31afcd (0x4ecf bytes) 001000[1] 1761222325.269806 fallocate 0x4046c4 thru 0x41391c (0xf259 bytes) 001100[0] 1761222328.426599 read 0x0c2000 thru 0x0c55a1 (0x35a2 bytes) 001200[1] 1761222332.182884 trunc from 0x3d4505 to 0x3e0ae5 (0xc5e1 bytes) 001300[1] 1761222334.754052 write_OD 0x57f000 thru 0x580fff (0x2000 bytes) 001400[0] 1761222337.990197 punch from 0x54fca9 to 0x55bf99 (0xc2f1 bytes) 001600[1] 1761222344.281657 read_OD 0x2c1000 thru 0x2c2fff (0x2000 bytes) 001800[0] 1761222349.895935 read 0x1d1000 thru 0x1e090d (0xf90e bytes) 001900[1] 1761222353.335972 read_OD 0x1d2000 thru 0x1ddfff (0xc000 bytes) 002000[1] 1761222356.830311 punch from 0x0059e2 to 0x00a917 (0x4f36 bytes) 002400[0] 1761222368.497110 punch from 0x269fc1 to 0x26b3c0 (0x1400 bytes) 002500[0] 1761222371.361028 mapread 0x036000 thru 0x041080 (0xb081 bytes) All operations completed A-OK! PASS 16a (169s) == sanityn test 16b: 2500 iterations of dual-mount fsx at small size ========================================================== 08:26:14 (1761222374) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.178892 s, 46.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.158259 s, 53.0 MB/s Chance of close/open is 1 in 50 Seed set to 2376 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn skipping zero size read skipping zero length punch hole fallocating to largest ever: 0x160776 fallocating to largest ever: 0x3619c8 fallocating to largest ever: 0x8594a1 fallocating to largest ever: 0x87ac83 fallocating to largest ever: 0x8b8225 fallocating to largest ever: 0x92baf9 truncating to largest ever: 0x955b8e 000100[1] 1761222381.494673 mapread 0x815f4e thru 0x816b25 (0xbd8 bytes) truncating to largest ever: 0x9c0cac fallocating to largest ever: 0x9f43bf 000200[0] 1761222385.498723 mapwrite 0x8a2ddc thru 0x8a316d (0x392 bytes) 000400[1] 1761222392.691282 trunc from 0x5ed707 to 0x57ad40 (0x729c6 bytes) 000600[0] 1761222398.966849 fallocate 0x2a917d thru 0x2ab00e (0x1e92 bytes) 000800[0] 1761222405.599535 punch from 0x408937 to 0x409592 (0xc5c bytes) 000900[0] 1761222408.487146 trunc from 0x0096b8 to 0x00ab2b (0x1474 bytes) 001300[1] 1761222422.552893 trunc from 0x381fc5 to 0x383df2 (0x1e2e bytes) 001600[1] 1761222433.049801 write 0x6c51dd thru 0x6c5218 (0x3c bytes) fallocating to largest ever: 0x9f554b 001700[1] 1761222436.604705 trunc from 0x03c9ae to 0x03cb2b (0x17e bytes) 001900[1] 1761222443.146501 trunc from 0x8a04d3 to 0x8a071e (0x24c bytes) 002000[0] 1761222446.528243 read 0x0dc1bc thru 0x0dced7 (0xd1c bytes) 002300[1] 1761222457.316200 read 0x057d60 thru 0x059812 (0x1ab3 bytes) 002400[1] 1761222460.851483 punch from 0x01def7 to 0x01f6d5 (0x17df bytes) 002500[1] 1761222464.384715 fallocate 0x57b8f7 thru 0x57c4ed (0xbf7 bytes) All operations completed A-OK! PASS 16b (95s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 08:27:49 (1761222469) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.236611 s, 35.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.179855 s, 46.6 MB/s Waiting for MDT destroys to complete Chance of close/open is 1 in 50 Seed set to 2488 fd 0: /mnt/lustre/f16c.sanityn fd 1: /mnt/lustre2/f16c.sanityn fallocating to largest ever: 0x50a5c4 truncating to largest ever: 0x5c6191 fallocating to largest ever: 0x65b19f truncating to largest ever: 0x74266b truncating to largest ever: 0x910d15 000100[0] 1761222492.919597 trunc from 0x23b99d to 0x23f8d1 (0x3f35 bytes) fallocating to largest ever: 0x9f6337 000200[0] 1761222496.156595 mapwrite 0x8398be thru 0x83ddb7 (0x44fa bytes) 000300[0] 1761222500.044628 mapread 0x47c7e3 thru 0x482da4 (0x65c2 bytes) fallocating to largest ever: 0xa00000 000500[1] 1761222506.496211 write 0x01aa2f thru 0x022300 (0x78d2 bytes) 000700[0] 1761222514.951183 trunc from 0x1fcfbe to 0x1b104b (0x4bf72 bytes) 000800[1] 1761222519.398633 write 0x031183 thru 0x03276c (0x15ea bytes) 000900[0] 1761222524.088108 read 0x080749 thru 0x084b6e (0x4426 bytes) 001000[1] 1761222528.208987 fallocate 0x88da90 thru 0x89d7c3 (0xfd34 bytes) 001100[1] 1761222531.711983 mapread 0x7d7a81 thru 0x7db2ac (0x382c bytes) 001200[0] 1761222535.908714 trunc from 0x03f9f6 to 0x047199 (0x77a4 bytes) 001300[1] 1761222539.640830 read 0x40c5f0 thru 0x416d8a (0xa79b bytes) 001400[0] 1761222544.082988 fallocate 0x351133 thru 0x35d2aa (0xc178 bytes) 001700[1] 1761222556.394492 fallocate 0x89eada thru 0x8a9b73 (0xb09a bytes) 001800[1] 1761222559.363034 write 0x5f623a thru 0x5fcd54 (0x6b1b bytes) 001900[0] 1761222563.269325 mapread 0x87b845 thru 0x87c626 (0xde2 bytes) 002000[0] 1761222567.310224 mapread 0x5af836 thru 0x5bb5fa (0xbdc5 bytes) 002100[1] 1761222571.528936 punch from 0x580906 to 0x5824ff (0x1bfa bytes) 002200[1] 1761222575.371616 trunc from 0x031c54 to 0x5f6774 (0x5c4b21 bytes) 002300[0] 1761222579.365461 fallocate 0x0ea6e9 thru 0x0f8306 (0xdc1e bytes) All operations completed A-OK! PASS 16c (123s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 08:29:52 (1761222592) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.132352 s, 63.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.143152 s, 58.6 MB/s ldlm.namespaces.MGC192.168.201.138@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff92d0c84b1800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff92d0cab1e000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff92d0c84b1800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff92d0cab1e000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff92d0c84b1800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff92d0cab1e000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff92d0c84b1800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff92d0cab1e000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.38232 s, 23.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.16547 s, 48.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.43287 s, 43.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.76525 s, 22.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.53015 s, 29.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.30951 s, 31.7 MB/s PASS 16d (30s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 08:30:22 (1761222622) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.227221 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463032 s, 22.6 MB/s PASS 16e (6s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 08:30:28 (1761222628) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 08:30:56 (1761222656) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 08:31:25 (1761222685) striped dir -i0 -c2 -H crush2 /mnt/lustre/d16h.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (6s) == sanityn test 16i: read after truncate file ============ 08:31:32 (1761222692) striped dir -i0 -c2 -H all_char /mnt/lustre/d16i.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (7s) == sanityn test 16j: race dio with buffered i/o ========== 08:31:38 (1761222698) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.787343 s, 33.3 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.973787 s, 26.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.42273 s, 18.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.753587 s, 34.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.04692 s, 25.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.92428 s, 28.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.32189 s, 19.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.44705 s, 18.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.9206 s, 13.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.00517 s, 13.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.04258 s, 12.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.36961 s, 19.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.50731 s, 17.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.04192 s, 12.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.08156 s, 12.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.22783 s, 21.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.32857 s, 19.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.75197 s, 15.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.95105 s, 13.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (27s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 08:32:05 (1761222725) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 fallocating to largest ever: 0x3428364 skipping zero size read skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Thu Oct 23 08:32:43 EDT 2025 + fsx 2 done: rc=0 Thu Oct 23 08:32:43 EDT 2025 + fsx 3 done: rc=0 Thu Oct 23 08:32:43 EDT 2025 + fsx 4 done: rc=0 Thu Oct 23 08:32:43 EDT 2025 + fsx 5 done: rc=0 Thu Oct 23 08:32:43 EDT 2025 PASS 16k (46s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 08:32:52 (1761222772) fail_loc=0x8000030a PASS 17 (10s) == sanityn test 18: mmap sanity check =========================================================================================== 08:33:02 (1761222782) mmap test1: basic mmap operation (PASS, 0.13583s) mmap test2: MAP_PRIVATE not write back (PASS, 0.084192s) mmap test3: concurrent mmap ops on two nodes (PASS, 18.092s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.6836s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.1761s) mmap test6: check mmap write/read content on two nodes (PASS, 0.37527s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.7409s) mmap test11: multi-thread shared mmap (PASS, 1.821s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1309s) PASS 18 (38s) == sanityn test 19: test concurrent uncached read races ========================================================================= 08:33:40 (1761222820) 32+0 records in 32+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.5979 s, 28.1 MB/s loop 5 loop 10 loop 15 loop 20 PASS 19 (30s) == sanityn test 20: test extra readahead page left in cache ============================================================== 08:34:10 (1761222850) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d20.sanityn PASS 20 (5s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 08:34:15 (1761222855) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d21.sanityn rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (5s) == sanityn test 23: others should see updated atime while another read============================================================== 08:34:20 (1761222860) now is 1761222863 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.8200 new atime is 1761222924 PASS 23 (68s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 08:35:28 (1761222928) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2032 1269272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 8168 3598852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 30680 3576340 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38848 7175192 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2032 1269272 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1788 1269516 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 8168 3598852 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 30680 3576340 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38848 7175192 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 354 999.7K 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 256.0K 402 255.6K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 256.0K 403 255.6K 1% /mnt/lustre[OST:1] filesystem_summary: 511.8K 633 511.2K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 354 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 402 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 403 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.8K 633 511.2K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 1.3G 2.0M 1.2G 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1.3G 1.7M 1.2G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.7G 8.0M 3.4G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.7G 30.0M 3.4G 1% /mnt/lustre[OST:1] filesystem_summary: 7.3G 37.9M 6.8G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 354 1023646 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 402 261742 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 403 261741 1% /mnt/lustre2[OST:1] filesystem_summary: 524116 633 523483 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2032 1269272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 8168 3598852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 30680 3576340 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38848 7175192 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 354 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 402 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 403 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.8K 633 511.2K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 354 1023646 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 262144 402 261742 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 262144 403 261741 1% /mnt/lustre[OST:1] filesystem_summary: 524116 633 523483 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 354 1023646 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 402 261742 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 403 261741 1% /mnt/lustre2[OST:1] filesystem_summary: 524116 633 523483 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2032 1269272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 8168 3598852 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 30680 3576340 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38848 7175192 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2032 1269272 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1788 1269516 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 8168 3598852 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 30680 3576340 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38848 7175192 1% /mnt/lustre2 PASS 24a (6s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 08:35:34 (1761222934) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 08:35:40 (1761222940) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (5s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 08:35:45 (1761222945) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] PASS 25b (7s) == sanityn test 26a: allow mtime to get older ============ 08:35:52 (1761222952) utime: good mknod times 1761222952 <= 1761222953 <= 1761222953 for /mnt/lustre/f26a utime: good mknod times 1761222952 <= 1761222953 <= 1761222953 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (6s) == sanityn test 26b: sync mtime between ost and mds ====== 08:35:58 (1761222958) PASS 26b (8s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 08:36:06 (1761222966) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8200 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279613 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-10-23 08:36:08.000000000 -0400 Birth: 2025-10-23 08:36:08.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279613 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-10-23 08:36:08.000000000 -0400 Birth: 2025-10-23 08:36:08.000000000 -0400 PASS 26c (7s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 08:36:14 (1761222974) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d30.sanityn 0 PASS 30 (7s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 08:36:21 (1761222981) striped dir -i1 -c2 -H crush2 /mnt/lustre/d31a.sanityn fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 08:36:29 (1761222989) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg138-server mds-ost sync done. Waiting for MDT destroys to complete striped dir -i1 -c2 -H all_char /mnt/lustre/d31b.sanityn fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.138@tcp:/lustre 7666232 39976 7174036 1% /mnt/lustre2 PASS 31b (16s) == sanityn test 31r: open-rename(replace) race =========== 08:36:46 (1761223006) -rw-r--r-- 1 root root 0 Oct 23 08:36 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (9s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 08:36:55 (1761223015) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 47555 Terminated tail -f $tfile PASS 31s (7s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 08:37:03 (1761223023) PASS 31t (7s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 08:37:12 (1761223032) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (2s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 08:37:14 (1761223034) Stopping /mnt/lustre-mds1 (opts:) on oleg138-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg138-server: oleg138-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg138-client: oleg138-server: ssh exited with exit code 1 Started lustre-MDT0000 create file on 2nd stripe under striped directory /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre/d33c.sanityn/striped touch /mnt/lustre2/d33c.sanityn/striped/subfile SoLC count 2 Below operations shouldn't trigger Solc: create file on 2nd stripe after setattr chmod 777 /mnt/lustre/d33c.sanityn/striped; touch /mnt/lustre2/d33c.sanityn/striped/subfile create remote dir and local dir /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote create local dir after remote dir creation transaction commit /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote.3; do_facet mds2 /home/green/git/lustre-release/lustre/utils/lctl set_param -n osd*.*MDT0001.force_sync 1; mkdir /mnt/lustre/d33c.sanityn/local.3 Solc count 0 PASS 33c (43s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 08:37:57 (1761223077) remote directory create /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre2/d33d.sanityn/subdir CoS count 2 cross-MDT rename mkdir /mnt/lustre/d33d.sanityn; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn/d1; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33d.sanityn/d2; touch /mnt/lustre/d33d.sanityn/d1/src mv /mnt/lustre2/d33d.sanityn/d1/src /mnt/lustre2/d33d.sanityn/d2/tgt CoS count 1 migrate /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs migrate -m 1 /mnt/lustre2/d33d.sanityn CoS count 2 PASS 33d (41s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 08:38:38 (1761223118) plain dir creation mkdir /mnt/lustre2/d33e.sanityn/plain open file and write echo abc > /mnt/lustre2/d33e.sanityn/f33e.sanityn append write echo dfg >> /mnt/lustre2/d33e.sanityn/f33e.sanityn setattr touch /mnt/lustre2/d33e.sanityn/f33e.sanityn file unlink rm /mnt/lustre2/d33e.sanityn/f33e.sanityn plain dir unlink rmdir /mnt/lustre2/d33e.sanityn/plain striped directory creation /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre2/d33e.sanityn/striped set default LMV to create striped subdir /home/green/git/lustre-release/lustre/utils/lfs setdirstripe -D -c 2 /mnt/lustre/d33e.sanityn striped subdir creation createmany -d /mnt/lustre/d33e.sanityn/subdir 100 total: 100 mkdir in 1.55 seconds: 64.66 ops/second sub file creation and write createmany -o /mnt/lustre/d33e.sanityn/subfile 100; echo abc > /mnt/lustre/d33e.sanityn/subfile1 total: 100 open/close in 1.35 seconds: 74.33 ops/second sub file append write echo dfg >> /mnt/lustre2/d33e.sanityn/subfile2 subdir setatttr touch /mnt/lustre2/d33e.sanityn/subdir1 subdir unlink unlinkmany -d /mnt/lustre/d33e.sanityn/subdir 100 - unlinked 0 (time 1761223124 ; total 0 ; last 0) total: 100 unlinks in 6 seconds: 16.666666 unlinks/second sub file unlink unlinkmany /mnt/lustre2/d33e.sanityn/subfile 100 - unlinked 0 (time 1761223130 ; total 0 ; last 0) total: 100 unlinks in 2 seconds: 50.000000 unlinks/second sub file creation follows striped dir chmod chmod 777 /mnt/lustre/d33e.sanityn/striped; touch /mnt/lustre/d33e.sanityn/striped/subfile striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn/striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn CoS count 0 PASS 33e (20s) == sanityn test 34: no lock timeout under IO ============= 08:38:58 (1761223138) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff92d0c84b1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92d0c84b1800.ost_server_uuid in FULL state after 0 sec oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92d0c84b1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92d0c84b1800.ost_server_uuid in IDLE state after 0 sec oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff92d0c84b1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92d0c84b1800.ost_server_uuid in IDLE state after 0 sec oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92d0c84b1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92d0c84b1800.ost_server_uuid in IDLE state after 0 sec oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff92d0c84b1800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff92d0c84b1800.ost_server_uuid in IDLE state after 0 sec oleg138-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff92d0c84b1800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff92d0c84b1800.ost_server_uuid in IDLE state after 0 sec PASS 34 (108s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 08:40:46 (1761223246) striped dir -i1 -c2 -H crush2 /mnt/lustre/d35.sanityn fail_loc=0x80000317 Race attempt 0 Wait for 57554 57642 for 60 sec... - open/close 866 (time 1761223261.26 total 10.01 last 86.55) - open/close 1641 (time 1761223271.26 total 20.01 last 77.49) - open/close 2445 (time 1761223281.27 total 30.02 last 80.28) - open/close 3159 (time 1761223291.28 total 40.03 last 71.34) - open/close 3902 (time 1761223301.29 total 50.04 last 74.23) total: 4000 open/close in 51.32 seconds: 77.95 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.201.138@tcp:/lustre 7.4G 40M 6.9G 1% /mnt/lustre 192.168.201.138@tcp:/lustre 7.4G 40M 6.9G 1% /mnt/lustre2 PASS 35 (70s) == sanityn test 36: handle ESTALE/open-unlink correctly == 08:41:56 (1761223316) striped dir -i0 -c2 -H crush2 /mnt/lustre/d36.sanityn Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg138-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.39623 s, 37.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8200 Waiting for MDT destroys to complete *** cycle(0) *** before(7174008) after_dd(7122808) after(7174008) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.40157 s, 37.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8200