-----============= acceptance-small: sanityn ============----- Thu Oct 23 08:04:58 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg141-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 08:05:17 (1761221117) === Starting client oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre2 Mount client oleg141-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre2 Started clients oleg141-client.virtnet: 192.168.201.141@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg141-client.virtnet: executing check_config_client /mnt/lustre oleg141-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg141-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff992c04ef3800.idle_timeout=debug osc.lustre-OST0000-osc-ffff992c19891800.idle_timeout=debug osc.lustre-OST0001-osc-ffff992c04ef3800.idle_timeout=debug osc.lustre-OST0001-osc-ffff992c19891800.idle_timeout=debug disable quota as required oleg141-server: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanityn: finish setup 08:05:56 (1761221156) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0175725 s, 59.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8239] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 08:05:59 (1761221159) PASS 0 (8s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 08:06:08 (1761221168) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (6s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 08:06:15 (1761221175) -rw-r--r-- 1 root root 0 Oct 23 08:06 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (6s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 08:06:21 (1761221181) -rw-r--r-- 1 root root 0 Oct 23 08:06 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 08:06:27 (1761221187) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 08:06:34 (1761221194) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (6s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 08:06:41 (1761221201) total 0 -rwxrwxrwx 1 root root 0 Oct 23 08:06 f2a -rwxrwxrwx 1 root root 0 Oct 23 08:06 f2b total 0 -rwxrwxrwx 1 root root 0 Oct 23 08:06 f2a -rwxrwxrwx 1 root root 0 Oct 23 08:06 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0196136 s, 26.1 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 08:06:48 (1761221208) /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has type file OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has perms 0777 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by user #500 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by group #500 OK Can't lstat /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn: No such file or directory PASS 2f (7s) == sanityn test 2g: check blocks update on sync write ==== 08:06:55 (1761221215) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.2954 s, 7.1 MB/s /mnt/lustre/f2g.sanityn has 4096 blocks /mnt/lustre2/f2g.sanityn has 4096 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 08:07:03 (1761221223) PASS 3 (8s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 08:07:10 (1761221230) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 08:07:19 (1761221239) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 08:07:25 (1761221245) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (6s) == sanityn test 7: remove of open directory on other node ======================================================================= 08:07:31 (1761221251) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (5s) == sanityn test 8: remove of open special file on other node ==================================================================== 08:07:36 (1761221256) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 08:07:43 (1761221263) PASS 9a (6s) == sanityn test 9b: append to striped sparse file ======== 08:07:50 (1761221270) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.048938 s, 21.4 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0581827 s, 0.1 kB/s Data read (expecting 'foo'): foo PASS 9b (6s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 08:07:56 (1761221276) 1+0 records in 1+0 records out 1 byte copied, 0.0125688 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0178013 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0157398 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0181058 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0173729 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0102804 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00876366 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0222395 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0144004 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0112417 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0097354 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00817269 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 08:08:04 (1761221284) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0185299 s, 166 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0386916 s, 106 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00510978 s, 601 kB/s PASS 10b (6s) == sanityn test 11: execution of file opened for write should return error ============================================================== 08:08:10 (1761221290) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d11 multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8239 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 08:08:18 (1761221298) warning: '-runtime' deprecated, use '-t runtime' instead - link 354 (time 1761221310.62 total 10.02 last 35.32) - link 694 (time 1761221320.63 total 20.03 last 33.97) - link 1081 (time 1761221330.64 total 30.04 last 38.64) - link 1458 (time 1761221340.65 total 40.05 last 37.68) - link 1844 (time 1761221350.68 total 50.08 last 38.49) - link 2175 (time 1761221360.69 total 60.09 last 33.07) - link 2492 (time 1761221370.71 total 70.12 last 31.61) - link 2822 (time 1761221380.72 total 80.12 last 32.97) - link 3159 (time 1761221390.77 total 90.17 last 33.54) total: 3495 link in 100.00 seconds: 34.95 ops/second using seed 2034049527 running for 100 seconds - stat 9568 (time 1761221312 ; total 11 ; last 11) - stat 10000 (time 1761221312 ; total 11 ; last 0) - stat 20000 (time 1761221322 ; total 21 ; last 10) - stat 30000 (time 1761221330 ; total 29 ; last 8) - stat 40000 (time 1761221339 ; total 38 ; last 9) - stat 50000 (time 1761221347 ; total 46 ; last 8) - stat 56755 (time 1761221358 ; total 57 ; last 11) - stat 60000 (time 1761221367 ; total 66 ; last 9) - stat 63776 (time 1761221378 ; total 77 ; last 11) - stat 68017 (time 1761221389 ; total 88 ; last 11) - stat 70000 (time 1761221393 ; total 92 ; last 4) total: 73175 stats in 100 seconds: 731.750000 stats/second - unlinked 0 (time 1761221351 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3495) error: No such file or directory total: 3494 unlinks in 87 seconds: 40.160919 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (22712) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (22714) - No such process PASS 12 (145s) == sanityn test 13: test directory page revocation ======= 08:10:44 (1761221444) striped dir -i1 -c2 -H all_char /mnt/lustre/d13 f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 08:10:52 (1761221452) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d14aa.sanityn multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.8239 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 23924 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 08:10:59 (1761221459) striped dir -i0 -c2 -H crush2 /mnt/lustre/d14ab.sanityn open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 24515 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (6s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 08:11:05 (1761221465) striped dir -i0 -c2 -H all_char /mnt/lustre/d14b.sanityn truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 25105 Terminated $DIR1/$tdir/sleep 60 PASS 14b (6s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 08:11:11 (1761221471) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d14c.sanityn cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 25700 Terminated $DIR1/$tdir/sleep 60 PASS 14c (6s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 08:11:17 (1761221477) striped dir -i0 -c2 -H crush2 /mnt/lustre/d14d.sanityn chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 26295 Terminated $DIR1/$tdir/sleep 60 PASS 14d (8s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 08:11:25 (1761221485) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky Waiting for MDT destroys to complete 0 STRIPECOUNT=2 ORIGFREE=7207412 MAXFREE=2097152000 0 lustre-OST0000 avl=1216 grnt=1235 diff=-19 limit=46331 FULL lustre-OST0001 avl=348 grnt=323 diff=25 limit=46331 FULL ERROR: dd blocks 7198252 != file size 7198268 LOG LOG2 file /tmp/oos2.log:dd: error writing '/mnt/lustre/oosfile': No space left on device /tmp/oos2.log:3588913+0 records in /tmp/oos2.log:3588912+0 records out /tmp/oos2.log:3675045888 bytes (3.7 GB, 3.4 GiB) copied, 687.098 s, 5.3 MB/s /tmp/oos2.log2:dd: error writing '/mnt/lustre2/oosfile2': No space left on device /tmp/oos2.log2:3609341+0 records in /tmp/oos2.log2:3609340+0 records out /tmp/oos2.log2:3695964160 bytes (3.7 GB, 3.4 GiB) copied, 687.138 s, 5.4 MB/s Waiting for MDT destroys to complete Waiting for MDT destroys to complete PASS 15 (738s) == sanityn test 16a: 2500 iterations of dual-mount fsx === 08:23:43 (1761222223) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.142011 s, 59.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.153099 s, 54.8 MB/s Chance of close/open is 1 in 50 Seed set to 2226 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fallocating to largest ever: 0x86dfc6 truncating to largest ever: 0x89c117 fallocating to largest ever: 0x9ce852 000100[1] 1761222229.865828 trunc from 0x22df9d to 0x5ed3da (0x3bf43e bytes) 000200[0] 1761222234.009020 punch from 0x1e79b1 to 0x1f3905 (0xbf55 bytes) 000300[1] 1761222238.526201 mapwrite 0x651928 thru 0x65a5b8 (0x8c91 bytes) 000400[1] 1761222242.632496 trunc from 0x0b1260 to 0x0b2809 (0x15aa bytes) fallocating to largest ever: 0x9ed7e1 000500[0] 1761222246.173968 fallocate 0x4438fc thru 0x4506d7 (0xcddc bytes) 000600[1] 1761222249.293230 fallocate 0x5f88cb thru 0x602d6d (0xa4a3 bytes) 000800[1] 1761222255.237662 mapwrite 0x2f0f57 thru 0x2f9443 (0x84ed bytes) 000900[0] 1761222258.661285 mapread 0x709977 thru 0x709b3a (0x1c4 bytes) 001100[0] 1761222265.247225 punch from 0x3ef01a to 0x3fc3dc (0xd3c3 bytes) 001400[0] 1761222275.492259 trunc from 0x6ffcd0 to 0x7dd8ca (0xddbfb bytes) 001600[1] 1761222282.999645 write 0x6ba762 thru 0x6be8f6 (0x4195 bytes) 001700[0] 1761222286.516899 write 0x506a5b thru 0x50ed59 (0x82ff bytes) 001800[1] 1761222289.947945 fallocate 0x1bacf1 thru 0x1bc066 (0x1376 bytes) 001900[1] 1761222293.434227 punch from 0x39337a to 0x3a3197 (0xfe1e bytes) 002000[0] 1761222297.146475 punch from 0x5810db to 0x5852ba (0x41e0 bytes) 002100[0] 1761222300.843317 trunc from 0x42417e to 0x4a89c0 (0x84843 bytes) 002200[1] 1761222303.945043 trunc from 0x156e51 to 0x15db0e (0x6cbe bytes) 002300[0] 1761222307.911272 punch from 0x671fbb to 0x67a34e (0x8394 bytes) truncating to largest ever: 0x9fdf9e 002400[1] 1761222312.341153 read 0x440e6d thru 0x445d5c (0x4ef0 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 2316 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fallocating to largest ever: 0x4f753d truncating to largest ever: 0x6bba18 fallocating to largest ever: 0x8036c8 fallocating to largest ever: 0x8fad72 truncating to largest ever: 0x9a0289 truncating to largest ever: 0x9b4edf truncating to largest ever: 0x9f2141 000100[0] 1761222319.466932 fallocate 0x2e72d4 thru 0x2f4f43 (0xdc70 bytes) truncating to largest ever: 0x9fb3a3 000200[0] 1761222322.047005 fallocate 0x45fe64 thru 0x4660d7 (0x6274 bytes) 000300[0] 1761222324.764201 mapread 0x535000 thru 0x53b203 (0x6204 bytes) 000400[1] 1761222327.982726 trunc from 0x2dd0f6 to 0x2e3e65 (0x6d70 bytes) 000500[1] 1761222331.203398 mapread 0x15e000 thru 0x16b72e (0xd72f bytes) 000800[0] 1761222339.433039 write 0x44e000 thru 0x45bf4d (0xdf4e bytes) 001000[0] 1761222345.409488 mapread 0x6e9000 thru 0x6efa20 (0x6a21 bytes) 001100[0] 1761222348.018702 mapread 0x6ba000 thru 0x6c6aec (0xcaed bytes) 001200[0] 1761222351.546176 punch from 0x6b3dd6 to 0x6bda1f (0x9c4a bytes) 001300[1] 1761222354.837008 read 0x104000 thru 0x1123c0 (0xe3c1 bytes) truncating to largest ever: 0x9ff403 fallocating to largest ever: 0xa00000 001700[0] 1761222366.826240 read 0x014000 thru 0x0216d5 (0xd6d6 bytes) 001800[0] 1761222369.702803 write 0x18f000 thru 0x19cfb9 (0xdfba bytes) 001900[1] 1761222372.310380 write 0x00d000 thru 0x00f958 (0x2959 bytes) 002000[1] 1761222375.329605 trunc from 0x556c4a to 0x3b670b (0x1a053e bytes) 002100[1] 1761222378.125770 fallocate 0x15f195 thru 0x161538 (0x23a4 bytes) 002200[1] 1761222381.348259 mapread 0x005000 thru 0x0148ec (0xf8ed bytes) 002300[0] 1761222385.198102 mapwrite 0x810000 thru 0x818b7e (0x8b7f bytes) 002400[0] 1761222388.433426 read_OD 0x45c000 thru 0x45cfff (0x1000 bytes) 002500[0] 1761222391.469610 read_OD 0x397000 thru 0x39dfff (0x7000 bytes) All operations completed A-OK! PASS 16a (172s) == sanityn test 16b: 2500 iterations of dual-mount fsx at small size ========================================================== 08:26:35 (1761222395) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.161658 s, 51.9 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.185989 s, 45.1 MB/s Chance of close/open is 1 in 50 Seed set to 2397 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn skipping zero size read skipping zero length zero range skipping zero length zero range skipping zero size read skipping zero length punch hole skipping zero length punch hole truncating to largest ever: 0x67f1b1 fallocating to largest ever: 0x6d92f9 fallocating to largest ever: 0x991dc9 truncating to largest ever: 0x9d39c0 000100[0] 1761222401.721596 mapwrite 0x7b7284 thru 0x7b8727 (0x14a4 bytes) fallocating to largest ever: 0x9fe36e 000400[0] 1761222412.716010 mapwrite 0x7e99a3 thru 0x7e9b34 (0x192 bytes) fallocating to largest ever: 0xa00000 000500[1] 1761222416.514041 read 0x16883f thru 0x16a4a7 (0x1c69 bytes) 000600[1] 1761222420.051191 mapread 0x3e40d4 thru 0x3e52be (0x11eb bytes) 000700[1] 1761222423.886227 mapwrite 0x681c80 thru 0x681db0 (0x131 bytes) 000800[0] 1761222427.592337 write 0x4a3e0d thru 0x4a42a0 (0x494 bytes) 000900[0] 1761222431.225249 mapread 0x9c80ee thru 0x9c8415 (0x328 bytes) 001000[1] 1761222434.771035 mapread 0x4d2dae thru 0x4d363b (0x88e bytes) 001100[0] 1761222438.146374 mapread 0x456c75 thru 0x45837b (0x1707 bytes) 001200[1] 1761222441.309795 mapwrite 0x1fe7f4 thru 0x2004cd (0x1cda bytes) 001300[1] 1761222444.913806 trunc from 0x5c8580 to 0x5c985b (0x12dc bytes) 001400[0] 1761222448.273764 trunc from 0x4fac0f to 0x0e3f5d (0x416cb1 bytes) 001500[0] 1761222451.943562 mapwrite 0x209a17 thru 0x20a1c7 (0x7b1 bytes) 001600[1] 1761222455.475199 trunc from 0x7df7c6 to 0x7e0e06 (0x1641 bytes) 001700[0] 1761222459.060904 read 0x009733 thru 0x00b6b7 (0x1f85 bytes) 001800[1] 1761222462.295975 trunc from 0x7f0d90 to 0x7f2a60 (0x1cd1 bytes) 001900[0] 1761222466.144184 trunc from 0x9ae5df to 0x8fc5e7 (0xb1ff7 bytes) 002100[1] 1761222473.433847 trunc from 0x76ae81 to 0x065279 (0x705c07 bytes) 002200[0] 1761222477.166064 punch from 0x4df13c to 0x4df5cb (0x490 bytes) skipping zero length punch hole 002400[0] 1761222484.168537 fallocate 0x78d200 thru 0x78db27 (0x928 bytes) 002500[1] 1761222488.037537 mapwrite 0x2cbc67 thru 0x2cbeba (0x254 bytes) All operations completed A-OK! PASS 16b (96s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 08:28:11 (1761222491) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.166518 s, 50.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.149621 s, 56.1 MB/s Waiting for MDT destroys to complete Chance of close/open is 1 in 50 Seed set to 2507 fd 0: /mnt/lustre/f16c.sanityn fd 1: /mnt/lustre2/f16c.sanityn zero_range to largest ever: 0x2d7d20 truncating to largest ever: 0x7d4283 fallocating to largest ever: 0x84aa77 truncating to largest ever: 0x934c96 fallocating to largest ever: 0x9d2c2b 000100[0] 1761222511.288946 trunc from 0x5ac8b0 to 0x0f7498 (0x4b5417 bytes) 000200[0] 1761222516.426790 mapread 0x371daf thru 0x379d7b (0x7fcd bytes) 000400[0] 1761222524.770359 mapread 0x491e1f thru 0x4a12f8 (0xf4da bytes) 000500[0] 1761222528.633792 mapwrite 0x9d4058 thru 0x9dd4c4 (0x946d bytes) 000600[0] 1761222532.801572 trunc from 0x977189 to 0x04670b (0x930a7d bytes) fallocating to largest ever: 0x9d7e89 000700[1] 1761222536.674899 trunc from 0x531606 to 0x333f50 (0x1fd6b5 bytes) fallocating to largest ever: 0x9e4920 000900[1] 1761222546.199368 fallocate 0x559d47 thru 0x561153 (0x740d bytes) fallocating to largest ever: 0xa00000 001100[1] 1761222554.204153 punch from 0x740353 to 0x74a488 (0xa136 bytes) 001200[0] 1761222557.983663 fallocate 0x4ba82c thru 0x4c5926 (0xb0fb bytes) 001300[1] 1761222562.346560 write 0x10b874 thru 0x1129cc (0x7159 bytes) 001400[1] 1761222566.387778 fallocate 0x1e4ead thru 0x1efeb3 (0xb007 bytes) 001500[1] 1761222570.931838 punch from 0x316be6 to 0x31e647 (0x7a62 bytes) 001600[1] 1761222575.254843 trunc from 0x894d91 to 0x275885 (0x61f50b bytes) 001700[0] 1761222579.182147 mapwrite 0x8c6282 thru 0x8cbf53 (0x5cd2 bytes) 001800[0] 1761222583.301732 trunc from 0x890608 to 0x2cc15c (0x5c44ab bytes) 001900[1] 1761222587.875864 trunc from 0x7d16af to 0x7d65fa (0x4f4c bytes) 002000[0] 1761222592.043677 read 0x4e16f8 thru 0x4e64af (0x4db8 bytes) 002100[0] 1761222595.896272 trunc from 0x2d18c9 to 0x2213d4 (0xb04f4 bytes) 002300[1] 1761222603.574854 mapread 0x032878 thru 0x0340f7 (0x1880 bytes) 002400[1] 1761222607.763499 read 0x75f071 thru 0x762ef4 (0x3e84 bytes) All operations completed A-OK! PASS 16c (128s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 08:30:19 (1761222619) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.197415 s, 42.5 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.176507 s, 47.5 MB/s ldlm.namespaces.MGC192.168.201.141@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff992c04ef3800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff992c19891800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff992c04ef3800.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff992c19891800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff992c04ef3800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff992c19891800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff992c04ef3800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff992c19891800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.72869 s, 18.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.719 s, 38.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.63894 s, 39.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.99525 s, 17.5 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.33718 s, 31.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.39576 s, 30.9 MB/s PASS 16d (36s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 08:30:55 (1761222655) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.244656 s, 42.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040931 s, 25.6 MB/s PASS 16e (6s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 08:31:01 (1761222661) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 08:31:30 (1761222690) TIMEOUT PASS 16g (28s) == sanityn test 16h: mmap read after truncate file ======= 08:31:58 (1761222718) striped dir -i0 -c2 -H crush /mnt/lustre/d16h.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (7s) == sanityn test 16i: read after truncate file ============ 08:32:04 (1761222724) striped dir -i0 -c2 -H crush2 /mnt/lustre/d16i.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (6s) == sanityn test 16j: race dio with buffered i/o ========== 08:32:10 (1761222730) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.854348 s, 30.7 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.16179 s, 22.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.67401 s, 15.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.880723 s, 29.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.33941 s, 19.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.10231 s, 23.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.80281 s, 14.5 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.12489 s, 12.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.29582 s, 11.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.59699 s, 10.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.64008 s, 9.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.54034 s, 17.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.62685 s, 16.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.06898 s, 12.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.14936 s, 12.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.6027 s, 16.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.62671 s, 16.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.11503 s, 12.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.16084 s, 12.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (33s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 08:32:43 (1761222763) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 fallocating to largest ever: 0x3428364 skipping zero size read skipping zero size read truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Thu Oct 23 08:33:16 EDT 2025 + fsx 2 done: rc=0 Thu Oct 23 08:33:16 EDT 2025 + fsx 3 done: rc=0 All operations completed A-OK! Thu Oct 23 08:33:17 EDT 2025 + fsx 4 done: rc=0 Thu Oct 23 08:33:17 EDT 2025 + fsx 5 done: rc=0 Thu Oct 23 08:33:17 EDT 2025 PASS 16k (43s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 08:33:26 (1761222806) fail_loc=0x8000030a PASS 17 (13s) == sanityn test 18: mmap sanity check =========================================================================================== 08:33:39 (1761222819) mmap test1: basic mmap operation (PASS, 0.16693s) mmap test2: MAP_PRIVATE not write back (PASS, 0.086368s) mmap test3: concurrent mmap ops on two nodes (PASS, 28.488s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.4519s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.15657s) mmap test6: check mmap write/read content on two nodes (PASS, 0.1964s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.0486s) mmap test11: multi-thread shared mmap (PASS, 1.0767s) mmap test10: mtime not change for readonly mmap access (PASS, 1.075s) PASS 18 (45s) == sanityn test 19: test concurrent uncached read races ========================================================================= 08:34:24 (1761222864) 32+0 records in 32+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.365369 s, 45.9 MB/s loop 5 loop 10 loop 15 loop 20 PASS 19 (31s) == sanityn test 20: test extra readahead page left in cache ============================================================== 08:34:55 (1761222895) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d20.sanityn PASS 20 (6s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 08:35:01 (1761222901) striped dir -i1 -c2 -H crush2 /mnt/lustre/d21.sanityn rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (6s) == sanityn test 23: others should see updated atime while another read============================================================== 08:35:07 (1761222907) now is 1761222909 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.8239 new atime is 1761222970 PASS 23 (67s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 08:36:14 (1761222974) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2032 1269272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 30588 3576432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 9024 3597996 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 39612 7174428 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2032 1269272 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1784 1269520 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 30588 3576432 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 9024 3597996 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 39612 7174428 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 354 999.7K 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 256.0K 404 255.6K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 256.0K 400 255.6K 1% /mnt/lustre[OST:1] filesystem_summary: 511.8K 633 511.2K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 354 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 404 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 400 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.8K 633 511.2K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 1.3G 2.0M 1.2G 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1.3G 1.7M 1.2G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.7G 29.9M 3.4G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.7G 8.8M 3.4G 1% /mnt/lustre[OST:1] filesystem_summary: 7.3G 38.7M 6.8G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 354 1023646 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 404 261740 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 400 261744 1% /mnt/lustre2[OST:1] filesystem_summary: 524117 633 523484 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2032 1269272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 30588 3576432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 9024 3597996 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 39612 7174428 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 354 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 279 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 404 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 400 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.8K 633 511.2K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 354 1023646 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 262144 404 261740 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 262144 400 261744 1% /mnt/lustre[OST:1] filesystem_summary: 524117 633 523484 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 354 1023646 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 279 1023721 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 404 261740 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 400 261744 1% /mnt/lustre2[OST:1] filesystem_summary: 524117 633 523484 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2032 1269272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 30588 3576432 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 9024 3597996 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 39612 7174428 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2032 1269272 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1784 1269520 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 30588 3576432 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 9024 3597996 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 39612 7174428 1% /mnt/lustre2 PASS 24a (5s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 08:36:19 (1761222979) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 08:36:25 (1761222985) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 08:36:33 (1761222993) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] PASS 25b (7s) == sanityn test 26a: allow mtime to get older ============ 08:36:40 (1761223000) utime: good mknod times 1761223000 <= 1761223001 <= 1761223001 for /mnt/lustre/f26a utime: good mknod times 1761223000 <= 1761223001 <= 1761223001 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (7s) == sanityn test 26b: sync mtime between ost and mds ====== 08:36:47 (1761223007) PASS 26b (7s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 08:36:54 (1761223014) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8239 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279613 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-10-23 08:36:56.000000000 -0400 Birth: 2025-10-23 08:36:57.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279613 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-10-23 08:36:56.000000000 -0400 Birth: 2025-10-23 08:36:57.000000000 -0400 PASS 26c (9s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 08:37:05 (1761223025) striped dir -i0 -c2 -H all_char /mnt/lustre/d30.sanityn 0 PASS 30 (8s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 08:37:13 (1761223033) striped dir -i1 -c2 -H all_char /mnt/lustre/d31a.sanityn fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 08:37:21 (1761223041) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg141-server mds-ost sync done. Waiting for MDT destroys to complete striped dir -i1 -c2 -H crush2 /mnt/lustre/d31b.sanityn fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.201.141@tcp:/lustre 7666232 40744 7173296 1% /mnt/lustre2 PASS 31b (17s) == sanityn test 31r: open-rename(replace) race =========== 08:37:38 (1761223058) -rw-r--r-- 1 root root 0 Oct 23 08:37 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (10s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 08:37:49 (1761223069) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 47603 Terminated tail -f $tfile PASS 31s (7s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 08:37:56 (1761223076) PASS 31t (7s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 08:38:04 (1761223084) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (2s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 08:38:07 (1761223087) Stopping /mnt/lustre-mds1 (opts:) on oleg141-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0000 create file on 2nd stripe under striped directory /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre/d33c.sanityn/striped touch /mnt/lustre2/d33c.sanityn/striped/subfile SoLC count 2 Below operations shouldn't trigger Solc: create file on 2nd stripe after setattr chmod 777 /mnt/lustre/d33c.sanityn/striped; touch /mnt/lustre2/d33c.sanityn/striped/subfile create remote dir and local dir /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote create local dir after remote dir creation transaction commit /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote.3; do_facet mds2 /home/green/git/lustre-release/lustre/utils/lctl set_param -n osd*.*MDT0001.force_sync 1; mkdir /mnt/lustre/d33c.sanityn/local.3 Solc count 0 PASS 33c (40s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 08:38:47 (1761223127) remote directory create /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre2/d33d.sanityn/subdir CoS count 2 cross-MDT rename mkdir /mnt/lustre/d33d.sanityn; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn/d1; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33d.sanityn/d2; touch /mnt/lustre/d33d.sanityn/d1/src mv /mnt/lustre2/d33d.sanityn/d1/src /mnt/lustre2/d33d.sanityn/d2/tgt CoS count 1 migrate /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs migrate -m 1 /mnt/lustre2/d33d.sanityn CoS count 2 PASS 33d (56s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 08:39:43 (1761223183) plain dir creation mkdir /mnt/lustre2/d33e.sanityn/plain open file and write echo abc > /mnt/lustre2/d33e.sanityn/f33e.sanityn append write echo dfg >> /mnt/lustre2/d33e.sanityn/f33e.sanityn setattr touch /mnt/lustre2/d33e.sanityn/f33e.sanityn file unlink rm /mnt/lustre2/d33e.sanityn/f33e.sanityn plain dir unlink rmdir /mnt/lustre2/d33e.sanityn/plain striped directory creation /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre2/d33e.sanityn/striped set default LMV to create striped subdir /home/green/git/lustre-release/lustre/utils/lfs setdirstripe -D -c 2 /mnt/lustre/d33e.sanityn striped subdir creation createmany -d /mnt/lustre/d33e.sanityn/subdir 100 total: 100 mkdir in 1.15 seconds: 87.01 ops/second sub file creation and write createmany -o /mnt/lustre/d33e.sanityn/subfile 100; echo abc > /mnt/lustre/d33e.sanityn/subfile1 total: 100 open/close in 1.52 seconds: 65.93 ops/second sub file append write echo dfg >> /mnt/lustre2/d33e.sanityn/subfile2 subdir setatttr touch /mnt/lustre2/d33e.sanityn/subdir1 subdir unlink unlinkmany -d /mnt/lustre/d33e.sanityn/subdir 100 - unlinked 0 (time 1761223189 ; total 0 ; last 0) total: 100 unlinks in 4 seconds: 25.000000 unlinks/second sub file unlink unlinkmany /mnt/lustre2/d33e.sanityn/subfile 100 - unlinked 0 (time 1761223193 ; total 0 ; last 0) total: 100 unlinks in 2 seconds: 50.000000 unlinks/second sub file creation follows striped dir chmod chmod 777 /mnt/lustre/d33e.sanityn/striped; touch /mnt/lustre/d33e.sanityn/striped/subfile striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn/striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn CoS count 0 PASS 33e (16s) == sanityn test 34: no lock timeout under IO ============= 08:39:59 (1761223199) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg141-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff992c04ef3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff992c04ef3800.ost_server_uuid in FULL state after 0 sec oleg141-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff992c04ef3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff992c04ef3800.ost_server_uuid in IDLE state after 0 sec oleg141-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff992c04ef3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff992c04ef3800.ost_server_uuid in IDLE state after 0 sec oleg141-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff992c04ef3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff992c04ef3800.ost_server_uuid in IDLE state after 0 sec oleg141-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff992c04ef3800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff992c04ef3800.ost_server_uuid in IDLE state after 0 sec oleg141-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff992c04ef3800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff992c04ef3800.ost_server_uuid in IDLE state after 0 sec PASS 34 (111s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 08:41:50 (1761223310) striped dir -i1 -c2 -H crush2 /mnt/lustre/d35.sanityn fail_loc=0x80000317 Race attempt 0 Wait for 57595 57650 for 60 sec...