-----============= acceptance-small: sanityn ============----- Thu Oct 23 08:04:34 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg219-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a skipping tests SLOW=no: 33a === sanityn: start setup 08:04:56 (1761221096) === Starting client oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre2 Mount client oleg219-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre2 Started clients oleg219-client.virtnet: 192.168.202.119@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg219-client.virtnet: executing check_config_client /mnt/lustre oleg219-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg219-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff951182f36000.idle_timeout=debug osc.lustre-OST0000-osc-ffff95118711c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff951182f36000.idle_timeout=debug osc.lustre-OST0001-osc-ffff95118711c000.idle_timeout=debug disable quota as required oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanityn: finish setup 08:05:36 (1761221136) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441259 s, 23.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8229] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 08:05:38 (1761221138) PASS 0 (9s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 08:05:47 (1761221147) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (7s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 08:05:55 (1761221155) -rw-r--r-- 1 root root 0 Oct 23 08:05 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (6s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 08:06:01 (1761221161) -rw-r--r-- 1 root root 0 Oct 23 08:06 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (6s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 08:06:07 (1761221167) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (6s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 08:06:13 (1761221173) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (6s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 08:06:20 (1761221180) total 0 -rwxrwxrwx 1 root root 0 Oct 23 08:05 f2a -rwxrwxrwx 1 root root 0 Oct 23 08:06 f2b total 0 -rwxrwxrwx 1 root root 0 Oct 23 08:05 f2a -rwxrwxrwx 1 root root 0 Oct 23 08:06 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0235058 s, 21.8 kB/s PASS 2e (7s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 08:06:26 (1761221186) /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has type file OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn has perms 0777 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by user #500 OK /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn is owned by group #500 OK Can't lstat /mnt/lustre2/d2f.sanityn/remote_dir/f2f.sanityn: No such file or directory PASS 2f (8s) == sanityn test 2g: check blocks update on sync write ==== 08:06:34 (1761221194) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.293005 s, 7.2 MB/s /mnt/lustre/f2g.sanityn has 4096 blocks /mnt/lustre2/f2g.sanityn has 4096 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 08:06:41 (1761221201) PASS 3 (6s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 08:06:48 (1761221208) PASS 4 (8s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 08:06:55 (1761221215) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (6s) == sanityn test 6: remove of open file on other node ============================================================================ 08:07:01 (1761221221) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (7s) == sanityn test 7: remove of open directory on other node ======================================================================= 08:07:08 (1761221228) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (6s) == sanityn test 8: remove of open special file on other node ==================================================================== 08:07:14 (1761221234) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (6s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 08:07:20 (1761221240) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 08:07:27 (1761221247) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672504 s, 15.6 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0610334 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (6s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 08:07:33 (1761221253) 1+0 records in 1+0 records out 1 byte copied, 0.0034673 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.033304 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0208903 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00836564 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0193039 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.013128 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0255098 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0128 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0170643 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00965599 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0128361 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0117173 s, 0.1 kB/s PASS 10a (7s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 08:07:40 (1761221260) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.014637 s, 210 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0644582 s, 63.5 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00440946 s, 697 kB/s PASS 10b (6s) == sanityn test 11: execution of file opened for write should return error ============================================================== 08:07:46 (1761221266) striped dir -i1 -c2 -H all_char /mnt/lustre/d11 multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8229 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (6s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 08:07:53 (1761221273) warning: '-runtime' deprecated, use '-t runtime' instead - link 397 (time 1761221285.14 total 10.01 last 39.64) - link 744 (time 1761221295.15 total 20.02 last 34.69) - link 1107 (time 1761221305.16 total 30.03 last 36.24) - link 1498 (time 1761221315.19 total 40.06 last 39.01) - link 1873 (time 1761221325.19 total 50.06 last 37.48) - link 2307 (time 1761221335.20 total 60.07 last 43.35) - link 2734 (time 1761221345.22 total 70.09 last 42.64) - link 3148 (time 1761221355.22 total 80.10 last 41.38) - link 3537 (time 1761221365.24 total 90.11 last 38.82) total: 3910 link in 100.00 seconds: 39.10 ops/second using seed 3805896412 running for 100 seconds - stat 10000 (time 1761221285 ; total 9 ; last 9) - stat 20000 (time 1761221295 ; total 19 ; last 10) - stat 30000 (time 1761221304 ; total 28 ; last 9) - stat 40000 (time 1761221313 ; total 37 ; last 9) - stat 50000 (time 1761221321 ; total 45 ; last 8) - stat 58337 (time 1761221332 ; total 56 ; last 11) - stat 60000 (time 1761221335 ; total 59 ; last 3) - stat 65742 (time 1761221346 ; total 70 ; last 11) - stat 70000 (time 1761221353 ; total 77 ; last 7) - stat 74586 (time 1761221364 ; total 88 ; last 11) - stat 79891 (time 1761221375 ; total 99 ; last 11) - stat 80000 (time 1761221375 ; total 99 ; last 0) total: 80511 stats in 100 seconds: 805.109985 stats/second - unlinked 0 (time 1761221326 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3910) error: No such file or directory total: 3909 unlinks in 86 seconds: 45.453487 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (22703) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (22706) - No such process PASS 12 (145s) == sanityn test 13: test directory page revocation ======= 08:10:17 (1761221417) striped dir -i1 -c2 -H crush /mnt/lustre/d13 f13 f13 PASS 13 (9s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 08:10:26 (1761221426) striped dir -i0 -c2 -H crush2 /mnt/lustre/d14aa.sanityn multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.8229 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 23914 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (7s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 08:10:33 (1761221433) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d14ab.sanityn open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 4923: 24506 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (7s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 08:10:41 (1761221441) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d14b.sanityn truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 25096 Terminated $DIR1/$tdir/sleep 60 PASS 14b (7s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 08:10:48 (1761221448) striped dir -i0 -c2 -H crush /mnt/lustre/d14c.sanityn cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 25691 Terminated $DIR1/$tdir/sleep 60 PASS 14c (6s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 08:10:54 (1761221454) striped dir -i0 -c2 -H crush /mnt/lustre/d14d.sanityn chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 26286 Terminated $DIR1/$tdir/sleep 60 PASS 14d (7s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 08:11:01 (1761221461) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky Waiting for MDT destroys to complete 0 STRIPECOUNT=2 ORIGFREE=7207412 MAXFREE=2097152000 0 lustre-OST0000 avl=780 grnt=427 diff=353 limit=46331 FULL lustre-OST0001 avl=72 grnt=4191 diff=-4119 limit=46331 FULL ERROR: dd blocks 7197357 != file size 7197376 LOG LOG2 file /tmp/oos2.log:dd: error writing '/mnt/lustre/oosfile': No space left on device /tmp/oos2.log:3598273+0 records in /tmp/oos2.log:3598272+0 records out /tmp/oos2.log:3684630528 bytes (3.7 GB, 3.4 GiB) copied, 632.824 s, 5.8 MB/s /tmp/oos2.log2:dd: error writing '/mnt/lustre2/oosfile2': No space left on device /tmp/oos2.log2:3599086+0 records in /tmp/oos2.log2:3599085+0 records out /tmp/oos2.log2:3685463040 bytes (3.7 GB, 3.4 GiB) copied, 636.335 s, 5.8 MB/s Waiting for MDT destroys to complete Waiting for MDT destroys to complete PASS 15 (692s) == sanityn test 16a: 2500 iterations of dual-mount fsx === 08:22:33 (1761222153) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.132403 s, 63.4 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.101688 s, 82.5 MB/s Chance of close/open is 1 in 50 Seed set to 2155 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn zero_range to largest ever: 0x72c658 truncating to largest ever: 0x8d4fc5 truncating to largest ever: 0x9df0b1 000200[0] 1761222161.972444 read 0x1b71e2 thru 0x1c174f (0xa56e bytes) truncating to largest ever: 0x9f652b 000500[0] 1761222172.535312 write 0x27e3ce thru 0x27eb68 (0x79b bytes) 000600[1] 1761222176.202576 read 0x40c24b thru 0x412912 (0x66c8 bytes) 000800[0] 1761222181.915682 trunc from 0x8d6640 to 0x2856ea (0x650f55 bytes) 001000[1] 1761222188.643649 write 0x3bece6 thru 0x3c71ae (0x84c9 bytes) 001200[1] 1761222195.890853 mapwrite 0x654d7b thru 0x658261 (0x34e7 bytes) 001300[0] 1761222199.679570 punch from 0x568027 to 0x56a456 (0x2430 bytes) 001400[1] 1761222204.267686 read 0x8ae043 thru 0x8b0e01 (0x2dbf bytes) 001500[1] 1761222207.869389 fallocate 0x7f3931 thru 0x7f5948 (0x2018 bytes) 001700[0] 1761222215.584413 write 0x81dbb3 thru 0x828d15 (0xb163 bytes) 001800[0] 1761222218.816898 trunc from 0x192fdc to 0x19e17e (0xb1a3 bytes) 001900[1] 1761222221.942182 write 0x7dce77 thru 0x7e778c (0xa916 bytes) fallocating to largest ever: 0xa00000 002200[1] 1761222230.724786 mapread 0x0ff46f thru 0x101b1c (0x26ae bytes) 002300[1] 1761222233.063612 punch from 0x23d408 to 0x24b8e9 (0xe4e2 bytes) 002400[0] 1761222236.015833 read 0x5285b0 thru 0x52b093 (0x2ae4 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 2239 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fallocating to largest ever: 0x5596a8 truncating to largest ever: 0x733175 truncating to largest ever: 0x81bd0f truncating to largest ever: 0x8ac335 truncating to largest ever: 0x8ad567 truncating to largest ever: 0x9ea7e3 000100[0] 1761222242.918216 punch from 0x0c7452 to 0x0ca053 (0x2c02 bytes) 000200[0] 1761222245.559095 punch from 0x2489b4 to 0x2566fd (0xdd4a bytes) 000300[1] 1761222248.643929 fallocate 0x9a684d thru 0x9a88b4 (0x2068 bytes) 000400[0] 1761222251.168358 mapwrite 0x99b000 thru 0x9a97c3 (0xe7c4 bytes) truncating to largest ever: 0x9f7645 000600[0] 1761222255.686749 mapread 0x138000 thru 0x13d963 (0x5964 bytes) 000700[0] 1761222257.580688 read_OD 0x446000 thru 0x452fff (0xd000 bytes) 000900[1] 1761222262.128510 trunc from 0x517aa7 to 0x525854 (0xddae bytes) 001000[0] 1761222264.553846 mapread 0x12b000 thru 0x1348e4 (0x98e5 bytes) 001100[0] 1761222266.795407 fallocate 0x3aba7d thru 0x3b08b0 (0x4e34 bytes) 001200[0] 1761222269.428735 write_OD 0x872000 thru 0x87bfff (0xa000 bytes) 001300[1] 1761222272.198334 mapread 0x476000 thru 0x480036 (0xa037 bytes) fallocating to largest ever: 0x9f857a 001400[0] 1761222275.253297 mapwrite 0x4b9000 thru 0x4c2288 (0x9289 bytes) 001500[0] 1761222277.620859 punch from 0x586d6b to 0x58cffa (0x6290 bytes) 001600[1] 1761222280.278396 write_OD 0x654000 thru 0x65bfff (0x8000 bytes) 001700[1] 1761222282.553179 trunc from 0x9d8022 to 0x879269 (0x15edb8 bytes) 001800[0] 1761222284.659728 punch from 0x455097 to 0x459c85 (0x4bef bytes) 002000[0] 1761222289.843107 trunc from 0x61b914 to 0x59a9f7 (0x80f1c bytes) 002100[0] 1761222292.022968 punch from 0x02fda4 to 0x03d906 (0xdb63 bytes) 002200[0] 1761222293.753844 fallocate 0x984be6 thru 0x98b64d (0x6a68 bytes) 002300[0] 1761222296.008063 punch from 0x276988 to 0x281449 (0xaac2 bytes) All operations completed A-OK! PASS 16a (151s) == sanityn test 16b: 2500 iterations of dual-mount fsx at small size ========================================================== 08:25:04 (1761222304) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.145305 s, 57.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.121138 s, 69.2 MB/s Chance of close/open is 1 in 50 Seed set to 2307 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn skipping zero size read skipping zero size read skipping zero size read skipping zero length zero range skipping zero size read zero_range to largest ever: 0x645c30 truncating to largest ever: 0x7abb80 fallocating to largest ever: 0x8210cf truncating to largest ever: 0x8889ef 000200[1] 1761222313.138207 mapread 0x3fcc4e thru 0x3fd81d (0xbd0 bytes) truncating to largest ever: 0x8fe716 truncating to largest ever: 0x997ba1 fallocating to largest ever: 0x9f4075 000300[1] 1761222316.622741 write 0x6bddde thru 0x6be61b (0x83e bytes) fallocating to largest ever: 0x9fd030 000400[0] 1761222320.236166 mapwrite 0x48c781 thru 0x48d824 (0x10a4 bytes) 000500[1] 1761222323.734444 write 0x253ecd thru 0x254ffe (0x1132 bytes) 000600[0] 1761222327.608289 trunc from 0x9131dc to 0x0ca437 (0x848da4 bytes) 000700[1] 1761222331.030028 read 0x256bd9 thru 0x2584f4 (0x191c bytes) 000800[1] 1761222334.731815 trunc from 0x7cc135 to 0x85e896 (0x92762 bytes) 000900[0] 1761222338.748821 trunc from 0x998b3a to 0x71b921 (0x27d218 bytes) 001000[0] 1761222342.100724 trunc from 0x18a62f to 0x22701c (0x9c9ee bytes) 001100[0] 1761222345.936230 punch from 0x4ebbc9 to 0x4ed8e9 (0x1d21 bytes) 001200[0] 1761222349.228570 fallocate 0x67b109 thru 0x67b675 (0x56d bytes) 001300[0] 1761222352.520831 mapwrite 0x017509 thru 0x019167 (0x1c5f bytes) 001400[0] 1761222355.663314 mapwrite 0x331bbd thru 0x333975 (0x1db9 bytes) 001500[0] 1761222359.017023 trunc from 0x1ac4d3 to 0x1ac6a1 (0x1cf bytes) 001600[1] 1761222362.644006 fallocate 0x8dd3d4 thru 0x8deb30 (0x175d bytes) 001700[1] 1761222365.558946 punch from 0x14277f to 0x1429de (0x260 bytes) 001800[0] 1761222369.109968 trunc from 0x7a00da to 0x7a0968 (0x88f bytes) 001900[0] 1761222372.572894 mapwrite 0x2974ce thru 0x298442 (0xf75 bytes) 002000[0] 1761222375.748573 mapread 0x378368 thru 0x3787b5 (0x44e bytes) truncating to largest ever: 0x9fe189 002100[1] 1761222378.726805 mapwrite 0x2d10f4 thru 0x2d2b63 (0x1a70 bytes) 002200[1] 1761222381.783208 fallocate 0x7390d5 thru 0x73990e (0x83a bytes) 002300[1] 1761222385.288095 mapread 0x0474cd thru 0x0476ce (0x202 bytes) 002400[0] 1761222388.605615 mapwrite 0x6c2fcd thru 0x6c3eef (0xf23 bytes) All operations completed A-OK! PASS 16b (90s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 08:26:35 (1761222395) keep default fallocate mode: 0 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.129361 s, 64.8 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.150079 s, 55.9 MB/s Waiting for MDT destroys to complete Chance of close/open is 1 in 50 Seed set to 2412 fd 0: /mnt/lustre/f16c.sanityn fd 1: /mnt/lustre2/f16c.sanityn zero_range to largest ever: 0x214eab truncating to largest ever: 0x26a7d4 fallocating to largest ever: 0x85fe3a fallocating to largest ever: 0x950453 fallocating to largest ever: 0x98db98 000100[0] 1761222415.773550 write 0x0ce9e7 thru 0x0d38d4 (0x4eee bytes) fallocating to largest ever: 0x99bcaa truncating to largest ever: 0x9eb809 truncating to largest ever: 0x9fb236 000300[0] 1761222422.474214 trunc from 0x152f29 to 0x15bedb (0x8fb3 bytes) 000400[0] 1761222425.616413 fallocate 0x280b1f thru 0x2812e4 (0x7c6 bytes) 000500[1] 1761222429.168641 write 0x6abc0f thru 0x6ae471 (0x2863 bytes) 000600[1] 1761222432.637678 mapread 0x8f3e0f thru 0x8fc2db (0x84cd bytes) 000700[0] 1761222436.940004 read 0x097b9c thru 0x09d83d (0x5ca2 bytes) 000800[0] 1761222440.816206 read 0x9bf21b thru 0x9cdf87 (0xed6d bytes) 000900[1] 1761222444.421559 punch from 0x0b5a33 to 0x0c5421 (0xf9ef bytes) 001000[1] 1761222448.175256 mapwrite 0x89d7b3 thru 0x8a7008 (0x9856 bytes) 001100[0] 1761222451.435588 fallocate 0x55f9b0 thru 0x56302c (0x367d bytes) 001200[0] 1761222454.539049 fallocate 0x46c3b0 thru 0x475536 (0x9187 bytes) 001400[1] 1761222462.329848 fallocate 0x068584 thru 0x070777 (0x81f4 bytes) 001500[1] 1761222466.606990 trunc from 0x3a4b91 to 0x5a5aec (0x200f5c bytes) 001600[1] 1761222470.368178 trunc from 0x45103f to 0x45f1d5 (0xe197 bytes) fallocating to largest ever: 0xa00000 001800[0] 1761222479.013318 trunc from 0x4e2e2c to 0x1a21c6 (0x340c65 bytes) 001900[0] 1761222482.585507 read 0x5894af thru 0x5956a4 (0xc1f6 bytes) 002000[1] 1761222486.351424 trunc from 0x5632f8 to 0x5642e0 (0xfe9 bytes) 002100[1] 1761222490.444888 read 0x2dadcd thru 0x2e5737 (0xa96b bytes) 002200[1] 1761222494.131309 write 0x08bffc thru 0x0973c9 (0xb3ce bytes) 002300[1] 1761222497.494080 punch from 0x12ba70 to 0x130655 (0x4be6 bytes) 002400[1] 1761222501.702746 punch from 0x5ab80e to 0x5acbc5 (0x13b8 bytes) 002500[0] 1761222505.232841 punch from 0x45b76b to 0x46ad3a (0xf5d0 bytes) All operations completed A-OK! PASS 16c (114s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 08:28:29 (1761222509) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.150629 s, 55.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.139209 s, 60.3 MB/s ldlm.namespaces.MGC192.168.202.119@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff951182f36000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff95118711c000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff951182f36000.lru_size=clear ldlm.namespaces.lustre-MDT0001-mdc-ffff95118711c000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff951182f36000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff95118711c000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff951182f36000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff95118711c000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.85696 s, 27.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.75534 s, 59.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.47654 s, 71.0 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.46698 s, 30.2 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.78522 s, 37.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.52659 s, 41.5 MB/s PASS 16d (23s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 08:28:53 (1761222533) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.241056 s, 43.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0489415 s, 21.4 MB/s PASS 16e (5s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 08:28:58 (1761222538) TIMEOUT PASS 16f (26s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 08:29:25 (1761222565) TIMEOUT PASS 16g (26s) == sanityn test 16h: mmap read after truncate file ======= 08:29:51 (1761222591) striped dir -i0 -c2 -H all_char /mnt/lustre/d16h.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (6s) == sanityn test 16i: read after truncate file ============ 08:29:57 (1761222597) striped dir -i0 -c2 -H crush2 /mnt/lustre/d16i.sanityn ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (6s) == sanityn test 16j: race dio with buffered i/o ========== 08:30:03 (1761222603) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.833424 s, 31.5 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.866199 s, 30.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.27159 s, 20.6 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.886396 s, 29.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.42786 s, 18.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.08591 s, 24.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.52419 s, 17.2 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.11284 s, 23.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.44606 s, 18.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.45452 s, 18.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.50788 s, 17.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.16582 s, 22.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.19793 s, 21.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.56857 s, 16.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.70159 s, 15.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.32913 s, 19.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.50095 s, 17.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.004 s, 13.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.00932 s, 13.0 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (26s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 08:30:29 (1761222629) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 skipping zero size read fallocating to largest ever: 0x3428364 fallocating to largest ever: 0x3428364 skipping zero size read skipping zero size read truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3ddbbf6 fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee fallocating to largest ever: 0x3fa9cee truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 truncating to largest ever: 0x3fb80f5 All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Thu Oct 23 08:30:58 EDT 2025 + fsx 2 done: rc=0 Thu Oct 23 08:30:58 EDT 2025 + fsx 3 done: rc=0 Thu Oct 23 08:30:58 EDT 2025 + fsx 4 done: rc=0 Thu Oct 23 08:30:59 EDT 2025 + fsx 5 done: rc=0 Thu Oct 23 08:30:59 EDT 2025 PASS 16k (37s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 08:31:06 (1761222666) fail_loc=0x8000030a PASS 17 (10s) == sanityn test 18: mmap sanity check =========================================================================================== 08:31:16 (1761222676) mmap test1: basic mmap operation (PASS, 0.14524s) mmap test2: MAP_PRIVATE not write back (PASS, 0.0879s) mmap test3: concurrent mmap ops on two nodes (PASS, 21.219s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.4541s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.13587s) mmap test6: check mmap write/read content on two nodes (PASS, 0.23535s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.1694s) mmap test11: multi-thread shared mmap (PASS, 1.1422s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0711s) PASS 18 (37s) == sanityn test 19: test concurrent uncached read races ========================================================================= 08:31:53 (1761222713) 32+0 records in 32+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.317134 s, 52.9 MB/s loop 5 loop 10 loop 15 loop 20 PASS 19 (27s) == sanityn test 20: test extra readahead page left in cache ============================================================== 08:32:20 (1761222740) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d20.sanityn PASS 20 (5s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 08:32:25 (1761222745) striped dir -i1 -c2 -H crush /mnt/lustre/d21.sanityn rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (5s) == sanityn test 23: others should see updated atime while another read============================================================== 08:32:30 (1761222750) now is 1761222752 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.8229 new atime is 1761222813 PASS 23 (68s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 08:33:38 (1761222818) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2032 1269272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 8240 3598780 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 30616 3576404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38856 7175184 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2032 1269272 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1784 1269520 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 8240 3598780 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 30616 3576404 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38856 7175184 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 353 999.7K 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1000.0K 280 999.7K 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 256.0K 403 255.6K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 256.0K 402 255.6K 1% /mnt/lustre[OST:1] filesystem_summary: 511.8K 633 511.2K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 353 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 280 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 403 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 402 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.8K 633 511.2K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 1.3G 2.0M 1.2G 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1.3G 1.7M 1.2G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.7G 8.0M 3.4G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.7G 29.9M 3.4G 1% /mnt/lustre[OST:1] filesystem_summary: 7.3G 37.9M 6.8G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 353 1023647 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 280 1023720 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 403 261741 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 402 261742 1% /mnt/lustre2[OST:1] filesystem_summary: 524116 633 523483 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2032 1269272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 8240 3598780 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 30616 3576404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38856 7175184 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 353 999.7K 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1000.0K 280 999.7K 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 256.0K 403 255.6K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 256.0K 402 255.6K 1% /mnt/lustre2[OST:1] filesystem_summary: 511.8K 633 511.2K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 353 1023647 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1024000 280 1023720 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 262144 403 261741 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 262144 402 261742 1% /mnt/lustre[OST:1] filesystem_summary: 524116 633 523483 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 353 1023647 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1024000 280 1023720 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 262144 403 261741 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 262144 402 261742 1% /mnt/lustre2[OST:1] filesystem_summary: 524116 633 523483 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2032 1269272 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1784 1269520 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 8240 3598780 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 30616 3576404 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 38856 7175184 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2032 1269272 1% /mnt/lustre2[MDT:0] lustre-MDT0001_UUID 1414116 1784 1269520 1% /mnt/lustre2[MDT:1] lustre-OST0000_UUID 3833116 8240 3598780 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3833116 30616 3576404 1% /mnt/lustre2[OST:1] filesystem_summary: 7666232 38856 7175184 1% /mnt/lustre2 PASS 24a (6s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 08:33:44 (1761222824) PASS 24b (6s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 08:33:50 (1761222830) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (7s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 08:33:57 (1761222837) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] Can't lstat /mnt/lustre2/d25b.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25b.sanityn/f1] PASS 25b (6s) == sanityn test 26a: allow mtime to get older ============ 08:34:03 (1761222843) utime: good mknod times 1761222843 <= 1761222844 <= 1761222844 for /mnt/lustre/f26a utime: good mknod times 1761222843 <= 1761222844 <= 1761222844 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (6s) == sanityn test 26b: sync mtime between ost and mds ====== 08:34:09 (1761222849) PASS 26b (7s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 08:34:16 (1761222856) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8229 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279612 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-10-23 08:34:18.000000000 -0400 Birth: 2025-10-23 08:34:17.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205289279612 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2025-10-23 08:34:18.000000000 -0400 Birth: 2025-10-23 08:34:17.000000000 -0400 PASS 26c (7s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 08:34:25 (1761222865) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d30.sanityn 0 PASS 30 (7s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 08:34:32 (1761222872) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d31a.sanityn fail_loc=0x314 PASS 31a (7s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 08:34:39 (1761222879) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg219-server mds-ost sync done. Waiting for MDT destroys to complete striped dir -i1 -c2 -H crush /mnt/lustre/d31b.sanityn fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.119@tcp:/lustre 7666232 39984 7174028 1% /mnt/lustre2 PASS 31b (15s) == sanityn test 31r: open-rename(replace) race =========== 08:34:54 (1761222894) -rw-r--r-- 1 root root 0 Oct 23 08:34 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (8s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 08:35:03 (1761222903) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 47729 Terminated tail -f $tfile PASS 31s (6s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 08:35:09 (1761222909) PASS 31t (6s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 08:35:16 (1761222916) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 08:35:19 (1761222919) Stopping /mnt/lustre-mds1 (opts:) on oleg219-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 create file on 2nd stripe under striped directory /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre/d33c.sanityn/striped touch /mnt/lustre2/d33c.sanityn/striped/subfile SoLC count 2 Below operations shouldn't trigger Solc: create file on 2nd stripe after setattr chmod 777 /mnt/lustre/d33c.sanityn/striped; touch /mnt/lustre2/d33c.sanityn/striped/subfile create remote dir and local dir /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote create local dir after remote dir creation transaction commit /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33c.sanityn/remote.3; do_facet mds2 /home/green/git/lustre-release/lustre/utils/lctl set_param -n osd*.*MDT0001.force_sync 1; mkdir /mnt/lustre/d33c.sanityn/local.3 Solc count 0 PASS 33c (100s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 08:36:59 (1761223019) remote directory create /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre2/d33d.sanityn/subdir CoS count 2 cross-MDT rename mkdir /mnt/lustre/d33d.sanityn; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn/d1; /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 1 /mnt/lustre/d33d.sanityn/d2; touch /mnt/lustre/d33d.sanityn/d1/src mv /mnt/lustre2/d33d.sanityn/d1/src /mnt/lustre2/d33d.sanityn/d2/tgt CoS count 1 migrate /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 /mnt/lustre/d33d.sanityn /home/green/git/lustre-release/lustre/utils/lfs migrate -m 1 /mnt/lustre2/d33d.sanityn CoS count 2 PASS 33d (40s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 08:37:39 (1761223059) plain dir creation mkdir /mnt/lustre2/d33e.sanityn/plain open file and write echo abc > /mnt/lustre2/d33e.sanityn/f33e.sanityn append write echo dfg >> /mnt/lustre2/d33e.sanityn/f33e.sanityn setattr touch /mnt/lustre2/d33e.sanityn/f33e.sanityn file unlink rm /mnt/lustre2/d33e.sanityn/f33e.sanityn plain dir unlink rmdir /mnt/lustre2/d33e.sanityn/plain striped directory creation /home/green/git/lustre-release/lustre/utils/lfs mkdir -i 0 -c 2 /mnt/lustre2/d33e.sanityn/striped set default LMV to create striped subdir /home/green/git/lustre-release/lustre/utils/lfs setdirstripe -D -c 2 /mnt/lustre/d33e.sanityn striped subdir creation createmany -d /mnt/lustre/d33e.sanityn/subdir 100 total: 100 mkdir in 1.38 seconds: 72.26 ops/second sub file creation and write createmany -o /mnt/lustre/d33e.sanityn/subfile 100; echo abc > /mnt/lustre/d33e.sanityn/subfile1 total: 100 open/close in 1.41 seconds: 70.74 ops/second sub file append write echo dfg >> /mnt/lustre2/d33e.sanityn/subfile2 subdir setatttr touch /mnt/lustre2/d33e.sanityn/subdir1 subdir unlink unlinkmany -d /mnt/lustre/d33e.sanityn/subdir 100 - unlinked 0 (time 1761223065 ; total 0 ; last 0) total: 100 unlinks in 6 seconds: 16.666666 unlinks/second sub file unlink unlinkmany /mnt/lustre2/d33e.sanityn/subfile 100 - unlinked 0 (time 1761223071 ; total 0 ; last 0) total: 100 unlinks in 2 seconds: 50.000000 unlinks/second sub file creation follows striped dir chmod chmod 777 /mnt/lustre/d33e.sanityn/striped; touch /mnt/lustre/d33e.sanityn/striped/subfile striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn/striped directory unlink rm -rf /mnt/lustre2/d33e.sanityn CoS count 0 PASS 33e (20s) == sanityn test 34: no lock timeout under IO ============= 08:37:59 (1761223079) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg219-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff951182f36000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff951182f36000.ost_server_uuid in IDLE state after 0 sec oleg219-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff951182f36000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff951182f36000.ost_server_uuid in FULL state after 0 sec oleg219-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff951182f36000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff951182f36000.ost_server_uuid in IDLE state after 0 sec oleg219-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff951182f36000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff951182f36000.ost_server_uuid in IDLE state after 0 sec oleg219-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff951182f36000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff951182f36000.ost_server_uuid in IDLE state after 0 sec oleg219-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff951182f36000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff951182f36000.ost_server_uuid in IDLE state after 0 sec PASS 34 (108s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 08:39:47 (1761223187) striped dir -i1 -c2 -H crush2 /mnt/lustre/d35.sanityn fail_loc=0x80000317 Race attempt 0 Wait for 57729 57815 for 60 sec... - open/close 883 (time 1761223202.50 total 10.00 last 88.30) - open/close 1750 (time 1761223212.51 total 20.00 last 86.66) - open/close 2605 (time 1761223222.51 total 30.01 last 85.48) - open/close 3481 (time 1761223232.52 total 40.01 last 87.53) total: 4000 open/close in 46.07 seconds: 86.82 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.119@tcp:/lustre 7.4G 40M 6.9G 1% /mnt/lustre 192.168.202.119@tcp:/lustre 7.4G 40M 6.9G 1% /mnt/lustre2 PASS 35 (71s) == sanityn test 36: handle ESTALE/open-unlink correctly == 08:40:59 (1761223259) striped dir -i0 -c2 -H crush2 /mnt/lustre/d36.sanityn Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg219-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.77343 s, 29.6 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8229 Waiting for MDT destroys to complete *** cycle(0) *** before(7174000) after_dd(7122800) after(7174000) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.844385 s, 62.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8229 Waiting for MDT destroys to complete *** cycle(1) *** before(7174000) after_dd(7122800) after(7174000) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.11497 s, 47.0 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8229 Waiting for MDT destroys to complete *** cycle(2) *** before(7174000) after_dd(7122800) after(7174000) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.37349 s, 38.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8229 Waiting for MDT destroys to complete *** cycle(3) *** before(7174000) after_dd(7122800) after(7174000) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.13627 s, 46.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.8229