-----============= acceptance-small: sanityn ============----- Thu Apr 30 14:10:04 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg209-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 14:10:11 (1777572611) === Starting client oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 Mount client oleg209-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 Started clients oleg209-client.virtnet: 192.168.202.109@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg209-client.virtnet: executing check_config_client /mnt/lustre oleg209-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg209-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88ac08b6f000.idle_timeout=debug osc.lustre-OST0000-osc-ffff88ac0ba33800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88ac08b6f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88ac0ba33800.idle_timeout=debug disable quota as required oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 14:10:22 (1777572622) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0133414 s, 78.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7580] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 14:10:23 (1777572623) PASS 0 (2s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 14:10:25 (1777572625) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (3s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 14:10:28 (1777572628) -rw-r--r-- 1 root root 0 Apr 30 14:10 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (2s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 14:10:30 (1777572630) -rw-r--r-- 1 root root 0 Apr 30 14:10 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (2s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 14:10:32 (1777572632) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (2s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 14:10:34 (1777572634) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (3s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 14:10:37 (1777572637) total 1 -rwxrwxrwx 1 root root 0 Apr 30 14:10 f2a -rwxrwxrwx 1 root root 0 Apr 30 14:10 f2b total 1 -rwxrwxrwx 1 root root 0 Apr 30 14:10 f2a -rwxrwxrwx 1 root root 0 Apr 30 14:10 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.00579737 s, 88.3 kB/s PASS 2e (2s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 14:10:39 (1777572639) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (1s) == sanityn test 2g: check blocks update on sync write ==== 14:10:41 (1777572641) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0765289 s, 27.4 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (3s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 14:10:43 (1777572643) PASS 3 (2s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 14:10:46 (1777572646) PASS 4 (4s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 14:10:49 (1777572649) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (3s) == sanityn test 6: remove of open file on other node ============================================================================ 14:10:52 (1777572652) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (2s) == sanityn test 7: remove of open directory on other node ======================================================================= 14:10:55 (1777572655) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (3s) == sanityn test 8: remove of open special file on other node ==================================================================== 14:10:57 (1777572657) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (2s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 14:10:59 (1777572659) PASS 9a (3s) == sanityn test 9b: append to striped sparse file ======== 14:11:02 (1777572662) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0247297 s, 42.4 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0170584 s, 0.2 kB/s Data read (expecting 'foo'): foo PASS 9b (3s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 14:11:05 (1777572665) 1+0 records in 1+0 records out 1 byte copied, 0.00237886 s, 0.4 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00373303 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00323753 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00360266 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00383727 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00608392 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0114563 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00910748 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00465161 s, 0.2 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00374204 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00349198 s, 0.3 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00403558 s, 0.2 kB/s PASS 10a (3s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 14:11:08 (1777572668) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00697473 s, 440 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0193885 s, 211 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000189676 s, 16.2 MB/s PASS 10b (2s) == sanityn test 11: execution of file opened for write should return error ============================================================== 14:11:10 (1777572670) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (3s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 14:11:13 (1777572673) warning: '-runtime' deprecated, use '-t runtime' instead - link 1457 (time 1777572684.03 total 10.00 last 145.63) - link 2736 (time 1777572694.04 total 20.01 last 127.78) - link 4104 (time 1777572704.04 total 30.02 last 136.78) - link 5332 (time 1777572714.05 total 40.02 last 122.76) - link 6569 (time 1777572724.05 total 50.02 last 123.65) - link 7854 (time 1777572734.06 total 60.03 last 128.43) - link 9071 (time 1777572744.06 total 70.03 last 121.67) - link 10000 (time 1777572750.27 total 76.24 last 149.56) - link 11234 (time 1777572760.28 total 86.25 last 123.32) - link 12357 (time 1777572770.28 total 96.25 last 112.25) total: 12833 link in 100.01 seconds: 128.32 ops/second using seed 2080077650 running for 100 seconds - stat 10000 (time 1777572683 ; total 8 ; last 8) - stat 20000 (time 1777572692 ; total 17 ; last 9) - stat 30000 (time 1777572700 ; total 25 ; last 8) - stat 40000 (time 1777572708 ; total 33 ; last 8) - stat 50000 (time 1777572717 ; total 42 ; last 9) - stat 60000 (time 1777572726 ; total 51 ; last 9) - stat 64864 (time 1777572737 ; total 62 ; last 11) - stat 70000 (time 1777572747 ; total 72 ; last 10) - stat 75197 (time 1777572758 ; total 83 ; last 11) - stat 80000 (time 1777572767 ; total 92 ; last 9) total: 84490 stats in 100 seconds: 844.900024 stats/second - unlinked 0 (time 1777572725 ; total 0 ; last 0) - unlinked 10000 (time 1777572820 ; total 95 ; last 95) unlink(/mnt/lustre2/lockdir/lockfile12833) error: No such file or directory total: 12832 unlinks in 110 seconds: 116.654549 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21346) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21348) - No such process PASS 12 (163s) == sanityn test 13: test directory page revocation ======= 14:13:56 (1777572836) f13 f13 PASS 13 (3s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 14:13:59 (1777572839) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 22535 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (2s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 14:14:01 (1777572841) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5002: 23114 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (2s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 14:14:03 (1777572843) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23692 Terminated $DIR1/$tdir/sleep 60 PASS 14b (2s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 14:14:05 (1777572845) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24273 Terminated $DIR1/$tdir/sleep 60 PASS 14c (2s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 14:14:07 (1777572847) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24855 Terminated $DIR1/$tdir/sleep 60 PASS 14d (2s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 14:14:09 (1777572849) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7527424 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7527424KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (30s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 14:14:39 (1777572879) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.107266 s, 78.2 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0746034 s, 112 MB/s Chance of close/open is 1 in 50 Seed set to 2880 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read 000004[0] 1777572880.883822 mapwrite 0x5757ee thru 0x57a3d2 (0x4be5 bytes) 000007[0] 1777572880.919463 read 0x1d4ff2 thru 0x1dea9d (0x9aac bytes) 000009[1] 1777572880.926687 read 0x1ea924 thru 0x1f6ad1 (0xc1ae bytes) 000013[1] 1777572880.936652 read 0x34aaf9 thru 0x350280 (0x5788 bytes) 000014[0] 1777572880.939529 read 0x457112 thru 0x45ef00 (0x7def bytes) 000016[1] 1777572880.946650 mapread 0x0946d4 thru 0x09c795 (0x80c2 bytes) 000017[0] 1777572880.953862 write 0x70cba3 thru 0x71a794 (0xdbf2 bytes) 000018[1] 1777572880.955340 write 0x00c701 thru 0x013b46 (0x7446 bytes) 000022[1] 1777572880.975282 read 0x1c2303 thru 0x1c2f6f (0xc6d bytes) truncating to largest ever: 0x8eb0db 000030[0] 1777572881.006048 trunc from 0x71a795 to 0x8eb0da (0x1d0946 bytes) 000037[1] 1777572881.029889 trunc from 0x8eb0db to 0x08d343 (0x85dd97 bytes) truncating to largest ever: 0x91a3a6 000040[1] 1777572881.048511 trunc from 0x08d344 to 0x91a3a5 (0x88d062 bytes) 000041[1] 1777572881.056236 write 0x713aff thru 0x7186d2 (0x4bd4 bytes) 000042[0] 1777572881.059161 mapwrite 0x57639e thru 0x579ac3 (0x3726 bytes) 000048[1] 1777572881.106271 mapwrite 0x7af2c4 thru 0x7b0226 (0xf63 bytes) 000050[0] 1777572881.137272 mapwrite 0x305c38 thru 0x309c8f (0x4058 bytes) 000051[1] 1777572881.165430 write 0x75b79d thru 0x761edf (0x6743 bytes) 000052[0] 1777572881.173542 trunc from 0x91a3a6 to 0x8d8a5a (0x4194b bytes) 000053[1] 1777572881.187592 mapwrite 0x0618a9 thru 0x069227 (0x797f bytes) 000054[1] 1777572881.218392 write 0x065392 thru 0x067ff1 (0x2c60 bytes) 000057[1] 1777572881.230173 trunc from 0x8d8a5b to 0x3a6064 (0x5329f6 bytes) 000059[0] 1777572881.237931 mapwrite 0x0f8942 thru 0x107b70 (0xf22f bytes) 000060[0] 1777572881.272817 trunc from 0x3a6065 to 0x78997d (0x3e3919 bytes) 000062[1] 1777572881.292644 read 0x1da81c thru 0x1e084c (0x6031 bytes) 000065[1] 1777572881.307623 read 0x19dca8 thru 0x1a49b2 (0x6d0b bytes) 000068[0] 1777572881.315260 mapwrite 0x222669 thru 0x225f5f (0x38f7 bytes) 000070[1] 1777572881.343791 read 0x171acf thru 0x180340 (0xe872 bytes) 000075[0] 1777572881.358627 mapread 0x6e91b1 thru 0x6efb2b (0x697b bytes) 000079[0] 1777572881.371600 write 0x55c1fc thru 0x56b0b8 (0xeebd bytes) 000081[1] 1777572881.375506 mapwrite 0x51d937 thru 0x51f61b (0x1ce5 bytes) 000082[0] 1777572881.403222 write 0x66f778 thru 0x67d4f0 (0xdd79 bytes) 000083[0] 1777572881.409126 trunc from 0x78997e to 0x1ccfed (0x5bc990 bytes) 000087[1] 1777572881.423308 trunc from 0x1ccfee to 0x353899 (0x1868ac bytes) 000089[1] 1777572881.441752 write 0x186195 thru 0x195613 (0xf47f bytes) 000090[1] 1777572881.446667 mapwrite 0x91c385 thru 0x91cafc (0x778 bytes) 000092[1] 1777572881.473653 write 0x547d67 thru 0x54f962 (0x7bfc bytes) 000093[0] 1777572881.480718 read 0x75c999 thru 0x7637c8 (0x6e30 bytes) 000098[1] 1777572881.495782 mapread 0x57ccf5 thru 0x57ed93 (0x209f bytes) 000101[1] 1777572881.502445 write 0x8e664e thru 0x8f1668 (0xb01b bytes) 000102[0] 1777572881.508941 trunc from 0x91cafd to 0x13ef7b (0x7ddb81 bytes) 000104[0] 1777572881.526527 trunc from 0x13ef7c to 0x4faa0b (0x3bba90 bytes) 000106[1] 1777572881.537765 read 0x25bd57 thru 0x25e392 (0x263c bytes) 000108[0] 1777572881.548548 trunc from 0x4faa0c to 0x900d2b (0x406320 bytes) 000110[0] 1777572881.557122 mapwrite 0x489b6f thru 0x49773f (0xdbd1 bytes) 000111[0] 1777572881.579403 mapread 0x592cf7 thru 0x59e3b4 (0xb6be bytes) 000113[1] 1777572881.592727 trunc from 0x900d2c to 0x8a58ed (0x5b43e bytes) 000116[1] 1777572881.607356 read 0x46807d thru 0x46f5e1 (0x7565 bytes) 000122[0] 1777572881.622433 trunc from 0x8a58ee to 0x1b94b9 (0x6ec434 bytes) 000123[1] 1777572881.634479 write 0x99454a thru 0x9a3efe (0xf9b5 bytes) 000124[1] 1777572881.643273 mapwrite 0x7f49f4 thru 0x7f812a (0x3737 bytes) 000126[1] 1777572881.671987 mapwrite 0x82c02d thru 0x8307b3 (0x4787 bytes) 000127[1] 1777572881.685287 mapwrite 0x4b4a26 thru 0x4baf04 (0x64df bytes) 000128[0] 1777572881.707138 read 0x179075 thru 0x17c9c2 (0x394e bytes) 000129[0] 1777572881.714116 read 0x0cde77 thru 0x0dc32c (0xe4b6 bytes) 000131[0] 1777572881.717893 mapread 0x4257e4 thru 0x42a3dc (0x4bf9 bytes) 000134[0] 1777572881.728774 read 0x53fbb0 thru 0x549172 (0x95c3 bytes) 000135[0] 1777572881.731390 write 0x674217 thru 0x6793c2 (0x51ac bytes) 000137[0] 1777572881.740048 mapread 0x964a67 thru 0x97466c (0xfc06 bytes) 000138[0] 1777572881.750873 trunc from 0x9a3eff to 0x34135d (0x662ba1 bytes) 000140[1] 1777572881.766112 read 0x062e07 thru 0x065445 (0x263f bytes) 000148[0] 1777572881.794740 write 0x2e30d1 thru 0x2ef505 (0xc435 bytes) 000149[0] 1777572881.802471 write 0x830e84 thru 0x835cd4 (0x4e51 bytes) 000150[1] 1777572881.804118 write 0x2ad8a0 thru 0x2bd0bb (0xf81c bytes) 000151[1] 1777572881.818974 mapread 0x4d35dd thru 0x4dc8e6 (0x930a bytes) 000152[1] 1777572881.838055 mapread 0x2b596c thru 0x2c29d4 (0xd069 bytes) 000153[1] 1777572881.844408 write 0x529e6b thru 0x53154b (0x76e1 bytes) 000155[0] 1777572881.853613 read 0x5b7b73 thru 0x5c35b4 (0xba42 bytes) 000156[0] 1777572881.873095 write 0x2b499c thru 0x2c1c9b (0xd300 bytes) 000157[1] 1777572881.884319 read 0x04d38f thru 0x05b9e6 (0xe658 bytes) 000162[1] 1777572881.892591 trunc from 0x835cd5 to 0x2878e6 (0x5ae3ee bytes) truncating to largest ever: 0x9b2e1c 000163[0] 1777572881.906606 trunc from 0x2878e7 to 0x9b2e1b (0x72b535 bytes) 000164[0] 1777572881.919429 write 0x7d156c thru 0x7d7bff (0x6694 bytes) 000167[1] 1777572881.930829 write 0x9d09ba thru 0x9d18a5 (0xeec bytes) 000168[1] 1777572881.936145 mapread 0x612391 thru 0x61fd5b (0xd9cb bytes) 000171[0] 1777572881.955054 mapwrite 0x976f6c thru 0x97a888 (0x391d bytes) 000172[0] 1777572881.977883 mapread 0x0cea69 thru 0x0d4568 (0x5b00 bytes) 000173[1] 1777572881.982811 mapwrite 0x6b90cc thru 0x6bfb0f (0x6a44 bytes) 000175[0] 1777572882.007299 write 0x953a49 thru 0x95b958 (0x7f10 bytes) 000178[1] 1777572882.019653 mapread 0x8e2634 thru 0x8ef57d (0xcf4a bytes) 000182[0] 1777572882.045309 read 0x643948 thru 0x649c97 (0x6350 bytes) 000185[0] 1777572882.050960 mapread 0x0fbd0f thru 0x103a9c (0x7d8e bytes) 000187[0] 1777572882.055981 mapread 0x6560ec thru 0x65e369 (0x827e bytes) 000189[0] 1777572882.061009 mapread 0x5d913b thru 0x5dcaad (0x3973 bytes) 000193[0] 1777572882.064277 read 0x334b50 thru 0x3409a3 (0xbe54 bytes) 000197[1] 1777572882.067652 write 0x07fedd thru 0x08d990 (0xdab4 bytes) 000200[0] 1777572882.073767 write 0x667121 thru 0x66b145 (0x4025 bytes) 000201[1] 1777572882.081660 read 0x285a3c thru 0x28ea94 (0x9059 bytes) 000204[1] 1777572882.093358 read 0x40aeee thru 0x41859e (0xd6b1 bytes) 000205[0] 1777572882.099837 mapwrite 0x404ad7 thru 0x4142a0 (0xf7ca bytes) 000208[1] 1777572882.141177 write 0x77e8e3 thru 0x782e88 (0x45a6 bytes) 000211[1] 1777572882.153896 mapwrite 0x6b395d thru 0x6b3e83 (0x527 bytes) 000212[1] 1777572882.167925 read 0x1c00f7 thru 0x1c5ea4 (0x5dae bytes) 000215[1] 1777572882.178579 mapwrite 0x914924 thru 0x916d4a (0x2427 bytes) 000217[1] 1777572882.206793 mapread 0x3d8642 thru 0x3e0857 (0x8216 bytes) 000218[0] 1777572882.213084 read 0x502b6e thru 0x50b35c (0x87ef bytes) 000219[0] 1777572882.218762 mapwrite 0x8e16ea thru 0x8e960e (0x7f25 bytes) 000222[1] 1777572882.243189 mapwrite 0x0e6e71 thru 0x0ed319 (0x64a9 bytes) 000224[1] 1777572882.270616 read 0x7653a6 thru 0x76eddb (0x9a36 bytes) 000226[0] 1777572882.273898 mapwrite 0x73f876 thru 0x7442e5 (0x4a70 bytes) truncating to largest ever: 0x9b6956 000227[0] 1777572882.300163 trunc from 0x9d18a6 to 0x9b6955 (0x1af50 bytes) 000228[0] 1777572882.307584 write 0x3ae58e thru 0x3ba2d6 (0xbd49 bytes) 000230[0] 1777572882.316046 read 0x415abf thru 0x41d9ae (0x7ef0 bytes) 000232[0] 1777572882.323604 write 0x065fa7 thru 0x06b354 (0x53ae bytes) 000235[0] 1777572882.327205 write 0x268206 thru 0x269c12 (0x1a0d bytes) 000237[1] 1777572882.340504 mapread 0x3d4908 thru 0x3e0e5c (0xc555 bytes) 000239[1] 1777572882.358633 read 0x98a652 thru 0x98e47e (0x3e2d bytes) 000240[0] 1777572882.364288 write 0x9e7167 thru 0x9e7f61 (0xdfb bytes) truncating to largest ever: 0x9eecd7 000243[0] 1777572882.371210 trunc from 0x9e7f62 to 0x9eecd6 (0x6d75 bytes) 000246[1] 1777572882.384777 write 0x151cf0 thru 0x155cb8 (0x3fc9 bytes) 000247[0] 1777572882.393932 write 0x1734b2 thru 0x174ac4 (0x1613 bytes) 000248[1] 1777572882.404236 trunc from 0x9eecd7 to 0x779106 (0x275bd0 bytes) 000249[0] 1777572882.417980 mapwrite 0x5cfb02 thru 0x5d1935 (0x1e34 bytes) 000251[0] 1777572882.449534 read 0x4f2377 thru 0x4f7157 (0x4de1 bytes) 000252[0] 1777572882.455425 read 0x1bf3f5 thru 0x1ca754 (0xb360 bytes) 000253[0] 1777572882.463695 mapwrite 0x70b93e thru 0x716674 (0xad37 bytes) 000254[0] 1777572882.484609 mapread 0x2812c2 thru 0x289577 (0x82b6 bytes) 000255[0] 1777572882.490697 trunc from 0x779107 to 0x694ec9 (0xe423d bytes) 000256[1] 1777572882.504495 trunc from 0x694eca to 0x8c65a2 (0x2316d9 bytes) 000258[0] 1777572882.525845 mapwrite 0x86ba83 thru 0x872c99 (0x7217 bytes) 000259[1] 1777572882.553938 mapwrite 0x422c43 thru 0x42fc43 (0xd001 bytes) 000263[0] 1777572882.596499 trunc from 0x8c65a3 to 0x7f8f4a (0xcd658 bytes) 000269[0] 1777572882.630474 mapwrite 0x039be9 thru 0x042fef (0x9407 bytes) 000272[0] 1777572882.653133 mapwrite 0x0b8c34 thru 0x0bf913 (0x6ce0 bytes) 000277[1] 1777572882.681635 write 0x617594 thru 0x621c42 (0xa6af bytes) 000278[0] 1777572882.690736 mapread 0x0a1790 thru 0x0a7bc4 (0x6435 bytes) 000280[0] 1777572882.703904 write 0x20b7b8 thru 0x214670 (0x8eb9 bytes) 000281[1] 1777572882.711247 trunc from 0x7f8f4b to 0x7de476 (0x1aad4 bytes) 000283[0] 1777572882.728936 mapwrite 0x825d75 thru 0x829eed (0x4179 bytes) 000285[0] 1777572882.773791 trunc from 0x829eee to 0x92b642 (0x101755 bytes) 000286[0] 1777572882.780842 mapread 0x5d2f94 thru 0x5dddaa (0xae17 bytes) 000288[0] 1777572882.808313 mapread 0x057de8 thru 0x060c46 (0x8e5f bytes) 000289[1] 1777572882.816378 mapwrite 0x2583b6 thru 0x263285 (0xaed0 bytes) 000290[0] 1777572882.857495 mapread 0x471068 thru 0x47c50a (0xb4a3 bytes) 000292[1] 1777572882.876858 write 0x93cfc4 thru 0x9445c5 (0x7602 bytes) 000293[1] 1777572882.881784 write 0x321af9 thru 0x327c99 (0x61a1 bytes) 000294[1] 1777572882.888645 read 0x4f8974 thru 0x500539 (0x7bc6 bytes) 000295[1] 1777572882.894097 mapread 0x057292 thru 0x058742 (0x14b1 bytes) 000297[0] 1777572882.897535 mapread 0x75de09 thru 0x760a47 (0x2c3f bytes) 000300[0] 1777572882.907886 write 0x4ae907 thru 0x4bb1fd (0xc8f7 bytes) 000301[0] 1777572882.916728 write 0x3d8d19 thru 0x3d9606 (0x8ee bytes) 000302[0] 1777572882.924715 trunc from 0x9445c6 to 0x222664 (0x721f61 bytes) 000303[0] 1777572882.934147 mapread 0x06d856 thru 0x07becb (0xe676 bytes) 000304[1] 1777572882.943798 mapwrite 0x6d691d thru 0x6d860e (0x1cf2 bytes) 000305[1] 1777572882.978107 mapwrite 0x4c0fb3 thru 0x4c857b (0x75c9 bytes) 000306[1] 1777572882.998598 write 0x102aa0 thru 0x111f10 (0xf471 bytes) 000308[1] 1777572883.001967 read 0x30703b thru 0x3070d1 (0x97 bytes) 000309[0] 1777572883.003542 read 0x63f79d thru 0x6410ac (0x1910 bytes) 000310[1] 1777572883.008373 mapwrite 0x9ef713 thru 0x9f19c5 (0x22b3 bytes) 000318[1] 1777572883.052894 mapwrite 0x269145 thru 0x275533 (0xc3ef bytes) 000319[1] 1777572883.073508 mapwrite 0x52ab5a thru 0x534869 (0x9d10 bytes) 000321[0] 1777572883.094241 read 0x9e4677 thru 0x9e9ffa (0x5984 bytes) 000322[1] 1777572883.101475 trunc from 0x9f19c6 to 0x60455e (0x3ed467 bytes) 000324[0] 1777572883.112589 trunc from 0x60455f to 0x0a3972 (0x560bec bytes) 000328[0] 1777572883.136054 mapread 0x02d9f2 thru 0x02e3d4 (0x9e3 bytes) 000332[1] 1777572883.151254 mapwrite 0x58fd15 thru 0x59d1f1 (0xd4dd bytes) 000334[0] 1777572883.183631 mapwrite 0x6e65f8 thru 0x6ebe56 (0x585f bytes) 000336[0] 1777572883.212707 mapwrite 0x7de703 thru 0x7e94f0 (0xadee bytes) 000338[0] 1777572883.246078 read 0x5ee4af thru 0x5f9c04 (0xb756 bytes) 000339[1] 1777572883.252499 trunc from 0x7e94f1 to 0x596b19 (0x2529d7 bytes) 000340[0] 1777572883.264008 trunc from 0x596b1a to 0x2ded86 (0x2b7d93 bytes) 000342[0] 1777572883.276181 mapwrite 0x81807c thru 0x8229cc (0xa951 bytes) 000343[0] 1777572883.299506 trunc from 0x8229cd to 0x29fc74 (0x582d58 bytes) 000345[0] 1777572883.308456 mapwrite 0x0e5ede thru 0x0ebb09 (0x5c2c bytes) 000348[1] 1777572883.328734 write 0x02eb3b thru 0x037f8b (0x9451 bytes) 000349[1] 1777572883.336913 mapwrite 0x1565a3 thru 0x165b1e (0xf57c bytes) 000353[0] 1777572883.373205 read 0x24c72a thru 0x254ea4 (0x877b bytes) 000355[0] 1777572883.389792 write 0x1a73ea thru 0x1ac7de (0x53f5 bytes) 000356[1] 1777572883.396963 mapread 0x0897b0 thru 0x092756 (0x8fa7 bytes) 000359[1] 1777572883.411199 mapwrite 0x501c3c thru 0x509597 (0x795c bytes) 000361[1] 1777572883.442549 mapread 0x2e4e85 thru 0x2f0d77 (0xbef3 bytes) 000362[1] 1777572883.453161 write 0x7b048e thru 0x7bc251 (0xbdc4 bytes) 000365[1] 1777572883.459270 read 0x031612 thru 0x03e1a1 (0xcb90 bytes) 000366[1] 1777572883.461814 mapread 0x2ef8b8 thru 0x2f66bd (0x6e06 bytes) 000368[1] 1777572883.462905 mapwrite 0x59cfc5 thru 0x5a21ea (0x5226 bytes) 000374[1] 1777572883.492289 trunc from 0x7bc252 to 0x4d5eed (0x2e6364 bytes) 000376[1] 1777572883.502888 mapwrite 0x03747b thru 0x03dd45 (0x68cb bytes) 000381[0] 1777572883.524495 trunc from 0x4d5eee to 0x003bea (0x4d2303 bytes) 000382[0] 1777572883.534643 mapwrite 0x782f4d thru 0x78beed (0x8fa1 bytes) 000384[0] 1777572883.579387 write 0x809da7 thru 0x80d479 (0x36d3 bytes) 000385[0] 1777572883.580404 trunc from 0x80d47a to 0x072c2f (0x79a84a bytes) 000387[1] 1777572883.595278 trunc from 0x072c30 to 0x2a4497 (0x231868 bytes) 000388[1] 1777572883.604825 trunc from 0x2a4498 to 0x65bf9a (0x3b7b03 bytes) 000389[0] 1777572883.609518 mapread 0x25b92a thru 0x265105 (0x97dc bytes) 000390[0] 1777572883.631173 mapwrite 0x0471ee thru 0x055143 (0xdf56 bytes) 000391[1] 1777572883.674294 read 0x0ab4f9 thru 0x0b6aba (0xb5c2 bytes) 000392[1] 1777572883.679483 mapwrite 0x8c413e thru 0x8cd4b3 (0x9376 bytes) 000395[0] 1777572883.712217 mapread 0x5723be thru 0x581ed2 (0xfb15 bytes) 000399[0] 1777572883.745457 mapwrite 0x88074b thru 0x88ea5d (0xe313 bytes) 000400[1] 1777572883.773066 trunc from 0x8cd4b4 to 0x87a430 (0x53083 bytes) 000402[1] 1777572883.788109 mapread 0x52539c thru 0x52c34a (0x6faf bytes) 000405[1] 1777572883.797489 read 0x074e0e thru 0x07c141 (0x7334 bytes) 000412[1] 1777572883.816788 write 0x1d1d04 thru 0x1d31e8 (0x14e5 bytes) 000414[0] 1777572883.823603 trunc from 0x87a431 to 0x60d3b4 (0x26d07c bytes) 000415[1] 1777572883.837334 write 0x8a3e2f thru 0x8b2878 (0xea4a bytes) 000421[0] 1777572883.862685 mapwrite 0x7b215d thru 0x7b8467 (0x630b bytes) 000422[0] 1777572883.915672 trunc from 0x8b2879 to 0x6be211 (0x1f4667 bytes) 000424[0] 1777572883.932432 write 0x666bf9 thru 0x6727af (0xbbb7 bytes) 000425[1] 1777572883.939063 mapread 0x2cd586 thru 0x2db75b (0xe1d6 bytes) 000426[1] 1777572883.957038 trunc from 0x6be212 to 0x3f151d (0x2cccf4 bytes) 000430[1] 1777572883.983589 trunc from 0x3f151e to 0x106fb6 (0x2ea567 bytes) 000432[1] 1777572883.997048 trunc from 0x106fb7 to 0x23e8f8 (0x137942 bytes) 000433[0] 1777572884.002345 trunc from 0x23e8f9 to 0x0bc623 (0x1822d5 bytes) 000436[0] 1777572884.020680 mapwrite 0x2bfdde thru 0x2c07fd (0xa20 bytes) 000437[1] 1777572884.047169 trunc from 0x2c07fe to 0x8f5aad (0x6352b0 bytes) 000438[0] 1777572884.062621 trunc from 0x8f5aae to 0x9c5381 (0xcf8d4 bytes) 000439[0] 1777572884.076445 mapread 0x715a44 thru 0x71a6bc (0x4c79 bytes) 000441[0] 1777572884.086883 mapwrite 0x94aafe thru 0x956ea4 (0xc3a7 bytes) 000442[0] 1777572884.111898 mapread 0x241d99 thru 0x2442d7 (0x253f bytes) 000443[0] 1777572884.122713 mapwrite 0x2544d2 thru 0x261e6d (0xd99c bytes) 000444[1] 1777572884.162579 trunc from 0x9c5382 to 0x651129 (0x374258 bytes) 000446[1] 1777572884.179632 mapwrite 0x32077a thru 0x32e179 (0xda00 bytes) 000451[1] 1777572884.208773 trunc from 0x65112a to 0x36b643 (0x2e5ae6 bytes) 000452[0] 1777572884.220038 read 0x335e5e thru 0x33e94a (0x8aed bytes) 000453[0] 1777572884.242282 mapread 0x29ffc5 thru 0x2a8506 (0x8542 bytes) 000455[1] 1777572884.246620 write 0x20478e thru 0x20a62b (0x5e9e bytes) 000456[0] 1777572884.257431 trunc from 0x36b644 to 0x179a22 (0x1f1c21 bytes) 000458[1] 1777572884.275386 trunc from 0x179a23 to 0x2f112b (0x177709 bytes) 000463[1] 1777572884.298476 read 0x21a3d7 thru 0x2253f8 (0xb022 bytes) 000464[1] 1777572884.316359 mapwrite 0x8bfa68 thru 0x8c9555 (0x9aee bytes) 000465[1] 1777572884.360881 write 0x872351 thru 0x87e680 (0xc330 bytes) 000466[0] 1777572884.361657 mapread 0x06fc16 thru 0x071fbe (0x23a9 bytes) 000468[1] 1777572884.387478 mapread 0x23188c thru 0x234763 (0x2ed8 bytes) 000469[0] 1777572884.393669 trunc from 0x8c9556 to 0x681ed6 (0x24767f bytes) 000470[1] 1777572884.413864 mapread 0x24e34a thru 0x2527fc (0x44b3 bytes) 000471[0] 1777572884.426916 trunc from 0x681ed7 to 0x720fb9 (0x9f0e3 bytes) 000475[1] 1777572884.436114 mapwrite 0x1d4b4a thru 0x1d6655 (0x1b0c bytes) 000482[1] 1777572884.482221 trunc from 0x720fba to 0x22e3da (0x4f2bdf bytes) 000484[1] 1777572884.495431 mapwrite 0x5d527f thru 0x5e4671 (0xf3f3 bytes) 000486[1] 1777572884.543169 trunc from 0x5e4672 to 0x07aad4 (0x569b9d bytes) 000490[1] 1777572884.562330 read 0x0575ba thru 0x065565 (0xdfac bytes) 000491[1] 1777572884.572070 write 0x406725 thru 0x412416 (0xbcf2 bytes) 000493[1] 1777572884.581977 mapwrite 0x352ade thru 0x357587 (0x4aaa bytes) 000494[0] 1777572884.614781 read 0x368f7d thru 0x3742a0 (0xb324 bytes) 000495[1] 1777572884.639456 mapread 0x009961 thru 0x0140be (0xa75e bytes) 000496[1] 1777572884.655871 write 0x9c6fbc thru 0x9d4214 (0xd259 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 2884 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000002[1] 1777572884.908897 write 0x26a000 thru 0x26d8b7 (0x38b8 bytes) 000003[0] 1777572884.913571 mapwrite 0x034000 thru 0x0400d9 (0xc0da bytes) 000004[1] 1777572884.945915 mapwrite 0x935000 thru 0x9379ec (0x29ed bytes) 000005[0] 1777572884.983571 write 0x6b2000 thru 0x6b3b07 (0x1b08 bytes) truncating to largest ever: 0x9ad95f 000006[0] 1777572884.988429 trunc from 0x9379ed to 0x9ad95e (0x75f72 bytes) 000007[0] 1777572884.993054 write 0x600000 thru 0x604ece (0x4ecf bytes) 000008[0] 1777572885.000054 read 0x2a1000 thru 0x2a8421 (0x7422 bytes) 000010[0] 1777572885.002996 read 0x93a000 thru 0x947fca (0xdfcb bytes) 000012[1] 1777572885.009382 trunc from 0x9ad95f to 0x2308ba (0x77d0a4 bytes) 000013[0] 1777572885.019226 read 0x01d000 thru 0x023257 (0x6258 bytes) 000014[0] 1777572885.024841 mapread 0x1b8000 thru 0x1b959d (0x159e bytes) 000015[1] 1777572885.028530 mapwrite 0x0ef000 thru 0x0f6b2b (0x7b2c bytes) 000017[0] 1777572885.057549 trunc from 0x2308bb to 0x97415a (0x7438a0 bytes) 000018[1] 1777572885.070931 mapwrite 0x5fa000 thru 0x606fe7 (0xcfe8 bytes) 000019[1] 1777572885.102873 write 0x884000 thru 0x8878ef (0x38f0 bytes) 000020[1] 1777572885.111120 mapwrite 0x783000 thru 0x791f1b (0xef1c bytes) 000021[1] 1777572885.137432 read 0x140000 thru 0x1428d7 (0x28d8 bytes) 000022[1] 1777572885.140357 mapread 0x8da000 thru 0x8e20a5 (0x80a6 bytes) 000024[1] 1777572885.149788 mapread 0x384000 thru 0x38f701 (0xb702 bytes) 000026[1] 1777572885.167737 trunc from 0x97415b to 0x83b4d2 (0x138c88 bytes) 000028[1] 1777572885.178419 mapwrite 0x3a4000 thru 0x3aa53d (0x653e bytes) 000030[0] 1777572885.197339 mapread 0x358000 thru 0x36345f (0xb460 bytes) 000031[0] 1777572885.209871 read 0x11b000 thru 0x11faf0 (0x4af1 bytes) 000034[1] 1777572885.214345 read 0x680000 thru 0x68caf2 (0xcaf3 bytes) 000040[1] 1777572885.218778 write 0x0aa000 thru 0x0aaaea (0xaeb bytes) 000041[0] 1777572885.231090 trunc from 0x83b4d3 to 0x41e6d8 (0x41cdfa bytes) 000042[0] 1777572885.240167 read 0x2ce000 thru 0x2dc98b (0xe98c bytes) 000047[0] 1777572885.250520 write 0x839000 thru 0x83cf27 (0x3f28 bytes) 000049[1] 1777572885.252729 write 0x2e7000 thru 0x2ef5e8 (0x85e9 bytes) 000051[0] 1777572885.260303 mapread 0x751000 thru 0x756b3b (0x5b3c bytes) 000052[0] 1777572885.273247 mapread 0x2e4000 thru 0x2eeede (0xaedf bytes) 000054[1] 1777572885.282056 trunc from 0x83cf28 to 0x116192 (0x726d95 bytes) 000055[0] 1777572885.291202 read 0x0dd000 thru 0x0ea872 (0xd873 bytes) 000056[1] 1777572885.299246 write 0x36d000 thru 0x377bd5 (0xabd6 bytes) 000058[0] 1777572885.311192 mapread 0x26b000 thru 0x277e7b (0xce7c bytes) 000061[1] 1777572885.326570 write 0x3a3000 thru 0x3a4a1b (0x1a1c bytes) 000066[1] 1777572885.337995 write 0x19f000 thru 0x1a7669 (0x866a bytes) 000069[1] 1777572885.343711 trunc from 0x3a4a1c to 0x231a38 (0x172fe3 bytes) 000070[1] 1777572885.348909 mapread 0x1e8000 thru 0x1ed027 (0x5028 bytes) 000071[0] 1777572885.357714 write 0x340000 thru 0x343a0c (0x3a0d bytes) 000076[0] 1777572885.374218 trunc from 0x343a0d to 0x05988e (0x2ea17e bytes) 000077[0] 1777572885.387778 write 0x0d5000 thru 0x0d90e1 (0x40e2 bytes) 000081[1] 1777572885.399130 write 0x6a5000 thru 0x6a5621 (0x622 bytes) 000083[1] 1777572885.416588 mapwrite 0x486000 thru 0x48d7ec (0x77ed bytes) 000086[1] 1777572885.453790 trunc from 0x6a5622 to 0x66c419 (0x39208 bytes) 000087[1] 1777572885.462572 mapread 0x17c000 thru 0x18b18f (0xf190 bytes) 000088[1] 1777572885.471729 mapwrite 0x01f000 thru 0x02c706 (0xd707 bytes) 000092[1] 1777572885.502564 trunc from 0x66c41a to 0x87f3f5 (0x212fdc bytes) 000094[1] 1777572885.516746 mapwrite 0x05b000 thru 0x0662d0 (0xb2d1 bytes) 000095[0] 1777572885.540381 write 0x409000 thru 0x418501 (0xf502 bytes) 000096[1] 1777572885.551387 read 0x44e000 thru 0x45ad0b (0xcd0c bytes) 000101[1] 1777572885.561697 mapwrite 0x1c7000 thru 0x1ce0d7 (0x70d8 bytes) 000104[0] 1777572885.593507 read 0x6c2000 thru 0x6cf20b (0xd20c bytes) 000105[0] 1777572885.599116 mapread 0x10d000 thru 0x112037 (0x5038 bytes) 000108[0] 1777572885.606201 trunc from 0x87f3f6 to 0x408648 (0x476dad bytes) 000110[1] 1777572885.621092 mapread 0x37e000 thru 0x38daa0 (0xfaa1 bytes) 000113[0] 1777572885.635107 trunc from 0x408649 to 0x38b839 (0x7ce0f bytes) 000116[1] 1777572885.649460 write 0x743000 thru 0x743161 (0x162 bytes) 000119[0] 1777572885.663573 read 0x3e5000 thru 0x3ef8b4 (0xa8b5 bytes) 000121[0] 1777572885.670958 mapread 0x0fe000 thru 0x108702 (0xa703 bytes) 000123[0] 1777572885.676960 write 0x3ac000 thru 0x3ae6b7 (0x26b8 bytes) 000124[0] 1777572885.681352 read 0x00c000 thru 0x00d030 (0x1031 bytes) 000126[1] 1777572885.690158 trunc from 0x743162 to 0x50dd10 (0x235451 bytes) 000127[0] 1777572885.699297 trunc from 0x50dd11 to 0x06e070 (0x49fca0 bytes) 000132[1] 1777572885.713072 mapread 0x05f000 thru 0x06d3d6 (0xe3d7 bytes) 000133[1] 1777572885.725915 read 0x00f000 thru 0x011dfb (0x2dfc bytes) 000134[0] 1777572885.730218 mapwrite 0x5da000 thru 0x5ddb6e (0x3b6f bytes) 000139[1] 1777572885.763806 read 0x597000 thru 0x5a1e2e (0xae2f bytes) 000143[1] 1777572885.770973 mapread 0x395000 thru 0x39ebac (0x9bad bytes) 000144[1] 1777572885.776732 mapread 0x3dd000 thru 0x3deede (0x1edf bytes) 000146[0] 1777572885.779939 trunc from 0x5ddb6f to 0x7a3f05 (0x1c6397 bytes) 000148[0] 1777572885.789723 write 0x884000 thru 0x88fb2f (0xbb30 bytes) 000150[0] 1777572885.791094 write 0x202000 thru 0x20ac34 (0x8c35 bytes) 000154[1] 1777572885.799581 mapread 0x3a1000 thru 0x3ac91b (0xb91c bytes) 000155[0] 1777572885.815652 mapwrite 0x899000 thru 0x8a1c97 (0x8c98 bytes) 000156[0] 1777572885.843839 read 0x1ad000 thru 0x1b1bdc (0x4bdd bytes) 000157[1] 1777572885.852154 mapread 0x61e000 thru 0x628dfa (0xadfb bytes) 000159[1] 1777572885.864077 mapwrite 0x001000 thru 0x00a4c8 (0x94c9 bytes) 000160[1] 1777572885.892423 read 0x7a7000 thru 0x7af404 (0x8405 bytes) 000166[1] 1777572885.905402 read 0x818000 thru 0x81ef4e (0x6f4f bytes) 000168[1] 1777572885.916349 mapread 0x27a000 thru 0x28401f (0xa020 bytes) 000169[0] 1777572885.921940 mapread 0x50f000 thru 0x51871d (0x971e bytes) 000171[0] 1777572885.932321 write 0x18c000 thru 0x192a92 (0x6a93 bytes) 000174[0] 1777572885.941045 mapwrite 0x8a2000 thru 0x8b17e0 (0xf7e1 bytes) 000179[0] 1777572885.981808 read 0x803000 thru 0x80e3fd (0xb3fe bytes) 000180[0] 1777572885.988856 mapwrite 0x9e8000 thru 0x9ec719 (0x471a bytes) 000181[0] 1777572886.020570 mapwrite 0x556000 thru 0x564c97 (0xec98 bytes) 000185[0] 1777572886.052244 mapwrite 0x0ac000 thru 0x0b5e6a (0x9e6b bytes) 000186[1] 1777572886.074129 write 0x49d000 thru 0x4a5f16 (0x8f17 bytes) 000189[1] 1777572886.087195 mapread 0x21b000 thru 0x2276fd (0xc6fe bytes) 000191[1] 1777572886.102252 write 0x9ae000 thru 0x9b30b7 (0x50b8 bytes) 000192[0] 1777572886.104343 read 0x60f000 thru 0x613c87 (0x4c88 bytes) 000193[0] 1777572886.112816 trunc from 0x9ec71a to 0x388a23 (0x663cf6 bytes) 000195[0] 1777572886.122103 read 0x222000 thru 0x22e514 (0xc515 bytes) 000197[0] 1777572886.142043 write 0x42e000 thru 0x430378 (0x2379 bytes) 000201[0] 1777572886.153579 read 0x1b2000 thru 0x1b6c20 (0x4c21 bytes) 000204[1] 1777572886.158972 mapwrite 0x62b000 thru 0x632881 (0x7882 bytes) 000206[0] 1777572886.198736 mapwrite 0x98e000 thru 0x98f5c6 (0x15c7 bytes) 000207[1] 1777572886.224181 mapread 0x910000 thru 0x915922 (0x5923 bytes) truncating to largest ever: 0x9d6fdb 000209[0] 1777572886.235433 trunc from 0x98f5c7 to 0x9d6fda (0x47a14 bytes) 000211[0] 1777572886.247839 trunc from 0x9d6fdb to 0x43c047 (0x59af93 bytes) 000212[1] 1777572886.260079 read 0x28d000 thru 0x296b66 (0x9b67 bytes) 000213[0] 1777572886.265480 read 0x05d000 thru 0x05f489 (0x248a bytes) 000214[0] 1777572886.269880 mapwrite 0x42c000 thru 0x43b1c7 (0xf1c8 bytes) 000215[0] 1777572886.296982 trunc from 0x43c048 to 0x5d781b (0x19b7d4 bytes) 000218[1] 1777572886.312354 write 0x421000 thru 0x425f1b (0x4f1c bytes) 000219[0] 1777572886.320698 trunc from 0x5d781c to 0x28192b (0x355ef0 bytes) 000220[0] 1777572886.332985 write 0x399000 thru 0x39d845 (0x4846 bytes) 000221[0] 1777572886.336961 trunc from 0x39d846 to 0x8b9d04 (0x51c4bf bytes) 000222[0] 1777572886.346270 write 0x366000 thru 0x36f015 (0x9016 bytes) 000223[0] 1777572886.351717 mapwrite 0x58b000 thru 0x592f2a (0x7f2b bytes) 000225[1] 1777572886.381937 mapread 0x324000 thru 0x328983 (0x4984 bytes) 000226[0] 1777572886.394002 mapwrite 0x094000 thru 0x0a2897 (0xe898 bytes) 000227[0] 1777572886.419501 write 0x85e000 thru 0x869a5a (0xba5b bytes) 000231[1] 1777572886.429305 mapread 0x77d000 thru 0x78cd9a (0xfd9b bytes) 000232[0] 1777572886.444240 trunc from 0x8b9d05 to 0x418e9b (0x4a0e69 bytes) 000234[0] 1777572886.458517 mapwrite 0x6be000 thru 0x6c8bc5 (0xabc6 bytes) 000236[0] 1777572886.495211 trunc from 0x6c8bc6 to 0x97e696 (0x2b5ad1 bytes) 000239[1] 1777572886.502830 write 0x206000 thru 0x21010d (0xa10e bytes) 000242[1] 1777572886.514520 mapread 0x7df000 thru 0x7e5e53 (0x6e54 bytes) 000243[0] 1777572886.525295 mapwrite 0x6f4000 thru 0x701929 (0xd92a bytes) 000245[1] 1777572886.566253 mapread 0x530000 thru 0x53b76e (0xb76f bytes) 000247[1] 1777572886.581764 mapwrite 0x5cb000 thru 0x5cca28 (0x1a29 bytes) 000248[0] 1777572886.604663 read 0x926000 thru 0x927fb0 (0x1fb1 bytes) 000250[0] 1777572886.610455 trunc from 0x97e697 to 0x8fc9d1 (0x81cc5 bytes) 000251[0] 1777572886.624831 read 0x854000 thru 0x85ee5f (0xae60 bytes) 000258[0] 1777572886.646345 write 0x561000 thru 0x570dc1 (0xfdc2 bytes) 000260[1] 1777572886.655445 mapwrite 0x367000 thru 0x36a147 (0x3148 bytes) 000261[1] 1777572886.693947 mapwrite 0x266000 thru 0x26ee78 (0x8e79 bytes) 000265[0] 1777572886.726224 write 0x8f9000 thru 0x90045a (0x745b bytes) 000266[0] 1777572886.731713 read 0x423000 thru 0x4272da (0x42db bytes) 000267[0] 1777572886.739417 mapread 0x4ca000 thru 0x4ceefe (0x4eff bytes) 000268[1] 1777572886.750250 write 0x100000 thru 0x105e3c (0x5e3d bytes) 000269[1] 1777572886.758757 mapread 0x691000 thru 0x6991e2 (0x81e3 bytes) 000274[1] 1777572886.774579 mapread 0x34d000 thru 0x353966 (0x6967 bytes) 000277[1] 1777572886.785583 trunc from 0x90045b to 0x37affb (0x58545f bytes) 000278[0] 1777572886.795555 mapwrite 0x9fc000 thru 0x9ff1d2 (0x31d3 bytes) 000279[1] 1777572886.829657 mapwrite 0x807000 thru 0x814555 (0xd556 bytes) 000281[1] 1777572886.865995 mapwrite 0x997000 thru 0x99b48f (0x4490 bytes) 000282[1] 1777572886.887330 mapwrite 0x213000 thru 0x222080 (0xf081 bytes) 000284[0] 1777572886.914866 trunc from 0x9ff1d3 to 0x18b297 (0x873f3b bytes) 000285[1] 1777572886.923240 mapwrite 0x672000 thru 0x67f204 (0xd205 bytes) 000287[0] 1777572886.964614 write 0x02b000 thru 0x03a798 (0xf799 bytes) 000289[0] 1777572886.976358 write 0x4b9000 thru 0x4c541c (0xc41d bytes) 000290[0] 1777572886.984052 mapwrite 0x334000 thru 0x336cde (0x2cdf bytes) 000292[0] 1777572887.016703 write 0x8fe000 thru 0x90b6b2 (0xd6b3 bytes) 000293[0] 1777572887.018780 write 0x06e000 thru 0x07d0fe (0xf0ff bytes) 000295[1] 1777572887.021588 write 0x457000 thru 0x461adb (0xaadc bytes) 000296[1] 1777572887.033233 mapwrite 0x86b000 thru 0x8702d4 (0x52d5 bytes) 000297[0] 1777572887.057564 trunc from 0x90b6b3 to 0x5656ef (0x3a5fc3 bytes) 000299[1] 1777572887.068048 mapwrite 0x107000 thru 0x113b1c (0xcb1d bytes) 000302[0] 1777572887.106764 mapwrite 0x974000 thru 0x976420 (0x2421 bytes) 000305[0] 1777572887.151278 mapread 0x138000 thru 0x1436f3 (0xb6f4 bytes) 000307[0] 1777572887.168348 read 0x5e0000 thru 0x5e839c (0x839d bytes) 000308[0] 1777572887.171778 mapwrite 0x449000 thru 0x458a0f (0xfa10 bytes) 000309[0] 1777572887.204740 mapwrite 0x4ac000 thru 0x4b9205 (0xd206 bytes) 000310[1] 1777572887.235783 trunc from 0x976421 to 0x4e8aff (0x48d921 bytes) 000311[1] 1777572887.253992 mapwrite 0x0fc000 thru 0x10505e (0x905f bytes) 000312[0] 1777572887.278677 write 0x542000 thru 0x54e424 (0xc425 bytes) 000313[1] 1777572887.288273 trunc from 0x54e425 to 0x0d35a5 (0x47ae7f bytes) 000315[0] 1777572887.306265 trunc from 0x0d35a6 to 0x804cc2 (0x73171d bytes) 000317[0] 1777572887.326433 read 0x573000 thru 0x582e25 (0xfe26 bytes) 000318[0] 1777572887.331168 write 0x3c5000 thru 0x3ccb75 (0x7b76 bytes) 000321[0] 1777572887.341670 write 0x01d000 thru 0x020f3d (0x3f3e bytes) 000322[1] 1777572887.348877 mapwrite 0x758000 thru 0x75bf47 (0x3f48 bytes) 000323[0] 1777572887.381783 read 0x691000 thru 0x693e87 (0x2e88 bytes) 000324[0] 1777572887.389245 read 0x138000 thru 0x1471e0 (0xf1e1 bytes) 000329[1] 1777572887.392863 mapwrite 0x762000 thru 0x76698c (0x498d bytes) 000332[1] 1777572887.418242 mapwrite 0x076000 thru 0x07ab07 (0x4b08 bytes) 000333[0] 1777572887.436133 write 0x821000 thru 0x8273d0 (0x63d1 bytes) 000335[0] 1777572887.446565 read 0x134000 thru 0x13ab61 (0x6b62 bytes) 000336[1] 1777572887.453216 read 0x3d2000 thru 0x3d5010 (0x3011 bytes) 000337[0] 1777572887.462151 write 0x740000 thru 0x74f8fa (0xf8fb bytes) 000343[1] 1777572887.477591 mapwrite 0x872000 thru 0x8806a7 (0xe6a8 bytes) 000344[0] 1777572887.514782 mapread 0x01f000 thru 0x026247 (0x7248 bytes) 000346[0] 1777572887.527176 mapwrite 0x8a5000 thru 0x8af9cb (0xa9cc bytes) 000347[0] 1777572887.556096 mapwrite 0x7e4000 thru 0x7ec128 (0x8129 bytes) 000348[1] 1777572887.574915 mapread 0x691000 thru 0x6952f1 (0x42f2 bytes) 000353[1] 1777572887.586126 mapread 0x0d4000 thru 0x0dd20a (0x920b bytes) 000354[1] 1777572887.591529 mapwrite 0x4ea000 thru 0x4f4516 (0xa517 bytes) 000355[1] 1777572887.615115 read 0x479000 thru 0x48252e (0x952f bytes) 000356[1] 1777572887.622242 trunc from 0x8af9cc to 0x5b6f0a (0x2f8ac1 bytes) 000357[0] 1777572887.628687 mapread 0x47f000 thru 0x485c13 (0x6c14 bytes) 000359[1] 1777572887.645479 mapwrite 0x056000 thru 0x0592d2 (0x32d3 bytes) 000360[0] 1777572887.674711 read 0x32e000 thru 0x331930 (0x3931 bytes) 000361[1] 1777572887.679163 trunc from 0x5b6f0b to 0x801d2e (0x24ae24 bytes) 000363[0] 1777572887.691355 mapread 0x328000 thru 0x32d5eb (0x55ec bytes) 000370[1] 1777572887.702720 write 0x7d8000 thru 0x7e331c (0xb31d bytes) 000381[1] 1777572887.730329 mapwrite 0x377000 thru 0x382457 (0xb458 bytes) 000384[1] 1777572887.763460 read 0x342000 thru 0x351db7 (0xfdb8 bytes) 000385[1] 1777572887.768292 mapread 0x167000 thru 0x16a51b (0x351c bytes) 000386[0] 1777572887.775342 mapread 0x605000 thru 0x60c64f (0x7650 bytes) 000389[1] 1777572887.794504 read 0x131000 thru 0x1331f6 (0x21f7 bytes) 000392[1] 1777572887.797414 mapwrite 0x888000 thru 0x8971d6 (0xf1d7 bytes) 000393[0] 1777572887.838492 mapwrite 0x269000 thru 0x273b55 (0xab56 bytes) 000394[0] 1777572887.873512 read 0x354000 thru 0x3563cf (0x23d0 bytes) 000397[0] 1777572887.879491 write 0x4ef000 thru 0x4f511e (0x611f bytes) 000400[0] 1777572887.882324 read 0x26b000 thru 0x275689 (0xa68a bytes) 000406[1] 1777572887.901485 mapread 0x317000 thru 0x317d44 (0xd45 bytes) 000408[0] 1777572887.910394 mapread 0x2ed000 thru 0x2f7b4f (0xab50 bytes) 000411[0] 1777572887.917456 mapwrite 0x64c000 thru 0x64d322 (0x1323 bytes) 000412[0] 1777572887.934268 mapwrite 0x439000 thru 0x4433c7 (0xa3c8 bytes) 000415[0] 1777572887.963708 read 0x732000 thru 0x733555 (0x1556 bytes) 000417[1] 1777572887.969713 mapwrite 0x5d3000 thru 0x5dc0b7 (0x90b8 bytes) 000418[0] 1777572887.997845 mapwrite 0x423000 thru 0x42db5f (0xab60 bytes) 000420[0] 1777572888.025968 mapwrite 0x6e8000 thru 0x6f34ec (0xb4ed bytes) 000421[1] 1777572888.050345 trunc from 0x8971d7 to 0x5a1300 (0x2f5ed6 bytes) 000422[0] 1777572888.062406 read 0x101000 thru 0x108601 (0x7602 bytes) 000424[1] 1777572888.070305 mapread 0x44d000 thru 0x45545c (0x845d bytes) 000425[0] 1777572888.076326 mapread 0x337000 thru 0x339a50 (0x2a51 bytes) 000427[1] 1777572888.083100 mapwrite 0x4f6000 thru 0x5021c0 (0xc1c1 bytes) 000429[0] 1777572888.109686 mapread 0x1e6000 thru 0x1edde9 (0x7dea bytes) 000431[0] 1777572888.121074 read 0x3a4000 thru 0x3acbfa (0x8bfb bytes) 000433[0] 1777572888.124328 read 0x2ad000 thru 0x2af607 (0x2608 bytes) truncating to largest ever: 0x9fe027 000435[1] 1777572888.126916 trunc from 0x5a1301 to 0x9fe026 (0x45cd26 bytes) 000436[0] 1777572888.137075 mapread 0x606000 thru 0x6155aa (0xf5ab bytes) 000437[0] 1777572888.152391 read 0x871000 thru 0x879d31 (0x8d32 bytes) 000438[1] 1777572888.155862 write 0x9cb000 thru 0x9d796a (0xc96b bytes) 000442[1] 1777572888.172337 trunc from 0x9fe027 to 0x4c03f1 (0x53dc35 bytes) 000443[0] 1777572888.178528 mapwrite 0x321000 thru 0x328ac4 (0x7ac5 bytes) 000445[0] 1777572888.204797 trunc from 0x4c03f2 to 0x834c77 (0x374886 bytes) 000446[0] 1777572888.211637 mapread 0x055000 thru 0x05620a (0x120b bytes) 000447[0] 1777572888.218915 mapread 0x1c5000 thru 0x1ce7e4 (0x97e5 bytes) 000448[0] 1777572888.223841 write 0x9b7000 thru 0x9b852f (0x1530 bytes) 000450[0] 1777572888.228445 read 0x990000 thru 0x99c295 (0xc296 bytes) 000452[0] 1777572888.231133 mapread 0x6fc000 thru 0x70546d (0x946e bytes) 000453[1] 1777572888.236649 write 0x25e000 thru 0x26246e (0x446f bytes) 000454[1] 1777572888.242990 read 0x791000 thru 0x7995ac (0x85ad bytes) 000455[1] 1777572888.249388 trunc from 0x9b8530 to 0x8b0717 (0x107e18 bytes) 000456[1] 1777572888.258310 trunc from 0x8b0718 to 0x4a2aa7 (0x40dc70 bytes) 000459[1] 1777572888.274219 mapwrite 0x0c8000 thru 0x0d015b (0x815c bytes) 000460[0] 1777572888.297946 mapread 0x1b7000 thru 0x1c51c7 (0xe1c8 bytes) 000464[1] 1777572888.311160 trunc from 0x4a2aa8 to 0x990c87 (0x4ee1e0 bytes) 000465[1] 1777572888.325745 mapread 0x5ab000 thru 0x5ab68b (0x68c bytes) 000468[1] 1777572888.332929 mapread 0x1f1000 thru 0x1f33e0 (0x23e1 bytes) 000469[1] 1777572888.341545 trunc from 0x990c88 to 0x843174 (0x14db13 bytes) 000471[1] 1777572888.346675 mapread 0x41c000 thru 0x426e2d (0xae2e bytes) 000472[1] 1777572888.351721 write 0x1d9000 thru 0x1e07aa (0x77ab bytes) 000474[1] 1777572888.358002 mapread 0x7c1000 thru 0x7c54bc (0x44bd bytes) 000476[1] 1777572888.366207 mapwrite 0x950000 thru 0x952ee4 (0x2ee5 bytes) 000477[0] 1777572888.388911 write 0x6ce000 thru 0x6d57cf (0x77d0 bytes) 000479[0] 1777572888.398703 read 0x904000 thru 0x9079d0 (0x39d1 bytes) 000482[1] 1777572888.404931 read 0x7c5000 thru 0x7d3055 (0xe056 bytes) 000483[1] 1777572888.413398 mapread 0x71c000 thru 0x7265d8 (0xa5d9 bytes) 000484[1] 1777572888.419248 mapread 0x43d000 thru 0x440088 (0x3089 bytes) 000485[1] 1777572888.424810 read 0x3a9000 thru 0x3af751 (0x6752 bytes) 000488[0] 1777572888.427560 read 0x6ec000 thru 0x6fadd4 (0xedd5 bytes) 000489[0] 1777572888.430301 mapwrite 0x656000 thru 0x663e5f (0xde60 bytes) 000492[1] 1777572888.458192 trunc from 0x952ee5 to 0x4bda30 (0x4954b4 bytes) 000493[0] 1777572888.469735 write 0x9bd000 thru 0x9c4853 (0x7854 bytes) 000494[1] 1777572888.477227 read 0x295000 thru 0x29e2b5 (0x92b6 bytes) 000498[0] 1777572888.488011 mapwrite 0x9a7000 thru 0x9ac137 (0x5138 bytes) All operations completed A-OK! PASS 16a (11s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 14:14:50 (1777572890) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0886395 s, 94.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0759895 s, 110 MB/s Chance of close/open is 1 in 50 Seed set to 2890 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read skipping zero size read truncating to largest ever: 0x5ea6c2 000008[0] 1777572891.038388 trunc from 00000000 to 0x5ea6c1 (0x5ea6c2 bytes) 000013[1] 1777572891.052063 mapwrite 0x86a633 thru 0x86b4d1 (0xe9f bytes) 000014[0] 1777572891.090725 mapwrite 0x862c6c thru 0x864696 (0x1a2b bytes) 000015[1] 1777572891.117693 mapwrite 0x10000a thru 0x1003ef (0x3e6 bytes) truncating to largest ever: 0x6f633a 000017[0] 1777572891.142738 trunc from 0x86b4d2 to 0x6f6339 (0x175198 bytes) 000020[1] 1777572891.161194 read 0x5a04fb thru 0x5a1cea (0x17f0 bytes) 000024[0] 1777572891.174082 read 0x2eeb87 thru 0x2f0345 (0x17bf bytes) 000025[1] 1777572891.175914 write 0x6dc5ef thru 0x6de0f6 (0x1b08 bytes) 000028[0] 1777572891.192597 mapread 0x6df5b0 thru 0x6df7d2 (0x223 bytes) 000029[1] 1777572891.199500 mapwrite 0x1383f2 thru 0x13a236 (0x1e45 bytes) 000030[0] 1777572891.224928 read 0x4efe93 thru 0x4f0f6b (0x10d9 bytes) 000031[0] 1777572891.230454 read 0x05ca07 thru 0x05ddd6 (0x13d0 bytes) 000034[0] 1777572891.236244 mapread 0x525ba7 thru 0x526dd6 (0x1230 bytes) 000035[0] 1777572891.239367 mapread 0x1f6f03 thru 0x1f8bfd (0x1cfb bytes) 000036[1] 1777572891.244738 write 0x2961a9 thru 0x296394 (0x1ec bytes) 000040[1] 1777572891.250080 mapread 0x6923ee thru 0x693865 (0x1478 bytes) 000042[0] 1777572891.261270 write 0x8eedd6 thru 0x8ef6fa (0x925 bytes) truncating to largest ever: 0x72737b 000043[1] 1777572891.266357 trunc from 0x8ef6fb to 0x72737a (0x1c8380 bytes) 000045[1] 1777572891.282515 mapwrite 0x3eb9f7 thru 0x3ed272 (0x187c bytes) 000046[1] 1777572891.298409 read 0x059713 thru 0x05a025 (0x913 bytes) 000047[1] 1777572891.300317 mapread 0x1a5f71 thru 0x1a7e9b (0x1f2b bytes) 000048[0] 1777572891.309907 mapwrite 0x92651e thru 0x92657a (0x5d bytes) 000050[0] 1777572891.337522 write 0x88fd62 thru 0x88fee9 (0x188 bytes) 000052[1] 1777572891.343399 mapread 0x69609f thru 0x69680d (0x76f bytes) 000053[1] 1777572891.349947 mapwrite 0x8bce69 thru 0x8be542 (0x16da bytes) 000055[0] 1777572891.378879 read 0x8bc782 thru 0x8bdea2 (0x1721 bytes) 000056[0] 1777572891.384040 mapwrite 0x5b2885 thru 0x5b4105 (0x1881 bytes) 000061[1] 1777572891.426480 write 0x260f9d thru 0x261621 (0x685 bytes) 000064[0] 1777572891.443217 read 0x5680bd thru 0x5692cd (0x1211 bytes) 000066[0] 1777572891.451867 mapread 0x3d3659 thru 0x3d46b0 (0x1058 bytes) 000068[0] 1777572891.463499 write 0x326c3b thru 0x32855a (0x1920 bytes) 000069[0] 1777572891.468805 mapread 0x15bc1e thru 0x15bf54 (0x337 bytes) 000072[1] 1777572891.475330 mapread 0x0a6601 thru 0x0a84a6 (0x1ea6 bytes) 000075[1] 1777572891.490962 trunc from 0x92657b to 0x6af027 (0x277553 bytes) 000076[0] 1777572891.501458 mapwrite 0x68a62c thru 0x68b0b7 (0xa8c bytes) 000078[1] 1777572891.528269 trunc from 0x6af028 to 0x34493b (0x36a6ec bytes) 000080[0] 1777572891.541772 trunc from 0x34493c to 0x2ae2b2 (0x96689 bytes) 000084[1] 1777572891.567143 read 0x0fbbbd thru 0x0fcb95 (0xfd9 bytes) 000087[0] 1777572891.595980 mapwrite 0x820337 thru 0x8212c6 (0xf90 bytes) 000092[1] 1777572891.630963 mapwrite 0x9337b1 thru 0x934bca (0x141a bytes) 000093[0] 1777572891.688098 write 0x8fb134 thru 0x8fc7be (0x168b bytes) 000095[1] 1777572891.708615 write 0x399bbf thru 0x39ab06 (0xf48 bytes) 000096[1] 1777572891.721849 mapread 0x4aa80b thru 0x4abbbc (0x13b2 bytes) 000098[1] 1777572891.733596 read 0x33adc4 thru 0x33c947 (0x1b84 bytes) 000099[0] 1777572891.740498 trunc from 0x934bcb to 0x2c7257 (0x66d973 bytes) 000100[1] 1777572891.757190 mapwrite 0x8a993d thru 0x8a9d83 (0x447 bytes) 000104[1] 1777572891.794447 mapwrite 0x8ac6c0 thru 0x8adb52 (0x1493 bytes) 000105[0] 1777572891.819004 mapwrite 0x784b4e thru 0x78552d (0x9e0 bytes) 000106[0] 1777572891.844903 mapread 0x3e47af thru 0x3e4e9e (0x6f0 bytes) 000110[1] 1777572891.863959 read 0x540da5 thru 0x5418f7 (0xb53 bytes) 000112[0] 1777572891.868413 write 0x48e80a thru 0x48f221 (0xa18 bytes) 000116[0] 1777572891.893852 mapread 0x4d4d7e thru 0x4d666d (0x18f0 bytes) 000117[0] 1777572891.901359 trunc from 0x8adb53 to 0x5c23e4 (0x2eb76e bytes) 000119[1] 1777572891.918603 read 0x01d24f thru 0x01e38a (0x113c bytes) 000120[1] 1777572891.924856 read 0x3df28e thru 0x3e0a10 (0x1783 bytes) 000122[1] 1777572891.949969 mapread 0x59f93e thru 0x5a13df (0x1aa2 bytes) 000127[0] 1777572891.971290 write 0x5d3995 thru 0x5d5377 (0x19e3 bytes) 000128[0] 1777572891.979990 mapread 0x555fdf thru 0x557533 (0x1555 bytes) 000130[1] 1777572891.995427 read 0x06633d thru 0x0677f5 (0x14b9 bytes) 000131[0] 1777572891.999797 mapwrite 0x58b84d thru 0x58cfed (0x17a1 bytes) 000132[0] 1777572892.017480 read 0x5bbd90 thru 0x5bd0a3 (0x1314 bytes) 000133[1] 1777572892.021287 write 0x74e7bf thru 0x74edeb (0x62d bytes) 000136[0] 1777572892.026439 write 0x115da5 thru 0x115f58 (0x1b4 bytes) 000137[0] 1777572892.038942 mapread 0x4630bf thru 0x463f63 (0xea5 bytes) 000139[0] 1777572892.046469 trunc from 0x74edec to 0x38c4c1 (0x3c292a bytes) 000140[1] 1777572892.058445 write 0x781ee8 thru 0x7837c8 (0x18e1 bytes) 000145[0] 1777572892.081968 write 0x8823d1 thru 0x88346a (0x109a bytes) 000146[1] 1777572892.086500 trunc from 0x88346b to 0x5fa96a (0x288b00 bytes) 000148[0] 1777572892.105483 mapread 0x26aeb0 thru 0x26af5e (0xaf bytes) 000150[0] 1777572892.121508 write 0x7c72e7 thru 0x7c75c4 (0x2de bytes) 000151[0] 1777572892.130525 mapread 0x11fabc thru 0x120e2e (0x1373 bytes) 000153[0] 1777572892.141545 mapread 0x66b836 thru 0x66b973 (0x13e bytes) 000156[0] 1777572892.144894 write 0x203813 thru 0x203f27 (0x715 bytes) 000160[0] 1777572892.159970 mapread 0x051209 thru 0x052cf5 (0x1aed bytes) 000164[1] 1777572892.176374 trunc from 0x7c75c5 to 0x562716 (0x264eae bytes) 000165[0] 1777572892.188906 mapread 0x25730e thru 0x258464 (0x1157 bytes) 000166[1] 1777572892.200179 read 0x2a8b53 thru 0x2a9637 (0xae5 bytes) 000167[1] 1777572892.206424 trunc from 0x562717 to 0x09baca (0x4c6c4c bytes) 000168[0] 1777572892.214126 mapwrite 0x3ab7e8 thru 0x3aba46 (0x25f bytes) 000171[1] 1777572892.245414 mapread 0x252f97 thru 0x253499 (0x503 bytes) 000174[1] 1777572892.257530 write 0x803075 thru 0x804796 (0x1722 bytes) 000175[1] 1777572892.264497 mapwrite 0x3f4679 thru 0x3f4cf2 (0x67a bytes) 000177[0] 1777572892.285809 write 0x3c73b0 thru 0x3c7857 (0x4a8 bytes) 000178[1] 1777572892.293665 mapread 0x5c16db thru 0x5c3673 (0x1f99 bytes) 000179[0] 1777572892.310843 write 0x9b143e thru 0x9b2a4e (0x1611 bytes) 000182[0] 1777572892.319599 mapread 0x33bfbe thru 0x33c799 (0x7dc bytes) 000188[0] 1777572892.330269 mapread 0x2b707d thru 0x2b852f (0x14b3 bytes) 000189[0] 1777572892.336492 trunc from 0x9b2a4f to 0x4a935c (0x5096f2 bytes) 000192[1] 1777572892.349206 write 0x179639 thru 0x17a914 (0x12dc bytes) 000193[1] 1777572892.359245 write 0x31c9d7 thru 0x31e69e (0x1cc8 bytes) 000198[0] 1777572892.380085 write 0x121561 thru 0x121d38 (0x7d8 bytes) 000199[0] 1777572892.386102 mapwrite 0x078eef thru 0x0791d0 (0x2e2 bytes) 000200[0] 1777572892.401189 write 0x964b25 thru 0x9653d3 (0x8af bytes) 000201[1] 1777572892.402722 write 0x2fb4ac thru 0x2fc29f (0xdf4 bytes) 000203[0] 1777572892.417140 trunc from 0x9653d4 to 0x44fc57 (0x51577c bytes) 000204[1] 1777572892.425992 mapread 0x29010c thru 0x2911aa (0x109f bytes) 000206[0] 1777572892.441379 write 0x705ddc thru 0x707d79 (0x1f9e bytes) 000207[0] 1777572892.446730 read 0x3f6d03 thru 0x3f7d22 (0x1020 bytes) 000208[0] 1777572892.449457 mapread 0x34fee9 thru 0x350e60 (0xf78 bytes) truncating to largest ever: 0x9eb3fb 000209[1] 1777572892.453053 trunc from 0x707d7a to 0x9eb3fa (0x2e3681 bytes) 000210[1] 1777572892.469322 read 0x7b03eb thru 0x7b0cba (0x8d0 bytes) 000213[0] 1777572892.477094 read 0x087323 thru 0x088ec0 (0x1b9e bytes) 000215[1] 1777572892.486830 write 0x366be8 thru 0x368b8d (0x1fa6 bytes) 000216[1] 1777572892.495920 trunc from 0x9eb3fb to 0x0b6d82 (0x934678 bytes) 000217[0] 1777572892.504964 write 0x46dd41 thru 0x46f9e8 (0x1ca8 bytes) 000218[1] 1777572892.512667 trunc from 0x46f9e9 to 0x48d1eb (0x1d803 bytes) 000219[1] 1777572892.520738 mapread 0x36f8ca thru 0x36febb (0x5f2 bytes) 000220[0] 1777572892.532506 trunc from 0x48d1ec to 0x8fe088 (0x470e9d bytes) 000222[1] 1777572892.550121 mapread 0x465e8f thru 0x4670e3 (0x1255 bytes) 000224[1] 1777572892.567335 mapread 0x5eb84b thru 0x5ed0c8 (0x187e bytes) 000228[0] 1777572892.589676 read 0x8abcc9 thru 0x8ac351 (0x689 bytes) 000229[0] 1777572892.592610 read 0x1f6387 thru 0x1f7446 (0x10c0 bytes) 000230[1] 1777572892.598446 trunc from 0x8fe089 to 0x2dc6c6 (0x6219c2 bytes) 000232[0] 1777572892.609567 mapread 0x1ca3b2 thru 0x1caf0d (0xb5c bytes) 000234[0] 1777572892.616766 mapread 0x1c2812 thru 0x1c3472 (0xc61 bytes) 000235[0] 1777572892.623012 write 0x6e7cbd thru 0x6e9279 (0x15bd bytes) 000236[0] 1777572892.630718 mapread 0x1155e1 thru 0x11588f (0x2af bytes) 000237[1] 1777572892.635142 write 0x4b8024 thru 0x4b8d5f (0xd3c bytes) 000240[1] 1777572892.652434 write 0x7a87d5 thru 0x7a9d75 (0x15a1 bytes) 000241[1] 1777572892.653280 trunc from 0x7a9d76 to 0x178074 (0x631d01 bytes) 000243[1] 1777572892.662092 read 0x13f593 thru 0x140463 (0xed1 bytes) 000246[0] 1777572892.678211 mapwrite 0x6224c3 thru 0x622d45 (0x883 bytes) 000247[0] 1777572892.705256 mapread 0x350a85 thru 0x350c8b (0x207 bytes) 000249[0] 1777572892.717964 read 0x2489fa thru 0x2494a3 (0xaaa bytes) 000254[0] 1777572892.735140 mapread 0x5d88fd thru 0x5d9fae (0x16b2 bytes) 000255[1] 1777572892.739096 trunc from 0x622d46 to 0x28f5e9 (0x39375c bytes) 000256[0] 1777572892.754260 read 0x054ab4 thru 0x0568e4 (0x1e31 bytes) 000259[1] 1777572892.782815 mapread 0x03eee3 thru 0x03ffe6 (0x1104 bytes) 000262[1] 1777572892.797846 read 0x160abc thru 0x161dd9 (0x131e bytes) 000265[1] 1777572892.809911 mapwrite 0x4e78eb thru 0x4e870f (0xe25 bytes) 000266[0] 1777572892.848425 write 0x5725fe thru 0x573902 (0x1305 bytes) 000268[0] 1777572892.860806 trunc from 0x573903 to 0x7d7b80 (0x26427e bytes) 000269[1] 1777572892.875415 write 0x471569 thru 0x47175a (0x1f2 bytes) 000270[0] 1777572892.888644 mapwrite 0x4e6684 thru 0x4e66dd (0x5a bytes) 000271[1] 1777572892.934112 mapwrite 0x64ac01 thru 0x64bf52 (0x1352 bytes) 000273[1] 1777572892.981263 mapwrite 0x1846aa thru 0x18520c (0xb63 bytes) 000276[0] 1777572893.008710 mapread 0x4a4775 thru 0x4a59f2 (0x127e bytes) 000277[1] 1777572893.025392 write 0x9d66b0 thru 0x9d816f (0x1ac0 bytes) 000278[1] 1777572893.027918 mapread 0x88e7bb thru 0x88feac (0x16f2 bytes) 000279[0] 1777572893.035596 trunc from 0x9d8170 to 0x2ecee9 (0x6eb286 bytes) 000280[1] 1777572893.053657 write 0x6f9374 thru 0x6fa637 (0x12c4 bytes) 000281[1] 1777572893.062279 read 0x067655 thru 0x067c8a (0x636 bytes) 000282[1] 1777572893.073651 mapwrite 0x539513 thru 0x53a289 (0xd77 bytes) 000283[0] 1777572893.089608 mapread 0x0ac1d9 thru 0x0ad3b5 (0x11dd bytes) 000284[0] 1777572893.098995 read 0x63c78b thru 0x63cec7 (0x73d bytes) 000285[1] 1777572893.103951 trunc from 0x6fa638 to 0x066045 (0x6945f2 bytes) 000286[0] 1777572893.117662 read 0x009a84 thru 0x00b671 (0x1bee bytes) 000288[1] 1777572893.130967 read 0x014d67 thru 0x01666e (0x1908 bytes) 000289[1] 1777572893.139229 read 0x03dbb0 thru 0x03f6ba (0x1b0b bytes) 000292[1] 1777572893.147959 mapwrite 0x0d2641 thru 0x0d2b97 (0x557 bytes) 000294[0] 1777572893.182218 write 0x5af341 thru 0x5af941 (0x601 bytes) 000296[0] 1777572893.194354 write 0x585aa2 thru 0x585f1e (0x47d bytes) 000297[1] 1777572893.200307 write 0x0dfe10 thru 0x0e08c9 (0xaba bytes) 000300[0] 1777572893.213593 read 0x3ea7d4 thru 0x3eb08d (0x8ba bytes) 000302[0] 1777572893.230515 read 0x181347 thru 0x183339 (0x1ff3 bytes) 000304[0] 1777572893.234993 mapread 0x0cbf9b thru 0x0cde59 (0x1ebf bytes) 000305[0] 1777572893.245380 write 0x08530e thru 0x0865e1 (0x12d4 bytes) 000306[1] 1777572893.251798 write 0x653d8d thru 0x65501a (0x128e bytes) 000310[1] 1777572893.272230 read 0x237627 thru 0x2391a8 (0x1b82 bytes) 000311[0] 1777572893.278954 trunc from 0x65501b to 0x63fa1b (0x155ff bytes) 000314[0] 1777572893.297725 read 0x4468af thru 0x4474bd (0xc0f bytes) 000315[0] 1777572893.299623 mapwrite 0x46cd3b thru 0x46ea7e (0x1d44 bytes) 000317[1] 1777572893.320452 read 0x0af1e1 thru 0x0b0485 (0x12a5 bytes) 000322[0] 1777572893.337727 trunc from 0x63fa1c to 0x3ea927 (0x2550f4 bytes) 000325[1] 1777572893.359087 read 0x31dfce thru 0x31e704 (0x737 bytes) 000326[0] 1777572893.386268 trunc from 0x3ea928 to 0x84b74b (0x460e24 bytes) 000327[1] 1777572893.402576 trunc from 0x84b74c to 0x063ad1 (0x7e7c7a bytes) 000330[1] 1777572893.415987 mapread 0x04d671 thru 0x04f0cf (0x1a5f bytes) 000331[1] 1777572893.424084 mapwrite 0x598768 thru 0x599792 (0x102b bytes) 000334[0] 1777572893.460174 mapread 0x498f6f thru 0x49ac61 (0x1cf3 bytes) 000337[1] 1777572893.474101 write 0x737c8e thru 0x737df6 (0x169 bytes) 000338[0] 1777572893.482856 mapread 0x36f605 thru 0x370955 (0x1351 bytes) 000339[0] 1777572893.492144 trunc from 0x737df7 to 0x4c3e6c (0x273f8a bytes) 000340[0] 1777572893.510489 write 0x87d360 thru 0x87eb8b (0x182c bytes) 000343[0] 1777572893.523266 write 0x0d6233 thru 0x0d6ccc (0xa9a bytes) 000346[0] 1777572893.530939 write 0x1f0aaa thru 0x1f1128 (0x67f bytes) 000348[1] 1777572893.538766 write 0x23079d thru 0x231da9 (0x160d bytes) 000354[1] 1777572893.567155 trunc from 0x87eb8c to 0x8355b2 (0x495d9 bytes) 000355[1] 1777572893.578366 trunc from 0x8355b3 to 0x3b8143 (0x47d46f bytes) 000356[0] 1777572893.583263 trunc from 0x3b8144 to 0x2c4047 (0xf40fc bytes) 000358[0] 1777572893.598197 mapwrite 0x5931f0 thru 0x594a93 (0x18a4 bytes) 000360[0] 1777572893.617575 read 0x394598 thru 0x396328 (0x1d91 bytes) 000363[1] 1777572893.624787 trunc from 0x594a94 to 0x47b666 (0x11942d bytes) 000366[0] 1777572893.639475 mapwrite 0x1935e1 thru 0x193cec (0x70c bytes) 000367[1] 1777572893.660710 trunc from 0x47b667 to 0x02dbd5 (0x44da91 bytes) 000368[0] 1777572893.672263 mapread 0x02c024 thru 0x02c47c (0x459 bytes) 000371[1] 1777572893.687515 mapread 0x00e39c thru 0x010222 (0x1e87 bytes) 000372[0] 1777572893.692184 write 0x7ef5d1 thru 0x7f0197 (0xbc7 bytes) 000374[1] 1777572893.701399 mapwrite 0x8967d1 thru 0x897fad (0x17dd bytes) 000376[1] 1777572893.748517 mapread 0x0a4b40 thru 0x0a53ae (0x86f bytes) 000378[1] 1777572893.753238 write 0x027cf9 thru 0x0286dc (0x9e4 bytes) 000380[1] 1777572893.760371 mapwrite 0x19233e thru 0x193419 (0x10dc bytes) 000381[1] 1777572893.777648 write 0x9f8bd7 thru 0x9f964a (0xa74 bytes) 000382[1] 1777572893.778635 mapread 0x0fc8cf thru 0x0fcd93 (0x4c5 bytes) 000383[0] 1777572893.780643 write 0x7bfec4 thru 0x7c0314 (0x451 bytes) 000385[1] 1777572893.794928 mapread 0x7e8a98 thru 0x7e954e (0xab7 bytes) 000386[0] 1777572893.807098 trunc from 0x9f964b to 0x413693 (0x5e5fb7 bytes) 000387[1] 1777572893.819490 read 0x1178db thru 0x117c66 (0x38c bytes) 000390[0] 1777572893.833489 mapread 0x028173 thru 0x029bd7 (0x1a65 bytes) 000394[1] 1777572893.848724 mapread 0x179d35 thru 0x17af9d (0x1269 bytes) 000396[1] 1777572893.856636 trunc from 0x413694 to 0x105f73 (0x30d720 bytes) 000399[0] 1777572893.871489 mapwrite 0x71ca12 thru 0x71db81 (0x1170 bytes) 000404[0] 1777572893.909546 mapread 0x501358 thru 0x502dfc (0x1aa5 bytes) 000405[0] 1777572893.916542 read 0x1f1c35 thru 0x1f2ca0 (0x106c bytes) 000407[1] 1777572893.922497 mapread 0x5a144b thru 0x5a28b6 (0x146c bytes) 000408[0] 1777572893.933452 mapwrite 0x61e4f5 thru 0x61e934 (0x440 bytes) 000409[1] 1777572893.955267 mapwrite 0x691308 thru 0x693077 (0x1d70 bytes) 000410[0] 1777572893.981437 mapread 0x54a488 thru 0x54b9b0 (0x1529 bytes) 000411[0] 1777572893.995329 write 0x7af7d1 thru 0x7afe6f (0x69f bytes) 000413[0] 1777572894.003322 read 0x2ca5e0 thru 0x2cbbec (0x160d bytes) 000414[0] 1777572894.005493 trunc from 0x7afe70 to 0x14376b (0x66c704 bytes) 000415[1] 1777572894.011816 mapwrite 0x9cacb5 thru 0x9cbe0a (0x1156 bytes) 000417[1] 1777572894.047139 mapwrite 0x23dd17 thru 0x23f0a8 (0x1392 bytes) 000420[0] 1777572894.065686 mapread 0x854f29 thru 0x855e25 (0xefd bytes) 000421[0] 1777572894.074125 trunc from 0x9cbe0b to 0x4a89c1 (0x523449 bytes) 000423[1] 1777572894.084953 read 0x20e6d3 thru 0x21053a (0x1e68 bytes) 000424[1] 1777572894.093792 write 0x97970a thru 0x979853 (0x14a bytes) 000425[0] 1777572894.103240 mapread 0x4e9815 thru 0x4e9b6f (0x35b bytes) 000428[0] 1777572894.119668 write 0x9de3a6 thru 0x9dee31 (0xa8c bytes) 000429[0] 1777572894.127632 mapwrite 0x142591 thru 0x142a6f (0x4df bytes) 000431[0] 1777572894.146458 trunc from 0x9dee32 to 0x7efdd6 (0x1ef05b bytes) 000432[1] 1777572894.156188 mapread 0x7cd485 thru 0x7cf034 (0x1bb0 bytes) 000433[0] 1777572894.169765 write 0x4c9791 thru 0x4ca7a4 (0x1014 bytes) 000434[1] 1777572894.179771 trunc from 0x7efdd7 to 0x49d644 (0x352792 bytes) 000436[0] 1777572894.196656 write 0x719a2a thru 0x719c36 (0x20d bytes) 000437[1] 1777572894.203710 write 0x974869 thru 0x976670 (0x1e08 bytes) 000440[1] 1777572894.215622 read 0x2ec79e thru 0x2ee560 (0x1dc3 bytes) 000441[0] 1777572894.220620 read 0x958e66 thru 0x959129 (0x2c4 bytes) 000442[0] 1777572894.225943 mapwrite 0x75428e thru 0x755d64 (0x1ad7 bytes) 000443[0] 1777572894.243667 mapwrite 0x1776ce thru 0x1793c4 (0x1cf7 bytes) 000444[1] 1777572894.263766 write 0x932c08 thru 0x934070 (0x1469 bytes) 000445[1] 1777572894.273256 mapwrite 0x8cf43a thru 0x8cfcbb (0x882 bytes) 000446[1] 1777572894.293394 mapwrite 0x819fa2 thru 0x81b658 (0x16b7 bytes) 000449[1] 1777572894.324871 read 0x189b3b thru 0x18b95b (0x1e21 bytes) 000451[0] 1777572894.334058 mapwrite 0x58ca98 thru 0x58d67d (0xbe6 bytes) 000452[1] 1777572894.354683 trunc from 0x976671 to 0x65b192 (0x31b4de bytes) 000453[1] 1777572894.366591 mapread 0x53960b thru 0x53b1d4 (0x1bca bytes) 000454[0] 1777572894.373941 mapwrite 0x898113 thru 0x898c84 (0xb72 bytes) 000455[0] 1777572894.396371 trunc from 0x898c85 to 0x5d8f30 (0x2bfd54 bytes) 000458[0] 1777572894.405736 read 0x36540c thru 0x366b3d (0x1732 bytes) 000459[0] 1777572894.411275 trunc from 0x5d8f31 to 0x7d27b4 (0x1f9884 bytes) 000462[1] 1777572894.423387 read 0x224737 thru 0x22562d (0xef7 bytes) 000467[1] 1777572894.434490 write 0x7dbd6d thru 0x7dd527 (0x17bb bytes) 000469[1] 1777572894.438461 mapwrite 0x32ce40 thru 0x32da81 (0xc42 bytes) 000473[0] 1777572894.462257 write 0x165ec6 thru 0x16670a (0x845 bytes) 000478[1] 1777572894.487184 trunc from 0x7dd528 to 0x0dc8ba (0x700c6d bytes) 000480[1] 1777572894.503756 trunc from 0x0dc8bb to 0x1e6ae0 (0x10a226 bytes) 000483[0] 1777572894.520360 mapread 0x0c16da thru 0x0c347e (0x1da5 bytes) 000485[0] 1777572894.528800 mapread 0x189193 thru 0x18920e (0x7c bytes) 000486[0] 1777572894.533657 mapread 0x04d9ba thru 0x04e811 (0xe58 bytes) 000487[0] 1777572894.538731 mapwrite 0x3ad026 thru 0x3aee5e (0x1e39 bytes) 000488[0] 1777572894.566427 read 0x0c705a thru 0x0c7101 (0xa8 bytes) 000489[1] 1777572894.587668 mapwrite 0x843b3d thru 0x845453 (0x1917 bytes) 000491[0] 1777572894.622768 mapread 0x715ef6 thru 0x716318 (0x423 bytes) 000493[1] 1777572894.631474 mapread 0x7e3a98 thru 0x7e5190 (0x16f9 bytes) 000494[1] 1777572894.634915 read 0x772348 thru 0x773ae1 (0x179a bytes) 000496[1] 1777572894.641154 mapwrite 0x4bf562 thru 0x4bf982 (0x421 bytes) 000498[0] 1777572894.657225 write 0x43f53c thru 0x4409a3 (0x1468 bytes) All operations completed A-OK! PASS 16b (6s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 14:14:56 (1777572896) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (1s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 14:14:57 (1777572897) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0942865 s, 89.0 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.0733769 s, 114 MB/s ldlm.namespaces.MGC192.168.202.109@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff88ac08b6f000.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff88ac0ba33800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff88ac0ba33800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff88ac0ba33800.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 2.5019 s, 41.9 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.810968 s, 129 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.798417 s, 131 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.42301 s, 73.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 1.47526 s, 71.1 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 0.988259 s, 106 MB/s PASS 16d (11s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 14:15:08 (1777572908) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.0908071 s, 115 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410891 s, 25.5 MB/s PASS 16e (3s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 14:15:11 (1777572911) TIMEOUT PASS 16f (22s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 14:15:33 (1777572933) TIMEOUT PASS 16g (23s) == sanityn test 16h: mmap read after truncate file ======= 14:15:56 (1777572956) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (3s) == sanityn test 16i: read after truncate file ============ 14:15:59 (1777572959) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (2s) == sanityn test 16j: race dio with buffered i/o ========== 14:16:02 (1777572962) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.684576 s, 38.3 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.457025 s, 57.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.687312 s, 38.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.505188 s, 51.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.669959 s, 39.1 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.560963 s, 46.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.750899 s, 34.9 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.782369 s, 33.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.798653 s, 32.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.951345 s, 27.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.996428 s, 26.3 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.738173 s, 35.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.827457 s, 31.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.962078 s, 27.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.02143 s, 25.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.757645 s, 34.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 0.908636 s, 28.9 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.11116 s, 23.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.15565 s, 22.7 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (12s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 14:16:14 (1777572974) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: : Operation not supported Operation not supported fsx: fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: : Operation not supported Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported truncating to largest ever: 0x1a7062c truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x232e1d3 skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Thu Apr 30 14:16:26 EDT 2026 All operations completed A-OK! All operations completed A-OK! + fsx 2 done: rc=0 Thu Apr 30 14:16:27 EDT 2026 + fsx 3 done: rc=0 Thu Apr 30 14:16:27 EDT 2026 All operations completed A-OK! + fsx 4 done: rc=0 Thu Apr 30 14:16:27 EDT 2026 + fsx 5 done: rc=0 Thu Apr 30 14:16:27 EDT 2026 PASS 16k (17s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 14:16:31 (1777572991) fail_loc=0x8000030a PASS 17 (5s) == sanityn test 18: mmap sanity check =========================================================================================== 14:16:36 (1777572996) mmap test1: basic mmap operation (PASS, 0.040653s) mmap test2: MAP_PRIVATE not write back (PASS, 0.035469s) mmap test3: concurrent mmap ops on two nodes (PASS, 4.2031s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.1512s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.055717s) mmap test6: check mmap write/read content on two nodes (PASS, 0.057088s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 0.47515s) mmap test11: multi-thread shared mmap (PASS, 0.46913s) mmap test10: mtime not change for readonly mmap access (PASS, 1.0413s) PASS 18 (14s) == sanityn test 19: test concurrent uncached read races ========================================================================= 14:16:50 (1777573010) oleg209-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (2s) == sanityn test 20: test extra readahead page left in cache ============================================================== 14:16:52 (1777573012) PASS 20 (3s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 14:16:55 (1777573015) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (3s) == sanityn test 23: others should see updated atime while another read============================================================== 14:16:58 (1777573018) now is 1777573019 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 new atime is 1777573080 PASS 23 (64s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 14:18:02 (1777573082) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.3K 400 114.9K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 401 114.1K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 400 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.0M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 7.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 31.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118032 400 117632 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117265 401 116864 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.9K 386 524.5K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.3K 400 114.9K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 401 114.1K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118032 400 117632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117265 401 116864 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537447 386 537061 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118032 400 117632 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117265 401 116864 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 7168 3762176 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 31744 3737600 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (2s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 14:18:04 (1777573084) PASS 24b (3s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 14:18:07 (1777573087) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (2s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 14:18:09 (1777573089) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (1s) == sanityn test 26a: allow mtime to get older ============ 14:18:10 (1777573090) utime: good mknod times 1777573090 <= 1777573091 <= 1777573091 for /mnt/lustre/f26a utime: good mknod times 1777573090 <= 1777573091 <= 1777573091 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (3s) == sanityn test 26b: sync mtime between ost and mds ====== 14:18:13 (1777573093) PASS 26b (4s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 14:18:18 (1777573098) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-04-30 14:18:18.000000000 -0400 Birth: 2026-04-30 14:18:19.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-04-30 14:18:18.000000000 -0400 Birth: 2026-04-30 14:18:19.000000000 -0400 PASS 26c (4s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 14:18:22 (1777573102) 0 PASS 30 (4s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 14:18:26 (1777573106) fail_loc=0x314 PASS 31a (3s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 14:18:29 (1777573109) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg209-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.202.109@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (13s) == sanityn test 31r: open-rename(replace) race =========== 14:18:42 (1777573122) -rw-r--r-- 1 root root 0 Apr 30 14:18 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 mv: cannot stat '/mnt/lustre2/f31r.sanityn.source': Input/output error PASS 31r (6s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 14:18:48 (1777573128) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 42978 Terminated tail -f $tfile PASS 31s (4s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 14:18:52 (1777573132) PASS 31t (6s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 14:18:59 (1777573139) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (2s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 14:19:01 (1777573141) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (2s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 14:19:03 (1777573143) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (2s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 14:19:05 (1777573145) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (2s) == sanityn test 34: no lock timeout under IO ============= 14:19:07 (1777573147) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg209-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88ac08b6f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88ac08b6f000.ost_server_uuid in IDLE state after 0 sec oleg209-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88ac08b6f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88ac08b6f000.ost_server_uuid in FULL state after 0 sec oleg209-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88ac08b6f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88ac08b6f000.ost_server_uuid in IDLE state after 0 sec oleg209-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88ac08b6f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88ac08b6f000.ost_server_uuid in IDLE state after 0 sec oleg209-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88ac08b6f000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88ac08b6f000.ost_server_uuid in IDLE state after 0 sec oleg209-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88ac08b6f000.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff88ac08b6f000.ost_server_uuid in IDLE state after 0 sec PASS 34 (102s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 14:20:49 (1777573249) fail_loc=0x80000317 Race attempt 0 Wait for 49028 49134 for 60 sec... - open/close 1469 (time 1777573262.41 total 10.00 last 146.90) - open/close 2782 (time 1777573272.41 total 20.00 last 131.29) total: 4000 open/close in 29.36 seconds: 136.23 ops/second fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.202.109@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.202.109@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 PASS 35 (66s) == sanityn test 36: handle ESTALE/open-unlink correctly == 14:21:55 (1777573315) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg209-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.495943 s, 106 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.486532 s, 108 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.464246 s, 113 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(2) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.591231 s, 88.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7497728) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.455154 s, 115 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7493632) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.478509 s, 110 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7497728) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.50241 s, 104 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.585506 s, 89.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7495680) after_dd(7446528) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.466948 s, 112 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(8) *** before(7497728) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.502524 s, 104 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7495680) after_dd(7444480) after(7495680) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.571895 s, 91.7 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7495680) after_dd(7443456) after(7495680) PASS 36 (234s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 14:25:49 (1777573549) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 - create 4887 (time 1777573561.23 total 10.00 last 488.64) - create 9520 (time 1777573571.23 total 20.00 last 463.18) total: 10000 create in 21.18 seconds: 472.08 ops/second PASS 37 (26s) == sanityn test 39a: file mtime does not change after rename ========================================================== 14:26:15 (1777573575) repeat after cancel_lru_locks PASS 39a (3s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 14:26:18 (1777573578) repeat after cancel_lru_locks PASS 39b (3s) == sanityn test 39c: check truncate mtime update ================================================================================ 14:26:21 (1777573581) repeat after cancel_lru_locks PASS 39c (4s) == sanityn test 39d: sync write should update mtime ====== 14:26:25 (1777573585) fail_loc=0x411 fail_loc=0 PASS 39d (2s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 14:26:28 (1777573588) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (9s) == sanityn test 40c: pdirops: link and others ======================================================================== 14:26:37 (1777573597) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (9s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 14:26:46 (1777573606) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (9s) == sanityn test 40e: pdirops: rename and others ======================================================================== 14:26:55 (1777573615) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (8s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 14:27:03 (1777573623) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (6s) == sanityn test 41b: pdirops: create vs create ======================================================================== 14:27:09 (1777573629) open: File exists Conflict PASS 41b (5s) == sanityn test 41c: pdirops: create vs link ======================================================================== 14:27:14 (1777573634) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (6s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 14:27:20 (1777573640) Conflict PASS 41d (5s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 14:27:25 (1777573645) Conflict PASS 41e (5s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 14:27:30 (1777573650) Conflict PASS 41f (5s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 14:27:35 (1777573655) Conflict PASS 41g (6s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 14:27:41 (1777573661) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (5s) == sanityn test 41i: reint_open: create vs create ======== 14:27:46 (1777573666) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (751s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 14:40:17 (1777574417) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (5s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 14:40:22 (1777574422) open: File exists Conflict PASS 42b (5s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 14:40:27 (1777574427) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (4s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 14:40:31 (1777574431) Conflict PASS 42d (5s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 14:40:36 (1777574436) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (5s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 14:40:41 (1777574441) Conflict PASS 42f (5s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 14:40:46 (1777574446) Conflict PASS 42g (5s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 14:40:51 (1777574451) Conflict PASS 42h (5s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 14:40:56 (1777574456) PASS 43a (16s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 14:41:12 (1777574472) Conflict PASS 43b (6s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 14:41:18 (1777574478) Conflict PASS 43c (5s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 14:41:23 (1777574483) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (5s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 14:41:28 (1777574488) Conflict PASS 43e (5s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 14:41:33 (1777574493) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (5s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 14:41:38 (1777574498) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (5s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 14:41:43 (1777574503) Conflict PASS 43h (5s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 14:41:48 (1777574508) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (1s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 14:41:49 (1777574509) mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (37s) == sanityn test 43k: unlink vs create ==================== 14:42:26 (1777574546) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (918s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 14:57:44 (1777575464) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Thu Apr 30 14:57:53 EDT 2026 PASS 44a (15s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 14:58:00 (1777575480) open: File exists Conflict PASS 44b (15s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 14:58:15 (1777575495) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (15s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 14:58:30 (1777575510) Conflict PASS 44d (13s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 14:58:43 (1777575523) Conflict PASS 44e (14s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 14:58:57 (1777575537) Conflict PASS 44f (18s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 14:59:15 (1777575555) Conflict PASS 44g (15s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 14:59:30 (1777575570) Conflict PASS 44h (15s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 14:59:45 (1777575585) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (4s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 14:59:50 (1777575590) PASS 45a (142s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 15:02:11 (1777575731) Conflict PASS 45b (14s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 15:02:25 (1777575745) Conflict PASS 45c (15s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 15:02:40 (1777575760) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (13s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 15:02:54 (1777575774) Conflict PASS 45e (14s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 15:03:07 (1777575787) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (14s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 15:03:22 (1777575802) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (16s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 15:03:37 (1777575817) Conflict PASS 45h (14s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 15:03:51 (1777575831) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (5s) == sanityn test 45j: read vs rename ====================== 15:03:56 (1777575836) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1223s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 15:24:19 (1777577059) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (15s) == sanityn test 46b: pdirops: link vs create ======================================================================== 15:24:34 (1777577074) open: File exists Conflict PASS 46b (15s) == sanityn test 46c: pdirops: link vs link ======================================================================== 15:24:50 (1777577090) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (15s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 15:25:06 (1777577106) Conflict PASS 46d (18s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 15:25:23 (1777577123) Conflict PASS 46e (14s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 15:25:37 (1777577137) Conflict PASS 46f (12s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 15:25:49 (1777577149) Conflict PASS 46g (13s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 15:26:03 (1777577163) Conflict PASS 46h (14s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 15:26:17 (1777577177) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (3s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 15:26:20 (1777577180) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (4s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 15:26:25 (1777577185) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (4s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 15:26:29 (1777577189) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 15:26:32 (1777577192) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (2s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 15:26:35 (1777577195) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 15:26:38 (1777577198) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (3s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 15:26:41 (1777577201) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 15:26:45 (1777577205) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.0939802 s, 109 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (13s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 15:26:57 (1777577217) 0+1 records in 0+1 records out 158 bytes copied, 0.0471942 s, 3.3 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (217467) - No such process PASS 51a (9s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 15:27:06 (1777577226) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0248851 s, 41.1 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117899 s, 8.9 MB/s 3145728 PASS 51b (21s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 15:27:27 (1777577247) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0747952 s, 13.7 kB/s PASS 51c (14s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 15:27:42 (1777577262) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640441 s, 16.4 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (9s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 15:27:50 (1777577270) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2451 0x993 0x280000400 read lease(1) has applied. PASS 51e (10s) == sanityn test 54: rename locking ======================= 15:28:00 (1777577280) llite.lustre-ffff88ac08b6f000.xattr_cache=1 llite.lustre-ffff88ac0ba33800.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-30 15:28:02.000000000 -0400 Modify: 2026-04-30 15:28:02.000000000 -0400 Change: 2026-04-30 15:28:02.000000000 -0400 Birth: 2026-04-30 15:28:03.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-30 15:28:08.000000000 -0400 Modify: 2026-04-30 15:28:09.000000000 -0400 Change: 2026-04-30 15:28:09.000000000 -0400 Birth: 2026-04-30 15:28:09.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-30 15:28:15.000000000 -0400 Modify: 2026-04-30 15:28:15.000000000 -0400 Change: 2026-04-30 15:28:15.000000000 -0400 Birth: 2026-04-30 15:28:15.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-30 15:28:14.000000000 -0400 Modify: 2026-04-30 15:28:16.000000000 -0400 Change: 2026-04-30 15:28:16.000000000 -0400 Birth: 2026-04-30 15:28:15.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-30 15:28:21.000000000 -0400 Modify: 2026-04-30 15:28:22.000000000 -0400 Change: 2026-04-30 15:28:22.000000000 -0400 Birth: 2026-04-30 15:28:22.000000000 -0400 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-30 15:28:21.000000000 -0400 Modify: 2026-04-30 15:28:22.000000000 -0400 Change: 2026-04-30 15:28:22.000000000 -0400 Birth: 2026-04-30 15:28:22.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory PASS 54 (32s) == sanityn test 55a: rename vs unlink target dir ========= 15:28:32 (1777577312) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (15s) == sanityn test 55b: rename vs unlink source dir ========= 15:28:47 (1777577327) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (15s) == sanityn test 55c: rename vs unlink orphan target dir == 15:29:02 (1777577342) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (21s) == sanityn test 55d: rename file vs link ================= 15:29:23 (1777577363) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 15:29:39 (1777577379) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (4s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 15:29:43 (1777577383) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (19s) == sanityn test 55g: rename: race with trylock =========== 15:30:03 (1777577403) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre/d55g.sanityn/a1/a2' to a subdirectory of itself, '/mnt/lustre/d55g.sanityn/b1/b2/b3/b4/a2' We survived race with trylock test fail_loc=0x0 PASS 55g (23s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 15:30:27 (1777577427) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1777577429 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.544992 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (122s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 15:32:28 (1777577548) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1777577555 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 0.953705 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (146s) == sanityn test 60: Verify data_version behaviour ======== 15:34:55 (1777577695) PASS 60 (11s) == sanityn test 70a: cd directory && rm directory ======== 15:35:06 (1777577706) PASS 70a (10s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 15:35:17 (1777577717) PASS 70b (11s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 15:35:27 (1777577727) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0195934 s, 3.3 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (4s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 15:35:31 (1777577731) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0157826 s, 4.2 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (5s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 15:35:36 (1777577736) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (4s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 15:35:40 (1777577740) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0224586 s, 2.9 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 15:35:44 (1777577744) llite.lustre-ffff88ac08b6f000.xattr_cache=1 llite.lustre-ffff88ac0ba33800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (7s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 15:35:51 (1777577751) llite.lustre-ffff88ac08b6f000.xattr_cache=1 llite.lustre-ffff88ac0ba33800.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (7s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 15:35:58 (1777577758) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0213637 s, 47.9 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0421895 s, 24.3 kB/s 230293: get lock2 230294: get lock1 230294: done 230294: get lock2 230293: done 230293: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 230293: cannot get lock1: Resource deadlock avoided 230294: done 230294: exit rc=0 230293: done 230293: exit rc=0 PASS 74 (11s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 15:36:09 (1777577769) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0863331 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0625835 s, 16.8 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 230874 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (14s) == sanityn test 76: Verify MDT open_files listing ======== 15:36:23 (1777577783) opening files: ........ closing files: ........ PASS 76 (89s) == sanityn test 77a: check FIFO NRS policy =============== 15:37:52 (1777577872) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.738778 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0506075 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0297264 s, 35.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0453868 s, 23.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800054 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200649 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202401 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250576 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.315161 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349307 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0415259 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0441842 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182674 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971714 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101391 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109724 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106948 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420497 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556738 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176889 s, 59.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121576 s, 86.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102189 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463529 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840416 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.464897 s, 2.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342958 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.476008 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0300583 s, 34.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.477133 s, 2.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214406 s, 48.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.533923 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10566 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.563334 s, 1.9 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (11s) == sanityn test 77b: check CRR-N NRS policy ============== 15:38:03 (1777577883) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.367215 s, 45.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0458917 s, 22.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650273 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670515 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159201 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119024 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168711 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110592 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18358 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212163 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197475 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125189 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197497 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175883 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276015 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24236 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.337038 s, 3.1 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0599917 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00737221 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12199 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0595139 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00674434 s, 155 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0507704 s, 20.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0815478 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503421 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0467988 s, 22.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504076 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510818 s, 20.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.046318 s, 22.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0418718 s, 25.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400033 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356156 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0346153 s, 30.3 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.65783 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0377354 s, 27.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0705744 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179231 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170267 s, 6.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231675 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200708 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200821 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282079 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289902 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.283851 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324274 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.37443 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335205 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378162 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338241 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.431053 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191213 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0933538 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102764 s, 10.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204414 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260902 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0807605 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652871 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195259 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112355 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182165 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13025 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145446 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155349 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157829 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128582 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108913 s, 9.6 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (13s) == sanityn test 77c: check ORR NRS policy ================ 15:38:16 (1777577896) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.493788 s, 34.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0722278 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0517273 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115476 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.05739 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205753 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287598 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.305017 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300969 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304001 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.352989 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384319 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.402009 s, 2.6 MB/s 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=3 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371064 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357623 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316407 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.464203 s, 2.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011792 s, 88.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0904248 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317369 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953789 s, 11.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457575 s, 22.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682225 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102048 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0623633 s, 16.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474394 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0540925 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746179 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0178401 s, 58.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107664 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333669 s, 31.4 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00373595 s, 281 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00256557 s, 409 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.632904 s, 26.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920151 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796844 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213152 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116237 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240895 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098262 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108256 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179637 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121546 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405287 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420091 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.511478 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.472076 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.611757 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.52805 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249077 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.555566 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0967301 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00944215 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178141 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163566 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105222 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968725 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00670951 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268917 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716807 s, 14.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247622 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384047 s, 27.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0310941 s, 33.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0327113 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211822 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314424 s, 33.3 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.574496 s, 29.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587107 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.071602 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0355001 s, 29.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109655 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159733 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147313 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225105 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130498 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262981 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257733 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236669 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261368 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238916 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327066 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340558 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.428266 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291206 s, 36.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0295123 s, 35.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765787 s, 13.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00654309 s, 160 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211728 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00213924 s, 490 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00203615 s, 515 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00621965 s, 169 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00217172 s, 483 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853513 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769245 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598191 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0363468 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158374 s, 66.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00738281 s, 142 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0457919 s, 22.9 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (17s) == sanityn test 77d: check TRR nrs policy ================ 15:38:33 (1777577913) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.535031 s, 31.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0505051 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0763034 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0958936 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165192 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187904 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258158 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.367038 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.336981 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321272 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267478 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363336 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302338 s, 3.5 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368859 s, 2.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3822 s, 2.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306194 s, 3.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.436263 s, 2.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11181 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802799 s, 13.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136895 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153963 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920444 s, 11.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136892 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940884 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700192 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804479 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796047 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179814 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158339 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171216 s, 6.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120924 s, 8.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0968303 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102232 s, 10.3 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.314643 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752154 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0471993 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137658 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172666 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236098 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279553 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307685 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306809 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272692 s, 3.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307321 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224767 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.333292 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324003 s, 3.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38788 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.349253 s, 3.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.499684 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0222619 s, 47.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00692866 s, 151 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0674814 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351045 s, 29.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123983 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0525041 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0812497 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0095464 s, 110 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0099351 s, 106 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236112 s, 44.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00782826 s, 134 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0654647 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.066143 s, 15.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112098 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683166 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0692722 s, 15.1 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.524032 s, 32.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745923 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532496 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115524 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232606 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262376 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293577 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.304183 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26599 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324013 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309662 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389084 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300056 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413764 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348399 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.451842 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364387 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0678614 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214036 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287579 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110029 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149295 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0132718 s, 79.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090053 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335362 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948401 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793708 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263663 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656738 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995765 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0588741 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546188 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562093 s, 18.7 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (18s) == sanityn test 77e: check TBF NID nrs policy ============ 15:38:51 (1777577931) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.202.9@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.628272 s, 26.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171525 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170651 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156997 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924204 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687102 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757193 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224775 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.34459 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373368 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147557 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29052 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357171 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334021 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330614 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.291619 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364604 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493008 s, 21.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780534 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859748 s, 12.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559064 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598245 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0882557 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0693873 s, 15.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0602702 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0638045 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962091 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103117 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962546 s, 10.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697669 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0846313 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769163 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834683 s, 12.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.504069 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521408 s, 20.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0493635 s, 21.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0503465 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374908 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0357288 s, 29.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661811 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12219 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121716 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181273 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189384 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225549 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209544 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231779 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225139 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180133 s, 5.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231722 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709867 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00626246 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547645 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0342968 s, 30.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0375122 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443043 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0109045 s, 96.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519405 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0569314 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0973683 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01835 s, 57.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00824178 s, 127 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0696166 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0673562 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721407 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0463009 s, 22.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.639595 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104236 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140629 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163842 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088156 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170996 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117567 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0866051 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424442 s, 24.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0759372 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21266 s, 4.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162375 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298233 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153973 s, 6.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365058 s, 2.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264167 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275325 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0022153 s, 473 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00178217 s, 588 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00175056 s, 599 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0018106 s, 579 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109474 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0949748 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162315 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928536 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589078 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16047 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474002 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451897 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165626 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00944459 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038455 s, 27.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.031383 s, 33.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.289594 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644446 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655182 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136461 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224842 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116878 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139436 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251919 s, 4.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275445 s, 3.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141595 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270734 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248319 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261777 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164715 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.278979 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206075 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412628 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136934 s, 76.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646182 s, 16.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559995 s, 18.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485655 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0317511 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981772 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0320185 s, 32.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104166 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00474107 s, 221 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103457 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10306 s, 10.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00633201 s, 166 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100061 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0551616 s, 19.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00490585 s, 214 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121383 s, 8.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (27s) == sanityn test 77f: check TBF JobID nrs policy ========== 15:39:18 (1777577958) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.463446 s, 36.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0666263 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.07474 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960492 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954792 s, 11.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141093 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236149 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210807 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290578 s, 3.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343846 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288466 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307426 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.363329 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168143 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.453381 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125119 s, 83.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.476449 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.483746 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0134099 s, 78.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0160621 s, 65.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25531 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0149633 s, 70.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0159836 s, 65.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179336 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0259644 s, 40.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0258113 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0519111 s, 20.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.526201 s, 2.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0411747 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00246547 s, 425 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0661675 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914339 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073254 s, 14.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.659902 s, 25.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173594 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227994 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32342 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32131 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326662 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.335704 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.434151 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.422537 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41376 s, 2.5 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.591011 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.571763 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.538156 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.432935 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.47216 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.611556 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.630918 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0751834 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280587 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113071 s, 92.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00487783 s, 215 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0121381 s, 86.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00918349 s, 114 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00253539 s, 414 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00201817 s, 520 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20161 s, 5.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106791 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655927 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00491399 s, 213 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132648 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0634073 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105681 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249556 s, 42.0 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.411568 s, 40.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116973 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0943505 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207029 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298076 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273928 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300703 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261851 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327952 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351989 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.403982 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.451943 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.3807 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.486705 s, 2.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.463086 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.493036 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.625944 s, 1.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154693 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202502 s, 5.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127909 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192064 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164931 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00626391 s, 167 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127073 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116214 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106899 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086668 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0840836 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723928 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629926 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527606 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0772101 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0709143 s, 14.8 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.769016 s, 21.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799222 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101599 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160642 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218251 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218052 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268863 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390032 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311248 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.385972 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401959 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429082 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411386 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374855 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.510671 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.553084 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.643588 s, 1.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00256977 s, 408 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267395 s, 39.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869686 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0197637 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00879702 s, 119 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159222 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221013 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0732276 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0767314 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0822063 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145637 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853731 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104574 s, 10.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891969 s, 11.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100483 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947675 s, 11.1 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (28s) == sanityn test 77g: Change TBF type directly ============ 15:39:46 (1777577986) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (15s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 15:40:01 (1777578001) oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg209-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg209-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg209-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg209-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 PASS 77h (13s) == sanityn test 77i: Change rank of TBF rule ============= 15:40:15 (1777578015) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (25s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 15:40:40 (1777578040) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.18312 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160963 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108709 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258557 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.235349 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157418 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319601 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161472 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332118 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284352 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342011 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254965 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387675 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350984 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440014 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.473765 s, 2.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.511531 s, 2.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184989 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166396 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224919 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14379 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172092 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173912 s, 6.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155252 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163581 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16496 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198333 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207364 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=2 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257915 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27081 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233941 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254175 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263324 s, 4.0 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1039 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7092 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (74s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 15:41:54 (1777578114) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg209-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg209-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg209-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg209-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.33293 s, 12.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129902 s, 8.1 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138682 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154472 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858609 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164952 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193602 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249333 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22498 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.316567 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.347927 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328346 s, 3.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.319106 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342973 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371865 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395324 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368104 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100383 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0810438 s, 12.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0995525 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998429 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112323 s, 9.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524703 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998297 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104443 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0998256 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112267 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887881 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108988 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123649 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902535 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119226 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109859 s, 9.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1652 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9017 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg209-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg209-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg209-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg209-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.3498 s, 12.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110356 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153624 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185443 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142767 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209545 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263587 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267422 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146222 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323732 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378562 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.383047 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135592 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228872 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447214 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247339 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227766 s, 4.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281996 s, 37.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0559303 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677565 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0589434 s, 17.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063753 s, 16.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0669323 s, 15.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560596 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711295 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925673 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643696 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937174 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0971479 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876304 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102858 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101041 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999548 s, 10.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1088 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0339 s, 5.2 MB/s Read runtime is 22 s, speed is 4.545454 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (160s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 15:44:35 (1777578275) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg209-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg209-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg209-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg209-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.31544 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126367 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108758 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140727 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145903 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22147 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246324 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246344 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340572 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31756 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328928 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.350324 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.332806 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357217 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327677 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.390247 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401664 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083476 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923257 s, 11.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110976 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114539 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0682678 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0865242 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0800352 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132587 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106313 s, 9.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149064 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133362 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119648 s, 8.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177629 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163632 s, 6.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127076 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142007 s, 7.4 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.5344 s, 5.1 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8166 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg209-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg209-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg209-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg209-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.28015 s, 13.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728631 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149739 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120557 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220105 s, 4.8 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190509 s, 5.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221014 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192648 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211687 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240074 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237557 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269925 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284734 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271253 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.267299 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248922 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286902 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929417 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105578 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105368 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11038 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119126 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805004 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12341 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127156 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100561 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118488 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13536 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105875 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12621 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142632 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111144 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138415 s, 7.6 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0044 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8241 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (160s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 15:47:15 (1777578435) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.08059 s, 15.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0950134 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158311 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180091 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171662 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253197 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413206 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334305 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361162 s, 2.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412992 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361999 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.415063 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.474368 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.536908 s, 2.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.566244 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.58135 s, 1.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991369 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0652456 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108831 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113148 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762814 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983699 s, 10.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806889 s, 13.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111683 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135598 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126464 s, 8.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0961067 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111556 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0930517 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116826 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119368 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133849 s, 7.8 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.3739 s, 8.5 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1368 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.9@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.9@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.92334 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0374307 s, 28.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426562 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814548 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0560426 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234324 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30059 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259221 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32836 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27172 s, 3.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293478 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297508 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.38885 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40438 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340129 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.339638 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.381905 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149806 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140657 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150426 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156427 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175921 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163371 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187145 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193612 s, 5.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201134 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227994 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208561 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214289 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223358 s, 4.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302802 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250488 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285199 s, 3.7 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.7902 s, 7.6 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0522 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.9@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.785233 s, 21.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0873939 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123709 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849585 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234191 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324303 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231148 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144595 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240823 s, 4.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.368099 s, 2.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.405551 s, 2.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171366 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317554 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366213 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243959 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249217 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389607 s, 2.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414052 s, 25.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0473641 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0397869 s, 26.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0321859 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.04469 s, 23.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106637 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871788 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140841 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104768 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927435 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970786 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0983565 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0066675 s, 157 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121107 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108523 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902847 s, 11.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.1859 s, 9.4 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.40077 s, 19.4 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.0374 s, 16.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542546 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805832 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221076 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280252 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169273 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138239 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221161 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268084 s, 3.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11961 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314215 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230307 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258542 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.432288 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195819 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258347 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.513539 s, 2.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123586 s, 8.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969995 s, 10.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773021 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0757952 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103686 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877868 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927951 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990229 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125989 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115006 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151647 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196178 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119213 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135797 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142858 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13922 s, 7.5 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.9167 s, 8.8 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0953 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1769 s, 10.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0337 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.03489 s, 16.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.061863 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147649 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0860691 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212799 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167035 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161546 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218207 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195553 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164208 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275218 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.213758 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234332 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276026 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775124 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247523 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.292611 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0827248 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049144 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379194 s, 27.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617442 s, 17.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100999 s, 10.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496379 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094726 s, 11.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140614 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075571 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752395 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109217 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844548 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124482 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219928 s, 47.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0758278 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935087 s, 11.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0126 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7293 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.636145 s, 26.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724495 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147491 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172229 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133916 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276204 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365753 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.373044 s, 2.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280876 s, 3.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412789 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298467 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.284182 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.453831 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306904 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.439756 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.406077 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.618082 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.012581 s, 83.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124767 s, 84.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00820195 s, 128 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0094759 s, 111 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00796708 s, 132 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0064291 s, 163 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108613 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139038 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.010018 s, 105 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173613 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0143791 s, 72.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133614 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920235 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210879 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0886225 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0728637 s, 14.4 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.9939 s, 8.7 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1517 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.06099 s, 15.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132097 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125352 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228867 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148633 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118115 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192767 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260634 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175719 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.239708 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394406 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321678 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.455406 s, 2.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353405 s, 3.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.482422 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.484337 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.389475 s, 2.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402515 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0410896 s, 25.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120048 s, 8.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546042 s, 19.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0785128 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0601026 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120363 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0561432 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948214 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776009 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0784852 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082786 s, 12.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114099 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0964393 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0990225 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106244 s, 9.9 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.4041 s, 8.5 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0642 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.4899 s, 7.8 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.2377 s, 10.2 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (487s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 15:55:22 (1777578922) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.763466 s, 22.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0887911 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158928 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261825 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.302245 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0903346 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378447 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226757 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259675 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.441756 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145375 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244264 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268598 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348427 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22048 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888282 s, 11.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.572378 s, 1.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125481 s, 83.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347197 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959317 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386878 s, 27.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0185349 s, 56.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207658 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0287341 s, 36.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354249 s, 29.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0547294 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393042 s, 26.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496529 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0333168 s, 31.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0487055 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501569 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00533386 s, 197 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00566597 s, 185 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.36985 s, 11.2 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0998 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (56s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 15:56:18 (1777578978) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.202.9@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.202.9@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.691612 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092329 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155975 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16981 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203211 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185731 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30307 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361752 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293621 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408722 s, 2.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241419 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395655 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.341679 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33783 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354269 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.475135 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.372139 s, 2.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0936764 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197234 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101987 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158329 s, 6.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180999 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152618 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129006 s, 8.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251102 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233678 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135708 s, 7.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221413 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158928 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222909 s, 4.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179822 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185838 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173667 s, 6.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=1 removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.80478 s, 11.9 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99667 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (55s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 15:57:13 (1777579033) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.202.9@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.883467 s, 19.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803257 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115183 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100975 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0897409 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117194 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274712 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301713 s, 3.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.269166 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22396 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225563 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376282 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.401051 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259681 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.309655 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.348841 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.465496 s, 2.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac08b6f000.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=3 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724128 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766182 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871489 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115412 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113374 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0984682 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137765 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134351 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144957 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113996 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144535 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101954 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111089 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121787 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130665 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128508 s, 8.2 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3105 s, 9.3 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.36168 s, 19.6 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (52s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 15:58:05 (1777579085) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.759614 s, 22.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0513944 s, 20.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108801 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114421 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223132 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313029 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330342 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.323031 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324449 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322776 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.441502 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.47195 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.512306 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.42534 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.48493 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.500113 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.548833 s, 1.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0024029 s, 436 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0127961 s, 81.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124368 s, 84.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0172904 s, 60.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.08576 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0290867 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.088537 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119506 s, 87.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152816 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0960807 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803153 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531681 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047793 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115673 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18401 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242469 s, 43.2 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.6657 s, 9.0 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9619 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.68129 s, 12.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.97394 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (100s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 15:59:45 (1777579185) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.07415 s, 15.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524475 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107461 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133049 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119333 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132667 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145101 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199616 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232203 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156519 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.365352 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.328455 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.398843 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326445 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325858 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.412432 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.454485 s, 2.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077398 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120514 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876293 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0713826 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102488 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783586 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850725 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834017 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095232 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0906025 s, 11.6 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991877 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132543 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138821 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108426 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110428 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119718 s, 8.8 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.2068 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0032 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (77s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 16:01:02 (1777579262) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.746284 s, 22.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075382 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947563 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122129 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162774 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169464 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276592 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185823 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301872 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.327188 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.331863 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310535 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.391205 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.32498 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.387041 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343198 s, 3.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.364117 s, 2.9 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88ac08b6f000.lock_unused_count=5 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0482972 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0579172 s, 18.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0591113 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687364 s, 15.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0712255 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753526 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0699051 s, 15.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0834032 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0902772 s, 11.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0859482 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133952 s, 7.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109086 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940702 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103345 s, 10.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119291 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0881041 s, 11.9 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.04859 s, 13.0 MB/s Write runtime is 9 s, speed is 11.111111 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.22 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg209-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.804234 s, 20.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058004 s, 18.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0461045 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0821492 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844019 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106426 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.060601 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.275944 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288541 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287672 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.311872 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362848 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.394651 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40429 s, 2.6 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.377083 s, 2.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421754 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.436068 s, 2.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158681 s, 66.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428159 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124108 s, 84.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914051 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0575459 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.033478 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036056 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738523 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743281 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764558 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0803765 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845499 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835703 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685712 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0432085 s, 24.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935771 s, 11.2 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 13.4673 s, 7.8 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0235 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.9484 s, 8.8 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0988 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (154s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 16:03:36 (1777579416) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg209-client.virtnet /mnt/lustre (opts:) Stopping client oleg209-client.virtnet /mnt/lustre opts: Stopping clients: oleg209-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg209-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg209-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg209-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.9,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg209-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.9,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg209-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.9,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/63203/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.9,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg209-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg209-server: detected 4 online CPUs by sysfs oleg209-server: MODOPTS_LIBCFS= oleg209-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre Starting client oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre Started clients oleg209-client.virtnet: 192.168.202.109@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 Starting client oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 Started clients oleg209-client.virtnet: 192.168.202.109@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88ac0334d800.idle_timeout=debug osc.lustre-OST0000-osc-ffff88ac04f6c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88ac0334d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88ac04f6c800.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.507722 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.073963 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.095103 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114556 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909996 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19678 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112983 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288437 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288496 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133454 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15021 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206267 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200296 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16639 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156877 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925673 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224628 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.571471 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.565813 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.549313 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.577296 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.531477 s, 2.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.551598 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.605942 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.54197 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.607195 s, 1.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.575673 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.590892 s, 1.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.559407 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.562718 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.616999 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.563253 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.633294 s, 1.7 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg209-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg209-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3675 s, 9.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.73612 s, 18.3 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.835372 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570814 s, 18.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0613088 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241869 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265003 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216217 s, 4.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27421 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257082 s, 4.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.277839 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.265039 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314921 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285847 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272454 s, 3.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac0334d800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff88ac0334d800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.358478 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.325655 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382119 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.395651 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.209461 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180817 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240633 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199146 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216753 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210474 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212445 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217842 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22776 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245196 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229905 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.231546 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238481 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22028 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229885 s, 4.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff88ac0334d800.lock_unused_count=3 ldlm.namespaces.lustre-OST0001-osc-ffff88ac0334d800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254092 s, 4.1 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.28546 s, 11.3 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0625 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (242s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 16:07:38 (1777579658) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg209-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (25s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 16:08:03 (1777579683) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (12s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 16:08:15 (1777579695) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg209-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.17895 s, 330 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg209-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.1167 s, 261 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (64s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 16:09:20 (1777579760) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.439641 s, 38.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0842107 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115867 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144835 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122406 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.274595 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285967 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217211 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161057 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.376014 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.353519 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.361515 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.296954 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.330061 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33165 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318131 s, 3.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787119 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643868 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185741 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0764883 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.421197 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.087796 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0539546 s, 19.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109267 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353403 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0226301 s, 46.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126694 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00232936 s, 450 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563322 s, 18.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00966349 s, 109 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0762877 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144795 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128324 s, 8.2 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.546 s, 9.1 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.74792 s, 18.2 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.723817 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14399 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124248 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25459 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279507 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234579 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.418633 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279574 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.342937 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.408165 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.443058 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.511434 s, 2.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.41187 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.613913 s, 1.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400233 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.447919 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.452669 s, 2.3 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff88ac0334d800.lock_unused_count=2 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.384788 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195984 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254459 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318981 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.318956 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229485 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.392797 s, 2.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.261173 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266915 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226302 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.340805 s, 3.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371131 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300035 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33975 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229351 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236666 s, 4.4 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg209-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg209-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.90205 s, 11.8 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.3237 s, 19.7 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (98s) == sanityn test 77o: Changing rank should not panic ====== 16:10:58 (1777579858) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (16s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 16:11:14 (1777579874) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_policies=fifo PASS 77q (143s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 16:13:37 (1777580017) mds.MDS.mdt.nrs_policies=tbf oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg209-client: oleg209-server: ssh exited with exit code 36 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg209-client: oleg209-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (47s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 16:14:24 (1777580064) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_policies=tbf jobid: Device or resource busy oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_policies'='tbf jobid': Device or resource busy pdsh@oleg209-client: oleg209-server: ssh exited with exit code 16 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 5465: kill: (269492) - No such process sanityn test_77r: @@@@@@ FAIL: timeout to start 'tbf jobid' policy Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7409:error() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:5516:test_77r() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7785:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7848:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7663:run_test() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:5534:main() Dumping lctl log to /tmp/testlogs//sanityn.test_77r.*.1777580306.log cleanup 77r 268572 rsync: chown "/tmp/testlogs/.sanityn.test_77r.debug_log.oleg209-server.1777580306.log.Vjf1Xs" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanityn.test_77r.dmesg.oleg209-server.1777580306.log.1OEq1d" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] kill md_thread xargs FAIL 77r (257s) == sanityn test 77s: Check TBF LRU shrinker ============== 16:18:54 (1777580334) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR oleg209-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_policies=tbf jobid: Resource temporarily unavailable oleg209-server: error: set_param: setting 'mds/MDS/mdt/nrs_policies'='tbf jobid': Resource temporarily unavailable pdsh@oleg209-client: oleg209-server: ssh exited with exit code 11 sanityn test_77s: @@@@@@ FAIL: fail to start 'tbf jobid' policy Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7409:error() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:5557:test_77s() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7785:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7848:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7663:run_test() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:5577:main() Dumping lctl log to /tmp/testlogs//sanityn.test_77s.*.1777580345.log rsync: chown "/tmp/testlogs/.sanityn.test_77s.debug_log.oleg209-server.1777580345.log.hol4jD" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanityn.test_77s.dmesg.oleg209-server.1777580345.log.jpAikP" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u FAIL 77s (23s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 16:19:29 (1777580369) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (12s) == sanityn test 79: xattr: intent error ================== 16:19:41 (1777580381) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826936406 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-30 16:19:43.000000000 -0400 Modify: 2026-04-30 16:19:43.000000000 -0400 Change: 2026-04-30 16:19:43.000000000 -0400 Birth: 2026-04-30 16:19:44.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (22s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 16:20:03 (1777580403) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (4s) == sanityn test 80b: Accessing directory during migration ========================================================== 16:20:07 (1777580407) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (5s) == sanityn test 81a: rename and stat under striped directory ========================================================== 16:20:12 (1777580412) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (4s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 16:20:16 (1777580416) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (4s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 16:20:20 (1777580420) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 16:20:24 (1777580424) - open/close 476 (time 1777580440.72 total 10.01 last 47.56) - open/close 962 (time 1777580450.74 total 20.02 last 48.54) - open/close 1462 (time 1777580460.76 total 30.04 last 49.90) - open/close 1892 (time 1777580470.78 total 40.06 last 42.90) - open/close 2282 (time 1777580480.82 total 50.10 last 38.85) total: 2500 open/close in 55.69 seconds: 44.89 ops/second snapshot_time 1777580606.895210057 secs.nsecs start_time 1777580428.429000088 secs.nsecs elapsed_time 178.466209969 secs.nsecs open 2505 samples [usecs] 177 25133 5812289 27089163071 close 2504 samples [usecs] 177 37007 1371452 4272096584 mknod 2501 samples [usecs] 691 24773 5015287 20800337555 unlink 2500 samples [usecs] 742 99573 6470138 71976205354 mkdir 2 samples [usecs] 2702 17539 20241 314917325 rmdir 2 samples [usecs] 9558 17570 27128 400060264 rename 2500 samples [usecs] 500 108844 22750761 469835242197 getattr 3764 samples [usecs] 6 21359 303716 1301165546 parallel_rename_file 2500 samples [usecs] 500 108844 22750761 469835242197 crossdir_rename 2500 samples [usecs] 500 108844 22750761 469835242197 PASS 81d (188s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 16:23:32 (1777580612) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7580 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7580 PASS 82 (8s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 16:23:40 (1777580620) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 16:23:45 (1777580625) fail_loc=0x8000060b PASS 84 (14s) == sanityn test 85: Lustre API root cache race =========== 16:23:59 (1777580639) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1777580641 Finishing test test1 at 1777580647 PASS 85 (14s) == sanityn test 90: open/create and unlink striped directory ========================================================== 16:24:13 (1777580653) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (5s) == sanityn test 91: chmod and unlink striped directory === 16:24:18 (1777580658) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (3s) == sanityn test 92: create remote directory under orphan directory ========================================================== 16:24:22 (1777580662) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (3s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 16:24:26 (1777580666) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6893 0x1aed 0x240000400 1 6856 0x1ac8 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6894 0x1aee 0x240000400 1 6857 0x1ac9 0x280000400 rc1=0 and rc2=0 PASS 93 (19s) == sanityn test 94: signal vs CP callback race =========== 16:24:44 (1777580684) osc.lustre-OST0000-osc-ffff88ac0334d800.idle_timeout=0 osc.lustre-OST0000-osc-ffff88ac04f6c800.idle_timeout=0 osc.lustre-OST0001-osc-ffff88ac0334d800.idle_timeout=0 osc.lustre-OST0001-osc-ffff88ac04f6c800.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.261284 s, 31.4 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.99837 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6178: 296397 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0948954 s, 43.2 kB/s osc.lustre-OST0000-osc-ffff88ac0334d800.idle_timeout=debug osc.lustre-OST0000-osc-ffff88ac04f6c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88ac0334d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88ac04f6c800.idle_timeout=debug PASS 94 (18s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 16:25:02 (1777580702) llite.lustre-ffff88ac0334d800.fast_read=0 llite.lustre-ffff88ac04f6c800.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0790087 s, 26.5 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff88ac0334d800.fast_read=1 llite.lustre-ffff88ac04f6c800.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 16:25:22 (1777580722) llite.lustre-ffff88ac0334d800.fast_read=0 llite.lustre-ffff88ac04f6c800.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0167998 s, 731 kB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff88ac0334d800.fast_read=1 llite.lustre-ffff88ac04f6c800.fast_read=1 PASS 95b (13s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 16:25:36 (1777580736) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (4s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 16:25:39 (1777580739) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00758706 s, 540 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (8s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 16:25:47 (1777580747) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.190142 s, 11.0 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (8s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 16:25:55 (1777580755) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.213862 s, 9.8 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (9s) == sanityn test 100e: DoM: read on open and file size ==== 16:26:04 (1777580764) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00890383 s, 1.3 MB/s PASS 100e (9s) == sanityn test 101a: Discard DoM data on unlink ========= 16:26:13 (1777580773) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0299483 s, 137 kB/s PASS 101a (9s) == sanityn test 101b: Discard DoM data on rename ========= 16:26:22 (1777580782) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00855412 s, 479 kB/s PASS 101b (8s) == sanityn test 101c: Discard DoM data on close-unlink === 16:26:30 (1777580790) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0174636 s, 235 kB/s PASS 101c (10s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 16:26:43 (1777580803) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1777580807 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1777580811 Finishing test test23 at 1777580811 PASS 103 (17s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 16:27:00 (1777580820) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 16:27:04 (1777580824) fail_loc=0x416 PASS 105 (25s) == sanityn test 106a: Verify the btime via statx() ======= 16:27:29 (1777580849) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 16:27:33 (1777580853) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0329024 s, 31.1 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938941 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-30 16:27:35.000000000 -0400 Modify: 2026-04-30 16:27:35.000000000 -0400 Change: 2026-04-30 16:27:35.000000000 -0400 Birth: 2026-04-30 16:27:36.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938941 -rw-r--r-- 1 0 0 1777580856 1777580855 1777580855 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938941 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-30 16:27:35.000000000 -0400 Modify: 2026-04-30 16:27:35.000000000 -0400 Change: 2026-04-30 16:27:35.000000000 -0400 Birth: 2026-04-30 16:27:36.000000000 -0400 1777580855 1024 1 PASS 106b (9s) == sanityn test 106c: Verify statx attributes mask ======= 16:27:42 (1777580862) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 16:27:50 (1777580870) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.325298 s, 32.2 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 6863 0x1acf 0x280000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 First grouplock blocks second one PASS 107a (9s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 16:27:59 (1777580879) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.25007 s, 41.9 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 6903 0x1af7 0x240000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7580 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (14s) == sanityn test 108a: lseek: parallel updates ============ 16:28:13 (1777580893) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.0187768 s, 533 kB/s llite.lustre-ffff88ac0334d800.hybrid_io=0 llite.lustre-ffff88ac04f6c800.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.69489 s, 1.8 MB/s PASS 108a (13s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 16:28:26 (1777580906) 192.168.202.109@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre (opts:) 192.168.202.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 192.168.202.109@tcp:/lustre /mnt/lustre lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.202.109@tcp:/lustre /mnt/lustre2 lustre rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg209-client.virtnet /mnt/lustre (opts:) Stopping client oleg209-client.virtnet /mnt/lustre2 (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg209-server: detected 4 online CPUs by sysfs oleg209-server: MODOPTS_LIBCFS= oleg209-server: Force libcfs to create 2 CPU partitions Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre2 PASS 109 (54s) == sanityn test 110: do not grant another lock on resend ========================================================== 16:29:20 (1777580960) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-04-30 16:29:22.000000000 -0400 Modify: 2026-04-30 16:29:22.000000000 -0400 Change: 2026-04-30 16:29:22.000000000 -0400 Birth: 2026-04-30 16:29:23.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg209-client.virtnet oleg209-client.virtnet PASS 110 (1551s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 16:55:12 (1777582512) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (6s) == sanityn test 112: update max-inherit in default LMV === 16:55:18 (1777582518) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (6s) == sanityn test 113: check servers of specified fs ======= 16:55:25 (1777582525) lustre-OST0000-osc-ffff88ac05e12800 active. lustre-OST0001-osc-ffff88ac05e12800 active. lustre-OST0000-osc-ffff88ac05e12800 active. lustre-OST0001-osc-ffff88ac05e12800 active. MGC192.168.202.109@tcp active. PASS 113 (12s) == sanityn test 114: implicit default LMV inherit ======== 16:55:36 (1777582536) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (4s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 16:55:40 (1777582540) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (4s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 16:55:44 (1777582544) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 16:55:48 (1777582548) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0309425 s, 662 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.102436 s, 200 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (12s) == sanityn test 200: service remains healthy while able to process request ========================================================== 16:56:00 (1777582560) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2744 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.2942 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3661 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3348 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.35 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3864 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.3228 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.2037 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3308 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3593 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3899 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3823 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.4547 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.3979 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4305 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.5534 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4701 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.6294 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg209-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88ac05e12800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff88ac05e12800.ost_server_uuid in FULL state after 0 sec PASS 200 (87s) cleanup: ====================================================== == sanityn test complete, duration 10045 sec ============= 16:57:29 (1777582649) sanityn: FAIL: test_77r timeout to start 'tbf jobid' policy sanityn: FAIL: test_77s fail to start 'tbf jobid' policy === sanityn: start cleanup 16:57:31 (1777582651) === Stopping clients: oleg209-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg209-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 17:02:47 (1777582967) ===