-----============= acceptance-small: sanityn ============----- Tue Jun 9 02:40:33 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg653-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/sanityn.*ex': No such file or directory excepting tests: 27 40a 102 skipping tests SLOW=no: 33a === sanityn: start setup 02:40:55 (1780987255) === Starting client oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre2 Mount client oleg653-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre2 Started clients oleg653-client.virtnet: 192.168.206.153@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg653-client.virtnet: executing check_config_client /mnt/lustre oleg653-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg653-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8a2a59a58800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8a2a60524000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a2a59a58800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a2a60524000.idle_timeout=debug disable quota as required oleg653-server: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all === sanityn: finish setup 02:41:35 (1780987295) === 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0868892 s, 12.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f7563] == sanityn test 0: do_node_vp() and do_facet_vp() do the right thing ========================================================== 02:41:38 (1780987298) PASS 0 (10s) == sanityn test 1: Check attribute updates on 2 mount points ========================================================== 02:41:48 (1780987308) /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0777 OK /mnt/lustre/f1.sanityn has type file OK /mnt/lustre/f1.sanityn has perms 0666 OK /mnt/lustre/f1.sanityn: absent OK PASS 1 (9s) == sanityn test 2a: check cached attribute updates on 2 mtpt's ================================================================== 02:41:57 (1780987317) -rw-r--r-- 1 root root 0 Jun 9 02:41 /mnt/lustre2/f2a /mnt/lustre/f2a has type file OK /mnt/lustre/f2a has perms 0777 OK PASS 2a (11s) == sanityn test 2b: check cached attribute updates on 2 mtpt's ================================================================== 02:42:09 (1780987329) -rw-r--r-- 1 root root 0 Jun 9 02:42 /mnt/lustre2/f2b /mnt/lustre2/f2b has type file OK /mnt/lustre2/f2b has perms 0777 OK PASS 2b (9s) == sanityn test 2c: check cached attribute updates on 2 mtpt's root ============================================================= 02:42:17 (1780987337) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0777 OK PASS 2c (9s) == sanityn test 2d: check cached attribute updates on 2 mtpt's root ============================================================= 02:42:27 (1780987347) /mnt/lustre2 has type dir OK /mnt/lustre2 has perms 0755 OK PASS 2d (11s) == sanityn test 2e: check chmod on root is propagated to others ========================================================== 02:42:37 (1780987357) total 1 -rwxrwxrwx 1 root root 0 Jun 9 02:41 f2a -rwxrwxrwx 1 root root 0 Jun 9 02:42 f2b total 1 -rwxrwxrwx 1 root root 0 Jun 9 02:41 f2a -rwxrwxrwx 1 root root 0 Jun 9 02:42 f2b running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre2/f2e.sanityn] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.020002 s, 25.6 kB/s PASS 2e (9s) == sanityn test 2f: check attr/owner updates on DNE with 2 mtpt's ========================================================== 02:42:46 (1780987366) SKIP: sanityn test_2f needs >= 2 MDTs SKIP 2f (4s) == sanityn test 2g: check blocks update on sync write ==== 02:42:50 (1780987370) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.302494 s, 6.9 MB/s /mnt/lustre/f2g.sanityn has 4103 blocks /mnt/lustre2/f2g.sanityn has 4103 blocks PASS 2g (7s) == sanityn test 3: symlink on one mtpt, readlink on another ===================================================================== 02:42:57 (1780987377) PASS 3 (7s) == sanityn test 4: fstat validation on multiple mount points ==================================================================== 02:43:04 (1780987384) PASS 4 (9s) == sanityn test 5: create a file on one mount, truncate it on the other ========================================================== 02:43:13 (1780987393) /mnt/lustre/f5 has type file OK /mnt/lustre/f5 has size 100 OK PASS 5 (8s) == sanityn test 6: remove of open file on other node ============================================================================ 02:43:21 (1780987401) opening writing unlinking /mnt/lustre2/f6.sanityn accessing (1) seeking (1) accessing (2) fstat... reading comparing data truncating seeking (2) writing again seeking (3) reading again comparing data again closing SUCCESS - goto beer PASS 6 (8s) == sanityn test 7: remove of open directory on other node ======================================================================= 02:43:29 (1780987409) creating directory /mnt/lustre/d7 opening directory unlinking /mnt/lustre/d7 Ok, everything goes well. PASS 7 (7s) == sanityn test 8: remove of open special file on other node ==================================================================== 02:43:36 (1780987416) creating special file /mnt/lustre/f8.sanityn opening file unlinking /mnt/lustre/f8.sanityn Ok, everything goes well. PASS 8 (8s) == sanityn test 9a: append of file with sub-page size on multiple mounts ========================================================== 02:43:44 (1780987424) PASS 9a (7s) == sanityn test 9b: append to striped sparse file ======== 02:43:51 (1780987431) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684529 s, 15.3 MB/s 3+0 records in 3+0 records out 3 bytes copied, 0.0741304 s, 0.0 kB/s Data read (expecting 'foo'): foo PASS 9b (8s) == sanityn test 10a: write of file with sub-page size on multiple mounts ========================================================== 02:43:59 (1780987439) 1+0 records in 1+0 records out 1 byte copied, 0.00922917 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0314152 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.010564 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.017774 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0201257 s, 0.0 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0110918 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0177091 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0153427 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.00906311 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0188083 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0138186 s, 0.1 kB/s 1+0 records in 1+0 records out 1 byte copied, 0.0127836 s, 0.1 kB/s PASS 10a (8s) == sanityn test 10b: write of file with sub-page size on multiple mounts ========================================================== 02:44:08 (1780987448) 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0170715 s, 180 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0602422 s, 68.0 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.000489588 s, 6.3 MB/s PASS 10b (8s) == sanityn test 11: execution of file opened for write should return error ============================================================== 02:44:15 (1780987455) multiop /mnt/lustre/d11/f vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 300: /mnt/lustre2/d11/f: Text file busy PASS 11 (8s) == sanityn test 12: test lock ordering (link, stat, unlink) ========================================================== 02:44:23 (1780987463) warning: '-runtime' deprecated, use '-t runtime' instead - link 319 (time 1780987475.71 total 10.01 last 31.88) - link 617 (time 1780987485.73 total 20.02 last 29.75) - link 942 (time 1780987495.74 total 30.03 last 32.46) - link 1305 (time 1780987505.76 total 40.05 last 36.24) - link 1621 (time 1780987515.77 total 50.06 last 31.57) - link 1921 (time 1780987525.77 total 60.06 last 29.99) - link 2234 (time 1780987535.78 total 70.07 last 31.26) - link 2557 (time 1780987545.78 total 80.08 last 32.29) - link 2879 (time 1780987555.80 total 90.09 last 32.15) total: 3172 link in 100.01 seconds: 31.72 ops/second using seed 2767258800 running for 100 seconds - stat 4087 (time 1780987477 ; total 11 ; last 11) - stat 8273 (time 1780987488 ; total 22 ; last 11) - stat 10000 (time 1780987491 ; total 25 ; last 3) - stat 14499 (time 1780987502 ; total 36 ; last 11) - stat 19459 (time 1780987513 ; total 47 ; last 11) - stat 20000 (time 1780987514 ; total 48 ; last 1) - stat 22453 (time 1780987525 ; total 59 ; last 11) - stat 24341 (time 1780987536 ; total 70 ; last 11) - stat 26554 (time 1780987547 ; total 81 ; last 11) - stat 28771 (time 1780987558 ; total 92 ; last 11) - stat 30000 (time 1780987564 ; total 98 ; last 6) total: 30395 stats in 100 seconds: 303.950012 stats/second - unlinked 0 (time 1780987516 ; total 0 ; last 0) unlink(/mnt/lustre2/lockdir/lockfile3172) error: No such file or directory total: 3171 unlinks in 87 seconds: 36.448277 unlinks/second /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 77: kill: (21329) - No such process /home/green/git/lustre-release/lustre/tests/lockorder.sh: line 78: kill: (21331) - No such process PASS 12 (147s) == sanityn test 13: test directory page revocation ======= 02:46:50 (1780987610) f13 f13 PASS 13 (8s) == sanityn test 14aa: execution of file open for write returns -ETXTBSY ========================================================== 02:46:59 (1780987619) multiop /mnt/lustre/d14aa.sanityn/f14aa.sanityn vOw_c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 335: /mnt/lustre2/d14aa.sanityn/f14aa.sanityn: Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5001: 22516 Terminated $MULTIOP_PROG $FILE v$ARGS > $TMPPIPE (wd: ~) PASS 14aa (8s) == sanityn test 14ab: open(RDWR) of executing file returns -ETXTBSY ========================================================== 02:47:06 (1780987626) open(O_RDWR|O_CREAT): Text file busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5001: 23096 Terminated $DIR1/$tdir/sleep 60 (wd: ~) PASS 14ab (8s) == sanityn test 14b: truncate of executing file returns -ETXTBSY ================================================================ 02:47:14 (1780987634) truncate: cannot truncate '/mnt/lustre2/d14b.sanityn/sleep' to length 60: Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 350: 23675 Terminated $DIR1/$tdir/sleep 60 PASS 14b (8s) == sanityn test 14c: open(O_TRUNC) of executing file return -ETXTBSY ============================================================ 02:47:22 (1780987642) cp: cannot create regular file '/mnt/lustre2/d14c.sanityn/sleep': Text file busy /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 362: 24257 Terminated $DIR1/$tdir/sleep 60 PASS 14c (7s) == sanityn test 14d: chmod of executing file is still possible ================================================================== 02:47:30 (1780987650) chmod /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 373: 24839 Terminated $DIR1/$tdir/sleep 60 PASS 14d (9s) == sanityn test 15: test out-of-space with multiple writers ===================================================================== 02:47:40 (1780987660) PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin: MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky sleep 5 for ZFS MDS Waiting for MDT destroys to complete fallocate on zfs doesn't consume space STRIPECOUNT=2 ORIGFREE=7531520 MAXFREE=800000 SKIP: oos2 test_15 oos2.sh: 7531520KB free > 800000KB, increase MAXFREE (or reduce fs size) sleep 5 for ZFS MDS Waiting for MDT destroys to complete SKIP 15 (50s) == sanityn test 16a: 500 iterations of dual-mount fsx ==== 02:48:29 (1780987709) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.83095 s, 10.1 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.486272 s, 17.3 MB/s Chance of close/open is 1 in 50 Seed set to 7714 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x448080 000008[0] 1780987715.453676 trunc from 00000000 to 0x44807f (0x448080 bytes) truncating to largest ever: 0x7d09b3 000009[0] 1780987715.545727 trunc from 0x448080 to 0x7d09b2 (0x388933 bytes) 000011[0] 1780987715.638585 mapwrite 0x00d287 thru 0x016f9e (0x9d18 bytes) 000013[1] 1780987715.764043 mapwrite 0x1225ab thru 0x131491 (0xeee7 bytes) 000016[1] 1780987716.026361 read 0x49c2dd thru 0x4aac27 (0xe94b bytes) 000017[1] 1780987716.101695 write 0x85bbdf thru 0x8694fe (0xd920 bytes) 000020[0] 1780987716.153929 mapread 0x7ed4e5 thru 0x7f8bfe (0xb71a bytes) 000021[1] 1780987716.561509 write 0x1cac87 thru 0x1d3eeb (0x9265 bytes) 000022[1] 1780987716.576792 mapwrite 0x780cb2 thru 0x78b35b (0xa6aa bytes) 000027[1] 1780987716.741764 trunc from 0x8694ff to 0x03b657 (0x82dea7 bytes) 000031[0] 1780987716.818670 read 0x0297ef thru 0x035c40 (0xc452 bytes) 000033[0] 1780987716.891447 write 0x6cb77e thru 0x6cc745 (0xfc8 bytes) 000034[0] 1780987716.907516 mapread 0x402f5a thru 0x40dd50 (0xadf7 bytes) 000037[0] 1780987716.956675 write 0x7dbb75 thru 0x7eb3de (0xf86a bytes) 000041[1] 1780987716.996546 read 0x5782d8 thru 0x57e921 (0x664a bytes) 000043[1] 1780987717.034061 trunc from 0x7eb3df to 0x07385e (0x777b80 bytes) 000045[0] 1780987717.137876 mapread 0x0582a6 thru 0x05cbd1 (0x492c bytes) 000048[0] 1780987717.219374 mapwrite 0x76867e thru 0x76ba42 (0x33c5 bytes) 000050[1] 1780987717.372737 read 0x0de285 thru 0x0eca3e (0xe7ba bytes) 000057[0] 1780987717.568461 write 0x24af8f thru 0x24ec73 (0x3ce5 bytes) 000062[0] 1780987717.703453 mapwrite 0x9be63e thru 0x9bf237 (0xbfa bytes) 000063[1] 1780987717.844106 mapwrite 0x2689db thru 0x26fe8a (0x74b0 bytes) 000064[1] 1780987717.962191 read 0x4b9319 thru 0x4ba5d6 (0x12be bytes) 000065[1] 1780987717.991642 write 0x21bee5 thru 0x22089b (0x49b7 bytes) 000067[1] 1780987718.070270 mapread 0x333c32 thru 0x335890 (0x1c5f bytes) 000068[0] 1780987718.112330 mapwrite 0x0bd79a thru 0x0c9eee (0xc755 bytes) 000071[0] 1780987718.310081 read 0x5b5579 thru 0x5b9672 (0x40fa bytes) 000073[1] 1780987718.349331 mapwrite 0x0dfc2f thru 0x0eae62 (0xb234 bytes) 000074[1] 1780987718.583015 write 0x258f33 thru 0x26579a (0xc868 bytes) 000075[1] 1780987718.609123 mapread 0x80190e thru 0x805245 (0x3938 bytes) 000076[1] 1780987718.682004 mapread 0x2124dc thru 0x21a43c (0x7f61 bytes) 000077[1] 1780987718.738867 read 0x86ee9f thru 0x87638b (0x74ed bytes) 000078[0] 1780987718.758330 trunc from 0x9bf238 to 0x2d4c93 (0x6ea5a4 bytes) 000080[1] 1780987718.875254 mapread 0x1aeac0 thru 0x1bd817 (0xed58 bytes) truncating to largest ever: 0x7f78b4 000081[1] 1780987718.940191 trunc from 0x2d4c94 to 0x7f78b3 (0x522c20 bytes) 000082[1] 1780987719.004179 read 0x51a13c thru 0x51ca14 (0x28d9 bytes) 000083[1] 1780987719.054707 trunc from 0x7f78b4 to 0x314ff7 (0x4e28bc bytes) 000087[1] 1780987719.198183 trunc from 0x314ff8 to 0x611dbd (0x2fcdc6 bytes) 000088[0] 1780987719.228100 mapread 0x0f9153 thru 0x0fcdea (0x3c98 bytes) 000089[1] 1780987719.326770 mapread 0x0d1aa9 thru 0x0d8e13 (0x736b bytes) 000093[1] 1780987719.499582 trunc from 0x611dbe to 0x1a7fb5 (0x469e08 bytes) 000095[0] 1780987719.588565 trunc from 0x1a7fb6 to 0x6b377c (0x50b7c7 bytes) 000100[1] 1780987719.806986 read 0x660143 thru 0x6636cf (0x358d bytes) 000101[1] 1780987719.841739 mapwrite 0x71a6d7 thru 0x726169 (0xba93 bytes) 000102[0] 1780987719.977829 mapwrite 0x0d0095 thru 0x0d355b (0x34c7 bytes) truncating to largest ever: 0x95933e 000104[1] 1780987720.202962 trunc from 0x72616a to 0x95933d (0x2331d4 bytes) 000105[0] 1780987720.296899 trunc from 0x95933e to 0x39426b (0x5c50d2 bytes) 000107[1] 1780987720.374971 mapread 0x37dc3b thru 0x382b63 (0x4f29 bytes) 000108[0] 1780987720.471920 mapread 0x15ed1c thru 0x16d3e0 (0xe6c5 bytes) 000109[1] 1780987720.513073 read 0x273089 thru 0x279b51 (0x6ac9 bytes) 000110[0] 1780987720.597935 mapread 0x2531dc thru 0x2619d8 (0xe7fd bytes) 000112[0] 1780987720.658510 trunc from 0x39426c to 0x5b0e2d (0x21cbc2 bytes) 000113[1] 1780987720.708301 mapwrite 0x7c87c9 thru 0x7d6de1 (0xe619 bytes) 000115[0] 1780987721.006525 mapread 0x02c552 thru 0x036db3 (0xa862 bytes) 000117[0] 1780987721.110014 write 0x882e7d thru 0x88767a (0x47fe bytes) 000121[1] 1780987721.270587 read 0x240913 thru 0x243781 (0x2e6f bytes) 000124[0] 1780987721.335229 trunc from 0x88767b to 0x0d79ac (0x7afcce bytes) 000125[0] 1780987721.405830 write 0x05c406 thru 0x06049d (0x4098 bytes) 000129[0] 1780987721.488763 write 0x962eb2 thru 0x96c1d8 (0x9327 bytes) 000135[1] 1780987721.535112 write 0x2264b2 thru 0x226dda (0x929 bytes) 000139[1] 1780987721.668120 read 0x0a6c7e thru 0x0b5901 (0xec84 bytes) 000143[1] 1780987721.755889 mapwrite 0x7619ac thru 0x76b1b3 (0x9808 bytes) 000146[1] 1780987722.036788 read 0x72d6c7 thru 0x73c75f (0xf099 bytes) 000147[0] 1780987722.048319 trunc from 0x96c1d9 to 0x4ccf1f (0x49f2b9 bytes) 000148[1] 1780987722.168469 write 0x7969d4 thru 0x7a2440 (0xba6d bytes) 000149[0] 1780987722.235529 write 0x938182 thru 0x93c639 (0x44b8 bytes) 000150[0] 1780987722.257602 mapwrite 0x4f516b thru 0x500dbe (0xbc54 bytes) 000152[1] 1780987722.478762 trunc from 0x93c63a to 0x1acadd (0x78fb5c bytes) 000153[0] 1780987722.604175 read 0x0b84b2 thru 0x0be3f1 (0x5f40 bytes) 000154[0] 1780987722.674566 read 0x13c9b0 thru 0x1437a9 (0x6dfa bytes) 000156[0] 1780987722.700889 write 0x85d325 thru 0x862b0c (0x57e8 bytes) 000157[1] 1780987722.745870 trunc from 0x862b0d to 0x562090 (0x300a7c bytes) truncating to largest ever: 0x96e0a5 000158[0] 1780987722.806127 trunc from 0x562091 to 0x96e0a4 (0x40c014 bytes) 000160[1] 1780987722.892677 write 0x8f346b thru 0x902304 (0xee9a bytes) 000162[0] 1780987723.009266 write 0x95f7a8 thru 0x9670be (0x7917 bytes) 000163[1] 1780987723.310807 read 0x74ff32 thru 0x75e3e0 (0xe4af bytes) 000166[0] 1780987723.485626 read 0x5a393c thru 0x5b2b08 (0xf1cd bytes) 000167[1] 1780987723.509263 trunc from 0x96e0a5 to 0x2757d8 (0x6f88cc bytes) 000173[0] 1780987723.667260 mapread 0x15c136 thru 0x165b73 (0x9a3e bytes) 000174[1] 1780987723.717642 trunc from 0x2757d9 to 0x169142 (0x10c696 bytes) 000175[1] 1780987723.751907 mapread 0x1203a4 thru 0x12bbdd (0xb83a bytes) 000176[1] 1780987723.783260 write 0x874fd9 thru 0x884fae (0xffd6 bytes) 000177[1] 1780987723.821217 read 0x49a854 thru 0x4a8d79 (0xe526 bytes) 000180[1] 1780987723.833790 read 0x68c8d8 thru 0x695b20 (0x9249 bytes) 000181[0] 1780987723.846303 mapread 0x656163 thru 0x665fd6 (0xfe74 bytes) 000182[1] 1780987724.106002 write 0x0a605a thru 0x0b384b (0xd7f2 bytes) 000184[0] 1780987724.130789 read 0x4cf4b8 thru 0x4d9f9f (0xaae8 bytes) 000186[0] 1780987724.230164 mapwrite 0x7c5247 thru 0x7c5df6 (0xbb0 bytes) 000187[0] 1780987724.345334 read 0x15180c thru 0x152456 (0xc4b bytes) 000188[0] 1780987724.456507 mapread 0x4fcfd4 thru 0x4fd430 (0x45d bytes) 000189[0] 1780987724.474011 write 0x6384bf thru 0x639835 (0x1377 bytes) 000190[1] 1780987724.500098 write 0x779112 thru 0x77a1e1 (0x10d0 bytes) 000195[1] 1780987724.599004 read 0x6d7658 thru 0x6e289b (0xb244 bytes) 000196[1] 1780987724.645255 write 0x4eb8c9 thru 0x4f0edf (0x5617 bytes) 000198[0] 1780987724.746526 read 0x579dcd thru 0x587a0f (0xdc43 bytes) 000199[1] 1780987724.786168 mapwrite 0x904441 thru 0x90580c (0x13cc bytes) 000200[0] 1780987725.010793 mapread 0x2cd5ab thru 0x2ce96d (0x13c3 bytes) 000202[1] 1780987725.076735 write 0x83a0fd thru 0x83ef80 (0x4e84 bytes) 000203[0] 1780987725.135676 write 0x9e32ae thru 0x9e9627 (0x637a bytes) 000205[0] 1780987725.247915 mapwrite 0x0e1e1f thru 0x0eadb9 (0x8f9b bytes) 000206[1] 1780987725.452159 mapread 0x0f161c thru 0x0f286d (0x1252 bytes) 000208[1] 1780987725.544701 mapwrite 0x910c54 thru 0x915e75 (0x5222 bytes) 000211[1] 1780987725.811734 write 0x8f0aaa thru 0x8fef7f (0xe4d6 bytes) 000214[1] 1780987725.856130 trunc from 0x9e9628 to 0x5125f3 (0x4d7034 bytes) 000215[0] 1780987725.938787 mapread 0x461ad2 thru 0x464292 (0x27c1 bytes) 000216[1] 1780987726.043192 trunc from 0x5125f4 to 0x44b5fa (0xc6ff9 bytes) 000217[0] 1780987726.096446 mapread 0x4008fe thru 0x403d46 (0x3449 bytes) 000219[1] 1780987726.168728 read 0x1ab202 thru 0x1b9fc8 (0xedc7 bytes) 000222[1] 1780987726.192748 mapread 0x27ad92 thru 0x2870cc (0xc33b bytes) 000223[1] 1780987726.222140 trunc from 0x44b5fb to 0x3d325c (0x7839e bytes) 000224[0] 1780987726.270368 read 0x30faa2 thru 0x3139a2 (0x3f01 bytes) 000225[0] 1780987726.299068 read 0x3cd204 thru 0x3d0863 (0x3660 bytes) 000226[0] 1780987726.385196 trunc from 0x3d325d to 0x66e1a9 (0x29af4d bytes) 000227[0] 1780987726.454103 trunc from 0x66e1aa to 0x076f8f (0x5f721a bytes) 000230[0] 1780987726.575462 read 0x071928 thru 0x076f8f (0x5668 bytes) 000231[1] 1780987726.612916 read 0x0338b6 thru 0x0397a3 (0x5eee bytes) 000238[0] 1780987726.745828 mapread 0x056d31 thru 0x05d373 (0x6643 bytes) 000239[0] 1780987726.828770 mapread 0x00a519 thru 0x0135de (0x90c6 bytes) 000240[0] 1780987726.922251 read 0x04cfec thru 0x051956 (0x496b bytes) 000243[0] 1780987726.991887 read 0x04d298 thru 0x0563c9 (0x9132 bytes) 000244[1] 1780987727.015593 write 0x50dd07 thru 0x5104cc (0x27c6 bytes) 000245[1] 1780987727.034794 trunc from 0x5104cd to 0x60bd2b (0xfb85f bytes) 000246[0] 1780987727.081450 mapwrite 0x8b2daa thru 0x8bb9fb (0x8c52 bytes) 000249[1] 1780987727.612312 write 0x0f4a3a thru 0x100097 (0xb65e bytes) 000250[0] 1780987727.809810 mapwrite 0x8abc68 thru 0x8b3c5f (0x7ff8 bytes) 000252[1] 1780987728.084113 write 0x7f4e95 thru 0x803e2f (0xef9b bytes) 000256[0] 1780987728.252137 read 0x7833bf thru 0x785e7e (0x2ac0 bytes) 000257[0] 1780987728.343966 mapread 0x870474 thru 0x871875 (0x1402 bytes) 000258[1] 1780987728.407377 mapread 0x3a678b thru 0x3b58a3 (0xf119 bytes) 000259[1] 1780987728.498332 trunc from 0x8bb9fc to 0x350cf9 (0x56ad02 bytes) 000260[1] 1780987728.566165 read 0x08612e thru 0x094b27 (0xe9fa bytes) 000264[0] 1780987728.726270 read 0x095dec thru 0x0a0f1b (0xb130 bytes) 000266[1] 1780987728.868753 mapread 0x29bb14 thru 0x2a4958 (0x8e45 bytes) 000267[1] 1780987728.938572 write 0x998167 thru 0x99d55e (0x53f8 bytes) 000269[0] 1780987728.976266 write 0x6b5c6a thru 0x6bcb96 (0x6f2d bytes) 000273[0] 1780987729.118545 trunc from 0x99d55f to 0x0f7077 (0x8a64e7 bytes) 000274[0] 1780987729.213370 write 0x6424a4 thru 0x64251c (0x79 bytes) 000275[0] 1780987729.243428 mapwrite 0x733f79 thru 0x73ca83 (0x8b0b bytes) 000276[0] 1780987729.507258 read 0x72eaba thru 0x7300a0 (0x15e7 bytes) 000277[1] 1780987729.531472 mapread 0x017cac thru 0x02690c (0xec61 bytes) 000278[1] 1780987729.697051 mapread 0x02d5e6 thru 0x03ce1a (0xf835 bytes) 000279[0] 1780987729.799649 write 0x58bab5 thru 0x595238 (0x9784 bytes) 000280[1] 1780987729.816920 mapread 0x67efd9 thru 0x68cd61 (0xdd89 bytes) 000281[0] 1780987729.912177 write 0x5be750 thru 0x5c461d (0x5ece bytes) 000282[1] 1780987729.965996 trunc from 0x73ca84 to 0x50683b (0x236248 bytes) 000284[1] 1780987730.068005 trunc from 0x50683c to 0x461404 (0xa5437 bytes) 000285[0] 1780987730.117721 mapwrite 0x1d496b thru 0x1d67ed (0x1e83 bytes) 000286[1] 1780987730.229296 read 0x1f11ea thru 0x1f4c50 (0x3a67 bytes) 000287[0] 1780987730.255625 write 0x0a0bdf thru 0x0ab290 (0xa6b2 bytes) 000290[0] 1780987730.331173 trunc from 0x461405 to 0x0fdac4 (0x363940 bytes) 000294[0] 1780987730.397642 mapread 0x021573 thru 0x026603 (0x5091 bytes) 000295[0] 1780987730.444223 trunc from 0x0fdac5 to 0x396509 (0x298a45 bytes) 000297[0] 1780987730.567988 mapwrite 0x785a80 thru 0x790224 (0xa7a5 bytes) 000298[0] 1780987730.778363 write 0x794cab thru 0x799ae0 (0x4e36 bytes) 000299[0] 1780987730.782887 trunc from 0x799ae1 to 0x7070d8 (0x92a08 bytes) 000300[0] 1780987730.802671 write 0x58db84 thru 0x59ae64 (0xd2e1 bytes) 000301[1] 1780987730.829032 mapread 0x5dd214 thru 0x5df5ee (0x23db bytes) 000304[0] 1780987730.993647 read 0x46673c thru 0x46e3f3 (0x7cb8 bytes) 000306[1] 1780987731.044314 write 0x382f3b thru 0x38abd5 (0x7c9b bytes) 000308[0] 1780987731.098717 trunc from 0x7070d9 to 0x7dcd13 (0xd5c3b bytes) 000311[1] 1780987731.242824 mapwrite 0x2416b4 thru 0x2509df (0xf32c bytes) 000312[0] 1780987731.526269 write 0x55c253 thru 0x5666dd (0xa48b bytes) 000313[1] 1780987731.578483 read 0x358a78 thru 0x35b55d (0x2ae6 bytes) 000314[1] 1780987731.617703 mapwrite 0x768e71 thru 0x7724d5 (0x9665 bytes) 000315[1] 1780987731.782850 write 0x43fdca thru 0x447895 (0x7acc bytes) 000318[1] 1780987731.840892 mapread 0x4e0c59 thru 0x4f0484 (0xf82c bytes) 000322[1] 1780987732.030034 write 0x16aa8f thru 0x170a7d (0x5fef bytes) 000323[0] 1780987732.078628 read 0x6f2253 thru 0x700e4c (0xebfa bytes) 000325[0] 1780987732.193588 mapread 0x42e2be thru 0x43a6d7 (0xc41a bytes) 000327[0] 1780987732.339806 mapread 0x4d23e4 thru 0x4d8d44 (0x6961 bytes) 000328[0] 1780987732.384768 trunc from 0x7dcd14 to 0x5629f3 (0x27a320 bytes) 000329[1] 1780987732.529741 mapwrite 0x177faa thru 0x181db2 (0x9e09 bytes) 000331[1] 1780987732.716649 read 0x439053 thru 0x43f014 (0x5fc2 bytes) 000332[0] 1780987732.794135 read 0x057095 thru 0x05e8d7 (0x7843 bytes) 000333[0] 1780987732.837273 write 0x3f6d60 thru 0x3fe36e (0x760f bytes) 000335[0] 1780987732.931465 read 0x35e8d3 thru 0x366a75 (0x81a3 bytes) 000339[1] 1780987732.978197 mapwrite 0x51030a thru 0x515786 (0x547d bytes) 000343[1] 1780987733.267948 read 0x372077 thru 0x37fd80 (0xdd0a bytes) 000344[0] 1780987733.351773 mapwrite 0x9db949 thru 0x9e8db2 (0xd46a bytes) 000345[0] 1780987733.577031 mapread 0x3782b0 thru 0x38701c (0xed6d bytes) 000346[0] 1780987733.648881 mapwrite 0x1044d6 thru 0x1098de (0x5409 bytes) 000348[0] 1780987733.805344 read 0x03627b thru 0x03a028 (0x3dae bytes) 000350[0] 1780987733.845107 read 0x14c572 thru 0x153e80 (0x790f bytes) 000351[1] 1780987733.874276 mapwrite 0x74c07d thru 0x75a0a2 (0xe026 bytes) 000352[0] 1780987734.037111 mapwrite 0x1ba6b2 thru 0x1c3e7b (0x97ca bytes) 000353[1] 1780987734.238715 read 0x84afa8 thru 0x85433c (0x9395 bytes) 000354[1] 1780987734.318602 trunc from 0x9e8db3 to 0x6b878e (0x330624 bytes) 000355[1] 1780987734.411152 trunc from 0x6b878f to 0x223c9c (0x494af2 bytes) 000359[1] 1780987734.608439 mapwrite 0x514ede thru 0x519850 (0x4973 bytes) 000360[1] 1780987734.834983 trunc from 0x519851 to 0x5850d1 (0x6b881 bytes) 000361[0] 1780987734.895937 mapread 0x0a7117 thru 0x0b4d58 (0xdc42 bytes) 000362[0] 1780987735.013239 mapwrite 0x7cd497 thru 0x7d492e (0x7498 bytes) 000366[0] 1780987735.228354 mapwrite 0x00705b thru 0x00e8df (0x7885 bytes) 000368[0] 1780987735.313858 trunc from 0x7d492f to 0x440be0 (0x393d4e bytes) 000369[0] 1780987735.335337 write 0x5660e0 thru 0x573b0b (0xda2c bytes) 000370[0] 1780987735.375344 write 0x67e05a thru 0x68c45f (0xe406 bytes) 000371[0] 1780987735.380325 read 0x63f346 thru 0x64e581 (0xf23c bytes) 000372[1] 1780987735.387528 mapwrite 0x0e0043 thru 0x0e7b2a (0x7ae8 bytes) 000374[0] 1780987735.511051 write 0x9f91d4 thru 0x9fffff (0x6e2c bytes) 000377[1] 1780987735.572519 mapwrite 0x4f8f20 thru 0x507aa8 (0xeb89 bytes) 000378[1] 1780987735.705240 write 0x091554 thru 0x098a27 (0x74d4 bytes) 000380[0] 1780987735.792371 mapread 0x5546c0 thru 0x561f0a (0xd84b bytes) 000382[0] 1780987735.963888 write 0x258ca7 thru 0x26206c (0x93c6 bytes) 000386[1] 1780987736.027706 read 0x0186a6 thru 0x01b2df (0x2c3a bytes) 000387[0] 1780987736.091021 trunc from 0xa00000 to 0x66ff7e (0x390081 bytes) 000388[0] 1780987736.182836 read 0x2ff69a thru 0x30071b (0x1082 bytes) 000390[0] 1780987736.219099 write 0x7e489c thru 0x7e7a3f (0x31a4 bytes) 000391[0] 1780987736.221550 read 0x618f7e thru 0x619052 (0xd5 bytes) 000394[1] 1780987736.277947 mapwrite 0x169de4 thru 0x172b22 (0x8d3f bytes) 000395[1] 1780987736.410555 trunc from 0x7e7a40 to 0x2fc3d5 (0x4eb66a bytes) 000396[0] 1780987736.458986 mapwrite 0x373f38 thru 0x37d292 (0x935b bytes) 000397[1] 1780987736.639188 mapwrite 0x1a4455 thru 0x1a84b4 (0x4060 bytes) 000400[1] 1780987736.882934 trunc from 0x37d293 to 0x3c95f2 (0x4c360 bytes) 000401[0] 1780987736.955102 mapread 0x1869b1 thru 0x192046 (0xb696 bytes) 000402[1] 1780987737.136350 read 0x102633 thru 0x10407d (0x1a4b bytes) 000408[0] 1780987737.309078 mapwrite 0x176d18 thru 0x1855bc (0xe8a5 bytes) 000409[1] 1780987737.478648 read 0x246321 thru 0x24aaf9 (0x47d9 bytes) 000411[1] 1780987737.615098 mapwrite 0x8a4fac thru 0x8a988e (0x48e3 bytes) 000412[1] 1780987737.844490 trunc from 0x8a988f to 0x2281da (0x6816b4 bytes) 000413[1] 1780987737.874013 mapwrite 0x907f81 thru 0x90c38c (0x440c bytes) 000415[0] 1780987738.039753 trunc from 0x90c38d to 0x71a8cf (0x1f1abd bytes) 000416[0] 1780987738.155293 trunc from 0x71a8d0 to 0x20caf8 (0x50ddd7 bytes) 000419[1] 1780987738.216177 write 0x88d6ee thru 0x8986f6 (0xb009 bytes) 000420[0] 1780987738.255268 mapwrite 0x24318f thru 0x243ee7 (0xd59 bytes) 000421[0] 1780987738.438984 trunc from 0x8986f7 to 0x14042f (0x7582c7 bytes) 000422[1] 1780987738.534246 write 0x211d76 thru 0x216ac3 (0x4d4e bytes) 000423[1] 1780987738.575668 trunc from 0x216ac4 to 0x76b4a4 (0x5549e1 bytes) 000424[1] 1780987738.616982 read 0x6890f2 thru 0x68cadf (0x39ee bytes) 000428[0] 1780987738.735800 write 0x61ea57 thru 0x622fcb (0x4575 bytes) 000429[0] 1780987738.841286 trunc from 0x76b4a5 to 0x4d5bfd (0x2958a7 bytes) 000430[0] 1780987738.907885 mapread 0x12c806 thru 0x135375 (0x8b70 bytes) 000431[1] 1780987738.948357 mapread 0x495f3b thru 0x49bb98 (0x5c5e bytes) 000432[1] 1780987739.055804 read 0x22d3c0 thru 0x2390d1 (0xbd12 bytes) 000433[1] 1780987739.094527 mapwrite 0x8a8495 thru 0x8b6410 (0xdf7c bytes) 000434[0] 1780987739.235960 mapwrite 0x33047f thru 0x333830 (0x33b2 bytes) 000435[1] 1780987739.342020 write 0x8ebd08 thru 0x8fa929 (0xec22 bytes) 000437[0] 1780987739.459563 mapwrite 0x8cb6df thru 0x8d77d3 (0xc0f5 bytes) 000438[0] 1780987739.679571 read 0x518d06 thru 0x526ccb (0xdfc6 bytes) 000440[0] 1780987739.743654 trunc from 0x8fa92a to 0x0cd670 (0x82d2b9 bytes) 000441[0] 1780987739.813825 trunc from 0x0cd671 to 0x1544d9 (0x86e69 bytes) 000445[0] 1780987739.885710 mapwrite 0x088940 thru 0x0954fa (0xcbbb bytes) 000446[0] 1780987739.973579 read 0x1339db thru 0x137715 (0x3d3b bytes) 000448[0] 1780987740.060019 trunc from 0x1544da to 0x70a202 (0x5b5d29 bytes) 000450[0] 1780987740.134924 write 0x6ae338 thru 0x6b54d7 (0x71a0 bytes) 000453[0] 1780987740.302345 trunc from 0x70a203 to 0x363764 (0x3a6a9e bytes) 000454[0] 1780987740.374900 mapwrite 0x2e2dc1 thru 0x2ebe62 (0x90a2 bytes) 000456[1] 1780987740.439206 mapwrite 0x3504b3 thru 0x35b47a (0xafc8 bytes) 000457[1] 1780987740.615022 read 0x2b0c5f thru 0x2b3bff (0x2fa1 bytes) 000460[0] 1780987740.784484 trunc from 0x363765 to 0x6eacbe (0x38755a bytes) 000462[0] 1780987740.834527 mapwrite 0x330ea5 thru 0x337486 (0x65e2 bytes) 000465[0] 1780987741.042186 write 0x46caa8 thru 0x479273 (0xc7cc bytes) 000466[1] 1780987741.061793 mapread 0x5972a2 thru 0x59b44e (0x41ad bytes) 000467[0] 1780987741.134697 mapread 0x135b9c thru 0x138011 (0x2476 bytes) 000470[0] 1780987741.174862 write 0x53ad0e thru 0x546789 (0xba7c bytes) 000472[0] 1780987741.203556 write 0x2efca7 thru 0x2f17b1 (0x1b0b bytes) 000478[1] 1780987741.402700 trunc from 0x6eacbf to 0x0bbb39 (0x62f185 bytes) 000479[1] 1780987741.595214 write 0x22c0ed thru 0x231be2 (0x5af6 bytes) 000483[1] 1780987741.677336 mapread 0x0e76d1 thru 0x0eaecf (0x37ff bytes) 000484[0] 1780987741.787719 mapwrite 0x1e6cba thru 0x1f1c71 (0xafb8 bytes) 000486[0] 1780987742.146191 mapread 0x04e770 thru 0x059dfa (0xb68b bytes) 000490[0] 1780987742.369270 read 0x20641a thru 0x2067ae (0x395 bytes) 000491[1] 1780987742.428662 mapwrite 0x28d42c thru 0x29d3d6 (0xffab bytes) 000492[0] 1780987742.711868 mapwrite 0x6c5c3c thru 0x6d283b (0xcc00 bytes) 000493[0] 1780987742.953256 write 0x3824a3 thru 0x38e803 (0xc361 bytes) truncating to largest ever: 0x9815fb 000495[1] 1780987742.984877 trunc from 0x6d283c to 0x9815fa (0x2aedbf bytes) 000497[0] 1780987743.108207 mapwrite 0x1552f2 thru 0x157161 (0x1e70 bytes) 000499[1] 1780987743.214630 read 0x354b70 thru 0x357a9e (0x2f2f bytes) 000500[1] 1780987743.271763 mapwrite 0x10f465 thru 0x118785 (0x9321 bytes) All operations completed A-OK! Chance of close/open is 1 in 50 Seed set to 7743 fd 0: /mnt/lustre/f16a.sanityn fd 1: /mnt/lustre2/f16a.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported 000001[0] 1780987744.455167 mapwrite 0x769000 thru 0x770b5f (0x7b60 bytes) 000002[0] 1780987744.603270 mapwrite 0x98e000 thru 0x9988eb (0xa8ec bytes) 000003[1] 1780987744.739336 read 0x0f2000 thru 0x0feb99 (0xcb9a bytes) 000004[0] 1780987744.808974 mapread 0x090000 thru 0x092070 (0x2071 bytes) truncating to largest ever: 0x3bf0b4 000005[0] 1780987744.854915 trunc from 0x9988ec to 0x3bf0b3 (0x5d9838 bytes) 000011[0] 1780987745.002134 read_OD 0x086000 thru 0x08afff (0x5000 bytes) 000016[0] 1780987745.031131 mapread 0x2b7000 thru 0x2b9742 (0x2743 bytes) 000019[0] 1780987745.067216 mapwrite 0x298000 thru 0x29f15c (0x715d bytes) 000020[0] 1780987745.160727 read_OD 0x268000 thru 0x274fff (0xd000 bytes) 000024[0] 1780987745.201899 write_OD 0x959000 thru 0x95bfff (0x3000 bytes) 000025[0] 1780987745.253560 mapwrite 0x545000 thru 0x548a60 (0x3a61 bytes) 000026[1] 1780987745.460246 write 0x090000 thru 0x091b54 (0x1b55 bytes) 000027[1] 1780987745.511619 write 0x51d000 thru 0x528f46 (0xbf47 bytes) 000030[0] 1780987745.536194 write_OD 0x0d2000 thru 0x0d6fff (0x5000 bytes) 000033[0] 1780987745.601726 write_OD 0x9bd000 thru 0x9c0fff (0x4000 bytes) 000035[1] 1780987745.666485 mapwrite 0x87f000 thru 0x885fc5 (0x6fc6 bytes) 000039[0] 1780987745.842037 read_OD 0x6d6000 thru 0x6dbfff (0x6000 bytes) 000040[0] 1780987745.875984 mapwrite 0x5b2000 thru 0x5bf3e9 (0xd3ea bytes) 000044[1] 1780987746.047209 mapread 0x257000 thru 0x25af26 (0x3f27 bytes) 000047[1] 1780987746.127750 read 0x452000 thru 0x456b73 (0x4b74 bytes) 000048[0] 1780987746.152670 mapwrite 0x205000 thru 0x213b88 (0xeb89 bytes) 000049[1] 1780987746.306935 write 0x0fb000 thru 0x100a0d (0x5a0e bytes) 000050[0] 1780987746.360271 mapwrite 0x7ce000 thru 0x7dacb5 (0xccb6 bytes) truncating to largest ever: 0x571812 000052[0] 1780987746.522252 trunc from 0x9c1000 to 0x571811 (0x44f7ee bytes) truncating to largest ever: 0x5d6833 000053[1] 1780987746.587964 trunc from 0x571812 to 0x5d6832 (0x65021 bytes) 000056[0] 1780987746.740914 trunc from 0x5d6833 to 0x13f0b9 (0x497779 bytes) 000059[0] 1780987746.829752 read_OD 0x10c000 thru 0x10dfff (0x2000 bytes) 000060[1] 1780987746.899218 mapwrite 0x1ef000 thru 0x1f81e5 (0x91e6 bytes) 000061[1] 1780987747.132130 read 0x099000 thru 0x09a170 (0x1171 bytes) 000063[1] 1780987747.218876 trunc from 0x1f81e6 to 0x486b6c (0x28e987 bytes) truncating to largest ever: 0x5dbf45 000064[0] 1780987747.264462 trunc from 0x486b6d to 0x5dbf44 (0x1553d8 bytes) 000065[1] 1780987747.328381 mapread 0x443000 thru 0x451a31 (0xea32 bytes) 000067[1] 1780987747.433812 read 0x1ab000 thru 0x1ab5b1 (0x5b2 bytes) 000068[1] 1780987747.447085 read 0x490000 thru 0x49b4cc (0xb4cd bytes) 000069[1] 1780987747.448255 trunc from 0x5dbf45 to 0x21c35e (0x3bfbe6 bytes) 000072[1] 1780987747.536312 mapwrite 0x819000 thru 0x821cf2 (0x8cf3 bytes) truncating to largest ever: 0x8adae1 000073[1] 1780987747.684300 trunc from 0x821cf3 to 0x8adae0 (0x8bdee bytes) 000074[1] 1780987747.699954 mapread 0x3b0000 thru 0x3bdeaa (0xdeab bytes) 000076[0] 1780987747.796283 write_OD 0x4c8000 thru 0x4cbfff (0x4000 bytes) 000077[1] 1780987747.876539 mapread 0x5cc000 thru 0x5dae6f (0xee70 bytes) 000078[1] 1780987747.922960 mapwrite 0x934000 thru 0x943ca5 (0xfca6 bytes) 000081[0] 1780987748.193307 read_OD 0x15f000 thru 0x168fff (0xa000 bytes) 000084[1] 1780987748.245871 mapwrite 0x08e000 thru 0x08eca2 (0xca3 bytes) 000087[1] 1780987748.327916 read 0x6e4000 thru 0x6f3ed4 (0xfed5 bytes) 000088[1] 1780987748.355753 read 0x4a0000 thru 0x4ad489 (0xd48a bytes) 000090[0] 1780987748.393898 mapread 0x436000 thru 0x444826 (0xe827 bytes) 000092[1] 1780987748.486784 write 0x807000 thru 0x813507 (0xc508 bytes) 000093[0] 1780987748.539629 mapread 0x7bb000 thru 0x7c1f71 (0x6f72 bytes) 000094[0] 1780987748.659489 mapread 0x2f5000 thru 0x302a27 (0xda28 bytes) 000095[0] 1780987748.716733 mapwrite 0x253000 thru 0x253068 (0x69 bytes) 000098[0] 1780987748.817643 mapwrite 0x3ae000 thru 0x3b6b4d (0x8b4e bytes) 000099[0] 1780987748.904899 mapwrite 0x389000 thru 0x38c211 (0x3212 bytes) 000100[0] 1780987748.985090 write_OD 0x5c2000 thru 0x5cbfff (0xa000 bytes) 000102[0] 1780987749.043260 mapwrite 0x084000 thru 0x08e727 (0xa728 bytes) 000103[1] 1780987749.168154 read 0x3ce000 thru 0x3d4e8e (0x6e8f bytes) 000104[1] 1780987749.192679 trunc from 0x943ca6 to 0x172515 (0x7d1790 bytes) 000105[0] 1780987749.225192 trunc from 0x172516 to 0x600e5f (0x48e94a bytes) 000108[0] 1780987749.297878 mapwrite 0x699000 thru 0x6a19cc (0x89cd bytes) truncating to largest ever: 0x8e7285 000109[1] 1780987749.404732 trunc from 0x6a19cd to 0x8e7284 (0x2458b8 bytes) 000110[1] 1780987749.499609 mapread 0x052000 thru 0x060c96 (0xec97 bytes) 000111[1] 1780987749.569760 mapread 0x228000 thru 0x23091f (0x8920 bytes) 000113[1] 1780987749.608162 mapread 0x4a8000 thru 0x4acd96 (0x4d97 bytes) 000114[1] 1780987749.660257 write 0x9b7000 thru 0x9bb174 (0x4175 bytes) 000116[1] 1780987749.680780 trunc from 0x9bb175 to 0x75032a (0x26ae4a bytes) 000118[0] 1780987749.702269 write_OD 0x7bb000 thru 0x7bcfff (0x2000 bytes) 000119[1] 1780987749.794749 write 0x763000 thru 0x76806f (0x5070 bytes) 000120[1] 1780987749.835107 read 0x391000 thru 0x39c7ee (0xb7ef bytes) 000122[0] 1780987749.877917 mapread 0x43a000 thru 0x441447 (0x7448 bytes) 000124[0] 1780987749.943644 mapread 0x388000 thru 0x38a95e (0x295f bytes) 000125[0] 1780987749.954797 mapread 0x4ce000 thru 0x4d1dde (0x3ddf bytes) 000126[1] 1780987749.996941 mapwrite 0x3a4000 thru 0x3af266 (0xb267 bytes) 000128[0] 1780987750.161798 mapwrite 0x84d000 thru 0x84fb59 (0x2b5a bytes) 000131[0] 1780987750.342155 mapwrite 0x037000 thru 0x046056 (0xf057 bytes) 000132[1] 1780987750.508665 mapread 0x719000 thru 0x7211c5 (0x81c6 bytes) 000134[1] 1780987750.611568 trunc from 0x84fb5a to 0x367f03 (0x4e7c56 bytes) 000135[1] 1780987750.685786 mapwrite 0x63d000 thru 0x63e171 (0x1172 bytes) 000138[0] 1780987750.830630 mapread 0x413000 thru 0x41f5c9 (0xc5ca bytes) 000139[1] 1780987750.881599 mapread 0x53d000 thru 0x53d66e (0x66f bytes) 000143[0] 1780987750.887332 trunc from 0x63e172 to 0x27d645 (0x3c0b2c bytes) 000145[1] 1780987750.960373 trunc from 0x27d646 to 0x3e7d0a (0x16a6c5 bytes) 000147[0] 1780987751.031543 mapwrite 0x12f000 thru 0x12f07e (0x7f bytes) 000149[1] 1780987751.145415 mapread 0x0c1000 thru 0x0cda5e (0xca5f bytes) 000151[1] 1780987751.268839 read 0x22d000 thru 0x234f3e (0x7f3f bytes) 000152[0] 1780987751.399498 trunc from 0x3e7d0b to 0x767a2b (0x37fd21 bytes) 000157[0] 1780987751.563063 mapread 0x2ae000 thru 0x2ba776 (0xc777 bytes) 000163[1] 1780987751.758426 read 0x652000 thru 0x656bae (0x4baf bytes) 000165[0] 1780987751.787438 read_OD 0x310000 thru 0x318fff (0x9000 bytes) 000166[0] 1780987751.803581 write_OD 0x138000 thru 0x140fff (0x9000 bytes) 000167[0] 1780987751.971967 read_OD 0x432000 thru 0x435fff (0x4000 bytes) 000173[1] 1780987752.004503 write 0x8ad000 thru 0x8b2db8 (0x5db9 bytes) 000177[0] 1780987752.094219 trunc from 0x8b2db9 to 0x11ec14 (0x7941a4 bytes) 000186[0] 1780987752.257671 mapread 0x0e9000 thru 0x0ebacf (0x2ad0 bytes) 000187[1] 1780987752.323725 read 0x00f000 thru 0x015739 (0x673a bytes) 000188[0] 1780987752.414153 read_OD 0x09d000 thru 0x09ffff (0x3000 bytes) 000190[0] 1780987752.426047 mapread 0x118000 thru 0x11e71d (0x671e bytes) 000191[0] 1780987752.463956 read_OD 0x0dd000 thru 0x0e6fff (0xa000 bytes) 000193[1] 1780987752.478363 mapread 0x082000 thru 0x0908ea (0xe8eb bytes) 000194[0] 1780987752.479492 read_OD 0x0ec000 thru 0x0f3fff (0x8000 bytes) 000196[1] 1780987752.496805 read 0x049000 thru 0x057240 (0xe241 bytes) 000200[0] 1780987752.498278 trunc from 0x11ec15 to 0x0fb0d8 (0x23b3c bytes) 000201[1] 1780987752.633565 write 0x79d000 thru 0x7a4e63 (0x7e64 bytes) 000202[0] 1780987752.658681 mapwrite 0x04e000 thru 0x05d07c (0xf07d bytes) 000205[0] 1780987752.943757 mapwrite 0x0de000 thru 0x0e0c7e (0x2c7f bytes) truncating to largest ever: 0x8ebfb1 000209[0] 1780987753.032489 trunc from 0x7a4e64 to 0x8ebfb0 (0x14714d bytes) 000210[1] 1780987753.057250 mapwrite 0x830000 thru 0x83762b (0x762c bytes) 000211[0] 1780987753.321161 write_OD 0x0f1000 thru 0x0f1fff (0x1000 bytes) 000215[1] 1780987753.445577 write 00000000 thru 0x008265 (0x8266 bytes) 000217[0] 1780987753.520736 mapwrite 0x7dc000 thru 0x7eb500 (0xf501 bytes) 000218[0] 1780987753.869185 trunc from 0x8ebfb1 to 0x4bb825 (0x43078b bytes) 000223[1] 1780987754.055800 read 0x428000 thru 0x42a6e0 (0x26e1 bytes) 000225[1] 1780987754.092955 write 0x2b8000 thru 0x2ba654 (0x2655 bytes) 000226[1] 1780987754.113565 mapread 0x3ec000 thru 0x3ef9e4 (0x39e5 bytes) 000227[1] 1780987754.139078 trunc from 0x4bb826 to 0x2e48f7 (0x1d6f2e bytes) 000230[0] 1780987754.164968 write_OD 0x977000 thru 0x977fff (0x1000 bytes) 000231[0] 1780987754.235870 mapwrite 0x735000 thru 0x7365b4 (0x15b5 bytes) 000233[1] 1780987754.332900 mapwrite 0x5c9000 thru 0x5c962e (0x62f bytes) 000235[0] 1780987754.437010 trunc from 0x978000 to 0x7f3999 (0x184666 bytes) 000237[1] 1780987754.527200 mapwrite 0x54a000 thru 0x552682 (0x8683 bytes) 000239[1] 1780987754.643621 mapread 0x121000 thru 0x1244a5 (0x34a6 bytes) 000242[1] 1780987754.699206 mapwrite 0x7d7000 thru 0x7e4e7a (0xde7b bytes) 000243[0] 1780987754.841965 trunc from 0x7f399a to 0x10cc2f (0x6e6d6a bytes) 000244[1] 1780987754.898627 mapread 0x09d000 thru 0x0a936d (0xc36e bytes) 000245[1] 1780987754.982942 mapwrite 0x15d000 thru 0x1648bb (0x78bc bytes) 000247[0] 1780987755.128881 mapwrite 0x6ab000 thru 0x6b196c (0x696d bytes) 000252[1] 1780987755.328765 mapread 0x10c000 thru 0x118ca6 (0xcca7 bytes) 000254[1] 1780987755.380914 write 0x15b000 thru 0x167f9e (0xcf9f bytes) 000255[1] 1780987755.406039 trunc from 0x6b196d to 0x643348 (0x6e624 bytes) 000257[0] 1780987755.448990 write_OD 0x826000 thru 0x82afff (0x5000 bytes) 000258[0] 1780987755.508979 mapwrite 0x492000 thru 0x4959b3 (0x39b4 bytes) 000259[1] 1780987755.607910 mapwrite 0x246000 thru 0x24ca85 (0x6a86 bytes) 000263[1] 1780987755.798650 trunc from 0x82b000 to 0x32a28b (0x500d74 bytes) truncating to largest ever: 0x9e245f 000268[1] 1780987755.882501 trunc from 0x32a28c to 0x9e245e (0x6b81d3 bytes) 000272[0] 1780987755.925975 mapread 0x7de000 thru 0x7ec8ba (0xe8bb bytes) 000275[0] 1780987756.034993 write_OD 0x456000 thru 0x459fff (0x4000 bytes) 000276[0] 1780987756.093988 write_OD 0x59a000 thru 0x59cfff (0x3000 bytes) 000278[1] 1780987756.151900 read 0x38b000 thru 0x39a8f2 (0xf8f3 bytes) 000280[1] 1780987756.165045 mapwrite 0x537000 thru 0x541907 (0xa908 bytes) 000283[0] 1780987756.284974 mapread 0x004000 thru 0x00a84b (0x684c bytes) 000284[0] 1780987756.336097 mapwrite 0x203000 thru 0x206976 (0x3977 bytes) 000285[1] 1780987756.498990 mapread 0x724000 thru 0x733c49 (0xfc4a bytes) 000286[0] 1780987756.575003 mapwrite 0x117000 thru 0x1187ec (0x17ed bytes) 000287[0] 1780987756.714933 trunc from 0x9e245f to 0x8b2bac (0x12f8b2 bytes) 000291[1] 1780987756.845832 trunc from 0x8b2bad to 0x3f6b85 (0x4bc027 bytes) 000293[1] 1780987756.989742 mapwrite 0x204000 thru 0x20e398 (0xa399 bytes) 000295[0] 1780987757.134883 write_OD 0x0e3000 thru 0x0e5fff (0x3000 bytes) 000296[1] 1780987757.256725 read 0x369000 thru 0x36a52d (0x152e bytes) 000297[1] 1780987757.351163 read 0x23f000 thru 0x241cc6 (0x2cc7 bytes) 000303[1] 1780987757.370753 trunc from 0x3f6b86 to 0x888302 (0x49177d bytes) 000304[0] 1780987757.431454 trunc from 0x888303 to 0x068eef (0x81f413 bytes) 000306[1] 1780987757.487136 read 0x032000 thru 0x03ad81 (0x8d82 bytes) 000307[0] 1780987757.519978 trunc from 0x068ef0 to 0x038cd3 (0x3021c bytes) 000310[0] 1780987757.579809 mapwrite 0x40c000 thru 0x40c35a (0x35b bytes) 000312[0] 1780987757.664024 mapread 0x103000 thru 0x1091a0 (0x61a1 bytes) 000314[1] 1780987757.699097 mapwrite 0x4ff000 thru 0x503b7b (0x4b7c bytes) 000316[0] 1780987757.872081 mapwrite 0x058000 thru 0x0623e6 (0xa3e7 bytes) 000319[1] 1780987758.018775 read 0x348000 thru 0x353181 (0xb182 bytes) 000325[1] 1780987758.050201 trunc from 0x503b7c to 0x936bff (0x433084 bytes) 000328[0] 1780987758.106675 trunc from 0x936c00 to 0x549086 (0x3edb79 bytes) 000329[0] 1780987758.161809 mapwrite 0x600000 thru 0x608e69 (0x8e6a bytes) 000331[1] 1780987758.312591 mapread 0x268000 thru 0x26d827 (0x5828 bytes) 000335[1] 1780987758.357096 write 0x2f7000 thru 0x3004e3 (0x94e4 bytes) 000337[0] 1780987758.380580 write_OD 0x2d2000 thru 0x2dffff (0xe000 bytes) 000339[1] 1780987758.507691 write 0x49b000 thru 0x4aa195 (0xf196 bytes) 000341[0] 1780987758.572830 read_OD 0x52f000 thru 0x533fff (0x5000 bytes) 000343[1] 1780987758.642120 write 0x126000 thru 0x12bffe (0x5fff bytes) 000344[0] 1780987758.691780 mapwrite 0x9a9000 thru 0x9ad0e4 (0x40e5 bytes) 000346[1] 1780987758.937060 write 0x207000 thru 0x20f36b (0x836c bytes) 000350[1] 1780987759.013343 mapread 0x166000 thru 0x1743f8 (0xe3f9 bytes) 000352[0] 1780987759.084510 trunc from 0x9ad0e5 to 0x45e0b3 (0x54f031 bytes) 000357[0] 1780987759.173920 mapread 0x1b5000 thru 0x1c39a1 (0xe9a2 bytes) 000359[0] 1780987759.237841 mapwrite 0x61f000 thru 0x623900 (0x4901 bytes) 000360[1] 1780987759.327053 mapwrite 0x16d000 thru 0x17c14e (0xf14f bytes) 000361[1] 1780987759.446131 mapread 0x3f8000 thru 0x3fd6d1 (0x56d2 bytes) 000363[1] 1780987759.507077 read 0x46f000 thru 0x4724bd (0x34be bytes) 000365[1] 1780987759.517462 mapwrite 0x52d000 thru 0x53a99b (0xd99c bytes) 000366[0] 1780987759.646033 read_OD 0x233000 thru 0x240fff (0xe000 bytes) 000368[0] 1780987759.681026 read_OD 0x0b0000 thru 0x0b4fff (0x5000 bytes) 000370[1] 1780987759.693707 trunc from 0x623901 to 0x134ccd (0x4eec33 bytes) 000371[1] 1780987759.744901 trunc from 0x134cce to 0x41ae19 (0x2e614c bytes) 000373[0] 1780987759.812492 mapwrite 0x025000 thru 0x030a38 (0xba39 bytes) 000375[1] 1780987760.001873 mapread 0x04e000 thru 0x05485a (0x685b bytes) 000376[1] 1780987760.109332 mapwrite 0x22b000 thru 0x234494 (0x9495 bytes) 000377[1] 1780987760.286439 mapread 0x20b000 thru 0x2199c5 (0xe9c6 bytes) 000378[0] 1780987760.327778 trunc from 0x41ae1a to 0x5892d0 (0x16e4b7 bytes) 000379[1] 1780987760.399962 trunc from 0x5892d1 to 0x61d376 (0x940a6 bytes) 000381[1] 1780987760.483928 write 0x7c9000 thru 0x7cfba7 (0x6ba8 bytes) 000383[0] 1780987760.491335 write_OD 0x118000 thru 0x11bfff (0x4000 bytes) 000384[1] 1780987760.592740 write 0x808000 thru 0x8141e1 (0xc1e2 bytes) 000385[1] 1780987760.613311 read 0x6ac000 thru 0x6b846b (0xc46c bytes) 000386[1] 1780987760.666349 mapwrite 0x76f000 thru 0x771549 (0x254a bytes) 000388[0] 1780987760.832255 mapwrite 0x508000 thru 0x516163 (0xe164 bytes) 000392[1] 1780987761.039348 write 0x90e000 thru 0x918f04 (0xaf05 bytes) 000394[0] 1780987761.052782 mapwrite 0x519000 thru 0x51a3eb (0x13ec bytes) 000395[1] 1780987761.166601 write 0x1fe000 thru 0x205a57 (0x7a58 bytes) 000398[1] 1780987761.224717 trunc from 0x918f05 to 0x7d175d (0x1477a7 bytes) 000399[0] 1780987761.246712 write_OD 0x9fe000 thru 0x9fefff (0x1000 bytes) 000400[0] 1780987761.333838 trunc from 0x9ff000 to 0x18586a (0x879795 bytes) 000401[1] 1780987761.437203 trunc from 0x18586b to 0x5d4854 (0x44efea bytes) 000403[1] 1780987761.516707 read 0x2af000 thru 0x2b17b8 (0x27b9 bytes) 000404[0] 1780987761.525885 read_OD 0x158000 thru 0x163fff (0xc000 bytes) 000405[0] 1780987761.581786 mapwrite 0x2f4000 thru 0x2f736a (0x336b bytes) 000409[1] 1780987761.802963 mapwrite 0x42b000 thru 0x430d93 (0x5d94 bytes) 000411[1] 1780987762.017959 write 0x208000 thru 0x215073 (0xd074 bytes) 000415[1] 1780987762.112856 read 0x188000 thru 0x196067 (0xe068 bytes) 000421[1] 1780987762.221067 read 0x49c000 thru 0x4a4fb5 (0x8fb6 bytes) 000422[1] 1780987762.252318 mapread 0x3cb000 thru 0x3d960c (0xe60d bytes) 000424[1] 1780987762.306929 mapwrite 0x4eb000 thru 0x4f037c (0x537d bytes) 000426[0] 1780987762.395722 write_OD 0x380000 thru 0x388fff (0x9000 bytes) 000427[1] 1780987762.488865 mapread 0x58b000 thru 0x58f6da (0x46db bytes) 000428[1] 1780987762.514683 trunc from 0x5d4855 to 0x902af2 (0x32e29e bytes) 000429[0] 1780987762.562817 mapwrite 0x0d1000 thru 0x0d1995 (0x996 bytes) 000430[1] 1780987762.640317 read 0x88c000 thru 0x88f0dd (0x30de bytes) 000432[0] 1780987762.667105 read_OD 0x1a4000 thru 0x1aefff (0xb000 bytes) 000434[0] 1780987762.677126 write_OD 0x13a000 thru 0x13ffff (0x6000 bytes) 000435[0] 1780987762.764164 trunc from 0x902af3 to 0x591e16 (0x370cdc bytes) 000438[1] 1780987762.841945 write 0x363000 thru 0x365fe6 (0x2fe7 bytes) 000439[0] 1780987762.859437 read_OD 0x1ef000 thru 0x1f2fff (0x4000 bytes) 000441[0] 1780987762.903013 write_OD 0x0d8000 thru 0x0e4fff (0xd000 bytes) 000442[1] 1780987762.995330 mapwrite 0x27a000 thru 0x281e3b (0x7e3c bytes) 000443[0] 1780987763.203955 mapwrite 0x963000 thru 0x96d0da (0xa0db bytes) 000444[0] 1780987763.429741 mapwrite 0x16d000 thru 0x16ffd6 (0x2fd7 bytes) 000445[1] 1780987763.561014 mapwrite 0x648000 thru 0x64a30e (0x230f bytes) 000446[1] 1780987763.679348 mapwrite 0x6cf000 thru 0x6d28de (0x38df bytes) 000447[0] 1780987763.801645 mapread 0x476000 thru 0x4819bf (0xb9c0 bytes) 000448[1] 1780987763.869884 trunc from 0x96d0db to 0x79b753 (0x1d1987 bytes) 000449[1] 1780987763.949136 mapwrite 0x505000 thru 0x50fed2 (0xaed3 bytes) 000450[1] 1780987764.066327 trunc from 0x79b754 to 0x3369d2 (0x464d81 bytes) 000452[0] 1780987764.134781 trunc from 0x3369d3 to 0x357d4d (0x2137b bytes) 000455[0] 1780987764.194980 trunc from 0x357d4e to 0x6afb32 (0x357de5 bytes) 000456[1] 1780987764.243529 mapread 0x54b000 thru 0x54e81a (0x381b bytes) 000457[1] 1780987764.302983 trunc from 0x6afb33 to 0x0272bd (0x688875 bytes) 000458[1] 1780987764.350125 mapread 0x01a000 thru 0x0254e4 (0xb4e5 bytes) 000461[1] 1780987764.426376 mapwrite 0x850000 thru 0x85d84b (0xd84c bytes) 000462[0] 1780987764.614626 read_OD 0x33e000 thru 0x340fff (0x3000 bytes) 000463[1] 1780987764.670897 mapwrite 0x2be000 thru 0x2cc5df (0xe5e0 bytes) 000466[1] 1780987764.933204 mapread 0x3e4000 thru 0x3f0586 (0xc587 bytes) 000470[1] 1780987765.049292 read 0x315000 thru 0x324bc0 (0xfbc1 bytes) 000472[1] 1780987765.077989 mapwrite 0x0c9000 thru 0x0d749c (0xe49d bytes) 000474[0] 1780987765.183111 mapread 0x285000 thru 0x292b28 (0xdb29 bytes) 000476[0] 1780987765.284869 read_OD 0x09b000 thru 0x0a3fff (0x9000 bytes) 000482[0] 1780987765.305138 write_OD 0x30b000 thru 0x30bfff (0x1000 bytes) 000483[1] 1780987765.436281 read 0x602000 thru 0x606d44 (0x4d45 bytes) 000485[0] 1780987765.454456 trunc from 0x85d84c to 0x7d9f73 (0x838d8 bytes) 000486[1] 1780987765.498459 write 0x85e000 thru 0x869ba3 (0xbba4 bytes) 000488[0] 1780987765.537447 mapwrite 0x488000 thru 0x49238c (0xa38d bytes) 000489[0] 1780987765.683562 read_OD 0x071000 thru 0x07ffff (0xf000 bytes) 000490[1] 1780987765.707984 mapread 0x3fb000 thru 0x3fc4fb (0x14fc bytes) 000491[1] 1780987765.724785 mapwrite 0x49c000 thru 0x4aab42 (0xeb43 bytes) 000495[1] 1780987765.853463 write 0x329000 thru 0x337401 (0xe402 bytes) 000499[1] 1780987765.923944 read 0x79a000 thru 0x79ee42 (0x4e43 bytes) 000500[1] 1780987765.959060 write 0x390000 thru 0x39f100 (0xf101 bytes) All operations completed A-OK! PASS 16a (63s) == sanityn test 16b: 500 iterations of dual-mount fsx at small size ========================================================== 02:49:32 (1780987772) fallocate on zfs doesn't consume space fallocate not supported 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.341345 s, 24.6 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.291757 s, 28.8 MB/s Chance of close/open is 1 in 50 Seed set to 7775 fd 0: /mnt/lustre/f16b.sanityn fd 1: /mnt/lustre2/f16b.sanityn fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read truncating to largest ever: 0x5938 000002[0] 1780987776.437796 trunc from 00000000 to 0x005937 (0x5938 bytes) 000003[0] 1780987776.479050 write 0x3b7f61 thru 0x3b866e (0x70e bytes) 000004[1] 1780987776.486787 mapread 0x2e1e4e thru 0x2e31b2 (0x1365 bytes) 000007[0] 1780987776.568351 read 0x33d353 thru 0x33f0db (0x1d89 bytes) 000008[0] 1780987776.575712 mapwrite 0x35bd7d thru 0x35d06c (0x12f0 bytes) 000012[1] 1780987776.896963 mapwrite 0x01b28c thru 0x01b2a9 (0x1e bytes) truncating to largest ever: 0x37d3a1 000013[1] 1780987777.082446 trunc from 0x3b866f to 0x37d3a0 (0x3b2ce bytes) 000014[1] 1780987777.173184 trunc from 0x37d3a1 to 0x068b1c (0x314884 bytes) 000016[1] 1780987777.204915 write 0x84ec1a thru 0x84fa5b (0xe42 bytes) 000017[1] 1780987777.252093 read 0x1c6731 thru 0x1c7271 (0xb41 bytes) 000018[1] 1780987777.260914 read 0x4eadcb thru 0x4eb687 (0x8bd bytes) 000019[0] 1780987777.269495 mapwrite 0x5ce5ce thru 0x5cfab7 (0x14ea bytes) 000021[0] 1780987777.451593 mapread 0x1b645c thru 0x1b6af4 (0x699 bytes) 000022[0] 1780987777.488675 mapwrite 0x0d138c thru 0x0d1c36 (0x8ab bytes) truncating to largest ever: 0x53ce21 000024[0] 1780987777.579023 trunc from 0x84fa5c to 0x53ce20 (0x312c3b bytes) truncating to largest ever: 0x946907 000026[1] 1780987777.644525 trunc from 0x53ce21 to 0x946906 (0x409ae6 bytes) 000029[1] 1780987777.759315 read 0x67fcab thru 0x681a72 (0x1dc8 bytes) 000030[1] 1780987777.791098 read 0x25db55 thru 0x25ed4b (0x11f7 bytes) 000031[0] 1780987777.819703 trunc from 0x946907 to 0x617db8 (0x32eb4e bytes) 000032[0] 1780987777.893350 mapwrite 0x36eb0b thru 0x370544 (0x1a3a bytes) 000033[1] 1780987778.051627 mapwrite 0x65e804 thru 0x65f200 (0x9fd bytes) 000037[1] 1780987778.322256 mapread 0x4f7f08 thru 0x4f89e2 (0xadb bytes) 000040[0] 1780987778.386970 mapwrite 0x4e000c thru 0x4e002b (0x20 bytes) 000041[0] 1780987778.510999 mapwrite 0x938efe thru 0x939eaa (0xfad bytes) 000042[1] 1780987778.656858 write 0x44c883 thru 0x44cede (0x65c bytes) 000043[0] 1780987778.698848 read 0x1ba35a thru 0x1bc027 (0x1cce bytes) 000047[0] 1780987778.835045 trunc from 0x939eab to 0x2c935e (0x670b4c bytes) 000048[1] 1780987778.879032 read 0x1f9469 thru 0x1faab2 (0x164a bytes) 000049[0] 1780987778.919508 write 0x0eb4a2 thru 0x0ed003 (0x1b62 bytes) 000055[0] 1780987779.009911 trunc from 0x2c935f to 0x774208 (0x4aaeaa bytes) 000059[0] 1780987779.179003 read 0x14fbf2 thru 0x151355 (0x1764 bytes) 000060[0] 1780987779.221481 write 0x4d6b4a thru 0x4d6db5 (0x26c bytes) 000061[0] 1780987779.260027 mapread 0x727cc0 thru 0x729c5a (0x1f9b bytes) 000062[0] 1780987779.299965 write 0x4c0a0a thru 0x4c170c (0xd03 bytes) 000064[1] 1780987779.359829 write 0x9eaaf2 thru 0x9ec33f (0x184e bytes) 000066[1] 1780987779.460742 write 0x6857ff thru 0x686d3c (0x153e bytes) 000070[0] 1780987779.572638 mapwrite 0x4962c6 thru 0x4973d9 (0x1114 bytes) 000071[1] 1780987779.674571 write 0x5bd1fd thru 0x5be73c (0x1540 bytes) 000072[1] 1780987779.700356 read 0x770fab thru 0x772f60 (0x1fb6 bytes) 000073[1] 1780987779.741919 trunc from 0x9ec340 to 0x5860af (0x466290 bytes) 000074[1] 1780987779.804918 write 0x3ad382 thru 0x3ad6c7 (0x346 bytes) 000078[0] 1780987779.907702 trunc from 0x5860b0 to 0x5f2bd0 (0x6cb21 bytes) 000081[1] 1780987779.985437 mapread 0x0d7de4 thru 0x0d943f (0x165c bytes) 000086[0] 1780987780.109728 mapread 0x194372 thru 0x195c18 (0x18a7 bytes) 000087[1] 1780987780.124898 mapwrite 0x222795 thru 0x224780 (0x1fec bytes) 000088[0] 1780987780.229928 write 0x60001c thru 0x6012f9 (0x12de bytes) 000089[1] 1780987780.255907 read 0x55c3ca thru 0x55cca6 (0x8dd bytes) 000090[1] 1780987780.316954 mapwrite 0x3895cd thru 0x389e88 (0x8bc bytes) 000091[0] 1780987780.375330 mapwrite 0x0bef4a thru 0x0c0ba5 (0x1c5c bytes) 000092[1] 1780987780.470341 read 0x522c6b thru 0x524360 (0x16f6 bytes) 000094[0] 1780987780.492529 mapwrite 0x879bb8 thru 0x87a59b (0x9e4 bytes) 000095[1] 1780987780.591508 mapread 0x7f68cb thru 0x7f86dc (0x1e12 bytes) 000096[1] 1780987780.671077 read 0x868a75 thru 0x8692e0 (0x86c bytes) 000097[1] 1780987780.703230 mapread 0x080edd thru 0x08137a (0x49e bytes) 000100[1] 1780987780.724801 trunc from 0x87a59c to 0x8ce4a5 (0x53f0a bytes) 000101[1] 1780987780.871773 write 0x06fe9b thru 0x0702da (0x440 bytes) 000102[0] 1780987780.931267 mapwrite 0x2f6fff thru 0x2f7c6c (0xc6e bytes) 000104[0] 1780987781.189796 trunc from 0x8ce4a6 to 0x4d5813 (0x3f8c92 bytes) 000105[1] 1780987781.259122 write 0x87dbe0 thru 0x87e173 (0x594 bytes) 000106[0] 1780987781.300443 write 0x4e9d03 thru 0x4ebaed (0x1deb bytes) 000107[1] 1780987781.326921 mapwrite 0x1c55a9 thru 0x1c6ff3 (0x1a4b bytes) 000109[0] 1780987781.523195 mapread 0x03cd2f thru 0x03da4b (0xd1d bytes) 000112[0] 1780987781.661667 read 0x86ff15 thru 0x8713a4 (0x1490 bytes) 000115[1] 1780987781.772962 trunc from 0x87e174 to 0x2593a0 (0x624dd3 bytes) 000116[0] 1780987781.946817 mapread 0x09a8c6 thru 0x09b147 (0x882 bytes) 000117[0] 1780987782.000005 write 0x0210dc thru 0x022d22 (0x1c47 bytes) 000119[0] 1780987782.105340 trunc from 0x2593a1 to 0x4d4ade (0x27b73e bytes) 000120[0] 1780987782.154731 trunc from 0x4d4adf to 0x3c01bc (0x114922 bytes) 000121[0] 1780987782.185711 write 0x89cf88 thru 0x89ec11 (0x1c8a bytes) 000124[1] 1780987782.193766 trunc from 0x89ec12 to 0x5e9fb5 (0x2b4c5c bytes) 000126[1] 1780987782.317875 mapwrite 0x4f9106 thru 0x4fa25a (0x1155 bytes) 000128[0] 1780987782.513022 write 0x08e1fe thru 0x08ef2b (0xd2e bytes) 000135[0] 1780987782.708896 read 0x066950 thru 0x0688ac (0x1f5d bytes) 000137[1] 1780987782.794497 write 0x298565 thru 0x29a269 (0x1d05 bytes) 000138[1] 1780987782.838058 write 0x7dc31c thru 0x7de23a (0x1f1f bytes) truncating to largest ever: 0x97ca92 000141[1] 1780987782.876896 trunc from 0x7de23b to 0x97ca91 (0x19e857 bytes) 000143[0] 1780987782.976352 write 0x633a7b thru 0x6353f9 (0x197f bytes) 000144[1] 1780987783.035307 mapwrite 0x3adbfc thru 0x3aee93 (0x1298 bytes) 000145[0] 1780987783.200959 trunc from 0x97ca92 to 0x3929b3 (0x5ea0de bytes) 000146[1] 1780987783.302285 mapwrite 0x2c62a5 thru 0x2c73f9 (0x1155 bytes) 000149[0] 1780987783.633830 trunc from 0x3929b4 to 0x6c942e (0x336a7b bytes) 000153[0] 1780987783.800989 mapread 0x5230e4 thru 0x523507 (0x424 bytes) 000155[0] 1780987783.900538 mapread 0x388874 thru 0x38a3b2 (0x1b3f bytes) 000157[1] 1780987784.019957 trunc from 0x6c942f to 0x813a52 (0x14a624 bytes) 000158[0] 1780987784.099081 mapread 0x376057 thru 0x376e5a (0xe04 bytes) 000159[0] 1780987784.184213 write 0x1525c8 thru 0x154147 (0x1b80 bytes) 000160[0] 1780987784.292086 write 0x013f48 thru 0x014895 (0x94e bytes) 000161[1] 1780987784.354158 mapread 0x450775 thru 0x450e89 (0x715 bytes) 000162[0] 1780987784.399625 trunc from 0x813a53 to 0x1fe8bc (0x615196 bytes) 000166[1] 1780987784.446543 write 0x5f1601 thru 0x5f210a (0xb0a bytes) 000172[1] 1780987784.627225 mapread 0x1669c4 thru 0x16879e (0x1ddb bytes) 000173[1] 1780987784.805276 read 0x0b7acd thru 0x0b845c (0x990 bytes) 000175[0] 1780987784.874752 mapread 0x0e2370 thru 0x0e436a (0x1ffb bytes) 000176[1] 1780987784.957666 mapwrite 0x617acb thru 0x6187d8 (0xd0e bytes) 000178[0] 1780987785.144942 trunc from 0x6187d9 to 0x8e15d1 (0x2c8df9 bytes) 000179[0] 1780987785.199921 write 0x97bde4 thru 0x97bdfc (0x19 bytes) 000183[0] 1780987785.249325 mapwrite 0x5df336 thru 0x5df3b4 (0x7f bytes) 000185[0] 1780987785.323805 mapwrite 0x19e37c thru 0x19ea66 (0x6eb bytes) 000188[1] 1780987785.468143 write 0x401a08 thru 0x40234e (0x947 bytes) 000192[0] 1780987785.623618 mapwrite 0x64b3cc thru 0x64bc5c (0x891 bytes) 000194[1] 1780987785.782179 mapread 0x1ec846 thru 0x1ee08b (0x1846 bytes) 000196[0] 1780987785.948757 read 0x35734c thru 0x358a8c (0x1741 bytes) 000203[1] 1780987786.113593 mapread 0x1f1714 thru 0x1f35ce (0x1ebb bytes) 000204[1] 1780987786.134614 read 0x3795fd thru 0x37ad86 (0x178a bytes) 000205[1] 1780987786.167933 trunc from 0x97bdfd to 0x8cd5dc (0xae820 bytes) 000207[1] 1780987786.281771 read 0x5569ce thru 0x557a71 (0x10a4 bytes) 000209[0] 1780987786.336172 read 0x562187 thru 0x562b65 (0x9df bytes) 000210[0] 1780987786.377130 read 0x43f6eb thru 0x4414b2 (0x1dc8 bytes) 000211[0] 1780987786.392357 mapread 0x81028b thru 0x81220c (0x1f82 bytes) 000212[0] 1780987786.466578 read 0x31aaa5 thru 0x31bf6b (0x14c7 bytes) 000213[1] 1780987786.472663 mapread 0x117385 thru 0x1186b8 (0x1334 bytes) 000216[0] 1780987786.490318 read 0x639fea thru 0x63a356 (0x36d bytes) 000218[1] 1780987786.498048 mapread 0x0b267a thru 0x0b2bb1 (0x538 bytes) 000221[0] 1780987786.508286 read 0x1b1f79 thru 0x1b3938 (0x19c0 bytes) 000222[1] 1780987786.515986 read 0x0ed707 thru 0x0edcea (0x5e4 bytes) 000224[0] 1780987786.523546 mapwrite 0x0d08e5 thru 0x0d0a61 (0x17d bytes) 000227[1] 1780987786.630049 mapwrite 0x3ba0a1 thru 0x3bb540 (0x14a0 bytes) 000228[0] 1780987786.811847 mapread 0x7df401 thru 0x7e0fb6 (0x1bb6 bytes) 000229[0] 1780987786.913641 mapwrite 0x039bd5 thru 0x03a1ed (0x619 bytes) 000230[0] 1780987787.065256 mapwrite 0x908259 thru 0x90893f (0x6e7 bytes) 000233[1] 1780987787.380925 write 0x94dd21 thru 0x94e29a (0x57a bytes) 000234[0] 1780987787.412044 write 0x1ab394 thru 0x1ac6ff (0x136c bytes) 000235[1] 1780987787.445323 mapwrite 0x9ce833 thru 0x9cfca4 (0x1472 bytes) 000237[1] 1780987787.662068 mapwrite 0x58630f thru 0x5870b3 (0xda5 bytes) 000239[0] 1780987787.840846 mapread 0x8faa72 thru 0x8fc17e (0x170d bytes) 000243[1] 1780987787.953755 mapread 0x159cbb thru 0x15a926 (0xc6c bytes) truncating to largest ever: 0x97f2be 000250[1] 1780987788.215480 trunc from 0x9cfca5 to 0x97f2bd (0x509e7 bytes) 000252[0] 1780987788.318016 mapread 0x83e8a7 thru 0x83eeef (0x649 bytes) 000254[1] 1780987788.388944 read 0x692754 thru 0x69380a (0x10b7 bytes) 000255[0] 1780987788.397979 mapread 0x4bcc5c thru 0x4be6ae (0x1a53 bytes) 000256[1] 1780987788.504369 mapread 0x59fd4a thru 0x5a05ca (0x881 bytes) 000257[1] 1780987788.521844 mapread 0x72444d thru 0x726241 (0x1df5 bytes) 000260[0] 1780987788.558193 trunc from 0x97f2be to 0x8e1eb1 (0x9d40c bytes) 000261[1] 1780987788.725419 write 0x2d1faf thru 0x2d2160 (0x1b2 bytes) 000262[0] 1780987788.791002 trunc from 0x8e1eb2 to 0x611377 (0x2d0b3a bytes) 000266[1] 1780987789.113364 mapwrite 0x2fb1a0 thru 0x2fbb7b (0x9dc bytes) 000267[0] 1780987789.360415 mapread 0x03ec2c thru 0x03edcd (0x1a2 bytes) 000273[1] 1780987789.454969 mapwrite 0x25fd05 thru 0x261806 (0x1b02 bytes) 000274[1] 1780987789.615560 mapwrite 0x60c577 thru 0x60e345 (0x1dcf bytes) 000275[1] 1780987789.772913 write 0x32fe29 thru 0x330577 (0x74f bytes) 000277[1] 1780987789.806586 mapread 0x03ca72 thru 0x03ca87 (0x16 bytes) 000278[1] 1780987789.814704 read 0x492bf9 thru 0x494416 (0x181e bytes) 000279[0] 1780987789.875260 mapwrite 0x42c299 thru 0x42cd51 (0xab9 bytes) 000280[1] 1780987790.046497 mapread 0x4e244d thru 0x4e283e (0x3f2 bytes) 000282[0] 1780987790.155685 read 0x49bd88 thru 0x49c365 (0x5de bytes) 000285[0] 1780987790.190111 write 0x23aba8 thru 0x23b166 (0x5bf bytes) 000288[0] 1780987790.325175 mapwrite 0x8e448e thru 0x8e4f91 (0xb04 bytes) 000290[1] 1780987790.475883 mapwrite 0x220f40 thru 0x222114 (0x11d5 bytes) 000292[0] 1780987790.622819 mapread 0x12958f thru 0x12a3b5 (0xe27 bytes) 000294[0] 1780987790.662145 read 0x527e7c thru 0x527edb (0x60 bytes) 000295[0] 1780987790.667801 trunc from 0x8e4f92 to 0x813615 (0xd197c bytes) 000297[0] 1780987790.704638 trunc from 0x813616 to 0x2bfa36 (0x553bdf bytes) 000300[1] 1780987790.805999 read 0x02c843 thru 0x02de10 (0x15ce bytes) 000301[1] 1780987790.895616 read 0x25bd58 thru 0x25c89e (0xb47 bytes) 000302[1] 1780987790.929358 write 0x517147 thru 0x51820d (0x10c7 bytes) 000303[0] 1780987790.953962 mapwrite 0x9b6a48 thru 0x9b74ea (0xaa3 bytes) 000304[0] 1780987791.214564 mapread 0x7dd531 thru 0x7dd6e0 (0x1b0 bytes) 000308[0] 1780987791.234878 trunc from 0x9b74eb to 0x251fd8 (0x765512 bytes) 000309[1] 1780987791.295885 write 0x428cd4 thru 0x429344 (0x671 bytes) 000311[1] 1780987791.371157 trunc from 0x429345 to 0x40198f (0x279b5 bytes) 000316[1] 1780987791.538178 read 0x050026 thru 0x0503a8 (0x383 bytes) 000318[1] 1780987791.625285 trunc from 0x401990 to 0x7cd146 (0x3cb7b7 bytes) 000320[1] 1780987791.703670 trunc from 0x7cd147 to 0x580fd5 (0x24c171 bytes) 000321[1] 1780987791.730022 write 0x2a1140 thru 0x2a2aea (0x19ab bytes) 000323[1] 1780987791.773344 write 0x64cd40 thru 0x64d08c (0x34d bytes) 000324[1] 1780987791.774852 mapwrite 0x160d18 thru 0x162812 (0x1afb bytes) 000325[1] 1780987791.856986 mapwrite 0x0b0fa2 thru 0x0b1547 (0x5a6 bytes) 000327[1] 1780987791.970477 mapwrite 0x960cc3 thru 0x9623d4 (0x1712 bytes) 000330[0] 1780987792.168070 write 0x6cb977 thru 0x6cc3a5 (0xa2f bytes) 000338[0] 1780987792.433665 mapread 0x0b3b38 thru 0x0b4b66 (0x102f bytes) 000339[0] 1780987792.531997 mapwrite 0x9de200 thru 0x9dfbb0 (0x19b1 bytes) 000340[0] 1780987792.734262 write 0x2f3dc6 thru 0x2f4317 (0x552 bytes) 000345[0] 1780987792.831488 read 0x27a3b9 thru 0x27af0e (0xb56 bytes) 000347[1] 1780987792.872528 read 0x43ff72 thru 0x44141a (0x14a9 bytes) 000351[1] 1780987792.937729 read 0x31dce2 thru 0x31ebc5 (0xee4 bytes) 000353[1] 1780987792.976341 read 0x1b01a4 thru 0x1b160e (0x146b bytes) 000354[0] 1780987792.982503 write 0x32879c thru 0x32966f (0xed4 bytes) 000357[1] 1780987793.124524 write 0x214466 thru 0x2162c1 (0x1e5c bytes) 000359[0] 1780987793.175366 mapwrite 0x02af33 thru 0x02c0e6 (0x11b4 bytes) 000360[1] 1780987793.306769 trunc from 0x9dfbb1 to 0x67ae1b (0x364d95 bytes) 000361[0] 1780987793.423679 mapwrite 0x31cc23 thru 0x31e8a3 (0x1c81 bytes) 000363[1] 1780987793.759308 mapread 0x01786a thru 0x017950 (0xe7 bytes) 000364[0] 1780987793.791354 mapwrite 0x8d87d5 thru 0x8d8807 (0x33 bytes) 000366[1] 1780987794.054742 trunc from 0x8d8808 to 0x2abc03 (0x62cc04 bytes) 000367[1] 1780987794.172446 write 0x0a7cc7 thru 0x0a814a (0x484 bytes) 000373[1] 1780987794.282353 write 0x78e422 thru 0x78e693 (0x272 bytes) 000374[1] 1780987794.314160 trunc from 0x78e694 to 0x4cd645 (0x2c104e bytes) 000375[1] 1780987794.388717 trunc from 0x4cd646 to 0x0399c8 (0x493c7d bytes) 000378[0] 1780987794.532626 mapread 0x0309b3 thru 0x032614 (0x1c62 bytes) 000379[1] 1780987794.586502 trunc from 0x0399c9 to 0x5d75b6 (0x59dbee bytes) 000380[1] 1780987794.620233 mapread 0x3f47b9 thru 0x3f55df (0xe27 bytes) 000381[1] 1780987794.638768 read 0x0a3d1f thru 0x0a423d (0x51f bytes) 000383[1] 1780987794.686706 trunc from 0x5d75b7 to 0x521ea9 (0xb570d bytes) 000384[0] 1780987794.725315 mapread 0x3252d9 thru 0x326467 (0x118f bytes) 000386[0] 1780987794.851979 trunc from 0x521eaa to 0x1554b8 (0x3cc9f1 bytes) 000387[1] 1780987794.933091 mapread 0x05a5a4 thru 0x05aa57 (0x4b4 bytes) 000388[1] 1780987794.986826 read 0x13f7ce thru 0x14042e (0xc61 bytes) 000389[0] 1780987795.036145 trunc from 0x1554b9 to 0x4fcaea (0x3a7632 bytes) 000390[1] 1780987795.090749 trunc from 0x4fcaeb to 0x8bfab4 (0x3c2fca bytes) 000393[1] 1780987795.195268 trunc from 0x8bfab5 to 0x226660 (0x699454 bytes) 000395[1] 1780987795.294183 mapwrite 0x40a56c thru 0x40adb6 (0x84b bytes) 000397[1] 1780987795.580291 mapwrite 0x11524c thru 0x115641 (0x3f6 bytes) 000398[1] 1780987795.725751 write 0x589a4d thru 0x58b1e5 (0x1799 bytes) 000399[0] 1780987795.728446 read 0x4d9bec thru 0x4db5af (0x19c4 bytes) 000400[1] 1780987795.858999 mapwrite 0x87bcc4 thru 0x87c8c1 (0xbfe bytes) 000401[1] 1780987796.143940 mapwrite 0x3ef254 thru 0x3ef3cf (0x17c bytes) 000403[0] 1780987796.205282 mapread 0x73e83c thru 0x74072c (0x1ef1 bytes) 000404[0] 1780987796.269916 write 0x356245 thru 0x357e54 (0x1c10 bytes) 000406[1] 1780987796.307008 write 0x2ba593 thru 0x2bb402 (0xe70 bytes) 000407[0] 1780987796.367545 read 0x64878b thru 0x649ea9 (0x171f bytes) 000408[1] 1780987796.376372 mapwrite 0x95870c thru 0x9596e9 (0xfde bytes) 000409[0] 1780987796.559594 read 0x1eeb08 thru 0x1f0406 (0x18ff bytes) 000413[0] 1780987796.672884 write 0x5a7bff thru 0x5a87ea (0xbec bytes) 000414[1] 1780987796.722571 read 0x3e53ab thru 0x3e64ea (0x1140 bytes) 000415[0] 1780987796.770210 read 0x9196ab thru 0x91a1e3 (0xb39 bytes) 000416[1] 1780987796.776982 trunc from 0x9596ea to 0x4acdb5 (0x4ac934 bytes) 000418[0] 1780987796.872162 mapwrite 0x2a403b thru 0x2a5a40 (0x1a06 bytes) 000419[1] 1780987797.036178 write 0x3b747b thru 0x3b7d51 (0x8d7 bytes) 000421[1] 1780987797.087155 trunc from 0x4acdb6 to 0x4c0a35 (0x13c80 bytes) 000428[1] 1780987797.319850 read 0x173af3 thru 0x1740ec (0x5fa bytes) 000429[0] 1780987797.328245 mapwrite 0x8fe952 thru 0x90082b (0x1eda bytes) 000431[0] 1780987797.562510 mapwrite 0x50654e thru 0x506d4b (0x7fe bytes) 000436[0] 1780987797.873657 read 0x86458e thru 0x866402 (0x1e75 bytes) 000442[1] 1780987798.078704 trunc from 0x90082c to 0x5859fa (0x37ae31 bytes) 000446[0] 1780987798.215634 mapwrite 0x1551cf thru 0x155f8b (0xdbd bytes) 000448[0] 1780987798.325539 read 0x27a91b thru 0x27c10d (0x17f3 bytes) 000450[1] 1780987798.387479 write 0x1af4c7 thru 0x1b0fb3 (0x1aed bytes) 000453[1] 1780987798.423245 trunc from 0x5859fb to 0x01f339 (0x5666c1 bytes) 000455[1] 1780987798.490213 trunc from 0x01f33a to 0x697664 (0x67832b bytes) 000457[0] 1780987798.582744 mapwrite 0x58fac7 thru 0x58fb57 (0x91 bytes) 000460[1] 1780987798.717270 mapread 0x4ab0fa thru 0x4ab91e (0x825 bytes) 000461[1] 1780987798.744120 read 0x2e312f thru 0x2e4d54 (0x1c26 bytes) 000462[1] 1780987798.749790 read 0x0c1bc4 thru 0x0c3826 (0x1c63 bytes) 000463[0] 1780987798.801428 read 0x62a263 thru 0x62a2e1 (0x7f bytes) 000464[1] 1780987798.816126 read 0x473ff7 thru 0x4746e3 (0x6ed bytes) 000465[1] 1780987798.830541 read 0x01fd3e thru 0x021ac2 (0x1d85 bytes) 000468[0] 1780987798.970423 mapread 0x294bfc thru 0x296ad8 (0x1edd bytes) 000471[0] 1780987799.045075 mapread 0x56c7b2 thru 0x56da84 (0x12d3 bytes) 000472[1] 1780987799.067313 trunc from 0x697665 to 0x409f96 (0x28d6ce bytes) 000473[1] 1780987799.166014 mapwrite 0x3f287f thru 0x3f291a (0x9c bytes) 000475[0] 1780987799.361176 mapwrite 0x20ad78 thru 0x20b34b (0x5d4 bytes) 000478[0] 1780987799.559324 mapwrite 0x6160c0 thru 0x617717 (0x1658 bytes) 000479[0] 1780987799.695720 read 0x0feeca thru 0x0ff8a8 (0x9df bytes) 000480[1] 1780987799.759547 mapwrite 0x9af60d thru 0x9b0d43 (0x1737 bytes) 000482[1] 1780987799.999308 write 0x662a44 thru 0x663d14 (0x12d1 bytes) 000483[0] 1780987800.053221 mapread 0x375cf0 thru 0x3766b7 (0x9c8 bytes) 000485[1] 1780987800.161360 write 0x34d083 thru 0x34db36 (0xab4 bytes) 000486[0] 1780987800.182379 read 0x971298 thru 0x972077 (0xde0 bytes) 000489[1] 1780987800.243985 trunc from 0x9b0d44 to 0x3b5a50 (0x5fb2f3 bytes) 000492[1] 1780987800.325261 read 0x279ed0 thru 0x27be24 (0x1f55 bytes) 000493[1] 1780987800.421187 mapwrite 0x15a403 thru 0x15ad6f (0x96d bytes) 000495[0] 1780987800.469545 read 0x0fc453 thru 0x0fcab7 (0x665 bytes) 000497[0] 1780987800.591976 read 0x17d702 thru 0x17da0c (0x30b bytes) 000498[0] 1780987800.592846 trunc from 0x3b5a51 to 0x348c3c (0x6ce14 bytes) 000499[1] 1780987800.679294 mapwrite 0x28196a thru 0x281a76 (0x10d bytes) 000500[0] 1780987800.794909 mapwrite 0x256831 thru 0x257d5b (0x152b bytes) All operations completed A-OK! PASS 16b (34s) == sanityn test 16c: verify data consistency on ldiskfs with cache disabled (b=17397) ========================================================== 02:50:06 (1780987806) SKIP: sanityn test_16c dio on ldiskfs only SKIP 16c (5s) == sanityn test 16d: Verify DIO and buffer IO with two clients ========================================================== 02:50:11 (1780987811) 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.302314 s, 27.7 MB/s 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.267883 s, 31.3 MB/s ldlm.namespaces.MGC192.168.206.153@tcp.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8a2a59a58800.lru_size=clear ldlm.namespaces.lustre-MDT0000-mdc-ffff8a2a60524000.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lru_size=clear ldlm.namespaces.lustre-OST0000-osc-ffff8a2a60524000.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8a2a59a58800.lru_size=clear ldlm.namespaces.lustre-OST0001-osc-ffff8a2a60524000.lru_size=clear 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.2507 s, 8.6 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.77514 s, 27.8 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 3.11283 s, 33.7 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 6.41886 s, 16.3 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 7.85154 s, 13.4 MB/s 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 4.46923 s, 23.5 MB/s PASS 16d (51s) == sanityn test 16e: Verify size consistency for O_DIRECT write ========================================================== 02:51:03 (1780987863) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.257992 s, 40.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142663 s, 7.4 MB/s PASS 16e (7s) == sanityn test 16f: rw sequential consistency vs drop_caches ========================================================== 02:51:10 (1780987870) TIMEOUT PASS 16f (28s) == sanityn test 16g: mmap rw sequential consistency vs drop_caches ========================================================== 02:51:39 (1780987899) TIMEOUT PASS 16g (29s) == sanityn test 16h: mmap read after truncate file ======= 02:52:09 (1780987929) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16h (10s) == sanityn test 16i: read after truncate file ============ 02:52:18 (1780987938) ------- before truncate -------- 0002700 6e65 7265 7461 726f 4420 6561 6f6d 3a6e 0002720 762f 7261 6c2f 6269 722f 676e 3a64 732f 0002740 6962 2f6e 6f6e 6f6c 6967 0a6e 0002754 ------- after truncate -------- 0001320 746e 7520 6573 2064 7962 7420 6568 7420 0001340 6f72 7375 7265 2073 6170 6b63 6761 2065 0001360 6f74 7320 6e61 0001366 PASS 16i (8s) == sanityn test 16j: race dio with buffered i/o ========== 02:52:26 (1780987946) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.66833 s, 9.8 MB/s bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.4741 s, 17.8 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.43137 s, 10.8 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.51014 s, 17.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.11452 s, 12.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.99379 s, 13.1 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.79364 s, 9.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1044480, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.36377 s, 19.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.80548 s, 14.5 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.2826 s, 8.0 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.52362 s, 7.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1048576, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.11617 s, 8.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.42392 s, 7.7 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.95382 s, 6.6 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.09305 s, 6.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK bs: 1049600, file_size 26214400 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.63951 s, 7.2 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.1505 s, 6.3 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.84157 s, 5.4 MB/s 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.89272 s, 5.4 MB/s /mnt/lustre/f16j.sanityn.2 has type file OK /mnt/lustre/f16j.sanityn.2 has size 26214400 OK /mnt/lustre2/f16j.sanityn.2 has type file OK /mnt/lustre2/f16j.sanityn.2 has size 26214400 OK PASS 16j (46s) == sanityn test 16k: Parallel FSX and drop caches should not panic ========================================================== 02:53:12 (1780987992) + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.1 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.2 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.3 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.4 -l 67108864 + /home/green/git/lustre-release/lustre/tests/fsx -N 1000 /mnt/lustre/fsxfile.5 -l 67108864 fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x0, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x3, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported truncating to largest ever: 0x1a7062c fsx: test_fallocate: filesystem does not support fallocate mode 0x10, disabling!: Operation not supported fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported skipping zero size read fsx: test_fallocate: filesystem does not support fallocate mode 0x1, disabling!: Operation not supported truncating to largest ever: 0x232e1d3 skipping zero size read truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x1a7062c truncating to largest ever: 0x232e1d3 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x32cce75 truncating to largest ever: 0x32cce75 truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c569db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3c711db truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f0e56b truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab truncating to largest ever: 0x3f706ab All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! All operations completed A-OK! + fsx 1 done: rc=0 Tue Jun 9 02:54:02 EDT 2026 + fsx 2 done: rc=0 Tue Jun 9 02:54:02 EDT 2026 + fsx 3 done: rc=0 Tue Jun 9 02:54:02 EDT 2026 + fsx 4 done: rc=0 Tue Jun 9 02:54:02 EDT 2026 + fsx 5 done: rc=0 Tue Jun 9 02:54:02 EDT 2026 PASS 16k (59s) == sanityn test 17: resource creation/LVB creation race ========================================================================= 02:54:12 (1780988052) fail_loc=0x8000030a PASS 17 (10s) == sanityn test 18: mmap sanity check =========================================================================================== 02:54:23 (1780988063) mmap test1: basic mmap operation (PASS, 0.2041s) mmap test2: MAP_PRIVATE not write back (PASS, 0.067135s) mmap test3: concurrent mmap ops on two nodes (PASS, 9.8718s) mmap test4: c1 write to f1 from mmapped f2, c2 write to f1 from mmapped f1 (PASS, 4.693s) mmap test5: read/write file to/from the buffer which mmapped to just this file (PASS, 0.18867s) mmap test6: check mmap write/read content on two nodes (PASS, 0.23931s) mmap test7: file i/o with an unmapped buffer (SKIPPED, 0s) mmap test8: SIGBUS for beyond file size (SKIPPED, 0s) mmap test9: SIGBUS for negative file offset (SKIPPED, 0s) mmap test10: multi-thread mmap access (PASS, 1.837s) mmap test11: multi-thread shared mmap (PASS, 1.7584s) mmap test10: mtime not change for readonly mmap access (PASS, 1.1083s) PASS 18 (29s) == sanityn test 19: test concurrent uncached read races ========================================================================= 02:54:52 (1780988092) oleg653-server: error: get_param: param_path 'osd-*/lustre-OST*/read_cache_enable': No such file or directory pdsh@oleg653-client: oleg653-server: ssh exited with exit code 2 SKIP: sanityn test_19 not cache-capable obdfilter SKIP 19 (4s) == sanityn test 20: test extra readahead page left in cache ============================================================== 02:54:56 (1780988096) PASS 20 (7s) == sanityn test 21: Try to remove mountpoint on another dir ============================================================== 02:55:03 (1780988103) rmdir: removing directory, '/mnt/lustre/d21.sanityn' rmdir: failed to remove '/mnt/lustre/d21.sanityn': Device or resource busy rmdir: removing directory, '/mnt/lustre2/d21.sanityn' Removed mounted directory from another mountpoint, needs to be fixed PASS 21 (8s) == sanityn test 23: others should see updated atime while another read============================================================== 02:55:11 (1780988111) now is 1780988114 starting reads multiop /mnt/lustre/f23.sanityn vor20_c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 new atime is 1780988175 PASS 23 (69s) == sanityn test 24a: lfs df [-ih] [path] test =================================================================================== 02:56:20 (1780988180) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 115.2K 401 114.8K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 401 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 2.1G 2.1M 2.1G 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3.6G 8.0M 3.6G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.6G 30.0M 3.6G 1% /mnt/lustre[OST:1] filesystem_summary: 7.2G 38.0M 7.2G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118001 401 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 524.8K 386 524.4K 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 115.2K 401 114.8K 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 114.5K 402 114.2K 1% /mnt/lustre2[OST:1] filesystem_summary: 229.4K 386 229.0K 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118001 401 117600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 537416 386 537030 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 118001 401 117600 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 117298 402 116896 1% /mnt/lustre2[OST:1] filesystem_summary: 234882 386 234496 1% /mnt/lustre2 Reader error: 'No devices found' at 0 No device found for name osc: Invalid argument UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre2[MDT:0] lustre-OST0000_UUID 3771392 8192 3761152 1% /mnt/lustre2[OST:0] lustre-OST0001_UUID 3771392 30720 3738624 1% /mnt/lustre2[OST:1] filesystem_summary: 7542784 38912 7499776 1% /mnt/lustre2 PASS 24a (8s) == sanityn test 24b: lfs df should show both filesystems ========================================================================= 02:56:28 (1780988188) PASS 24b (7s) == sanityn test 25a: change ACL on one mountpoint be seen on another ============================================================= 02:56:35 (1780988195) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] Can't lstat /mnt/lustre2/d25a.sanityn/f1: Permission denied running as uid/gid/euid/egid 500/500/500/500, groups: 500 [checkstat] [-v] [/mnt/lustre2/d25a.sanityn/f1] PASS 25a (8s) == sanityn test 25b: change ACL under remote dir on one mountpoint be seen on another ========================================================== 02:56:43 (1780988203) SKIP: sanityn test_25b needs >= 2 MDTs SKIP 25b (4s) == sanityn test 26a: allow mtime to get older ============ 02:56:47 (1780988207) utime: good mknod times 1780988208 <= 1780988209 <= 1780988209 for /mnt/lustre/f26a utime: good mknod times 1780988208 <= 1780988209 <= 1780988209 for /mnt/lustre2/f26a utime: good utime mtimes 100000, atime 200000 utime: updated times for /mnt/lustre2/f26a PASS 26a (8s) == sanityn test 26b: sync mtime between ost and mds ====== 02:56:55 (1780988215) PASS 26b (9s) == sanityn test 26c: set-in-past on open file is not lost on close ========================================================== 02:57:04 (1780988224) multiop /mnt/lustre/f26c.sanityn vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 File: /mnt/lustre/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-06-09 02:57:05.000000000 -0400 Birth: 2026-06-09 02:57:05.000000000 -0400 File: /mnt/lustre2/f26c.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205272502398 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2000-12-31 06:12:59.000000000 -0500 Modify: 2000-12-31 06:12:59.000000000 -0500 Change: 2026-06-09 02:57:05.000000000 -0400 Birth: 2026-06-09 02:57:05.000000000 -0400 PASS 26c (7s) SKIP: sanityn test_27 skipping excluded test 27 == sanityn test 30: recreate file race =================== 02:57:13 (1780988233) 0 PASS 30 (9s) == sanityn test 31a: voluntary cancel / blocking ast race======================================================================== 02:57:23 (1780988243) fail_loc=0x314 PASS 31a (8s) == sanityn test 31b: voluntary OST cancel / blocking ast race======================================================================== 02:57:30 (1780988250) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg653-server mds-ost sync done. sleep 5 for ZFS MDS Waiting for MDT destroys to complete fail_loc=0x314 fail_loc=0x316 fail_loc=0 fail_loc=0 Filesystem 1K-blocks Used Available Use% Mounted on 192.168.206.153@tcp:/lustre 7542784 39936 7498752 1% /mnt/lustre2 PASS 31b (24s) == sanityn test 31r: open-rename(replace) race =========== 02:57:54 (1780988274) -rw-r--r-- 1 root root 0 Jun 9 02:57 /mnt/lustre/f31r.sanityn.target fail_loc=0x1419 fail_val=3 PASS 31r (11s) == sanityn test 31s: open should not revalidate invalid dentry ========================================================== 02:58:05 (1780988285) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1207: 43008 Terminated tail -f $tfile PASS 31s (8s) == sanityn test 31t: getattr should not revalidate invalid dentry ========================================================== 02:58:13 (1780988293) PASS 31t (9s) SKIP: sanityn test_33a skipping SLOW test 33a == sanityn test 33b: COS: cross create/delete, 2 clients, benchmark under remote dir ========================================================== 02:58:24 (1780988304) SKIP: sanityn test_33b Need two or more clients, have 1 SKIP 33b (3s) == sanityn test 33c: Cancel cross-MDT lock should trigger Sync-on-Lock-Cancel ========================================================== 02:58:28 (1780988308) SKIP: sanityn test_33c needs >= 2 MDTs SKIP 33c (4s) == sanityn test 33d: dependent transactions should trigger COS ========================================================== 02:58:32 (1780988312) SKIP: sanityn test_33d needs >= 2 MDTs SKIP 33d (4s) == sanityn test 33e: independent transactions shouldn't trigger COS ========================================================== 02:58:36 (1780988316) SKIP: sanityn test_33e needs >= 2 MDTs SKIP 33e (4s) == sanityn test 34: no lock timeout under IO ============= 02:58:40 (1780988320) fail_loc=0x512 fail_loc=0x512 lock should not expire writing on client1 reading on client2 success fail_loc=0x511 fail_loc=0x511 lock should expire writing on client1 reading on client2 success oleg653-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a2a59a58800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a2a59a58800.ost_server_uuid in IDLE state after 0 sec oleg653-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a2a59a58800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a2a59a58800.ost_server_uuid in IDLE state after 0 sec oleg653-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a2a59a58800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a2a59a58800.ost_server_uuid in IDLE state after 0 sec oleg653-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a2a59a58800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a2a59a58800.ost_server_uuid in IDLE state after 0 sec oleg653-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a2a59a58800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a2a59a58800.ost_server_uuid in IDLE state after 0 sec oleg653-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a2a59a58800.ost_server_uuid 50 osc.lustre-OST0001-osc-ffff8a2a59a58800.ost_server_uuid in IDLE state after 0 sec PASS 34 (135s) == sanityn test 35: -EINTR cp_ast vs. bl_ast race does not evict client ========================================================== 03:00:55 (1780988455) fail_loc=0x80000317 Race attempt 0 Wait for 49062 49089 for 60 sec... /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 1746: 49062 Killed createmany -o $MOUNT2/$tdir/a 4000 fail_loc=0x0 Filesystem Size Used Avail Use% Mounted on 192.168.206.153@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre 192.168.206.153@tcp:/lustre 7.2G 40M 7.2G 1% /mnt/lustre2 - open/close 576 (time 1780988474.40 total 10.00 last 57.58) - open/close 1136 (time 1780988484.41 total 20.01 last 55.95) - open/close 1661 (time 1780988494.41 total 30.02 last 52.47) - open/close 2239 (time 1780988504.44 total 40.04 last 57.65) - open/close 2812 (time 1780988514.46 total 50.07 last 57.16) - open/close 3422 (time 1780988524.49 total 60.09 last 60.85) - open/close 3938 (time 1780988534.54 total 70.14 last 51.35) total: 4000 open/close in 70.91 seconds: 56.41 ops/second PASS 35 (80s) == sanityn test 36: handle ESTALE/open-unlink correctly == 03:02:15 (1780988535) Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed wait 40 secs maximumly for oleg653-server mds-ost sync done. Waiting for MDT destroys to complete 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.840458 s, 62.4 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(0) *** before(7497728) after_dd(7427072) after(7497728) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.987035 s, 53.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(1) *** before(7497728) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.733601 s, 71.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(2) *** before(7490560) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 1.1846 s, 44.3 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(3) *** before(7490560) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.74402 s, 70.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(4) *** before(7490560) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.820618 s, 63.9 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(5) *** before(7490560) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.751336 s, 69.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(6) *** before(7490560) after_dd(7432192) after(7483392) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.969112 s, 54.1 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(7) *** before(7483392) after_dd(7442432) after(7493632) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.803977 s, 65.2 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS Waiting for MDT destroys to complete *** cycle(8) *** before(7493632) after_dd(7439360) after(7490560) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.753882 s, 69.5 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(9) *** before(7490560) after_dd(7440384) after(7491584) 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 0.656956 s, 79.8 MB/s multiop /mnt/lustre2/d36.sanityn/f36.sanityn vO_r52428800c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete *** cycle(10) *** before(7491584) after_dd(7437312) after(7488512) PASS 36 (314s) == sanityn test 37: check i_size is not updated for directory on close (bug 18695) ======================================================================== 03:07:29 (1780988849) multiop /mnt/lustre/d37.sanityn vD_c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 - create 1267 (time 1780988863.93 total 10.01 last 126.58) - create 2468 (time 1780988873.93 total 20.01 last 120.09) - create 3661 (time 1780988883.94 total 30.02 last 119.22) - create 4827 (time 1780988893.94 total 40.02 last 116.54) - create 5747 (time 1780988903.95 total 50.03 last 91.91) - create 7013 (time 1780988913.95 total 60.03 last 126.57) - create 8208 (time 1780988923.96 total 70.04 last 119.44) - create 9436 (time 1780988933.97 total 80.06 last 122.60) total: 10000 create in 84.54 seconds: 118.28 ops/second PASS 37 (98s) == sanityn test 39a: file mtime does not change after rename ========================================================== 03:09:07 (1780988947) repeat after cancel_lru_locks PASS 39a (8s) == sanityn test 39b: file mtime the same on clients with/out lock ========================================================== 03:09:15 (1780988955) repeat after cancel_lru_locks PASS 39b (10s) == sanityn test 39c: check truncate mtime update ================================================================================ 03:09:25 (1780988965) repeat after cancel_lru_locks PASS 39c (9s) == sanityn test 39d: sync write should update mtime ====== 03:09:34 (1780988974) fail_loc=0x411 fail_loc=0 PASS 39d (8s) SKIP: sanityn test_40a skipping ALWAYS excluded test 40a == sanityn test 40b: pdirops: open|create and others ======================================================================== 03:09:44 (1780988984) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40b (18s) == sanityn test 40c: pdirops: link and others ======================================================================== 03:10:03 (1780989003) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40c (18s) == sanityn test 40d: pdirops: unlink and others ======================================================================== 03:10:22 (1780989022) No conflict No conflict No conflict No conflict No conflict No conflict No conflict PASS 40d (17s) == sanityn test 40e: pdirops: rename and others ======================================================================== 03:10:40 (1780989040) No conflict No conflict No conflict No conflict No conflict No conflict PASS 40e (17s) == sanityn test 41a: pdirops: create vs mkdir ======================================================================== 03:10:57 (1780989057) mkdir: cannot create directory '/mnt/lustre2/f41a.sanityn': File exists Conflict PASS 41a (15s) == sanityn test 41b: pdirops: create vs create ======================================================================== 03:11:13 (1780989073) open: File exists Conflict PASS 41b (14s) == sanityn test 41c: pdirops: create vs link ======================================================================== 03:11:27 (1780989087) link: cannot create link '/mnt/lustre2/f41c.sanityn' to '/mnt/lustre2/f41c.sanityn-2': File exists Conflict PASS 41c (14s) == sanityn test 41d: pdirops: create vs unlink ======================================================================== 03:11:41 (1780989101) Conflict PASS 41d (16s) == sanityn test 41e: pdirops: create and rename (tgt) ======================================================================== 03:11:57 (1780989117) Conflict PASS 41e (17s) == sanityn test 41f: pdirops: create and rename (src) ======================================================================== 03:12:14 (1780989134) Conflict PASS 41f (15s) == sanityn test 41g: pdirops: create vs getattr ======================================================================== 03:12:29 (1780989149) Conflict PASS 41g (13s) == sanityn test 41h: pdirops: create vs readdir ======================================================================== 03:12:42 (1780989162) /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 2318: lctl set_param -n fail_loc=0 2>/dev/null || true: No such file or directory Conflict PASS 41h (14s) == sanityn test 41i: reint_open: create vs create ======== 03:12:57 (1780989177) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 41i (1110s) == sanityn test 42a: pdirops: mkdir vs mkdir ======================================================================== 03:31:27 (1780990287) mkdir: cannot create directory '/mnt/lustre2/f42a.sanityn': File exists Conflict PASS 42a (15s) == sanityn test 42b: pdirops: mkdir vs create ======================================================================== 03:31:42 (1780990302) open: File exists Conflict PASS 42b (16s) == sanityn test 42c: pdirops: mkdir vs link ======================================================================== 03:31:58 (1780990318) link: cannot create link '/mnt/lustre2/f42c.sanityn' to '/mnt/lustre2/f42c.sanityn-2': File exists Conflict PASS 42c (15s) == sanityn test 42d: pdirops: mkdir vs unlink ======================================================================== 03:32:13 (1780990333) Conflict PASS 42d (15s) == sanityn test 42e: pdirops: mkdir and rename (tgt) ======================================================================== 03:32:29 (1780990349) mv: cannot overwrite directory '/mnt/lustre2/f42e.sanityn' with non-directory Conflict PASS 42e (14s) == sanityn test 42f: pdirops: mkdir and rename (src) ======================================================================== 03:32:43 (1780990363) Conflict PASS 42f (17s) == sanityn test 42g: pdirops: mkdir vs getattr ======================================================================== 03:33:00 (1780990380) Conflict PASS 42g (15s) == sanityn test 42h: pdirops: mkdir vs readdir ======================================================================== 03:33:15 (1780990395) Conflict PASS 42h (16s) == sanityn test 43a: rmdir,mkdir doesn't return -EEXIST ======================================================================== 03:33:31 (1780990411) PASS 43a (84s) == sanityn test 43b: pdirops: unlink vs create ======================================================================== 03:34:55 (1780990495) Conflict PASS 43b (16s) == sanityn test 43c: pdirops: unlink vs link ======================================================================== 03:35:11 (1780990511) Conflict PASS 43c (14s) == sanityn test 43d: pdirops: unlink vs unlink ======================================================================== 03:35:25 (1780990525) rm: cannot remove '/mnt/lustre2/f43d.sanityn': No such file or directory Conflict PASS 43d (15s) == sanityn test 43e: pdirops: unlink and rename (tgt) ======================================================================== 03:35:40 (1780990540) Conflict PASS 43e (15s) == sanityn test 43f: pdirops: unlink and rename (src) ======================================================================== 03:35:55 (1780990555) mv: cannot stat '/mnt/lustre2/f43f.sanityn': No such file or directory Conflict PASS 43f (14s) == sanityn test 43g: pdirops: unlink vs getattr ======================================================================== 03:36:09 (1780990569) stat: cannot statx '/mnt/lustre2/f43g.sanityn': No such file or directory Conflict PASS 43g (14s) == sanityn test 43h: pdirops: unlink vs readdir ======================================================================== 03:36:23 (1780990583) Conflict PASS 43h (15s) == sanityn test 43i: pdirops: unlink vs remote mkdir ===== 03:36:38 (1780990598) SKIP: sanityn test_43i needs >= 2 MDTs SKIP 43i (5s) == sanityn test 43j: racy mkdir return EEXIST ======================================================================== 03:36:43 (1780990603) mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre2/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists mkdir: cannot create directory '/mnt/lustre/d43j.sanityn/sub': File exists PASS 43j (153s) == sanityn test 43k: unlink vs create ==================== 03:39:16 (1780990756) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 43k (1357s) == sanityn test 44a: pdirops: rename tgt vs mkdir ======================================================================== 04:01:53 (1780992113) mkdir: cannot create directory '/mnt/lustre2/f44a.sanityn': File exists Conflict Tue Jun 9 04:02:02 EDT 2026 PASS 44a (15s) == sanityn test 44b: pdirops: rename tgt vs create ======================================================================== 04:02:08 (1780992128) open: File exists Conflict PASS 44b (16s) == sanityn test 44c: pdirops: rename tgt vs link ======================================================================== 04:02:24 (1780992144) link: cannot create link '/mnt/lustre2/f44c.sanityn' to '/mnt/lustre2/f44c.sanityn-3': File exists Conflict PASS 44c (13s) == sanityn test 44d: pdirops: rename tgt vs unlink ======================================================================== 04:02:37 (1780992157) Conflict PASS 44d (15s) == sanityn test 44e: pdirops: rename tgt and rename (tgt) ======================================================================== 04:02:52 (1780992172) Conflict PASS 44e (15s) == sanityn test 44f: pdirops: rename tgt and rename (src) ======================================================================== 04:03:07 (1780992187) Conflict PASS 44f (15s) == sanityn test 44g: pdirops: rename tgt vs getattr ======================================================================== 04:03:22 (1780992202) Conflict PASS 44g (14s) == sanityn test 44h: pdirops: rename tgt vs readdir ======================================================================== 04:03:36 (1780992216) Conflict PASS 44h (14s) == sanityn test 44i: pdirops: rename tgt vs remote mkdir ========================================================== 04:03:51 (1780992231) SKIP: sanityn test_44i needs >= 2 MDTs SKIP 44i (3s) == sanityn test 45a: rename,mkdir doesn't return -EEXIST ======================================================================== 04:03:55 (1780992235) PASS 45a (133s) == sanityn test 45b: pdirops: rename src vs create ======================================================================== 04:06:08 (1780992368) Conflict PASS 45b (17s) == sanityn test 45c: pdirops: rename src vs link ======================================================================== 04:06:25 (1780992385) Conflict PASS 45c (15s) == sanityn test 45d: pdirops: rename src vs unlink ======================================================================== 04:06:40 (1780992400) rm: cannot remove '/mnt/lustre2/f45d.sanityn': No such file or directory Conflict PASS 45d (15s) == sanityn test 45e: pdirops: rename src and rename (tgt) ======================================================================== 04:06:55 (1780992415) Conflict PASS 45e (17s) == sanityn test 45f: pdirops: rename src and rename (src) ======================================================================== 04:07:12 (1780992432) mv: cannot stat '/mnt/lustre2/f45f.sanityn': No such file or directory Conflict PASS 45f (18s) == sanityn test 45g: pdirops: rename src vs getattr ======================================================================== 04:07:31 (1780992451) stat: cannot statx '/mnt/lustre2/f45g.sanityn': No such file or directory Conflict PASS 45g (16s) == sanityn test 45h: pdirops: unlink vs readdir ======================================================================== 04:07:47 (1780992467) Conflict PASS 45h (13s) == sanityn test 45i: pdirops: rename src vs remote mkdir ========================================================== 04:08:01 (1780992481) SKIP: sanityn test_45i needs >= 2 MDTs SKIP 45i (4s) == sanityn test 45j: read vs rename ====================== 04:08:05 (1780992485) Begin 100 tests with fail_loc=0x80000169 Progress: **************************************************************************************************** Begin 100 tests with fail_loc=0x8000016a Progress: **************************************************************************************************** PASS 45j (1257s) == sanityn test 46a: pdirops: link vs mkdir ======================================================================== 04:29:02 (1780993742) mkdir: cannot create directory '/mnt/lustre2/f46a.sanityn': File exists Conflict PASS 46a (13s) == sanityn test 46b: pdirops: link vs create ======================================================================== 04:29:15 (1780993755) open: File exists Conflict PASS 46b (13s) == sanityn test 46c: pdirops: link vs link ======================================================================== 04:29:28 (1780993768) link: cannot create link '/mnt/lustre2/f46c.sanityn' to '/mnt/lustre2/f46c.sanityn': File exists Conflict PASS 46c (12s) == sanityn test 46d: pdirops: link vs unlink ======================================================================== 04:29:40 (1780993780) Conflict PASS 46d (14s) == sanityn test 46e: pdirops: link and rename (tgt) ======================================================================== 04:29:54 (1780993794) Conflict PASS 46e (13s) == sanityn test 46f: pdirops: link and rename (src) ======================================================================== 04:30:07 (1780993807) Conflict PASS 46f (13s) == sanityn test 46g: pdirops: link vs getattr ======================================================================== 04:30:20 (1780993820) Conflict PASS 46g (12s) == sanityn test 46h: pdirops: link vs readdir ======================================================================== 04:30:32 (1780993832) Conflict PASS 46h (14s) == sanityn test 46i: pdirops: link vs remote mkdir ======= 04:30:46 (1780993846) SKIP: sanityn test_46i needs >= 2 MDTs SKIP 46i (4s) == sanityn test 47a: pdirops: remote mkdir vs mkdir ====== 04:30:50 (1780993850) SKIP: sanityn test_47a needs >= 2 MDTs SKIP 47a (3s) == sanityn test 47b: pdirops: remote mkdir vs create ===== 04:30:53 (1780993853) SKIP: sanityn test_47b needs >= 2 MDTs SKIP 47b (3s) == sanityn test 47c: pdirops: remote mkdir vs link ======= 04:30:57 (1780993857) SKIP: sanityn test_47c needs >= 2 MDTs SKIP 47c (3s) == sanityn test 47d: pdirops: remote mkdir vs unlink ===== 04:31:00 (1780993860) SKIP: sanityn test_47d needs >= 2 MDTs SKIP 47d (3s) == sanityn test 47e: pdirops: remote mkdir and rename (tgt) ========================================================== 04:31:03 (1780993863) SKIP: sanityn test_47e needs >= 2 MDTs SKIP 47e (3s) == sanityn test 47f: pdirops: remote mkdir and rename (src) ========================================================== 04:31:07 (1780993867) SKIP: sanityn test_47f needs >= 2 MDTs SKIP 47f (4s) == sanityn test 47g: pdirops: remote mkdir vs getattr ==== 04:31:10 (1780993870) SKIP: sanityn test_47g needs >= 2 MDTs SKIP 47g (3s) == sanityn test 50: osc lvb attrs: enqueue vs. CP AST ======================================================================== 04:31:13 (1780993873) 10+0 records in 10+0 records out 10240 bytes (10 kB, 10 KiB) copied, 0.122372 s, 83.7 kB/s fail_loc=0x410 fail_loc=0x0 PASS 50 (13s) == sanityn test 51a: layout lock: refresh layout should work ========================================================== 04:31:26 (1780993886) 0+1 records in 0+1 records out 158 bytes copied, 0.0555301 s, 2.8 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 3581: kill: (217177) - No such process PASS 51a (9s) == sanityn test 51b: layout lock: glimpse should be able to restart if layout changed ========================================================== 04:31:35 (1780993895) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0262638 s, 39.0 kB/s fail_loc=0x1404 fail_val=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0412555 s, 25.4 MB/s 3145728 PASS 51b (20s) == sanityn test 51c: layout lock: IT_LAYOUT blocked and correct layout can be returned ========================================================== 04:31:55 (1780993915) fail_loc=0x172 Setting layout to have 2 stripes ... 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0237551 s, 43.1 kB/s PASS 51c (12s) == sanityn test 51d: layout lock: losing layout lock should clean up memory map region ========================================================== 04:32:07 (1780993927) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0422897 s, 24.8 MB/s Before revoking layout lock: 1024 KB mapped PASS 51d (8s) == sanityn test 51e: lfs getstripe does not break leases, part 2 ========================================================== 04:32:15 (1780993935) /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2450 0x992 0x280000400 write lease(2) has applied. /mnt/lustre2/f51e.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2450 0x992 0x280000400 read lease(1) has applied. PASS 51e (9s) == sanityn test 54: rename locking ======================= 04:32:24 (1780993944) llite.lustre-ffff8a2a59a58800.xattr_cache=1 llite.lustre-ffff8a2a60524000.xattr_cache=1 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505011 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-09 04:32:26.000000000 -0400 Modify: 2026-06-09 04:32:26.000000000 -0400 Change: 2026-06-09 04:32:26.000000000 -0400 Birth: 2026-06-09 04:32:25.000000000 -0400 ==> rename vs getattr vs setxattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505014 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-09 04:32:32.000000000 -0400 Modify: 2026-06-09 04:32:33.000000000 -0400 Change: 2026-06-09 04:32:33.000000000 -0400 Birth: 2026-06-09 04:32:32.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000153 File: /mnt/lustre/d1/d2 Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505017 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-09 04:32:38.000000000 -0400 Modify: 2026-06-09 04:32:38.000000000 -0400 Change: 2026-06-09 04:32:38.000000000 -0400 Birth: 2026-06-09 04:32:38.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505016 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-09 04:32:38.000000000 -0400 Modify: 2026-06-09 04:32:39.000000000 -0400 Change: 2026-06-09 04:32:39.000000000 -0400 Birth: 2026-06-09 04:32:38.000000000 -0400 ==> rename vs getattr vs open vs getattr should not deadlock fail_loc=0x80000154 File: /mnt/lustre/d1 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505019 Links: 4 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-09 04:32:44.000000000 -0400 Modify: 2026-06-09 04:32:45.000000000 -0400 Change: 2026-06-09 04:32:45.000000000 -0400 Birth: 2026-06-09 04:32:44.000000000 -0400 File: /mnt/lustre/d1/d2 Size: 7680 Blocks: 15 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272505020 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-09 04:32:44.000000000 -0400 Modify: 2026-06-09 04:32:45.000000000 -0400 Change: 2026-06-09 04:32:45.000000000 -0400 Birth: 2026-06-09 04:32:44.000000000 -0400 open(O_RDWR|O_CREAT): Is a directory PASS 54 (32s) == sanityn test 55a: rename vs unlink target dir ========= 04:32:57 (1780993977) fail_loc=0x80000156 mv: cannot move '/mnt/lustre/d55a.sanityn/d1/d2' to '/mnt/lustre/d55a.sanityn/d3/d2': No such file or directory PASS 55a (13s) == sanityn test 55b: rename vs unlink source dir ========= 04:33:10 (1780993990) fail_loc=0x80000156 mv: cannot stat '/mnt/lustre/d55b.sanityn/d1/d2': No such file or directory PASS 55b (13s) == sanityn test 55c: rename vs unlink orphan target dir == 04:33:23 (1780994003) fail_loc=0x156 mv: cannot move '/mnt/lustre/d55c.sanityn/d1/d2' to '/mnt/lustre/d55c.sanityn/d3/d2': No such file or directory PASS 55c (18s) == sanityn test 55d: rename file vs link ================= 04:33:41 (1780994021) fail_loc=0x155 mv: '/mnt/lustre/d55d.sanityn/f1' and '/mnt/lustre/d55d.sanityn/d55d.sanityn/f1' are the same file PASS 55d (16s) == sanityn test 55e: rename race AB/BA under the same parent dir ========================================================== 04:33:57 (1780994037) SKIP: sanityn test_55e needs >= 2 MDTs SKIP 55e (3s) == sanityn test 55f: rename: (P1/A -> P2/B) race with (P2/B -> P1/A) ========================================================== 04:34:00 (1780994040) fail_loc=0x155 We survived after P1A_P2B/P2B_P1A race test fail_loc=0x0 PASS 55f (21s) == sanityn test 55g: rename: race with trylock =========== 04:34:21 (1780994061) mdt.lustre-MDT0000.enable_rename_trylock=1 fail_loc=0x155 mv: cannot move '/mnt/lustre2/d55g.sanityn/b1/b2' to a subdirectory of itself, '/mnt/lustre2/d55g.sanityn/a1/a2/a3/a4/b2' We survived race with trylock test fail_loc=0x0 PASS 55g (26s) == sanityn test 56a: test llverdev with single large stripe ========================================================== 04:34:47 (1780994087) run_llverdev /mnt/lustre/f56a.sanityn -p -s 16000000 -c 4k llverdev: /mnt/lustre/f56a.sanityn is 16777216000000 bytes (15625 GB) in size timestamp: 1780994089 chunksize: 4096 size: 16777216000000 write offset: 16384000000kB 0.622848 MB/s read offset: 16384000000kB inf MB/s llverdev: data verified successfully PASS 56a (108s) == sanityn test 56b: test llverdev and partial verify of wide stripe file ========================================================== 04:36:35 (1780994195) run_llverdev /mnt/lustre/f56b.sanityn -p -s 32000000 -c 4k llverdev: /mnt/lustre/f56b.sanityn is 33554432000000 bytes (31250 GB) in size timestamp: 1780994201 chunksize: 4096 size: 33554432000000 write offset: 32768000000kB 1.17379 MB/s read offset: 32768000000kB inf MB/s llverdev: data verified successfully PASS 56b (119s) == sanityn test 60: Verify data_version behaviour ======== 04:38:35 (1780994315) PASS 60 (9s) == sanityn test 70a: cd directory && rm directory ======== 04:38:44 (1780994324) PASS 70a (8s) == sanityn test 70b: remove files after calling rm_entry ========================================================== 04:38:52 (1780994332) PASS 70b (9s) == sanityn test 71a: correct file map just after write operation is finished ========================================================== 04:39:01 (1780994341) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.00957641 s, 6.8 MB/s SKIP: sanityn test_71a ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71a (3s) == sanityn test 71b: check fiemap support for stripecount > 1 ========================================================== 04:39:04 (1780994344) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0259814 s, 2.5 MB/s SKIP: sanityn test_71b ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71b (4s) == sanityn test 71c: check FIEMAP_EXTENT_LAST flag with different extents number ========================================================== 04:39:08 (1780994348) SKIP: sanityn test_71c support only ldiskfs ost SKIP 71c (3s) == sanityn test 71d: fiemap corruption test with fm_extent_count=0 ========================================================== 04:39:11 (1780994351) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0209935 s, 3.1 MB/s SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS SKIP 71d (4s) == sanityn test 72: getxattr/setxattr cache should be consistent between nodes ========================================================== 04:39:15 (1780994355) llite.lustre-ffff8a2a59a58800.xattr_cache=1 llite.lustre-ffff8a2a60524000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names user.attr1="value1" getfattr: Removing leading '/' from absolute path names user.attr1="value2" getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names b362740ad371eae580e7f85452e8830b - 870e072da1ebfc20ea8c0cd3d6bdf3b6 - PASS 72 (8s) == sanityn test 73: getxattr should not cause xattr lock cancellation ========================================================== 04:39:23 (1780994363) llite.lustre-ffff8a2a59a58800.xattr_cache=1 llite.lustre-ffff8a2a60524000.xattr_cache=1 getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f73.sanityn user.attr1="value1" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre2/f73.sanityn user.attr1="value1" PASS 73 (8s) == sanityn test 74: flock deadlock: different mounts ======================================================================== 04:39:31 (1780994371) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00984783 s, 104 kB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.024281 s, 42.2 kB/s 229997: get lock2 229998: get lock1 229998: done 229998: get lock2 229997: done 229997: get lock1 fcntl cmd 7 failed: Resource deadlock avoided 229997: cannot get lock1: Resource deadlock avoided 229998: done 229998: exit rc=0 229997: done 229997: exit rc=0 PASS 74 (12s) == sanityn test 75: osc: upcall after unuse lock============================================================================= 04:39:43 (1780994383) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0297748 s, 70.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249837 s, 42.0 MB/s fail_loc=0x31d /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 4264: 230579 Killed stat -c %s $DIR1/$tfile fail_loc=0 PASS 75 (11s) == sanityn test 76: Verify MDT open_files listing ======== 04:39:54 (1780994394) opening files: ........ closing files: ........ PASS 76 (66s) == sanityn test 77a: check FIFO NRS policy =============== 04:41:00 (1780994460) ost.OSS.ost.nrs_policies=fifo ost.OSS.ost_create.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_out.nrs_policies=fifo ost.OSS.ost_seq.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.25822 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233631 s, 44.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0354234 s, 29.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584126 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486138 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331447 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532889 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132382 s, 7.9 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112019 s, 9.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083468 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116207 s, 9.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858316 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144126 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160613 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149207 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174144 s, 6.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8a2a59a58800.lock_unused_count=4 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180018 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00671489 s, 156 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0209063 s, 50.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608707 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0449342 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688581 s, 15.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130692 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0556306 s, 18.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0642135 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0802161 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0831228 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153077 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711004 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15702 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730877 s, 14.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915942 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118414 s, 8.9 MB/s removed '/mnt/lustre/d77a.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77a.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77a.sanityn' PASS 77a (10s) == sanityn test 77b: check CRR-N NRS policy ============== 04:41:10 (1780994470) ost.OSS.ost.nrs_policies=crrn ost.OSS.ost_create.nrs_policies=crrn ost.OSS.ost_io.nrs_policies=crrn ost.OSS.ost_out.nrs_policies=crrn ost.OSS.ost_seq.nrs_policies=crrn ost.OSS.ost.nrs_crrn_quantum=1 ost.OSS.ost_create.nrs_crrn_quantum=1 ost.OSS.ost_io.nrs_crrn_quantum=1 ost.OSS.ost_out.nrs_crrn_quantum=1 ost.OSS.ost_seq.nrs_crrn_quantum=1 policy: crr-n, crrn_quantum 1 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.230929 s, 72.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534111 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0656398 s, 16.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644718 s, 16.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509302 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0291442 s, 36.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131791 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880095 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123543 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118613 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0582775 s, 18.0 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8a2a59a58800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129154 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.266569 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29112 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256914 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264299 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182138 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583401 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0119898 s, 87.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0475751 s, 22.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486974 s, 21.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532471 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0027177 s, 386 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212169 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176316 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137261 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15951 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0173481 s, 60.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103174 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146386 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0789859 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527803 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0383222 s, 27.4 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost.nrs_crrn_quantum=64 ost.OSS.ost_create.nrs_crrn_quantum=64 ost.OSS.ost_io.nrs_crrn_quantum=64 ost.OSS.ost_out.nrs_crrn_quantum=64 ost.OSS.ost_seq.nrs_crrn_quantum=64 policy: crr-n, crrn_quantum 64 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.24716 s, 67.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364349 s, 28.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890801 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117996 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155312 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138872 s, 7.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123508 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130263 s, 8.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.139738 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22473 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195921 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.218646 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20467 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157451 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.258545 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191408 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263526 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0426084 s, 24.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0610852 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118716 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100756 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132899 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.094778 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801599 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142952 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115409 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120638 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117457 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151775 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132658 s, 7.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180412 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145866 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15725 s, 6.7 MB/s removed '/mnt/lustre/d77b.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77b.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77b.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77b (14s) == sanityn test 77c: check ORR NRS policy ================ 04:41:24 (1780994484) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_orr_offset_type=physical ost.OSS.ost_io.nrs_orr_supported=reads policy: orr, orr_quantum 1, orr_offset_type physical, orr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.353186 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0388532 s, 27.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110888 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144684 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217123 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22709 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.217803 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0646053 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268459 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28915 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160525 s, 6.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297975 s, 3.5 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240297 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185117 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326647 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.317277 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.321007 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8a2a59a58800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0227229 s, 46.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0331902 s, 31.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062107 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00430071 s, 244 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0267416 s, 39.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0437983 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.076502 s, 13.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0260396 s, 40.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0703867 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0136613 s, 76.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0219872 s, 47.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510165 s, 20.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183008 s, 57.3 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532681 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12314 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0330159 s, 31.8 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=writes ost.OSS.ost_io.nrs_orr_quantum=64 policy: orr, orr_quantum 64, orr_offset_type physical, orr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.592403 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0486063 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0635688 s, 16.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0577452 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594003 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200864 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766431 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.30405 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.308864 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205144 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205109 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.192942 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288971 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.378935 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0490236 s, 21.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.298469 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307018 s, 3.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0417053 s, 25.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0338516 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839105 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0496209 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0869335 s, 12.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.057976 s, 18.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0237193 s, 44.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0883292 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098967 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122122 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104637 s, 10.0 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0940501 s, 11.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102614 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107491 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10054 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115818 s, 9.1 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_orr_supported=reads_and_writes ost.OSS.ost_io.nrs_orr_offset_type=logical policy: orr, orr_quantum 64, orr_offset_type logical, orr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.273757 s, 61.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0550049 s, 19.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596933 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.090421 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0798509 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118185 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200253 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208846 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166651 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152941 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168588 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186069 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.20338 s, 5.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273898 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196168 s, 5.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250065 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313769 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0408043 s, 25.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0479416 s, 21.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677551 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0891408 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0249719 s, 42.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318151 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.045592 s, 23.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.147422 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0832593 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00488391 s, 215 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00794851 s, 132 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0336976 s, 31.1 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745438 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.069599 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0980586 s, 10.7 MB/s removed '/mnt/lustre/d77c.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77c.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77c.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77c (20s) == sanityn test 77d: check TRR nrs policy ================ 04:41:44 (1780994504) ost.OSS.ost_io.nrs_policies=trr ost.OSS.ost_io.nrs_trr_quantum=1 ost.OSS.ost_io.nrs_trr_offset_type=physical ost.OSS.ost_io.nrs_trr_supported=reads policy: trr, trr_quantum 1, trr_offset_type physical, trr_supported reads 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.291836 s, 57.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440662 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474617 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0745689 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12794 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158588 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121776 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0966959 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101405 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142296 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204067 s, 5.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132671 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234106 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240505 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171127 s, 6.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161247 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236491 s, 4.4 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8a2a59a58800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0176069 s, 59.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00251396 s, 417 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158327 s, 6.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175136 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0161826 s, 64.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124045 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143114 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00905396 s, 116 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117716 s, 8.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221623 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196444 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853426 s, 12.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.098695 s, 10.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0334514 s, 31.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0645518 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083644 s, 12.5 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=writes ost.OSS.ost_io.nrs_trr_quantum=64 policy: trr, trr_quantum 64, trr_offset_type physical, trr_supported writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.349154 s, 48.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0307041 s, 34.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0508995 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067386 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122445 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181681 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12165 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986383 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152737 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145359 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206383 s, 5.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362221 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28468 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208307 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.299072 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.33242 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380453 s, 2.8 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8a2a59a58800.lock_unused_count=6 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197875 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124456 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220458 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.201679 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.236366 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178728 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19035 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182626 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219819 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175023 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18317 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207048 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179911 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187194 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22479 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.202705 s, 5.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_trr_supported=reads_and_writes ost.OSS.ost_io.nrs_trr_offset_type=logical policy: trr, trr_quantum 64, trr_offset_type logical, trr_supported reads_and_writes 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.300547 s, 55.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0243511 s, 43.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541972 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301269 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371392 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0351945 s, 29.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0312424 s, 33.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0481488 s, 21.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182126 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160009 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115329 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164498 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185592 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165052 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104454 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126595 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162 s, 6.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8a2a59a58800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0576263 s, 18.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0600812 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0597967 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0542832 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.039956 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0683388 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0608068 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664673 s, 15.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0697263 s, 15.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0924823 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0659891 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102615 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0650736 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851438 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0844491 s, 12.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113881 s, 9.2 MB/s removed '/mnt/lustre/d77d.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77d.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77d.sanityn' ost.OSS.ost_io.nrs_policies=fifo PASS 77d (18s) == sanityn test 77e: check TBF NID nrs policy ============ 04:42:02 (1780994522) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_tbf_rule=start localhost nid={0@lo} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start clients nid={192.168.206.53@tcp} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start others nid={*.*.*.*@tcp} rate=50 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.307641 s, 54.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0137637 s, 76.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0239441 s, 43.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0316708 s, 33.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0372389 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563326 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0596718 s, 17.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776193 s, 13.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104492 s, 100 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067774 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113265 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142049 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154386 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143857 s, 7.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1349 s, 7.8 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8a2a59a58800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163092 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020058 s, 52.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402759 s, 26.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386412 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0920785 s, 11.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0314435 s, 33.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0484218 s, 21.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.047444 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421611 s, 24.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0689908 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724975 s, 14.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0723553 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124707 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761862 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143519 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00945385 s, 111 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194679 s, 5.4 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change localhost rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change clients rate=101 ost.OSS.ost_io.nrs_tbf_rule=change others rate=51 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.315681 s, 53.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0624261 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553567 s, 18.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223421 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184585 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186454 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207067 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148763 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.240254 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247926 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264611 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.224575 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.294616 s, 3.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.306063 s, 3.4 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.255565 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.31408 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249908 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.054336 s, 19.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607694 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.01015 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0113058 s, 92.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0799907 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0229014 s, 45.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0301312 s, 34.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0793418 s, 13.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107485 s, 9.8 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0746831 s, 14.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0736611 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112028 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128722 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0199664 s, 52.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207714 s, 50.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0218732 s, 47.9 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop localhost ost.OSS.ost_io.nrs_tbf_rule=stop clients ost.OSS.ost_io.nrs_tbf_rule=stop others 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.351063 s, 47.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359733 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970334 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833844 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129032 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102468 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143931 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172956 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223048 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.276381 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198537 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.295106 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225831 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263566 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.256954 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233109 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8a2a59a58800.lock_unused_count=7 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.382985 s, 2.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.063947 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0954281 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107351 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0101574 s, 103 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00544856 s, 192 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00283648 s, 370 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21032 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131098 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752339 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0155675 s, 67.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794648 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725059 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0485725 s, 21.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.017004 s, 61.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400156 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0533695 s, 19.6 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' ost.OSS.ost_io.nrs_policies=fifo 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.279367 s, 60.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0509201 s, 20.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0537765 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155218 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120565 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197187 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191367 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183527 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227396 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154195 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186908 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182625 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180844 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187746 s, 5.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8a2a59a58800.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254322 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23399 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.313247 s, 3.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00612221 s, 171 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0105546 s, 99.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00303584 s, 345 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00269173 s, 390 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00306289 s, 342 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00299109 s, 351 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00298214 s, 352 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.171492 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0215865 s, 48.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160636 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.351171 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119039 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123854 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10734 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0941673 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0424013 s, 24.7 MB/s removed '/mnt/lustre/d77e.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77e.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77e.sanityn' PASS 77e (29s) == sanityn test 77f: check TBF JobID nrs policy ========== 04:42:31 (1780994551) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start runas jobid={iozone.500 dd.500 tiotest.500} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start iozone_runas jobid={iozone.500} rate=100 ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.224302 s, 74.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0245109 s, 42.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0125729 s, 83.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.032129 s, 32.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0516733 s, 20.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0581468 s, 18.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0858613 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0979786 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.058475 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0668477 s, 15.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170508 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149318 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216158 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208507 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157363 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.241912 s, 4.3 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=6 ldlm.namespaces.lustre-OST0001-osc-ffff8a2a59a58800.lock_unused_count=2 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.281599 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0104665 s, 100 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0277878 s, 37.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0206667 s, 50.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0384883 s, 27.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116064 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717231 s, 14.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152918 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0214918 s, 48.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0207112 s, 50.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168104 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0235832 s, 44.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00903584 s, 116 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991413 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00712311 s, 147 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0280479 s, 37.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113707 s, 9.2 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=change runas rate=1001 ost.OSS.ost_io.nrs_tbf_rule=change iozone_runas rate=101 ost.OSS.ost_io.nrs_tbf_rule=change dd_runas rate=51 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.197933 s, 84.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0545281 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0923552 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120823 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113891 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125562 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127707 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140299 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13471 s, 7.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.137768 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181346 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189255 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163366 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165397 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177682 s, 5.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198575 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176372 s, 5.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0369971 s, 28.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0196594 s, 53.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326294 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0629869 s, 16.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379242 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0393746 s, 26.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0414332 s, 25.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0440493 s, 23.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0702156 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.049793 s, 21.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0442384 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0501448 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0512202 s, 20.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0499316 s, 21.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0462071 s, 22.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0510829 s, 20.5 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_tbf_rule=stop runas ost.OSS.ost_io.nrs_tbf_rule=stop iozone_runas ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.255555 s, 65.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0685243 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0309664 s, 33.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187653 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0328218 s, 31.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0353111 s, 29.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0294929 s, 35.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0491185 s, 21.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0347427 s, 30.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123512 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203251 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189388 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.203417 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.175097 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191415 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242966 s, 4.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167007 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00458025 s, 229 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108211 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11638 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108961 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120918 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119703 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131458 s, 8.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0845079 s, 12.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700467 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101443 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797867 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0438514 s, 23.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0183449 s, 57.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.166254 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185614 s, 5.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=2 ldlm.namespaces.lustre-OST0001-osc-ffff8a2a59a58800.lock_unused_count=10 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196514 s, 5.3 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' ost.OSS.ost_io.nrs_policies=fifo running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77f.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.370561 s, 45.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0520832 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.020544 s, 51.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0780694 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0658473 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819772 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173654 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0875058 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221729 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.163053 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207467 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220272 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.243096 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232284 s, 4.5 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=4 ldlm.namespaces.lustre-OST0001-osc-ffff8a2a59a58800.lock_unused_count=4 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.223121 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205895 s, 5.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.466085 s, 2.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0106518 s, 98.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15107 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110211 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0341014 s, 30.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126929 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639101 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00331777 s, 316 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00563137 s, 186 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8a2a59a58800.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0162791 s, 64.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0058473 s, 179 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00990329 s, 106 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0103427 s, 101 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0191144 s, 54.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299522 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0281625 s, 37.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00677585 s, 155 MB/s removed '/mnt/lustre/d77f.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77f.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77f.sanityn' PASS 77f (26s) == sanityn test 77g: Change TBF type directly ============ 04:42:57 (1780994577) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.500} rate=50 ost.OSS.ost_io.nrs_policies=fifo PASS 77g (13s) == sanityn test 77h: Wrong policy name should report error, not LBUG ========================================================== 04:43:10 (1780994590) oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=abc: No such file or directory oleg653-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='abc': No such file or directory pdsh@oleg653-client: oleg653-server: ssh exited with exit code 2 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc: Invalid argument oleg653-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf reg abc: Invalid argument oleg653-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf reg abc': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_policies=tbf abc efg: Invalid argument oleg653-server: error: set_param: setting 'ost/OSS/ost_io/nrs_policies'='tbf abc efg': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 PASS 77h (14s) == sanityn test 77i: Change rank of TBF rule ============= 04:43:24 (1780994604) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=start before jobid={jobid} rate=1000 ost.OSS.ost_io.nrs_tbf_rule=start after jobid={jobid} rate=1000 rank=default ost.OSS.ost_io.nrs_tbf_rule=start target jobid={jobid} rate=1000 rank=after Move before itself ost.OSS.ost_io.nrs_tbf_rule=change target rank=target Move to higher rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=before Move to lower rank ost.OSS.ost_io.nrs_tbf_rule=change target rank=after Move before default ost.OSS.ost_io.nrs_tbf_rule=change target rank=default ost.OSS.ost_io.nrs_policies=fifo PASS 77i (27s) == sanityn test 77j: check TBF-OPCode NRS policy ========= 04:43:51 (1780994631) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf opcode ost.OSS.ost_io.nrs_tbf_rule=start ost_r opcode={ost_read} rate=5 ost.OSS.ost_io.nrs_tbf_rule=start ost_w opcode={ost_write} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.522921 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.034447 s, 30.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0240701 s, 43.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0605052 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0730147 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0806812 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0553809 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0909776 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176559 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169635 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.207774 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.189341 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.193011 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214599 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.24072 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197051 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.226134 s, 4.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584746 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752468 s, 13.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0587382 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839865 s, 12.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118474 s, 8.9 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0721836 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105698 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107425 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917204 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0716532 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118239 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0986709 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105873 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122717 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122032 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127489 s, 8.2 MB/s removed '/mnt/lustre/d77j.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77j.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77j.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 5 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.06109 s, 11.6 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.6353 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77j.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_r ost.OSS.ost_io.nrs_tbf_rule=stop ost_w ost.OSS.ost_io.nrs_policies=fifo PASS 77j (71s) == sanityn test 77ja: check TBF-UID/GID NRS policy ======= 04:45:02 (1780994702) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg653-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg653-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg653-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg653-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30818 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0609103 s, 17.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0130102 s, 80.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0959863 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819627 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0604266 s, 17.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155854 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.178333 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.159274 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.200579 s, 5.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257117 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263313 s, 4.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263543 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263547 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.26914 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27466 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.301327 s, 3.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0640162 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573898 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0805426 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0534245 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0554234 s, 18.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573284 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059231 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0715118 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0531016 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.111672 s, 9.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0694612 s, 15.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124151 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103297 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116507 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127428 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119181 s, 8.8 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.98 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0046 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg653-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg653-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/d77ja.sanityn/nrs_r_oleg653-client.virtnet] [/mnt/lustre/d77ja.sanityn/nrs_w_oleg653-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.29029 s, 13.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801865 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0765596 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.086319 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0808433 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138529 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112999 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245558 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.260895 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.250782 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185247 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182444 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190376 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289722 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221583 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.289467 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263041 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0915542 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0794471 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.070053 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118139 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0871626 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.092082 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103244 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0917586 s, 11.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108464 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0969992 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108066 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121869 s, 8.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0999246 s, 10.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114619 s, 9.1 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125228 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123053 s, 8.5 MB/s removed '/mnt/lustre/d77ja.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77ja.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77ja.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d77ja.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ja.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.1962 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ja.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9732 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77ja.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77ja (159s) == sanityn test 77jb: check TBF-UID/GID NRS policy on files that don't belong to us ========================================================== 04:47:41 (1780994861) jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf uid ost.OSS.ost_io.nrs_tbf_rule=start ost_uid uid={500} rate=5 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg653-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg653-client.virtnet] running as uid/gid/euid/egid 0/500/0/500, groups: 500 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg653-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg653-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30832 s, 12.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124498 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117923 s, 8.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109321 s, 9.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0335871 s, 31.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113569 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105977 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.199206 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173639 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861067 s, 12.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.180061 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.29057 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.25686 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.380883 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.324822 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.371434 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.249756 s, 4.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0791192 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11299 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113958 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118056 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10214 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0514049 s, 20.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0888364 s, 11.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.116967 s, 9.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117299 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105151 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100736 s, 10.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103754 s, 10.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0585217 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0782113 s, 13.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0797082 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123952 s, 8.5 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 0/500/0/500, groups: 500 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0207 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8942 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_uid ost.OSS.ost_io.nrs_policies=fifo jobid_var=procname_uid ost.OSS.ost_io.nrs_policies=tbf gid ost.OSS.ost_io.nrs_tbf_rule=start ost_gid gid={500} rate=5 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg653-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg653-client.virtnet] running as uid/gid/euid/egid 500/0/500/0, groups: 0 [chmod] [777] [/mnt/lustre/d77jb.sanityn/nrs_r_oleg653-client.virtnet] [/mnt/lustre/d77jb.sanityn/nrs_w_oleg653-client.virtnet] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.30407 s, 12.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0809394 s, 13.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0818313 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067659 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775232 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185302 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196991 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.185665 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164174 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.212836 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188927 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205153 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.205099 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.220399 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247497 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293981 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.288607 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801263 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00991566 s, 106 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0738695 s, 14.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0495457 s, 21.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0364238 s, 28.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112689 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0718234 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0521014 s, 20.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056082 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.016139 s, 65.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.075537 s, 13.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0708248 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0929772 s, 11.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0776728 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109544 s, 9.6 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113509 s, 9.2 MB/s removed '/mnt/lustre/d77jb.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77jb.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77jb.sanityn' cpu_npartitions on ost1 is 1 running as uid/gid/euid/egid 500/0/500/0, groups: 0 [touch] [/mnt/lustre/d77jb.sanityn/tbf] Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77jb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0958 s, 5.2 MB/s Write runtime is 22 s, speed is 4.545454 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77jb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9894 s, 5.2 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77jb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ost_gid ost.OSS.ost_io.nrs_policies=fifo PASS 77jb (165s) == sanityn test 77k: check TBF policy with UID/GID/JobID/OPCode expression ========================================================== 04:50:27 (1780995027) ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.731171 s, 22.9 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0626442 s, 16.7 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0428297 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752288 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.117105 s, 9.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182671 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.196314 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120748 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198727 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167204 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195179 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15095 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.237493 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144148 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257125 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165828 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174931 s, 6.0 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=1 ldlm.namespaces.lustre-OST0001-osc-ffff8a2a59a58800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0439917 s, 23.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0465446 s, 22.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0574136 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0813009 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0655527 s, 16.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.041427 s, 25.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504704 s, 20.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0220534 s, 47.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.093768 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820063 s, 12.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852066 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0851846 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.110774 s, 9.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170376 s, 6.2 MB/s1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128183 s, 8.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0253773 s, 41.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.68891 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.98963 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.53@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.53@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.467612 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0329562 s, 31.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0292189 s, 35.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.077037 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108594 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.131131 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0991585 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140047 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127623 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.128984 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15431 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107256 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148003 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179269 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204131 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181261 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.187118 s, 5.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.038524 s, 27.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0251468 s, 41.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0953023 s, 11.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0423349 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0567738 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0472974 s, 22.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0504149 s, 20.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0607155 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.103052 s, 10.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0766472 s, 13.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0676096 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717052 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0880804 s, 11.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0939731 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0850894 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935931 s, 11.2 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.3863 s, 9.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0371 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.53@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.473208 s, 35.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0242887 s, 43.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0371208 s, 28.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162441 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157589 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17246 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.179053 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198017 s, 5.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19688 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.219664 s, 4.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.216248 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.229189 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2196 s, 4.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.225345 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.228125 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.233682 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262011 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0627119 s, 16.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0405598 s, 25.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0187561 s, 55.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0660749 s, 15.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0783691 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687668 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211934 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0348019 s, 30.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598994 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0474936 s, 22.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450353 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0665573 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0356315 s, 29.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0523567 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0324867 s, 32.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.036023 s, 29.1 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.66523 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.28831 s, 19.8 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.920901 s, 18.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0688571 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125046 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127396 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.181541 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.164429 s, 6.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13194 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153584 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123229 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211804 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.194583 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.227669 s, 4.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.300547 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287777 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230563 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.354401 s, 3.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.374537 s, 2.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00432631 s, 242 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126153 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123107 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0639248 s, 16.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649072 s, 16.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833666 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.133308 s, 7.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0541449 s, 19.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614478 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706394 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079883 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114381 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926175 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0981659 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145632 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130541 s, 8.0 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.0568 s, 8.7 MB/s Write runtime is 14 s, speed is 7.142857 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.94429 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.74683 s, 10.8 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.90539 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.33063 s, 50.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0211746 s, 49.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0527781 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717906 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0379627 s, 27.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0402125 s, 26.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786725 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0867181 s, 12.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0664564 s, 15.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0861437 s, 12.2 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.169005 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.17719 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184392 s, 5.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0852123 s, 12.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710866 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149491 s, 7.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170935 s, 6.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0617114 s, 17.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0804123 s, 13.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0777483 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710617 s, 14.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0910713 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0546784 s, 19.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0796595 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10016 s, 10.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0849066 s, 12.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0814322 s, 12.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0711531 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0801126 s, 13.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925703 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114911 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105934 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122323 s, 8.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.8901 s, 5.3 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.7058 s, 5.3 MB/s Read runtime is 20 s, speed is 5.000000 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_ug ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.40815 s, 41.1 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0675907 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0592222 s, 17.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119334 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0841752 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0686817 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114362 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14437 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142084 s, 7.4 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120446 s, 8.7 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126852 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.153318 s, 6.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.158398 s, 6.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.214087 s, 4.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155988 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.230514 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211267 s, 5.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00272216 s, 385 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00556377 s, 188 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0570601 s, 18.4 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00812358 s, 129 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0166572 s, 63.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0657979 s, 15.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0146391 s, 71.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0820315 s, 12.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00554576 s, 189 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00846026 s, 124 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0819252 s, 12.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.097192 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0885441 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0632351 s, 16.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0057128 s, 184 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0853675 s, 12.3 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.9 s, 9.6 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95195 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.312992 s, 53.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0710426 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0532044 s, 19.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672719 s, 15.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118893 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0862041 s, 12.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614933 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0752361 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216804 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614561 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.120577 s, 8.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118109 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0951289 s, 11.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.150514 s, 7.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177574 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264624 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264042 s, 4.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0927584 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0970717 s, 10.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0644541 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978958 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101656 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.135781 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107969 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.143733 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109517 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0877369 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12379 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0914385 s, 11.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.123921 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105061 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140064 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122454 s, 8.6 MB/s removed '/mnt/lustre/d77k.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77k.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77k.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.9483 s, 8.8 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0383 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77k.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.1024 s, 7.4 MB/s Write runtime is 16 s, speed is 6.250000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77k.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.88699 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77k.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_a ost.OSS.ost_io.nrs_tbf_rule=stop ext_b ost.OSS.ost_io.nrs_policies=fifo PASS 77k (455s) == sanityn test 77kb: Check different granular TBF type combination: jobid+opcode ========================================================== 04:58:02 (1780995482) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.00751 s, 16.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.059803 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.132998 s, 7.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127213 s, 8.2 MB/s1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.142475 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.062401 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130355 s, 8.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165136 s, 6.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113371 s, 9.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124538 s, 84.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0362406 s, 28.9 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238491 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.272724 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211739 s, 5.0 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.242543 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.174745 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.290127 s, 3.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.027041 s, 38.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0279497 s, 37.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0700393 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0359892 s, 29.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0427789 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0400747 s, 26.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10806 s, 9.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0469211 s, 22.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769959 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0558143 s, 18.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0594835 s, 17.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0614653 s, 17.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0714139 s, 14.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00806468 s, 130 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0988223 s, 10.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0811549 s, 12.9 MB/s removed '/mnt/lustre/d77kb.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77kb.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kb.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kb.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 8.63283 s, 12.1 MB/s Write runtime is 10 s, speed is 10.000000 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kb.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9646 s, 10.5 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kb.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kb (56s) == sanityn test 77kc: Check different granular TBF type combination: nid+opcode ========================================================== 04:58:59 (1780995539) ost.OSS.ost_io.nrs_policies=tbf nid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w nid={0@lo 192.168.206.53@tcp}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r nid={0@lo 192.168.206.53@tcp}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.48503 s, 34.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0528178 s, 19.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0535608 s, 19.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0833005 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0901255 s, 11.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0621005 s, 16.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0925225 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.113828 s, 9.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106179 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144898 s, 7.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.157002 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.162035 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170655 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.151047 s, 6.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.160541 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145467 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.176741 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0773872 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00719762 s, 146 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0564945 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.136771 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0568277 s, 18.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.082776 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.015073 s, 69.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126568 s, 82.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173935 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.106041 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107256 s, 9.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.115698 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0181101 s, 57.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.118356 s, 8.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.188496 s, 5.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134029 s, 7.8 MB/s removed '/mnt/lustre/d77kc.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77kc.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kc.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.95644 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93571 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kc.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kc (57s) == sanityn test 77kd: Check different granular TBF type combination: nid+jobid ========================================================== 04:59:56 (1780995596) ost.OSS.ost_io.nrs_policies=tbf nid+jobid ost.OSS.ost_io.nrs_tbf_rule=start ext nid={0@lo 192.168.206.53@tcp}&jobid={dd.500} rate=20 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.516323 s, 32.5 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0524841 s, 20.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0216497 s, 48.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0584785 s, 17.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.114974 s, 9.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0337616 s, 31.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0179411 s, 58.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0268132 s, 39.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146781 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195261 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.222043 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.322834 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362046 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247314 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.461122 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.462593 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.251391 s, 4.2 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a59a58800.lock_unused_count=7 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.040899 s, 25.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0698906 s, 15.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0727835 s, 14.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0724565 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0421984 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.168382 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839285 s, 12.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0744264 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0774881 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126874 s, 8.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101633 s, 10.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0965637 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108487 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.10919 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.125058 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165324 s, 6.3 MB/s removed '/mnt/lustre/d77kd.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77kd.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kd.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kd.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.4622 s, 9.1 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kd.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.37725 s, 19.5 MB/s Read runtime is 6 s, speed is 16.666666 IOPS cleanup_tbf /mnt/lustre/d77kd.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kd (58s) == sanityn test 77ke: Check different granular TBF type combination: jobid+opcode ========================================================== 05:00:54 (1780995654) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_a jobid={dd.500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b jobid={dd.500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.639554 s, 26.2 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0299475 s, 35.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0893145 s, 11.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.03378 s, 31.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0158267 s, 66.3 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0450249 s, 23.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0266374 s, 39.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.186636 s, 5.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146781 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0443169 s, 23.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0319465 s, 32.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.27092 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.16039 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.247836 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.18167 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.248071 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.345145 s, 3.0 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.096123 s, 10.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.100508 s, 10.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107466 s, 9.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.155479 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0775993 s, 13.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122788 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129789 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177963 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.141044 s, 7.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.161989 s, 6.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144503 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.12281 s, 8.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.170957 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165967 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.183475 s, 5.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.190064 s, 5.5 MB/s removed '/mnt/lustre/d77ke.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77ke.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77ke.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77ke.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 14.1736 s, 7.4 MB/s Write runtime is 15 s, speed is 6.666666 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77ke.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0389 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.56527 s, 11.0 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.93534 s, 10.6 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77ke.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77ke (105s) == sanityn test 77kf: Check different granular TBF type combination: uid+gid ========================================================== 05:02:39 (1780995759) ost.OSS.ost_io.nrs_policies=tbf uid+gid ost.OSS.ost_io.nrs_tbf_rule=start ext_ug uid={500}&gid={500} rate=5 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.09018 s, 15.4 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0573605 s, 18.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0583407 s, 18.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.167924 s, 6.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.152545 s, 6.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.1948 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232901 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.2131 s, 4.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221341 s, 4.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.253251 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234861 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.257485 s, 4.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314035 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.314175 s, 3.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.355474 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.343187 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.40057 s, 2.6 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0701478 s, 14.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0741959 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0717691 s, 14.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0786848 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0890433 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0670888 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0876686 s, 12.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0937605 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.146478 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105398 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11336 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0926447 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.130103 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122221 s, 8.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.124494 s, 8.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15455 s, 6.8 MB/s removed '/mnt/lustre/d77kf.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77kf.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kf.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 5, read rate: 5 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kf.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 20.0183 s, 5.2 MB/s Write runtime is 21 s, speed is 4.761904 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kf.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 19.9663 s, 5.3 MB/s Read runtime is 21 s, speed is 4.761904 IOPS cleanup_tbf /mnt/lustre/d77kf.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kf (80s) == sanityn test 77kg: check different granular TBF type combination: uid+opcode ========================================================== 05:03:59 (1780995839) ost.OSS.ost_io.nrs_policies=tbf uid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_uw uid={500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_ur uid={500}&opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.753847 s, 22.3 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0563582 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0934452 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.145801 s, 7.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182133 s, 5.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.246739 s, 4.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.172121 s, 6.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.22814 s, 4.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19415 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.262591 s, 4.0 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195435 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264426 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.23377 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.254757 s, 4.1 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270808 s, 3.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270571 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.326646 s, 3.2 MB/s ldlm.namespaces.lustre-OST0001-osc-ffff8a2a59a58800.lock_unused_count=1 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.056183 s, 18.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.129871 s, 8.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.148122 s, 7.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127632 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0649895 s, 16.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706991 s, 14.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.119248 s, 8.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0835062 s, 12.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0947083 s, 11.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0962234 s, 10.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.140556 s, 7.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105214 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107801 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.208483 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0928224 s, 11.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127449 s, 8.2 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.99606 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0387 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_uw ost.OSS.ost_io.nrs_tbf_rule=stop ext_ur ost.OSS.ost_io.nrs_tbf_rule=start ext_a uid={500},opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_b uid={500},opcode={ost_read} rate=10 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/nrs_r_oleg653-client.virtnet] [bs=1M] [count=16] 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.676729 s, 24.8 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_w] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0318043 s, 33.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0538296 s, 19.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0935476 s, 11.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0725044 s, 14.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.067952 s, 15.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.138298 s, 7.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.244729 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206451 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.263799 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.273478 s, 3.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.182089 s, 5.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286637 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.359665 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.338766 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404928 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286133 s, 3.7 MB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [declare] [-a] [pids_r] execvp fails running declare (13): Permission denied 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.06254 s, 16.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0672187 s, 15.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0643766 s, 16.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.037237 s, 28.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105675 s, 9.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0684766 s, 15.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.050134 s, 20.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0565628 s, 18.5 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0606662 s, 17.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0677202 s, 15.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.107716 s, 9.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.14452 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.078101 s, 13.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761611 s, 13.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0976168 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0839525 s, 12.5 MB/s removed '/mnt/lustre/d77kg.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77kg.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kg.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 10, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.3002 s, 8.5 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1358 s, 10.3 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77kg.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.4452 s, 9.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77kg.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.0568 s, 10.4 MB/s Read runtime is 11 s, speed is 9.090909 IOPS cleanup_tbf /mnt/lustre/d77kg.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete PASS 77kg (156s) == sanityn test 77kh: Verify that Project ID support for NRS TBF rule ========================================================== 05:06:36 (1780995996) using SAVE_PROJECT_SUPPORTED=0 Stopping clients: oleg653-client.virtnet /mnt/lustre (opts:) Stopping client oleg653-client.virtnet /mnt/lustre opts: Stopping clients: oleg653-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg653-client.virtnet /mnt/lustre2 opts: Stopping /mnt/lustre-mds1 (opts:-f) on oleg653-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg653-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg653-server sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) devtmpfs on /dev type devtmpfs (rw,nosuid,mode=755) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755) tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb) cgroup on /sys/fs/cgroup/misc type cgroup (rw,nosuid,nodev,noexec,relatime,misc) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) none on /sys/kernel/tracing type tracefs (rw,relatime) configfs on /sys/kernel/config type configfs (rw,relatime) /dev/nbd0 on / type ext4 (ro,relatime,stripe=32) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) debugfs on /sys/kernel/debug type debugfs (rw,relatime) mqueue on /dev/mqueue type mqueue (rw,relatime) none on /mnt type ramfs (rw,relatime) /dev/vda on /home/green/git/lustre-release type squashfs (ro,relatime) none on /var/lib/stateless/writable type tmpfs (rw,relatime) none on /var/cache/man type tmpfs (rw,relatime) none on /var/log type tmpfs (rw,relatime) none on /var/lib/dbus type tmpfs (rw,relatime) none on /tmp type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) none on /var/tmp type tmpfs (rw,relatime) none on /var/lib/NetworkManager type tmpfs (rw,relatime) none on /var/lib/systemd/random-seed type tmpfs (rw,relatime) none on /var/spool type tmpfs (rw,relatime) none on /var/lib/nfs type tmpfs (rw,relatime) none on /var/lib/gssproxy type tmpfs (rw,relatime) none on /var/lib/logrotate type tmpfs (rw,relatime) none on /etc type tmpfs (rw,relatime) none on /var/lib/rsyslog type tmpfs (rw,relatime) none on /root/.ssh type tmpfs (rw,relatime) none on /var/lib/dhclient type tmpfs (rw,relatime) 192.168.200.253:/exports/state/oleg653-client.virtnet on /var/lib/stateless/state type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.53,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg653-client.virtnet/boot on /boot type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.53,local_lock=none,addr=192.168.200.253) 192.168.200.253:/exports/state/oleg653-client.virtnet/etc/kdump.conf on /etc/kdump.conf type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.53,local_lock=none,addr=192.168.200.253) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=378180k,mode=700) 192.168.200.253:/exports/testreports/64863/testresults/sanityn-zfs-rocky8.10_x86_64-rocky8.10_x86_64 on /tmp/testlogs type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.206.53,local_lock=none,addr=192.168.200.253) /dev/vda on /usr/sbin/mount.lustre type squashfs (ro,relatime) Checking servers environments Checking clients oleg653-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg653-server' oleg653-server: oleg653-server.virtnet: executing load_modules_local oleg653-server: Loading modules from /home/green/git/lustre-release/lustre oleg653-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg653-server: detected 4 online CPUs by sysfs oleg653-server: MODOPTS_LIBCFS= oleg653-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg653-server: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg653-client: oleg653-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg653-server: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg653-client: oleg653-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg653-server: oleg653-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg653-client: oleg653-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre Starting client oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre Started clients oleg653-client.virtnet: 192.168.206.153@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre2 Starting client oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre2 Started clients oleg653-client.virtnet: 192.168.206.153@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8a2a48e8c000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8a2a7302d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a2a48e8c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a2a7302d000.idle_timeout=debug disable quota as required ost.OSS.ost_io.nrs_policies=tbf projid ost.OSS.ost_io.nrs_tbf_rule=start ext_proj_rw projid={100} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.68438 s, 24.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104825 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.13537 s, 7.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.191791 s, 5.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.238069 s, 4.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.19641 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234881 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.21945 s, 4.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293468 s, 3.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.282126 s, 3.7 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a48e8c000.lock_unused_count=5 ldlm.namespaces.lustre-OST0001-osc-ffff8a2a48e8c000.lock_unused_count=5 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.245363 s, 4.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.329912 s, 3.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.271305 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.310112 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.334797 s, 3.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.379298 s, 2.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.293437 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.529199 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.553941 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.518304 s, 2.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.541322 s, 1.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.541517 s, 1.9 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.552927 s, 1.9 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.532115 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.531166 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.534874 s, 2.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.576491 s, 1.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.583225 s, 1.8 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.5393 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.55654 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.62586 s, 1.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.561429 s, 1.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.588032 s, 1.8 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.9597 s, 10.5 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.52136 s, 19.0 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_proj_rw ost.OSS.ost_io.nrs_policies=fifo ost.OSS.ost_io.nrs_policies=tbf projid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w projid={100}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r projid={100}&opcode={ost_read} rate=10 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.677865 s, 24.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0742025 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.122351 s, 8.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.144122 s, 7.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.184115 s, 5.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280239 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.195824 s, 5.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.234016 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.177021 s, 5.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.286526 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287239 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.279663 s, 3.7 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.232604 s, 4.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287209 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.221655 s, 4.7 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.268949 s, 3.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.270005 s, 3.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a48e8c000.lock_unused_count=1 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0236991 s, 44.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0823866 s, 12.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0753604 s, 13.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0326972 s, 32.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0100931 s, 104 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0233999 s, 44.8 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.011904 s, 88.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120619 s, 86.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00833856 s, 126 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0978318 s, 10.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0788307 s, 13.3 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00960358 s, 109 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0230653 s, 45.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.112293 s, 9.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00448869 s, 234 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101612 s, 10.3 MB/s removed '/mnt/lustre/d77kh.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77kh.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77kh.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 10 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 12.4486 s, 8.4 MB/s Write runtime is 13 s, speed is 7.692307 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 10.1594 s, 10.3 MB/s Read runtime is 12 s, speed is 8.333333 IOPS cleanup_tbf /mnt/lustre/d77kh.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77kh (231s) == sanityn test 77ki: Add rule with unsupported TBF type should fail for generic TBF ========================================================== 05:10:26 (1780996226) ost.OSS.ost_io.nrs_policies=tbf jobid+opcode ost.OSS.ost_io.nrs_tbf_rule=start ext_w jobid={dd.500}&opcode={ost_write} rate=20 ost.OSS.ost_io.nrs_tbf_rule=start ext_r jobid={dd.500}&opcode={ost_read} rate=10 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/ost/OSS/ost_io/nrs_tbf_rule=start ext_fail projid={100}&opcode={ost_write} rate=20: Invalid argument oleg653-server: error: set_param: setting 'ost/OSS/ost_io/nrs_tbf_rule'='start ext_fail projid={100}&opcode={ost_write} rate=20': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 ost.OSS.ost_io.nrs_tbf_rule=stop ext_w ost.OSS.ost_io.nrs_tbf_rule=stop ext_r ost.OSS.ost_io.nrs_policies=fifo PASS 77ki (21s) == sanityn test 77l: check the output of NRS policies for generic TBF ========================================================== 05:10:47 (1780996247) ost.OSS.ost_io.nrs_policies=tbf nid ost.OSS.ost_io.nrs_policies=tbf ost.OSS.ost_io.nrs_policies=fifo PASS 77l (12s) == sanityn test 77m: check NRS Delay slows write RPC processing ========================================================== 05:10:59 (1780996259) ost.OSS.ost_io.nrs_policies=delay ost.OSS.ost_io.nrs_delay_min=4 ost.OSS.ost_io.nrs_delay_max=4 ost.OSS.ost_io.nrs_delay_pct=100 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg653-client.virtnet] [bs=1M] [count=1] [oflag=direct] [conv=fdatasync] 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 3.50406 s, 299 kB/s running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77m.sanityn/nrs_delay_oleg653-client.virtnet] [bs=1M] [count=10] [oflag=direct] [conv=fdatasync] 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 40.0975 s, 262 kB/s ost.OSS.ost_io.nrs_policies=fifo PASS 77m (59s) == sanityn test 77n: check wildcard support for TBF JobID NRS policy ========================================================== 05:11:58 (1780996318) ost.OSS.ost_io.nrs_policies=tbf jobid ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={*.500} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.596572 s, 28.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0771896 s, 13.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0386335 s, 27.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0889411 s, 11.8 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.173613 s, 6.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.259148 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.156596 s, 6.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.285653 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.366161 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.397067 s, 2.6 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.404501 s, 2.6 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.45 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433476 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.444114 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.362345 s, 2.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.494346 s, 2.1 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a48e8c000.lock_unused_count=7 ldlm.namespaces.lustre-OST0001-osc-ffff8a2a48e8c000.lock_unused_count=5 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.433528 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.198836 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.211369 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.206427 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.079161 s, 13.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0451455 s, 23.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0562824 s, 18.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.101971 s, 10.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0598024 s, 17.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105163 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.108514 s, 9.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.197968 s, 5.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.109046 s, 9.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.165304 s, 6.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0761402 s, 13.8 MB/s1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0787085 s, 13.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.121804 s, 8.6 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg653-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg653-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/d77n.sanityn/tbf] [bs=1M] [count=100] [oflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 11.4524 s, 9.2 MB/s Write runtime is 12 s, speed is 8.333333 IOPS Verify the read rate is under TBF control running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/mnt/lustre/d77n.sanityn/tbf] [of=/dev/null] [bs=1M] [count=100] [iflag=direct] 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.53639 s, 18.9 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_tbf_rule=start dd_runas jobid={dd.*} rate=20 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.696386 s, 24.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.11369 s, 9.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.127638 s, 8.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0687808 s, 15.2 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.104726 s, 10.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.204439 s, 5.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.210164 s, 5.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.28803 s, 3.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.126852 s, 8.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.307359 s, 3.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.264237 s, 4.0 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.280389 s, 3.7 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.357431 s, 2.9 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff8a2a48e8c000.lock_unused_count=3 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.105786 s, 9.9 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0743104 s, 14.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.297713 s, 3.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.149131 s, 7.0 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.400656 s, 2.6 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.420352 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.434515 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.413173 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.411832 s, 2.5 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.429937 s, 2.4 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.481517 s, 2.2 MB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.444525 s, 2.4 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.470283 s, 2.2 MB/s 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.425362 s, 2.5 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.456263 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.459254 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.458337 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.456104 s, 2.3 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.497756 s, 2.1 MB/s 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.482168 s, 2.2 MB/s removed '/mnt/lustre/d77n.sanityn/nrs_r_oleg653-client.virtnet' removed '/mnt/lustre/d77n.sanityn/nrs_w_oleg653-client.virtnet' removed directory '/mnt/lustre/d77n.sanityn' cpu_npartitions on ost1 is 1 Limited write rate: 20, read rate: 20 Verify the write rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 9.48292 s, 11.1 MB/s Write runtime is 11 s, speed is 9.090909 IOPS Verify the read rate is under TBF control 100+0 records in 100+0 records out 104857600 bytes (105 MB, 100 MiB) copied, 5.64452 s, 18.6 MB/s Read runtime is 7 s, speed is 14.285714 IOPS cleanup_tbf /mnt/lustre/d77n.sanityn sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete ost.OSS.ost_io.nrs_tbf_rule=stop dd_runas ost.OSS.ost_io.nrs_policies=fifo PASS 77n (100s) == sanityn test 77o: Changing rank should not panic ====== 05:13:38 (1780996418) mds.MDS.mdt.nrs_policies=tbf nid mds.MDS.mdt.nrs_tbf_rule=start name nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=start name1 nid={192.168.*.*@tcp} rate=10000 mds.MDS.mdt.nrs_tbf_rule=change name1 rank=name mds.MDS.mdt.nrs_tbf_rule=stop name mds.MDS.mdt.nrs_policies=fifo PASS 77o (16s) == sanityn test 77q: Parallel TBF rule definitions should not panic ========================================================== 05:13:54 (1780996434) mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=start rule77q_2 uid={1000}&gid={1000} rate=100 mds.MDS.mdt.nrs_tbf_rule=start rule77q_1 uid={ 500 11 3}&gid={500 10 33 100 } rate=100 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_1 mds.MDS.mdt.nrs_tbf_rule=stop rule77q_2 mds.MDS.mdt.nrs_policies=fifo PASS 77q (127s) == sanityn test 77p: Check validity of rule names for TBF policies ========================================================== 05:16:01 (1780996561) mds.MDS.mdt.nrs_policies=tbf oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_overflo uid={500} rate=500: File name too long oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_overflo uid={500} rate=500': File name too long pdsh@oleg653-client: oleg653-server: ssh exited with exit code 36 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test_77p_empty: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test_77p_empty': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p@spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p@spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p.spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p.spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p~spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p~spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p#spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p#spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p/spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p/spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p^spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p^spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p%spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p%spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p*spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p*spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p;spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p;spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p,spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p,spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p?spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p?spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77pspec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p>spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p:spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p:spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p+spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p+spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p=spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p=spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p)spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p)spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p(spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p(spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p{spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p{spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p}spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p}spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p|spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p|spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p[spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p[spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p]spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p]spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p!spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p!spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p&spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p&spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p$spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p$spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p`spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p`spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 oleg653-server: error: set_param: setting /sys/kernel/debug/lustre/mds/MDS/mdt/nrs_tbf_rule=start test77p\spec uid={500} rate=500: Invalid argument oleg653-server: error: set_param: setting 'mds/MDS/mdt/nrs_tbf_rule'='start test77p\spec uid={500} rate=500': Invalid argument pdsh@oleg653-client: oleg653-server: ssh exited with exit code 22 mds.MDS.mdt.nrs_policies=fifo PASS 77p (41s) == sanityn test 77r: Change type of tbf policy at run time ========================================================== 05:16:43 (1780996603) mds.MDS.mdt.nrs_policies=tbf uid mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf gid mds.MDS.mdt.nrs_tbf_rule=start md_rule gid={500} rate=1 mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start md_rule jobid={*.500} rate=1 mds.MDS.mdt.nrs_policies=tbf mds.MDS.mdt.nrs_tbf_rule=start md_rule uid={500} rate=1 mds.MDS.mdt.nrs_policies=fifo check the number of requests in queue: fifo: 0 crrn: 0 tbf: 0 delay: 0 fifo: 0 crrn: 0 tbf: 0 delay: 0 cleanup 77r 268639 kill md_thread xargs PASS 77r (60s) == sanityn test 77s: Check TBF LRU shrinker ============== 05:17:43 (1780996663) jobid_name=%e.%p jobid_var=TEST_77s_JOBVAR mds.MDS.mdt.nrs_policies=tbf jobid mds.MDS.mdt.nrs_tbf_rule=start touch jobid={touch.*} rate=1000 mds.MDS.mdt.nrs_tbf_rule=start unlink jobid={unlink.*} rate=1000 touch jobid={touch.*} 1000, ref 10 touch jobid={touch.*} 1000, ref 0 unlink jobid={unlink.*} 1000, ref 10 unlink jobid={unlink.*} 1000, ref 0 mds.MDS.mdt.nrs_policies=fifo jobid_var=procname_uid jobid_name=%e.%u PASS 77s (82s) == sanityn test 78: Enable policy and specify tunings right away ========================================================== 05:19:05 (1780996745) ost.OSS.ost_io.nrs_policies=orr ost.OSS.ost_io.nrs_orr_quantum=1 ost.OSS.ost_io.nrs_policies=fifo PASS 78 (11s) == sanityn test 79: xattr: intent error ================== 05:19:16 (1780996756) File: /mnt/lustre/d79.sanityn Size: 512 Blocks: 1 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115238826935626 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-09 05:19:18.000000000 -0400 Modify: 2026-06-09 05:19:18.000000000 -0400 Change: 2026-06-09 05:19:18.000000000 -0400 Birth: 2026-06-09 05:19:18.000000000 -0400 fail_val=0 fail_loc=0x80000160 fail_val=0 fail_loc=0x80000131 # file: mnt/lustre/d79.sanityn trusted.name1="value1" PASS 79 (20s) == sanityn test 80a: migrate directory when some children is being opened ========================================================== 05:19:36 (1780996776) SKIP: sanityn test_80a needs >= 2 MDTs SKIP 80a (3s) == sanityn test 80b: Accessing directory during migration ========================================================== 05:19:39 (1780996779) SKIP: sanityn test_80b needs >= 2 MDTs SKIP 80b (4s) == sanityn test 81a: rename and stat under striped directory ========================================================== 05:19:43 (1780996783) SKIP: sanityn test_81a needs >= 2 MDTs SKIP 81a (3s) == sanityn test 81b: rename under striped directory doesn't deadlock ========================================================== 05:19:46 (1780996786) SKIP: sanityn test_81b We need at least 2 MDTs for this test SKIP 81b (4s) == sanityn test 81c: rename revoke LOOKUP lock for remote object ========================================================== 05:19:50 (1780996790) SKIP: sanityn test_81c needs >= 4 MDTs SKIP 81c (4s) == sanityn test 81d: parallel rename file cross-dir on same MDT ========================================================== 05:19:54 (1780996794) - open/close 621 (time 1780996810.73 total 10.02 last 61.99) - open/close 1236 (time 1780996820.74 total 20.02 last 61.48) - open/close 1895 (time 1780996830.74 total 30.02 last 65.87) total: 2500 open/close in 38.92 seconds: 64.24 ops/second snapshot_time 1780996938.406423831 secs.nsecs start_time 1780996797.529236027 secs.nsecs elapsed_time 140.877187804 secs.nsecs open 2505 samples [usecs] 141 17181 4327350 12708695098 close 2505 samples [usecs] 164 10983 1041188 1104755036 mknod 2501 samples [usecs] 616 14015 3776126 9711456400 unlink 2500 samples [usecs] 780 151107 8094042 188457125582 mkdir 2 samples [usecs] 980 2004 2984 4976416 rmdir 2 samples [usecs] 2438 10239 12677 110780965 rename 2500 samples [usecs] 500 68230 19101366 299650370894 getattr 3762 samples [usecs] 7 10522 259370 490116610 parallel_rename_file 2500 samples [usecs] 500 68230 19101366 299650370894 crossdir_rename 2500 samples [usecs] 500 68230 19101366 299650370894 PASS 81d (150s) == sanityn test 82: fsetxattr and fgetxattr on orphan files ========================================================== 05:22:24 (1780996944) multiop /mnt/lustre/f82.sanityn vO_ac TMPPIPE=/tmp/multiop_open_wait_pipe.7563 multiop /mnt/lustre2/f82.sanityn vo_Ac TMPPIPE=/tmp/multiop_open_wait_pipe.7563 PASS 82 (8s) == sanityn test 83: access striped directory while it is being created/unlinked ========================================================== 05:22:32 (1780996952) SKIP: sanityn test_83 needs >= 2 MDTs SKIP 83 (4s) == sanityn test 84: 0-nlink race in lu_object_find() ===== 05:22:36 (1780996956) fail_loc=0x8000060b PASS 84 (12s) == sanityn test 85: Lustre API root cache race =========== 05:22:49 (1780996969) Starting: llapi_root_test /mnt/lustre/f85.sanityn /mnt/lustre2/f85.sanityn Starting test test1 at 1780996970 Finishing test test1 at 1780996976 PASS 85 (14s) == sanityn test 90: open/create and unlink striped directory ========================================================== 05:23:03 (1780996983) SKIP: sanityn test_90 needs >= 2 MDTs SKIP 90 (5s) == sanityn test 91: chmod and unlink striped directory === 05:23:08 (1780996988) SKIP: sanityn test_91 needs >= 2 MDTs SKIP 91 (4s) == sanityn test 92: create remote directory under orphan directory ========================================================== 05:23:13 (1780996993) SKIP: sanityn test_92 needs >= 2 MDTs SKIP 92 (4s) == sanityn test 93: alloc_rr should not allocate on same ost ========================================================== 05:23:17 (1780996997) fail_loc=0x00000163 fail_loc=0x0 /mnt/lustre/f93.sanityn-1/file1 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5761 0x1681 0x240000400 1 5724 0x165c 0x280000400 /mnt/lustre2/f93.sanityn-2/file2 lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5725 0x165d 0x280000400 0 5762 0x1682 0x240000400 rc1=0 and rc2=0 PASS 93 (19s) == sanityn test 94: signal vs CP callback race =========== 05:23:36 (1780997016) osc.lustre-OST0000-osc-ffff8a2a48e8c000.idle_timeout=0 osc.lustre-OST0000-osc-ffff8a2a7302d000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8a2a48e8c000.idle_timeout=0 osc.lustre-OST0001-osc-ffff8a2a7302d000.idle_timeout=0 2+0 records in 2+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.320888 s, 25.5 kB/s fail_val=5 fail_loc=0x80000312 fail_val=6 fail_loc=0x80000329 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 2.99758 s, 1.4 kB/s /home/green/git/lustre-release/lustre/tests/sanityn.sh: line 6178: 285380 Alarm clock dd if=/dev/zero of=$DIR/$tfile conv=notrunc oflag=append bs=4k count=1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0467874 s, 87.5 kB/s osc.lustre-OST0000-osc-ffff8a2a48e8c000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8a2a7302d000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a2a48e8c000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8a2a7302d000.idle_timeout=debug PASS 94 (16s) == sanityn test 95a: Check readpage() on a page that was removed from page cache ========================================================== 05:23:52 (1780997032) llite.lustre-ffff8a2a48e8c000.fast_read=0 llite.lustre-ffff8a2a7302d000.fast_read=0 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.098734 s, 21.2 MB/s fail_loc=0x80001421 fail_val=7 fail_loc=0x80001422 fail_val=10 llite.lustre-ffff8a2a48e8c000.fast_read=1 llite.lustre-ffff8a2a7302d000.fast_read=1 PASS 95a (20s) == sanityn test 95b: Check readpage() on a page that is no longer uptodate ========================================================== 05:24:12 (1780997052) llite.lustre-ffff8a2a48e8c000.fast_read=0 llite.lustre-ffff8a2a7302d000.fast_read=0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00881719 s, 1.4 MB/s fail_loc=0x80001424 fail_val=5 fail_loc=0 llite.lustre-ffff8a2a48e8c000.fast_read=1 llite.lustre-ffff8a2a7302d000.fast_read=1 PASS 95b (14s) == sanityn test 100a: DoM: glimpse RPCs for stat without IO lock (DoM only file) ========================================================== 05:24:26 (1780997066) SKIP: sanityn test_100a Reserved for glimpse-ahead SKIP 100a (3s) == sanityn test 100b: DoM: no glimpse RPC for stat with IO lock (DoM only file) ========================================================== 05:24:29 (1780997069) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00417948 s, 980 kB/s /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK /mnt/lustre/d100b.sanityn/dom has type file OK /mnt/lustre/d100b.sanityn/dom has size 4096 OK PASS 100b (7s) == sanityn test 100c: DoM: write vs stat without IO lock (combined file) ========================================================== 05:24:36 (1780997076) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.18139 s, 11.6 MB/s /mnt/lustre/d100c.sanityn/dom has type file OK /mnt/lustre/d100c.sanityn/dom has size 2097152 OK PASS 100c (8s) == sanityn test 100d: DoM: write+truncate vs stat without IO lock (combined file) ========================================================== 05:24:44 (1780997084) 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.207231 s, 10.1 MB/s /mnt/lustre/d100d.sanityn/dom has type file OK /mnt/lustre/d100d.sanityn/dom has size 4096 OK PASS 100d (8s) == sanityn test 100e: DoM: read on open and file size ==== 05:24:52 (1780997092) 1+0 records in 1+0 records out 12000 bytes (12 kB, 12 KiB) copied, 0.00584845 s, 2.1 MB/s PASS 100e (7s) == sanityn test 101a: Discard DoM data on unlink ========= 05:24:59 (1780997099) /mnt/lustre/f101a.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0117594 s, 348 kB/s PASS 101a (8s) == sanityn test 101b: Discard DoM data on rename ========= 05:25:07 (1780997107) /mnt/lustre/f101b.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0179815 s, 228 kB/s PASS 101b (7s) == sanityn test 101c: Discard DoM data on close-unlink === 05:25:14 (1780997114) /mnt/lustre/f101c.sanityn has type file OK 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0117435 s, 349 kB/s PASS 101c (9s) SKIP: sanityn test_102 skipping ALWAYS excluded test 102 == sanityn test 103: Test size correctness with lockahead ========================================================== 05:25:25 (1780997125) fail_loc=0x415 fail_loc=0x415 fail_val=2 Incorrect size expected (no glimpse fix): Starting test test23 at 1780997128 lockahead_test: lockahead_test.c:1171: test23: assertion 'sb.st_size == sb2.st_size' failed: size on /mnt/lustre/d103.sanityn/f103.sanityn and /mnt/lustre2/d103.sanityn/f103.sanityn differs: 0 vs 1048576 fail_loc=0 fail_loc=0x214 fail_val=2 Starting test test23 at 1780997131 Finishing test test23 at 1780997131 PASS 103 (14s) == sanityn test 104: Verify that MDS stores atime/mtime/ctime during close ========================================================== 05:25:40 (1780997140) SKIP: sanityn test_104 ldiskfs only test SKIP 104 (3s) == sanityn test 105: Glimpse and lock cancel race ======== 05:25:43 (1780997143) fail_loc=0x416 PASS 105 (28s) == sanityn test 106a: Verify the btime via statx() ======= 05:26:11 (1780997171) SKIP: sanityn test_106a Test only for ldiskfs and statx() supported SKIP 106a (4s) == sanityn test 106b: Glimpse RPCs test for statx ======== 05:26:15 (1780997175) 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0136724 s, 74.9 kB/s File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938161 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-09 05:26:17.000000000 -0400 Modify: 2026-06-09 05:26:17.000000000 -0400 Change: 2026-06-09 05:26:17.000000000 -0400 Birth: 2026-06-09 05:26:17.000000000 -0400 /mnt/lustre/f106b.sanityn 144115238826938161 -rw-r--r-- 1 0 0 1780997177 1780997177 1780997177 File: /mnt/lustre/f106b.sanityn Size: 1024 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115238826938161 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-09 05:26:17.000000000 -0400 Modify: 2026-06-09 05:26:17.000000000 -0400 Change: 2026-06-09 05:26:17.000000000 -0400 Birth: 2026-06-09 05:26:17.000000000 -0400 1780997177 1024 1 PASS 106b (8s) == sanityn test 106c: Verify statx attributes mask ======= 05:26:23 (1780997183) PASS 106c (8s) == sanityn test 107a: Basic grouplock conflict =========== 05:26:31 (1780997191) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.264301 s, 39.7 MB/s /mnt/lustre/f107a.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 5770 0x168a 0x240000400 multiop /mnt/lustre/f107a.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 multiop /mnt/lustre2/f107a.sanityn vO_G16022000r10g16022000c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 First grouplock blocks second one PASS 107a (9s) == sanityn test 107b: Grouplock is added to the head of waiting list ========================================================== 05:26:41 (1780997201) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.217568 s, 48.2 MB/s /mnt/lustre/f107b.sanityn lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 5732 0x1664 0x280000400 multiop /mnt/lustre/f107b.sanityn vOG14091995_g14091995c TMPPIPE=/tmp/multiop_open_wait_pipe.7563 Grouplock blocks IO First grouplock blocks second one Second grouplock blocks IO PASS 107b (15s) == sanityn test 108a: lseek: parallel updates ============ 05:26:57 (1780997217) 1+0 records in 1+0 records out 10000 bytes (10 kB, 9.8 KiB) copied, 0.00857037 s, 1.2 MB/s llite.lustre-ffff8a2a48e8c000.hybrid_io=0 llite.lustre-ffff8a2a7302d000.hybrid_io=0 fail_val=4 fail_loc=0x80000414 1+0 records in 1+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 4.85643 s, 1.7 MB/s PASS 108a (12s) == sanityn test 109: Race with several mount instances on 1 node ========================================================== 05:27:09 (1780997229) 192.168.206.153@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg653-client.virtnet /mnt/lustre (opts:) 192.168.206.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg653-client.virtnet /mnt/lustre2 (opts:) Starting race between client mounts (duration 30) Iteration 0 Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre2 Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre 192.168.206.153@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg653-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg653-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg653-server' oleg653-server: oleg653-server.virtnet: executing load_modules_local oleg653-server: Loading modules from /home/green/git/lustre-release/lustre oleg653-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg653-server: detected 4 online CPUs by sysfs oleg653-server: MODOPTS_LIBCFS= oleg653-server: Force libcfs to create 2 CPU partitions Iteration 1 Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre2 192.168.206.153@tcp:/lustre /mnt/lustre2 lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 192.168.206.153@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg653-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg653-client.virtnet /mnt/lustre (opts:) /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' loading modules on: 'oleg653-server' oleg653-server: oleg653-server.virtnet: executing load_modules_local oleg653-server: Loading modules from /home/green/git/lustre-release/lustre oleg653-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg653-server: detected 4 online CPUs by sysfs oleg653-server: MODOPTS_LIBCFS= oleg653-server: Force libcfs to create 2 CPU partitions Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre2 PASS 109 (69s) == sanityn test 110: do not grant another lock on resend ========================================================== 05:28:18 (1780997298) fail_loc=0x525 fail_val=3 fail_loc=0x534 File: /mnt/lustre/f110.sanityn Size: 0 Blocks: 1 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115238860488707 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-06-09 05:28:20.000000000 -0400 Modify: 2026-06-09 05:28:20.000000000 -0400 Change: 2026-06-09 05:28:20.000000000 -0400 Birth: 2026-06-09 05:28:20.000000000 -0400 fail_loc=0 fail_val=0 Connected clients: oleg653-client.virtnet oleg653-client.virtnet PASS 110 (72s) == sanityn test 111: A racy rename/link an open file should not cause fs corruption ========================================================== 05:29:30 (1780997370) SKIP: sanityn test_111 needs >= 2 MDTs SKIP 111 (3s) == sanityn test 112: update max-inherit in default LMV === 05:29:33 (1780997373) SKIP: sanityn test_112 We need at least 2 MDTs for this test SKIP 112 (4s) == sanityn test 113: check servers of specified fs ======= 05:29:37 (1780997377) lustre-OST0000-osc-ffff8a2a48e2f800 active. lustre-OST0001-osc-ffff8a2a48e2f800 active. lustre-OST0000-osc-ffff8a2a48e2f800 active. lustre-OST0001-osc-ffff8a2a48e2f800 active. MGC192.168.206.153@tcp active. PASS 113 (6s) == sanityn test 114: implicit default LMV inherit ======== 05:29:43 (1780997383) SKIP: sanityn test_114 We need at least 2 MDTs for this test SKIP 114 (3s) == sanityn test 115: ldiskfs doesn't check direntry for uniqueness ========================================================== 05:29:47 (1780997387) SKIP: sanityn test_115 ldiskfs only test SKIP 115 (3s) == sanityn test 116: DNE: Set default LMV layout from a remote client ========================================================== 05:29:50 (1780997390) SKIP: sanityn test_116 needs >= 2 MDTs SKIP 116 (4s) == sanityn test 121: trunc append race =================== 05:29:54 (1780997394) 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0430532 s, 476 kB/s fail_loc=0x80001436 1+0 records in 1+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.136272 s, 150 kB/s 0000000 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0005000 0000 0000 0000 0000 0000 0000 0000 0000 * 000c800 0a79 0a79 0a79 0a79 0a79 0a79 0a79 0a79 * 0011800 PASS 121 (9s) == sanityn test 122: Multiple lock ASTs enter into critical section of invalidate_lock ========================================================== 05:30:04 (1780997404) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.073676 s, 28.5 MB/s ldlm.ldlm_enqueue_min=100 timout: 100 pause: 65 obdfilter.lustre-OST0000.eviction_count=0 obdfilter.lustre-OST0001.eviction_count=0 fail_loc=0x1437 fail_val=65 17179876635 obdfilter.lustre-OST0000.eviction_count=0 obdfilter.lustre-OST0001.eviction_count=0 PASS 122 (14s) == sanityn test 200: service remains healthy while able to process request ========================================================== 05:30:17 (1780997417) at_max=10 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=9 fail_loc=0x1e1 fail_val=20 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1436 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.119 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1895 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1661 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1689 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1598 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 20.1675 s, 0.2 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.0963 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1747 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1564 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1612 s, 0.1 kB/s 1+0 records in 1+0 records out 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1561 s, 0.1 kB/s1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1787 s, 0.1 kB/s 4096 bytes (4.1 kB, 4.0 KiB) copied, 40.1644 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3184 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3593 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.3128 s, 0.1 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 60.4247 s, 0.1 kB/s at_max=600 at_unhealthy_factor=3 ost.OSS.ost_io.threads_max=116 oleg653-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff8a2a48e2f800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff8a2a48e2f800.ost_server_uuid in FULL state after 0 sec PASS 200 (82s) cleanup: ====================================================== == sanityn test complete, duration 10266 sec ============= 05:31:41 (1780997501) === sanityn: start cleanup 05:31:42 (1780997502) === Stopping clients: oleg653-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg653-client.virtnet /mnt/lustre2 opts: === sanityn: finish cleanup 05:36:14 (1780997774) ===