== sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 15:03:57 (1788635037)
osp.lustre-MDT0000-osp-MDT0001.rpc_stats
osp.lustre-MDT0001-osp-MDT0000.rpc_stats
total: 10 mkdir in 0.58 seconds: 17.35 ops/second
 - mkdir 150 (time 1788635057.65 total 10.01 last 14.99)
 - mkdir 313 (time 1788635067.68 total 20.03 last 16.26)
 - mkdir 493 (time 1788635077.70 total 30.06 last 17.96)
 - mkdir 666 (time 1788635087.73 total 40.09 last 17.24)
 - mkdir 813 (time 1788635097.78 total 50.14 last 14.63)
 - mkdir 951 (time 1788635107.79 total 60.14 last 13.79)
total: 1000 mkdir in 63.10 seconds: 15.85 ops/second
 - mkdir 157 (time 1788635059.45 total 10.03 last 15.65)
 - mkdir 321 (time 1788635069.50 total 20.08 last 16.33)
 - mkdir 518 (time 1788635079.50 total 30.08 last 19.69)
 - mkdir 673 (time 1788635089.51 total 40.09 last 15.49)
 - mkdir 830 (time 1788635099.52 total 50.10 last 15.69)
 - mkdir 974 (time 1788635109.57 total 60.15 last 14.32)
total: 1000 mkdir in 62.24 seconds: 16.07 ops/second
 - mkdir 153 (time 1788635059.71 total 10.01 last 15.28)
 - mkdir 322 (time 1788635069.71 total 20.01 last 16.90)
 - mkdir 514 (time 1788635079.74 total 30.04 last 19.15)
 - mkdir 671 (time 1788635089.78 total 40.08 last 15.64)
 - mkdir 825 (time 1788635099.87 total 50.17 last 15.25)
 - mkdir 968 (time 1788635109.92 total 60.22 last 14.23)
total: 1000 mkdir in 63.84 seconds: 15.66 ops/second
 - mkdir 158 (time 1788635059.51 total 10.00 last 15.80)
 - mkdir 316 (time 1788635069.56 total 20.05 last 15.72)
 - mkdir 509 (time 1788635079.59 total 30.08 last 19.24)
 - mkdir 667 (time 1788635089.63 total 40.12 last 15.74)
 - mkdir 821 (time 1788635099.72 total 50.21 last 15.26)
 - mkdir 965 (time 1788635109.75 total 60.25 last 14.35)
total: 1000 mkdir in 64.12 seconds: 15.60 ops/second
 - mkdir 155 (time 1788635058.01 total 10.06 last 15.40)
 - mkdir 308 (time 1788635068.07 total 20.11 last 15.22)
 - mkdir 487 (time 1788635078.08 total 30.13 last 17.87)
 - mkdir 653 (time 1788635088.12 total 40.17 last 16.54)
 - mkdir 798 (time 1788635098.16 total 50.21 last 14.43)
 - mkdir 935 (time 1788635108.19 total 60.24 last 13.66)
total: 1000 mkdir in 65.99 seconds: 15.15 ops/second
 - mkdir 152 (time 1788635059.17 total 10.07 last 15.09)
 - mkdir 311 (time 1788635069.20 total 20.10 last 15.85)
 - mkdir 505 (time 1788635079.27 total 30.18 last 19.26)
 - mkdir 660 (time 1788635089.28 total 40.19 last 15.48)
 - mkdir 816 (time 1788635099.33 total 50.24 last 15.51)
 - mkdir 955 (time 1788635109.34 total 60.25 last 13.89)
total: 1000 mkdir in 64.98 seconds: 15.39 ops/second
 - mkdir 155 (time 1788635059.57 total 10.04 last 15.43)
 - mkdir 313 (time 1788635069.58 total 20.06 last 15.78)
 - mkdir 508 (time 1788635079.61 total 30.09 last 19.43)
 - mkdir 664 (time 1788635089.63 total 40.10 last 15.58)
 - mkdir 814 (time 1788635099.64 total 50.11 last 14.98)
 - mkdir 958 (time 1788635109.65 total 60.13 last 14.38)
total: 1000 mkdir in 64.66 seconds: 15.47 ops/second
 - mkdir 154 (time 1788635059.45 total 10.02 last 15.37)
 - mkdir 308 (time 1788635069.47 total 20.03 last 15.38)
 - mkdir 493 (time 1788635079.47 total 30.03 last 18.49)
 - mkdir 651 (time 1788635089.54 total 40.10 last 15.70)
 - mkdir 803 (time 1788635099.61 total 50.17 last 15.09)
 - mkdir 946 (time 1788635109.64 total 60.20 last 14.26)
total: 1000 mkdir in 66.51 seconds: 15.04 ops/second
 - mkdir 151 (time 1788635059.49 total 10.06 last 15.01)
 - mkdir 303 (time 1788635069.52 total 20.10 last 15.15)
 - mkdir 491 (time 1788635079.53 total 30.10 last 18.80)
 - mkdir 643 (time 1788635089.58 total 40.15 last 15.12)
 - mkdir 791 (time 1788635099.62 total 50.19 last 14.74)
 - mkdir 929 (time 1788635109.68 total 60.25 last 13.73)
total: 1000 mkdir in 67.82 seconds: 14.74 ops/second
 - mkdir 144 (time 1788635059.00 total 10.03 last 14.36)
 - mkdir 298 (time 1788635069.06 total 20.09 last 15.31)
 - mkdir 485 (time 1788635079.08 total 30.11 last 18.66)
 - mkdir 636 (time 1788635089.14 total 40.17 last 15.01)
 - mkdir 785 (time 1788635099.20 total 50.23 last 14.81)
 - mkdir 919 (time 1788635109.24 total 60.26 last 13.35)
total: 1000 mkdir in 68.96 seconds: 14.50 ops/second
Time for file creation 78 sec for 8 parallel RPCs
rmdir(/mnt/lustre/d182b.sanity/8/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/0/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/2/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/9/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/3/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/5/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/1/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/6/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/4/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/7/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
Time for file removal 13 sec for 8 parallel RPCs
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=1
total: 10 mkdir in 0.94 seconds: 10.65 ops/second
 - mkdir 86 (time 1788635680.17 total 10.10 last 8.52)
 - mkdir 210 (time 1788635690.21 total 20.14 last 12.35)
 - mkdir 356 (time 1788635700.21 total 30.14 last 14.60)
 - mkdir 511 (time 1788635710.25 total 40.18 last 15.44)
 - mkdir 678 (time 1788635720.28 total 50.22 last 16.64)
 - mkdir 809 (time 1788635730.30 total 60.23 last 13.08)
 - mkdir 985 (time 1788635740.31 total 70.24 last 17.59)
total: 1000 mkdir in 71.16 seconds: 14.05 ops/second
 - mkdir 87 (time 1788635682.87 total 10.02 last 8.69)
 - mkdir 219 (time 1788635692.92 total 20.07 last 13.13)
 - mkdir 379 (time 1788635702.95 total 30.10 last 15.94)
 - mkdir 534 (time 1788635712.98 total 40.13 last 15.45)
 - mkdir 699 (time 1788635722.99 total 50.14 last 16.48)
 - mkdir 841 (time 1788635733.08 total 60.23 last 14.08)
total: 1000 mkdir in 69.19 seconds: 14.45 ops/second
 - mkdir 83 (time 1788635681.79 total 10.00 last 8.30)
 - mkdir 211 (time 1788635691.86 total 20.07 last 12.71)
 - mkdir 369 (time 1788635701.87 total 30.08 last 15.78)
 - mkdir 528 (time 1788635711.89 total 40.10 last 15.87)
 - mkdir 691 (time 1788635721.91 total 50.12 last 16.27)
 - mkdir 820 (time 1788635731.94 total 60.15 last 12.86)
 - mkdir 991 (time 1788635741.96 total 70.17 last 17.06)
total: 1000 mkdir in 71.05 seconds: 14.07 ops/second
 - mkdir 94 (time 1788635682.99 total 10.01 last 9.39)
 - mkdir 224 (time 1788635693.02 total 20.04 last 12.97)
 - mkdir 379 (time 1788635703.06 total 30.08 last 15.43)
 - mkdir 536 (time 1788635713.06 total 40.08 last 15.70)
 - mkdir 695 (time 1788635723.09 total 50.11 last 15.86)
 - mkdir 834 (time 1788635733.19 total 60.22 last 13.75)
 - mkdir 997 (time 1788635743.21 total 70.23 last 16.28)
total: 1000 mkdir in 70.51 seconds: 14.18 ops/second
 - mkdir 80 (time 1788635682.22 total 10.01 last 7.99)
 - mkdir 210 (time 1788635692.25 total 20.03 last 12.97)
 - mkdir 368 (time 1788635702.27 total 30.06 last 15.76)
 - mkdir 526 (time 1788635712.30 total 40.09 last 15.76)
 - mkdir 688 (time 1788635722.35 total 50.14 last 16.11)
 - mkdir 822 (time 1788635732.38 total 60.17 last 13.36)
 - mkdir 986 (time 1788635742.50 total 70.28 last 16.22)
total: 1000 mkdir in 71.77 seconds: 13.93 ops/second
 - mkdir 85 (time 1788635682.80 total 10.02 last 8.49)
 - mkdir 206 (time 1788635692.84 total 20.06 last 12.04)
 - mkdir 356 (time 1788635702.85 total 30.07 last 15.00)
 - mkdir 511 (time 1788635712.85 total 40.07 last 15.49)
 - mkdir 674 (time 1788635722.92 total 50.14 last 16.20)
 - mkdir 816 (time 1788635732.97 total 60.19 last 14.12)
 - mkdir 984 (time 1788635743.06 total 70.28 last 16.65)
total: 1000 mkdir in 72.24 seconds: 13.84 ops/second
 - mkdir 87 (time 1788635683.07 total 10.01 last 8.69)
 - mkdir 209 (time 1788635693.10 total 20.04 last 12.16)
 - mkdir 369 (time 1788635703.16 total 30.10 last 15.91)
 - mkdir 524 (time 1788635713.17 total 40.10 last 15.49)
 - mkdir 677 (time 1788635723.31 total 50.25 last 15.08)
 - mkdir 820 (time 1788635733.31 total 60.25 last 14.30)
 - mkdir 981 (time 1788635743.40 total 70.34 last 15.96)
total: 1000 mkdir in 72.41 seconds: 13.81 ops/second
 - mkdir 78 (time 1788635682.20 total 10.05 last 7.76)
 - mkdir 204 (time 1788635692.23 total 20.07 last 12.57)
 - mkdir 354 (time 1788635702.28 total 30.12 last 14.93)
 - mkdir 508 (time 1788635712.32 total 40.16 last 15.34)
 - mkdir 669 (time 1788635722.37 total 50.21 last 16.01)
 - mkdir 807 (time 1788635732.39 total 60.23 last 13.77)
 - mkdir 970 (time 1788635742.44 total 70.28 last 16.22)
total: 1000 mkdir in 73.60 seconds: 13.59 ops/second
 - mkdir 78 (time 1788635681.64 total 10.04 last 7.77)
 - mkdir 198 (time 1788635691.72 total 20.12 last 11.91)
 - mkdir 342 (time 1788635701.77 total 30.18 last 14.32)
 - mkdir 495 (time 1788635711.79 total 40.19 last 15.28)
 - mkdir 658 (time 1788635721.79 total 50.19 last 16.29)
 - mkdir 795 (time 1788635731.82 total 60.22 last 13.66)
 - mkdir 958 (time 1788635741.91 total 70.32 last 16.15)
total: 1000 mkdir in 75.02 seconds: 13.33 ops/second
 - mkdir 77 (time 1788635681.76 total 10.00 last 7.70)
 - mkdir 195 (time 1788635691.78 total 20.02 last 11.78)
 - mkdir 340 (time 1788635701.82 total 30.06 last 14.44)
 - mkdir 490 (time 1788635711.83 total 40.06 last 15.00)
 - mkdir 647 (time 1788635721.86 total 50.10 last 15.64)
 - mkdir 777 (time 1788635731.90 total 60.14 last 12.95)
 - mkdir 938 (time 1788635741.97 total 70.21 last 15.99)
total: 1000 mkdir in 76.05 seconds: 13.15 ops/second
Time for file creation 88 sec for 1 RPC sent at a time
 - unlinked 0 (time 1788635755 ; total 0 ; last 0)
total: 1000 unlinks in 37 seconds: 27.027027 unlinks/second
 - unlinked 0 (time 1788635755 ; total 0 ; last 0)
total: 1000 unlinks in 37 seconds: 27.027027 unlinks/second
 - unlinked 0 (time 1788635756 ; total 0 ; last 0)
total: 1000 unlinks in 37 seconds: 27.027027 unlinks/second
 - unlinked 0 (time 1788635756 ; total 0 ; last 0)
total: 1000 unlinks in 37 seconds: 27.027027 unlinks/second
 - unlinked 0 (time 1788635756 ; total 0 ; last 0)
total: 1000 unlinks in 37 seconds: 27.027027 unlinks/second
 - unlinked 0 (time 1788635756 ; total 0 ; last 0)
total: 1000 unlinks in 37 seconds: 27.027027 unlinks/second
 - unlinked 0 (time 1788635756 ; total 0 ; last 0)
total: 1000 unlinks in 37 seconds: 27.027027 unlinks/second
 - unlinked 0 (time 1788635756 ; total 0 ; last 0)
total: 1000 unlinks in 37 seconds: 27.027027 unlinks/second
 - unlinked 0 (time 1788635756 ; total 0 ; last 0)
total: 1000 unlinks in 38 seconds: 26.315790 unlinks/second
 - unlinked 0 (time 1788635756 ; total 0 ; last 0)
total: 1000 unlinks in 38 seconds: 26.315790 unlinks/second
Time for file removal 49 sec for 1 RPC sent at a time
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=8