== sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 10:24:15 (1788531855)
osp.lustre-MDT0000-osp-MDT0001.rpc_stats
osp.lustre-MDT0001-osp-MDT0000.rpc_stats
total: 10 mkdir in 0.46 seconds: 21.84 ops/second
 - mkdir 145 (time 1788531879.90 total 10.06 last 14.42)
 - mkdir 286 (time 1788531889.98 total 20.14 last 13.99)
 - mkdir 429 (time 1788531900.02 total 30.18 last 14.25)
 - mkdir 576 (time 1788531910.06 total 40.21 last 14.65)
 - mkdir 733 (time 1788531920.10 total 50.26 last 15.63)
 - mkdir 890 (time 1788531930.14 total 60.29 last 15.65)
total: 1000 mkdir in 67.11 seconds: 14.90 ops/second
 - mkdir 147 (time 1788531878.16 total 10.02 last 14.67)
 - mkdir 289 (time 1788531888.20 total 20.06 last 14.14)
 - mkdir 426 (time 1788531898.25 total 30.11 last 13.64)
 - mkdir 569 (time 1788531908.28 total 40.14 last 14.26)
 - mkdir 729 (time 1788531918.29 total 50.14 last 15.99)
 - mkdir 870 (time 1788531928.30 total 60.16 last 14.08)
total: 1000 mkdir in 68.85 seconds: 14.52 ops/second
 - mkdir 138 (time 1788531879.09 total 10.03 last 13.76)
 - mkdir 279 (time 1788531889.18 total 20.13 last 13.97)
 - mkdir 417 (time 1788531899.21 total 30.15 last 13.76)
 - mkdir 572 (time 1788531909.31 total 40.26 last 15.34)
 - mkdir 723 (time 1788531919.36 total 50.30 last 15.03)
 - mkdir 876 (time 1788531929.36 total 60.31 last 15.29)
total: 1000 mkdir in 68.62 seconds: 14.57 ops/second
 - mkdir 138 (time 1788531879.11 total 10.07 last 13.70)
 - mkdir 281 (time 1788531889.22 total 20.18 last 14.15)
 - mkdir 418 (time 1788531899.29 total 30.25 last 13.61)
 - mkdir 569 (time 1788531909.33 total 40.29 last 15.04)
 - mkdir 720 (time 1788531919.39 total 50.35 last 15.01)
 - mkdir 870 (time 1788531929.46 total 60.42 last 14.89)
total: 1000 mkdir in 69.99 seconds: 14.29 ops/second
 - mkdir 131 (time 1788531878.87 total 10.03 last 13.06)
 - mkdir 269 (time 1788531888.93 total 20.09 last 13.72)
 - mkdir 400 (time 1788531899.01 total 30.17 last 13.00)
 - mkdir 549 (time 1788531909.03 total 40.19 last 14.87)
 - mkdir 699 (time 1788531919.09 total 50.25 last 14.91)
 - mkdir 851 (time 1788531929.12 total 60.28 last 15.16)
 - mkdir 991 (time 1788531939.19 total 70.35 last 13.90)
total: 1000 mkdir in 71.25 seconds: 14.04 ops/second
 - mkdir 132 (time 1788531880.05 total 10.01 last 13.18)
 - mkdir 273 (time 1788531890.16 total 20.12 last 13.95)
 - mkdir 412 (time 1788531900.20 total 30.16 last 13.85)
 - mkdir 561 (time 1788531910.28 total 40.24 last 14.78)
 - mkdir 716 (time 1788531920.30 total 50.26 last 15.47)
 - mkdir 865 (time 1788531930.32 total 60.28 last 14.87)
 - mkdir 999 (time 1788531940.38 total 70.34 last 13.31)
total: 1000 mkdir in 70.34 seconds: 14.22 ops/second
 - mkdir 136 (time 1788531879.95 total 10.04 last 13.54)
 - mkdir 271 (time 1788531890.02 total 20.10 last 13.42)
 - mkdir 411 (time 1788531900.06 total 30.15 last 13.94)
 - mkdir 560 (time 1788531910.16 total 40.25 last 14.75)
 - mkdir 708 (time 1788531920.17 total 50.26 last 14.78)
 - mkdir 859 (time 1788531930.18 total 60.27 last 15.10)
 - mkdir 990 (time 1788531940.27 total 70.36 last 12.98)
total: 1000 mkdir in 71.37 seconds: 14.01 ops/second
 - mkdir 143 (time 1788531880.49 total 10.02 last 14.27)
 - mkdir 271 (time 1788531890.53 total 20.05 last 12.76)
 - mkdir 410 (time 1788531900.61 total 30.13 last 13.79)
 - mkdir 557 (time 1788531910.62 total 40.14 last 14.69)
 - mkdir 712 (time 1788531920.68 total 50.20 last 15.41)
 - mkdir 856 (time 1788531930.71 total 60.23 last 14.36)
 - mkdir 979 (time 1788531940.82 total 70.35 last 12.16)
total: 1000 mkdir in 72.55 seconds: 13.78 ops/second
 - mkdir 136 (time 1788531880.39 total 10.01 last 13.59)
 - mkdir 266 (time 1788531890.45 total 20.06 last 12.93)
 - mkdir 395 (time 1788531900.46 total 30.07 last 12.89)
 - mkdir 540 (time 1788531910.49 total 40.11 last 14.45)
 - mkdir 685 (time 1788531920.51 total 50.12 last 14.47)
 - mkdir 829 (time 1788531930.56 total 60.17 last 14.33)
 - mkdir 953 (time 1788531940.64 total 70.26 last 12.29)
total: 1000 mkdir in 74.51 seconds: 13.42 ops/second
 - mkdir 127 (time 1788531879.52 total 10.01 last 12.69)
 - mkdir 261 (time 1788531889.59 total 20.08 last 13.30)
 - mkdir 385 (time 1788531899.62 total 30.11 last 12.36)
 - mkdir 527 (time 1788531909.63 total 40.12 last 14.19)
 - mkdir 669 (time 1788531919.69 total 50.18 last 14.12)
 - mkdir 814 (time 1788531929.77 total 60.26 last 14.38)
 - mkdir 944 (time 1788531939.80 total 70.29 last 12.96)
total: 1000 mkdir in 75.55 seconds: 13.24 ops/second
Time for file creation 86 sec for 8 parallel RPCs
rmdir(/mnt/lustre/d182b.sanity/4/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/9/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/1/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/8/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/0/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/5/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/2/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/7/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/6/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/3/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
Time for file removal 14 sec for 8 parallel RPCs
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=1
total: 10 mkdir in 0.54 seconds: 18.68 ops/second
 - mkdir 146 (time 1788532505.91 total 10.01 last 14.59)
 - mkdir 310 (time 1788532516.02 total 20.12 last 16.21)
 - mkdir 480 (time 1788532526.02 total 30.13 last 17.00)
 - mkdir 655 (time 1788532536.03 total 40.13 last 17.49)
 - mkdir 833 (time 1788532546.08 total 50.18 last 17.71)
total: 1000 mkdir in 58.64 seconds: 17.05 ops/second
 - mkdir 148 (time 1788532505.79 total 10.02 last 14.77)
 - mkdir 307 (time 1788532515.80 total 20.03 last 15.89)
 - mkdir 471 (time 1788532525.87 total 30.10 last 16.28)
 - mkdir 653 (time 1788532535.89 total 40.12 last 18.15)
 - mkdir 826 (time 1788532545.94 total 50.17 last 17.22)
total: 1000 mkdir in 59.52 seconds: 16.80 ops/second
 - mkdir 147 (time 1788532505.93 total 10.00 last 14.70)
 - mkdir 308 (time 1788532516.12 total 20.18 last 15.81)
 - mkdir 473 (time 1788532526.20 total 30.26 last 16.37)
 - mkdir 648 (time 1788532536.24 total 40.31 last 17.42)
 - mkdir 825 (time 1788532546.27 total 50.34 last 17.65)
total: 1000 mkdir in 60.17 seconds: 16.62 ops/second
 - mkdir 154 (time 1788532506.49 total 10.02 last 15.37)
 - mkdir 313 (time 1788532516.52 total 20.05 last 15.85)
 - mkdir 471 (time 1788532526.53 total 30.06 last 15.79)
 - mkdir 654 (time 1788532536.53 total 40.06 last 18.30)
 - mkdir 826 (time 1788532546.59 total 50.12 last 17.10)
total: 1000 mkdir in 60.10 seconds: 16.64 ops/second
 - mkdir 150 (time 1788532507.05 total 10.02 last 14.97)
 - mkdir 309 (time 1788532517.05 total 20.02 last 15.90)
 - mkdir 470 (time 1788532527.06 total 30.03 last 16.09)
 - mkdir 655 (time 1788532537.12 total 40.10 last 18.37)
 - mkdir 837 (time 1788532547.15 total 50.12 last 18.15)
total: 1000 mkdir in 59.57 seconds: 16.79 ops/second
 - mkdir 157 (time 1788532507.31 total 10.04 last 15.64)
 - mkdir 319 (time 1788532517.32 total 20.04 last 16.19)
 - mkdir 487 (time 1788532527.34 total 30.06 last 16.77)
 - mkdir 664 (time 1788532537.37 total 40.09 last 17.65)
 - mkdir 837 (time 1788532547.42 total 50.14 last 17.21)
total: 1000 mkdir in 59.65 seconds: 16.77 ops/second
 - mkdir 152 (time 1788532507.17 total 10.05 last 15.13)
 - mkdir 306 (time 1788532517.19 total 20.07 last 15.37)
 - mkdir 467 (time 1788532527.23 total 30.11 last 16.03)
 - mkdir 646 (time 1788532537.26 total 40.14 last 17.84)
 - mkdir 821 (time 1788532547.29 total 50.17 last 17.45)
 - mkdir 995 (time 1788532557.36 total 60.24 last 17.28)
total: 1000 mkdir in 60.55 seconds: 16.51 ops/second
 - mkdir 154 (time 1788532507.03 total 10.08 last 15.28)
 - mkdir 311 (time 1788532517.04 total 20.09 last 15.69)
 - mkdir 468 (time 1788532527.06 total 30.11 last 15.67)
 - mkdir 649 (time 1788532537.12 total 40.17 last 18.00)
 - mkdir 823 (time 1788532547.14 total 50.19 last 17.35)
 - mkdir 993 (time 1788532557.20 total 60.25 last 16.90)
total: 1000 mkdir in 60.80 seconds: 16.45 ops/second
 - mkdir 145 (time 1788532507.21 total 10.06 last 14.42)
 - mkdir 296 (time 1788532517.22 total 20.07 last 15.08)
 - mkdir 455 (time 1788532527.25 total 30.10 last 15.86)
 - mkdir 635 (time 1788532537.25 total 40.10 last 17.99)
 - mkdir 812 (time 1788532547.32 total 50.17 last 17.58)
 - mkdir 980 (time 1788532557.41 total 60.26 last 16.65)
total: 1000 mkdir in 62.48 seconds: 16.01 ops/second
 - mkdir 138 (time 1788532506.06 total 10.01 last 13.79)
 - mkdir 285 (time 1788532516.11 total 20.06 last 14.63)
 - mkdir 443 (time 1788532526.22 total 30.16 last 15.63)
 - mkdir 618 (time 1788532536.22 total 40.17 last 17.49)
 - mkdir 787 (time 1788532546.25 total 50.20 last 16.85)
 - mkdir 968 (time 1788532556.32 total 60.27 last 17.97)
total: 1000 mkdir in 63.63 seconds: 15.72 ops/second
Time for file creation 72 sec for 1 RPC sent at a time
 - unlinked 0 (time 1788532568 ; total 0 ; last 0)
total: 1000 unlinks in 35 seconds: 28.571428 unlinks/second
 - unlinked 0 (time 1788532568 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1788532568 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1788532569 ; total 0 ; last 0)
total: 1000 unlinks in 35 seconds: 28.571428 unlinks/second
 - unlinked 0 (time 1788532569 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1788532568 ; total 0 ; last 0)
total: 1000 unlinks in 37 seconds: 27.027027 unlinks/second
 - unlinked 0 (time 1788532569 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1788532570 ; total 0 ; last 0)
total: 1000 unlinks in 35 seconds: 28.571428 unlinks/second
 - unlinked 0 (time 1788532570 ; total 1 ; last 1)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1788532570 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
Time for file removal 48 sec for 1 RPC sent at a time
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=8