== sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 02:48:48 (1788936528)
osp.lustre-MDT0000-osp-MDT0001.rpc_stats
osp.lustre-MDT0001-osp-MDT0000.rpc_stats
total: 10 mkdir in 0.61 seconds: 16.29 ops/second
 - mkdir 160 (time 1788936552.42 total 10.00 last 16.00)
 - mkdir 322 (time 1788936562.43 total 20.01 last 16.19)
 - mkdir 489 (time 1788936572.48 total 30.06 last 16.62)
 - mkdir 634 (time 1788936582.48 total 40.06 last 14.50)
 - mkdir 779 (time 1788936592.48 total 50.06 last 14.50)
 - mkdir 956 (time 1788936602.49 total 60.07 last 17.68)
total: 1000 mkdir in 62.60 seconds: 15.97 ops/second
 - mkdir 165 (time 1788936553.53 total 10.01 last 16.48)
 - mkdir 329 (time 1788936563.55 total 20.03 last 16.37)
 - mkdir 481 (time 1788936573.55 total 30.03 last 15.20)
 - mkdir 625 (time 1788936583.62 total 40.10 last 14.31)
 - mkdir 773 (time 1788936593.63 total 50.11 last 14.78)
 - mkdir 950 (time 1788936603.63 total 60.11 last 17.70)
total: 1000 mkdir in 65.30 seconds: 15.31 ops/second
 - mkdir 160 (time 1788936553.75 total 10.02 last 15.96)
 - mkdir 322 (time 1788936563.78 total 20.06 last 16.14)
 - mkdir 477 (time 1788936573.84 total 30.12 last 15.41)
 - mkdir 623 (time 1788936583.89 total 40.16 last 14.54)
 - mkdir 767 (time 1788936593.91 total 50.18 last 14.37)
 - mkdir 944 (time 1788936603.94 total 60.21 last 17.65)
total: 1000 mkdir in 66.71 seconds: 14.99 ops/second
 - mkdir 163 (time 1788936553.49 total 10.04 last 16.24)
 - mkdir 324 (time 1788936563.54 total 20.09 last 16.01)
 - mkdir 472 (time 1788936573.59 total 30.14 last 14.73)
 - mkdir 613 (time 1788936583.64 total 40.19 last 14.02)
 - mkdir 760 (time 1788936593.66 total 50.20 last 14.68)
 - mkdir 934 (time 1788936603.71 total 60.26 last 17.31)
total: 1000 mkdir in 67.55 seconds: 14.80 ops/second
 - mkdir 155 (time 1788936553.20 total 10.02 last 15.48)
 - mkdir 314 (time 1788936563.23 total 20.04 last 15.86)
 - mkdir 466 (time 1788936573.23 total 30.05 last 15.20)
 - mkdir 609 (time 1788936583.29 total 40.11 last 14.21)
 - mkdir 754 (time 1788936593.32 total 50.14 last 14.46)
 - mkdir 929 (time 1788936603.36 total 60.18 last 17.42)
total: 1000 mkdir in 68.08 seconds: 14.69 ops/second
 - mkdir 157 (time 1788936553.02 total 10.04 last 15.63)
 - mkdir 318 (time 1788936563.04 total 20.06 last 16.08)
 - mkdir 468 (time 1788936573.12 total 30.14 last 14.88)
 - mkdir 612 (time 1788936583.16 total 40.18 last 14.35)
 - mkdir 754 (time 1788936593.19 total 50.21 last 14.15)
 - mkdir 925 (time 1788936603.19 total 60.21 last 17.09)
total: 1000 mkdir in 68.30 seconds: 14.64 ops/second
 - mkdir 155 (time 1788936552.40 total 10.01 last 15.49)
 - mkdir 313 (time 1788936562.47 total 20.07 last 15.70)
 - mkdir 465 (time 1788936572.50 total 30.10 last 15.16)
 - mkdir 606 (time 1788936582.57 total 40.17 last 14.01)
 - mkdir 745 (time 1788936592.58 total 50.19 last 13.88)
 - mkdir 916 (time 1788936602.58 total 60.19 last 17.10)
total: 1000 mkdir in 69.17 seconds: 14.46 ops/second
 - mkdir 156 (time 1788936553.26 total 10.05 last 15.52)
 - mkdir 313 (time 1788936563.27 total 20.06 last 15.69)
 - mkdir 464 (time 1788936573.34 total 30.13 last 14.99)
 - mkdir 611 (time 1788936583.38 total 40.17 last 14.64)
 - mkdir 752 (time 1788936593.41 total 50.20 last 14.06)
 - mkdir 926 (time 1788936603.43 total 60.22 last 17.36)
total: 1000 mkdir in 68.41 seconds: 14.62 ops/second
 - mkdir 159 (time 1788936554.00 total 10.04 last 15.83)
 - mkdir 312 (time 1788936564.02 total 20.07 last 15.26)
 - mkdir 463 (time 1788936574.05 total 30.10 last 15.06)
 - mkdir 605 (time 1788936584.11 total 40.16 last 14.12)
 - mkdir 750 (time 1788936594.16 total 50.20 last 14.43)
 - mkdir 923 (time 1788936604.19 total 60.24 last 17.24)
total: 1000 mkdir in 69.72 seconds: 14.34 ops/second
 - mkdir 150 (time 1788936553.81 total 10.05 last 14.93)
 - mkdir 302 (time 1788936563.88 total 20.11 last 15.10)
 - mkdir 444 (time 1788936573.95 total 30.18 last 14.10)
 - mkdir 587 (time 1788936584.02 total 40.25 last 14.21)
 - mkdir 730 (time 1788936594.05 total 50.28 last 14.25)
 - mkdir 903 (time 1788936604.09 total 60.32 last 17.23)
 - mkdir 989 (time 1788936614.21 total 70.44 last 8.50)
total: 1000 mkdir in 71.16 seconds: 14.05 ops/second
Time for file creation 84 sec for 8 parallel RPCs
rmdir(/mnt/lustre/d182b.sanity/7/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/9/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/8/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/1/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/2/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/0/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/3/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/4/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/6/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/5/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
Time for file removal 12 sec for 8 parallel RPCs
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=1
total: 10 mkdir in 0.47 seconds: 21.34 ops/second
 - mkdir 129 (time 1788937208.08 total 10.03 last 12.86)
 - mkdir 269 (time 1788937218.13 total 20.09 last 13.92)
 - mkdir 435 (time 1788937228.14 total 30.10 last 16.59)
 - mkdir 583 (time 1788937238.22 total 40.17 last 14.69)
 - mkdir 745 (time 1788937248.24 total 50.19 last 16.17)
 - mkdir 884 (time 1788937258.27 total 60.23 last 13.85)
total: 1000 mkdir in 67.83 seconds: 14.74 ops/second
 - mkdir 118 (time 1788937206.26 total 10.06 last 11.73)
 - mkdir 259 (time 1788937216.28 total 20.08 last 14.07)
 - mkdir 414 (time 1788937226.34 total 30.13 last 15.41)
 - mkdir 555 (time 1788937236.44 total 40.24 last 13.96)
 - mkdir 718 (time 1788937246.45 total 50.25 last 16.28)
 - mkdir 866 (time 1788937256.52 total 60.31 last 14.71)
total: 1000 mkdir in 70.11 seconds: 14.26 ops/second
 - mkdir 128 (time 1788937208.71 total 10.04 last 12.75)
 - mkdir 273 (time 1788937218.74 total 20.07 last 14.46)
 - mkdir 443 (time 1788937228.82 total 30.15 last 16.86)
 - mkdir 588 (time 1788937238.87 total 40.19 last 14.43)
 - mkdir 754 (time 1788937248.87 total 50.20 last 16.60)
 - mkdir 889 (time 1788937258.96 total 60.29 last 13.38)
total: 1000 mkdir in 68.03 seconds: 14.70 ops/second
 - mkdir 122 (time 1788937207.42 total 10.01 last 12.19)
 - mkdir 257 (time 1788937217.46 total 20.05 last 13.44)
 - mkdir 423 (time 1788937227.47 total 30.06 last 16.58)
 - mkdir 564 (time 1788937237.59 total 40.18 last 13.93)
 - mkdir 724 (time 1788937247.60 total 50.19 last 15.99)
 - mkdir 870 (time 1788937257.64 total 60.23 last 14.54)
total: 1000 mkdir in 70.15 seconds: 14.26 ops/second
 - mkdir 125 (time 1788937209.79 total 10.08 last 12.40)
 - mkdir 277 (time 1788937219.85 total 20.15 last 15.10)
 - mkdir 438 (time 1788937229.88 total 30.18 last 16.05)
 - mkdir 582 (time 1788937239.94 total 40.23 last 14.32)
 - mkdir 748 (time 1788937249.97 total 50.26 last 16.55)
 - mkdir 880 (time 1788937259.97 total 60.26 last 13.19)
total: 1000 mkdir in 69.86 seconds: 14.31 ops/second
 - mkdir 128 (time 1788937209.88 total 10.09 last 12.69)
 - mkdir 281 (time 1788937219.99 total 20.20 last 15.13)
 - mkdir 434 (time 1788937230.00 total 30.21 last 15.29)
 - mkdir 584 (time 1788937240.02 total 40.23 last 14.96)
 - mkdir 753 (time 1788937250.03 total 50.24 last 16.89)
 - mkdir 881 (time 1788937260.16 total 60.37 last 12.64)
 - mkdir 998 (time 1788937270.17 total 70.38 last 11.69)
total: 1000 mkdir in 70.72 seconds: 14.14 ops/second
 - mkdir 129 (time 1788937209.64 total 10.01 last 12.89)
 - mkdir 276 (time 1788937219.70 total 20.07 last 14.61)
 - mkdir 432 (time 1788937229.75 total 30.13 last 15.52)
 - mkdir 578 (time 1788937239.76 total 40.14 last 14.59)
 - mkdir 744 (time 1788937249.78 total 50.15 last 16.58)
 - mkdir 873 (time 1788937259.87 total 60.25 last 12.78)
 - mkdir 994 (time 1788937269.94 total 70.31 last 12.02)
total: 1000 mkdir in 71.29 seconds: 14.03 ops/second
 - mkdir 129 (time 1788937209.71 total 10.06 last 12.82)
 - mkdir 274 (time 1788937219.76 total 20.12 last 14.42)
 - mkdir 427 (time 1788937229.78 total 30.13 last 15.28)
 - mkdir 572 (time 1788937239.86 total 40.21 last 14.39)
 - mkdir 740 (time 1788937249.88 total 50.23 last 16.76)
 - mkdir 867 (time 1788937259.97 total 60.32 last 12.58)
 - mkdir 988 (time 1788937270.02 total 70.37 last 12.05)
total: 1000 mkdir in 72.01 seconds: 13.89 ops/second
 - mkdir 124 (time 1788937209.36 total 10.06 last 12.32)
 - mkdir 270 (time 1788937219.39 total 20.10 last 14.55)
 - mkdir 426 (time 1788937229.42 total 30.13 last 15.55)
 - mkdir 571 (time 1788937239.45 total 40.16 last 14.45)
 - mkdir 735 (time 1788937249.47 total 50.17 last 16.38)
 - mkdir 863 (time 1788937259.50 total 60.21 last 12.76)
 - mkdir 979 (time 1788937269.56 total 70.27 last 11.53)
total: 1000 mkdir in 72.80 seconds: 13.74 ops/second
 - mkdir 119 (time 1788937209.48 total 10.07 last 11.82)
 - mkdir 258 (time 1788937219.51 total 20.10 last 13.86)
 - mkdir 408 (time 1788937229.59 total 30.18 last 14.88)
 - mkdir 551 (time 1788937239.62 total 40.21 last 14.25)
 - mkdir 716 (time 1788937249.63 total 50.22 last 16.49)
 - mkdir 849 (time 1788937259.71 total 60.30 last 13.19)
 - mkdir 964 (time 1788937269.78 total 70.37 last 11.42)
total: 1000 mkdir in 74.32 seconds: 13.45 ops/second
Time for file creation 86 sec for 1 RPC sent at a time
 - unlinked 0 (time 1788937281 ; total 0 ; last 0)
total: 1000 unlinks in 34 seconds: 29.411764 unlinks/second
 - unlinked 0 (time 1788937283 ; total 0 ; last 0)
total: 1000 unlinks in 35 seconds: 28.571428 unlinks/second
 - unlinked 0 (time 1788937282 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1788937283 ; total 0 ; last 0)
total: 1000 unlinks in 35 seconds: 28.571428 unlinks/second
 - unlinked 0 (time 1788937283 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1788937284 ; total 0 ; last 0)
total: 1000 unlinks in 35 seconds: 28.571428 unlinks/second
 - unlinked 0 (time 1788937283 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1788937283 ; total 0 ; last 0)
total: 1000 unlinks in 37 seconds: 27.027027 unlinks/second
 - unlinked 0 (time 1788937283 ; total 0 ; last 0)
total: 1000 unlinks in 37 seconds: 27.027027 unlinks/second
 - unlinked 0 (time 1788937283 ; total 0 ; last 0)
total: 1000 unlinks in 38 seconds: 26.315790 unlinks/second
Time for file removal 49 sec for 1 RPC sent at a time
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=8