== sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 07:44:38 (1788867878)
osp.lustre-MDT0000-osp-MDT0001.rpc_stats
osp.lustre-MDT0001-osp-MDT0000.rpc_stats
total: 10 mkdir in 0.65 seconds: 15.39 ops/second
 - mkdir 145 (time 1788867903.25 total 10.08 last 14.38)
 - mkdir 303 (time 1788867913.34 total 20.17 last 15.66)
 - mkdir 472 (time 1788867923.46 total 30.29 last 16.70)
 - mkdir 605 (time 1788867933.55 total 40.38 last 13.19)
 - mkdir 741 (time 1788867943.57 total 50.40 last 13.57)
 - mkdir 895 (time 1788867953.67 total 60.51 last 15.24)
total: 1000 mkdir in 68.81 seconds: 14.53 ops/second
 - mkdir 137 (time 1788867902.27 total 10.03 last 13.66)
 - mkdir 288 (time 1788867912.30 total 20.05 last 15.07)
 - mkdir 446 (time 1788867922.32 total 30.07 last 15.77)
 - mkdir 590 (time 1788867932.34 total 40.10 last 14.36)
 - mkdir 709 (time 1788867942.36 total 50.12 last 11.88)
 - mkdir 858 (time 1788867952.39 total 60.14 last 14.86)
 - mkdir 986 (time 1788867962.47 total 70.23 last 12.69)
total: 1000 mkdir in 71.72 seconds: 13.94 ops/second
 - mkdir 142 (time 1788867901.80 total 10.08 last 14.09)
 - mkdir 291 (time 1788867911.82 total 20.10 last 14.87)
 - mkdir 443 (time 1788867921.85 total 30.13 last 15.15)
 - mkdir 582 (time 1788867931.90 total 40.18 last 13.84)
 - mkdir 705 (time 1788867941.99 total 50.27 last 12.19)
 - mkdir 851 (time 1788867952.07 total 60.35 last 14.49)
 - mkdir 981 (time 1788867962.12 total 70.40 last 12.94)
total: 1000 mkdir in 72.74 seconds: 13.75 ops/second
 - mkdir 132 (time 1788867902.16 total 10.02 last 13.17)
 - mkdir 287 (time 1788867912.18 total 20.04 last 15.46)
 - mkdir 436 (time 1788867922.21 total 30.07 last 14.86)
 - mkdir 577 (time 1788867932.23 total 40.09 last 14.08)
 - mkdir 701 (time 1788867942.29 total 50.15 last 12.32)
 - mkdir 845 (time 1788867952.31 total 60.17 last 14.38)
 - mkdir 975 (time 1788867962.38 total 70.24 last 12.91)
total: 1000 mkdir in 73.14 seconds: 13.67 ops/second
 - mkdir 141 (time 1788867903.51 total 10.05 last 14.02)
 - mkdir 297 (time 1788867913.59 total 20.13 last 15.48)
 - mkdir 465 (time 1788867923.60 total 30.14 last 16.78)
 - mkdir 599 (time 1788867933.68 total 40.22 last 13.29)
 - mkdir 720 (time 1788867943.73 total 50.27 last 12.04)
 - mkdir 872 (time 1788867953.81 total 60.35 last 15.08)
 - mkdir 988 (time 1788867963.84 total 70.38 last 11.57)
total: 1000 mkdir in 72.31 seconds: 13.83 ops/second
 - mkdir 140 (time 1788867903.70 total 10.05 last 13.93)
 - mkdir 290 (time 1788867913.70 total 20.05 last 15.00)
 - mkdir 455 (time 1788867923.77 total 30.12 last 16.38)
 - mkdir 593 (time 1788867933.80 total 40.15 last 13.76)
 - mkdir 716 (time 1788867943.85 total 50.20 last 12.24)
 - mkdir 866 (time 1788867953.87 total 60.22 last 14.97)
 - mkdir 980 (time 1788867963.88 total 70.23 last 11.38)
total: 1000 mkdir in 73.91 seconds: 13.53 ops/second
 - mkdir 139 (time 1788867903.46 total 10.08 last 13.79)
 - mkdir 289 (time 1788867913.48 total 20.09 last 14.98)
 - mkdir 450 (time 1788867923.53 total 30.14 last 16.02)
 - mkdir 579 (time 1788867933.55 total 40.16 last 12.87)
 - mkdir 705 (time 1788867943.57 total 50.19 last 12.57)
 - mkdir 855 (time 1788867953.68 total 60.29 last 14.84)
 - mkdir 971 (time 1788867963.86 total 70.47 last 11.40)
total: 1000 mkdir in 75.84 seconds: 13.19 ops/second
 - mkdir 137 (time 1788867903.12 total 10.01 last 13.69)
 - mkdir 286 (time 1788867913.20 total 20.08 last 14.79)
 - mkdir 445 (time 1788867923.23 total 30.11 last 15.85)
 - mkdir 575 (time 1788867933.38 total 40.26 last 12.81)
 - mkdir 691 (time 1788867943.39 total 50.28 last 11.58)
 - mkdir 840 (time 1788867953.45 total 60.34 last 14.81)
 - mkdir 953 (time 1788867963.48 total 70.36 last 11.27)
total: 1000 mkdir in 78.44 seconds: 12.75 ops/second
 - mkdir 129 (time 1788867903.15 total 10.07 last 12.82)
 - mkdir 278 (time 1788867913.18 total 20.10 last 14.85)
 - mkdir 435 (time 1788867923.20 total 30.12 last 15.68)
 - mkdir 565 (time 1788867933.24 total 40.16 last 12.94)
 - mkdir 688 (time 1788867943.33 total 50.25 last 12.19)
 - mkdir 831 (time 1788867953.36 total 60.28 last 14.25)
 - mkdir 945 (time 1788867963.55 total 70.46 last 11.20)
total: 1000 mkdir in 79.77 seconds: 12.54 ops/second
 - mkdir 131 (time 1788867903.65 total 10.02 last 13.08)
 - mkdir 270 (time 1788867913.70 total 20.06 last 13.84)
 - mkdir 427 (time 1788867923.79 total 30.15 last 15.56)
 - mkdir 551 (time 1788867933.85 total 40.22 last 12.32)
 - mkdir 668 (time 1788867943.91 total 50.28 last 11.63)
 - mkdir 814 (time 1788867953.92 total 60.28 last 14.59)
 - mkdir 919 (time 1788867963.96 total 70.32 last 10.46)
 - mkdir 985 (time 1788867973.99 total 80.36 last 6.58)
total: 1000 mkdir in 81.34 seconds: 12.29 ops/second
Time for file creation 94 sec for 8 parallel RPCs
rmdir(/mnt/lustre/d182b.sanity/0/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/1/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/3/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/9/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/5/d-0) error: No such file or directory
total: 0 unlinks in 1 seconds: 0.000000 unlinks/second
rmdir(/mnt/lustre/d182b.sanity/8/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/6/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/2/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/4/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/7/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
Time for file removal 20 sec for 8 parallel RPCs
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=1
total: 10 mkdir in 0.50 seconds: 20.16 ops/second
 - mkdir 142 (time 1788868549.40 total 10.01 last 14.18)
 - mkdir 301 (time 1788868559.47 total 20.08 last 15.80)
 - mkdir 462 (time 1788868569.49 total 30.10 last 16.06)
 - mkdir 613 (time 1788868579.55 total 40.16 last 15.01)
 - mkdir 788 (time 1788868589.57 total 50.18 last 17.47)
 - mkdir 939 (time 1788868599.62 total 60.23 last 15.03)
total: 1000 mkdir in 64.09 seconds: 15.60 ops/second
 - mkdir 140 (time 1788868550.44 total 10.02 last 13.97)
 - mkdir 305 (time 1788868560.45 total 20.03 last 16.48)
 - mkdir 459 (time 1788868570.50 total 30.08 last 15.32)
 - mkdir 621 (time 1788868580.52 total 40.10 last 16.18)
 - mkdir 799 (time 1788868590.56 total 50.14 last 17.72)
 - mkdir 946 (time 1788868600.65 total 60.23 last 14.57)
total: 1000 mkdir in 63.48 seconds: 15.75 ops/second
 - mkdir 135 (time 1788868549.81 total 10.01 last 13.49)
 - mkdir 300 (time 1788868559.85 total 20.05 last 16.43)
 - mkdir 456 (time 1788868569.89 total 30.09 last 15.54)
 - mkdir 613 (time 1788868579.92 total 40.12 last 15.65)
 - mkdir 787 (time 1788868589.93 total 50.13 last 17.38)
 - mkdir 937 (time 1788868600.02 total 60.22 last 14.88)
total: 1000 mkdir in 64.70 seconds: 15.46 ops/second
 - mkdir 137 (time 1788868550.36 total 10.04 last 13.65)
 - mkdir 303 (time 1788868560.44 total 20.11 last 16.48)
 - mkdir 461 (time 1788868570.44 total 30.12 last 15.79)
 - mkdir 620 (time 1788868580.47 total 40.14 last 15.87)
 - mkdir 795 (time 1788868590.48 total 50.16 last 17.47)
 - mkdir 942 (time 1788868600.53 total 60.20 last 14.64)
total: 1000 mkdir in 64.29 seconds: 15.55 ops/second
 - mkdir 140 (time 1788868550.66 total 10.04 last 13.95)
 - mkdir 306 (time 1788868560.69 total 20.07 last 16.55)
 - mkdir 466 (time 1788868570.78 total 30.16 last 15.84)
 - mkdir 626 (time 1788868580.87 total 40.25 last 15.87)
 - mkdir 799 (time 1788868590.88 total 50.26 last 17.27)
 - mkdir 942 (time 1788868600.96 total 60.34 last 14.19)
total: 1000 mkdir in 64.78 seconds: 15.44 ops/second
 - mkdir 136 (time 1788868549.85 total 10.02 last 13.57)
 - mkdir 295 (time 1788868559.89 total 20.07 last 15.83)
 - mkdir 444 (time 1788868569.93 total 30.11 last 14.84)
 - mkdir 596 (time 1788868579.94 total 40.11 last 15.19)
 - mkdir 774 (time 1788868589.94 total 50.12 last 17.79)
 - mkdir 921 (time 1788868600.04 total 60.21 last 14.56)
total: 1000 mkdir in 66.99 seconds: 14.93 ops/second
 - mkdir 130 (time 1788868550.04 total 10.07 last 12.91)
 - mkdir 292 (time 1788868560.04 total 20.08 last 16.19)
 - mkdir 447 (time 1788868570.08 total 30.12 last 15.44)
 - mkdir 605 (time 1788868580.09 total 40.13 last 15.79)
 - mkdir 777 (time 1788868590.17 total 50.20 last 17.07)
 - mkdir 923 (time 1788868600.26 total 60.29 last 14.47)
total: 1000 mkdir in 66.95 seconds: 14.94 ops/second
 - mkdir 132 (time 1788868550.04 total 10.03 last 13.16)
 - mkdir 293 (time 1788868560.04 total 20.03 last 16.10)
 - mkdir 443 (time 1788868570.08 total 30.08 last 14.93)
 - mkdir 599 (time 1788868580.09 total 40.08 last 15.60)
 - mkdir 775 (time 1788868590.14 total 50.14 last 17.50)
 - mkdir 921 (time 1788868600.16 total 60.16 last 14.57)
total: 1000 mkdir in 67.00 seconds: 14.92 ops/second
 - mkdir 130 (time 1788868549.02 total 10.01 last 12.99)
 - mkdir 280 (time 1788868559.04 total 20.02 last 14.98)
 - mkdir 431 (time 1788868569.05 total 30.03 last 15.09)
 - mkdir 580 (time 1788868579.11 total 40.09 last 14.81)
 - mkdir 752 (time 1788868589.11 total 50.10 last 17.19)
 - mkdir 903 (time 1788868599.16 total 60.15 last 15.02)
total: 1000 mkdir in 69.29 seconds: 14.43 ops/second
 - mkdir 122 (time 1788868549.68 total 10.02 last 12.17)
 - mkdir 278 (time 1788868559.71 total 20.04 last 15.56)
 - mkdir 426 (time 1788868569.73 total 30.06 last 14.77)
 - mkdir 579 (time 1788868579.81 total 40.14 last 15.18)
 - mkdir 754 (time 1788868589.85 total 50.19 last 17.42)
 - mkdir 902 (time 1788868599.88 total 60.22 last 14.75)
total: 1000 mkdir in 69.19 seconds: 14.45 ops/second
Time for file creation 80 sec for 1 RPC sent at a time
 - unlinked 0 (time 1788868617 ; total 1 ; last 1)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1788868620 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1788868621 ; total 0 ; last 0)
total: 1000 unlinks in 35 seconds: 28.571428 unlinks/second
 - unlinked 0 (time 1788868620 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1788868622 ; total 0 ; last 0)
total: 1000 unlinks in 36 seconds: 27.777779 unlinks/second
 - unlinked 0 (time 1788868622 ; total 1 ; last 1)
total: 1000 unlinks in 37 seconds: 27.027027 unlinks/second
 - unlinked 0 (time 1788868621 ; total 0 ; last 0)
total: 1000 unlinks in 38 seconds: 26.315790 unlinks/second
 - unlinked 0 (time 1788868622 ; total 1 ; last 1)
total: 1000 unlinks in 38 seconds: 26.315790 unlinks/second
 - unlinked 0 (time 1788868621 ; total 0 ; last 0)
total: 1000 unlinks in 38 seconds: 26.315790 unlinks/second
 - unlinked 0 (time 1788868621 ; total 0 ; last 0)
total: 1000 unlinks in 39 seconds: 25.641026 unlinks/second
Time for file removal 52 sec for 1 RPC sent at a time
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=8