== sanity test 182b: Test parallel modify metadata operations from osp ========================================================== 07:51:24 (1784202684)
osp.lustre-MDT0000-osp-MDT0001.rpc_stats
osp.lustre-MDT0001-osp-MDT0000.rpc_stats
total: 10 mkdir in 0.39 seconds: 25.40 ops/second
 - mkdir 137 (time 1784202709.46 total 10.02 last 13.67)
 - mkdir 313 (time 1784202719.53 total 20.09 last 17.48)
 - mkdir 473 (time 1784202729.61 total 30.17 last 15.88)
 - mkdir 619 (time 1784202739.67 total 40.24 last 14.50)
 - mkdir 759 (time 1784202749.68 total 50.24 last 13.99)
 - mkdir 916 (time 1784202759.70 total 60.26 last 15.67)
total: 1000 mkdir in 64.91 seconds: 15.41 ops/second
 - mkdir 148 (time 1784202711.37 total 10.05 last 14.72)
 - mkdir 332 (time 1784202721.42 total 20.11 last 18.30)
 - mkdir 490 (time 1784202731.44 total 30.13 last 15.77)
 - mkdir 640 (time 1784202741.52 total 40.21 last 14.88)
 - mkdir 783 (time 1784202751.54 total 50.23 last 14.27)
 - mkdir 941 (time 1784202761.55 total 60.24 last 15.79)
total: 1000 mkdir in 63.04 seconds: 15.86 ops/second
 - mkdir 137 (time 1784202709.71 total 10.13 last 13.53)
 - mkdir 318 (time 1784202719.79 total 20.20 last 17.96)
 - mkdir 472 (time 1784202729.83 total 30.25 last 15.33)
 - mkdir 617 (time 1784202739.90 total 40.32 last 14.40)
 - mkdir 751 (time 1784202749.96 total 50.38 last 13.32)
 - mkdir 912 (time 1784202760.01 total 60.43 last 16.02)
total: 1000 mkdir in 66.11 seconds: 15.13 ops/second
 - mkdir 149 (time 1784202710.98 total 10.05 last 14.82)
 - mkdir 331 (time 1784202721.02 total 20.09 last 18.12)
 - mkdir 480 (time 1784202731.07 total 30.14 last 14.83)
 - mkdir 628 (time 1784202741.15 total 40.22 last 14.69)
 - mkdir 768 (time 1784202751.25 total 50.32 last 13.86)
 - mkdir 919 (time 1784202761.31 total 60.38 last 15.02)
total: 1000 mkdir in 66.58 seconds: 15.02 ops/second
 - mkdir 146 (time 1784202711.83 total 10.02 last 14.58)
 - mkdir 330 (time 1784202721.85 total 20.03 last 18.37)
 - mkdir 476 (time 1784202731.86 total 30.04 last 14.58)
 - mkdir 624 (time 1784202741.92 total 40.11 last 14.70)
 - mkdir 769 (time 1784202751.95 total 50.13 last 14.47)
 - mkdir 919 (time 1784202761.98 total 60.16 last 14.96)
total: 1000 mkdir in 67.34 seconds: 14.85 ops/second
 - mkdir 141 (time 1784202711.88 total 10.03 last 14.06)
 - mkdir 321 (time 1784202721.93 total 20.07 last 17.92)
 - mkdir 465 (time 1784202731.98 total 30.12 last 14.33)
 - mkdir 611 (time 1784202742.00 total 40.15 last 14.56)
 - mkdir 754 (time 1784202752.01 total 50.15 last 14.29)
 - mkdir 909 (time 1784202762.02 total 60.16 last 15.48)
total: 1000 mkdir in 67.99 seconds: 14.71 ops/second
 - mkdir 143 (time 1784202711.11 total 10.04 last 14.25)
 - mkdir 323 (time 1784202721.13 total 20.05 last 17.97)
 - mkdir 466 (time 1784202731.17 total 30.09 last 14.25)
 - mkdir 616 (time 1784202741.20 total 40.13 last 14.95)
 - mkdir 751 (time 1784202751.23 total 50.15 last 13.47)
 - mkdir 903 (time 1784202761.24 total 60.16 last 15.18)
total: 1000 mkdir in 68.89 seconds: 14.52 ops/second
 - mkdir 148 (time 1784202712.12 total 10.03 last 14.75)
 - mkdir 329 (time 1784202722.16 total 20.07 last 18.03)
 - mkdir 477 (time 1784202732.18 total 30.09 last 14.77)
 - mkdir 624 (time 1784202742.25 total 40.17 last 14.59)
 - mkdir 761 (time 1784202752.26 total 50.17 last 13.69)
 - mkdir 911 (time 1784202762.31 total 60.23 last 14.92)
total: 1000 mkdir in 68.26 seconds: 14.65 ops/second
 - mkdir 136 (time 1784202711.78 total 10.03 last 13.56)
 - mkdir 311 (time 1784202721.79 total 20.04 last 17.49)
 - mkdir 451 (time 1784202731.79 total 30.04 last 13.99)
 - mkdir 594 (time 1784202741.86 total 40.11 last 14.20)
 - mkdir 731 (time 1784202751.90 total 50.15 last 13.65)
 - mkdir 877 (time 1784202761.93 total 60.18 last 14.54)
 - mkdir 986 (time 1784202772.03 total 70.28 last 10.79)
total: 1000 mkdir in 71.08 seconds: 14.07 ops/second
 - mkdir 133 (time 1784202711.28 total 10.06 last 13.22)
 - mkdir 305 (time 1784202721.32 total 20.10 last 17.13)
 - mkdir 445 (time 1784202731.34 total 30.11 last 13.98)
 - mkdir 582 (time 1784202741.38 total 40.16 last 13.64)
 - mkdir 713 (time 1784202751.42 total 50.20 last 13.04)
 - mkdir 859 (time 1784202761.49 total 60.27 last 14.50)
 - mkdir 967 (time 1784202771.50 total 70.28 last 10.79)
total: 1000 mkdir in 72.31 seconds: 13.83 ops/second
Time for file creation 86 sec for 8 parallel RPCs
rmdir(/mnt/lustre/d182b.sanity/3/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/5/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/0/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/1/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/4/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/9/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/7/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/8/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/6/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
rmdir(/mnt/lustre/d182b.sanity/2/d-0) error: No such file or directory
total: 0 unlinks in 0 seconds: -nan unlinks/second
Time for file removal 17 sec for 8 parallel RPCs
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=1
total: 10 mkdir in 0.59 seconds: 16.93 ops/second
 - mkdir 101 (time 1784203403.05 total 10.01 last 10.09)
 - mkdir 203 (time 1784203413.10 total 20.07 last 10.14)
 - mkdir 318 (time 1784203423.14 total 30.11 last 11.46)
 - mkdir 425 (time 1784203433.18 total 40.14 last 10.66)
 - mkdir 552 (time 1784203443.19 total 50.15 last 12.69)
 - mkdir 663 (time 1784203453.31 total 60.28 last 10.96)
 - mkdir 744 (time 1784203463.33 total 70.29 last 8.09)
 - mkdir 856 (time 1784203473.34 total 80.30 last 11.19)
total: 1000 mkdir in 90.26 seconds: 11.08 ops/second
 - mkdir 96 (time 1784203401.67 total 10.06 last 9.54)
 - mkdir 200 (time 1784203411.69 total 20.09 last 10.38)
 - mkdir 313 (time 1784203421.79 total 30.19 last 11.18)
 - mkdir 413 (time 1784203431.88 total 40.27 last 9.92)
 - mkdir 544 (time 1784203441.91 total 50.30 last 13.07)
 - mkdir 658 (time 1784203451.94 total 60.34 last 11.36)
 - mkdir 743 (time 1784203462.04 total 70.44 last 8.41)
 - mkdir 836 (time 1784203472.08 total 80.48 last 9.26)
 - mkdir 971 (time 1784203482.12 total 90.52 last 13.45)
total: 1000 mkdir in 92.14 seconds: 10.85 ops/second
 - mkdir 103 (time 1784203405.15 total 10.03 last 10.27)
 - mkdir 207 (time 1784203415.20 total 20.08 last 10.35)
 - mkdir 319 (time 1784203425.26 total 30.14 last 11.13)
 - mkdir 432 (time 1784203435.32 total 40.20 last 11.24)
 - mkdir 561 (time 1784203445.35 total 50.23 last 12.86)
 - mkdir 672 (time 1784203455.45 total 60.32 last 11.00)
 - mkdir 753 (time 1784203465.51 total 70.38 last 8.05)
 - mkdir 880 (time 1784203475.56 total 80.44 last 12.63)
total: 1000 mkdir in 88.89 seconds: 11.25 ops/second
 - mkdir 97 (time 1784203404.78 total 10.03 last 9.67)
 - mkdir 203 (time 1784203414.83 total 20.08 last 10.55)
 - mkdir 312 (time 1784203424.88 total 30.14 last 10.84)
 - mkdir 421 (time 1784203434.89 total 40.15 last 10.89)
 - mkdir 552 (time 1784203444.96 total 50.22 last 13.01)
 - mkdir 662 (time 1784203455.07 total 60.33 last 10.88)
 - mkdir 745 (time 1784203465.15 total 70.40 last 8.24)
 - mkdir 870 (time 1784203475.22 total 80.47 last 12.42)
total: 1000 mkdir in 89.51 seconds: 11.17 ops/second
 - mkdir 93 (time 1784203404.67 total 10.04 last 9.27)
 - mkdir 197 (time 1784203414.72 total 20.09 last 10.35)
 - mkdir 312 (time 1784203424.77 total 30.14 last 11.44)
 - mkdir 421 (time 1784203434.81 total 40.18 last 10.86)
 - mkdir 554 (time 1784203444.91 total 50.28 last 13.18)
 - mkdir 660 (time 1784203454.95 total 60.32 last 10.55)
 - mkdir 744 (time 1784203465.03 total 70.40 last 8.33)
 - mkdir 866 (time 1784203475.05 total 80.42 last 12.18)
 - mkdir 994 (time 1784203485.18 total 90.55 last 12.63)
total: 1000 mkdir in 91.60 seconds: 10.92 ops/second
 - mkdir 93 (time 1784203405.65 total 10.02 last 9.28)
 - mkdir 196 (time 1784203415.68 total 20.05 last 10.26)
 - mkdir 306 (time 1784203426.09 total 30.46 last 10.57)
 - mkdir 422 (time 1784203436.20 total 40.57 last 11.48)
 - mkdir 556 (time 1784203446.21 total 50.58 last 13.39)
 - mkdir 652 (time 1784203456.27 total 60.64 last 9.54)
 - mkdir 736 (time 1784203466.28 total 70.65 last 8.39)
 - mkdir 861 (time 1784203476.36 total 80.73 last 12.40)
 - mkdir 991 (time 1784203486.48 total 90.85 last 12.84)
total: 1000 mkdir in 92.53 seconds: 10.81 ops/second
 - mkdir 102 (time 1784203405.59 total 10.08 last 10.12)
 - mkdir 206 (time 1784203415.65 total 20.15 last 10.33)
 - mkdir 317 (time 1784203425.91 total 30.40 last 10.82)
 - mkdir 432 (time 1784203435.96 total 40.45 last 11.44)
 - mkdir 559 (time 1784203445.99 total 50.48 last 12.67)
 - mkdir 660 (time 1784203456.22 total 60.71 last 9.87)
 - mkdir 740 (time 1784203466.24 total 70.73 last 7.98)
 - mkdir 860 (time 1784203476.25 total 80.75 last 11.99)
 - mkdir 988 (time 1784203486.36 total 90.85 last 12.67)
total: 1000 mkdir in 93.09 seconds: 10.74 ops/second
 - mkdir 101 (time 1784203405.50 total 10.07 last 10.03)
 - mkdir 199 (time 1784203415.71 total 20.28 last 9.60)
 - mkdir 307 (time 1784203425.94 total 30.51 last 10.56)
 - mkdir 420 (time 1784203435.96 total 40.53 last 11.27)
 - mkdir 548 (time 1784203445.98 total 50.55 last 12.78)
 - mkdir 650 (time 1784203456.18 total 60.75 last 10.00)
 - mkdir 733 (time 1784203466.18 total 70.75 last 8.30)
 - mkdir 856 (time 1784203476.25 total 80.83 last 12.21)
 - mkdir 983 (time 1784203486.27 total 90.84 last 12.68)
total: 1000 mkdir in 93.75 seconds: 10.67 ops/second
 - mkdir 92 (time 1784203404.93 total 10.06 last 9.15)
 - mkdir 190 (time 1784203415.00 total 20.12 last 9.74)
 - mkdir 300 (time 1784203425.07 total 30.19 last 10.92)
 - mkdir 408 (time 1784203435.12 total 40.25 last 10.74)
 - mkdir 534 (time 1784203445.12 total 50.25 last 12.60)
 - mkdir 648 (time 1784203455.17 total 60.30 last 11.35)
 - mkdir 727 (time 1784203465.22 total 70.34 last 7.86)
 - mkdir 846 (time 1784203475.31 total 80.43 last 11.79)
 - mkdir 968 (time 1784203485.45 total 90.58 last 12.03)
total: 1000 mkdir in 95.54 seconds: 10.47 ops/second
 - mkdir 93 (time 1784203404.03 total 10.10 last 9.20)
 - mkdir 190 (time 1784203414.26 total 20.34 last 9.48)
 - mkdir 296 (time 1784203424.42 total 30.49 last 10.44)
 - mkdir 404 (time 1784203434.49 total 40.57 last 10.72)
 - mkdir 532 (time 1784203444.50 total 50.57 last 12.80)
 - mkdir 641 (time 1784203454.66 total 60.73 last 10.73)
 - mkdir 725 (time 1784203464.77 total 70.85 last 8.31)
 - mkdir 832 (time 1784203474.77 total 80.85 last 10.70)
 - mkdir 962 (time 1784203485.01 total 91.09 last 12.70)
total: 1000 mkdir in 97.08 seconds: 10.30 ops/second
Time for file creation 110 sec for 1 RPC sent at a time
 - unlinked 0 (time 1784203499 ; total 0 ; last 0)
total: 1000 unlinks in 34 seconds: 29.411764 unlinks/second
 - unlinked 0 (time 1784203499 ; total 0 ; last 0)
total: 1000 unlinks in 34 seconds: 29.411764 unlinks/second
 - unlinked 0 (time 1784203501 ; total 0 ; last 0)
total: 1000 unlinks in 34 seconds: 29.411764 unlinks/second
 - unlinked 0 (time 1784203502 ; total 0 ; last 0)
total: 1000 unlinks in 33 seconds: 30.303030 unlinks/second
 - unlinked 0 (time 1784203502 ; total 0 ; last 0)
total: 1000 unlinks in 34 seconds: 29.411764 unlinks/second
 - unlinked 0 (time 1784203502 ; total 0 ; last 0)
total: 1000 unlinks in 34 seconds: 29.411764 unlinks/second
 - unlinked 0 (time 1784203502 ; total 1 ; last 1)
total: 1000 unlinks in 35 seconds: 28.571428 unlinks/second
 - unlinked 0 (time 1784203502 ; total 1 ; last 1)
total: 1000 unlinks in 35 seconds: 28.571428 unlinks/second
 - unlinked 0 (time 1784203502 ; total 0 ; last 0)
total: 1000 unlinks in 34 seconds: 29.411764 unlinks/second
 - unlinked 0 (time 1784203502 ; total 0 ; last 0)
total: 1000 unlinks in 35 seconds: 28.571428 unlinks/second
Time for file removal 48 sec for 1 RPC sent at a time
osp.lustre-MDT0001-osp-MDT0000.max_mod_rpcs_in_flight=8