== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:35:36 (1788543336) start mds service on oleg341-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 22:00 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing load_modules_local oleg341-server: kmemleak disabled oleg341-server: Loading modules from /home/green/git/lustre-release/lustre oleg341-server: detected 4 online CPUs by sysfs oleg341-server: MODOPTS_LIBCFS= oleg341-server: Force libcfs to create 2 CPU partitions oleg341-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg341-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg341-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg341-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg341-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg341-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg341-server: oleg341-server.virtnet: executing set_default_debug -1 all pdsh@oleg341-client: oleg341-server: ssh exited with exit code 1 Started lustre-OST0000 oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg341-client.virtnet: -o user_xattr,flock 192.168.203.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 203 (time 1788543419.21 total 10.93 last 18.57) - open/close 398 (time 1788543429.21 total 20.94 last 19.49) - open/close 620 (time 1788543439.56 total 31.28 last 21.46) - open/close 852 (time 1788543449.57 total 41.30 last 23.17) - open/close 1110 (time 1788543460.43 total 52.16 last 23.75) - open/close 1217 (time 1788543472.33 total 64.06 last 8.99) - open/close 1461 (time 1788543482.85 total 74.58 last 23.19) - open/close 1676 (time 1788543493.00 total 84.72 last 21.20) - open/close 1892 (time 1788543503.62 total 95.35 last 20.33) - open/close 2119 (time 1788543513.66 total 105.39 last 22.61) - open/close 2301 (time 1788543523.92 total 115.65 last 17.73) - open/close 2496 (time 1788543534.73 total 126.46 last 18.05) - open/close 2729 (time 1788543545.31 total 137.04 last 22.01) - open/close 2974 (time 1788543556.11 total 147.84 last 22.69) - open/close 3213 (time 1788543566.36 total 158.09 last 23.32) - open/close 3465 (time 1788543577.30 total 169.03 last 23.04) - open/close 3639 (time 1788543587.73 total 179.45 last 16.69) - open/close 3883 (time 1788543598.47 total 190.20 last 22.71) - open/close 4118 (time 1788543608.52 total 200.24 last 23.39) - open/close 4351 (time 1788543619.23 total 210.96 last 21.74) - open/close 4592 (time 1788543629.24 total 220.96 last 24.09) - open/close 4827 (time 1788543639.25 total 230.98 last 23.47) - open/close 5091 (time 1788543649.25 total 240.98 last 26.39) - open/close 5320 (time 1788543659.28 total 251.01 last 22.83) - open/close 5562 (time 1788543670.17 total 261.89 last 22.23) - open/close 5748 (time 1788543680.65 total 272.38 last 17.74) - open/close 5963 (time 1788543690.66 total 282.39 last 21.48) - open/close 6215 (time 1788543700.67 total 292.40 last 25.18) - open/close 6486 (time 1788543711.04 total 302.77 last 26.12) - open/close 6712 (time 1788543721.05 total 312.77 last 22.60) - open/close 6929 (time 1788543731.06 total 322.79 last 21.67) - open/close 7146 (time 1788543741.08 total 332.81 last 21.65) - open/close 7327 (time 1788543751.43 total 343.16 last 17.49) - open/close 7589 (time 1788543764.50 total 356.22 last 20.06) - open/close 7841 (time 1788543774.50 total 366.23 last 25.19) - open/close 8002 (time 1788543784.52 total 376.25 last 16.06) - open/close 8248 (time 1788543795.28 total 387.01 last 22.86) - open/close 8457 (time 1788543806.16 total 397.88 last 19.22) - open/close 8634 (time 1788543816.20 total 407.93 last 17.62) - open/close 8705 (time 1788543826.22 total 417.94 last 7.09) - open/close 8889 (time 1788543836.23 total 427.96 last 18.38) - open/close 9109 (time 1788543846.72 total 438.45 last 20.97) - open/close 9372 (time 1788543856.74 total 448.46 last 26.27) - open/close 9621 (time 1788543867.39 total 459.12 last 23.37) - open/close 9878 (time 1788543878.38 total 470.10 last 23.39) - open/close 10000 (time 1788543882.61 total 474.34 last 28.83) - open/close 10188 (time 1788543893.28 total 485.00 last 17.62) - open/close 10400 (time 1788543903.56 total 495.29 last 20.61) - open/close 10544 (time 1788543914.37 total 506.09 last 13.33) - open/close 10772 (time 1788543924.40 total 516.13 last 22.71) - open/close 10955 (time 1788543934.41 total 526.13 last 18.29) - open/close 11103 (time 1788543944.41 total 536.14 last 14.79) - open/close 11313 (time 1788543954.42 total 546.15 last 20.99) - open/close 11531 (time 1788543965.00 total 556.72 last 20.61) - open/close 11735 (time 1788543975.69 total 567.42 last 19.08) - open/close 11974 (time 1788543986.24 total 577.97 last 22.64) - open/close 12202 (time 1788543996.95 total 588.67 last 21.31) - open/close 12426 (time 1788544006.95 total 598.67 last 22.40) - open/close 12515 (time 1788544017.26 total 608.99 last 8.63) - open/close 12773 (time 1788544027.28 total 619.00 last 25.76) - open/close 12962 (time 1788544037.31 total 629.04 last 18.84) - open/close 13159 (time 1788544047.61 total 639.34 last 19.12) - open/close 13355 (time 1788544057.62 total 649.35 last 19.58) - open/close 13599 (time 1788544067.65 total 659.38 last 24.32) - open/close 13762 (time 1788544078.30 total 670.03 last 15.31) - open/close 13904 (time 1788544088.46 total 680.19 last 13.97) - open/close 14019 (time 1788544099.97 total 691.70 last 9.99) - open/close 14197 (time 1788544109.99 total 701.71 last 17.77) - open/close 14283 (time 1788544120.00 total 711.72 last 8.59) - open/close 14445 (time 1788544130.03 total 721.76 last 16.14) - open/close 14602 (time 1788544140.17 total 731.89 last 15.49) - open/close 14806 (time 1788544150.36 total 742.08 last 20.02) - open/close 15038 (time 1788544160.41 total 752.14 last 23.07) - open/close 15260 (time 1788544170.42 total 762.14 last 22.19) - open/close 15460 (time 1788544180.45 total 772.18 last 19.93) - open/close 15725 (time 1788544190.60 total 782.33 last 26.11) - open/close 15978 (time 1788544201.35 total 793.08 last 23.53) - open/close 16255 (time 1788544211.36 total 803.09 last 27.68) - open/close 16471 (time 1788544221.37 total 813.09 last 21.58) - open/close 16713 (time 1788544231.89 total 823.62 last 22.99) - open/close 16927 (time 1788544241.90 total 833.63 last 21.37) - open/close 17168 (time 1788544251.91 total 843.64 last 24.08) - open/close 17415 (time 1788544261.93 total 853.65 last 24.66) - open/close 17580 (time 1788544272.07 total 863.79 last 16.27) - open/close 17742 (time 1788544283.11 total 874.84 last 14.67) - open/close 17911 (time 1788544293.13 total 884.85 last 16.87) - open/close 18116 (time 1788544303.24 total 894.97 last 20.26) - open/close 18338 (time 1788544313.36 total 905.08 last 21.95) - open/close 18856 (time 1788544323.37 total 915.09 last 51.75) - open/close 19564 (time 1788544333.38 total 925.11 last 70.70) - open/close 20000 (time 1788544340.14 total 931.87 last 64.51) - open/close 20579 (time 1788544350.14 total 941.87 last 57.88) - open/close 21234 (time 1788544360.15 total 951.87 last 65.47) - open/close 21938 (time 1788544370.17 total 961.90 last 70.22) - open/close 22589 (time 1788544380.18 total 971.91 last 65.06) - open/close 23301 (time 1788544390.20 total 981.92 last 71.08) - open/close 24035 (time 1788544400.21 total 991.93 last 73.33) - open/close 24753 (time 1788544410.22 total 1001.95 last 71.70) - open/close 25317 (time 1788544420.23 total 1011.96 last 56.35) - open/close 25970 (time 1788544430.25 total 1021.98 last 65.17) - open/close 26574 (time 1788544440.26 total 1031.99 last 60.33) - open/close 27202 (time 1788544450.27 total 1042.00 last 62.74) - open/close 27989 (time 1788544460.27 total 1052.00 last 78.70) - open/close 28601 (time 1788544470.28 total 1062.00 last 61.17) - open/close 29193 (time 1788544480.28 total 1072.01 last 59.18) - open/close 29880 (time 1788544490.28 total 1082.01 last 68.67) - open/close 30618 (time 1788544500.29 total 1092.02 last 73.75) - open/close 31290 (time 1788544510.30 total 1102.02 last 67.17) - open/close 31972 (time 1788544520.30 total 1112.02 last 68.19) - open/close 32700 (time 1788544530.30 total 1122.03 last 72.79) - open/close 33351 (time 1788544540.31 total 1132.03 last 65.04) - open/close 33988 (time 1788544550.31 total 1142.04 last 63.69) - open/close 34527 (time 1788544560.31 total 1152.04 last 53.88) - open/close 35160 (time 1788544570.32 total 1162.05 last 63.24) - open/close 35835 (time 1788544580.33 total 1172.06 last 67.45) - open/close 36586 (time 1788544590.34 total 1182.07 last 75.01) - open/close 37317 (time 1788544600.35 total 1192.08 last 73.03) - open/close 37948 (time 1788544610.37 total 1202.10 last 62.96) - open/close 38636 (time 1788544620.37 total 1212.10 last 68.79) - open/close 39326 (time 1788544630.38 total 1222.10 last 68.97) - open/close 40000 (time 1788544639.68 total 1231.41 last 72.43) - open/close 40648 (time 1788544649.70 total 1241.43 last 64.69) - open/close 41321 (time 1788544659.71 total 1251.44 last 67.24) - open/close 41909 (time 1788544669.72 total 1261.45 last 58.73) - open/close 42586 (time 1788544679.74 total 1271.47 last 67.55) - open/close 43280 (time 1788544689.75 total 1281.48 last 69.33) - open/close 43948 (time 1788544699.77 total 1291.50 last 66.69) - open/close 44646 (time 1788544709.78 total 1301.51 last 69.74) - open/close 45292 (time 1788544719.79 total 1311.52 last 64.54) - open/close 46008 (time 1788544729.79 total 1321.52 last 71.56) - open/close 46690 (time 1788544739.80 total 1331.53 last 68.13) - open/close 47324 (time 1788544749.81 total 1341.54 last 63.33) - open/close 47949 (time 1788544759.82 total 1351.55 last 62.47) - open/close 48543 (time 1788544769.84 total 1361.57 last 59.26) - open/close 49127 (time 1788544779.87 total 1371.59 last 58.25) - open/close 49732 (time 1788544789.89 total 1381.62 last 60.36) total: 50000 open/close in 1386.11 seconds: 36.07 ops/second - unlinked 0 (time 1788544798 ; total 0 ; last 0) - unlinked 10000 (time 1788544946 ; total 148 ; last 148) - unlinked 20000 (time 1788545100 ; total 302 ; last 154) - unlinked 30000 (time 1788545261 ; total 463 ; last 161) - unlinked 40000 (time 1788545417 ; total 619 ; last 156) total: 50000 unlinks in 779 seconds: 64.184853 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg341-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg341-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg341-server stop mds service on oleg341-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg341-server unloading modules via unload_modules_local on: 'oleg341-server' oleg341-server: oleg341-server.virtnet: executing unload_modules_local oleg341-server: kmemleak disabled modules unloaded. pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-client: ssh exited with exit code 2 pdsh@oleg341-client: oleg341-server: ssh exited with exit code 2