== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:18:04 (1788571084) start mds service on oleg405-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 4 03:00 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing load_modules_local oleg405-server: kmemleak disabled oleg405-server: Loading modules from /home/green/git/lustre-release/lustre oleg405-server: detected 4 online CPUs by sysfs oleg405-server: MODOPTS_LIBCFS= oleg405-server: Force libcfs to create 2 CPU partitions oleg405-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg405-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg405-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg405-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg405-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 Started lustre-OST0000 oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 207 (time 1788571164.76 total 10.01 last 20.68) - open/close 413 (time 1788571174.83 total 20.08 last 20.46) - open/close 665 (time 1788571184.84 total 30.09 last 25.17) - open/close 891 (time 1788571194.84 total 40.09 last 22.59) - open/close 1150 (time 1788571205.67 total 50.91 last 23.93) - open/close 1355 (time 1788571215.67 total 60.91 last 20.50) - open/close 1580 (time 1788571226.29 total 71.53 last 21.18) - open/close 1867 (time 1788571236.38 total 81.63 last 28.43) - open/close 2077 (time 1788571246.81 total 92.06 last 20.14) - open/close 2282 (time 1788571257.84 total 103.09 last 18.58) - open/close 2537 (time 1788571267.85 total 113.10 last 25.47) - open/close 2720 (time 1788571279.80 total 125.05 last 15.32) - open/close 2929 (time 1788571289.81 total 135.05 last 20.89) - open/close 3134 (time 1788571299.82 total 145.07 last 20.47) - open/close 3382 (time 1788571310.57 total 155.82 last 23.07) - open/close 3634 (time 1788571320.57 total 165.82 last 25.20) - open/close 3880 (time 1788571330.75 total 176.00 last 24.16) - open/close 4090 (time 1788571341.05 total 186.29 last 20.40) - open/close 4330 (time 1788571352.04 total 197.28 last 21.84) - open/close 4552 (time 1788571362.67 total 207.92 last 20.88) - open/close 4757 (time 1788571373.19 total 218.43 last 19.49) - open/close 4995 (time 1788571383.22 total 228.47 last 23.72) - open/close 5266 (time 1788571394.17 total 239.42 last 24.75) - open/close 5536 (time 1788571404.17 total 249.42 last 27.00) - open/close 5737 (time 1788571414.18 total 259.42 last 20.09) - open/close 5997 (time 1788571424.94 total 270.18 last 24.16) - open/close 6224 (time 1788571434.95 total 280.20 last 22.67) - open/close 6464 (time 1788571444.95 total 290.20 last 23.99) - open/close 6602 (time 1788571455.87 total 301.11 last 12.65) - open/close 6729 (time 1788571466.03 total 311.28 last 12.49) - open/close 6808 (time 1788571476.30 total 321.54 last 7.70) - open/close 6921 (time 1788571486.71 total 331.96 last 10.85) - open/close 7109 (time 1788571496.72 total 341.97 last 18.78) - open/close 7298 (time 1788571506.73 total 351.97 last 18.90) - open/close 7566 (time 1788571516.74 total 361.98 last 26.78) - open/close 7782 (time 1788571527.03 total 372.28 last 20.98) - open/close 8031 (time 1788571539.19 total 384.44 last 20.48) - open/close 8295 (time 1788571549.20 total 394.45 last 26.36) - open/close 8473 (time 1788571559.47 total 404.72 last 17.34) - open/close 8720 (time 1788571570.42 total 415.66 last 22.56) - open/close 8885 (time 1788571581.20 total 426.45 last 15.30) - open/close 9065 (time 1788571591.22 total 436.46 last 17.97) - open/close 9200 (time 1788571601.33 total 446.58 last 13.35) - open/close 9343 (time 1788571613.95 total 459.19 last 11.33) - open/close 9604 (time 1788571623.98 total 469.23 last 26.01) - open/close 9842 (time 1788571634.87 total 480.12 last 21.85) - open/close 9989 (time 1788571645.30 total 490.54 last 14.10) - open/close 10188 (time 1788571655.31 total 500.55 last 19.87) - open/close 10381 (time 1788571665.32 total 510.56 last 19.29) - open/close 10584 (time 1788571675.83 total 521.07 last 19.31) - open/close 10834 (time 1788571685.84 total 531.08 last 24.97) - open/close 11103 (time 1788571695.84 total 541.09 last 26.88) - open/close 11331 (time 1788571705.86 total 551.10 last 22.77) - open/close 11554 (time 1788571716.20 total 561.44 last 21.57) - open/close 11800 (time 1788571726.21 total 571.46 last 24.56) - open/close 11996 (time 1788571736.23 total 581.48 last 19.56) - open/close 12253 (time 1788571746.26 total 591.50 last 25.64) - open/close 12515 (time 1788571758.98 total 604.23 last 20.59) - open/close 12705 (time 1788571769.71 total 614.95 last 17.71) - open/close 12871 (time 1788571780.09 total 625.33 last 16.00) - open/close 13048 (time 1788571790.59 total 635.83 last 16.85) - open/close 13209 (time 1788571801.19 total 646.43 last 15.19) - open/close 13408 (time 1788571811.75 total 657.00 last 18.84) - open/close 13729 (time 1788571821.78 total 667.02 last 32.02) - open/close 13988 (time 1788571832.11 total 677.36 last 25.06) - open/close 14252 (time 1788571842.35 total 687.60 last 25.78) - open/close 14450 (time 1788571852.37 total 697.61 last 19.77) - open/close 14594 (time 1788571863.41 total 708.65 last 13.04) - open/close 14874 (time 1788571873.46 total 718.71 last 27.85) - open/close 15124 (time 1788571884.09 total 729.34 last 23.51) - open/close 15236 (time 1788571894.79 total 740.04 last 10.47) - open/close 15353 (time 1788571904.91 total 750.16 last 11.56) - open/close 15494 (time 1788571916.77 total 762.02 last 11.89) - open/close 15721 (time 1788571927.39 total 772.64 last 21.37) - open/close 15961 (time 1788571937.42 total 782.66 last 23.94) - open/close 16115 (time 1788571947.50 total 792.74 last 15.28) - open/close 16313 (time 1788571957.54 total 802.79 last 19.71) - open/close 16515 (time 1788571967.57 total 812.81 last 20.16) - open/close 16783 (time 1788571977.57 total 822.82 last 26.78) - open/close 17001 (time 1788571988.11 total 833.35 last 20.70) - open/close 17244 (time 1788571999.01 total 844.26 last 22.28) - open/close 17493 (time 1788572009.02 total 854.26 last 24.89) - open/close 17711 (time 1788572019.15 total 864.39 last 21.53) - open/close 17817 (time 1788572029.74 total 874.99 last 10.00) - open/close 18112 (time 1788572041.70 total 886.95 last 24.66) - open/close 18346 (time 1788572052.35 total 897.59 last 21.98) - open/close 18514 (time 1788572062.50 total 907.74 last 16.55) - open/close 18780 (time 1788572073.53 total 918.77 last 24.11) - open/close 18872 (time 1788572083.54 total 928.78 last 9.19) - open/close 19060 (time 1788572094.66 total 939.90 last 16.91) - open/close 19195 (time 1788572105.28 total 950.53 last 12.70) - open/close 19510 (time 1788572115.29 total 960.53 last 31.50) - open/close 20000 (time 1788572121.33 total 966.58 last 81.03) - open/close 20697 (time 1788572131.33 total 976.58 last 69.69) - open/close 21410 (time 1788572141.34 total 986.58 last 71.28) - open/close 22221 (time 1788572151.35 total 996.59 last 81.03) - open/close 22981 (time 1788572161.36 total 1006.61 last 75.89) - open/close 23698 (time 1788572171.36 total 1016.61 last 71.69) - open/close 24344 (time 1788572181.38 total 1026.62 last 64.50) - open/close 25040 (time 1788572191.40 total 1036.64 last 69.45) - open/close 25693 (time 1788572201.41 total 1046.65 last 65.25) - open/close 26403 (time 1788572211.42 total 1056.67 last 70.88) - open/close 27174 (time 1788572221.44 total 1066.68 last 77.00) - open/close 27958 (time 1788572231.44 total 1076.68 last 78.39) - open/close 28699 (time 1788572241.44 total 1086.68 last 74.08) - open/close 29526 (time 1788572251.44 total 1096.69 last 82.67) - open/close 30000 (time 1788572258.01 total 1103.25 last 72.22) - open/close 30754 (time 1788572268.02 total 1113.27 last 75.29) - open/close 31464 (time 1788572278.02 total 1123.27 last 70.97) - open/close 32255 (time 1788572288.03 total 1133.27 last 79.07) - open/close 33022 (time 1788572298.03 total 1143.28 last 76.65) - open/close 33755 (time 1788572308.04 total 1153.29 last 73.25) - open/close 34535 (time 1788572318.05 total 1163.30 last 77.92) - open/close 35319 (time 1788572328.06 total 1173.30 last 78.34) - open/close 35991 (time 1788572338.06 total 1183.31 last 67.17) - open/close 36762 (time 1788572348.07 total 1193.31 last 77.05) - open/close 37471 (time 1788572358.07 total 1203.32 last 70.89) - open/close 38228 (time 1788572368.09 total 1213.33 last 75.58) - open/close 38976 (time 1788572378.10 total 1223.34 last 74.71) - open/close 39584 (time 1788572388.12 total 1233.36 last 60.67) - open/close 40000 (time 1788572393.86 total 1239.10 last 72.48) - open/close 40718 (time 1788572403.88 total 1249.12 last 71.68) - open/close 41394 (time 1788572413.89 total 1259.13 last 67.51) - open/close 42100 (time 1788572423.91 total 1269.16 last 70.44) - open/close 42797 (time 1788572433.91 total 1279.16 last 69.68) - open/close 43592 (time 1788572443.92 total 1289.16 last 79.47) - open/close 44372 (time 1788572453.95 total 1299.19 last 77.74) - open/close 45111 (time 1788572463.96 total 1309.20 last 73.85) - open/close 45844 (time 1788572473.96 total 1319.20 last 73.28) - open/close 46516 (time 1788572483.97 total 1329.21 last 67.15) - open/close 47211 (time 1788572493.99 total 1339.23 last 69.35) - open/close 47918 (time 1788572504.01 total 1349.26 last 70.52) - open/close 48648 (time 1788572514.02 total 1359.26 last 72.97) - open/close 49392 (time 1788572524.03 total 1369.27 last 74.33) total: 50000 open/close in 1377.38 seconds: 36.30 ops/second - unlinked 0 (time 1788572535 ; total 0 ; last 0) - unlinked 10000 (time 1788572677 ; total 142 ; last 142) - unlinked 20000 (time 1788572827 ; total 292 ; last 150) - unlinked 30000 (time 1788572969 ; total 434 ; last 142) - unlinked 40000 (time 1788573110 ; total 575 ; last 141) total: 50000 unlinks in 718 seconds: 69.637886 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg405-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg405-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg405-server stop mds service on oleg405-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg405-server unloading modules via unload_modules_local on: 'oleg405-server' oleg405-server: oleg405-server.virtnet: executing unload_modules_local oleg405-server: kmemleak disabled modules unloaded. pdsh@oleg405-client: oleg405-client: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-client: ssh exited with exit code 2 pdsh@oleg405-client: oleg405-server: ssh exited with exit code 2