== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:12:07 (1788487927) start mds service on oleg347-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1754920 Sep 3 10:02 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing load_modules_local oleg347-server: kmemleak disabled oleg347-server: Loading modules from /home/green/git/lustre-release/lustre oleg347-server: detected 4 online CPUs by sysfs oleg347-server: MODOPTS_LIBCFS= oleg347-server: Force libcfs to create 2 CPU partitions oleg347-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg347-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg347-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg347-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg347-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock 192.168.203.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 137 (time 1788488009.17 total 10.96 last 12.50) - open/close 359 (time 1788488019.18 total 20.97 last 22.18) - open/close 490 (time 1788488029.19 total 30.97 last 13.09) - open/close 738 (time 1788488039.20 total 40.99 last 24.76) - open/close 846 (time 1788488049.76 total 51.55 last 10.23) - open/close 1103 (time 1788488059.77 total 61.56 last 25.69) - open/close 1321 (time 1788488069.97 total 71.76 last 21.36) - open/close 1574 (time 1788488080.98 total 82.77 last 22.98) - open/close 1782 (time 1788488091.35 total 93.14 last 20.06) - open/close 1974 (time 1788488106.02 total 107.81 last 13.09) - open/close 2190 (time 1788488116.68 total 118.46 last 20.27) - open/close 2416 (time 1788488126.70 total 128.49 last 22.54) - open/close 2591 (time 1788488137.78 total 139.57 last 15.80) - open/close 2752 (time 1788488148.31 total 150.10 last 15.28) - open/close 2981 (time 1788488158.81 total 160.60 last 21.81) - open/close 3138 (time 1788488168.85 total 170.64 last 15.64) - open/close 3289 (time 1788488178.88 total 180.67 last 15.05) - open/close 3442 (time 1788488190.00 total 191.79 last 13.76) - open/close 3655 (time 1788488200.02 total 201.80 last 21.26) - open/close 3802 (time 1788488211.67 total 213.46 last 12.61) - open/close 3989 (time 1788488221.68 total 223.47 last 18.69) - open/close 4207 (time 1788488231.69 total 233.47 last 21.79) - open/close 4379 (time 1788488242.66 total 244.45 last 15.68) - open/close 4537 (time 1788488252.88 total 254.66 last 15.46) - open/close 4704 (time 1788488263.31 total 265.10 last 16.01) - open/close 4962 (time 1788488274.25 total 276.04 last 23.58) - open/close 5221 (time 1788488284.25 total 286.04 last 25.90) - open/close 5477 (time 1788488294.25 total 296.04 last 25.59) - open/close 5692 (time 1788488304.29 total 306.08 last 21.42) - open/close 5900 (time 1788488314.52 total 316.31 last 20.33) - open/close 6121 (time 1788488324.54 total 326.32 last 22.07) - open/close 6300 (time 1788488334.79 total 336.58 last 17.45) - open/close 6542 (time 1788488345.35 total 347.14 last 22.92) - open/close 6812 (time 1788488355.37 total 357.15 last 26.96) - open/close 6944 (time 1788488365.38 total 367.17 last 13.18) - open/close 7157 (time 1788488376.01 total 377.80 last 20.03) - open/close 7359 (time 1788488386.47 total 388.26 last 19.31) - open/close 7637 (time 1788488396.50 total 398.28 last 27.73) - open/close 7825 (time 1788488407.12 total 408.91 last 17.70) - open/close 8056 (time 1788488417.13 total 418.91 last 23.08) - open/close 8294 (time 1788488427.52 total 429.31 last 22.89) - open/close 8508 (time 1788488437.58 total 439.37 last 21.27) - open/close 8742 (time 1788488447.59 total 449.37 last 23.39) - open/close 8878 (time 1788488462.73 total 464.52 last 8.98) - open/close 9057 (time 1788488472.76 total 474.55 last 17.85) - open/close 9211 (time 1788488482.97 total 484.76 last 15.09) - open/close 9382 (time 1788488493.01 total 494.80 last 17.02) - open/close 9488 (time 1788488503.13 total 504.92 last 10.48) - open/close 9609 (time 1788488513.13 total 514.92 last 12.10) - open/close 9817 (time 1788488525.20 total 526.99 last 17.24) - open/close 9935 (time 1788488535.32 total 537.11 last 11.66) - open/close 10000 (time 1788488540.07 total 541.85 last 13.69) - open/close 10200 (time 1788488550.36 total 552.15 last 19.43) - open/close 10428 (time 1788488561.03 total 562.82 last 21.37) - open/close 10557 (time 1788488571.42 total 573.21 last 12.42) - open/close 10711 (time 1788488582.35 total 584.14 last 14.08) - open/close 10816 (time 1788488594.06 total 595.85 last 8.97) - open/close 10892 (time 1788488604.08 total 605.86 last 7.59) - open/close 11109 (time 1788488614.87 total 616.66 last 20.10) - open/close 11242 (time 1788488624.87 total 626.66 last 13.30) - open/close 11449 (time 1788488634.89 total 636.68 last 20.66) - open/close 11543 (time 1788488645.71 total 647.50 last 8.69) - open/close 11711 (time 1788488655.77 total 657.56 last 16.70) - open/close 11857 (time 1788488666.40 total 668.19 last 13.73) - open/close 11884 (time 1788488677.78 total 679.57 last 2.37) - open/close 11983 (time 1788488688.48 total 690.26 last 9.26) - open/close 12059 (time 1788488698.77 total 700.55 last 7.39) - open/close 12174 (time 1788488709.84 total 711.63 last 10.39) - open/close 12283 (time 1788488719.86 total 721.64 last 10.88) - open/close 12406 (time 1788488729.87 total 731.66 last 12.28) - open/close 12503 (time 1788488740.27 total 742.06 last 9.33) - open/close 12729 (time 1788488750.94 total 752.72 last 21.19) - open/close 12898 (time 1788488761.24 total 763.03 last 16.40) - open/close 13007 (time 1788488771.41 total 773.20 last 10.72) - open/close 13078 (time 1788488782.30 total 784.09 last 6.52) - open/close 13316 (time 1788488797.53 total 799.32 last 15.63) - open/close 13513 (time 1788488808.02 total 809.80 last 18.78) - open/close 13674 (time 1788488818.34 total 820.13 last 15.60) - open/close 13843 (time 1788488829.35 total 831.14 last 15.35) - open/close 14032 (time 1788488839.71 total 841.50 last 18.24) - open/close 14254 (time 1788488850.14 total 851.93 last 21.28) - open/close 14456 (time 1788488861.81 total 863.60 last 17.31) - open/close 14694 (time 1788488872.79 total 874.57 last 21.68) - open/close 14830 (time 1788488882.80 total 884.58 last 13.59) - open/close 15007 (time 1788488893.36 total 895.14 last 16.76) - open/close 15078 (time 1788488903.36 total 905.15 last 7.10) - open/close 15191 (time 1788488915.24 total 917.03 last 9.51) - open/close 15267 (time 1788488925.24 total 927.03 last 7.60) - open/close 15367 (time 1788488936.10 total 937.89 last 9.21) - open/close 15492 (time 1788488946.19 total 947.98 last 12.39) - open/close 15595 (time 1788488956.20 total 957.99 last 10.29) - open/close 15680 (time 1788488966.23 total 968.02 last 8.47) - open/close 15861 (time 1788488977.62 total 979.41 last 15.89) - open/close 15936 (time 1788488987.63 total 989.41 last 7.50) - open/close 16090 (time 1788488998.43 total 1000.22 last 14.25) - open/close 16286 (time 1788489009.11 total 1010.90 last 18.35) - open/close 16402 (time 1788489019.24 total 1021.02 last 11.46) - open/close 16783 (time 1788489029.25 total 1031.03 last 38.06) - open/close 17368 (time 1788489039.25 total 1041.04 last 58.46) - open/close 17949 (time 1788489049.26 total 1051.05 last 58.07) - open/close 18534 (time 1788489059.27 total 1061.05 last 58.46) - open/close 19165 (time 1788489069.27 total 1071.06 last 63.04) - open/close 19736 (time 1788489079.28 total 1081.07 last 57.05) - open/close 20000 (time 1788489083.93 total 1085.72 last 56.78) - open/close 20560 (time 1788489093.95 total 1095.74 last 55.88) - open/close 21193 (time 1788489103.97 total 1105.75 last 63.22) - open/close 21781 (time 1788489113.97 total 1115.75 last 58.80) - open/close 22438 (time 1788489123.98 total 1125.77 last 65.62) - open/close 23107 (time 1788489133.98 total 1135.77 last 66.86) - open/close 23710 (time 1788489144.00 total 1145.79 last 60.20) - open/close 24348 (time 1788489154.01 total 1155.80 last 63.76) - open/close 25002 (time 1788489164.01 total 1165.80 last 65.38) - open/close 25560 (time 1788489174.02 total 1175.81 last 55.76) - open/close 26125 (time 1788489184.03 total 1185.81 last 56.45) - open/close 26726 (time 1788489194.04 total 1195.83 last 60.01) - open/close 27303 (time 1788489204.05 total 1205.84 last 57.64) - open/close 27888 (time 1788489214.06 total 1215.85 last 58.44) - open/close 28453 (time 1788489224.06 total 1225.85 last 56.49) - open/close 29065 (time 1788489234.07 total 1235.86 last 61.16) - open/close 29625 (time 1788489244.07 total 1245.86 last 55.99) - open/close 30000 (time 1788489250.22 total 1252.01 last 60.94) - open/close 30631 (time 1788489260.23 total 1262.02 last 63.04) - open/close 31267 (time 1788489270.25 total 1272.04 last 63.51) - open/close 31869 (time 1788489280.25 total 1282.04 last 60.19) - open/close 32422 (time 1788489290.26 total 1292.04 last 55.27) - open/close 33092 (time 1788489300.26 total 1302.05 last 66.96) - open/close 33792 (time 1788489310.26 total 1312.05 last 69.98) - open/close 34475 (time 1788489320.28 total 1322.06 last 68.21) - open/close 35171 (time 1788489330.28 total 1332.07 last 69.56) - open/close 35816 (time 1788489340.30 total 1342.09 last 64.40) - open/close 36413 (time 1788489350.32 total 1352.11 last 59.56) - open/close 36959 (time 1788489360.36 total 1362.15 last 54.37) - open/close 37508 (time 1788489370.36 total 1372.15 last 54.90) - open/close 38173 (time 1788489380.37 total 1382.15 last 66.49) - open/close 38869 (time 1788489390.38 total 1392.16 last 69.52) - open/close 39624 (time 1788489400.38 total 1402.17 last 75.47) - open/close 40000 (time 1788489405.58 total 1407.36 last 72.35) - open/close 40715 (time 1788489415.58 total 1417.37 last 71.47) - open/close 41390 (time 1788489425.59 total 1427.38 last 67.44) - open/close 42075 (time 1788489435.60 total 1437.39 last 68.45) - open/close 42820 (time 1788489445.60 total 1447.39 last 74.47) - open/close 43479 (time 1788489455.61 total 1457.40 last 65.85) - open/close 44178 (time 1788489465.62 total 1467.41 last 69.84) - open/close 44822 (time 1788489475.64 total 1477.43 last 64.26) - open/close 45497 (time 1788489485.65 total 1487.44 last 67.43) - open/close 46179 (time 1788489495.66 total 1497.45 last 68.09) - open/close 46861 (time 1788489505.67 total 1507.46 last 68.13) - open/close 47537 (time 1788489515.68 total 1517.46 last 67.59) - open/close 48218 (time 1788489525.68 total 1527.47 last 68.05) - open/close 48897 (time 1788489535.69 total 1537.47 last 67.88) - open/close 49563 (time 1788489545.70 total 1547.49 last 66.53) total: 50000 open/close in 1553.17 seconds: 32.19 ops/second - unlinked 0 (time 1788489556 ; total 0 ; last 0) - unlinked 10000 (time 1788489713 ; total 157 ; last 157) - unlinked 20000 (time 1788489866 ; total 310 ; last 153) - unlinked 30000 (time 1788490029 ; total 473 ; last 163) - unlinked 40000 (time 1788490188 ; total 632 ; last 159) total: 50000 unlinks in 792 seconds: 63.131313 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server unloading modules via unload_modules_local on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing unload_modules_local oleg347-server: kmemleak disabled modules unloaded. pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2