== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:09:17 (1788523757) start mds service on oleg401-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 3 15:04 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg401-server' oleg401-server: oleg401-server.virtnet: executing load_modules_local oleg401-server: kmemleak disabled oleg401-server: Loading modules from /home/green/git/lustre-release/lustre oleg401-server: detected 4 online CPUs by sysfs oleg401-server: MODOPTS_LIBCFS= oleg401-server: Force libcfs to create 2 CPU partitions oleg401-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg401-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg401-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg401-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg401-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Started lustre-OST0000 oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg401-client.virtnet: -o user_xattr,flock 192.168.204.101@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 237 (time 1788523841.03 total 10.98 last 21.58) - open/close 389 (time 1788523851.23 total 21.19 last 14.89) - open/close 654 (time 1788523862.07 total 32.03 last 24.46) - open/close 847 (time 1788523872.49 total 42.45 last 18.53) - open/close 1092 (time 1788523882.50 total 52.46 last 24.47) - open/close 1356 (time 1788523892.78 total 62.73 last 25.69) - open/close 1640 (time 1788523903.34 total 73.29 last 26.90) - open/close 1876 (time 1788523913.97 total 83.93 last 22.18) - open/close 2077 (time 1788523926.34 total 96.29 last 16.26) - open/close 2240 (time 1788523936.36 total 106.31 last 16.27) - open/close 2495 (time 1788523947.19 total 117.15 last 23.53) - open/close 2720 (time 1788523957.75 total 127.71 last 21.31) - open/close 2889 (time 1788523968.51 total 138.46 last 15.71) - open/close 3049 (time 1788523979.05 total 149.00 last 15.18) - open/close 3307 (time 1788523989.55 total 159.51 last 24.55) - open/close 3504 (time 1788524000.31 total 170.27 last 18.32) - open/close 3774 (time 1788524011.32 total 181.28 last 24.52) - open/close 4019 (time 1788524021.33 total 191.29 last 24.47) - open/close 4233 (time 1788524031.36 total 201.31 last 21.35) - open/close 4459 (time 1788524041.37 total 211.33 last 22.56) - open/close 4670 (time 1788524051.94 total 221.90 last 19.96) - open/close 4908 (time 1788524062.64 total 232.60 last 22.24) - open/close 5201 (time 1788524073.26 total 243.22 last 27.59) - open/close 5429 (time 1788524083.33 total 253.29 last 22.66) - open/close 5579 (time 1788524093.58 total 263.54 last 14.63) - open/close 5763 (time 1788524104.17 total 274.13 last 17.38) - open/close 6038 (time 1788524114.43 total 284.38 last 26.81) - open/close 6280 (time 1788524124.43 total 294.39 last 24.19) - open/close 6507 (time 1788524135.08 total 305.04 last 21.32) - open/close 6801 (time 1788524145.08 total 315.04 last 29.40) - open/close 7055 (time 1788524155.37 total 325.33 last 24.68) - open/close 7245 (time 1788524165.75 total 335.71 last 18.31) - open/close 7489 (time 1788524176.18 total 346.14 last 23.39) - open/close 7716 (time 1788524186.19 total 356.15 last 22.69) - open/close 7964 (time 1788524196.21 total 366.16 last 24.75) - open/close 8097 (time 1788524206.21 total 376.17 last 13.29) - open/close 8346 (time 1788524216.22 total 386.18 last 24.88) - open/close 8557 (time 1788524226.60 total 396.56 last 20.33) - open/close 8775 (time 1788524236.61 total 406.56 last 21.79) - open/close 8966 (time 1788524247.60 total 417.55 last 17.38) - open/close 9219 (time 1788524257.65 total 427.61 last 25.17) - open/close 9405 (time 1788524267.90 total 437.86 last 18.14) - open/close 9526 (time 1788524278.65 total 448.61 last 11.26) - open/close 9602 (time 1788524288.66 total 458.62 last 7.59) - open/close 9822 (time 1788524298.66 total 468.62 last 21.99) - open/close 10000 (time 1788524305.68 total 475.64 last 25.36) - open/close 10192 (time 1788524315.69 total 485.64 last 19.19) - open/close 10408 (time 1788524326.58 total 496.54 last 19.83) - open/close 10543 (time 1788524337.52 total 507.47 last 12.34) - open/close 10709 (time 1788524347.54 total 517.49 last 16.57) - open/close 11000 (time 1788524357.55 total 527.50 last 29.07) - open/close 11212 (time 1788524367.56 total 537.52 last 21.17) - open/close 11414 (time 1788524378.23 total 548.19 last 18.93) - open/close 11540 (time 1788524389.06 total 559.02 last 11.63) - open/close 11735 (time 1788524399.08 total 569.04 last 19.46) - open/close 11900 (time 1788524411.51 total 581.46 last 13.28) - open/close 12069 (time 1788524421.56 total 591.51 last 16.82) - open/close 12264 (time 1788524431.57 total 601.52 last 19.48) - open/close 12479 (time 1788524442.16 total 612.12 last 20.29) - open/close 12607 (time 1788524452.66 total 622.62 last 12.19) - open/close 12717 (time 1788524462.68 total 632.63 last 10.98) - open/close 12829 (time 1788524472.70 total 642.66 last 11.17) - open/close 13015 (time 1788524483.64 total 653.60 last 17.00) - open/close 13255 (time 1788524494.55 total 664.51 last 22.00) - open/close 13504 (time 1788524505.02 total 674.98 last 23.79) - open/close 13762 (time 1788524516.09 total 686.05 last 23.30) - open/close 14005 (time 1788524527.05 total 697.01 last 22.16) - open/close 14232 (time 1788524537.47 total 707.43 last 21.79) - open/close 14438 (time 1788524547.48 total 717.44 last 20.57) - open/close 14627 (time 1788524558.38 total 728.33 last 17.35) - open/close 14814 (time 1788524568.38 total 738.33 last 18.70) - open/close 14999 (time 1788524578.49 total 748.45 last 18.29) - open/close 15169 (time 1788524589.56 total 759.51 last 15.36) - open/close 15385 (time 1788524601.07 total 771.03 last 18.75) - open/close 15552 (time 1788524611.09 total 781.05 last 16.67) - open/close 15673 (time 1788524626.24 total 796.20 last 7.99) - open/close 15935 (time 1788524636.25 total 806.21 last 26.17) - open/close 16046 (time 1788524646.76 total 816.72 last 10.56) - open/close 16263 (time 1788524657.22 total 827.18 last 20.75) - open/close 16516 (time 1788524668.28 total 838.23 last 22.89) - open/close 16761 (time 1788524679.34 total 849.30 last 22.14) - open/close 16969 (time 1788524689.80 total 859.76 last 19.89) - open/close 17006 (time 1788524700.11 total 870.07 last 3.59) - open/close 17137 (time 1788524710.71 total 880.66 last 12.36) - open/close 17375 (time 1788524721.30 total 891.26 last 22.47) - open/close 17568 (time 1788524731.82 total 901.78 last 18.34) - open/close 17715 (time 1788524742.84 total 912.80 last 13.34) - open/close 17820 (time 1788524753.69 total 923.65 last 9.68) - open/close 17928 (time 1788524764.60 total 934.56 last 9.90) - open/close 18058 (time 1788524774.98 total 944.94 last 12.52) - open/close 18163 (time 1788524785.01 total 954.97 last 10.47) - open/close 18250 (time 1788524795.05 total 965.01 last 8.66) - open/close 18732 (time 1788524805.05 total 975.01 last 48.20) - open/close 19356 (time 1788524815.07 total 985.02 last 62.31) - open/close 20000 (time 1788524824.57 total 994.53 last 67.75) - open/close 20677 (time 1788524834.58 total 1004.54 last 67.64) - open/close 21273 (time 1788524844.58 total 1014.54 last 59.58) - open/close 22010 (time 1788524854.59 total 1024.55 last 73.63) - open/close 22729 (time 1788524864.60 total 1034.56 last 71.84) - open/close 23331 (time 1788524874.61 total 1044.57 last 60.12) - open/close 23884 (time 1788524884.63 total 1054.59 last 55.22) - open/close 24569 (time 1788524894.63 total 1064.59 last 68.50) - open/close 25221 (time 1788524904.63 total 1074.59 last 65.20) - open/close 25832 (time 1788524914.64 total 1084.60 last 61.04) - open/close 26514 (time 1788524924.65 total 1094.60 last 68.15) - open/close 27256 (time 1788524934.66 total 1104.62 last 74.10) - open/close 27983 (time 1788524944.66 total 1114.62 last 72.69) - open/close 28735 (time 1788524954.66 total 1124.62 last 75.20) - open/close 29447 (time 1788524964.67 total 1134.63 last 71.16) - open/close 30000 (time 1788524972.67 total 1142.63 last 69.10) - open/close 30677 (time 1788524982.68 total 1152.64 last 67.65) - open/close 31398 (time 1788524992.69 total 1162.65 last 72.02) - open/close 32072 (time 1788525002.70 total 1172.66 last 67.30) - open/close 32845 (time 1788525012.71 total 1182.67 last 77.26) - open/close 33574 (time 1788525022.72 total 1192.68 last 72.80) - open/close 34291 (time 1788525032.73 total 1202.69 last 71.64) - open/close 34989 (time 1788525042.73 total 1212.69 last 69.79) - open/close 35768 (time 1788525052.74 total 1222.69 last 77.87) - open/close 36474 (time 1788525062.74 total 1232.70 last 70.57) - open/close 37141 (time 1788525072.75 total 1242.71 last 66.65) - open/close 37858 (time 1788525082.76 total 1252.72 last 71.63) - open/close 38562 (time 1788525092.77 total 1262.73 last 70.33) - open/close 39208 (time 1788525102.78 total 1272.74 last 64.54) - open/close 39869 (time 1788525112.78 total 1282.74 last 66.10) - open/close 40000 (time 1788525114.80 total 1284.76 last 64.83) - open/close 40665 (time 1788525124.81 total 1294.76 last 66.45) - open/close 41324 (time 1788525134.82 total 1304.78 last 65.78) - open/close 41972 (time 1788525144.89 total 1314.85 last 64.34) - open/close 42624 (time 1788525154.90 total 1324.86 last 65.15) - open/close 43357 (time 1788525164.91 total 1334.87 last 73.23) - open/close 44025 (time 1788525174.92 total 1344.88 last 66.76) - open/close 44700 (time 1788525184.93 total 1354.89 last 67.42) - open/close 45389 (time 1788525194.94 total 1364.90 last 68.84) - open/close 46068 (time 1788525204.94 total 1374.90 last 67.87) - open/close 46749 (time 1788525214.96 total 1384.91 last 68.01) - open/close 47429 (time 1788525224.97 total 1394.93 last 67.90) - open/close 48126 (time 1788525235.00 total 1404.95 last 69.52) - open/close 48792 (time 1788525245.01 total 1414.96 last 66.54) - open/close 49465 (time 1788525255.01 total 1424.96 last 67.30) total: 50000 open/close in 1432.45 seconds: 34.91 ops/second - unlinked 0 (time 1788525267 ; total 0 ; last 0) - unlinked 10000 (time 1788525414 ; total 147 ; last 147) - unlinked 20000 (time 1788525571 ; total 304 ; last 157) - unlinked 30000 (time 1788525718 ; total 451 ; last 147) - unlinked 40000 (time 1788525875 ; total 608 ; last 157) total: 50000 unlinks in 755 seconds: 66.225166 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg401-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg401-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg401-server stop mds service on oleg401-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg401-server unloading modules via unload_modules_local on: 'oleg401-server' oleg401-server: oleg401-server.virtnet: executing unload_modules_local oleg401-server: kmemleak disabled modules unloaded. pdsh@oleg401-client: oleg401-client: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-client: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2