== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:46:25 (1787795185) start mds service on oleg313-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1755968 Aug 26 20:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing load_modules_local oleg313-server: kmemleak disabled oleg313-server: Loading modules from /home/green/git/lustre-release/lustre oleg313-server: detected 4 online CPUs by sysfs oleg313-server: MODOPTS_LIBCFS= oleg313-server: Force libcfs to create 2 CPU partitions oleg313-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg313-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg313-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg313-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg313-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg313-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg313-server: oleg313-server.virtnet: executing set_default_debug -1 all pdsh@oleg313-client: oleg313-server: ssh exited with exit code 1 Started lustre-OST0000 oleg313-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg313-client.virtnet: -o user_xattr,flock 192.168.203.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 101 (time 1787795277.98 total 10.73 last 9.41) - open/close 196 (time 1787795288.28 total 21.03 last 9.22) - open/close 313 (time 1787795298.28 total 31.03 last 11.70) - open/close 560 (time 1787795308.61 total 41.36 last 23.92) - open/close 744 (time 1787795318.61 total 51.36 last 18.40) - open/close 915 (time 1787795329.28 total 62.03 last 16.03) - open/close 1177 (time 1787795339.69 total 72.44 last 25.16) - open/close 1433 (time 1787795350.59 total 83.34 last 23.49) - open/close 1668 (time 1787795360.62 total 93.37 last 23.43) - open/close 1785 (time 1787795370.66 total 103.42 last 11.65) - open/close 1963 (time 1787795380.67 total 113.42 last 17.80) - open/close 2127 (time 1787795391.05 total 123.80 last 15.79) - open/close 2288 (time 1787795401.32 total 134.07 last 15.69) - open/close 2472 (time 1787795412.10 total 144.85 last 17.07) - open/close 2683 (time 1787795422.15 total 154.90 last 21.00) - open/close 2882 (time 1787795432.33 total 165.08 last 19.54) - open/close 2949 (time 1787795444.51 total 177.26 last 5.50) - open/close 3158 (time 1787795454.52 total 187.27 last 20.89) - open/close 3316 (time 1787795465.87 total 198.62 last 13.92) - open/close 3556 (time 1787795475.91 total 208.66 last 23.90) - open/close 3805 (time 1787795486.66 total 219.41 last 23.16) - open/close 4076 (time 1787795496.67 total 229.42 last 27.07) - open/close 4209 (time 1787795506.69 total 239.44 last 13.28) - open/close 4437 (time 1787795516.70 total 249.45 last 22.77) - open/close 4636 (time 1787795527.81 total 260.56 last 17.91) - open/close 4817 (time 1787795537.83 total 270.58 last 18.06) - open/close 4900 (time 1787795547.83 total 280.59 last 8.30) - open/close 5111 (time 1787795557.85 total 290.60 last 21.07) - open/close 5273 (time 1787795568.03 total 300.78 last 15.91) - open/close 5417 (time 1787795578.34 total 311.09 last 13.98) - open/close 5610 (time 1787795588.89 total 321.64 last 18.28) - open/close 5747 (time 1787795599.62 total 332.37 last 12.76) - open/close 5943 (time 1787795609.63 total 342.38 last 19.58) - open/close 6019 (time 1787795620.04 total 352.79 last 7.30) - open/close 6257 (time 1787795630.68 total 363.43 last 22.38) - open/close 6461 (time 1787795641.50 total 374.25 last 18.85) - open/close 6640 (time 1787795651.74 total 384.49 last 17.49) - open/close 6765 (time 1787795661.97 total 394.72 last 12.21) - open/close 6889 (time 1787795672.65 total 405.40 last 11.61) - open/close 6983 (time 1787795683.36 total 416.11 last 8.78) - open/close 7131 (time 1787795693.38 total 426.13 last 14.77) - open/close 7203 (time 1787795703.78 total 436.54 last 6.92) - open/close 7396 (time 1787795713.81 total 446.56 last 19.25) - open/close 7530 (time 1787795723.92 total 456.67 last 13.26) - open/close 7668 (time 1787795734.68 total 467.44 last 12.82) - open/close 7871 (time 1787795744.69 total 477.44 last 20.29) - open/close 8081 (time 1787795755.72 total 488.47 last 19.04) - open/close 8295 (time 1787795765.72 total 498.47 last 21.39) - open/close 8500 (time 1787795776.32 total 509.07 last 19.34) - open/close 8611 (time 1787795787.08 total 519.83 last 10.32) - open/close 8681 (time 1787795797.83 total 530.59 last 6.51) - open/close 8844 (time 1787795808.06 total 540.81 last 15.94) - open/close 8982 (time 1787795819.26 total 552.01 last 12.32) - open/close 9123 (time 1787795829.85 total 562.60 last 13.31) - open/close 9308 (time 1787795840.34 total 573.09 last 17.64) - open/close 9457 (time 1787795851.98 total 584.73 last 12.80) - open/close 9551 (time 1787795862.48 total 595.23 last 8.96) - open/close 9650 (time 1787795872.85 total 605.60 last 9.55) - open/close 9728 (time 1787795882.85 total 615.60 last 7.79) - open/close 9832 (time 1787795892.87 total 625.62 last 10.39) - open/close 10000 (time 1787795900.20 total 632.96 last 22.89) - open/close 10211 (time 1787795910.99 total 643.74 last 19.57) - open/close 10390 (time 1787795921.75 total 654.50 last 16.64) - open/close 10580 (time 1787795931.85 total 664.60 last 18.81) - open/close 10819 (time 1787795941.86 total 674.61 last 23.87) - open/close 11025 (time 1787795951.94 total 684.69 last 20.44) - open/close 11261 (time 1787795962.23 total 694.98 last 22.92) - open/close 11502 (time 1787795972.24 total 704.99 last 24.10) - open/close 11633 (time 1787795982.27 total 715.02 last 13.05) - open/close 11803 (time 1787795992.85 total 725.60 last 16.07) - open/close 11989 (time 1787796007.26 total 740.01 last 12.91) - open/close 12220 (time 1787796018.00 total 750.75 last 21.51) - open/close 12475 (time 1787796028.66 total 761.41 last 23.91) - open/close 12705 (time 1787796039.44 total 772.19 last 21.34) - open/close 12825 (time 1787796049.48 total 782.23 last 11.96) - open/close 13055 (time 1787796059.50 total 792.25 last 22.96) - open/close 13210 (time 1787796069.78 total 802.53 last 15.07) - open/close 13382 (time 1787796079.79 total 812.54 last 17.18) - open/close 13606 (time 1787796089.79 total 822.54 last 22.40) - open/close 13819 (time 1787796099.80 total 832.55 last 21.28) - open/close 14038 (time 1787796109.83 total 842.58 last 21.84) - open/close 14253 (time 1787796120.54 total 853.29 last 20.08) - open/close 14429 (time 1787796130.54 total 863.30 last 17.59) - open/close 14656 (time 1787796141.19 total 873.94 last 21.32) - open/close 14874 (time 1787796151.20 total 883.95 last 21.79) - open/close 15102 (time 1787796161.21 total 893.96 last 22.77) - open/close 15310 (time 1787796171.33 total 904.08 last 20.56) - open/close 15555 (time 1787796181.35 total 914.10 last 24.44) - open/close 15748 (time 1787796191.36 total 924.11 last 19.27) - open/close 15962 (time 1787796201.37 total 934.12 last 21.39) - open/close 16191 (time 1787796212.22 total 944.97 last 21.10) - open/close 16355 (time 1787796222.40 total 955.15 last 16.12) - open/close 16586 (time 1787796232.91 total 965.66 last 21.97) - open/close 16769 (time 1787796244.69 total 977.44 last 15.54) - open/close 16900 (time 1787796255.05 total 987.80 last 12.64) - open/close 17112 (time 1787796265.42 total 998.17 last 20.44) - open/close 17351 (time 1787796275.47 total 1008.22 last 23.79) - open/close 17965 (time 1787796285.48 total 1018.23 last 61.32) - open/close 18568 (time 1787796295.49 total 1028.24 last 60.25) - open/close 19219 (time 1787796305.50 total 1038.25 last 65.06) - open/close 19911 (time 1787796315.50 total 1048.25 last 69.20) - open/close 20609 (time 1787796325.50 total 1058.25 last 69.74) - open/close 21268 (time 1787796335.51 total 1068.26 last 65.89) - open/close 21960 (time 1787796345.53 total 1078.28 last 69.05) - open/close 22653 (time 1787796355.54 total 1088.29 last 69.25) - open/close 23324 (time 1787796365.55 total 1098.30 last 67.03) - open/close 24033 (time 1787796375.55 total 1108.31 last 70.84) - open/close 24700 (time 1787796385.56 total 1118.31 last 66.64) - open/close 25336 (time 1787796395.58 total 1128.33 last 63.51) - open/close 26051 (time 1787796405.60 total 1138.35 last 71.36) - open/close 26725 (time 1787796415.60 total 1148.35 last 67.38) - open/close 27296 (time 1787796425.61 total 1158.36 last 57.04) - open/close 27994 (time 1787796435.62 total 1168.37 last 69.76) - open/close 28645 (time 1787796445.62 total 1178.37 last 65.06) - open/close 29343 (time 1787796455.63 total 1188.38 last 69.75) - open/close 29995 (time 1787796465.63 total 1198.39 last 65.17) - open/close 30692 (time 1787796475.65 total 1208.40 last 69.61) - open/close 31355 (time 1787796485.65 total 1218.40 last 66.25) - open/close 32025 (time 1787796495.72 total 1228.47 last 66.58) - open/close 32651 (time 1787796505.72 total 1238.47 last 62.56) - open/close 33184 (time 1787796515.74 total 1248.49 last 53.24) - open/close 33751 (time 1787796525.74 total 1258.49 last 56.68) - open/close 34422 (time 1787796535.75 total 1268.50 last 67.02) - open/close 35045 (time 1787796545.75 total 1278.50 last 62.30) - open/close 35749 (time 1787796555.75 total 1288.50 last 70.40) - open/close 36400 (time 1787796565.76 total 1298.51 last 65.06) - open/close 37000 (time 1787796575.78 total 1308.53 last 59.88) - open/close 37595 (time 1787796585.78 total 1318.53 last 59.49) - open/close 38262 (time 1787796595.79 total 1328.54 last 66.62) - open/close 38840 (time 1787796605.79 total 1338.54 last 57.79) - open/close 39509 (time 1787796615.79 total 1348.54 last 66.89) - open/close 40000 (time 1787796623.26 total 1356.01 last 65.77) - open/close 40599 (time 1787796633.26 total 1366.01 last 59.89) - open/close 41181 (time 1787796643.26 total 1376.01 last 58.18) - open/close 41763 (time 1787796653.27 total 1386.02 last 58.17) - open/close 42376 (time 1787796663.29 total 1396.04 last 61.18) - open/close 42979 (time 1787796673.30 total 1406.05 last 60.22) - open/close 43500 (time 1787796683.30 total 1416.05 last 52.09) - open/close 44080 (time 1787796693.31 total 1426.06 last 57.96) - open/close 44693 (time 1787796703.32 total 1436.07 last 61.24) - open/close 45353 (time 1787796713.32 total 1446.07 last 66.00) - open/close 45997 (time 1787796723.32 total 1456.08 last 64.37) - open/close 46652 (time 1787796733.33 total 1466.08 last 65.47) - open/close 47294 (time 1787796743.34 total 1476.09 last 64.13) - open/close 47932 (time 1787796753.35 total 1486.10 last 63.74) - open/close 48560 (time 1787796763.36 total 1496.11 last 62.76) - open/close 49193 (time 1787796773.36 total 1506.11 last 63.28) - open/close 49799 (time 1787796783.38 total 1516.13 last 60.51) total: 50000 open/close in 1519.20 seconds: 32.91 ops/second - unlinked 0 (time 1787796791 ; total 0 ; last 0) - unlinked 10000 (time 1787796962 ; total 171 ; last 171) - unlinked 20000 (time 1787797120 ; total 329 ; last 158) - unlinked 30000 (time 1787797283 ; total 492 ; last 163) - unlinked 40000 (time 1787797454 ; total 663 ; last 171) total: 50000 unlinks in 823 seconds: 60.753342 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg313-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg313-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg313-server stop mds service on oleg313-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg313-server unloading modules via unload_modules_local on: 'oleg313-server' oleg313-server: oleg313-server.virtnet: executing unload_modules_local oleg313-server: kmemleak disabled modules unloaded. pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-client: ssh exited with exit code 2 pdsh@oleg313-client: oleg313-server: ssh exited with exit code 2