== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:31:56 (1789605116) start mds service on oleg109-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 16 12:13 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing load_modules_local oleg109-server: kmemleak disabled oleg109-server: Loading modules from /home/green/git/lustre-release/lustre oleg109-server: detected 4 online CPUs by sysfs oleg109-server: MODOPTS_LIBCFS= oleg109-server: Force libcfs to create 2 CPU partitions oleg109-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg109-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg109-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg109-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg109-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg109-server: oleg109-server.virtnet: executing set_default_debug -1 all pdsh@oleg109-client: oleg109-server: ssh exited with exit code 1 Started lustre-OST0000 oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg109-client.virtnet: -o user_xattr,flock 192.168.201.109@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 142 (time 1789605205.33 total 10.80 last 13.14) - open/close 299 (time 1789605216.28 total 21.75 last 14.34) - open/close 367 (time 1789605226.29 total 31.76 last 6.79) - open/close 527 (time 1789605237.29 total 42.76 last 14.55) - open/close 670 (time 1789605248.35 total 53.82 last 12.93) - open/close 841 (time 1789605258.99 total 64.46 last 16.07) - open/close 1018 (time 1789605270.69 total 76.16 last 15.12) - open/close 1159 (time 1789605280.71 total 86.18 last 14.07) - open/close 1336 (time 1789605290.74 total 96.20 last 17.66) - open/close 1482 (time 1789605301.60 total 107.07 last 13.44) - open/close 1596 (time 1789605312.57 total 118.04 last 10.39) - open/close 1769 (time 1789605323.42 total 128.89 last 15.94) - open/close 1869 (time 1789605334.06 total 139.53 last 9.40) - open/close 2022 (time 1789605344.42 total 149.89 last 14.78) - open/close 2193 (time 1789605354.43 total 159.90 last 17.08) - open/close 2445 (time 1789605365.23 total 170.70 last 23.34) - open/close 2665 (time 1789605375.72 total 181.19 last 20.96) - open/close 2882 (time 1789605386.08 total 191.55 last 20.96) - open/close 3036 (time 1789605397.29 total 202.76 last 13.74) - open/close 3209 (time 1789605407.30 total 212.77 last 17.29) - open/close 3370 (time 1789605417.84 total 223.31 last 15.27) - open/close 3515 (time 1789605428.87 total 234.34 last 13.15) - open/close 3719 (time 1789605439.66 total 245.12 last 18.91) - open/close 3878 (time 1789605449.66 total 255.13 last 15.89) - open/close 4062 (time 1789605460.52 total 265.99 last 16.94) - open/close 4157 (time 1789605470.56 total 276.03 last 9.46) - open/close 4253 (time 1789605480.67 total 286.14 last 9.50) - open/close 4359 (time 1789605491.44 total 296.91 last 9.84) - open/close 4477 (time 1789605502.46 total 307.93 last 10.71) - open/close 4616 (time 1789605512.57 total 318.04 last 13.75) - open/close 4708 (time 1789605528.29 total 333.75 last 5.85) - open/close 4867 (time 1789605538.40 total 343.86 last 15.73) - open/close 5087 (time 1789605554.47 total 359.94 last 13.69) - open/close 5206 (time 1789605564.48 total 369.95 last 11.88) - open/close 5399 (time 1789605575.89 total 381.36 last 16.92) - open/close 5505 (time 1789605588.84 total 394.31 last 8.18) - open/close 5577 (time 1789605598.85 total 404.32 last 7.19) - open/close 5750 (time 1789605608.87 total 414.34 last 17.27) - open/close 5986 (time 1789605619.88 total 425.35 last 21.44) - open/close 6193 (time 1789605632.49 total 437.96 last 16.42) - open/close 6393 (time 1789605642.73 total 448.20 last 19.53) - open/close 6447 (time 1789605654.04 total 459.51 last 4.77) - open/close 6548 (time 1789605664.42 total 469.89 last 9.73) - open/close 6716 (time 1789605674.43 total 479.90 last 16.79) - open/close 6788 (time 1789605684.46 total 489.93 last 7.18) - open/close 7011 (time 1789605695.40 total 500.87 last 20.39) - open/close 7264 (time 1789605705.42 total 510.89 last 25.26) - open/close 7531 (time 1789605715.81 total 521.28 last 25.70) - open/close 7679 (time 1789605726.05 total 531.52 last 14.44) - open/close 7923 (time 1789605736.06 total 541.53 last 24.38) - open/close 8119 (time 1789605746.66 total 552.13 last 18.49) - open/close 8224 (time 1789605757.41 total 562.88 last 9.77) - open/close 8388 (time 1789605768.35 total 573.82 last 15.00) - open/close 8507 (time 1789605779.87 total 585.34 last 10.33) - open/close 8685 (time 1789605790.25 total 595.72 last 17.15) - open/close 8931 (time 1789605800.27 total 605.74 last 24.56) - open/close 9208 (time 1789605810.74 total 616.21 last 26.46) - open/close 9361 (time 1789605821.29 total 626.76 last 14.50) - open/close 9605 (time 1789605834.11 total 639.58 last 19.04) - open/close 9815 (time 1789605844.12 total 649.58 last 20.98) - open/close 9987 (time 1789605856.81 total 662.28 last 13.55) - open/close 10211 (time 1789605866.82 total 672.29 last 22.38) - open/close 10357 (time 1789605877.14 total 682.61 last 14.15) - open/close 10493 (time 1789605887.18 total 692.65 last 13.54) - open/close 10659 (time 1789605897.20 total 702.67 last 16.57) - open/close 10949 (time 1789605907.77 total 713.23 last 27.45) - open/close 11122 (time 1789605918.30 total 723.77 last 16.42) - open/close 11301 (time 1789605928.30 total 733.77 last 17.89) - open/close 11419 (time 1789605939.49 total 744.96 last 10.55) - open/close 11554 (time 1789605949.63 total 755.10 last 13.32) - open/close 11778 (time 1789605959.63 total 765.10 last 22.39) - open/close 12027 (time 1789605969.64 total 775.11 last 24.88) - open/close 12172 (time 1789605979.64 total 785.11 last 14.50) - open/close 12321 (time 1789605989.73 total 795.20 last 14.77) - open/close 12606 (time 1789605999.73 total 805.20 last 28.49) - open/close 12746 (time 1789606010.46 total 815.93 last 13.05) - open/close 12882 (time 1789606021.54 total 827.00 last 12.28) - open/close 13003 (time 1789606031.88 total 837.35 last 11.69) - open/close 13118 (time 1789606041.92 total 847.39 last 11.46) - open/close 13230 (time 1789606052.32 total 857.79 last 10.77) - open/close 13408 (time 1789606062.33 total 867.80 last 17.78) - open/close 13542 (time 1789606073.50 total 878.97 last 12.00) - open/close 13637 (time 1789606083.50 total 888.97 last 9.50) - open/close 13745 (time 1789606093.54 total 899.01 last 10.76) - open/close 13851 (time 1789606103.54 total 909.01 last 10.60) - open/close 14046 (time 1789606114.79 total 920.26 last 17.33) - open/close 14207 (time 1789606126.67 total 932.14 last 13.55) - open/close 14382 (time 1789606136.68 total 942.15 last 17.49) - open/close 14540 (time 1789606147.10 total 952.57 last 15.15) - open/close 14633 (time 1789606157.68 total 963.15 last 8.79) - open/close 14880 (time 1789606168.17 total 973.64 last 23.53) - open/close 15070 (time 1789606178.18 total 983.64 last 19.00) - open/close 15160 (time 1789606190.95 total 996.41 last 7.05) - open/close 15330 (time 1789606200.96 total 1006.43 last 16.97) - open/close 15471 (time 1789606211.00 total 1016.47 last 14.05) - open/close 15540 (time 1789606223.28 total 1028.75 last 5.62) - open/close 15781 (time 1789606234.08 total 1039.55 last 22.32) - open/close 15887 (time 1789606244.26 total 1049.73 last 10.41) - open/close 16089 (time 1789606254.27 total 1059.74 last 20.18) - open/close 16807 (time 1789606264.29 total 1069.76 last 71.71) - open/close 17576 (time 1789606274.29 total 1079.76 last 76.89) - open/close 18386 (time 1789606284.30 total 1089.77 last 80.90) - open/close 19116 (time 1789606294.32 total 1099.79 last 72.87) - open/close 19737 (time 1789606304.33 total 1109.80 last 61.99) - open/close 20000 (time 1789606307.99 total 1113.46 last 71.89) - open/close 20740 (time 1789606318.00 total 1123.47 last 73.94) - open/close 21440 (time 1789606328.01 total 1133.48 last 69.95) - open/close 22225 (time 1789606338.02 total 1143.49 last 78.41) - open/close 22922 (time 1789606348.03 total 1153.50 last 69.60) - open/close 23534 (time 1789606358.04 total 1163.51 last 61.18) - open/close 24139 (time 1789606368.04 total 1173.51 last 60.48) - open/close 24817 (time 1789606378.05 total 1183.52 last 67.72) - open/close 25617 (time 1789606388.06 total 1193.53 last 79.96) - open/close 26462 (time 1789606398.06 total 1203.53 last 84.47) - open/close 27262 (time 1789606408.07 total 1213.54 last 79.94) - open/close 27968 (time 1789606418.08 total 1223.55 last 70.54) - open/close 28764 (time 1789606428.09 total 1233.56 last 79.54) - open/close 29429 (time 1789606438.10 total 1243.56 last 66.44) - open/close 30000 (time 1789606445.27 total 1250.74 last 79.55) - open/close 30699 (time 1789606455.28 total 1260.75 last 69.87) - open/close 31411 (time 1789606465.28 total 1270.75 last 71.18) - open/close 32186 (time 1789606475.28 total 1280.75 last 77.47) - open/close 32941 (time 1789606485.30 total 1290.77 last 75.41) - open/close 33653 (time 1789606495.30 total 1300.77 last 71.18) - open/close 34460 (time 1789606505.30 total 1310.77 last 80.69) - open/close 35174 (time 1789606515.31 total 1320.78 last 71.33) - open/close 35714 (time 1789606525.34 total 1330.81 last 53.85) - open/close 36372 (time 1789606535.34 total 1340.81 last 65.78) - open/close 37144 (time 1789606545.36 total 1350.83 last 77.06) - open/close 37768 (time 1789606555.37 total 1360.84 last 62.32) - open/close 38394 (time 1789606565.41 total 1370.88 last 62.39) - open/close 38959 (time 1789606575.41 total 1380.88 last 56.50) - open/close 39651 (time 1789606585.42 total 1390.89 last 69.13) - open/close 40000 (time 1789606590.36 total 1395.83 last 70.66) - open/close 40750 (time 1789606600.36 total 1405.83 last 74.96) - open/close 41428 (time 1789606610.36 total 1415.83 last 67.78) - open/close 42042 (time 1789606620.37 total 1425.84 last 61.39) - open/close 42743 (time 1789606630.38 total 1435.85 last 70.02) - open/close 43410 (time 1789606640.39 total 1445.85 last 66.65) - open/close 44124 (time 1789606650.39 total 1455.85 last 71.40) - open/close 44837 (time 1789606660.40 total 1465.87 last 71.21) - open/close 45493 (time 1789606670.41 total 1475.88 last 65.52) - open/close 46162 (time 1789606680.41 total 1485.88 last 66.89) - open/close 46967 (time 1789606690.42 total 1495.89 last 80.42) - open/close 47673 (time 1789606700.42 total 1505.89 last 70.58) - open/close 48437 (time 1789606710.45 total 1515.92 last 76.23) - open/close 48993 (time 1789606720.45 total 1525.92 last 55.58) - open/close 49572 (time 1789606730.46 total 1535.93 last 57.86) total: 50000 open/close in 1542.30 seconds: 32.42 ops/second - unlinked 0 (time 1789606741 ; total 0 ; last 0) - unlinked 10000 (time 1789606896 ; total 155 ; last 155) - unlinked 20000 (time 1789607060 ; total 319 ; last 164) - unlinked 30000 (time 1789607218 ; total 477 ; last 158) - unlinked 40000 (time 1789607377 ; total 636 ; last 159) total: 50000 unlinks in 792 seconds: 63.131313 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg109-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg109-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg109-server stop mds service on oleg109-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg109-server unloading modules via unload_modules_local on: 'oleg109-server' oleg109-server: oleg109-server.virtnet: executing unload_modules_local oleg109-server: kmemleak disabled modules unloaded. pdsh@oleg109-client: oleg109-client: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-client: ssh exited with exit code 2 pdsh@oleg109-client: oleg109-server: ssh exited with exit code 2