== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:54:41 (1789610081) start mds service on oleg227-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931640 Sep 16 13:21 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing load_modules_local oleg227-server: kmemleak disabled oleg227-server: Loading modules from /home/green/git/lustre-release/lustre oleg227-server: detected 4 online CPUs by sysfs oleg227-server: MODOPTS_LIBCFS= oleg227-server: Force libcfs to create 2 CPU partitions oleg227-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg227-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg227-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg227-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg227-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg227-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 258 (time 1789610186.78 total 10.02 last 25.74) - open/close 468 (time 1789610197.13 total 20.37 last 20.29) - open/close 759 (time 1789610207.69 total 30.93 last 27.56) - open/close 1029 (time 1789610217.69 total 40.93 last 27.00) - open/close 1161 (time 1789610227.85 total 51.10 last 12.98) - open/close 1322 (time 1789610237.86 total 61.10 last 16.09) - open/close 1527 (time 1789610247.86 total 71.11 last 20.50) - open/close 1785 (time 1789610258.26 total 81.51 last 24.80) - open/close 2021 (time 1789610269.22 total 92.46 last 21.55) - open/close 2258 (time 1789610279.24 total 102.48 last 23.65) - open/close 2432 (time 1789610289.72 total 112.96 last 16.61) - open/close 2658 (time 1789610299.73 total 122.98 last 22.56) - open/close 2888 (time 1789610310.59 total 133.84 last 21.17) - open/close 3124 (time 1789610321.35 total 144.60 last 21.94) - open/close 3386 (time 1789610331.53 total 154.78 last 25.73) - open/close 3655 (time 1789610342.35 total 165.59 last 24.88) - open/close 3894 (time 1789610353.11 total 176.36 last 22.20) - open/close 4104 (time 1789610364.34 total 187.59 last 18.71) - open/close 4319 (time 1789610374.35 total 197.60 last 21.47) - open/close 4601 (time 1789610384.39 total 207.63 last 28.11) - open/close 4761 (time 1789610394.40 total 217.64 last 15.98) - open/close 4988 (time 1789610404.76 total 228.01 last 21.90) - open/close 5109 (time 1789610414.78 total 238.02 last 12.08) - open/close 5301 (time 1789610425.17 total 248.41 last 18.48) - open/close 5486 (time 1789610435.95 total 259.19 last 17.16) - open/close 5675 (time 1789610445.95 total 269.20 last 18.89) - open/close 5900 (time 1789610456.87 total 280.12 last 20.60) - open/close 6031 (time 1789610467.86 total 291.11 last 11.92) - open/close 6247 (time 1789610477.88 total 301.13 last 21.56) - open/close 6421 (time 1789610487.89 total 311.13 last 17.39) - open/close 6683 (time 1789610498.90 total 322.15 last 23.79) - open/close 6930 (time 1789610512.04 total 335.28 last 18.80) - open/close 7201 (time 1789610522.72 total 345.96 last 25.38) - open/close 7403 (time 1789610532.95 total 356.20 last 19.74) - open/close 7630 (time 1789610543.26 total 366.50 last 22.03) - open/close 7875 (time 1789610553.26 total 376.51 last 24.49) - open/close 8119 (time 1789610563.27 total 386.52 last 24.38) - open/close 8380 (time 1789610573.28 total 396.52 last 26.09) - open/close 8506 (time 1789610583.39 total 406.64 last 12.46) - open/close 8667 (time 1789610594.13 total 417.38 last 14.99) - open/close 8868 (time 1789610604.18 total 427.43 last 19.99) - open/close 9056 (time 1789610614.82 total 438.06 last 17.68) - open/close 9211 (time 1789610626.21 total 449.46 last 13.60) - open/close 9373 (time 1789610636.22 total 459.47 last 16.18) - open/close 9576 (time 1789610646.36 total 469.60 last 20.03) - open/close 9773 (time 1789610656.37 total 479.61 last 19.68) - open/close 9873 (time 1789610668.10 total 491.35 last 8.53) - open/close 10000 (time 1789610674.06 total 497.30 last 21.32) - open/close 10210 (time 1789610684.30 total 507.55 last 20.49) - open/close 10367 (time 1789610695.96 total 519.21 last 13.47) - open/close 10605 (time 1789610705.97 total 529.22 last 23.77) - open/close 10698 (time 1789610716.25 total 539.49 last 9.05) - open/close 10831 (time 1789610727.19 total 550.43 last 12.16) - open/close 11084 (time 1789610737.19 total 560.44 last 25.29) - open/close 11321 (time 1789610747.20 total 570.45 last 23.67) - open/close 11550 (time 1789610757.51 total 580.76 last 22.21) - open/close 11762 (time 1789610767.53 total 590.77 last 21.17) - open/close 12004 (time 1789610777.62 total 600.86 last 23.99) - open/close 12231 (time 1789610788.26 total 611.51 last 21.32) - open/close 12460 (time 1789610799.07 total 622.32 last 21.18) - open/close 12655 (time 1789610809.25 total 632.50 last 19.16) - open/close 12828 (time 1789610822.09 total 645.34 last 13.48) - open/close 12984 (time 1789610832.61 total 655.86 last 14.82) - open/close 13103 (time 1789610843.54 total 666.79 last 10.89) - open/close 13293 (time 1789610854.04 total 677.29 last 18.10) - open/close 13443 (time 1789610864.86 total 688.11 last 13.86) - open/close 13622 (time 1789610874.88 total 698.12 last 17.88) - open/close 13728 (time 1789610885.51 total 708.76 last 9.97) - open/close 13905 (time 1789610895.51 total 718.76 last 17.70) - open/close 14156 (time 1789610905.52 total 728.77 last 25.07) - open/close 14267 (time 1789610916.18 total 739.42 last 10.42) - open/close 14419 (time 1789610927.16 total 750.40 last 13.85) - open/close 14624 (time 1789610938.71 total 761.96 last 17.74) - open/close 14873 (time 1789610949.16 total 772.40 last 23.83) - open/close 15110 (time 1789610960.21 total 783.46 last 21.44) - open/close 15351 (time 1789610970.23 total 793.47 last 24.06) - open/close 15572 (time 1789610980.24 total 803.49 last 22.07) - open/close 15711 (time 1789610990.68 total 813.92 last 13.32) - open/close 15963 (time 1789611000.72 total 823.96 last 25.10) - open/close 16053 (time 1789611013.33 total 836.57 last 7.14) - open/close 16262 (time 1789611023.33 total 846.57 last 20.90) - open/close 16396 (time 1789611033.34 total 856.59 last 13.38) - open/close 16504 (time 1789611043.48 total 866.73 last 10.65) - open/close 16568 (time 1789611054.41 total 877.66 last 5.85) - open/close 16825 (time 1789611065.38 total 888.62 last 23.44) - open/close 16982 (time 1789611075.94 total 899.19 last 14.86) - open/close 17169 (time 1789611085.95 total 909.20 last 18.69) - open/close 17307 (time 1789611096.99 total 920.23 last 12.50) - open/close 17507 (time 1789611106.99 total 930.24 last 19.99) - open/close 17654 (time 1789611117.15 total 940.39 last 14.47) - open/close 17799 (time 1789611127.75 total 951.00 last 13.67) - open/close 17941 (time 1789611137.77 total 961.01 last 14.18) - open/close 18085 (time 1789611147.79 total 971.04 last 14.37) - open/close 18179 (time 1789611158.69 total 981.93 last 8.63) - open/close 18307 (time 1789611168.69 total 991.93 last 12.80) - open/close 18906 (time 1789611178.72 total 1001.97 last 59.70) - open/close 19563 (time 1789611188.73 total 1011.98 last 65.64) - open/close 20000 (time 1789611195.55 total 1018.80 last 64.04) - open/close 20728 (time 1789611205.57 total 1028.82 last 72.64) - open/close 21451 (time 1789611215.58 total 1038.82 last 72.30) - open/close 22274 (time 1789611225.59 total 1048.84 last 82.18) - open/close 22916 (time 1789611235.59 total 1058.84 last 64.18) - open/close 23620 (time 1789611245.60 total 1068.85 last 70.33) - open/close 24273 (time 1789611255.61 total 1078.85 last 65.26) - open/close 24998 (time 1789611265.61 total 1088.86 last 72.49) - open/close 25794 (time 1789611275.63 total 1098.88 last 79.44) - open/close 26490 (time 1789611285.64 total 1108.88 last 69.57) - open/close 27139 (time 1789611295.64 total 1118.88 last 64.89) - open/close 27947 (time 1789611305.65 total 1128.89 last 80.73) - open/close 28669 (time 1789611315.65 total 1138.89 last 72.20) - open/close 29414 (time 1789611325.65 total 1148.89 last 74.49) - open/close 30000 (time 1789611333.82 total 1157.07 last 71.69) - open/close 30663 (time 1789611343.84 total 1167.09 last 66.16) - open/close 31412 (time 1789611353.85 total 1177.09 last 74.86) - open/close 32057 (time 1789611363.86 total 1187.10 last 64.43) - open/close 32766 (time 1789611373.86 total 1197.11 last 70.87) - open/close 33503 (time 1789611383.87 total 1207.11 last 73.66) - open/close 34218 (time 1789611393.88 total 1217.13 last 71.36) - open/close 34799 (time 1789611403.88 total 1227.13 last 58.10) - open/close 35384 (time 1789611413.89 total 1237.13 last 58.50) - open/close 36039 (time 1789611423.90 total 1247.14 last 65.42) - open/close 36738 (time 1789611433.90 total 1257.15 last 69.87) - open/close 37375 (time 1789611443.90 total 1267.15 last 63.70) - open/close 38117 (time 1789611453.91 total 1277.15 last 74.16) - open/close 38835 (time 1789611463.92 total 1287.16 last 71.73) - open/close 39583 (time 1789611473.92 total 1297.17 last 74.78) - open/close 40000 (time 1789611479.75 total 1303.00 last 71.47) - open/close 40705 (time 1789611489.76 total 1313.01 last 70.43) - open/close 41400 (time 1789611499.77 total 1323.01 last 69.47) - open/close 42166 (time 1789611509.77 total 1333.02 last 76.55) - open/close 42774 (time 1789611519.82 total 1343.07 last 60.51) - open/close 43357 (time 1789611529.83 total 1353.08 last 58.24) - open/close 44055 (time 1789611539.83 total 1363.08 last 69.80) - open/close 44735 (time 1789611549.84 total 1373.09 last 67.95) - open/close 45329 (time 1789611559.85 total 1383.09 last 59.36) - open/close 45870 (time 1789611569.86 total 1393.11 last 54.02) - open/close 46513 (time 1789611579.86 total 1403.11 last 64.29) - open/close 47281 (time 1789611589.87 total 1413.12 last 76.74) - open/close 47989 (time 1789611599.88 total 1423.13 last 70.74) - open/close 48705 (time 1789611609.91 total 1433.16 last 71.38) - open/close 49406 (time 1789611619.92 total 1443.16 last 70.06) total: 50000 open/close in 1451.21 seconds: 34.45 ops/second - unlinked 0 (time 1789611632 ; total 0 ; last 0) - unlinked 10000 (time 1789611798 ; total 166 ; last 166) - unlinked 20000 (time 1789611942 ; total 310 ; last 144) - unlinked 30000 (time 1789612092 ; total 460 ; last 150) - unlinked 40000 (time 1789612237 ; total 605 ; last 145) total: 50000 unlinks in 757 seconds: 66.050201 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server unloading modules via unload_modules_local on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing unload_modules_local oleg227-server: kmemleak disabled modules unloaded. pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2