== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:06:59 (1788689219) start mds service on oleg110-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 5 18:57 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg110-server' oleg110-server: oleg110-server.virtnet: executing load_modules_local oleg110-server: kmemleak disabled oleg110-server: Loading modules from /home/green/git/lustre-release/lustre oleg110-server: detected 4 online CPUs by sysfs oleg110-server: MODOPTS_LIBCFS= oleg110-server: Force libcfs to create 2 CPU partitions oleg110-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg110-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg110-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg110-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg110-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg110-client.virtnet: -o user_xattr,flock 192.168.201.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 160 (time 1788689307.86 total 10.08 last 15.87) - open/close 373 (time 1788689318.24 total 20.46 last 20.52) - open/close 439 (time 1788689328.25 total 30.48 last 6.59) - open/close 636 (time 1788689338.27 total 40.49 last 19.68) - open/close 825 (time 1788689348.45 total 50.68 last 18.55) - open/close 956 (time 1788689358.80 total 61.02 last 12.66) - open/close 1175 (time 1788689369.60 total 71.82 last 20.28) - open/close 1412 (time 1788689379.60 total 81.83 last 23.68) - open/close 1616 (time 1788689389.62 total 91.84 last 20.37) - open/close 1790 (time 1788689399.62 total 101.85 last 17.39) - open/close 1880 (time 1788689410.73 total 112.96 last 8.10) - open/close 2047 (time 1788689421.60 total 123.82 last 15.37) - open/close 2218 (time 1788689432.58 total 134.81 last 15.57) - open/close 2405 (time 1788689443.33 total 145.55 last 17.40) - open/close 2614 (time 1788689453.33 total 155.56 last 20.89) - open/close 2769 (time 1788689463.35 total 165.58 last 15.47) - open/close 2960 (time 1788689473.36 total 175.59 last 19.08) - open/close 3094 (time 1788689483.37 total 185.59 last 13.39) - open/close 3316 (time 1788689494.11 total 196.33 last 20.67) - open/close 3433 (time 1788689504.27 total 206.49 last 11.52) - open/close 3637 (time 1788689514.90 total 217.12 last 19.19) - open/close 3831 (time 1788689524.94 total 227.16 last 19.32) - open/close 4005 (time 1788689534.95 total 237.18 last 17.37) - open/close 4229 (time 1788689544.97 total 247.20 last 22.37) - open/close 4445 (time 1788689554.99 total 257.22 last 21.55) - open/close 4603 (time 1788689566.95 total 269.18 last 13.21) - open/close 4730 (time 1788689577.04 total 279.27 last 12.59) - open/close 4905 (time 1788689587.04 total 289.27 last 17.50) - open/close 5066 (time 1788689597.81 total 300.03 last 14.96) - open/close 5198 (time 1788689607.81 total 310.04 last 13.19) - open/close 5419 (time 1788689618.52 total 320.74 last 20.64) - open/close 5554 (time 1788689628.95 total 331.18 last 12.94) - open/close 5696 (time 1788689639.55 total 341.77 last 13.40) - open/close 5947 (time 1788689650.27 total 352.50 last 23.40) - open/close 6177 (time 1788689661.08 total 363.31 last 21.28) - open/close 6376 (time 1788689672.15 total 374.37 last 17.98) - open/close 6527 (time 1788689682.27 total 384.50 last 14.92) - open/close 6665 (time 1788689692.32 total 394.55 last 13.73) - open/close 6867 (time 1788689702.32 total 404.55 last 20.20) - open/close 7075 (time 1788689714.20 total 416.43 last 17.52) - open/close 7193 (time 1788689724.22 total 426.45 last 11.77) - open/close 7460 (time 1788689735.25 total 437.47 last 24.22) - open/close 7601 (time 1788689745.41 total 447.64 last 13.87) - open/close 7773 (time 1788689756.25 total 458.47 last 15.87) - open/close 8009 (time 1788689766.73 total 468.96 last 22.51) - open/close 8176 (time 1788689776.76 total 478.99 last 16.65) - open/close 8321 (time 1788689786.77 total 488.99 last 14.49) - open/close 8630 (time 1788689796.79 total 499.02 last 30.83) - open/close 8782 (time 1788689807.11 total 509.34 last 14.73) - open/close 9020 (time 1788689817.13 total 519.35 last 23.77) - open/close 9274 (time 1788689827.14 total 529.37 last 25.37) - open/close 9396 (time 1788689837.28 total 539.51 last 12.03) - open/close 9634 (time 1788689847.64 total 549.87 last 22.98) - open/close 9767 (time 1788689857.65 total 559.88 last 13.29) - open/close 9897 (time 1788689868.37 total 570.60 last 12.12) - open/close 9994 (time 1788689878.55 total 580.77 last 9.53) - open/close 10274 (time 1788689888.55 total 590.78 last 27.99) - open/close 10353 (time 1788689899.01 total 601.24 last 7.55) - open/close 10468 (time 1788689909.20 total 611.43 last 11.29) - open/close 10668 (time 1788689919.20 total 621.43 last 19.99) - open/close 10903 (time 1788689929.59 total 631.81 last 22.63) - open/close 11100 (time 1788689939.99 total 642.21 last 18.94) - open/close 11271 (time 1788689950.02 total 652.25 last 17.04) - open/close 11384 (time 1788689960.04 total 662.26 last 11.28) - open/close 11571 (time 1788689970.60 total 672.83 last 17.71) - open/close 11690 (time 1788689980.83 total 683.05 last 11.64) - open/close 11875 (time 1788689991.50 total 693.72 last 17.33) - open/close 12011 (time 1788690002.31 total 704.54 last 12.57) - open/close 12147 (time 1788690012.64 total 714.87 last 13.17) - open/close 12422 (time 1788690023.63 total 725.86 last 25.03) - open/close 12635 (time 1788690034.38 total 736.60 last 19.82) - open/close 12750 (time 1788690046.48 total 748.70 last 9.51) - open/close 12888 (time 1788690056.98 total 759.21 last 13.14) - open/close 13118 (time 1788690066.99 total 769.21 last 22.99) - open/close 13273 (time 1788690077.83 total 780.06 last 14.29) - open/close 13420 (time 1788690088.47 total 790.70 last 13.82) - open/close 13542 (time 1788690098.50 total 800.73 last 12.17) - open/close 13673 (time 1788690109.28 total 811.51 last 12.15) - open/close 13819 (time 1788690119.52 total 821.75 last 14.25) - open/close 14003 (time 1788690129.54 total 831.77 last 18.36) - open/close 14180 (time 1788690139.79 total 842.01 last 17.27) - open/close 14221 (time 1788690150.04 total 852.27 last 4.00) - open/close 14339 (time 1788690162.24 total 864.47 last 9.67) - open/close 14460 (time 1788690172.26 total 874.49 last 12.08) - open/close 14570 (time 1788690183.71 total 885.94 last 9.61) - open/close 14733 (time 1788690193.74 total 895.96 last 16.26) - open/close 14969 (time 1788690204.63 total 906.86 last 21.66) - open/close 15214 (time 1788690214.64 total 916.87 last 24.48) - open/close 15344 (time 1788690224.65 total 926.88 last 12.99) - open/close 15414 (time 1788690235.42 total 937.65 last 6.50) - open/close 15625 (time 1788690245.44 total 947.67 last 21.06) - open/close 15741 (time 1788690255.47 total 957.69 last 11.57) - open/close 15795 (time 1788690268.05 total 970.27 last 4.29) - open/close 16010 (time 1788690278.07 total 980.30 last 21.44) - open/close 16098 (time 1788690290.47 total 992.69 last 7.10) - open/close 16290 (time 1788690300.50 total 1002.72 last 19.14) - open/close 16501 (time 1788690310.80 total 1013.02 last 20.48) - open/close 16611 (time 1788690322.84 total 1025.07 last 9.13) - open/close 16682 (time 1788690333.77 total 1036.00 last 6.50) - open/close 16940 (time 1788690344.35 total 1046.57 last 24.40) - open/close 17537 (time 1788690354.35 total 1056.58 last 59.69) - open/close 18205 (time 1788690364.37 total 1066.59 last 66.69) - open/close 18846 (time 1788690374.37 total 1076.59 last 64.10) - open/close 19531 (time 1788690384.39 total 1086.61 last 68.36) - open/close 20000 (time 1788690391.61 total 1093.84 last 64.90) - open/close 20590 (time 1788690401.62 total 1103.85 last 58.95) - open/close 21160 (time 1788690411.67 total 1113.90 last 56.71) - open/close 21738 (time 1788690421.68 total 1123.91 last 57.74) - open/close 22369 (time 1788690431.69 total 1133.92 last 63.03) - open/close 23033 (time 1788690441.70 total 1143.93 last 66.34) - open/close 23690 (time 1788690451.72 total 1153.94 last 65.61) - open/close 24362 (time 1788690461.72 total 1163.95 last 67.16) - open/close 25036 (time 1788690471.73 total 1173.96 last 67.35) - open/close 25777 (time 1788690481.74 total 1183.97 last 74.03) - open/close 26480 (time 1788690491.76 total 1193.99 last 70.12) - open/close 27157 (time 1788690501.77 total 1203.99 last 67.69) - open/close 27904 (time 1788690511.77 total 1214.00 last 74.68) - open/close 28584 (time 1788690521.78 total 1224.00 last 67.94) - open/close 29392 (time 1788690531.79 total 1234.02 last 80.67) - open/close 29985 (time 1788690541.80 total 1244.03 last 59.24) - open/close 30664 (time 1788690551.81 total 1254.04 last 67.84) - open/close 31347 (time 1788690561.82 total 1264.05 last 68.22) - open/close 32117 (time 1788690571.84 total 1274.06 last 76.90) - open/close 32790 (time 1788690581.84 total 1284.06 last 67.28) - open/close 33437 (time 1788690591.85 total 1294.07 last 64.65) - open/close 34151 (time 1788690601.85 total 1304.07 last 71.39) - open/close 34916 (time 1788690611.85 total 1314.07 last 76.49) - open/close 35708 (time 1788690621.85 total 1324.08 last 79.18) - open/close 36470 (time 1788690631.85 total 1334.08 last 76.18) - open/close 37146 (time 1788690641.87 total 1344.09 last 67.50) - open/close 37750 (time 1788690651.87 total 1354.10 last 60.36) - open/close 38389 (time 1788690661.89 total 1364.12 last 63.81) - open/close 39017 (time 1788690671.89 total 1374.12 last 62.79) - open/close 39676 (time 1788690681.89 total 1384.12 last 65.88) - open/close 40000 (time 1788690686.69 total 1388.91 last 67.59) - open/close 40761 (time 1788690696.69 total 1398.92 last 76.05) - open/close 41584 (time 1788690706.71 total 1408.94 last 82.17) - open/close 42288 (time 1788690716.72 total 1418.95 last 70.32) - open/close 42952 (time 1788690726.74 total 1428.96 last 66.29) - open/close 43744 (time 1788690736.74 total 1438.96 last 79.19) - open/close 44395 (time 1788690746.74 total 1448.97 last 65.07) - open/close 45070 (time 1788690756.75 total 1458.97 last 67.48) - open/close 45733 (time 1788690766.75 total 1468.98 last 66.27) - open/close 46491 (time 1788690776.77 total 1479.00 last 75.65) - open/close 47170 (time 1788690786.77 total 1489.00 last 67.87) - open/close 47792 (time 1788690796.78 total 1499.01 last 62.17) - open/close 48480 (time 1788690806.79 total 1509.02 last 68.72) - open/close 49101 (time 1788690816.80 total 1519.02 last 62.07) - open/close 49755 (time 1788690826.81 total 1529.03 last 65.32) total: 50000 open/close in 1533.53 seconds: 32.60 ops/second - unlinked 0 (time 1788690836 ; total 0 ; last 0) - unlinked 10000 (time 1788690993 ; total 157 ; last 157) - unlinked 20000 (time 1788691151 ; total 315 ; last 158) - unlinked 30000 (time 1788691308 ; total 472 ; last 157) - unlinked 40000 (time 1788691469 ; total 633 ; last 161) total: 50000 unlinks in 788 seconds: 63.451778 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg110-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg110-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server stop mds service on oleg110-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg110-server unloading modules via unload_modules_local on: 'oleg110-server' oleg110-server: oleg110-server.virtnet: executing unload_modules_local oleg110-server: kmemleak disabled modules unloaded. pdsh@oleg110-client: oleg110-client: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-client: ssh exited with exit code 2