== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:57:07 (1788537427) start mds service on oleg207-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1923184 Sep 3 20:45 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: kmemleak disabled oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: detected 4 online CPUs by sysfs oleg207-server: MODOPTS_LIBCFS= oleg207-server: Force libcfs to create 2 CPU partitions oleg207-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg207-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg207-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg207-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 259 (time 1788537508.71 total 10.51 last 24.64) - open/close 353 (time 1788537519.46 total 21.27 last 8.74) - open/close 464 (time 1788537529.75 total 31.55 last 10.79) - open/close 600 (time 1788537540.06 total 41.86 last 13.19) - open/close 749 (time 1788537550.36 total 52.17 last 14.46) - open/close 924 (time 1788537561.04 total 62.84 last 16.39) - open/close 1133 (time 1788537571.04 total 72.84 last 20.90) - open/close 1388 (time 1788537581.84 total 83.64 last 23.61) - open/close 1614 (time 1788537592.35 total 94.15 last 21.51) - open/close 1834 (time 1788537602.63 total 104.43 last 21.39) - open/close 2152 (time 1788537613.46 total 115.26 last 29.38) - open/close 2393 (time 1788537623.47 total 125.27 last 24.06) - open/close 2525 (time 1788537633.50 total 135.30 last 13.16) - open/close 2764 (time 1788537643.52 total 145.32 last 23.86) - open/close 2920 (time 1788537653.54 total 155.34 last 15.57) - open/close 3129 (time 1788537663.54 total 165.35 last 20.88) - open/close 3320 (time 1788537673.56 total 175.37 last 19.06) - open/close 3445 (time 1788537684.48 total 186.28 last 11.45) - open/close 3639 (time 1788537694.49 total 196.29 last 19.39) - open/close 3779 (time 1788537705.43 total 207.23 last 12.79) - open/close 4002 (time 1788537715.43 total 217.24 last 22.30) - open/close 4180 (time 1788537725.44 total 227.24 last 17.79) - open/close 4370 (time 1788537736.33 total 238.14 last 17.44) - open/close 4607 (time 1788537747.04 total 248.84 last 22.13) - open/close 4792 (time 1788537757.05 total 258.85 last 18.49) - open/close 5043 (time 1788537767.51 total 269.31 last 23.99) - open/close 5245 (time 1788537778.01 total 279.81 last 19.24) - open/close 5469 (time 1788537788.02 total 289.82 last 22.38) - open/close 5571 (time 1788537798.02 total 299.83 last 10.19) - open/close 5775 (time 1788537808.85 total 310.65 last 18.85) - open/close 6037 (time 1788537819.84 total 321.64 last 23.83) - open/close 6266 (time 1788537829.85 total 331.65 last 22.88) - open/close 6517 (time 1788537841.81 total 343.61 last 20.99) - open/close 6651 (time 1788537852.62 total 354.43 last 12.39) - open/close 6816 (time 1788537863.43 total 365.24 last 15.27) - open/close 7053 (time 1788537873.67 total 375.48 last 23.14) - open/close 7222 (time 1788537883.69 total 385.49 last 16.87) - open/close 7303 (time 1788537894.34 total 396.14 last 7.61) - open/close 7548 (time 1788537904.36 total 406.16 last 24.45) - open/close 7680 (time 1788537914.39 total 416.19 last 13.16) - open/close 7928 (time 1788537924.94 total 426.75 last 23.50) - open/close 8118 (time 1788537934.94 total 436.75 last 19.00) - open/close 8360 (time 1788537945.49 total 447.29 last 22.95) - open/close 8591 (time 1788537956.36 total 458.16 last 21.25) - open/close 8767 (time 1788537966.37 total 468.17 last 17.59) - open/close 8966 (time 1788537976.38 total 478.18 last 19.88) - open/close 9222 (time 1788537986.39 total 488.19 last 25.56) - open/close 9422 (time 1788537998.80 total 500.60 last 16.12) - open/close 9690 (time 1788538008.80 total 510.61 last 26.78) - open/close 9879 (time 1788538018.82 total 520.62 last 18.88) - open/close 10000 (time 1788538023.24 total 525.05 last 27.33) - open/close 10248 (time 1788538034.02 total 535.82 last 23.01) - open/close 10387 (time 1788538044.04 total 545.84 last 13.88) - open/close 10572 (time 1788538054.04 total 555.85 last 18.48) - open/close 10831 (time 1788538064.58 total 566.38 last 24.58) - open/close 11086 (time 1788538074.60 total 576.41 last 25.45) - open/close 11307 (time 1788538084.94 total 586.74 last 21.38) - open/close 11489 (time 1788538095.64 total 597.44 last 17.01) - open/close 11672 (time 1788538105.65 total 607.45 last 18.28) - open/close 11881 (time 1788538116.65 total 618.45 last 19.00) - open/close 12034 (time 1788538127.76 total 629.56 last 13.78) - open/close 12252 (time 1788538138.33 total 640.14 last 20.61) - open/close 12456 (time 1788538148.36 total 650.16 last 20.34) - open/close 12679 (time 1788538158.68 total 660.48 last 21.61) - open/close 12865 (time 1788538170.78 total 672.59 last 15.36) - open/close 13089 (time 1788538180.79 total 682.60 last 22.38) - open/close 13387 (time 1788538192.87 total 694.67 last 24.68) - open/close 13668 (time 1788538203.65 total 705.46 last 26.05) - open/close 13868 (time 1788538213.97 total 715.77 last 19.40) - open/close 14175 (time 1788538224.98 total 726.78 last 27.88) - open/close 14310 (time 1788538235.92 total 737.73 last 12.33) - open/close 14507 (time 1788538246.66 total 748.46 last 18.35) - open/close 14676 (time 1788538257.22 total 759.03 last 16.00) - open/close 14843 (time 1788538267.91 total 769.72 last 15.62) - open/close 15099 (time 1788538277.92 total 779.73 last 25.57) - open/close 15343 (time 1788538287.93 total 789.73 last 24.39) - open/close 15611 (time 1788538297.94 total 799.74 last 26.78) - open/close 15824 (time 1788538307.94 total 809.74 last 21.30) - open/close 16031 (time 1788538318.93 total 820.73 last 18.84) - open/close 16283 (time 1788538329.65 total 831.45 last 23.51) - open/close 16481 (time 1788538339.67 total 841.47 last 19.75) - open/close 16735 (time 1788538350.76 total 852.56 last 22.91) - open/close 16969 (time 1788538360.78 total 862.58 last 23.35) - open/close 17193 (time 1788538370.79 total 872.59 last 22.37) - open/close 17453 (time 1788538381.72 total 883.53 last 23.78) - open/close 17640 (time 1788538391.73 total 893.54 last 18.68) - open/close 17911 (time 1788538401.87 total 903.67 last 26.73) - open/close 18115 (time 1788538412.74 total 914.54 last 18.77) - open/close 18244 (time 1788538423.46 total 925.26 last 12.03) - open/close 18397 (time 1788538433.48 total 935.28 last 15.27) - open/close 18610 (time 1788538444.26 total 946.06 last 19.76) - open/close 18759 (time 1788538454.26 total 956.06 last 14.90) - open/close 18989 (time 1788538464.27 total 966.07 last 22.97) - open/close 19230 (time 1788538474.27 total 976.08 last 24.09) - open/close 20000 (time 1788538482.56 total 984.36 last 92.98) - open/close 20984 (time 1788538492.56 total 994.36 last 98.35) - open/close 21757 (time 1788538502.56 total 1004.36 last 77.28) - open/close 22689 (time 1788538512.57 total 1014.37 last 93.14) - open/close 23549 (time 1788538522.57 total 1024.37 last 86.00) - open/close 24464 (time 1788538532.58 total 1034.38 last 91.38) - open/close 25338 (time 1788538542.58 total 1044.39 last 87.37) - open/close 26196 (time 1788538552.59 total 1054.39 last 85.77) - open/close 27050 (time 1788538562.60 total 1064.40 last 85.32) - open/close 27872 (time 1788538572.61 total 1074.41 last 82.13) - open/close 28748 (time 1788538582.62 total 1084.42 last 87.53) - open/close 29598 (time 1788538592.62 total 1094.43 last 84.93) - open/close 30000 (time 1788538598.23 total 1100.03 last 71.76) - open/close 30576 (time 1788538608.23 total 1110.03 last 57.58) - open/close 31289 (time 1788538618.24 total 1120.04 last 71.22) - open/close 32087 (time 1788538628.24 total 1130.05 last 79.76) - open/close 32944 (time 1788538638.25 total 1140.05 last 85.66) - open/close 33847 (time 1788538648.25 total 1150.06 last 90.26) - open/close 34721 (time 1788538658.26 total 1160.06 last 87.35) - open/close 35531 (time 1788538668.27 total 1170.07 last 80.93) - open/close 36202 (time 1788538678.31 total 1180.11 last 66.84) - open/close 36855 (time 1788538688.32 total 1190.12 last 65.21) - open/close 37722 (time 1788538698.33 total 1200.13 last 86.65) - open/close 38587 (time 1788538708.33 total 1210.14 last 86.43) - open/close 39432 (time 1788538718.34 total 1220.14 last 84.49) - open/close 40000 (time 1788538724.91 total 1226.72 last 86.34) - open/close 40837 (time 1788538734.92 total 1236.72 last 83.68) - open/close 41637 (time 1788538744.92 total 1246.72 last 79.98) - open/close 42383 (time 1788538754.92 total 1256.73 last 74.55) - open/close 43116 (time 1788538764.93 total 1266.73 last 73.30) - open/close 43893 (time 1788538774.94 total 1276.74 last 77.61) - open/close 44898 (time 1788538784.94 total 1286.75 last 100.43) - open/close 45776 (time 1788538794.96 total 1296.76 last 87.68) - open/close 46699 (time 1788538804.97 total 1306.77 last 92.20) - open/close 47553 (time 1788538814.97 total 1316.78 last 85.35) - open/close 48389 (time 1788538824.97 total 1326.78 last 83.60) - open/close 49260 (time 1788538834.98 total 1336.78 last 87.07) - open/close 49962 (time 1788538844.98 total 1346.78 last 70.17) total: 50000 open/close in 1347.53 seconds: 37.10 ops/second - unlinked 0 (time 1788538850 ; total 0 ; last 0) - unlinked 10000 (time 1788538983 ; total 133 ; last 133) - unlinked 20000 (time 1788539108 ; total 258 ; last 125) - unlinked 30000 (time 1788539236 ; total 386 ; last 128) - unlinked 40000 (time 1788539367 ; total 517 ; last 131) total: 50000 unlinks in 649 seconds: 77.041603 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server unloading modules via unload_modules_local on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local oleg207-server: kmemleak disabled modules unloaded. pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2