== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:31:39 (1788841899) start mds service on oleg432-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 7 17:37 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: kmemleak disabled oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions oleg432-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg432-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg432-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg432-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg432-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-OST0000 oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 148 (time 1788841996.27 total 10.89 last 13.59) - open/close 264 (time 1788842006.29 total 20.90 last 11.58) - open/close 442 (time 1788842016.31 total 30.92 last 17.76) - open/close 612 (time 1788842026.32 total 40.94 last 16.98) - open/close 823 (time 1788842037.31 total 51.93 last 19.19) - open/close 1040 (time 1788842048.83 total 63.45 last 18.84) - open/close 1219 (time 1788842058.84 total 73.45 last 17.89) - open/close 1454 (time 1788842068.85 total 83.47 last 23.46) - open/close 1648 (time 1788842078.86 total 93.48 last 19.38) - open/close 1830 (time 1788842089.64 total 104.26 last 16.88) - open/close 1955 (time 1788842099.91 total 114.53 last 12.17) - open/close 2139 (time 1788842110.73 total 125.34 last 17.01) - open/close 2258 (time 1788842120.74 total 135.36 last 11.89) - open/close 2438 (time 1788842130.75 total 145.36 last 17.99) - open/close 2597 (time 1788842142.64 total 157.26 last 13.37) - open/close 2809 (time 1788842152.64 total 167.26 last 21.20) - open/close 3045 (time 1788842162.65 total 177.27 last 23.57) - open/close 3266 (time 1788842173.49 total 188.11 last 20.39) - open/close 3401 (time 1788842183.92 total 198.53 last 12.95) - open/close 3661 (time 1788842193.93 total 208.55 last 25.97) - open/close 3886 (time 1788842204.53 total 219.15 last 21.21) - open/close 4085 (time 1788842214.55 total 229.17 last 19.86) - open/close 4248 (time 1788842224.56 total 239.17 last 16.29) - open/close 4388 (time 1788842235.32 total 249.94 last 13.01) - open/close 4504 (time 1788842245.64 total 260.26 last 11.24) - open/close 4632 (time 1788842255.65 total 270.27 last 12.79) - open/close 4815 (time 1788842266.01 total 280.63 last 17.66) - open/close 4994 (time 1788842276.03 total 290.65 last 17.86) - open/close 5180 (time 1788842286.06 total 300.67 last 18.56) - open/close 5409 (time 1788842297.07 total 311.69 last 20.78) - open/close 5680 (time 1788842307.08 total 321.69 last 27.10) - open/close 5831 (time 1788842317.48 total 332.10 last 14.51) - open/close 5989 (time 1788842327.50 total 342.11 last 15.78) - open/close 6067 (time 1788842338.78 total 353.39 last 6.92) - open/close 6195 (time 1788842348.78 total 363.40 last 12.79) - open/close 6341 (time 1788842359.34 total 373.96 last 13.82) - open/close 6526 (time 1788842370.29 total 384.91 last 16.89) - open/close 6793 (time 1788842381.16 total 395.78 last 24.57) - open/close 6947 (time 1788842393.80 total 408.41 last 12.19) - open/close 7146 (time 1788842403.80 total 418.41 last 19.90) - open/close 7344 (time 1788842413.80 total 428.42 last 19.80) - open/close 7589 (time 1788842424.83 total 439.45 last 22.21) - open/close 7847 (time 1788842434.84 total 449.46 last 25.76) - open/close 7993 (time 1788842445.18 total 459.79 last 14.13) - open/close 8163 (time 1788842455.99 total 470.60 last 15.73) - open/close 8391 (time 1788842465.99 total 480.60 last 22.80) - open/close 8582 (time 1788842476.01 total 490.63 last 19.05) - open/close 8749 (time 1788842486.72 total 501.34 last 15.60) - open/close 8902 (time 1788842496.73 total 511.34 last 15.29) - open/close 9127 (time 1788842507.70 total 522.31 last 20.51) - open/close 9264 (time 1788842517.71 total 532.32 last 13.69) - open/close 9499 (time 1788842527.71 total 542.33 last 23.49) - open/close 9544 (time 1788842538.29 total 552.91 last 4.25) - open/close 9613 (time 1788842549.37 total 563.98 last 6.23) - open/close 9796 (time 1788842559.54 total 574.16 last 17.99) - open/close 9908 (time 1788842569.54 total 584.16 last 11.20) - open/close 10000 (time 1788842573.38 total 588.00 last 23.97) - open/close 10201 (time 1788842583.38 total 598.00 last 20.09) - open/close 10423 (time 1788842593.86 total 608.48 last 21.18) - open/close 10553 (time 1788842604.23 total 618.85 last 12.54) - open/close 10686 (time 1788842614.91 total 629.53 last 12.45) - open/close 10802 (time 1788842625.91 total 640.53 last 10.54) - open/close 10961 (time 1788842636.67 total 651.29 last 14.78) - open/close 11107 (time 1788842647.35 total 661.96 last 13.68) - open/close 11326 (time 1788842658.11 total 672.72 last 20.35) - open/close 11548 (time 1788842668.12 total 682.74 last 22.17) - open/close 11744 (time 1788842678.13 total 692.75 last 19.58) - open/close 11861 (time 1788842689.79 total 704.41 last 10.04) - open/close 12006 (time 1788842700.14 total 714.76 last 14.01) - open/close 12100 (time 1788842710.21 total 724.82 last 9.34) - open/close 12231 (time 1788842722.89 total 737.51 last 10.32) - open/close 12311 (time 1788842733.11 total 747.72 last 7.83) - open/close 12419 (time 1788842743.30 total 757.92 last 10.60) - open/close 12645 (time 1788842754.16 total 768.78 last 20.80) - open/close 12870 (time 1788842764.23 total 778.85 last 22.35) - open/close 13116 (time 1788842774.23 total 788.85 last 24.60) - open/close 13343 (time 1788842784.25 total 798.87 last 22.66) - open/close 13556 (time 1788842794.37 total 808.99 last 21.05) - open/close 13705 (time 1788842806.59 total 821.21 last 12.19) - open/close 13948 (time 1788842817.46 total 832.08 last 22.36) - open/close 14188 (time 1788842828.02 total 842.63 last 22.73) - open/close 14413 (time 1788842838.51 total 853.12 last 21.45) - open/close 14545 (time 1788842848.65 total 863.27 last 13.01) - open/close 14781 (time 1788842858.67 total 873.28 last 23.57) - open/close 15015 (time 1788842869.05 total 883.67 last 22.53) - open/close 15245 (time 1788842879.07 total 893.68 last 22.97) - open/close 15417 (time 1788842889.09 total 903.71 last 17.16) - open/close 15557 (time 1788842899.10 total 913.72 last 13.98) - open/close 15732 (time 1788842909.73 total 924.35 last 16.46) - open/close 15798 (time 1788842920.26 total 934.88 last 6.27) - open/close 15869 (time 1788842930.35 total 944.96 last 7.04) - open/close 16065 (time 1788842940.35 total 954.97 last 19.59) - open/close 16233 (time 1788842950.78 total 965.40 last 16.10) - open/close 16459 (time 1788842961.02 total 975.63 last 22.09) - open/close 16697 (time 1788842972.12 total 986.74 last 21.43) - open/close 16935 (time 1788842982.13 total 996.75 last 23.77) - open/close 17164 (time 1788842992.15 total 1006.76 last 22.87) - open/close 17281 (time 1788843003.05 total 1017.67 last 10.73) - open/close 17603 (time 1788843013.07 total 1027.68 last 32.16) - open/close 18245 (time 1788843023.07 total 1037.68 last 64.20) - open/close 18837 (time 1788843033.08 total 1047.69 last 59.16) - open/close 19391 (time 1788843043.09 total 1057.70 last 55.33) - open/close 19996 (time 1788843053.11 total 1067.73 last 60.36) - open/close 20630 (time 1788843063.11 total 1077.73 last 63.38) - open/close 21282 (time 1788843073.13 total 1087.75 last 65.10) - open/close 21752 (time 1788843083.14 total 1097.76 last 46.95) - open/close 22229 (time 1788843093.15 total 1107.77 last 47.63) - open/close 22772 (time 1788843103.16 total 1117.77 last 54.29) - open/close 23382 (time 1788843113.17 total 1127.79 last 60.89) - open/close 23991 (time 1788843123.18 total 1137.79 last 60.88) - open/close 24614 (time 1788843133.19 total 1147.80 last 62.25) - open/close 25304 (time 1788843143.21 total 1157.82 last 68.84) - open/close 25955 (time 1788843153.21 total 1167.83 last 65.09) - open/close 26614 (time 1788843163.21 total 1177.83 last 65.89) - open/close 27225 (time 1788843173.22 total 1187.83 last 61.06) - open/close 27781 (time 1788843183.23 total 1197.84 last 55.54) - open/close 28339 (time 1788843193.23 total 1207.84 last 55.79) - open/close 28935 (time 1788843203.24 total 1217.86 last 59.53) - open/close 29528 (time 1788843213.25 total 1227.86 last 59.25) - open/close 30000 (time 1788843221.04 total 1235.65 last 60.59) - open/close 30442 (time 1788843231.04 total 1245.66 last 44.17) - open/close 31022 (time 1788843241.05 total 1255.66 last 57.98) - open/close 31670 (time 1788843251.07 total 1265.68 last 64.67) - open/close 32348 (time 1788843261.08 total 1275.69 last 67.73) - open/close 32953 (time 1788843271.09 total 1285.71 last 60.41) - open/close 33527 (time 1788843281.09 total 1295.71 last 57.39) - open/close 34031 (time 1788843291.10 total 1305.72 last 50.36) - open/close 34606 (time 1788843301.14 total 1315.76 last 57.28) - open/close 35181 (time 1788843311.14 total 1325.76 last 57.48) - open/close 35772 (time 1788843321.15 total 1335.77 last 59.04) - open/close 36324 (time 1788843331.16 total 1345.78 last 55.15) - open/close 36910 (time 1788843341.17 total 1355.78 last 58.58) - open/close 37727 (time 1788843351.17 total 1365.79 last 81.67) - open/close 38380 (time 1788843361.18 total 1375.80 last 65.23) - open/close 39051 (time 1788843371.19 total 1385.80 last 67.06) - open/close 39712 (time 1788843381.19 total 1395.81 last 66.09) - open/close 40000 (time 1788843386.03 total 1400.65 last 59.50) - open/close 40619 (time 1788843396.04 total 1410.66 last 61.84) - open/close 41243 (time 1788843406.04 total 1420.66 last 62.40) - open/close 41876 (time 1788843416.08 total 1430.69 last 63.07) - open/close 42503 (time 1788843426.08 total 1440.70 last 62.65) - open/close 43160 (time 1788843436.09 total 1450.71 last 65.63) - open/close 43855 (time 1788843446.11 total 1460.72 last 69.41) - open/close 44487 (time 1788843456.11 total 1470.73 last 63.16) - open/close 45124 (time 1788843466.13 total 1480.75 last 63.58) - open/close 45743 (time 1788843476.14 total 1490.76 last 61.85) - open/close 46389 (time 1788843486.16 total 1500.78 last 64.45) - open/close 46982 (time 1788843496.17 total 1510.79 last 59.24) - open/close 47639 (time 1788843506.19 total 1520.80 last 65.61) - open/close 48291 (time 1788843516.20 total 1530.82 last 65.12) - open/close 48979 (time 1788843526.21 total 1540.82 last 68.76) - open/close 49572 (time 1788843536.21 total 1550.83 last 59.25) total: 50000 open/close in 1557.31 seconds: 32.11 ops/second - unlinked 0 (time 1788843548 ; total 0 ; last 0) - unlinked 10000 (time 1788843726 ; total 178 ; last 178) - unlinked 20000 (time 1788843892 ; total 344 ; last 166) - unlinked 30000 (time 1788844069 ; total 521 ; last 177) - unlinked 40000 (time 1788844231 ; total 683 ; last 162) total: 50000 unlinks in 855 seconds: 58.479530 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg432-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server stop mds service on oleg432-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server unloading modules via unload_modules_local on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing unload_modules_local oleg432-server: kmemleak disabled modules unloaded. pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2