== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:58:15 (1790729895) start mds service on oleg213-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1946856 Sep 29 10:18 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: kmemleak disabled oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions oleg213-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg213-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg213-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg213-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg213-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg213-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 248 (time 1790729999.84 total 10.79 last 22.98) - open/close 467 (time 1790730010.22 total 21.17 last 21.10) - open/close 720 (time 1790730020.58 total 31.53 last 24.42) - open/close 970 (time 1790730031.00 total 41.95 last 23.99) - open/close 1232 (time 1790730041.52 total 52.47 last 24.92) - open/close 1486 (time 1790730051.53 total 62.49 last 25.35) - open/close 1673 (time 1790730061.56 total 72.52 last 18.65) - open/close 1884 (time 1790730071.57 total 82.53 last 21.08) - open/close 2128 (time 1790730081.82 total 92.78 last 23.80) - open/close 2435 (time 1790730092.91 total 103.86 last 27.70) - open/close 2631 (time 1790730103.75 total 114.70 last 18.08) - open/close 2839 (time 1790730113.75 total 124.70 last 20.80) - open/close 3072 (time 1790730124.39 total 135.34 last 21.90) - open/close 3304 (time 1790730134.40 total 145.35 last 23.18) - open/close 3556 (time 1790730145.12 total 156.07 last 23.50) - open/close 3764 (time 1790730156.12 total 167.07 last 18.92) - open/close 4012 (time 1790730166.13 total 177.08 last 24.77) - open/close 4239 (time 1790730176.79 total 187.74 last 21.29) - open/close 4493 (time 1790730186.79 total 197.74 last 25.40) - open/close 4746 (time 1790730196.97 total 207.92 last 24.86) - open/close 4974 (time 1790730207.46 total 218.41 last 21.73) - open/close 5185 (time 1790730217.57 total 228.53 last 20.86) - open/close 5391 (time 1790730228.08 total 239.03 last 19.62) - open/close 5674 (time 1790730238.49 total 249.44 last 27.18) - open/close 5911 (time 1790730249.50 total 260.45 last 21.52) - open/close 6159 (time 1790730260.07 total 271.02 last 23.47) - open/close 6423 (time 1790730270.09 total 281.05 last 26.34) - open/close 6684 (time 1790730280.10 total 291.05 last 26.09) - open/close 6858 (time 1790730290.10 total 301.05 last 17.39) - open/close 7043 (time 1790730302.24 total 313.19 last 15.24) - open/close 7257 (time 1790730312.26 total 323.21 last 21.36) - open/close 7458 (time 1790730324.33 total 335.29 last 16.65) - open/close 7636 (time 1790730335.41 total 346.36 last 16.07) - open/close 7786 (time 1790730345.83 total 356.78 last 14.40) - open/close 7932 (time 1790730355.94 total 366.89 last 14.44) - open/close 8117 (time 1790730366.62 total 377.57 last 17.32) - open/close 8272 (time 1790730377.18 total 388.14 last 14.67) - open/close 8383 (time 1790730387.86 total 398.81 last 10.40) - open/close 8613 (time 1790730397.92 total 408.88 last 22.86) - open/close 8823 (time 1790730408.49 total 419.44 last 19.88) - open/close 8994 (time 1790730419.95 total 430.90 last 14.92) - open/close 9143 (time 1790730430.63 total 441.58 last 13.95) - open/close 9267 (time 1790730440.63 total 451.58 last 12.40) - open/close 9386 (time 1790730450.64 total 461.60 last 11.88) - open/close 9436 (time 1790730463.26 total 474.21 last 3.96) - open/close 9559 (time 1790730473.27 total 484.22 last 12.29) - open/close 9682 (time 1790730483.28 total 494.23 last 12.29) - open/close 9863 (time 1790730493.29 total 504.24 last 18.08) - open/close 10000 (time 1790730499.95 total 510.90 last 20.58) - open/close 10244 (time 1790730510.18 total 521.13 last 23.85) - open/close 10453 (time 1790730520.18 total 531.13 last 20.90) - open/close 10603 (time 1790730530.65 total 541.60 last 14.33) - open/close 10767 (time 1790730540.66 total 551.61 last 16.39) - open/close 11012 (time 1790730551.48 total 562.43 last 22.63) - open/close 11210 (time 1790730561.67 total 572.63 last 19.43) - open/close 11446 (time 1790730571.95 total 582.91 last 22.95) - open/close 11568 (time 1790730582.99 total 593.95 last 11.05) - open/close 11731 (time 1790730593.64 total 604.59 last 15.31) - open/close 11782 (time 1790730604.31 total 615.27 last 4.78) - open/close 11993 (time 1790730615.24 total 626.19 last 19.31) - open/close 12111 (time 1790730625.52 total 636.48 last 11.48) - open/close 12263 (time 1790730635.94 total 646.89 last 14.60) - open/close 12416 (time 1790730648.63 total 659.58 last 12.05) - open/close 12521 (time 1790730659.51 total 670.46 last 9.65) - open/close 12623 (time 1790730669.51 total 680.46 last 10.20) - open/close 12825 (time 1790730679.78 total 690.73 last 19.67) - open/close 13060 (time 1790730689.79 total 700.75 last 23.46) - open/close 13170 (time 1790730700.33 total 711.29 last 10.44) - open/close 13294 (time 1790730710.36 total 721.31 last 12.37) - open/close 13541 (time 1790730720.42 total 731.37 last 24.55) - open/close 13775 (time 1790730730.86 total 741.81 last 22.41) - open/close 13871 (time 1790730741.86 total 752.82 last 8.72) - open/close 14024 (time 1790730751.90 total 762.85 last 15.25) - open/close 14172 (time 1790730762.65 total 773.61 last 13.76) - open/close 14336 (time 1790730772.78 total 783.74 last 16.19) - open/close 14438 (time 1790730782.95 total 793.91 last 10.03) - open/close 14702 (time 1790730793.62 total 804.57 last 24.76) - open/close 14905 (time 1790730803.80 total 814.75 last 19.94) - open/close 15141 (time 1790730813.81 total 824.76 last 23.58) - open/close 15362 (time 1790730826.40 total 837.36 last 17.54) - open/close 15536 (time 1790730837.22 total 848.17 last 16.09) - open/close 15739 (time 1790730847.22 total 858.17 last 20.29) - open/close 15915 (time 1790730857.23 total 868.18 last 17.58) - open/close 16155 (time 1790730867.23 total 878.19 last 24.00) - open/close 16382 (time 1790730877.53 total 888.49 last 22.04) - open/close 16645 (time 1790730887.54 total 898.49 last 26.28) - open/close 16812 (time 1790730897.94 total 908.90 last 16.06) - open/close 17058 (time 1790730908.51 total 919.46 last 23.28) - open/close 17325 (time 1790730919.53 total 930.48 last 24.24) - open/close 17462 (time 1790730929.56 total 940.51 last 13.66) - open/close 17649 (time 1790730939.69 total 950.64 last 18.46) - open/close 17778 (time 1790730949.70 total 960.65 last 12.89) - open/close 17902 (time 1790730959.70 total 970.65 last 12.40) - open/close 18136 (time 1790730970.01 total 980.96 last 22.70) - open/close 18550 (time 1790730980.02 total 990.97 last 41.35) - open/close 19403 (time 1790730990.02 total 1000.98 last 85.26) - open/close 20000 (time 1790730996.51 total 1007.47 last 92.00) - open/close 20916 (time 1790731006.52 total 1017.47 last 91.55) - open/close 21809 (time 1790731016.52 total 1027.48 last 89.26) - open/close 22676 (time 1790731026.53 total 1037.49 last 86.63) - open/close 23520 (time 1790731036.53 total 1047.49 last 84.39) - open/close 24522 (time 1790731046.55 total 1057.50 last 100.08) - open/close 25424 (time 1790731056.56 total 1067.51 last 90.10) - open/close 26117 (time 1790731066.56 total 1077.51 last 69.30) - open/close 26929 (time 1790731076.56 total 1087.52 last 81.15) - open/close 27768 (time 1790731086.56 total 1097.52 last 83.90) - open/close 28504 (time 1790731096.56 total 1107.52 last 73.59) - open/close 29261 (time 1790731106.57 total 1117.52 last 75.68) - open/close 30000 (time 1790731115.16 total 1126.11 last 86.03) - open/close 30823 (time 1790731125.16 total 1136.11 last 82.25) - open/close 31654 (time 1790731135.17 total 1146.13 last 82.99) - open/close 32573 (time 1790731145.18 total 1156.13 last 91.89) - open/close 33506 (time 1790731155.18 total 1166.13 last 93.27) - open/close 34300 (time 1790731165.18 total 1176.14 last 79.37) - open/close 35104 (time 1790731175.19 total 1186.14 last 80.34) - open/close 35942 (time 1790731185.20 total 1196.15 last 83.75) - open/close 36815 (time 1790731195.20 total 1206.15 last 87.26) - open/close 37700 (time 1790731205.21 total 1216.16 last 88.44) - open/close 38520 (time 1790731215.21 total 1226.17 last 81.95) - open/close 39466 (time 1790731225.23 total 1236.18 last 94.46) - open/close 40000 (time 1790731231.55 total 1242.50 last 84.51) - open/close 40905 (time 1790731241.56 total 1252.51 last 90.40) - open/close 41733 (time 1790731251.56 total 1262.51 last 82.78) - open/close 42621 (time 1790731261.56 total 1272.52 last 88.79) - open/close 43368 (time 1790731271.57 total 1282.52 last 74.67) - open/close 44199 (time 1790731281.59 total 1292.54 last 82.92) - open/close 44805 (time 1790731291.61 total 1302.56 last 60.49) - open/close 45491 (time 1790731301.63 total 1312.58 last 68.45) - open/close 46253 (time 1790731311.64 total 1322.60 last 76.08) - open/close 47000 (time 1790731321.65 total 1332.60 last 74.67) - open/close 47857 (time 1790731331.65 total 1342.60 last 85.68) - open/close 48722 (time 1790731341.66 total 1352.62 last 86.38) - open/close 49421 (time 1790731351.67 total 1362.62 last 69.89) total: 50000 open/close in 1369.78 seconds: 36.50 ops/second - unlinked 0 (time 1790731363 ; total 0 ; last 0) - unlinked 10000 (time 1790731491 ; total 128 ; last 128) - unlinked 20000 (time 1790731619 ; total 256 ; last 128) - unlinked 30000 (time 1790731759 ; total 396 ; last 140) - unlinked 40000 (time 1790731881 ; total 518 ; last 122) total: 50000 unlinks in 637 seconds: 78.492935 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg213-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg213-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg213-server stop mds service on oleg213-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg213-server stop mds service on oleg213-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg213-server unloading modules via unload_modules_local on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing unload_modules_local oleg213-server: kmemleak disabled modules unloaded. pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2