== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:10:17 (1788739817) start mds service on oleg460-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921104 Sep 6 19:17 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing load_modules_local oleg460-server: kmemleak disabled oleg460-server: Loading modules from /home/green/git/lustre-release/lustre oleg460-server: detected 4 online CPUs by sysfs oleg460-server: MODOPTS_LIBCFS= oleg460-server: Force libcfs to create 2 CPU partitions oleg460-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg460-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg460-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg460-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg460-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 248 (time 1788739901.42 total 10.74 last 23.10) - open/close 510 (time 1788739911.78 total 21.09 last 25.29) - open/close 752 (time 1788739922.61 total 31.93 last 22.34) - open/close 1005 (time 1788739932.61 total 41.93 last 25.29) - open/close 1260 (time 1788739943.21 total 52.53 last 24.05) - open/close 1482 (time 1788739953.22 total 62.54 last 22.18) - open/close 1690 (time 1788739963.82 total 73.14 last 19.62) - open/close 1954 (time 1788739973.84 total 83.16 last 26.36) - open/close 2091 (time 1788739984.03 total 93.35 last 13.44) - open/close 2263 (time 1788739994.05 total 103.37 last 17.16) - open/close 2489 (time 1788740004.06 total 113.37 last 22.60) - open/close 2722 (time 1788740014.82 total 124.14 last 21.64) - open/close 2873 (time 1788740024.83 total 134.15 last 15.10) - open/close 3051 (time 1788740035.57 total 144.89 last 16.57) - open/close 3138 (time 1788740045.77 total 155.08 last 8.53) - open/close 3326 (time 1788740056.19 total 165.51 last 18.04) - open/close 3538 (time 1788740066.87 total 176.19 last 19.85) - open/close 3727 (time 1788740077.02 total 186.34 last 18.61) - open/close 3890 (time 1788740087.08 total 196.40 last 16.21) - open/close 4120 (time 1788740097.08 total 206.40 last 23.00) - open/close 4365 (time 1788740107.11 total 216.43 last 24.43) - open/close 4588 (time 1788740117.17 total 226.49 last 22.17) - open/close 4819 (time 1788740127.53 total 236.85 last 22.30) - open/close 5104 (time 1788740137.53 total 246.85 last 28.49) - open/close 5355 (time 1788740148.33 total 257.65 last 23.23) - open/close 5572 (time 1788740159.39 total 268.71 last 19.62) - open/close 5802 (time 1788740169.70 total 279.02 last 22.31) - open/close 5933 (time 1788740179.73 total 289.04 last 13.07) - open/close 6135 (time 1788740191.29 total 300.61 last 17.47) - open/close 6338 (time 1788740202.30 total 311.62 last 18.43) - open/close 6487 (time 1788740212.39 total 321.71 last 14.77) - open/close 6656 (time 1788740222.39 total 331.71 last 16.89) - open/close 6806 (time 1788740232.40 total 341.72 last 14.99) - open/close 7018 (time 1788740242.42 total 351.74 last 21.16) - open/close 7235 (time 1788740253.29 total 362.61 last 19.97) - open/close 7458 (time 1788740263.86 total 373.18 last 21.09) - open/close 7667 (time 1788740273.87 total 383.19 last 20.88) - open/close 7885 (time 1788740283.87 total 393.19 last 21.80) - open/close 8072 (time 1788740293.98 total 403.30 last 18.49) - open/close 8256 (time 1788740303.99 total 413.30 last 18.40) - open/close 8496 (time 1788740314.98 total 424.29 last 21.84) - open/close 8655 (time 1788740325.21 total 434.53 last 15.53) - open/close 8843 (time 1788740335.48 total 444.80 last 18.31) - open/close 9051 (time 1788740345.96 total 455.28 last 19.85) - open/close 9242 (time 1788740355.97 total 465.29 last 19.08) - open/close 9447 (time 1788740365.98 total 475.30 last 20.48) - open/close 9628 (time 1788740376.56 total 485.87 last 17.11) - open/close 9830 (time 1788740386.57 total 495.89 last 20.17) - open/close 10000 (time 1788740393.49 total 502.81 last 24.58) - open/close 10217 (time 1788740403.50 total 512.82 last 21.68) - open/close 10427 (time 1788740413.50 total 522.82 last 21.00) - open/close 10626 (time 1788740423.51 total 532.83 last 19.89) - open/close 10801 (time 1788740434.23 total 543.55 last 16.32) - open/close 11062 (time 1788740444.24 total 553.56 last 26.08) - open/close 11271 (time 1788740456.05 total 565.37 last 17.69) - open/close 11491 (time 1788740466.48 total 575.80 last 21.09) - open/close 11751 (time 1788740476.70 total 586.02 last 25.45) - open/close 12000 (time 1788740486.71 total 596.03 last 24.88) - open/close 12131 (time 1788740497.71 total 607.03 last 11.91) - open/close 12354 (time 1788740508.21 total 617.53 last 21.23) - open/close 12596 (time 1788740518.22 total 627.54 last 24.19) - open/close 12735 (time 1788740528.35 total 637.67 last 13.71) - open/close 12935 (time 1788740539.38 total 648.70 last 18.14) - open/close 13114 (time 1788740549.60 total 658.92 last 17.51) - open/close 13344 (time 1788740559.63 total 668.95 last 22.95) - open/close 13518 (time 1788740569.64 total 678.96 last 17.38) - open/close 13784 (time 1788740580.14 total 689.46 last 25.33) - open/close 14062 (time 1788740590.83 total 700.15 last 26.01) - open/close 14292 (time 1788740600.94 total 710.25 last 22.76) - open/close 14519 (time 1788740611.91 total 721.23 last 20.69) - open/close 14737 (time 1788740621.92 total 731.24 last 21.76) - open/close 14953 (time 1788740632.17 total 741.49 last 21.08) - open/close 15149 (time 1788740642.60 total 751.91 last 18.80) - open/close 15352 (time 1788740652.62 total 761.94 last 20.25) - open/close 15588 (time 1788740662.76 total 772.08 last 23.29) - open/close 15819 (time 1788740673.45 total 782.77 last 21.60) - open/close 16041 (time 1788740683.94 total 793.25 last 21.18) - open/close 16247 (time 1788740694.83 total 804.15 last 18.91) - open/close 16436 (time 1788740704.85 total 814.17 last 18.86) - open/close 16663 (time 1788740715.71 total 825.03 last 20.91) - open/close 16906 (time 1788740725.71 total 835.03 last 24.30) - open/close 17136 (time 1788740735.72 total 845.04 last 22.97) - open/close 17357 (time 1788740745.73 total 855.04 last 22.09) - open/close 17588 (time 1788740755.73 total 865.05 last 23.10) - open/close 17812 (time 1788740765.73 total 875.05 last 22.39) - open/close 18547 (time 1788740775.74 total 885.06 last 73.42) - open/close 19225 (time 1788740785.75 total 895.07 last 67.77) - open/close 19907 (time 1788740795.76 total 905.08 last 68.08) - open/close 20578 (time 1788740805.77 total 915.09 last 67.07) - open/close 21308 (time 1788740815.79 total 925.11 last 72.81) - open/close 22027 (time 1788740825.83 total 935.15 last 71.67) - open/close 22691 (time 1788740835.83 total 945.15 last 66.39) - open/close 23295 (time 1788740845.84 total 955.16 last 60.32) - open/close 23930 (time 1788740855.85 total 965.17 last 63.42) - open/close 24509 (time 1788740865.86 total 975.18 last 57.85) - open/close 25194 (time 1788740875.87 total 985.19 last 68.45) - open/close 25808 (time 1788740885.89 total 995.21 last 61.27) - open/close 26513 (time 1788740895.89 total 1005.21 last 70.49) - open/close 27227 (time 1788740905.90 total 1015.22 last 71.37) - open/close 27935 (time 1788740915.92 total 1025.24 last 70.64) - open/close 28583 (time 1788740925.93 total 1035.25 last 64.73) - open/close 29298 (time 1788740935.93 total 1045.25 last 71.48) - open/close 29987 (time 1788740945.95 total 1055.27 last 68.78) - open/close 30630 (time 1788740955.96 total 1065.28 last 64.23) - open/close 31109 (time 1788740965.98 total 1075.30 last 47.82) - open/close 31520 (time 1788740976.00 total 1085.32 last 41.02) - open/close 31978 (time 1788740986.00 total 1095.32 last 45.80) - open/close 32631 (time 1788740996.00 total 1105.32 last 65.27) - open/close 33351 (time 1788741006.00 total 1115.32 last 71.99) - open/close 33918 (time 1788741016.01 total 1125.33 last 56.66) - open/close 34514 (time 1788741026.01 total 1135.33 last 59.59) - open/close 34970 (time 1788741036.04 total 1145.36 last 45.46) - open/close 35366 (time 1788741046.06 total 1155.38 last 39.53) - open/close 35914 (time 1788741056.08 total 1165.40 last 54.67) - open/close 36466 (time 1788741066.09 total 1175.41 last 55.16) - open/close 36939 (time 1788741076.09 total 1185.41 last 47.30) - open/close 37304 (time 1788741086.10 total 1195.42 last 36.45) - open/close 37701 (time 1788741096.13 total 1205.45 last 39.60) - open/close 38126 (time 1788741106.14 total 1215.46 last 42.45) - open/close 38633 (time 1788741116.15 total 1225.47 last 50.64) - open/close 39185 (time 1788741126.16 total 1235.48 last 55.18) - open/close 39764 (time 1788741136.18 total 1245.50 last 57.79) - open/close 40000 (time 1788741139.78 total 1249.10 last 65.40) - open/close 40630 (time 1788741149.78 total 1259.10 last 63.00) - open/close 41184 (time 1788741159.78 total 1269.10 last 55.40) - open/close 41764 (time 1788741169.79 total 1279.11 last 57.96) - open/close 42336 (time 1788741179.80 total 1289.12 last 57.15) - open/close 42958 (time 1788741189.80 total 1299.12 last 62.20) - open/close 43580 (time 1788741199.81 total 1309.13 last 62.15) - open/close 44221 (time 1788741209.81 total 1319.13 last 64.07) - open/close 44896 (time 1788741219.81 total 1329.13 last 67.49) - open/close 45564 (time 1788741229.83 total 1339.14 last 66.72) - open/close 46208 (time 1788741239.84 total 1349.16 last 64.32) - open/close 46908 (time 1788741249.85 total 1359.17 last 69.89) - open/close 47472 (time 1788741259.88 total 1369.20 last 56.27) - open/close 48095 (time 1788741269.89 total 1379.21 last 62.24) - open/close 48684 (time 1788741279.89 total 1389.21 last 58.90) - open/close 49299 (time 1788741289.90 total 1399.22 last 61.42) - open/close 49903 (time 1788741299.90 total 1409.22 last 60.37) total: 50000 open/close in 1410.90 seconds: 35.44 ops/second - unlinked 0 (time 1788741307 ; total 0 ; last 0) - unlinked 10000 (time 1788741486 ; total 179 ; last 179) - unlinked 20000 (time 1788741671 ; total 364 ; last 185) - unlinked 30000 (time 1788741835 ; total 528 ; last 164) - unlinked 40000 (time 1788741993 ; total 686 ; last 158) total: 50000 unlinks in 842 seconds: 59.382423 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server unloading modules via unload_modules_local on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing unload_modules_local oleg460-server: kmemleak disabled modules unloaded. pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2