== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:05:42 (1790982342) start mds service on oleg454-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1946968 Oct 2 10:36 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg454-server' oleg454-server: oleg454-server.virtnet: executing load_modules_local oleg454-server: kmemleak disabled oleg454-server: Loading modules from /home/green/git/lustre-release/lustre oleg454-server: detected 4 online CPUs by sysfs oleg454-server: MODOPTS_LIBCFS= oleg454-server: Force libcfs to create 2 CPU partitions oleg454-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg454-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg454-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg454-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg454-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-OST0000 oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg454-client.virtnet: -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 163 (time 1790982423.57 total 10.01 last 16.28) - open/close 367 (time 1790982433.60 total 20.04 last 20.34) - open/close 657 (time 1790982443.60 total 30.05 last 28.99) - open/close 870 (time 1790982453.62 total 40.06 last 21.27) - open/close 1086 (time 1790982463.63 total 50.07 last 21.58) - open/close 1310 (time 1790982473.63 total 60.07 last 22.39) - open/close 1473 (time 1790982484.00 total 70.44 last 15.71) - open/close 1696 (time 1790982494.69 total 81.13 last 20.87) - open/close 1866 (time 1790982505.71 total 92.15 last 15.43) - open/close 2100 (time 1790982515.73 total 102.17 last 23.35) - open/close 2272 (time 1790982526.75 total 113.19 last 15.61) - open/close 2336 (time 1790982537.95 total 124.39 last 5.71) - open/close 2421 (time 1790982548.01 total 134.45 last 8.45) - open/close 2515 (time 1790982558.56 total 145.01 last 8.91) - open/close 2647 (time 1790982568.69 total 155.13 last 13.04) - open/close 2869 (time 1790982581.48 total 167.92 last 17.35) - open/close 3105 (time 1790982592.16 total 178.60 last 22.10) - open/close 3266 (time 1790982604.25 total 190.69 last 13.32) - open/close 3405 (time 1790982614.57 total 201.02 last 13.46) - open/close 3684 (time 1790982624.98 total 211.43 last 26.80) - open/close 3821 (time 1790982637.17 total 223.61 last 11.24) - open/close 4035 (time 1790982647.46 total 233.90 last 20.79) - open/close 4222 (time 1790982657.46 total 243.91 last 18.69) - open/close 4460 (time 1790982669.92 total 256.36 last 19.10) - open/close 4662 (time 1790982680.60 total 267.05 last 18.91) - open/close 4891 (time 1790982690.62 total 277.06 last 22.87) - open/close 5066 (time 1790982701.69 total 288.13 last 15.81) - open/close 5183 (time 1790982712.51 total 298.95 last 10.81) - open/close 5448 (time 1790982722.53 total 308.97 last 26.45) - open/close 5597 (time 1790982732.53 total 318.98 last 14.89) - open/close 5820 (time 1790982743.09 total 329.53 last 21.13) - open/close 6072 (time 1790982753.38 total 339.82 last 24.49) - open/close 6330 (time 1790982764.39 total 350.83 last 23.43) - open/close 6570 (time 1790982774.39 total 360.84 last 23.98) - open/close 6829 (time 1790982785.31 total 371.76 last 23.72) - open/close 7041 (time 1790982795.93 total 382.37 last 19.97) - open/close 7246 (time 1790982808.42 total 394.86 last 16.42) - open/close 7363 (time 1790982818.43 total 404.87 last 11.69) - open/close 7548 (time 1790982829.28 total 415.72 last 17.05) - open/close 7596 (time 1790982840.36 total 426.80 last 4.33) - open/close 7745 (time 1790982851.06 total 437.50 last 13.93) - open/close 7995 (time 1790982862.05 total 448.49 last 22.74) - open/close 8188 (time 1790982872.73 total 459.18 last 18.07) - open/close 8418 (time 1790982883.26 total 469.70 last 21.85) - open/close 8648 (time 1790982893.68 total 480.12 last 22.07) - open/close 8844 (time 1790982903.68 total 490.12 last 19.60) - open/close 9072 (time 1790982913.77 total 500.21 last 22.61) - open/close 9388 (time 1790982924.40 total 510.84 last 29.72) - open/close 9638 (time 1790982935.40 total 521.84 last 22.73) - open/close 9867 (time 1790982946.10 total 532.54 last 21.40) - open/close 10000 (time 1790982951.77 total 538.22 last 23.43) - open/close 10242 (time 1790982964.59 total 551.03 last 18.88) - open/close 10501 (time 1790982975.40 total 561.84 last 23.97) - open/close 10725 (time 1790982985.41 total 571.85 last 22.38) - open/close 10965 (time 1790982995.43 total 581.87 last 23.95) - open/close 11188 (time 1790983005.43 total 591.88 last 22.29) - open/close 11500 (time 1790983015.44 total 601.88 last 31.17) - open/close 11688 (time 1790983026.17 total 612.61 last 17.52) - open/close 11922 (time 1790983036.44 total 622.89 last 22.78) - open/close 12141 (time 1790983046.45 total 632.89 last 21.89) - open/close 12356 (time 1790983056.46 total 642.90 last 21.48) - open/close 12613 (time 1790983066.93 total 653.37 last 24.55) - open/close 12862 (time 1790983076.93 total 663.38 last 24.88) - open/close 13031 (time 1790983086.96 total 673.40 last 16.86) - open/close 13059 (time 1790983096.98 total 683.42 last 2.79) - open/close 13294 (time 1790983107.11 total 693.56 last 23.19) - open/close 13538 (time 1790983118.12 total 704.57 last 22.17) - open/close 13773 (time 1790983128.13 total 714.57 last 23.48) - open/close 13906 (time 1790983138.16 total 724.60 last 13.27) - open/close 14164 (time 1790983148.26 total 734.70 last 25.53) - open/close 14415 (time 1790983158.76 total 745.20 last 23.91) - open/close 14684 (time 1790983168.76 total 755.20 last 26.90) - open/close 14823 (time 1790983179.16 total 765.60 last 13.36) - open/close 15040 (time 1790983189.35 total 775.79 last 21.30) - open/close 15281 (time 1790983199.36 total 785.80 last 24.08) - open/close 15520 (time 1790983209.38 total 795.82 last 23.85) - open/close 15679 (time 1790983219.77 total 806.21 last 15.30) - open/close 15852 (time 1790983230.44 total 816.89 last 16.21) - open/close 16005 (time 1790983240.51 total 826.96 last 15.20) - open/close 16159 (time 1790983251.33 total 837.77 last 14.24) - open/close 16392 (time 1790983261.33 total 847.78 last 23.29) - open/close 16569 (time 1790983271.35 total 857.79 last 17.67) - open/close 16794 (time 1790983281.38 total 867.82 last 22.44) - open/close 17039 (time 1790983292.12 total 878.56 last 22.81) - open/close 17185 (time 1790983302.47 total 888.92 last 14.10) - open/close 17340 (time 1790983312.51 total 898.96 last 15.44) - open/close 17513 (time 1790983323.53 total 909.97 last 15.70) - open/close 17631 (time 1790983334.59 total 921.03 last 10.67) - open/close 17726 (time 1790983345.64 total 932.09 last 8.59) - open/close 17768 (time 1790983356.26 total 942.70 last 3.96) - open/close 17871 (time 1790983367.29 total 953.74 last 9.34) - open/close 18012 (time 1790983377.70 total 964.15 last 13.54) - open/close 18412 (time 1790983387.70 total 974.15 last 40.00) - open/close 18990 (time 1790983397.73 total 984.17 last 57.65) - open/close 19605 (time 1790983407.73 total 994.18 last 61.48) - open/close 20000 (time 1790983414.21 total 1000.66 last 60.95) - open/close 20564 (time 1790983424.22 total 1010.66 last 56.37) - open/close 21067 (time 1790983434.23 total 1020.68 last 50.22) - open/close 21623 (time 1790983444.25 total 1030.69 last 55.52) - open/close 22216 (time 1790983454.26 total 1040.70 last 59.24) - open/close 22802 (time 1790983464.26 total 1050.71 last 58.57) - open/close 23418 (time 1790983474.27 total 1060.72 last 61.54) - open/close 24003 (time 1790983484.28 total 1070.72 last 58.49) - open/close 24584 (time 1790983494.30 total 1080.74 last 57.99) - open/close 25186 (time 1790983504.30 total 1090.75 last 60.14) - open/close 25825 (time 1790983514.31 total 1100.75 last 63.87) - open/close 26450 (time 1790983524.32 total 1110.77 last 62.42) - open/close 27100 (time 1790983534.34 total 1120.78 last 64.91) - open/close 27698 (time 1790983544.35 total 1130.79 last 59.71) - open/close 28321 (time 1790983554.36 total 1140.80 last 62.25) - open/close 28922 (time 1790983564.36 total 1150.80 last 60.10) - open/close 29505 (time 1790983574.37 total 1160.82 last 58.23) - open/close 30000 (time 1790983582.90 total 1169.34 last 58.03) - open/close 30524 (time 1790983592.91 total 1179.35 last 52.36) - open/close 31140 (time 1790983602.92 total 1189.36 last 61.55) - open/close 31711 (time 1790983612.93 total 1199.38 last 57.01) - open/close 32232 (time 1790983622.94 total 1209.38 last 52.09) - open/close 32881 (time 1790983632.96 total 1219.40 last 64.74) - open/close 33477 (time 1790983642.96 total 1229.41 last 59.58) - open/close 34007 (time 1790983652.97 total 1239.41 last 52.97) - open/close 34610 (time 1790983662.97 total 1249.42 last 60.28) - open/close 35225 (time 1790983672.99 total 1259.44 last 61.37) - open/close 35854 (time 1790983683.01 total 1269.45 last 62.81) - open/close 36447 (time 1790983693.04 total 1279.48 last 59.12) - open/close 37025 (time 1790983703.05 total 1289.49 last 57.72) - open/close 37630 (time 1790983713.06 total 1299.50 last 60.45) - open/close 38207 (time 1790983723.07 total 1309.52 last 57.63) - open/close 38789 (time 1790983733.09 total 1319.53 last 58.12) - open/close 39343 (time 1790983743.11 total 1329.55 last 55.29) - open/close 39975 (time 1790983753.12 total 1339.57 last 63.09) - open/close 40615 (time 1790983763.15 total 1349.59 last 63.86) - open/close 41253 (time 1790983773.15 total 1359.59 last 63.77) - open/close 41877 (time 1790983783.16 total 1369.60 last 62.35) - open/close 42426 (time 1790983793.18 total 1379.62 last 54.79) - open/close 43030 (time 1790983803.18 total 1389.62 last 60.39) - open/close 43604 (time 1790983813.18 total 1399.62 last 57.40) - open/close 44208 (time 1790983823.19 total 1409.63 last 60.34) - open/close 44849 (time 1790983833.20 total 1419.64 last 64.01) - open/close 45465 (time 1790983843.21 total 1429.65 last 61.55) - open/close 46117 (time 1790983853.22 total 1439.66 last 65.16) - open/close 46728 (time 1790983863.22 total 1449.66 last 61.08) - open/close 47314 (time 1790983873.22 total 1459.67 last 58.57) - open/close 47912 (time 1790983883.23 total 1469.67 last 59.79) - open/close 48464 (time 1790983893.28 total 1479.72 last 54.91) - open/close 48998 (time 1790983903.28 total 1489.72 last 53.40) - open/close 49485 (time 1790983913.29 total 1499.73 last 48.67) - open/close 49967 (time 1790983923.32 total 1509.76 last 48.06) total: 50000 open/close in 1510.56 seconds: 33.10 ops/second - unlinked 0 (time 1790983929 ; total 0 ; last 0) - unlinked 10000 (time 1790984097 ; total 168 ; last 168) - unlinked 20000 (time 1790984264 ; total 335 ; last 167) - unlinked 30000 (time 1790984433 ; total 504 ; last 169) - unlinked 40000 (time 1790984606 ; total 677 ; last 173) total: 50000 unlinks in 847 seconds: 59.031876 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg454-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg454-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg454-server stop mds service on oleg454-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg454-server unloading modules via unload_modules_local on: 'oleg454-server' oleg454-server: oleg454-server.virtnet: executing unload_modules_local oleg454-server: kmemleak disabled modules unloaded. pdsh@oleg454-client: oleg454-client: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-server: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-client: ssh exited with exit code 2 pdsh@oleg454-client: oleg454-server: ssh exited with exit code 2