== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:39:28 (1790948368) start mds service on oleg630-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1946968 Oct 2 03:41 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing load_modules_local oleg630-server: kmemleak disabled oleg630-server: Loading modules from /home/green/git/lustre-release/lustre oleg630-server: detected 4 online CPUs by sysfs oleg630-server: MODOPTS_LIBCFS= oleg630-server: Force libcfs to create 2 CPU partitions oleg630-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg630-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg630-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg630-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg630-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg630-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg630-server: oleg630-server.virtnet: executing set_default_debug -1 all pdsh@oleg630-client: oleg630-server: ssh exited with exit code 1 Started lustre-OST0000 oleg630-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg630-client.virtnet: -o user_xattr,flock 192.168.206.130@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 205 (time 1790948453.44 total 10.91 last 18.80) - open/close 447 (time 1790948464.25 total 21.71 last 22.39) - open/close 763 (time 1790948474.26 total 31.73 last 31.56) - open/close 946 (time 1790948484.56 total 42.03 last 17.76) - open/close 1214 (time 1790948494.57 total 52.03 last 26.79) - open/close 1415 (time 1790948504.58 total 62.04 last 20.08) - open/close 1626 (time 1790948514.75 total 72.21 last 20.74) - open/close 1737 (time 1790948525.18 total 82.65 last 10.64) - open/close 1987 (time 1790948535.80 total 93.26 last 23.55) - open/close 2204 (time 1790948545.81 total 103.28 last 21.67) - open/close 2324 (time 1790948556.35 total 113.82 last 11.38) - open/close 2459 (time 1790948566.36 total 123.83 last 13.49) - open/close 2548 (time 1790948576.37 total 133.84 last 8.89) - open/close 2652 (time 1790948586.42 total 143.88 last 10.35) - open/close 2848 (time 1790948596.43 total 153.90 last 19.57) - open/close 3068 (time 1790948606.43 total 163.90 last 22.00) - open/close 3266 (time 1790948616.44 total 173.91 last 19.78) - open/close 3424 (time 1790948626.61 total 184.08 last 15.53) - open/close 3639 (time 1790948636.63 total 194.09 last 21.47) - open/close 3879 (time 1790948647.12 total 204.58 last 22.88) - open/close 4125 (time 1790948657.78 total 215.25 last 23.06) - open/close 4344 (time 1790948668.04 total 225.50 last 21.36) - open/close 4547 (time 1790948678.27 total 235.74 last 19.84) - open/close 4683 (time 1790948688.27 total 245.74 last 13.60) - open/close 4850 (time 1790948698.88 total 256.35 last 15.74) - open/close 4950 (time 1790948711.22 total 268.69 last 8.10) - open/close 5082 (time 1790948721.23 total 278.70 last 13.18) - open/close 5294 (time 1790948731.25 total 288.72 last 21.16) - open/close 5529 (time 1790948741.26 total 298.73 last 23.47) - open/close 5754 (time 1790948751.46 total 308.93 last 22.05) - open/close 5962 (time 1790948762.24 total 319.71 last 19.29) - open/close 6165 (time 1790948772.25 total 329.72 last 20.28) - open/close 6364 (time 1790948782.27 total 339.73 last 19.87) - open/close 6541 (time 1790948792.66 total 350.13 last 17.03) - open/close 6761 (time 1790948802.67 total 360.14 last 21.98) - open/close 6896 (time 1790948813.56 total 371.03 last 12.40) - open/close 7092 (time 1790948824.34 total 381.80 last 18.19) - open/close 7349 (time 1790948834.71 total 392.18 last 24.77) - open/close 7602 (time 1790948845.33 total 402.80 last 23.83) - open/close 7862 (time 1790948855.94 total 413.40 last 24.51) - open/close 8147 (time 1790948865.96 total 423.43 last 28.44) - open/close 8385 (time 1790948875.96 total 433.43 last 23.79) - open/close 8574 (time 1790948886.41 total 443.87 last 18.10) - open/close 8814 (time 1790948896.42 total 453.89 last 23.96) - open/close 9081 (time 1790948909.37 total 466.84 last 20.62) - open/close 9355 (time 1790948919.39 total 476.85 last 27.36) - open/close 9595 (time 1790948929.68 total 487.14 last 23.33) - open/close 9849 (time 1790948940.49 total 497.96 last 23.48) - open/close 9972 (time 1790948951.80 total 509.26 last 10.88) - open/close 10199 (time 1790948962.12 total 519.58 last 21.99) - open/close 10425 (time 1790948972.22 total 529.69 last 22.37) - open/close 10636 (time 1790948982.23 total 539.70 last 21.07) - open/close 10857 (time 1790948992.42 total 549.89 last 21.69) - open/close 10991 (time 1790949002.49 total 559.96 last 13.31) - open/close 11148 (time 1790949012.82 total 570.29 last 15.20) - open/close 11456 (time 1790949022.84 total 580.31 last 30.74) - open/close 11680 (time 1790949033.67 total 591.13 last 20.69) - open/close 11934 (time 1790949043.69 total 601.16 last 25.34) - open/close 12151 (time 1790949053.79 total 611.25 last 21.50) - open/close 12342 (time 1790949064.83 total 622.30 last 17.29) - open/close 12465 (time 1790949075.19 total 632.66 last 11.88) - open/close 12735 (time 1790949085.21 total 642.68 last 26.94) - open/close 12996 (time 1790949095.98 total 653.45 last 24.23) - open/close 13181 (time 1790949106.80 total 664.27 last 17.09) - open/close 13401 (time 1790949116.80 total 674.27 last 22.00) - open/close 13699 (time 1790949127.16 total 684.63 last 28.77) - open/close 13942 (time 1790949137.42 total 694.89 last 23.68) - open/close 14165 (time 1790949148.34 total 705.81 last 20.42) - open/close 14425 (time 1790949159.16 total 716.63 last 24.03) - open/close 14654 (time 1790949169.78 total 727.25 last 21.58) - open/close 14867 (time 1790949180.35 total 737.82 last 20.14) - open/close 15045 (time 1790949190.66 total 748.12 last 17.27) - open/close 15234 (time 1790949201.23 total 758.70 last 17.88) - open/close 15425 (time 1790949212.30 total 769.76 last 17.26) - open/close 15622 (time 1790949222.37 total 779.84 last 19.55) - open/close 15751 (time 1790949233.19 total 790.66 last 11.92) - open/close 15887 (time 1790949243.40 total 800.86 last 13.32) - open/close 16122 (time 1790949253.89 total 811.36 last 22.39) - open/close 16324 (time 1790949263.89 total 821.36 last 20.20) - open/close 16491 (time 1790949276.33 total 833.80 last 13.43) - open/close 16670 (time 1790949286.33 total 843.80 last 17.90) - open/close 16842 (time 1790949296.33 total 853.80 last 17.20) - open/close 16929 (time 1790949308.36 total 865.83 last 7.23) - open/close 17178 (time 1790949319.23 total 876.70 last 22.90) - open/close 17423 (time 1790949329.66 total 887.13 last 23.49) - open/close 17691 (time 1790949340.53 total 898.00 last 24.64) - open/close 17882 (time 1790949351.50 total 908.97 last 17.41) - open/close 18139 (time 1790949361.52 total 918.98 last 25.66) - open/close 18506 (time 1790949371.54 total 929.00 last 36.63) - open/close 19254 (time 1790949381.54 total 939.01 last 74.78) - open/close 20000 (time 1790949391.14 total 948.61 last 77.70) - open/close 20778 (time 1790949401.16 total 958.62 last 77.67) - open/close 21475 (time 1790949411.16 total 968.63 last 69.67) - open/close 22188 (time 1790949421.18 total 978.64 last 71.20) - open/close 22944 (time 1790949431.18 total 988.65 last 75.56) - open/close 23627 (time 1790949441.18 total 998.65 last 68.28) - open/close 24364 (time 1790949451.19 total 1008.66 last 73.63) - open/close 25089 (time 1790949461.20 total 1018.67 last 72.44) - open/close 25774 (time 1790949471.21 total 1028.67 last 68.47) - open/close 26428 (time 1790949481.21 total 1038.67 last 65.39) - open/close 27105 (time 1790949491.21 total 1048.67 last 67.70) - open/close 27875 (time 1790949501.24 total 1058.71 last 76.77) - open/close 28580 (time 1790949511.25 total 1068.72 last 70.42) - open/close 29318 (time 1790949521.25 total 1078.72 last 73.80) - open/close 30000 (time 1790949530.74 total 1088.20 last 71.89) - open/close 30747 (time 1790949540.75 total 1098.21 last 74.63) - open/close 31495 (time 1790949550.76 total 1108.22 last 74.71) - open/close 32248 (time 1790949560.76 total 1118.23 last 75.29) - open/close 33018 (time 1790949570.76 total 1128.23 last 76.98) - open/close 33787 (time 1790949580.76 total 1138.23 last 76.89) - open/close 34432 (time 1790949590.77 total 1148.24 last 64.46) - open/close 35114 (time 1790949600.78 total 1158.24 last 68.16) - open/close 35829 (time 1790949610.79 total 1168.26 last 71.39) - open/close 36553 (time 1790949620.79 total 1178.26 last 72.39) - open/close 37243 (time 1790949630.80 total 1188.26 last 68.98) - open/close 37935 (time 1790949640.80 total 1198.26 last 69.19) - open/close 38562 (time 1790949650.80 total 1208.27 last 62.67) - open/close 39162 (time 1790949660.80 total 1218.27 last 60.00) - open/close 39858 (time 1790949670.81 total 1228.28 last 69.51) - open/close 40677 (time 1790949680.82 total 1238.29 last 81.85) - open/close 41415 (time 1790949690.83 total 1248.30 last 73.71) - open/close 42159 (time 1790949700.88 total 1258.35 last 74.04) - open/close 42796 (time 1790949710.89 total 1268.35 last 63.66) - open/close 43530 (time 1790949720.92 total 1278.38 last 73.19) - open/close 44268 (time 1790949730.92 total 1288.39 last 73.74) - open/close 44982 (time 1790949740.93 total 1298.40 last 71.37) - open/close 45732 (time 1790949750.94 total 1308.40 last 74.94) - open/close 46462 (time 1790949760.95 total 1318.41 last 72.92) - open/close 47156 (time 1790949770.95 total 1328.42 last 69.36) - open/close 47877 (time 1790949780.97 total 1338.44 last 71.97) - open/close 48543 (time 1790949790.98 total 1348.45 last 66.55) - open/close 49147 (time 1790949800.98 total 1358.45 last 60.38) - open/close 49833 (time 1790949810.99 total 1368.45 last 68.57) total: 50000 open/close in 1371.31 seconds: 36.46 ops/second - unlinked 0 (time 1790949819 ; total 0 ; last 0) - unlinked 10000 (time 1790949970 ; total 151 ; last 151) - unlinked 20000 (time 1790950118 ; total 299 ; last 148) - unlinked 30000 (time 1790950268 ; total 449 ; last 150) - unlinked 40000 (time 1790950415 ; total 596 ; last 147) total: 50000 unlinks in 751 seconds: 66.577896 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg630-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg630-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg630-server stop mds service on oleg630-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg630-server unloading modules via unload_modules_local on: 'oleg630-server' oleg630-server: oleg630-server.virtnet: executing unload_modules_local oleg630-server: kmemleak disabled modules unloaded. pdsh@oleg630-client: oleg630-server: ssh exited with exit code 2 pdsh@oleg630-client: oleg630-client: ssh exited with exit code 2 pdsh@oleg630-client: oleg630-client: ssh exited with exit code 2 pdsh@oleg630-client: oleg630-server: ssh exited with exit code 2