== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:46:25 (1790948785) start mds service on oleg114-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1946968 Oct 2 07:21 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing load_modules_local oleg114-server: kmemleak disabled oleg114-server: Loading modules from /home/green/git/lustre-release/lustre oleg114-server: detected 4 online CPUs by sysfs oleg114-server: MODOPTS_LIBCFS= oleg114-server: Force libcfs to create 2 CPU partitions oleg114-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg114-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg114-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg114-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg114-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg114-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg114-server: oleg114-server.virtnet: executing set_default_debug -1 all pdsh@oleg114-client: oleg114-server: ssh exited with exit code 1 Started lustre-OST0000 oleg114-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg114-client.virtnet: -o user_xattr,flock 192.168.201.114@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 213 (time 1790948860.96 total 10.01 last 21.28) - open/close 437 (time 1790948871.46 total 20.50 last 21.34) - open/close 667 (time 1790948882.20 total 31.24 last 21.42) - open/close 911 (time 1790948892.70 total 41.75 last 23.23) - open/close 1116 (time 1790948902.71 total 51.76 last 20.47) - open/close 1361 (time 1790948912.73 total 61.77 last 24.47) - open/close 1629 (time 1790948922.73 total 71.78 last 26.79) - open/close 1882 (time 1790948933.55 total 82.60 last 23.38) - open/close 2113 (time 1790948944.28 total 93.32 last 21.54) - open/close 2351 (time 1790948954.96 total 104.00 last 22.28) - open/close 2434 (time 1790948964.96 total 114.01 last 8.30) - open/close 2586 (time 1790948974.96 total 124.01 last 15.20) - open/close 2803 (time 1790948985.31 total 134.35 last 20.98) - open/close 2915 (time 1790948995.31 total 144.36 last 11.19) - open/close 3177 (time 1790949005.99 total 155.04 last 24.53) - open/close 3375 (time 1790949016.01 total 165.06 last 19.76) - open/close 3644 (time 1790949027.04 total 176.09 last 24.39) - open/close 3890 (time 1790949037.67 total 186.71 last 23.15) - open/close 4101 (time 1790949047.76 total 196.81 last 20.90) - open/close 4346 (time 1790949058.14 total 207.19 last 23.61) - open/close 4561 (time 1790949068.31 total 217.36 last 21.13) - open/close 4763 (time 1790949078.31 total 227.36 last 20.20) - open/close 4983 (time 1790949088.32 total 237.36 last 22.00) - open/close 5243 (time 1790949098.93 total 247.98 last 24.49) - open/close 5484 (time 1790949109.59 total 258.64 last 22.61) - open/close 5633 (time 1790949119.63 total 268.68 last 14.84) - open/close 5886 (time 1790949129.74 total 278.79 last 25.03) - open/close 6155 (time 1790949139.75 total 288.79 last 26.89) - open/close 6349 (time 1790949151.58 total 300.63 last 16.39) - open/close 6559 (time 1790949164.06 total 313.10 last 16.84) - open/close 6706 (time 1790949177.96 total 327.01 last 10.57) - open/close 6909 (time 1790949189.02 total 338.07 last 18.36) - open/close 7082 (time 1790949200.92 total 349.97 last 14.54) - open/close 7350 (time 1790949211.60 total 360.65 last 25.09) - open/close 7647 (time 1790949222.10 total 371.15 last 28.28) - open/close 7900 (time 1790949232.79 total 381.84 last 23.67) - open/close 8208 (time 1790949243.75 total 392.80 last 28.11) - open/close 8400 (time 1790949253.84 total 402.89 last 19.02) - open/close 8574 (time 1790949263.85 total 412.90 last 17.39) - open/close 8721 (time 1790949273.86 total 422.90 last 14.69) - open/close 8850 (time 1790949286.75 total 435.80 last 10.00) - open/close 9097 (time 1790949297.31 total 446.35 last 23.40) - open/close 9280 (time 1790949307.32 total 456.37 last 18.27) - open/close 9468 (time 1790949317.75 total 466.79 last 18.04) - open/close 9710 (time 1790949328.37 total 477.41 last 22.79) - open/close 9965 (time 1790949339.35 total 488.39 last 23.22) - open/close 10204 (time 1790949349.70 total 498.74 last 23.09) - open/close 10430 (time 1790949359.85 total 508.90 last 22.25) - open/close 10668 (time 1790949370.45 total 519.49 last 22.46) - open/close 10927 (time 1790949381.28 total 530.33 last 23.90) - open/close 11199 (time 1790949392.17 total 541.21 last 25.00) - open/close 11424 (time 1790949402.66 total 551.71 last 21.44) - open/close 11666 (time 1790949412.95 total 562.00 last 23.52) - open/close 11941 (time 1790949423.66 total 572.71 last 25.68) - open/close 12245 (time 1790949433.66 total 582.71 last 30.39) - open/close 12424 (time 1790949443.94 total 592.99 last 17.41) - open/close 12656 (time 1790949453.96 total 603.00 last 23.16) - open/close 12878 (time 1790949464.15 total 613.20 last 21.78) - open/close 13117 (time 1790949474.72 total 623.77 last 22.61) - open/close 13336 (time 1790949484.97 total 634.01 last 21.37) - open/close 13567 (time 1790949494.98 total 644.02 last 23.07) - open/close 13769 (time 1790949505.51 total 654.56 last 19.17) - open/close 14009 (time 1790949515.80 total 664.84 last 23.34) - open/close 14242 (time 1790949525.80 total 674.84 last 23.30) - open/close 14459 (time 1790949535.80 total 684.85 last 21.69) - open/close 14695 (time 1790949546.77 total 695.82 last 21.52) - open/close 14925 (time 1790949556.79 total 705.84 last 22.95) - open/close 15169 (time 1790949567.52 total 716.57 last 22.74) - open/close 15378 (time 1790949577.53 total 726.58 last 20.88) - open/close 15631 (time 1790949588.50 total 737.55 last 23.07) - open/close 15937 (time 1790949598.51 total 747.56 last 30.57) - open/close 16165 (time 1790949609.11 total 758.15 last 21.52) - open/close 16463 (time 1790949619.18 total 768.23 last 29.59) - open/close 16696 (time 1790949629.65 total 778.70 last 22.24) - open/close 16928 (time 1790949639.67 total 788.72 last 23.16) - open/close 17226 (time 1790949649.67 total 798.72 last 29.79) - open/close 17515 (time 1790949660.11 total 809.16 last 27.69) - open/close 17758 (time 1790949670.61 total 819.66 last 23.14) - open/close 17962 (time 1790949681.44 total 830.48 last 18.85) - open/close 18187 (time 1790949692.18 total 841.23 last 20.94) - open/close 18334 (time 1790949703.51 total 852.56 last 12.98) - open/close 18466 (time 1790949714.00 total 863.04 last 12.59) - open/close 18748 (time 1790949728.87 total 877.91 last 18.97) - open/close 18912 (time 1790949739.05 total 888.10 last 16.10) - open/close 19003 (time 1790949750.03 total 899.08 last 8.29) - open/close 19206 (time 1790949760.60 total 909.65 last 19.21) - open/close 19907 (time 1790949770.61 total 919.66 last 70.05) - open/close 20718 (time 1790949780.62 total 929.66 last 81.06) - open/close 21615 (time 1790949790.62 total 939.67 last 89.65) - open/close 22352 (time 1790949800.62 total 949.67 last 73.68) - open/close 23339 (time 1790949810.63 total 959.67 last 98.68) - open/close 24111 (time 1790949820.64 total 969.68 last 77.13) - open/close 24997 (time 1790949830.64 total 979.69 last 88.55) - open/close 25818 (time 1790949840.65 total 989.69 last 82.06) - open/close 26608 (time 1790949850.67 total 999.71 last 78.83) - open/close 27457 (time 1790949860.68 total 1009.72 last 84.82) - open/close 28351 (time 1790949870.68 total 1019.73 last 89.37) - open/close 29267 (time 1790949880.69 total 1029.73 last 91.56) - open/close 30000 (time 1790949888.67 total 1037.71 last 91.82) - open/close 30928 (time 1790949898.68 total 1047.73 last 92.67) - open/close 31906 (time 1790949908.69 total 1057.73 last 97.73) - open/close 32784 (time 1790949918.70 total 1067.75 last 87.70) - open/close 33716 (time 1790949928.70 total 1077.75 last 93.17) - open/close 34588 (time 1790949938.71 total 1087.76 last 87.14) - open/close 35469 (time 1790949948.72 total 1097.76 last 88.04) - open/close 36347 (time 1790949958.72 total 1107.77 last 87.75) - open/close 37196 (time 1790949968.74 total 1117.78 last 84.81) - open/close 38034 (time 1790949978.76 total 1127.80 last 83.62) - open/close 38754 (time 1790949988.79 total 1137.83 last 71.78) - open/close 39692 (time 1790949998.80 total 1147.84 last 93.74) - open/close 40000 (time 1790950002.49 total 1151.54 last 83.29) - open/close 40804 (time 1790950012.49 total 1161.54 last 80.39) - open/close 41722 (time 1790950022.49 total 1171.54 last 91.79) - open/close 42514 (time 1790950032.50 total 1181.55 last 79.12) - open/close 43303 (time 1790950042.50 total 1191.55 last 78.90) - open/close 44197 (time 1790950052.51 total 1201.56 last 89.33) - open/close 45054 (time 1790950062.51 total 1211.56 last 85.69) - open/close 45622 (time 1790950072.52 total 1221.56 last 56.78) - open/close 46402 (time 1790950082.52 total 1231.57 last 77.96) - open/close 47219 (time 1790950092.53 total 1241.57 last 81.66) - open/close 48060 (time 1790950102.53 total 1251.58 last 84.05) - open/close 48897 (time 1790950112.55 total 1261.60 last 83.55) - open/close 49762 (time 1790950122.56 total 1271.60 last 86.46) total: 50000 open/close in 1273.94 seconds: 39.25 ops/second - unlinked 0 (time 1790950128 ; total 0 ; last 0) - unlinked 10000 (time 1790950250 ; total 122 ; last 122) - unlinked 20000 (time 1790950364 ; total 236 ; last 114) - unlinked 30000 (time 1790950436 ; total 308 ; last 72) - unlinked 40000 (time 1790950513 ; total 385 ; last 77) total: 50000 unlinks in 454 seconds: 110.132156 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg114-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg114-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg114-server stop mds service on oleg114-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg114-server unloading modules via unload_modules_local on: 'oleg114-server' oleg114-server: oleg114-server.virtnet: executing unload_modules_local oleg114-server: kmemleak disabled modules unloaded. pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-client: ssh exited with exit code 2 pdsh@oleg114-client: oleg114-server: ssh exited with exit code 2