== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:43:55 (1790966635) start mds service on oleg215-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Oct 2 13:44 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing load_modules_local oleg215-server: kmemleak disabled oleg215-server: Loading modules from /home/green/git/lustre-release/lustre oleg215-server: detected 4 online CPUs by sysfs oleg215-server: MODOPTS_LIBCFS= oleg215-server: Force libcfs to create 2 CPU partitions oleg215-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg215-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg215-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg215-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg215-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Started lustre-OST0000 oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 271 (time 1790966726.77 total 10.01 last 27.09) - open/close 385 (time 1790966737.60 total 20.83 last 10.53) - open/close 562 (time 1790966747.95 total 31.19 last 17.10) - open/close 730 (time 1790966758.58 total 41.81 last 15.81) - open/close 959 (time 1790966768.70 total 51.94 last 22.62) - open/close 1168 (time 1790966779.25 total 62.49 last 19.80) - open/close 1313 (time 1790966789.56 total 72.80 last 14.06) - open/close 1507 (time 1790966799.58 total 82.81 last 19.37) - open/close 1757 (time 1790966810.36 total 93.59 last 23.20) - open/close 1999 (time 1790966820.55 total 103.78 last 23.75) - open/close 2217 (time 1790966831.42 total 114.65 last 20.06) - open/close 2296 (time 1790966841.42 total 124.65 last 7.90) - open/close 2426 (time 1790966851.63 total 134.87 last 12.72) - open/close 2635 (time 1790966862.01 total 145.24 last 20.15) - open/close 2870 (time 1790966872.03 total 155.26 last 23.45) - open/close 3064 (time 1790966882.04 total 165.28 last 19.37) - open/close 3166 (time 1790966892.40 total 175.64 last 9.85) - open/close 3478 (time 1790966902.90 total 186.13 last 29.73) - open/close 3715 (time 1790966912.93 total 196.17 last 23.61) - open/close 3965 (time 1790966922.94 total 206.18 last 24.98) - open/close 4105 (time 1790966932.94 total 216.18 last 14.00) - open/close 4251 (time 1790966943.66 total 226.90 last 13.62) - open/close 4317 (time 1790966953.93 total 237.16 last 6.43) - open/close 4455 (time 1790966963.94 total 247.18 last 13.78) - open/close 4554 (time 1790966973.97 total 257.20 last 9.87) - open/close 4695 (time 1790966984.76 total 268.00 last 13.06) - open/close 4824 (time 1790966995.76 total 279.00 last 11.73) - open/close 5005 (time 1790967005.77 total 289.01 last 18.08) - open/close 5134 (time 1790967016.03 total 299.27 last 12.58) - open/close 5267 (time 1790967028.35 total 311.58 last 10.80) - open/close 5454 (time 1790967038.44 total 321.68 last 18.53) - open/close 5568 (time 1790967049.33 total 332.57 last 10.47) - open/close 5731 (time 1790967059.33 total 342.57 last 16.30) - open/close 5870 (time 1790967070.36 total 353.59 last 12.61) - open/close 6001 (time 1790967080.36 total 363.59 last 13.10) - open/close 6159 (time 1790967090.65 total 373.88 last 15.36) - open/close 6314 (time 1790967101.65 total 384.88 last 14.09) - open/close 6495 (time 1790967112.18 total 395.42 last 17.18) - open/close 6639 (time 1790967123.19 total 406.42 last 13.09) - open/close 6823 (time 1790967133.19 total 416.43 last 18.40) - open/close 7037 (time 1790967143.36 total 426.60 last 21.04) - open/close 7209 (time 1790967154.09 total 437.33 last 16.03) - open/close 7446 (time 1790967164.59 total 447.83 last 22.56) - open/close 7617 (time 1790967174.60 total 457.84 last 17.09) - open/close 7842 (time 1790967185.20 total 468.43 last 21.24) - open/close 8031 (time 1790967197.55 total 480.79 last 15.30) - open/close 8228 (time 1790967207.59 total 490.82 last 19.63) - open/close 8397 (time 1790967220.56 total 503.79 last 13.03) - open/close 8550 (time 1790967230.57 total 513.81 last 15.28) - open/close 8707 (time 1790967240.85 total 524.09 last 15.27) - open/close 8916 (time 1790967251.77 total 535.00 last 19.15) - open/close 9112 (time 1790967261.79 total 545.03 last 19.56) - open/close 9335 (time 1790967271.83 total 555.06 last 22.21) - open/close 9502 (time 1790967282.51 total 565.75 last 15.63) - open/close 9755 (time 1790967295.48 total 578.71 last 19.51) - open/close 9958 (time 1790967305.53 total 588.77 last 20.19) - open/close 10000 (time 1790967307.72 total 590.96 last 19.20) - open/close 10127 (time 1790967318.59 total 601.82 last 11.69) - open/close 10378 (time 1790967328.63 total 611.87 last 24.99) - open/close 10605 (time 1790967338.91 total 622.14 last 22.09) - open/close 10881 (time 1790967349.68 total 632.92 last 25.62) - open/close 11054 (time 1790967360.12 total 643.36 last 16.56) - open/close 11276 (time 1790967370.68 total 653.91 last 21.04) - open/close 11552 (time 1790967381.50 total 664.73 last 25.50) - open/close 11742 (time 1790967391.52 total 674.75 last 18.96) - open/close 11926 (time 1790967405.33 total 688.56 last 13.32) - open/close 12079 (time 1790967418.04 total 701.28 last 12.03) - open/close 12221 (time 1790967428.06 total 711.29 last 14.17) - open/close 12323 (time 1790967438.08 total 721.32 last 10.18) - open/close 12467 (time 1790967448.15 total 731.38 last 14.30) - open/close 12647 (time 1790967458.55 total 741.79 last 17.30) - open/close 12872 (time 1790967468.60 total 751.84 last 22.39) - open/close 12958 (time 1790967479.29 total 762.53 last 8.04) - open/close 13075 (time 1790967489.33 total 772.57 last 11.65) - open/close 13183 (time 1790967500.16 total 783.40 last 9.97) - open/close 13281 (time 1790967510.22 total 793.45 last 9.74) - open/close 13424 (time 1790967520.22 total 803.45 last 14.30) - open/close 13517 (time 1790967530.24 total 813.48 last 9.28) - open/close 13778 (time 1790967540.67 total 823.90 last 25.04) - open/close 13965 (time 1790967551.00 total 834.23 last 18.10) - open/close 14085 (time 1790967561.02 total 844.26 last 11.97) - open/close 14190 (time 1790967571.21 total 854.44 last 10.31) - open/close 14317 (time 1790967581.30 total 864.53 last 12.59) - open/close 14520 (time 1790967591.30 total 874.53 last 20.29) - open/close 14719 (time 1790967601.97 total 885.21 last 18.64) - open/close 14768 (time 1790967612.68 total 895.91 last 4.58) - open/close 14876 (time 1790967623.62 total 906.85 last 9.87) - open/close 14983 (time 1790967634.43 total 917.66 last 9.90) - open/close 15153 (time 1790967645.40 total 928.64 last 15.48) - open/close 15302 (time 1790967655.41 total 938.65 last 14.89) - open/close 15377 (time 1790967665.79 total 949.03 last 7.23) - open/close 15499 (time 1790967675.80 total 959.04 last 12.18) - open/close 15696 (time 1790967686.36 total 969.60 last 18.65) - open/close 15799 (time 1790967697.02 total 980.25 last 9.67) - open/close 15921 (time 1790967707.35 total 990.58 last 11.81) - open/close 15999 (time 1790967717.68 total 1000.92 last 7.55) - open/close 16137 (time 1790967727.69 total 1010.93 last 13.79) - open/close 16332 (time 1790967738.32 total 1021.56 last 18.35) - open/close 16475 (time 1790967748.83 total 1032.07 last 13.60) - open/close 16582 (time 1790967758.85 total 1042.08 last 10.69) - open/close 16711 (time 1790967768.86 total 1052.09 last 12.89) - open/close 16880 (time 1790967778.86 total 1062.10 last 16.90) - open/close 16997 (time 1790967789.07 total 1072.30 last 11.46) - open/close 17080 (time 1790967800.14 total 1083.37 last 7.50) - open/close 17199 (time 1790967810.79 total 1094.03 last 11.17) - open/close 17352 (time 1790967821.19 total 1104.42 last 14.72) - open/close 17870 (time 1790967831.19 total 1114.42 last 51.79) - open/close 18600 (time 1790967841.20 total 1124.43 last 72.92) - open/close 19257 (time 1790967851.21 total 1134.44 last 65.65) - open/close 19984 (time 1790967861.21 total 1144.44 last 72.69) - open/close 20721 (time 1790967871.21 total 1154.45 last 73.66) - open/close 21336 (time 1790967881.22 total 1164.45 last 61.47) - open/close 22013 (time 1790967891.22 total 1174.46 last 67.66) - open/close 22646 (time 1790967901.22 total 1184.46 last 63.28) - open/close 23360 (time 1790967911.25 total 1194.48 last 71.25) - open/close 24021 (time 1790967921.25 total 1204.48 last 66.08) - open/close 24664 (time 1790967931.26 total 1214.50 last 64.20) - open/close 25385 (time 1790967941.27 total 1224.51 last 72.03) - open/close 26088 (time 1790967951.29 total 1234.53 last 70.16) - open/close 26779 (time 1790967961.30 total 1244.53 last 69.07) - open/close 27505 (time 1790967971.32 total 1254.56 last 72.41) - open/close 28080 (time 1790967981.34 total 1264.57 last 57.43) - open/close 28464 (time 1790967991.34 total 1274.58 last 38.38) - open/close 28981 (time 1790968001.35 total 1284.58 last 51.67) - open/close 29563 (time 1790968011.39 total 1294.62 last 57.97) - open/close 30000 (time 1790968017.55 total 1300.78 last 70.93) - open/close 30689 (time 1790968027.56 total 1310.80 last 68.83) - open/close 31428 (time 1790968037.56 total 1320.80 last 73.88) - open/close 32028 (time 1790968047.60 total 1330.83 last 59.79) - open/close 32654 (time 1790968057.62 total 1340.85 last 62.48) - open/close 33277 (time 1790968067.66 total 1350.90 last 62.04) - open/close 33901 (time 1790968077.67 total 1360.91 last 62.32) - open/close 34620 (time 1790968087.68 total 1370.92 last 71.85) - open/close 35178 (time 1790968097.69 total 1380.92 last 55.76) - open/close 35640 (time 1790968107.69 total 1390.93 last 46.19) - open/close 36204 (time 1790968117.70 total 1400.94 last 56.33) - open/close 36816 (time 1790968127.72 total 1410.96 last 61.06) - open/close 37479 (time 1790968137.73 total 1420.97 last 66.23) - open/close 38208 (time 1790968147.74 total 1430.98 last 72.86) - open/close 38938 (time 1790968157.74 total 1440.98 last 72.97) - open/close 39699 (time 1790968167.75 total 1450.98 last 76.08) - open/close 40000 (time 1790968171.63 total 1454.87 last 77.49) - open/close 40699 (time 1790968181.64 total 1464.87 last 69.86) - open/close 41361 (time 1790968191.65 total 1474.89 last 66.11) - open/close 42070 (time 1790968201.66 total 1484.89 last 70.86) - open/close 42825 (time 1790968211.66 total 1494.89 last 75.48) - open/close 43549 (time 1790968221.67 total 1504.90 last 72.34) - open/close 44289 (time 1790968231.67 total 1514.91 last 73.97) - open/close 45008 (time 1790968241.67 total 1524.91 last 71.89) - open/close 45674 (time 1790968251.69 total 1534.92 last 66.50) - open/close 46300 (time 1790968261.72 total 1544.95 last 62.42) - open/close 46994 (time 1790968271.73 total 1554.96 last 69.31) - open/close 47701 (time 1790968281.75 total 1564.99 last 70.53) - open/close 48432 (time 1790968291.77 total 1575.00 last 72.99) - open/close 49020 (time 1790968301.77 total 1585.00 last 58.80) - open/close 49677 (time 1790968311.77 total 1595.01 last 65.69) total: 50000 open/close in 1600.07 seconds: 31.25 ops/second - unlinked 0 (time 1790968322 ; total 0 ; last 0) - unlinked 10000 (time 1790968463 ; total 141 ; last 141) - unlinked 20000 (time 1790968608 ; total 286 ; last 145) - unlinked 30000 (time 1790968764 ; total 442 ; last 156) - unlinked 40000 (time 1790968932 ; total 610 ; last 168) total: 50000 unlinks in 779 seconds: 64.184853 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg215-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg215-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg215-server stop mds service on oleg215-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg215-server unloading modules via unload_modules_local on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing unload_modules_local oleg215-server: kmemleak disabled modules unloaded. pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2