== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:04:12 (1789581852) start mds service on oleg225-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1932672 Sep 16 08:21 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing load_modules_local oleg225-server: kmemleak disabled oleg225-server: Loading modules from /home/green/git/lustre-release/lustre oleg225-server: detected 4 online CPUs by sysfs oleg225-server: MODOPTS_LIBCFS= oleg225-server: Force libcfs to create 2 CPU partitions oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg225-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg225-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 112 (time 1789581947.52 total 10.06 last 11.14) - open/close 235 (time 1789581957.53 total 20.06 last 12.29) - open/close 324 (time 1789581967.76 total 30.30 last 8.70) - open/close 489 (time 1789581978.48 total 41.02 last 15.39) - open/close 570 (time 1789581988.65 total 51.19 last 7.96) - open/close 626 (time 1789581999.34 total 61.88 last 5.24) - open/close 720 (time 1789582009.35 total 71.89 last 9.39) - open/close 883 (time 1789582019.35 total 81.89 last 16.30) - open/close 983 (time 1789582030.32 total 92.86 last 9.11) - open/close 1207 (time 1789582040.56 total 103.10 last 21.88) - open/close 1302 (time 1789582051.00 total 113.54 last 9.10) - open/close 1378 (time 1789582061.61 total 124.15 last 7.16) - open/close 1479 (time 1789582072.47 total 135.01 last 9.30) - open/close 1599 (time 1789582085.30 total 147.84 last 9.36) - open/close 1780 (time 1789582095.30 total 157.84 last 18.10) - open/close 1946 (time 1789582105.30 total 167.84 last 16.60) - open/close 2095 (time 1789582116.28 total 178.82 last 13.56) - open/close 2153 (time 1789582127.27 total 189.81 last 5.28) - open/close 2314 (time 1789582137.43 total 199.97 last 15.85) - open/close 2460 (time 1789582148.27 total 210.81 last 13.47) - open/close 2697 (time 1789582158.89 total 221.43 last 22.32) - open/close 2955 (time 1789582170.72 total 233.26 last 21.81) - open/close 3199 (time 1789582182.25 total 244.79 last 21.17) - open/close 3336 (time 1789582192.27 total 254.81 last 13.67) - open/close 3548 (time 1789582202.28 total 264.82 last 21.18) - open/close 3772 (time 1789582212.45 total 274.99 last 22.03) - open/close 4030 (time 1789582223.04 total 285.58 last 24.36) - open/close 4110 (time 1789582233.27 total 295.81 last 7.82) - open/close 4248 (time 1789582243.69 total 306.23 last 13.24) - open/close 4391 (time 1789582254.61 total 317.15 last 13.10) - open/close 4651 (time 1789582264.65 total 327.19 last 25.88) - open/close 4874 (time 1789582274.66 total 337.20 last 22.29) - open/close 5056 (time 1789582284.67 total 347.21 last 18.18) - open/close 5220 (time 1789582295.05 total 357.59 last 15.80) - open/close 5388 (time 1789582305.85 total 368.39 last 15.56) - open/close 5598 (time 1789582316.00 total 378.54 last 20.68) - open/close 5860 (time 1789582326.02 total 388.56 last 26.16) - open/close 6046 (time 1789582336.10 total 398.64 last 18.45) - open/close 6186 (time 1789582346.13 total 408.67 last 13.97) - open/close 6432 (time 1789582357.12 total 419.66 last 22.38) - open/close 6667 (time 1789582367.54 total 430.08 last 22.54) - open/close 6814 (time 1789582379.96 total 442.50 last 11.84) - open/close 7053 (time 1789582390.90 total 453.44 last 21.84) - open/close 7321 (time 1789582400.92 total 463.46 last 26.76) - open/close 7515 (time 1789582411.82 total 474.36 last 17.80) - open/close 7762 (time 1789582421.83 total 484.37 last 24.67) - open/close 7960 (time 1789582432.21 total 494.75 last 19.08) - open/close 8200 (time 1789582442.98 total 505.51 last 22.29) - open/close 8455 (time 1789582452.98 total 515.52 last 25.50) - open/close 8506 (time 1789582463.62 total 526.16 last 4.79) - open/close 8599 (time 1789582476.46 total 539.00 last 7.25) - open/close 8704 (time 1789582487.47 total 550.01 last 9.54) - open/close 8850 (time 1789582497.90 total 560.44 last 14.00) - open/close 8944 (time 1789582508.87 total 571.40 last 8.57) - open/close 9152 (time 1789582519.48 total 582.02 last 19.60) - open/close 9342 (time 1789582529.52 total 592.06 last 18.93) - open/close 9479 (time 1789582540.04 total 602.58 last 13.02) - open/close 9661 (time 1789582551.09 total 613.63 last 16.47) - open/close 9876 (time 1789582562.02 total 624.56 last 19.67) - open/close 9970 (time 1789582572.27 total 634.81 last 9.17) - open/close 10000 (time 1789582575.55 total 638.09 last 9.15) - open/close 10150 (time 1789582585.89 total 648.43 last 14.51) - open/close 10291 (time 1789582596.59 total 659.13 last 13.18) - open/close 10505 (time 1789582608.12 total 670.66 last 18.56) - open/close 10676 (time 1789582618.48 total 681.02 last 16.50) - open/close 10771 (time 1789582628.97 total 691.51 last 9.06) - open/close 10991 (time 1789582639.79 total 702.33 last 20.32) - open/close 11141 (time 1789582650.71 total 713.25 last 13.74) - open/close 11273 (time 1789582661.67 total 724.21 last 12.05) - open/close 11368 (time 1789582672.36 total 734.90 last 8.88) - open/close 11474 (time 1789582682.62 total 745.16 last 10.33) - open/close 11727 (time 1789582692.91 total 755.44 last 24.60) - open/close 11865 (time 1789582702.91 total 765.45 last 13.79) - open/close 12076 (time 1789582713.03 total 775.57 last 20.85) - open/close 12249 (time 1789582723.70 total 786.24 last 16.22) - open/close 12481 (time 1789582733.71 total 796.25 last 23.17) - open/close 12619 (time 1789582744.56 total 807.10 last 12.72) - open/close 12726 (time 1789582755.58 total 818.12 last 9.70) - open/close 12853 (time 1789582766.57 total 829.11 last 11.56) - open/close 12936 (time 1789582777.15 total 839.69 last 7.85) - open/close 13034 (time 1789582787.15 total 849.69 last 9.80) - open/close 13233 (time 1789582797.37 total 859.91 last 19.47) - open/close 13354 (time 1789582807.39 total 869.93 last 12.08) - open/close 13465 (time 1789582817.40 total 879.94 last 11.08) - open/close 13703 (time 1789582827.43 total 889.97 last 23.75) - open/close 13904 (time 1789582837.44 total 899.98 last 20.07) - open/close 14082 (time 1789582847.99 total 910.53 last 16.88) - open/close 14221 (time 1789582857.99 total 920.53 last 13.89) - open/close 14355 (time 1789582868.00 total 930.54 last 13.39) - open/close 14550 (time 1789582878.00 total 940.54 last 19.49) - open/close 14677 (time 1789582891.14 total 953.68 last 9.67) - open/close 14787 (time 1789582901.68 total 964.22 last 10.44) - open/close 14887 (time 1789582911.86 total 974.40 last 9.83) - open/close 14958 (time 1789582922.42 total 984.96 last 6.72) - open/close 15078 (time 1789582933.27 total 995.81 last 11.05) - open/close 15175 (time 1789582943.67 total 1006.21 last 9.33) - open/close 15268 (time 1789582954.18 total 1016.71 last 8.85) - open/close 15389 (time 1789582965.00 total 1027.54 last 11.18) - open/close 15497 (time 1789582975.01 total 1037.55 last 10.79) - open/close 15671 (time 1789582985.39 total 1047.93 last 16.76) - open/close 15773 (time 1789582995.40 total 1057.94 last 10.19) - open/close 15976 (time 1789583005.43 total 1067.97 last 20.24) - open/close 16160 (time 1789583015.99 total 1078.53 last 17.42) - open/close 16191 (time 1789583026.18 total 1088.72 last 3.04) - open/close 16322 (time 1789583036.19 total 1098.73 last 13.09) - open/close 16422 (time 1789583047.14 total 1109.67 last 9.13) - open/close 16629 (time 1789583057.14 total 1119.67 last 20.70) - open/close 17268 (time 1789583067.14 total 1129.68 last 63.85) - open/close 18086 (time 1789583077.14 total 1139.68 last 81.78) - open/close 18861 (time 1789583087.15 total 1149.68 last 77.50) - open/close 19723 (time 1789583097.15 total 1159.69 last 86.16) - open/close 20000 (time 1789583101.43 total 1163.97 last 64.71) - open/close 20621 (time 1789583111.43 total 1173.97 last 62.09) - open/close 21287 (time 1789583121.44 total 1183.98 last 66.57) - open/close 22029 (time 1789583131.44 total 1193.98 last 74.14) - open/close 22856 (time 1789583141.44 total 1203.98 last 82.70) - open/close 23805 (time 1789583151.45 total 1213.99 last 94.87) - open/close 24592 (time 1789583161.48 total 1224.02 last 78.48) - open/close 25259 (time 1789583171.48 total 1234.02 last 66.66) - open/close 25903 (time 1789583181.49 total 1244.03 last 64.37) - open/close 26436 (time 1789583191.50 total 1254.04 last 53.21) - open/close 27213 (time 1789583201.53 total 1264.07 last 77.50) - open/close 27963 (time 1789583211.53 total 1274.07 last 74.97) - open/close 28766 (time 1789583221.55 total 1284.08 last 80.20) - open/close 29572 (time 1789583231.55 total 1294.09 last 80.55) - open/close 30000 (time 1789583236.57 total 1299.11 last 85.28) - open/close 30710 (time 1789583246.58 total 1309.12 last 70.95) - open/close 31545 (time 1789583256.59 total 1319.13 last 83.42) - open/close 32390 (time 1789583266.59 total 1329.13 last 84.47) - open/close 33073 (time 1789583276.60 total 1339.14 last 68.24) - open/close 33671 (time 1789583286.60 total 1349.14 last 59.79) - open/close 34363 (time 1789583296.60 total 1359.14 last 69.18) - open/close 35182 (time 1789583306.61 total 1369.15 last 81.85) - open/close 35854 (time 1789583316.68 total 1379.22 last 66.70) - open/close 36473 (time 1789583326.69 total 1389.23 last 61.86) - open/close 36871 (time 1789583336.70 total 1399.23 last 39.78) - open/close 37492 (time 1789583346.70 total 1409.24 last 62.08) - open/close 38297 (time 1789583356.71 total 1419.25 last 80.42) - open/close 39006 (time 1789583366.71 total 1429.25 last 70.86) - open/close 39917 (time 1789583376.71 total 1439.25 last 91.09) - open/close 40737 (time 1789583386.72 total 1449.26 last 81.96) - open/close 41575 (time 1789583396.72 total 1459.26 last 83.78) - open/close 42509 (time 1789583406.73 total 1469.27 last 93.35) - open/close 43344 (time 1789583416.74 total 1479.28 last 83.38) - open/close 44176 (time 1789583426.74 total 1489.28 last 83.18) - open/close 44884 (time 1789583436.75 total 1499.28 last 70.78) - open/close 45665 (time 1789583446.76 total 1509.30 last 78.00) - open/close 46442 (time 1789583456.76 total 1519.30 last 77.68) - open/close 47239 (time 1789583466.76 total 1529.30 last 79.69) - open/close 47969 (time 1789583476.77 total 1539.31 last 72.97) - open/close 48785 (time 1789583486.77 total 1549.31 last 81.56) - open/close 49578 (time 1789583496.77 total 1559.31 last 79.28) total: 50000 open/close in 1564.25 seconds: 31.96 ops/second - unlinked 0 (time 1789583505 ; total 0 ; last 0) - unlinked 10000 (time 1789583633 ; total 128 ; last 128) - unlinked 20000 (time 1789583769 ; total 264 ; last 136) - unlinked 30000 (time 1789583919 ; total 414 ; last 150) - unlinked 40000 (time 1789584059 ; total 554 ; last 140) total: 50000 unlinks in 687 seconds: 72.780205 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg225-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server unloading modules via unload_modules_local on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing unload_modules_local oleg225-server: kmemleak disabled modules unloaded. pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2