== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:26:32 (1790922392) start mds service on oleg231-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1946968 Oct 1 19:15 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing load_modules_local oleg231-server: kmemleak disabled oleg231-server: Loading modules from /home/green/git/lustre-release/lustre oleg231-server: detected 4 online CPUs by sysfs oleg231-server: MODOPTS_LIBCFS= oleg231-server: Force libcfs to create 2 CPU partitions oleg231-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg231-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg231-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg231-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg231-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg231-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg231-server: oleg231-server.virtnet: executing set_default_debug -1 all pdsh@oleg231-client: oleg231-server: ssh exited with exit code 1 Started lustre-OST0000 oleg231-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg231-client.virtnet: -o user_xattr,flock 192.168.202.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 120 (time 1790922487.39 total 10.37 last 11.58) - open/close 305 (time 1790922497.79 total 20.76 last 17.79) - open/close 552 (time 1790922508.04 total 31.01 last 24.10) - open/close 794 (time 1790922518.12 total 41.10 last 23.99) - open/close 1027 (time 1790922528.80 total 51.77 last 21.83) - open/close 1141 (time 1790922538.95 total 61.93 last 11.23) - open/close 1267 (time 1790922549.23 total 72.21 last 12.26) - open/close 1452 (time 1790922559.54 total 82.52 last 17.94) - open/close 1572 (time 1790922570.02 total 92.99 last 11.45) - open/close 1738 (time 1790922580.89 total 103.87 last 15.27) - open/close 1862 (time 1790922591.51 total 114.49 last 11.67) - open/close 2007 (time 1790922601.83 total 124.80 last 14.06) - open/close 2106 (time 1790922614.55 total 137.53 last 7.78) - open/close 2256 (time 1790922625.57 total 148.55 last 13.61) - open/close 2524 (time 1790922635.61 total 158.59 last 26.69) - open/close 2702 (time 1790922645.62 total 168.59 last 17.79) - open/close 2758 (time 1790922656.66 total 179.64 last 5.07) - open/close 2825 (time 1790922666.68 total 189.65 last 6.69) - open/close 2927 (time 1790922678.03 total 201.01 last 8.98) - open/close 3090 (time 1790922688.04 total 211.02 last 16.29) - open/close 3297 (time 1790922698.76 total 221.74 last 19.30) - open/close 3393 (time 1790922710.23 total 233.20 last 8.37) - open/close 3531 (time 1790922720.33 total 243.31 last 13.66) - open/close 3621 (time 1790922731.15 total 254.12 last 8.32) - open/close 3729 (time 1790922741.17 total 264.15 last 10.78) - open/close 3833 (time 1790922751.29 total 274.26 last 10.28) - open/close 4017 (time 1790922761.32 total 284.29 last 18.35) - open/close 4168 (time 1790922772.00 total 294.98 last 14.13) - open/close 4277 (time 1790922782.01 total 304.99 last 10.89) - open/close 4412 (time 1790922792.62 total 315.60 last 12.72) - open/close 4670 (time 1790922802.92 total 325.90 last 25.05) - open/close 4824 (time 1790922812.95 total 335.93 last 15.35) - open/close 5072 (time 1790922822.96 total 345.94 last 24.77) - open/close 5241 (time 1790922832.96 total 355.94 last 16.90) - open/close 5392 (time 1790922843.71 total 366.69 last 14.04) - open/close 5513 (time 1790922853.87 total 376.84 last 11.92) - open/close 5655 (time 1790922863.99 total 386.96 last 14.03) - open/close 5826 (time 1790922875.02 total 398.00 last 15.50) - open/close 5917 (time 1790922885.08 total 408.06 last 9.05) - open/close 6023 (time 1790922895.56 total 418.53 last 10.12) - open/close 6269 (time 1790922906.08 total 429.05 last 23.38) - open/close 6376 (time 1790922918.31 total 441.28 last 8.75) - open/close 6426 (time 1790922929.29 total 452.27 last 4.55) - open/close 6490 (time 1790922939.31 total 462.28 last 6.39) - open/close 6613 (time 1790922949.31 total 472.29 last 12.29) - open/close 6699 (time 1790922959.83 total 482.80 last 8.18) - open/close 6803 (time 1790922970.09 total 493.07 last 10.13) - open/close 6964 (time 1790922980.43 total 503.40 last 15.58) - open/close 7006 (time 1790922990.45 total 513.43 last 4.19) - open/close 7129 (time 1790923000.86 total 523.84 last 11.82) - open/close 7241 (time 1790923010.97 total 533.95 last 11.08) - open/close 7350 (time 1790923021.02 total 543.99 last 10.85) - open/close 7509 (time 1790923031.02 total 553.99 last 15.90) - open/close 7645 (time 1790923041.36 total 564.33 last 13.16) - open/close 7815 (time 1790923051.37 total 574.35 last 16.97) - open/close 7937 (time 1790923062.19 total 585.16 last 11.28) - open/close 8132 (time 1790923072.85 total 595.82 last 18.29) - open/close 8234 (time 1790923083.24 total 606.21 last 9.81) - open/close 8433 (time 1790923093.71 total 616.68 last 19.01) - open/close 8547 (time 1790923103.89 total 626.86 last 11.20) - open/close 8670 (time 1790923113.89 total 636.87 last 12.29) - open/close 8780 (time 1790923126.79 total 649.77 last 8.53) - open/close 8818 (time 1790923137.15 total 660.12 last 3.67) - open/close 9002 (time 1790923147.69 total 670.66 last 17.45) - open/close 9059 (time 1790923157.89 total 680.87 last 5.59) - open/close 9190 (time 1790923167.91 total 690.88 last 13.08) - open/close 9281 (time 1790923178.08 total 701.06 last 8.94) - open/close 9492 (time 1790923191.08 total 714.06 last 16.23) - open/close 9626 (time 1790923201.69 total 724.67 last 12.63) - open/close 9879 (time 1790923211.73 total 734.70 last 25.21) - open/close 10000 (time 1790923219.50 total 742.47 last 15.57) - open/close 10088 (time 1790923230.05 total 753.03 last 8.34) - open/close 10180 (time 1790923240.86 total 763.84 last 8.51) - open/close 10267 (time 1790923251.35 total 774.33 last 8.30) - open/close 10467 (time 1790923262.03 total 785.00 last 18.73) - open/close 10583 (time 1790923275.15 total 798.13 last 8.84) - open/close 10769 (time 1790923285.78 total 808.75 last 17.51) - open/close 10973 (time 1790923296.27 total 819.24 last 19.44) - open/close 11089 (time 1790923306.59 total 829.57 last 11.24) - open/close 11197 (time 1790923316.65 total 839.62 last 10.74) - open/close 11301 (time 1790923327.18 total 850.15 last 9.87) - open/close 11336 (time 1790923337.80 total 860.77 last 3.30) - open/close 11557 (time 1790923347.81 total 870.79 last 22.07) - open/close 11649 (time 1790923357.83 total 880.80 last 9.18) - open/close 11818 (time 1790923368.09 total 891.06 last 16.47) - open/close 11927 (time 1790923378.22 total 901.19 last 10.76) - open/close 12007 (time 1790923388.70 total 911.68 last 7.63) - open/close 12200 (time 1790923399.50 total 922.48 last 17.87) - open/close 12319 (time 1790923412.06 total 935.04 last 9.47) - open/close 12530 (time 1790923422.07 total 945.05 last 21.08) - open/close 12680 (time 1790923433.58 total 956.55 last 13.04) - open/close 12818 (time 1790923444.52 total 967.49 last 12.61) - open/close 12944 (time 1790923454.55 total 977.52 last 12.56) - open/close 13129 (time 1790923465.07 total 988.04 last 17.59) - open/close 13267 (time 1790923475.92 total 998.89 last 12.72) - open/close 13454 (time 1790923486.25 total 1009.22 last 18.10) - open/close 13608 (time 1790923496.25 total 1019.22 last 15.40) - open/close 13766 (time 1790923506.25 total 1029.23 last 15.79) - open/close 13889 (time 1790923516.44 total 1039.42 last 12.07) - open/close 14043 (time 1790923526.74 total 1049.72 last 14.96) - open/close 14192 (time 1790923536.77 total 1059.74 last 14.86) - open/close 14337 (time 1790923548.90 total 1071.87 last 11.95) - open/close 14476 (time 1790923559.76 total 1082.74 last 12.79) - open/close 14640 (time 1790923570.02 total 1092.99 last 16.00) - open/close 14787 (time 1790923580.51 total 1103.48 last 14.01) - open/close 14917 (time 1790923590.53 total 1113.51 last 12.96) - open/close 15295 (time 1790923600.54 total 1123.52 last 37.77) - open/close 15925 (time 1790923610.56 total 1133.53 last 62.90) - open/close 16555 (time 1790923620.56 total 1143.53 last 62.99) - open/close 17144 (time 1790923630.56 total 1153.54 last 58.87) - open/close 17816 (time 1790923640.57 total 1163.54 last 67.17) - open/close 18394 (time 1790923650.57 total 1173.55 last 57.79) - open/close 18995 (time 1790923660.57 total 1183.55 last 60.10) - open/close 19671 (time 1790923670.57 total 1193.55 last 67.59) - open/close 20000 (time 1790923676.89 total 1199.87 last 52.06) - open/close 20512 (time 1790923686.90 total 1209.88 last 51.15) - open/close 21109 (time 1790923696.93 total 1219.91 last 59.52) - open/close 21693 (time 1790923706.95 total 1229.92 last 58.32) - open/close 22292 (time 1790923716.95 total 1239.92 last 59.89) - open/close 22924 (time 1790923726.96 total 1249.93 last 63.13) - open/close 23506 (time 1790923736.98 total 1259.96 last 58.06) - open/close 24095 (time 1790923747.00 total 1269.97 last 58.81) - open/close 24829 (time 1790923757.00 total 1279.98 last 73.36) - open/close 25440 (time 1790923767.02 total 1290.00 last 60.99) - open/close 26095 (time 1790923777.02 total 1300.00 last 65.48) - open/close 26606 (time 1790923787.03 total 1310.00 last 51.09) - open/close 27106 (time 1790923797.03 total 1320.01 last 49.98) - open/close 27549 (time 1790923807.04 total 1330.02 last 44.25) - open/close 28178 (time 1790923817.04 total 1340.02 last 62.88) - open/close 28957 (time 1790923827.05 total 1350.02 last 77.87) - open/close 29552 (time 1790923837.08 total 1360.06 last 59.29) - open/close 30000 (time 1790923843.45 total 1366.43 last 70.35) - open/close 30675 (time 1790923853.46 total 1376.44 last 67.41) - open/close 31235 (time 1790923863.48 total 1386.45 last 55.93) - open/close 31818 (time 1790923873.48 total 1396.45 last 58.28) - open/close 32448 (time 1790923883.48 total 1406.46 last 62.98) - open/close 33124 (time 1790923893.48 total 1416.46 last 67.60) - open/close 33725 (time 1790923903.49 total 1426.47 last 60.05) - open/close 34382 (time 1790923913.51 total 1436.49 last 65.57) - open/close 34983 (time 1790923923.51 total 1446.49 last 60.09) - open/close 35525 (time 1790923933.52 total 1456.49 last 54.19) - open/close 36123 (time 1790923943.53 total 1466.51 last 59.71) - open/close 36556 (time 1790923953.55 total 1476.53 last 43.21) - open/close 37042 (time 1790923963.56 total 1486.53 last 48.58) - open/close 37610 (time 1790923973.56 total 1496.54 last 56.76) - open/close 38186 (time 1790923983.61 total 1506.59 last 57.31) - open/close 38775 (time 1790923993.63 total 1516.60 last 58.81) - open/close 39405 (time 1790924003.64 total 1526.61 last 62.95) - open/close 40000 (time 1790924012.56 total 1535.53 last 66.71) - open/close 40620 (time 1790924022.56 total 1545.53 last 61.98) - open/close 41251 (time 1790924032.56 total 1555.53 last 63.09) - open/close 41804 (time 1790924042.56 total 1565.54 last 55.27) - open/close 42410 (time 1790924052.57 total 1575.54 last 60.57) - open/close 42992 (time 1790924062.59 total 1585.56 last 58.11) - open/close 43612 (time 1790924072.59 total 1595.56 last 61.97) - open/close 44208 (time 1790924082.59 total 1605.57 last 59.59) - open/close 44774 (time 1790924092.66 total 1615.63 last 56.23) - open/close 45450 (time 1790924102.68 total 1625.65 last 67.46) - open/close 46100 (time 1790924112.69 total 1635.67 last 64.92) - open/close 46784 (time 1790924122.69 total 1645.67 last 68.40) - open/close 47481 (time 1790924132.70 total 1655.68 last 69.61) - open/close 48140 (time 1790924142.71 total 1665.69 last 65.86) - open/close 48743 (time 1790924152.72 total 1675.69 last 60.26) - open/close 49314 (time 1790924162.74 total 1685.71 last 57.00) - open/close 49993 (time 1790924172.74 total 1695.72 last 67.85) total: 50000 open/close in 1695.84 seconds: 29.48 ops/second - unlinked 0 (time 1790924178 ; total 0 ; last 0) - unlinked 10000 (time 1790924351 ; total 173 ; last 173) - unlinked 20000 (time 1790924527 ; total 349 ; last 176) - unlinked 30000 (time 1790924710 ; total 532 ; last 183) - unlinked 40000 (time 1790924873 ; total 695 ; last 163) total: 50000 unlinks in 857 seconds: 58.343056 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg231-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg231-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg231-server stop mds service on oleg231-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg231-server unloading modules via unload_modules_local on: 'oleg231-server' oleg231-server: oleg231-server.virtnet: executing unload_modules_local oleg231-server: kmemleak disabled modules unloaded. pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-server: ssh exited with exit code 2 pdsh@oleg231-client: oleg231-client: ssh exited with exit code 2