== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:35:32 (1789378532) start mds service on oleg228-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 04:14 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing load_modules_local oleg228-server: kmemleak disabled oleg228-server: Loading modules from /home/green/git/lustre-release/lustre oleg228-server: detected 4 online CPUs by sysfs oleg228-server: MODOPTS_LIBCFS= oleg228-server: Force libcfs to create 2 CPU partitions oleg228-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg228-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg228-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg228-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg228-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg228-client: oleg228-server: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-server: ssh exited with exit code 123 allow_register=unlimited oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg228-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 201 (time 1789378629.34 total 10.00 last 20.09) - open/close 361 (time 1789378639.34 total 20.01 last 15.99) - open/close 494 (time 1789378649.35 total 30.01 last 13.29) - open/close 609 (time 1789378659.35 total 40.02 last 11.49) - open/close 790 (time 1789378669.73 total 50.40 last 17.44) - open/close 1027 (time 1789378680.08 total 60.75 last 22.90) - open/close 1241 (time 1789378691.00 total 71.66 last 19.60) - open/close 1389 (time 1789378701.66 total 82.33 last 13.88) - open/close 1515 (time 1789378711.67 total 92.34 last 12.58) - open/close 1652 (time 1789378722.36 total 103.03 last 12.82) - open/close 1865 (time 1789378732.37 total 113.04 last 21.27) - open/close 1955 (time 1789378743.40 total 124.07 last 8.16) - open/close 2026 (time 1789378753.43 total 134.10 last 7.08) - open/close 2192 (time 1789378763.95 total 144.61 last 15.78) - open/close 2302 (time 1789378773.96 total 154.62 last 10.99) - open/close 2544 (time 1789378783.98 total 164.65 last 24.15) - open/close 2770 (time 1789378793.99 total 174.65 last 22.58) - open/close 2984 (time 1789378805.03 total 185.69 last 19.38) - open/close 3199 (time 1789378815.65 total 196.32 last 20.24) - open/close 3425 (time 1789378826.42 total 207.09 last 20.99) - open/close 3624 (time 1789378837.80 total 218.46 last 17.49) - open/close 3779 (time 1789378847.91 total 228.57 last 15.33) - open/close 3922 (time 1789378858.62 total 239.28 last 13.35) - open/close 4105 (time 1789378868.67 total 249.33 last 18.21) - open/close 4195 (time 1789378878.67 total 259.34 last 9.00) - open/close 4317 (time 1789378889.64 total 270.31 last 11.12) - open/close 4514 (time 1789378899.77 total 280.43 last 19.45) - open/close 4694 (time 1789378910.18 total 290.84 last 17.29) - open/close 4807 (time 1789378920.21 total 300.88 last 11.26) - open/close 4923 (time 1789378931.01 total 311.68 last 10.74) - open/close 5031 (time 1789378941.02 total 321.68 last 10.79) - open/close 5217 (time 1789378951.50 total 332.16 last 17.75) - open/close 5427 (time 1789378961.85 total 342.51 last 20.29) - open/close 5548 (time 1789378973.81 total 354.48 last 10.11) - open/close 5634 (time 1789378984.39 total 365.05 last 8.13) - open/close 5740 (time 1789378994.40 total 375.07 last 10.59) - open/close 5874 (time 1789379005.23 total 385.89 last 12.38) - open/close 5991 (time 1789379016.05 total 396.72 last 10.81) - open/close 6160 (time 1789379026.05 total 406.72 last 16.89) - open/close 6296 (time 1789379036.07 total 416.74 last 13.58) - open/close 6418 (time 1789379046.55 total 427.21 last 11.65) - open/close 6532 (time 1789379059.40 total 440.07 last 8.87) - open/close 6692 (time 1789379070.18 total 450.85 last 14.84) - open/close 6821 (time 1789379080.20 total 460.86 last 12.88) - open/close 6933 (time 1789379090.22 total 470.88 last 11.18) - open/close 7043 (time 1789379100.85 total 481.51 last 10.35) - open/close 7214 (time 1789379110.85 total 491.52 last 17.10) - open/close 7336 (time 1789379120.85 total 501.52 last 12.20) - open/close 7485 (time 1789379130.85 total 511.52 last 14.90) - open/close 7569 (time 1789379141.74 total 522.41 last 7.71) - open/close 7783 (time 1789379151.97 total 532.64 last 20.93) - open/close 7943 (time 1789379162.42 total 543.09 last 15.31) - open/close 8159 (time 1789379172.43 total 553.09 last 21.59) - open/close 8245 (time 1789379182.58 total 563.25 last 8.47) - open/close 8411 (time 1789379192.58 total 573.25 last 16.60) - open/close 8510 (time 1789379203.29 total 583.96 last 9.24) - open/close 8705 (time 1789379213.30 total 593.97 last 19.48) - open/close 8895 (time 1789379223.67 total 604.34 last 18.32) - open/close 9039 (time 1789379233.67 total 614.34 last 14.40) - open/close 9221 (time 1789379243.70 total 624.36 last 18.15) - open/close 9410 (time 1789379253.79 total 634.46 last 18.72) - open/close 9630 (time 1789379263.82 total 644.49 last 21.94) - open/close 9813 (time 1789379273.83 total 654.49 last 18.29) - open/close 9966 (time 1789379284.47 total 665.14 last 14.37) - open/close 10113 (time 1789379295.41 total 676.08 last 13.44) - open/close 10331 (time 1789379306.11 total 686.77 last 20.39) - open/close 10554 (time 1789379316.20 total 696.86 last 22.10) - open/close 10814 (time 1789379326.22 total 706.88 last 25.95) - open/close 10956 (time 1789379336.37 total 717.04 last 13.98) - open/close 11145 (time 1789379348.06 total 728.73 last 16.17) - open/close 11267 (time 1789379358.58 total 739.25 last 11.60) - open/close 11417 (time 1789379369.01 total 749.67 last 14.39) - open/close 11556 (time 1789379379.02 total 759.68 last 13.88) - open/close 11757 (time 1789379389.11 total 769.77 last 19.92) - open/close 11954 (time 1789379399.12 total 779.78 last 19.68) - open/close 12185 (time 1789379412.10 total 792.76 last 17.79) - open/close 12332 (time 1789379422.70 total 803.37 last 13.87) - open/close 12533 (time 1789379433.61 total 814.27 last 18.43) - open/close 12645 (time 1789379443.61 total 824.28 last 11.19) - open/close 12842 (time 1789379453.69 total 834.36 last 19.55) - open/close 13094 (time 1789379463.72 total 844.38 last 25.13) - open/close 13225 (time 1789379474.72 total 855.39 last 11.91) - open/close 13292 (time 1789379484.93 total 865.59 last 6.57) - open/close 13500 (time 1789379494.95 total 875.62 last 20.75) - open/close 13662 (time 1789379505.74 total 886.40 last 15.02) - open/close 13860 (time 1789379515.77 total 896.44 last 19.73) - open/close 14037 (time 1789379526.13 total 906.80 last 17.08) - open/close 14158 (time 1789379536.74 total 917.41 last 11.41) - open/close 14302 (time 1789379546.75 total 927.42 last 14.38) - open/close 14465 (time 1789379556.76 total 937.43 last 16.29) - open/close 14621 (time 1789379567.32 total 947.99 last 14.77) - open/close 14767 (time 1789379577.66 total 958.32 last 14.13) - open/close 14974 (time 1789379587.67 total 968.33 last 20.68) - open/close 15091 (time 1789379597.68 total 978.34 last 11.69) - open/close 15233 (time 1789379608.60 total 989.27 last 12.99) - open/close 15357 (time 1789379619.30 total 999.96 last 11.60) - open/close 15459 (time 1789379629.35 total 1010.02 last 10.15) - open/close 15569 (time 1789379640.44 total 1021.11 last 9.92) - open/close 15646 (time 1789379651.45 total 1032.11 last 7.00) - open/close 15879 (time 1789379661.96 total 1042.62 last 22.17) - open/close 16025 (time 1789379671.96 total 1052.63 last 14.59) - open/close 16242 (time 1789379682.48 total 1063.15 last 20.63) - open/close 16418 (time 1789379692.87 total 1073.53 last 16.95) - open/close 16515 (time 1789379702.88 total 1083.55 last 9.69) - open/close 16622 (time 1789379712.92 total 1093.59 last 10.66) - open/close 16740 (time 1789379723.13 total 1103.80 last 11.56) - open/close 16838 (time 1789379736.45 total 1117.12 last 7.36) - open/close 17361 (time 1789379746.48 total 1127.15 last 52.13) - open/close 17954 (time 1789379756.49 total 1137.15 last 59.30) - open/close 18456 (time 1789379766.52 total 1147.18 last 50.05) - open/close 18944 (time 1789379776.53 total 1157.19 last 48.75) - open/close 19336 (time 1789379786.54 total 1167.20 last 39.16) - open/close 19735 (time 1789379796.54 total 1177.21 last 39.88) - open/close 20000 (time 1789379801.63 total 1182.30 last 52.01) - open/close 20504 (time 1789379811.64 total 1192.31 last 50.37) - open/close 21055 (time 1789379821.67 total 1202.33 last 54.95) - open/close 21588 (time 1789379831.68 total 1212.34 last 53.24) - open/close 22114 (time 1789379841.68 total 1222.35 last 52.57) - open/close 22654 (time 1789379851.70 total 1232.37 last 53.88) - open/close 23230 (time 1789379861.73 total 1242.40 last 57.44) - open/close 23813 (time 1789379871.74 total 1252.41 last 58.25) - open/close 24365 (time 1789379881.77 total 1262.43 last 55.06) - open/close 24835 (time 1789379891.81 total 1272.48 last 46.79) - open/close 25269 (time 1789379901.82 total 1282.49 last 43.36) - open/close 25695 (time 1789379911.82 total 1292.49 last 42.60) - open/close 26140 (time 1789379921.83 total 1302.49 last 44.48) - open/close 26743 (time 1789379931.83 total 1312.50 last 60.27) - open/close 27302 (time 1789379941.86 total 1322.52 last 55.76) - open/close 27872 (time 1789379951.86 total 1332.52 last 56.99) - open/close 28450 (time 1789379961.87 total 1342.53 last 57.76) - open/close 28999 (time 1789379971.87 total 1352.54 last 54.85) - open/close 29502 (time 1789379981.88 total 1362.54 last 50.28) - open/close 30000 (time 1789379989.86 total 1370.53 last 62.40) - open/close 30562 (time 1789379999.87 total 1380.54 last 56.13) - open/close 31101 (time 1789380009.89 total 1390.56 last 53.78) - open/close 31741 (time 1789380019.93 total 1400.60 last 63.75) - open/close 32316 (time 1789380029.98 total 1410.65 last 57.23) - open/close 32883 (time 1789380039.99 total 1420.66 last 56.64) - open/close 33269 (time 1789380050.00 total 1430.66 last 38.59) - open/close 33819 (time 1789380060.01 total 1440.67 last 54.93) - open/close 34226 (time 1789380070.03 total 1450.70 last 40.61) - open/close 34722 (time 1789380080.04 total 1460.70 last 49.56) - open/close 35288 (time 1789380090.05 total 1470.71 last 56.54) - open/close 35818 (time 1789380100.05 total 1480.72 last 52.98) - open/close 36430 (time 1789380110.05 total 1490.72 last 61.19) - open/close 36928 (time 1789380120.05 total 1500.72 last 49.80) - open/close 37595 (time 1789380130.06 total 1510.72 last 66.68) - open/close 38055 (time 1789380140.06 total 1520.73 last 45.97) - open/close 38494 (time 1789380150.07 total 1530.73 last 43.87) - open/close 38899 (time 1789380160.08 total 1540.75 last 40.46) - open/close 39425 (time 1789380170.08 total 1550.75 last 52.59) - open/close 39977 (time 1789380180.09 total 1560.76 last 55.15) - open/close 40399 (time 1789380190.10 total 1570.76 last 42.17) - open/close 40765 (time 1789380200.14 total 1580.81 last 36.44) - open/close 41144 (time 1789380210.14 total 1590.81 last 37.88) - open/close 41663 (time 1789380220.15 total 1600.82 last 51.85) - open/close 42305 (time 1789380230.16 total 1610.82 last 64.18) - open/close 42863 (time 1789380240.16 total 1620.83 last 55.79) - open/close 43520 (time 1789380250.17 total 1630.83 last 65.65) - open/close 44060 (time 1789380260.17 total 1640.84 last 53.97) - open/close 44574 (time 1789380270.18 total 1650.85 last 51.34) - open/close 44951 (time 1789380280.19 total 1660.85 last 37.69) - open/close 45377 (time 1789380290.20 total 1670.87 last 42.53) - open/close 45932 (time 1789380300.21 total 1680.87 last 55.48) - open/close 46401 (time 1789380310.22 total 1690.88 last 46.85) - open/close 46868 (time 1789380320.25 total 1700.91 last 46.56) - open/close 47409 (time 1789380330.27 total 1710.93 last 53.99) - open/close 48086 (time 1789380340.28 total 1720.95 last 67.60) - open/close 48638 (time 1789380350.30 total 1730.97 last 55.08) - open/close 49159 (time 1789380360.31 total 1740.97 last 52.08) - open/close 49724 (time 1789380370.32 total 1750.98 last 56.44) total: 50000 open/close in 1755.95 seconds: 28.47 ops/second - unlinked 0 (time 1789380380 ; total 0 ; last 0) - unlinked 10000 (time 1789380566 ; total 186 ; last 186) - unlinked 20000 (time 1789380742 ; total 362 ; last 176) - unlinked 30000 (time 1789380923 ; total 543 ; last 181) - unlinked 40000 (time 1789381110 ; total 730 ; last 187) total: 50000 unlinks in 915 seconds: 54.644810 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg228-client.virtnet /mnt/lustre (opts:) oleg228-server: params: OBD_IOC_LLOG_PRINT failed: No such file or directory pdsh@oleg228-client: oleg228-server: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-server: ssh exited with exit code 123 allow_register=unlimited stop ost1 service on oleg228-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg228-server stop mds service on oleg228-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg228-server unloading modules via unload_modules_local on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing unload_modules_local oleg228-server: kmemleak disabled modules unloaded. pdsh@oleg228-client: oleg228-server: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-client: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-client: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-server: ssh exited with exit code 2