== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:38:39 (1789285119) start mds service on oleg103-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 13 02:26 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg103-server' oleg103-server: oleg103-server.virtnet: executing load_modules_local oleg103-server: kmemleak disabled oleg103-server: Loading modules from /home/green/git/lustre-release/lustre oleg103-server: detected 4 online CPUs by sysfs oleg103-server: MODOPTS_LIBCFS= oleg103-server: Force libcfs to create 2 CPU partitions oleg103-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg103-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg103-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg103-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg103-server: oleg103-server.virtnet: executing set_default_debug -1 all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg103-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg103-server: oleg103-server.virtnet: executing set_default_debug -1 all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Started lustre-OST0000 oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg103-client.virtnet: -o user_xattr,flock 192.168.201.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 152 (time 1789285208.92 total 11.57 last 13.14) - open/close 289 (time 1789285218.92 total 21.57 last 13.70) - open/close 405 (time 1789285228.92 total 31.57 last 11.60) - open/close 604 (time 1789285238.95 total 41.60 last 19.85) - open/close 792 (time 1789285249.96 total 52.61 last 17.08) - open/close 971 (time 1789285259.99 total 62.64 last 17.84) - open/close 1247 (time 1789285270.39 total 73.04 last 26.55) - open/close 1469 (time 1789285280.40 total 83.05 last 22.18) - open/close 1598 (time 1789285290.41 total 93.06 last 12.89) - open/close 1811 (time 1789285300.43 total 103.07 last 21.26) - open/close 1973 (time 1789285311.23 total 113.88 last 15.00) - open/close 2204 (time 1789285322.02 total 124.67 last 21.40) - open/close 2429 (time 1789285332.04 total 134.69 last 22.46) - open/close 2643 (time 1789285342.88 total 145.53 last 19.74) - open/close 2851 (time 1789285353.63 total 156.27 last 19.36) - open/close 2946 (time 1789285364.33 total 166.98 last 8.87) - open/close 3081 (time 1789285374.89 total 177.53 last 12.79) - open/close 3254 (time 1789285385.02 total 187.67 last 17.07) - open/close 3428 (time 1789285395.03 total 197.68 last 17.38) - open/close 3604 (time 1789285405.03 total 207.68 last 17.60) - open/close 3837 (time 1789285415.03 total 217.68 last 23.29) - open/close 4120 (time 1789285425.70 total 228.35 last 26.53) - open/close 4376 (time 1789285435.71 total 238.35 last 25.59) - open/close 4594 (time 1789285446.15 total 248.80 last 20.88) - open/close 4776 (time 1789285456.16 total 258.81 last 18.18) - open/close 4933 (time 1789285466.74 total 269.39 last 14.83) - open/close 5049 (time 1789285476.75 total 279.40 last 11.59) - open/close 5276 (time 1789285487.62 total 290.27 last 20.89) - open/close 5446 (time 1789285498.49 total 301.14 last 15.63) - open/close 5554 (time 1789285508.51 total 311.16 last 10.78) - open/close 5720 (time 1789285519.09 total 321.73 last 15.69) - open/close 5853 (time 1789285529.89 total 332.54 last 12.30) - open/close 5985 (time 1789285540.13 total 342.78 last 12.90) - open/close 6102 (time 1789285550.95 total 353.60 last 10.81) - open/close 6253 (time 1789285561.95 total 364.60 last 13.73) - open/close 6366 (time 1789285571.97 total 374.61 last 11.29) - open/close 6585 (time 1789285582.69 total 385.34 last 20.42) - open/close 6804 (time 1789285593.06 total 395.71 last 21.12) - open/close 6990 (time 1789285603.50 total 406.15 last 17.82) - open/close 7177 (time 1789285614.53 total 417.17 last 16.96) - open/close 7324 (time 1789285624.75 total 427.40 last 14.37) - open/close 7550 (time 1789285634.95 total 437.60 last 22.16) - open/close 7782 (time 1789285644.99 total 447.64 last 23.12) - open/close 7909 (time 1789285657.24 total 459.89 last 10.36) - open/close 8106 (time 1789285667.95 total 470.60 last 18.40) - open/close 8226 (time 1789285678.77 total 481.41 last 11.10) - open/close 8409 (time 1789285689.64 total 492.29 last 16.82) - open/close 8438 (time 1789285699.90 total 502.54 last 2.83) - open/close 8558 (time 1789285710.63 total 513.28 last 11.18) - open/close 8703 (time 1789285721.06 total 523.71 last 13.90) - open/close 8828 (time 1789285731.31 total 533.96 last 12.20) - open/close 8925 (time 1789285741.87 total 544.52 last 9.18) - open/close 9021 (time 1789285752.54 total 555.19 last 9.00) - open/close 9175 (time 1789285763.37 total 566.02 last 14.21) - open/close 9225 (time 1789285774.25 total 576.90 last 4.60) - open/close 9318 (time 1789285784.25 total 586.90 last 9.30) - open/close 9429 (time 1789285794.26 total 596.91 last 11.10) - open/close 9584 (time 1789285805.15 total 607.79 last 14.24) - open/close 9802 (time 1789285815.16 total 617.80 last 21.78) - open/close 10000 (time 1789285824.42 total 627.06 last 21.38) - open/close 10211 (time 1789285834.44 total 637.08 last 21.06) - open/close 10280 (time 1789285844.47 total 647.12 last 6.88) - open/close 10416 (time 1789285854.48 total 657.13 last 13.58) - open/close 10580 (time 1789285864.49 total 667.14 last 16.38) - open/close 10718 (time 1789285874.98 total 677.63 last 13.16) - open/close 10846 (time 1789285884.98 total 687.63 last 12.79) - open/close 11044 (time 1789285894.99 total 697.64 last 19.79) - open/close 11228 (time 1789285907.81 total 710.45 last 14.35) - open/close 11378 (time 1789285918.52 total 721.17 last 14.00) - open/close 11465 (time 1789285928.57 total 731.22 last 8.66) - open/close 11711 (time 1789285939.48 total 742.13 last 22.54) - open/close 11856 (time 1789285950.19 total 752.84 last 13.54) - open/close 11914 (time 1789285960.22 total 762.86 last 5.78) - open/close 12039 (time 1789285970.23 total 772.87 last 12.49) - open/close 12112 (time 1789285980.40 total 783.05 last 7.17) - open/close 12319 (time 1789285991.36 total 794.00 last 18.90) - open/close 12397 (time 1789286001.64 total 804.28 last 7.59) - open/close 12605 (time 1789286011.66 total 814.30 last 20.76) - open/close 12833 (time 1789286022.43 total 825.08 last 21.16) - open/close 13078 (time 1789286032.43 total 835.08 last 24.49) - open/close 13298 (time 1789286043.38 total 846.03 last 20.10) - open/close 13443 (time 1789286053.38 total 856.03 last 14.50) - open/close 13522 (time 1789286063.49 total 866.14 last 7.82) - open/close 13644 (time 1789286075.56 total 878.21 last 10.11) - open/close 13826 (time 1789286086.06 total 888.71 last 17.33) - open/close 14071 (time 1789286096.16 total 898.81 last 24.25) - open/close 14243 (time 1789286106.16 total 908.81 last 17.19) - open/close 14419 (time 1789286116.18 total 918.83 last 17.57) - open/close 14602 (time 1789286127.01 total 929.65 last 16.90) - open/close 14748 (time 1789286139.17 total 941.82 last 12.00) - open/close 14825 (time 1789286150.19 total 952.84 last 6.99) - open/close 15024 (time 1789286160.43 total 963.08 last 19.43) - open/close 15123 (time 1789286170.55 total 973.19 last 9.79) - open/close 15179 (time 1789286180.55 total 983.20 last 5.60) - open/close 15273 (time 1789286192.16 total 994.81 last 8.10) - open/close 15426 (time 1789286202.22 total 1004.86 last 15.22) - open/close 15629 (time 1789286212.98 total 1015.62 last 18.87) - open/close 15822 (time 1789286223.00 total 1025.64 last 19.26) - open/close 15911 (time 1789286233.45 total 1036.10 last 8.51) - open/close 16054 (time 1789286243.94 total 1046.59 last 13.63) - open/close 16319 (time 1789286253.94 total 1056.59 last 26.50) - open/close 16906 (time 1789286263.95 total 1066.60 last 58.65) - open/close 17441 (time 1789286273.97 total 1076.62 last 53.39) - open/close 18208 (time 1789286283.98 total 1086.63 last 76.63) - open/close 18815 (time 1789286294.00 total 1096.64 last 60.61) - open/close 19581 (time 1789286304.01 total 1106.66 last 76.52) - open/close 20000 (time 1789286311.01 total 1113.66 last 59.82) - open/close 20620 (time 1789286321.02 total 1123.67 last 61.93) - open/close 21264 (time 1789286331.03 total 1133.68 last 64.33) - open/close 22009 (time 1789286341.04 total 1143.68 last 74.48) - open/close 22767 (time 1789286351.04 total 1153.69 last 75.76) - open/close 23464 (time 1789286361.05 total 1163.70 last 69.62) - open/close 24080 (time 1789286371.07 total 1173.72 last 61.50) - open/close 24750 (time 1789286381.08 total 1183.72 last 66.97) - open/close 25379 (time 1789286391.09 total 1193.73 last 62.84) - open/close 26117 (time 1789286401.09 total 1203.74 last 73.76) - open/close 26845 (time 1789286411.10 total 1213.74 last 72.77) - open/close 27539 (time 1789286421.10 total 1223.75 last 69.37) - open/close 28163 (time 1789286431.11 total 1233.76 last 62.33) - open/close 28650 (time 1789286441.12 total 1243.77 last 48.67) - open/close 29128 (time 1789286451.12 total 1253.77 last 47.77) - open/close 29633 (time 1789286461.13 total 1263.78 last 50.46) - open/close 30000 (time 1789286466.29 total 1268.94 last 71.19) - open/close 30835 (time 1789286476.29 total 1278.94 last 83.44) - open/close 31394 (time 1789286486.30 total 1288.95 last 55.88) - open/close 32016 (time 1789286496.31 total 1298.96 last 62.10) - open/close 32789 (time 1789286506.34 total 1308.98 last 77.13) - open/close 33489 (time 1789286516.34 total 1318.98 last 69.99) - open/close 34237 (time 1789286526.34 total 1328.99 last 74.77) - open/close 34873 (time 1789286536.35 total 1339.00 last 63.52) - open/close 35430 (time 1789286546.36 total 1349.00 last 55.68) - open/close 35886 (time 1789286556.37 total 1359.01 last 45.56) - open/close 36437 (time 1789286566.38 total 1369.03 last 55.02) - open/close 37039 (time 1789286576.39 total 1379.03 last 60.16) - open/close 37634 (time 1789286586.39 total 1389.04 last 59.50) - open/close 38215 (time 1789286596.39 total 1399.04 last 58.06) - open/close 38867 (time 1789286606.42 total 1409.06 last 65.07) - open/close 39418 (time 1789286616.43 total 1419.07 last 55.04) - open/close 39921 (time 1789286626.48 total 1429.13 last 50.01) - open/close 40333 (time 1789286636.49 total 1439.13 last 41.19) - open/close 40863 (time 1789286646.49 total 1449.14 last 52.99) - open/close 41505 (time 1789286656.49 total 1459.14 last 64.18) - open/close 42160 (time 1789286666.50 total 1469.15 last 65.43) - open/close 42827 (time 1789286676.51 total 1479.16 last 66.64) - open/close 43478 (time 1789286686.52 total 1489.17 last 65.06) - open/close 43993 (time 1789286696.52 total 1499.17 last 51.49) - open/close 44650 (time 1789286706.53 total 1509.18 last 65.65) - open/close 45363 (time 1789286716.53 total 1519.18 last 71.30) - open/close 46035 (time 1789286726.53 total 1529.18 last 67.16) - open/close 46721 (time 1789286736.54 total 1539.19 last 68.55) - open/close 47472 (time 1789286746.55 total 1549.20 last 75.01) - open/close 48080 (time 1789286756.56 total 1559.21 last 60.77) - open/close 48671 (time 1789286766.56 total 1569.21 last 59.07) - open/close 49499 (time 1789286776.57 total 1579.22 last 82.75) total: 50000 open/close in 1586.74 seconds: 31.51 ops/second - unlinked 0 (time 1789286789 ; total 0 ; last 0) - unlinked 10000 (time 1789286939 ; total 150 ; last 150) - unlinked 20000 (time 1789287110 ; total 321 ; last 171) - unlinked 30000 (time 1789287265 ; total 476 ; last 155) - unlinked 40000 (time 1789287420 ; total 631 ; last 155) total: 50000 unlinks in 788 seconds: 63.451778 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg103-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg103-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg103-server stop mds service on oleg103-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg103-server unloading modules via unload_modules_local on: 'oleg103-server' oleg103-server: oleg103-server.virtnet: executing unload_modules_local oleg103-server: kmemleak disabled modules unloaded. pdsh@oleg103-client: oleg103-client: ssh exited with exit code 2 pdsh@oleg103-client: oleg103-server: ssh exited with exit code 2 pdsh@oleg103-client: oleg103-client: ssh exited with exit code 2 pdsh@oleg103-client: oleg103-server: ssh exited with exit code 2