== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:41:25 (1788590485) start mds service on oleg631-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 4 11:05 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg631-server' oleg631-server: oleg631-server.virtnet: executing load_modules_local oleg631-server: kmemleak disabled oleg631-server: Loading modules from /home/green/git/lustre-release/lustre oleg631-server: detected 4 online CPUs by sysfs oleg631-server: MODOPTS_LIBCFS= oleg631-server: Force libcfs to create 2 CPU partitions oleg631-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg631-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg631-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg631-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg631-server: oleg631-server.virtnet: executing set_default_debug -1 all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg631-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg631-server: oleg631-server.virtnet: executing set_default_debug -1 all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg631-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug -1 all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0000 oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 270 (time 1788590591.29 total 10.57 last 25.55) - open/close 470 (time 1788590601.30 total 20.58 last 19.98) - open/close 655 (time 1788590611.30 total 30.58 last 18.50) - open/close 861 (time 1788590623.96 total 43.24 last 16.27) - open/close 1166 (time 1788590634.57 total 53.84 last 28.76) - open/close 1448 (time 1788590645.51 total 64.79 last 25.76) - open/close 1692 (time 1788590655.51 total 74.79 last 24.40) - open/close 1834 (time 1788590665.88 total 85.15 last 13.70) - open/close 1942 (time 1788590678.82 total 98.09 last 8.35) - open/close 2098 (time 1788590688.82 total 108.09 last 15.60) - open/close 2209 (time 1788590700.18 total 119.46 last 9.77) - open/close 2355 (time 1788590710.20 total 129.47 last 14.58) - open/close 2521 (time 1788590720.20 total 139.47 last 16.60) - open/close 2769 (time 1788590730.20 total 149.48 last 24.79) - open/close 3061 (time 1788590741.09 total 160.36 last 26.83) - open/close 3284 (time 1788590751.24 total 170.51 last 21.97) - open/close 3520 (time 1788590762.18 total 181.45 last 21.57) - open/close 3797 (time 1788590772.18 total 191.46 last 27.69) - open/close 4047 (time 1788590782.34 total 201.62 last 24.61) - open/close 4336 (time 1788590792.77 total 212.05 last 27.71) - open/close 4603 (time 1788590802.77 total 222.05 last 26.70) - open/close 4856 (time 1788590812.94 total 232.22 last 24.88) - open/close 5129 (time 1788590823.10 total 242.38 last 26.86) - open/close 5383 (time 1788590833.11 total 252.38 last 25.39) - open/close 5643 (time 1788590843.79 total 263.07 last 24.33) - open/close 5918 (time 1788590854.16 total 273.43 last 26.54) - open/close 6110 (time 1788590864.65 total 283.92 last 18.31) - open/close 6290 (time 1788590875.27 total 294.54 last 16.95) - open/close 6515 (time 1788590886.67 total 305.94 last 19.74) - open/close 6711 (time 1788590896.68 total 315.95 last 19.57) - open/close 6924 (time 1788590906.72 total 326.00 last 21.21) - open/close 7160 (time 1788590916.72 total 336.00 last 23.59) - open/close 7347 (time 1788590927.26 total 346.54 last 17.75) - open/close 7550 (time 1788590938.14 total 357.42 last 18.66) - open/close 7844 (time 1788590948.14 total 367.42 last 29.40) - open/close 8102 (time 1788590958.92 total 378.19 last 23.95) - open/close 8348 (time 1788590969.30 total 388.57 last 23.69) - open/close 8591 (time 1788590979.31 total 398.58 last 24.28) - open/close 8890 (time 1788590989.31 total 408.59 last 29.88) - open/close 9194 (time 1788590999.32 total 418.60 last 30.37) - open/close 9413 (time 1788591010.32 total 429.59 last 19.92) - open/close 9662 (time 1788591020.32 total 439.60 last 24.89) - open/close 9923 (time 1788591030.34 total 449.62 last 26.04) - open/close 10000 (time 1788591035.13 total 454.40 last 16.09) - open/close 10241 (time 1788591046.03 total 465.30 last 22.11) - open/close 10539 (time 1788591056.27 total 475.54 last 29.10) - open/close 10743 (time 1788591067.21 total 486.48 last 18.64) - open/close 10965 (time 1788591077.21 total 496.49 last 22.18) - open/close 11214 (time 1788591088.03 total 507.30 last 23.03) - open/close 11481 (time 1788591098.59 total 517.86 last 25.28) - open/close 11682 (time 1788591111.31 total 530.59 last 15.79) - open/close 11948 (time 1788591123.02 total 542.29 last 22.73) - open/close 12216 (time 1788591133.21 total 552.49 last 26.29) - open/close 12497 (time 1788591143.22 total 562.50 last 28.07) - open/close 12750 (time 1788591153.23 total 572.51 last 25.27) - open/close 13007 (time 1788591163.24 total 582.52 last 25.67) - open/close 13299 (time 1788591173.25 total 592.52 last 29.19) - open/close 13577 (time 1788591183.57 total 602.84 last 26.94) - open/close 13803 (time 1788591194.18 total 613.46 last 21.29) - open/close 14040 (time 1788591204.19 total 623.46 last 23.68) - open/close 14255 (time 1788591214.20 total 633.47 last 21.48) - open/close 14357 (time 1788591224.41 total 643.68 last 9.99) - open/close 14570 (time 1788591234.90 total 654.18 last 20.29) - open/close 14833 (time 1788591247.57 total 666.84 last 20.77) - open/close 15048 (time 1788591259.48 total 678.75 last 18.05) - open/close 15369 (time 1788591269.87 total 689.15 last 30.88) - open/close 15627 (time 1788591280.79 total 700.07 last 23.63) - open/close 15842 (time 1788591291.59 total 710.87 last 19.91) - open/close 16108 (time 1788591301.78 total 721.06 last 26.11) - open/close 16361 (time 1788591311.78 total 731.06 last 25.29) - open/close 16648 (time 1788591321.79 total 741.06 last 28.69) - open/close 16950 (time 1788591331.81 total 751.08 last 30.14) - open/close 17217 (time 1788591341.81 total 761.09 last 26.69) - open/close 17435 (time 1788591352.58 total 771.86 last 20.24) - open/close 17707 (time 1788591365.75 total 785.02 last 20.66) - open/close 17827 (time 1788591376.33 total 795.61 last 11.34) - open/close 17967 (time 1788591387.01 total 806.29 last 13.11) - open/close 18088 (time 1788591397.02 total 816.30 last 12.08) - open/close 18337 (time 1788591407.32 total 826.59 last 24.19) - open/close 18634 (time 1788591417.89 total 837.16 last 28.09) - open/close 18870 (time 1788591428.03 total 847.31 last 23.27) - open/close 19163 (time 1788591438.93 total 858.21 last 26.88) - open/close 19381 (time 1788591448.94 total 868.22 last 21.78) - open/close 19569 (time 1788591458.96 total 878.24 last 18.76) - open/close 19755 (time 1788591469.53 total 888.81 last 17.60) - open/close 20000 (time 1788591478.03 total 897.30 last 28.83) - open/close 20250 (time 1788591488.45 total 907.73 last 23.98) - open/close 20455 (time 1788591499.34 total 918.61 last 18.84) - open/close 20668 (time 1788591509.51 total 928.79 last 20.93) - open/close 20855 (time 1788591520.17 total 939.45 last 17.55) - open/close 21089 (time 1788591530.84 total 950.11 last 21.94) - open/close 21353 (time 1788591540.84 total 960.11 last 26.39) - open/close 22206 (time 1788591550.87 total 970.15 last 85.02) - open/close 23111 (time 1788591560.89 total 980.16 last 90.39) - open/close 23980 (time 1788591570.89 total 990.17 last 86.86) - open/close 24857 (time 1788591580.89 total 1000.17 last 87.68) - open/close 25766 (time 1788591590.89 total 1010.17 last 90.88) - open/close 26656 (time 1788591600.90 total 1020.18 last 88.92) - open/close 27581 (time 1788591610.91 total 1030.19 last 92.42) - open/close 28433 (time 1788591620.92 total 1040.19 last 85.14) - open/close 29353 (time 1788591630.92 total 1050.20 last 91.96) - open/close 30000 (time 1788591638.73 total 1058.01 last 82.84) - open/close 30842 (time 1788591648.75 total 1068.03 last 84.06) - open/close 31676 (time 1788591658.75 total 1078.03 last 83.39) - open/close 32478 (time 1788591668.76 total 1088.03 last 80.16) - open/close 33319 (time 1788591678.78 total 1098.05 last 83.92) - open/close 34126 (time 1788591688.79 total 1108.06 last 80.65) - open/close 34905 (time 1788591698.80 total 1118.07 last 77.80) - open/close 35772 (time 1788591708.80 total 1128.07 last 86.68) - open/close 36645 (time 1788591718.80 total 1138.08 last 87.27) - open/close 37508 (time 1788591728.81 total 1148.08 last 86.24) - open/close 38347 (time 1788591738.81 total 1158.09 last 83.87) - open/close 39051 (time 1788591748.84 total 1168.11 last 70.23) - open/close 39909 (time 1788591758.84 total 1178.12 last 85.77) - open/close 40832 (time 1788591768.86 total 1188.14 last 92.10) - open/close 41714 (time 1788591778.87 total 1198.15 last 88.09) - open/close 42633 (time 1788591788.87 total 1208.15 last 91.89) - open/close 43493 (time 1788591798.88 total 1218.15 last 85.97) - open/close 44360 (time 1788591808.88 total 1228.16 last 86.66) - open/close 45255 (time 1788591818.89 total 1238.16 last 89.46) - open/close 46136 (time 1788591828.89 total 1248.17 last 88.05) - open/close 46823 (time 1788591838.90 total 1258.18 last 68.65) - open/close 47707 (time 1788591848.90 total 1268.18 last 88.39) - open/close 48562 (time 1788591858.91 total 1278.19 last 85.41) - open/close 49470 (time 1788591868.93 total 1288.21 last 90.63) total: 50000 open/close in 1293.31 seconds: 38.66 ops/second - unlinked 0 (time 1788591878 ; total 0 ; last 0) - unlinked 10000 (time 1788592006 ; total 128 ; last 128) - unlinked 20000 (time 1788592131 ; total 253 ; last 125) - unlinked 30000 (time 1788592266 ; total 388 ; last 135) - unlinked 40000 (time 1788592395 ; total 517 ; last 129) total: 50000 unlinks in 641 seconds: 78.003120 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg631-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg631-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg631-server stop mds service on oleg631-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg631-server stop mds service on oleg631-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg631-server unloading modules via unload_modules_local on: 'oleg631-server' oleg631-server: oleg631-server.virtnet: executing unload_modules_local oleg631-server: kmemleak disabled modules unloaded. pdsh@oleg631-client: oleg631-client: ssh exited with exit code 2 pdsh@oleg631-client: oleg631-server: ssh exited with exit code 2 pdsh@oleg631-client: oleg631-server: ssh exited with exit code 2 pdsh@oleg631-client: oleg631-client: ssh exited with exit code 2