== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:00:28 (1788645628) start mds service on oleg631-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 4 18:25 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg631-server' oleg631-server: oleg631-server.virtnet: executing load_modules_local oleg631-server: kmemleak disabled oleg631-server: Loading modules from /home/green/git/lustre-release/lustre oleg631-server: detected 4 online CPUs by sysfs oleg631-server: MODOPTS_LIBCFS= oleg631-server: Force libcfs to create 2 CPU partitions oleg631-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg631-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg631-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg631-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg631-server: oleg631-server.virtnet: executing set_default_debug -1 all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg631-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug -1 all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0000 oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 181 (time 1788645711.47 total 10.44 last 17.34) - open/close 427 (time 1788645722.21 total 21.18 last 22.90) - open/close 601 (time 1788645732.40 total 31.36 last 17.09) - open/close 802 (time 1788645742.49 total 41.46 last 19.92) - open/close 1003 (time 1788645752.55 total 51.52 last 19.97) - open/close 1225 (time 1788645763.30 total 62.27 last 20.66) - open/close 1448 (time 1788645774.19 total 73.16 last 20.48) - open/close 1702 (time 1788645784.19 total 83.16 last 25.40) - open/close 1876 (time 1788645795.05 total 94.02 last 16.02) - open/close 2146 (time 1788645805.06 total 104.03 last 26.97) - open/close 2318 (time 1788645815.07 total 114.04 last 17.19) - open/close 2465 (time 1788645825.44 total 124.41 last 14.17) - open/close 2737 (time 1788645835.46 total 134.43 last 27.15) - open/close 2885 (time 1788645846.51 total 145.48 last 13.39) - open/close 2988 (time 1788645857.88 total 156.85 last 9.06) - open/close 3170 (time 1788645867.89 total 166.86 last 18.19) - open/close 3419 (time 1788645878.11 total 177.08 last 24.35) - open/close 3618 (time 1788645888.59 total 187.56 last 18.99) - open/close 3712 (time 1788645899.31 total 198.28 last 8.77) - open/close 3820 (time 1788645910.47 total 209.44 last 9.67) - open/close 4048 (time 1788645921.23 total 220.20 last 21.19) - open/close 4201 (time 1788645931.25 total 230.22 last 15.27) - open/close 4351 (time 1788645942.05 total 241.02 last 13.88) - open/close 4452 (time 1788645952.06 total 251.02 last 10.10) - open/close 4573 (time 1788645962.79 total 261.76 last 11.27) - open/close 4693 (time 1788645973.31 total 272.28 last 11.40) - open/close 4892 (time 1788645983.33 total 282.29 last 19.87) - open/close 4989 (time 1788645993.34 total 292.31 last 9.69) - open/close 5188 (time 1788646003.35 total 302.31 last 19.89) - open/close 5428 (time 1788646013.36 total 312.33 last 23.96) - open/close 5654 (time 1788646023.38 total 322.35 last 22.56) - open/close 5866 (time 1788646033.38 total 332.35 last 21.19) - open/close 6090 (time 1788646043.38 total 342.35 last 22.40) - open/close 6320 (time 1788646053.60 total 352.57 last 22.51) - open/close 6571 (time 1788646063.60 total 362.57 last 25.10) - open/close 6812 (time 1788646074.50 total 373.47 last 22.11) - open/close 7062 (time 1788646085.43 total 384.40 last 22.88) - open/close 7310 (time 1788646095.84 total 394.81 last 23.82) - open/close 7569 (time 1788646106.84 total 405.81 last 23.54) - open/close 7803 (time 1788646116.85 total 415.82 last 23.39) - open/close 8069 (time 1788646127.53 total 426.50 last 24.91) - open/close 8317 (time 1788646138.15 total 437.12 last 23.36) - open/close 8521 (time 1788646148.16 total 447.13 last 20.37) - open/close 8730 (time 1788646158.24 total 457.20 last 20.75) - open/close 8924 (time 1788646169.38 total 468.34 last 17.41) - open/close 9154 (time 1788646180.21 total 479.18 last 21.23) - open/close 9388 (time 1788646190.22 total 489.18 last 23.39) - open/close 9557 (time 1788646200.75 total 499.71 last 16.05) - open/close 9697 (time 1788646211.71 total 510.68 last 12.77) - open/close 9879 (time 1788646221.73 total 520.70 last 18.16) - open/close 10000 (time 1788646227.57 total 526.54 last 20.73) - open/close 10110 (time 1788646238.05 total 537.01 last 10.50) - open/close 10323 (time 1788646248.07 total 547.04 last 21.25) - open/close 10473 (time 1788646258.33 total 557.30 last 14.61) - open/close 10622 (time 1788646268.65 total 567.61 last 14.45) - open/close 10720 (time 1788646279.71 total 578.68 last 8.86) - open/close 10864 (time 1788646289.72 total 588.69 last 14.39) - open/close 10957 (time 1788646300.15 total 599.12 last 8.92) - open/close 11069 (time 1788646312.22 total 611.19 last 9.28) - open/close 11257 (time 1788646322.40 total 621.37 last 18.47) - open/close 11515 (time 1788646332.53 total 631.50 last 25.46) - open/close 11669 (time 1788646342.55 total 641.52 last 15.38) - open/close 11862 (time 1788646353.40 total 652.37 last 17.78) - open/close 11942 (time 1788646363.69 total 662.66 last 7.77) - open/close 12178 (time 1788646373.99 total 672.96 last 22.92) - open/close 12414 (time 1788646384.55 total 683.52 last 22.33) - open/close 12650 (time 1788646395.42 total 694.39 last 21.72) - open/close 12877 (time 1788646405.69 total 704.66 last 22.11) - open/close 13084 (time 1788646415.85 total 714.82 last 20.37) - open/close 13260 (time 1788646425.87 total 724.84 last 17.57) - open/close 13462 (time 1788646436.79 total 735.76 last 18.50) - open/close 13600 (time 1788646446.80 total 745.77 last 13.78) - open/close 13794 (time 1788646456.83 total 755.80 last 19.35) - open/close 13937 (time 1788646467.70 total 766.67 last 13.15) - open/close 14024 (time 1788646477.76 total 776.73 last 8.65) - open/close 14223 (time 1788646488.55 total 787.52 last 18.44) - open/close 14503 (time 1788646499.48 total 798.45 last 25.61) - open/close 14730 (time 1788646510.07 total 809.04 last 21.45) - open/close 14994 (time 1788646521.06 total 820.03 last 24.01) - open/close 15147 (time 1788646531.37 total 830.34 last 14.85) - open/close 15397 (time 1788646542.24 total 841.21 last 23.00) - open/close 15648 (time 1788646552.95 total 851.92 last 23.44) - open/close 15915 (time 1788646564.01 total 862.98 last 24.14) - open/close 16139 (time 1788646574.25 total 873.22 last 21.86) - open/close 16388 (time 1788646584.48 total 883.45 last 24.34) - open/close 16612 (time 1788646594.66 total 893.63 last 22.02) - open/close 16853 (time 1788646605.38 total 904.35 last 22.47) - open/close 17054 (time 1788646615.39 total 914.36 last 20.07) - open/close 17319 (time 1788646626.08 total 925.04 last 24.81) - open/close 17564 (time 1788646636.91 total 935.88 last 22.62) - open/close 17823 (time 1788646647.60 total 946.56 last 24.23) - open/close 18030 (time 1788646658.03 total 957.00 last 19.83) - open/close 18701 (time 1788646668.04 total 967.01 last 67.03) - open/close 19410 (time 1788646678.05 total 977.01 last 70.88) - open/close 20000 (time 1788646686.57 total 985.53 last 69.25) - open/close 20729 (time 1788646696.57 total 995.54 last 72.85) - open/close 21455 (time 1788646706.58 total 1005.55 last 72.53) - open/close 22161 (time 1788646716.59 total 1015.56 last 70.56) - open/close 22874 (time 1788646726.59 total 1025.56 last 71.29) - open/close 23631 (time 1788646736.59 total 1035.56 last 75.69) - open/close 24410 (time 1788646746.60 total 1045.57 last 77.80) - open/close 25179 (time 1788646756.61 total 1055.58 last 76.84) - open/close 25925 (time 1788646766.62 total 1065.58 last 74.56) - open/close 26676 (time 1788646776.62 total 1075.59 last 75.09) - open/close 27470 (time 1788646786.62 total 1085.59 last 79.35) - open/close 28233 (time 1788646796.63 total 1095.60 last 76.24) - open/close 28989 (time 1788646806.64 total 1105.61 last 75.55) - open/close 29760 (time 1788646816.65 total 1115.62 last 77.03) - open/close 30000 (time 1788646819.91 total 1118.88 last 73.66) - open/close 30750 (time 1788646829.92 total 1128.89 last 74.91) - open/close 31495 (time 1788646839.92 total 1138.89 last 74.47) - open/close 32297 (time 1788646849.92 total 1148.89 last 80.19) - open/close 33072 (time 1788646859.94 total 1158.91 last 77.39) - open/close 33904 (time 1788646869.94 total 1168.91 last 83.20) - open/close 34658 (time 1788646879.94 total 1178.91 last 75.38) - open/close 35269 (time 1788646889.95 total 1188.92 last 61.06) - open/close 35933 (time 1788646899.95 total 1198.92 last 66.36) - open/close 36580 (time 1788646909.97 total 1208.93 last 64.62) - open/close 37278 (time 1788646919.98 total 1218.95 last 69.71) - open/close 37981 (time 1788646929.98 total 1228.95 last 70.28) - open/close 38577 (time 1788646939.99 total 1238.96 last 59.56) - open/close 39303 (time 1788646950.00 total 1248.97 last 72.53) - open/close 40000 (time 1788646959.15 total 1258.12 last 76.19) - open/close 40778 (time 1788646969.17 total 1268.14 last 77.61) - open/close 41506 (time 1788646979.20 total 1278.17 last 72.57) - open/close 42295 (time 1788646989.22 total 1288.19 last 78.79) - open/close 43024 (time 1788646999.24 total 1298.21 last 72.74) - open/close 43784 (time 1788647009.24 total 1308.21 last 75.99) - open/close 44543 (time 1788647019.24 total 1318.21 last 75.89) - open/close 45287 (time 1788647029.25 total 1328.22 last 74.34) - open/close 45991 (time 1788647039.25 total 1338.22 last 70.38) - open/close 46752 (time 1788647049.26 total 1348.22 last 76.08) - open/close 47528 (time 1788647059.26 total 1358.23 last 77.56) - open/close 48234 (time 1788647069.27 total 1368.24 last 70.52) - open/close 48864 (time 1788647079.29 total 1378.25 last 62.91) - open/close 49585 (time 1788647089.30 total 1388.27 last 72.01) total: 50000 open/close in 1393.99 seconds: 35.87 ops/second - unlinked 0 (time 1788647099 ; total 0 ; last 0) - unlinked 10000 (time 1788647240 ; total 141 ; last 141) - unlinked 20000 (time 1788647388 ; total 289 ; last 148) - unlinked 30000 (time 1788647541 ; total 442 ; last 153) - unlinked 40000 (time 1788647691 ; total 592 ; last 150) total: 50000 unlinks in 731 seconds: 68.399452 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg631-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg631-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg631-server stop mds service on oleg631-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg631-server unloading modules via unload_modules_local on: 'oleg631-server' oleg631-server: oleg631-server.virtnet: executing unload_modules_local oleg631-server: kmemleak disabled modules unloaded. pdsh@oleg631-client: oleg631-client: ssh exited with exit code 2 pdsh@oleg631-client: oleg631-server: ssh exited with exit code 2 pdsh@oleg631-client: oleg631-client: ssh exited with exit code 2 pdsh@oleg631-client: oleg631-server: ssh exited with exit code 2