== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:33:17 (1790659997) start mds service on oleg653-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1940568 Sep 28 12:40 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg653-server' oleg653-server: oleg653-server.virtnet: executing load_modules_local oleg653-server: kmemleak disabled oleg653-server: Loading modules from /home/green/git/lustre-release/lustre oleg653-server: detected 4 online CPUs by sysfs oleg653-server: MODOPTS_LIBCFS= oleg653-server: Force libcfs to create 2 CPU partitions oleg653-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg653-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg653-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg653-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg653-server: oleg653-server.virtnet: executing set_default_debug -1 all pdsh@oleg653-client: oleg653-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg653-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg653-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg653-server: oleg653-server.virtnet: executing set_default_debug -1 all pdsh@oleg653-client: oleg653-server: ssh exited with exit code 1 Started lustre-OST0000 oleg653-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg653-client.virtnet: -o user_xattr,flock 192.168.206.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 218 (time 1790660085.43 total 10.01 last 21.78) - open/close 445 (time 1790660095.45 total 20.02 last 22.67) - open/close 660 (time 1790660105.93 total 30.51 last 20.50) - open/close 898 (time 1790660115.93 total 40.51 last 23.80) - open/close 1139 (time 1790660125.94 total 50.51 last 24.09) - open/close 1390 (time 1790660136.47 total 61.04 last 23.84) - open/close 1601 (time 1790660146.55 total 71.12 last 20.93) - open/close 1852 (time 1790660156.56 total 81.13 last 25.08) - open/close 1998 (time 1790660166.57 total 91.14 last 14.59) - open/close 2219 (time 1790660177.37 total 101.94 last 20.46) - open/close 2380 (time 1790660187.50 total 112.07 last 15.89) - open/close 2544 (time 1790660197.50 total 122.08 last 16.39) - open/close 2754 (time 1790660208.22 total 132.80 last 19.59) - open/close 2902 (time 1790660218.51 total 143.08 last 14.39) - open/close 3130 (time 1790660230.04 total 154.61 last 19.78) - open/close 3346 (time 1790660240.05 total 164.63 last 21.56) - open/close 3538 (time 1790660250.71 total 175.28 last 18.02) - open/close 3793 (time 1790660261.23 total 185.80 last 24.25) - open/close 4020 (time 1790660271.51 total 196.08 last 22.08) - open/close 4261 (time 1790660281.51 total 206.09 last 24.09) - open/close 4482 (time 1790660291.52 total 216.10 last 22.07) - open/close 4703 (time 1790660302.11 total 226.68 last 20.88) - open/close 4929 (time 1790660312.75 total 237.33 last 21.23) - open/close 5194 (time 1790660322.76 total 247.34 last 26.48) - open/close 5414 (time 1790660333.41 total 257.99 last 20.66) - open/close 5596 (time 1790660343.42 total 268.00 last 18.18) - open/close 5792 (time 1790660353.42 total 278.00 last 19.60) - open/close 6030 (time 1790660363.43 total 288.00 last 23.78) - open/close 6226 (time 1790660375.82 total 300.39 last 15.82) - open/close 6368 (time 1790660385.88 total 310.46 last 14.11) - open/close 6582 (time 1790660396.57 total 321.15 last 20.02) - open/close 6773 (time 1790660406.59 total 331.17 last 19.06) - open/close 6986 (time 1790660417.62 total 342.19 last 19.32) - open/close 7223 (time 1790660427.62 total 352.20 last 23.68) - open/close 7409 (time 1790660438.52 total 363.10 last 17.06) - open/close 7582 (time 1790660449.05 total 373.62 last 16.44) - open/close 7661 (time 1790660459.47 total 384.05 last 7.58) - open/close 7858 (time 1790660469.49 total 394.06 last 19.68) - open/close 8063 (time 1790660479.50 total 404.08 last 20.47) - open/close 8235 (time 1790660489.55 total 414.13 last 17.11) - open/close 8400 (time 1790660500.00 total 424.57 last 15.79) - open/close 8502 (time 1790660510.42 total 435.00 last 9.78) - open/close 8706 (time 1790660520.42 total 445.00 last 20.40) - open/close 8907 (time 1790660531.49 total 456.06 last 18.17) - open/close 9125 (time 1790660541.66 total 466.24 last 21.42) - open/close 9365 (time 1790660552.30 total 476.87 last 22.56) - open/close 9663 (time 1790660562.50 total 487.08 last 29.20) - open/close 9903 (time 1790660572.50 total 497.08 last 24.00) - open/close 10000 (time 1790660577.50 total 502.08 last 19.41) - open/close 10232 (time 1790660587.51 total 512.08 last 23.19) - open/close 10397 (time 1790660597.55 total 522.13 last 16.43) - open/close 10583 (time 1790660608.59 total 533.16 last 16.85) - open/close 10739 (time 1790660619.06 total 543.64 last 14.89) - open/close 10894 (time 1790660629.07 total 553.64 last 15.49) - open/close 10976 (time 1790660639.19 total 563.77 last 8.10) - open/close 11151 (time 1790660649.20 total 573.78 last 17.49) - open/close 11366 (time 1790660659.21 total 583.78 last 21.49) - open/close 11627 (time 1790660669.94 total 594.52 last 24.31) - open/close 11945 (time 1790660680.91 total 605.48 last 29.01) - open/close 12198 (time 1790660690.93 total 615.50 last 25.25) - open/close 12398 (time 1790660700.94 total 625.52 last 19.97) - open/close 12676 (time 1790660710.94 total 635.52 last 27.79) - open/close 12892 (time 1790660722.97 total 647.55 last 17.96) - open/close 13141 (time 1790660734.05 total 658.62 last 22.48) - open/close 13356 (time 1790660744.67 total 669.24 last 20.24) - open/close 13592 (time 1790660754.67 total 679.25 last 23.60) - open/close 13666 (time 1790660765.41 total 689.99 last 6.89) - open/close 13905 (time 1790660776.47 total 701.04 last 21.62) - open/close 14160 (time 1790660786.51 total 711.08 last 25.40) - open/close 14436 (time 1790660796.53 total 721.10 last 27.54) - open/close 14674 (time 1790660806.53 total 731.11 last 23.79) - open/close 14932 (time 1790660817.17 total 741.75 last 24.24) - open/close 15169 (time 1790660827.18 total 751.76 last 23.69) - open/close 15402 (time 1790660837.74 total 762.32 last 22.06) - open/close 15622 (time 1790660847.89 total 772.47 last 21.68) - open/close 15848 (time 1790660858.23 total 782.81 last 21.85) - open/close 16035 (time 1790660868.81 total 793.39 last 17.67) - open/close 16246 (time 1790660878.82 total 803.40 last 21.08) - open/close 16423 (time 1790660889.37 total 813.94 last 16.79) - open/close 16613 (time 1790660901.66 total 826.24 last 15.45) - open/close 16838 (time 1790660911.97 total 836.54 last 21.84) - open/close 17048 (time 1790660923.23 total 847.81 last 18.64) - open/close 17316 (time 1790660934.18 total 858.75 last 24.49) - open/close 17573 (time 1790660944.19 total 868.76 last 25.67) - open/close 17820 (time 1790660954.19 total 878.77 last 24.68) - open/close 18072 (time 1790660965.14 total 889.71 last 23.02) - open/close 18234 (time 1790660975.67 total 900.24 last 15.39) - open/close 18446 (time 1790660985.68 total 910.26 last 21.17) - open/close 18606 (time 1790660995.95 total 920.52 last 15.59) - open/close 19284 (time 1790661005.95 total 930.53 last 67.79) - open/close 20000 (time 1790661015.26 total 939.83 last 76.94) - open/close 20720 (time 1790661025.27 total 949.84 last 71.94) - open/close 21376 (time 1790661035.27 total 959.85 last 65.56) - open/close 21984 (time 1790661045.27 total 969.85 last 60.80) - open/close 22626 (time 1790661055.28 total 979.85 last 64.16) - open/close 23289 (time 1790661065.29 total 989.86 last 66.25) - open/close 23964 (time 1790661075.31 total 999.88 last 67.35) - open/close 24686 (time 1790661085.32 total 1009.90 last 72.10) - open/close 25408 (time 1790661095.33 total 1019.90 last 72.17) - open/close 26054 (time 1790661105.33 total 1029.91 last 64.56) - open/close 26724 (time 1790661115.34 total 1039.91 last 66.98) - open/close 27397 (time 1790661125.35 total 1049.93 last 67.20) - open/close 28044 (time 1790661135.36 total 1059.93 last 64.65) - open/close 28717 (time 1790661145.38 total 1069.95 last 67.19) - open/close 29364 (time 1790661155.40 total 1079.97 last 64.57) - open/close 29984 (time 1790661165.41 total 1089.98 last 61.94) - open/close 30659 (time 1790661175.42 total 1099.99 last 67.43) - open/close 31321 (time 1790661185.42 total 1109.99 last 66.19) - open/close 31991 (time 1790661195.43 total 1120.00 last 66.93) - open/close 32643 (time 1790661205.43 total 1130.01 last 65.19) - open/close 33342 (time 1790661215.44 total 1140.01 last 69.86) - open/close 34071 (time 1790661225.45 total 1150.02 last 72.81) - open/close 34754 (time 1790661235.47 total 1160.04 last 68.16) - open/close 35451 (time 1790661245.47 total 1170.05 last 69.65) - open/close 36167 (time 1790661255.48 total 1180.05 last 71.57) - open/close 36859 (time 1790661265.48 total 1190.06 last 69.17) - open/close 37509 (time 1790661275.50 total 1200.08 last 64.90) - open/close 38040 (time 1790661285.53 total 1210.10 last 52.96) - open/close 38668 (time 1790661295.54 total 1220.12 last 62.69) - open/close 39356 (time 1790661305.55 total 1230.13 last 68.74) - open/close 40000 (time 1790661314.95 total 1239.52 last 68.56) - open/close 40725 (time 1790661324.96 total 1249.54 last 72.36) - open/close 41369 (time 1790661334.97 total 1259.55 last 64.34) - open/close 42107 (time 1790661344.98 total 1269.55 last 73.76) - open/close 42800 (time 1790661354.98 total 1279.55 last 69.29) - open/close 43463 (time 1790661364.99 total 1289.56 last 66.25) - open/close 44225 (time 1790661374.99 total 1299.57 last 76.14) - open/close 44886 (time 1790661385.00 total 1309.57 last 66.08) - open/close 45565 (time 1790661395.01 total 1319.59 last 67.79) - open/close 46320 (time 1790661405.08 total 1329.65 last 75.03) - open/close 47022 (time 1790661415.09 total 1339.66 last 70.13) - open/close 47714 (time 1790661425.13 total 1349.70 last 68.91) - open/close 48439 (time 1790661435.13 total 1359.71 last 72.46) - open/close 49121 (time 1790661445.14 total 1369.72 last 68.14) - open/close 49707 (time 1790661455.16 total 1379.73 last 58.52) total: 50000 open/close in 1384.50 seconds: 36.11 ops/second - unlinked 0 (time 1790661464 ; total 0 ; last 0) - unlinked 10000 (time 1790661622 ; total 158 ; last 158) - unlinked 20000 (time 1790661779 ; total 315 ; last 157) - unlinked 30000 (time 1790661929 ; total 465 ; last 150) - unlinked 40000 (time 1790662078 ; total 614 ; last 149) total: 50000 unlinks in 751 seconds: 66.577896 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg653-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg653-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg653-server stop mds service on oleg653-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg653-server unloading modules via unload_modules_local on: 'oleg653-server' oleg653-server: oleg653-server.virtnet: executing unload_modules_local oleg653-server: kmemleak disabled modules unloaded. pdsh@oleg653-client: oleg653-client: ssh exited with exit code 2 pdsh@oleg653-client: oleg653-server: ssh exited with exit code 2 pdsh@oleg653-client: oleg653-client: ssh exited with exit code 2 pdsh@oleg653-client: oleg653-server: ssh exited with exit code 2