== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:15:41 (1788466541) start mds service on oleg210-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 3 03:55 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: kmemleak disabled oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: detected 4 online CPUs by sysfs oleg210-server: MODOPTS_LIBCFS= oleg210-server: Force libcfs to create 2 CPU partitions oleg210-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg210-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg210-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg210-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg210-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 105 (time 1788466637.50 total 10.98 last 9.57) - open/close 227 (time 1788466647.53 total 21.01 last 12.16) - open/close 485 (time 1788466658.55 total 32.02 last 23.42) - open/close 726 (time 1788466668.55 total 42.03 last 24.09) - open/close 826 (time 1788466678.57 total 52.04 last 9.98) - open/close 896 (time 1788466688.75 total 62.22 last 6.88) - open/close 945 (time 1788466700.04 total 73.51 last 4.34) - open/close 1147 (time 1788466710.98 total 84.46 last 18.46) - open/close 1360 (time 1788466721.45 total 94.92 last 20.36) - open/close 1567 (time 1788466733.88 total 107.35 last 16.65) - open/close 1664 (time 1788466744.50 total 117.97 last 9.13) - open/close 1824 (time 1788466755.38 total 128.85 last 14.71) - open/close 1963 (time 1788466765.86 total 139.34 last 13.26) - open/close 2168 (time 1788466775.87 total 149.34 last 20.48) - open/close 2301 (time 1788466787.63 total 161.11 last 11.31) - open/close 2429 (time 1788466798.52 total 172.00 last 11.75) - open/close 2548 (time 1788466808.57 total 182.04 last 11.84) - open/close 2719 (time 1788466818.58 total 192.06 last 17.07) - open/close 2858 (time 1788466829.25 total 202.73 last 13.03) - open/close 2952 (time 1788466839.70 total 213.18 last 8.99) - open/close 3093 (time 1788466849.71 total 223.19 last 14.09) - open/close 3194 (time 1788466860.35 total 233.82 last 9.50) - open/close 3294 (time 1788466871.42 total 244.89 last 9.03) - open/close 3495 (time 1788466881.43 total 254.90 last 20.08) - open/close 3678 (time 1788466891.93 total 265.40 last 17.43) - open/close 3857 (time 1788466904.95 total 278.43 last 13.74) - open/close 3984 (time 1788466916.14 total 289.62 last 11.35) - open/close 4101 (time 1788466928.97 total 302.45 last 9.12) - open/close 4287 (time 1788466938.99 total 312.47 last 18.56) - open/close 4472 (time 1788466949.04 total 322.51 last 18.42) - open/close 4701 (time 1788466959.85 total 333.32 last 21.18) - open/close 4848 (time 1788466970.23 total 343.71 last 14.15) - open/close 5133 (time 1788466980.28 total 353.76 last 28.36) - open/close 5333 (time 1788466990.57 total 364.05 last 19.44) - open/close 5487 (time 1788467000.58 total 374.05 last 15.40) - open/close 5668 (time 1788467010.69 total 384.17 last 17.89) - open/close 5893 (time 1788467021.61 total 395.09 last 20.60) - open/close 6086 (time 1788467032.93 total 406.41 last 17.05) - open/close 6209 (time 1788467042.98 total 416.45 last 12.24) - open/close 6374 (time 1788467052.98 total 426.46 last 16.50) - open/close 6590 (time 1788467062.98 total 436.46 last 21.60) - open/close 6775 (time 1788467073.86 total 447.34 last 17.00) - open/close 6997 (time 1788467084.39 total 457.87 last 21.09) - open/close 7287 (time 1788467095.28 total 468.75 last 26.64) - open/close 7507 (time 1788467106.72 total 480.20 last 19.22) - open/close 7760 (time 1788467116.73 total 490.21 last 25.27) - open/close 7926 (time 1788467126.96 total 500.44 last 16.23) - open/close 8118 (time 1788467137.98 total 511.46 last 17.42) - open/close 8340 (time 1788467148.13 total 521.61 last 21.87) - open/close 8552 (time 1788467158.14 total 531.62 last 21.19) - open/close 8730 (time 1788467168.15 total 541.63 last 17.78) - open/close 8950 (time 1788467178.16 total 551.64 last 21.98) - open/close 9205 (time 1788467188.26 total 561.74 last 25.26) - open/close 9433 (time 1788467198.76 total 572.24 last 21.71) - open/close 9655 (time 1788467209.24 total 582.72 last 21.18) - open/close 9904 (time 1788467219.26 total 592.73 last 24.86) - open/close 10000 (time 1788467226.53 total 600.01 last 13.20) - open/close 10226 (time 1788467236.54 total 610.02 last 22.57) - open/close 10394 (time 1788467246.90 total 620.37 last 16.22) - open/close 10578 (time 1788467256.90 total 630.38 last 18.39) - open/close 10783 (time 1788467267.49 total 640.97 last 19.36) - open/close 11062 (time 1788467277.50 total 650.98 last 27.87) - open/close 11298 (time 1788467288.36 total 661.83 last 21.75) - open/close 11585 (time 1788467299.17 total 672.65 last 26.53) - open/close 11795 (time 1788467309.61 total 683.09 last 20.12) - open/close 12055 (time 1788467320.05 total 693.53 last 24.91) - open/close 12291 (time 1788467330.09 total 703.56 last 23.51) - open/close 12487 (time 1788467340.54 total 714.02 last 18.74) - open/close 12687 (time 1788467350.60 total 724.07 last 19.89) - open/close 12916 (time 1788467361.32 total 734.79 last 21.36) - open/close 13164 (time 1788467371.76 total 745.24 last 23.74) - open/close 13424 (time 1788467381.76 total 755.24 last 26.00) - open/close 13661 (time 1788467391.79 total 765.27 last 23.63) - open/close 13893 (time 1788467402.38 total 775.86 last 21.91) - open/close 14111 (time 1788467412.84 total 786.32 last 20.84) - open/close 14363 (time 1788467423.26 total 796.74 last 24.18) - open/close 14580 (time 1788467433.52 total 806.99 last 21.16) - open/close 14866 (time 1788467443.52 total 816.99 last 28.60) - open/close 15093 (time 1788467454.41 total 827.88 last 20.85) - open/close 15307 (time 1788467464.42 total 837.89 last 21.38) - open/close 15505 (time 1788467474.92 total 848.40 last 18.84) - open/close 15747 (time 1788467484.93 total 858.40 last 24.19) - open/close 15917 (time 1788467495.68 total 869.16 last 15.81) - open/close 16153 (time 1788467505.68 total 879.16 last 23.59) - open/close 16321 (time 1788467516.39 total 889.87 last 15.69) - open/close 16458 (time 1788467526.88 total 900.36 last 13.06) - open/close 16639 (time 1788467537.69 total 911.17 last 16.74) - open/close 16841 (time 1788467547.72 total 921.20 last 20.15) - open/close 17024 (time 1788467557.92 total 931.40 last 17.94) - open/close 17662 (time 1788467567.93 total 941.41 last 63.74) - open/close 18418 (time 1788467577.93 total 951.41 last 75.58) - open/close 19179 (time 1788467587.94 total 961.42 last 76.07) - open/close 20000 (time 1788467597.66 total 971.14 last 84.43) - open/close 20793 (time 1788467607.68 total 981.16 last 79.16) - open/close 21622 (time 1788467617.68 total 991.16 last 82.89) - open/close 22512 (time 1788467627.68 total 1001.16 last 88.98) - open/close 23327 (time 1788467637.69 total 1011.17 last 81.45) - open/close 24168 (time 1788467647.73 total 1021.21 last 83.75) - open/close 24957 (time 1788467657.75 total 1031.23 last 78.76) - open/close 25742 (time 1788467667.76 total 1041.24 last 78.41) - open/close 26494 (time 1788467677.77 total 1051.25 last 75.14) - open/close 27133 (time 1788467687.78 total 1061.25 last 63.86) - open/close 27845 (time 1788467697.78 total 1071.26 last 71.19) - open/close 28549 (time 1788467707.79 total 1081.27 last 70.30) - open/close 29256 (time 1788467717.80 total 1091.28 last 70.65) - open/close 30000 (time 1788467727.68 total 1101.16 last 75.27) - open/close 30714 (time 1788467737.69 total 1111.17 last 71.35) - open/close 31476 (time 1788467747.69 total 1121.17 last 76.19) - open/close 32161 (time 1788467757.69 total 1131.17 last 68.49) - open/close 32810 (time 1788467767.70 total 1141.18 last 64.85) - open/close 33541 (time 1788467777.70 total 1151.18 last 73.08) - open/close 34307 (time 1788467787.71 total 1161.19 last 76.53) - open/close 35005 (time 1788467797.72 total 1171.20 last 69.75) - open/close 35708 (time 1788467807.73 total 1181.21 last 70.25) - open/close 36441 (time 1788467817.76 total 1191.23 last 73.09) - open/close 37146 (time 1788467827.76 total 1201.24 last 70.46) - open/close 37885 (time 1788467837.77 total 1211.24 last 73.87) - open/close 38660 (time 1788467847.78 total 1221.25 last 77.42) - open/close 39413 (time 1788467857.80 total 1231.28 last 75.10) - open/close 40000 (time 1788467865.56 total 1239.03 last 75.71) - open/close 40651 (time 1788467875.56 total 1249.04 last 65.10) - open/close 41368 (time 1788467885.58 total 1259.06 last 71.55) - open/close 41998 (time 1788467895.60 total 1269.07 last 62.90) - open/close 42610 (time 1788467905.62 total 1279.09 last 61.08) - open/close 43297 (time 1788467915.66 total 1289.13 last 68.41) - open/close 44004 (time 1788467925.67 total 1299.14 last 70.65) - open/close 44699 (time 1788467935.67 total 1309.15 last 69.44) - open/close 45486 (time 1788467945.68 total 1319.16 last 78.67) - open/close 46249 (time 1788467955.69 total 1329.16 last 76.25) - open/close 46962 (time 1788467965.70 total 1339.17 last 71.22) - open/close 47685 (time 1788467975.70 total 1349.18 last 72.24) - open/close 48421 (time 1788467985.72 total 1359.19 last 73.50) - open/close 49226 (time 1788467995.73 total 1369.21 last 80.41) - open/close 49969 (time 1788468005.74 total 1379.22 last 74.22) total: 50000 open/close in 1379.93 seconds: 36.23 ops/second - unlinked 0 (time 1788468010 ; total 0 ; last 0) - unlinked 10000 (time 1788468155 ; total 145 ; last 145) - unlinked 20000 (time 1788468313 ; total 303 ; last 158) - unlinked 30000 (time 1788468478 ; total 468 ; last 165) - unlinked 40000 (time 1788468633 ; total 623 ; last 155) total: 50000 unlinks in 772 seconds: 64.766838 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server unloading modules via unload_modules_local on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing unload_modules_local oleg210-server: kmemleak disabled modules unloaded. pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2