== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:24:37 (1788877477) start mds service on oleg115-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 8 06:56 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg115-server' oleg115-server: oleg115-server.virtnet: executing load_modules_local oleg115-server: kmemleak disabled oleg115-server: Loading modules from /home/green/git/lustre-release/lustre oleg115-server: detected 4 online CPUs by sysfs oleg115-server: MODOPTS_LIBCFS= oleg115-server: Force libcfs to create 2 CPU partitions oleg115-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg115-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg115-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg115-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg115-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg115-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg115-server: oleg115-server.virtnet: executing set_default_debug -1 all pdsh@oleg115-client: oleg115-server: ssh exited with exit code 1 Started lustre-OST0000 oleg115-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg115-client.virtnet: -o user_xattr,flock 192.168.201.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 215 (time 1788877532.68 total 10.37 last 20.72) - open/close 486 (time 1788877542.70 total 20.39 last 27.06) - open/close 741 (time 1788877552.71 total 30.41 last 25.46) - open/close 925 (time 1788877562.85 total 40.54 last 18.16) - open/close 1180 (time 1788877572.86 total 50.55 last 25.48) - open/close 1331 (time 1788877582.87 total 60.57 last 15.07) - open/close 1525 (time 1788877593.26 total 70.95 last 18.68) - open/close 1632 (time 1788877603.29 total 80.98 last 10.67) - open/close 1792 (time 1788877613.37 total 91.07 last 15.87) - open/close 1934 (time 1788877623.42 total 101.12 last 14.13) - open/close 2076 (time 1788877634.05 total 111.74 last 13.37) - open/close 2238 (time 1788877644.30 total 121.99 last 15.81) - open/close 2356 (time 1788877655.97 total 133.66 last 10.11) - open/close 2575 (time 1788877666.15 total 143.84 last 21.52) - open/close 2827 (time 1788877676.15 total 153.84 last 25.20) - open/close 3020 (time 1788877686.16 total 163.85 last 19.28) - open/close 3315 (time 1788877696.17 total 173.86 last 29.48) - open/close 3458 (time 1788877706.47 total 184.16 last 13.88) - open/close 3724 (time 1788877716.90 total 194.59 last 25.51) - open/close 3871 (time 1788877727.20 total 204.89 last 14.28) - open/close 4034 (time 1788877737.94 total 215.63 last 15.18) - open/close 4219 (time 1788877750.10 total 227.80 last 15.20) - open/close 4472 (time 1788877763.09 total 240.78 last 19.48) - open/close 4613 (time 1788877775.24 total 252.93 last 11.60) - open/close 4824 (time 1788877786.66 total 264.35 last 18.48) - open/close 5086 (time 1788877796.67 total 274.36 last 26.19) - open/close 5318 (time 1788877806.67 total 284.36 last 23.19) - open/close 5560 (time 1788877817.75 total 295.45 last 21.84) - open/close 5725 (time 1788877832.13 total 309.82 last 11.48) - open/close 5929 (time 1788877842.14 total 319.84 last 20.36) - open/close 6127 (time 1788877853.01 total 330.70 last 18.22) - open/close 6264 (time 1788877863.04 total 340.73 last 13.66) - open/close 6407 (time 1788877874.09 total 351.79 last 12.93) - open/close 6614 (time 1788877884.13 total 361.82 last 20.63) - open/close 6734 (time 1788877894.60 total 372.29 last 11.46) - open/close 6978 (time 1788877905.57 total 383.26 last 22.24) - open/close 7122 (time 1788877915.80 total 393.50 last 14.06) - open/close 7308 (time 1788877925.82 total 403.52 last 18.57) - open/close 7522 (time 1788877935.92 total 413.61 last 21.20) - open/close 7811 (time 1788877945.92 total 423.61 last 28.89) - open/close 8039 (time 1788877956.50 total 434.19 last 21.56) - open/close 8266 (time 1788877966.79 total 444.49 last 22.05) - open/close 8523 (time 1788877977.83 total 455.52 last 23.30) - open/close 8720 (time 1788877988.81 total 466.50 last 17.93) - open/close 8860 (time 1788877998.82 total 476.51 last 13.99) - open/close 9125 (time 1788878009.75 total 487.44 last 24.25) - open/close 9390 (time 1788878019.78 total 497.48 last 26.41) - open/close 9598 (time 1788878029.78 total 507.48 last 20.80) - open/close 9822 (time 1788878039.79 total 517.49 last 22.38) - open/close 10000 (time 1788878047.87 total 525.57 last 22.03) - open/close 10244 (time 1788878058.31 total 536.01 last 23.37) - open/close 10561 (time 1788878068.81 total 546.50 last 30.20) - open/close 10834 (time 1788878079.31 total 557.01 last 25.99) - open/close 11096 (time 1788878089.35 total 567.04 last 26.11) - open/close 11282 (time 1788878100.04 total 577.73 last 17.39) - open/close 11560 (time 1788878111.07 total 588.76 last 25.20) - open/close 11727 (time 1788878123.67 total 601.37 last 13.25) - open/close 11959 (time 1788878134.93 total 612.62 last 20.61) - open/close 12138 (time 1788878144.94 total 622.64 last 17.88) - open/close 12346 (time 1788878154.95 total 632.65 last 20.78) - open/close 12603 (time 1788878165.91 total 643.60 last 23.47) - open/close 12823 (time 1788878176.41 total 654.10 last 20.95) - open/close 13033 (time 1788878186.52 total 664.21 last 20.77) - open/close 13226 (time 1788878196.80 total 674.50 last 18.77) - open/close 13381 (time 1788878206.81 total 684.50 last 15.49) - open/close 13665 (time 1788878216.82 total 694.51 last 28.38) - open/close 13952 (time 1788878227.34 total 705.03 last 27.29) - open/close 14400 (time 1788878237.34 total 715.03 last 44.77) - open/close 14561 (time 1788878248.58 total 726.28 last 14.32) - open/close 14884 (time 1788878259.00 total 736.70 last 31.00) - open/close 15101 (time 1788878269.76 total 747.46 last 20.17) - open/close 15304 (time 1788878280.58 total 758.27 last 18.77) - open/close 15544 (time 1788878291.58 total 769.28 last 21.80) - open/close 15782 (time 1788878301.63 total 779.32 last 23.69) - open/close 16038 (time 1788878312.58 total 790.27 last 23.39) - open/close 16230 (time 1788878323.26 total 800.96 last 17.96) - open/close 16998 (time 1788878333.27 total 810.96 last 76.76) - open/close 18926 (time 1788878343.27 total 820.97 last 192.74) - open/close 20000 (time 1788878347.39 total 825.08 last 261.12) - open/close 22236 (time 1788878357.40 total 835.09 last 223.38) - open/close 24359 (time 1788878367.40 total 845.09 last 212.27) - open/close 27095 (time 1788878377.40 total 855.09 last 273.59) - open/close 29504 (time 1788878387.40 total 865.09 last 240.90) - open/close 32845 (time 1788878397.40 total 875.09 last 334.09) - open/close 35392 (time 1788878407.40 total 885.09 last 254.68) - open/close 37749 (time 1788878417.40 total 895.09 last 235.70) - open/close 39595 (time 1788878427.40 total 905.09 last 184.56) - open/close 40000 (time 1788878429.56 total 907.25 last 187.38) - open/close 41563 (time 1788878439.57 total 917.26 last 156.23) - open/close 43677 (time 1788878449.57 total 927.26 last 211.36) - open/close 45857 (time 1788878459.57 total 937.27 last 217.89) - open/close 47724 (time 1788878469.58 total 947.27 last 186.59) - open/close 49557 (time 1788878479.58 total 957.27 last 183.29) total: 50000 open/close in 960.00 seconds: 52.08 ops/second - unlinked 0 (time 1788878484 ; total 0 ; last 0) - unlinked 10000 (time 1788878532 ; total 48 ; last 48) - unlinked 20000 (time 1788878564 ; total 80 ; last 32) - unlinked 30000 (time 1788878589 ; total 105 ; last 25) - unlinked 40000 (time 1788878618 ; total 134 ; last 29) total: 50000 unlinks in 163 seconds: 306.748474 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg115-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg115-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg115-server stop mds service on oleg115-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg115-server unloading modules via unload_modules_local on: 'oleg115-server' oleg115-server: oleg115-server.virtnet: executing unload_modules_local oleg115-server: kmemleak disabled modules unloaded. pdsh@oleg115-client: oleg115-client: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-server: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-client: ssh exited with exit code 2 pdsh@oleg115-client: oleg115-server: ssh exited with exit code 2