== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:29:02 (1789270142) start mds service on oleg444-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 12 19:14 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing load_modules_local oleg444-server: kmemleak disabled oleg444-server: Loading modules from /home/green/git/lustre-release/lustre oleg444-server: detected 4 online CPUs by sysfs oleg444-server: MODOPTS_LIBCFS= oleg444-server: Force libcfs to create 2 CPU partitions oleg444-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg444-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg444-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg444-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg444-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg444-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg444-server: oleg444-server.virtnet: executing set_default_debug -1 all pdsh@oleg444-client: oleg444-server: ssh exited with exit code 1 Started lustre-OST0000 oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg444-client.virtnet: -o user_xattr,flock 192.168.204.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 231 (time 1789270247.79 total 10.25 last 22.54) - open/close 483 (time 1789270258.15 total 20.61 last 24.32) - open/close 786 (time 1789270269.09 total 31.55 last 27.71) - open/close 1067 (time 1789270279.09 total 41.55 last 28.08) - open/close 1326 (time 1789270289.66 total 52.12 last 24.52) - open/close 1386 (time 1789270299.95 total 62.41 last 5.83) - open/close 1575 (time 1789270310.06 total 72.52 last 18.69) - open/close 1834 (time 1789270320.12 total 82.58 last 25.74) - open/close 2144 (time 1789270330.13 total 92.59 last 30.97) - open/close 2395 (time 1789270340.33 total 102.78 last 24.62) - open/close 2596 (time 1789270350.76 total 113.22 last 19.26) - open/close 2868 (time 1789270361.60 total 124.06 last 25.10) - open/close 3058 (time 1789270371.60 total 134.06 last 18.99) - open/close 3306 (time 1789270382.37 total 144.83 last 23.02) - open/close 3389 (time 1789270392.50 total 154.96 last 8.20) - open/close 3569 (time 1789270402.66 total 165.12 last 17.71) - open/close 3830 (time 1789270412.67 total 175.13 last 26.07) - open/close 3970 (time 1789270422.68 total 185.13 last 14.00) - open/close 4194 (time 1789270432.69 total 195.15 last 22.37) - open/close 4430 (time 1789270443.57 total 206.03 last 21.69) - open/close 4679 (time 1789270453.58 total 216.04 last 24.88) - open/close 4879 (time 1789270463.60 total 226.05 last 19.97) - open/close 5129 (time 1789270474.12 total 236.58 last 23.75) - open/close 5377 (time 1789270484.28 total 246.74 last 24.41) - open/close 5516 (time 1789270495.33 total 257.79 last 12.58) - open/close 5663 (time 1789270505.35 total 267.80 last 14.68) - open/close 5828 (time 1789270515.70 total 278.16 last 15.94) - open/close 6124 (time 1789270525.71 total 288.16 last 29.58) - open/close 6316 (time 1789270536.12 total 298.58 last 18.43) - open/close 6477 (time 1789270548.26 total 310.72 last 13.26) - open/close 6719 (time 1789270558.27 total 320.72 last 24.19) - open/close 6910 (time 1789270568.94 total 331.40 last 17.89) - open/close 7067 (time 1789270578.95 total 341.40 last 15.69) - open/close 7321 (time 1789270589.42 total 351.88 last 24.26) - open/close 7531 (time 1789270599.42 total 361.88 last 21.00) - open/close 7692 (time 1789270609.53 total 371.98 last 15.93) - open/close 7831 (time 1789270619.55 total 382.01 last 13.87) - open/close 8002 (time 1789270629.92 total 392.38 last 16.48) - open/close 8169 (time 1789270640.49 total 402.94 last 15.81) - open/close 8390 (time 1789270650.57 total 413.03 last 21.91) - open/close 8583 (time 1789270661.42 total 423.87 last 17.80) - open/close 8811 (time 1789270672.18 total 434.64 last 21.17) - open/close 9024 (time 1789270682.18 total 444.64 last 21.30) - open/close 9278 (time 1789270692.19 total 454.65 last 25.39) - open/close 9505 (time 1789270702.52 total 464.97 last 21.98) - open/close 9722 (time 1789270712.52 total 474.98 last 21.69) - open/close 9942 (time 1789270722.53 total 484.99 last 21.97) - open/close 10205 (time 1789270732.65 total 495.11 last 25.99) - open/close 10490 (time 1789270742.67 total 505.12 last 28.46) - open/close 10551 (time 1789270754.16 total 516.62 last 5.31) - open/close 10684 (time 1789270764.71 total 527.17 last 12.60) - open/close 10955 (time 1789270775.23 total 537.69 last 25.76) - open/close 11224 (time 1789270785.72 total 548.18 last 25.65) - open/close 11512 (time 1789270795.72 total 558.18 last 28.80) - open/close 11743 (time 1789270805.82 total 568.28 last 22.88) - open/close 12011 (time 1789270815.82 total 578.28 last 26.78) - open/close 12327 (time 1789270826.10 total 588.55 last 30.76) - open/close 12588 (time 1789270836.11 total 598.56 last 26.07) - open/close 12909 (time 1789270847.03 total 609.49 last 29.38) - open/close 13069 (time 1789270858.12 total 620.57 last 14.43) - open/close 13188 (time 1789270868.12 total 630.58 last 11.89) - open/close 13426 (time 1789270878.46 total 640.92 last 23.02) - open/close 13717 (time 1789270888.47 total 650.93 last 29.07) - open/close 13976 (time 1789270900.36 total 662.82 last 21.78) - open/close 14245 (time 1789270910.37 total 672.83 last 26.87) - open/close 14491 (time 1789270921.26 total 683.72 last 22.59) - open/close 14693 (time 1789270931.27 total 693.73 last 20.18) - open/close 14922 (time 1789270942.34 total 704.79 last 20.70) - open/close 15195 (time 1789270952.46 total 714.92 last 26.97) - open/close 15432 (time 1789270962.48 total 724.94 last 23.65) - open/close 15598 (time 1789270972.60 total 735.06 last 16.40) - open/close 15866 (time 1789270983.35 total 745.81 last 24.93) - open/close 16147 (time 1789270994.12 total 756.58 last 26.09) - open/close 16359 (time 1789271004.13 total 766.58 last 21.19) - open/close 16618 (time 1789271014.14 total 776.60 last 25.87) - open/close 16760 (time 1789271024.94 total 787.40 last 13.14) - open/close 16947 (time 1789271034.95 total 797.41 last 18.69) - open/close 17057 (time 1789271045.23 total 807.69 last 10.70) - open/close 17305 (time 1789271055.24 total 817.70 last 24.78) - open/close 17527 (time 1789271065.27 total 827.73 last 22.14) - open/close 17783 (time 1789271075.91 total 838.37 last 24.05) - open/close 18032 (time 1789271086.26 total 848.71 last 24.07) - open/close 18189 (time 1789271097.70 total 860.16 last 13.72) - open/close 18386 (time 1789271107.72 total 870.18 last 19.66) - open/close 18636 (time 1789271117.94 total 880.40 last 24.47) - open/close 18909 (time 1789271127.95 total 890.41 last 27.27) - open/close 19115 (time 1789271137.99 total 900.44 last 20.53) - open/close 19301 (time 1789271148.98 total 911.44 last 16.91) - open/close 19551 (time 1789271159.98 total 922.43 last 22.75) - open/close 19804 (time 1789271169.99 total 932.44 last 25.27) - open/close 20000 (time 1789271179.86 total 942.32 last 19.85) - open/close 20241 (time 1789271189.93 total 952.39 last 23.92) - open/close 20489 (time 1789271200.24 total 962.70 last 24.05) - open/close 20709 (time 1789271210.25 total 972.71 last 21.99) - open/close 21493 (time 1789271220.26 total 982.71 last 78.35) - open/close 22348 (time 1789271230.26 total 992.72 last 85.47) - open/close 23187 (time 1789271240.27 total 1002.72 last 83.85) - open/close 24100 (time 1789271250.28 total 1012.73 last 91.21) - open/close 24971 (time 1789271260.29 total 1022.75 last 86.99) - open/close 25837 (time 1789271270.29 total 1032.75 last 86.58) - open/close 26719 (time 1789271280.30 total 1042.75 last 88.15) - open/close 27530 (time 1789271290.30 total 1052.76 last 81.08) - open/close 28361 (time 1789271300.32 total 1062.78 last 82.90) - open/close 29214 (time 1789271310.32 total 1072.78 last 85.30) - open/close 29965 (time 1789271320.33 total 1082.79 last 75.06) - open/close 30640 (time 1789271330.34 total 1092.80 last 67.43) - open/close 31441 (time 1789271340.35 total 1102.80 last 80.05) - open/close 32240 (time 1789271350.35 total 1112.81 last 79.85) - open/close 33143 (time 1789271360.36 total 1122.82 last 90.21) - open/close 33993 (time 1789271370.37 total 1132.83 last 84.90) - open/close 34842 (time 1789271380.38 total 1142.84 last 84.83) - open/close 35642 (time 1789271390.39 total 1152.85 last 79.92) - open/close 36497 (time 1789271400.39 total 1162.85 last 85.49) - open/close 37371 (time 1789271410.41 total 1172.87 last 87.25) - open/close 38256 (time 1789271420.44 total 1182.89 last 88.26) - open/close 39133 (time 1789271430.44 total 1192.89 last 87.70) - open/close 39966 (time 1789271440.45 total 1202.90 last 83.22) - open/close 40887 (time 1789271450.46 total 1212.91 last 92.03) - open/close 41748 (time 1789271460.47 total 1222.93 last 85.99) - open/close 42617 (time 1789271470.47 total 1232.93 last 86.88) - open/close 43406 (time 1789271480.47 total 1242.93 last 78.89) - open/close 44311 (time 1789271490.48 total 1252.93 last 90.46) - open/close 45118 (time 1789271500.48 total 1262.94 last 80.64) - open/close 45953 (time 1789271510.49 total 1272.95 last 83.46) - open/close 46829 (time 1789271520.50 total 1282.96 last 87.51) - open/close 47661 (time 1789271530.52 total 1292.98 last 82.98) - open/close 48515 (time 1789271540.53 total 1302.99 last 85.33) - open/close 49382 (time 1789271550.53 total 1312.99 last 86.68) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1315.77 seconds: 37.72 ops/second - unlinked 0 (time 1789271557 ; total 0 ; last 0) - unlinked 10000 (time 1789271682 ; total 125 ; last 125) - unlinked 20000 (time 1789271805 ; total 248 ; last 123) - unlinked 30000 (time 1789271927 ; total 370 ; last 122) - unlinked 40000 (time 1789272042 ; total 485 ; last 115) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 602 seconds: 82.443520 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg444-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg444-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg444-server stop mds service on oleg444-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg444-server unloading modules via unload_modules_local on: 'oleg444-server' oleg444-server: oleg444-server.virtnet: executing unload_modules_local oleg444-server: kmemleak disabled modules unloaded. pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-server: ssh exited with exit code 2 pdsh@oleg444-client: oleg444-client: ssh exited with exit code 2