== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:43:52 (1788587032) start mds service on oleg251-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1765344 Sep 4 08:26 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: kmemleak disabled oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: detected 4 online CPUs by sysfs oleg251-server: MODOPTS_LIBCFS= oleg251-server: Force libcfs to create 2 CPU partitions oleg251-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg251-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg251-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg251-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg251-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg251-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock 192.168.202.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 192 (time 1788587147.49 total 10.01 last 19.18) - open/close 276 (time 1788587157.52 total 20.03 last 8.38) - open/close 482 (time 1788587167.52 total 30.03 last 20.60) - open/close 730 (time 1788587180.39 total 42.91 last 19.26) - open/close 959 (time 1788587191.09 total 53.61 last 21.40) - open/close 1177 (time 1788587201.92 total 64.43 last 20.14) - open/close 1430 (time 1788587211.92 total 74.43 last 25.30) - open/close 1682 (time 1788587222.45 total 84.96 last 23.93) - open/close 1963 (time 1788587232.47 total 94.98 last 28.05) - open/close 2104 (time 1788587243.43 total 105.94 last 12.86) - open/close 2287 (time 1788587254.32 total 116.83 last 16.81) - open/close 2510 (time 1788587264.33 total 126.85 last 22.26) - open/close 2746 (time 1788587274.34 total 136.85 last 23.59) - open/close 3013 (time 1788587284.35 total 146.86 last 26.68) - open/close 3252 (time 1788587294.35 total 156.86 last 23.89) - open/close 3454 (time 1788587304.58 total 167.10 last 19.73) - open/close 3743 (time 1788587314.87 total 177.39 last 28.09) - open/close 4007 (time 1788587325.29 total 187.80 last 25.35) - open/close 4329 (time 1788587336.05 total 198.56 last 29.92) - open/close 4562 (time 1788587346.06 total 208.57 last 23.28) - open/close 4832 (time 1788587356.77 total 219.28 last 25.20) - open/close 5068 (time 1788587367.18 total 229.69 last 22.67) - open/close 5379 (time 1788587378.03 total 240.54 last 28.67) - open/close 5641 (time 1788587388.65 total 251.17 last 24.66) - open/close 5949 (time 1788587398.67 total 261.19 last 30.74) - open/close 6174 (time 1788587408.82 total 271.34 last 22.17) - open/close 6291 (time 1788587419.51 total 282.02 last 10.95) - open/close 6399 (time 1788587429.52 total 292.04 last 10.78) - open/close 6642 (time 1788587439.85 total 302.36 last 23.54) - open/close 6853 (time 1788587450.93 total 313.45 last 19.03) - open/close 7040 (time 1788587460.95 total 323.46 last 18.67) - open/close 7199 (time 1788587470.95 total 333.47 last 15.89) - open/close 7435 (time 1788587481.85 total 344.36 last 21.66) - open/close 7649 (time 1788587492.02 total 354.54 last 21.03) - open/close 7802 (time 1788587502.13 total 364.64 last 15.14) - open/close 7936 (time 1788587513.09 total 375.60 last 12.23) - open/close 8153 (time 1788587524.01 total 386.52 last 19.86) - open/close 8439 (time 1788587534.39 total 396.90 last 27.56) - open/close 8702 (time 1788587544.40 total 406.91 last 26.27) - open/close 9033 (time 1788587555.21 total 417.72 last 30.62) - open/close 9290 (time 1788587565.22 total 427.73 last 25.68) - open/close 9446 (time 1788587577.23 total 439.74 last 12.98) - open/close 9661 (time 1788587587.24 total 449.75 last 21.48) - open/close 9934 (time 1788587600.09 total 462.61 last 21.24) - open/close 10000 (time 1788587602.21 total 464.73 last 31.14) - open/close 10287 (time 1788587614.83 total 477.35 last 22.74) - open/close 10505 (time 1788587624.85 total 487.37 last 21.76) - open/close 10715 (time 1788587635.12 total 497.64 last 20.45) - open/close 10781 (time 1788587646.07 total 508.59 last 6.03) - open/close 10920 (time 1788587656.11 total 518.62 last 13.85) - open/close 11074 (time 1788587666.48 total 528.99 last 14.85) - open/close 11188 (time 1788587676.80 total 539.32 last 11.04) - open/close 11378 (time 1788587687.36 total 549.88 last 17.99) - open/close 11499 (time 1788587697.93 total 560.44 last 11.45) - open/close 11704 (time 1788587707.94 total 570.45 last 20.48) - open/close 11957 (time 1788587717.94 total 580.45 last 25.30) - open/close 12119 (time 1788587728.13 total 590.64 last 15.90) - open/close 12266 (time 1788587738.13 total 600.65 last 14.70) - open/close 12529 (time 1788587748.14 total 610.66 last 26.26) - open/close 12767 (time 1788587758.27 total 620.78 last 23.52) - open/close 12909 (time 1788587771.27 total 633.78 last 10.92) - open/close 13031 (time 1788587782.07 total 644.59 last 11.29) - open/close 13166 (time 1788587792.95 total 655.47 last 12.41) - open/close 13423 (time 1788587803.77 total 666.29 last 23.74) - open/close 13603 (time 1788587814.40 total 676.91 last 16.95) - open/close 13876 (time 1788587825.25 total 687.76 last 25.16) - open/close 14089 (time 1788587835.26 total 697.78 last 21.27) - open/close 14350 (time 1788587845.69 total 708.20 last 25.03) - open/close 14644 (time 1788587856.49 total 719.01 last 27.21) - open/close 14934 (time 1788587867.45 total 729.96 last 26.47) - open/close 15207 (time 1788587880.43 total 742.95 last 21.03) - open/close 15331 (time 1788587891.32 total 753.83 last 11.39) - open/close 15546 (time 1788587901.33 total 763.84 last 21.48) - open/close 15681 (time 1788587911.95 total 774.47 last 12.71) - open/close 15819 (time 1788587922.41 total 784.92 last 13.20) - open/close 15930 (time 1788587933.49 total 796.01 last 10.02) - open/close 16110 (time 1788587944.26 total 806.78 last 16.71) - open/close 16300 (time 1788587954.76 total 817.27 last 18.11) - open/close 16566 (time 1788587965.80 total 828.32 last 24.08) - open/close 16751 (time 1788587976.82 total 839.33 last 16.79) - open/close 16927 (time 1788587987.77 total 850.28 last 16.08) - open/close 17268 (time 1788587997.78 total 860.29 last 34.08) - open/close 17482 (time 1788588007.78 total 870.30 last 21.38) - open/close 17704 (time 1788588017.80 total 880.31 last 22.17) - open/close 17913 (time 1788588030.84 total 893.35 last 16.03) - open/close 18083 (time 1788588041.71 total 904.23 last 15.63) - open/close 18228 (time 1788588051.75 total 914.26 last 14.45) - open/close 18319 (time 1788588062.21 total 924.72 last 8.70) - open/close 18540 (time 1788588072.22 total 934.73 last 22.07) - open/close 18695 (time 1788588082.90 total 945.41 last 14.52) - open/close 18920 (time 1788588095.64 total 958.16 last 17.65) - open/close 19136 (time 1788588106.58 total 969.10 last 19.75) - open/close 19390 (time 1788588117.24 total 979.75 last 23.84) - open/close 19658 (time 1788588128.18 total 990.70 last 24.48) - open/close 20000 (time 1788588136.15 total 998.66 last 42.95) - open/close 20786 (time 1788588146.16 total 1008.67 last 78.51) - open/close 21437 (time 1788588156.16 total 1018.68 last 65.05) - open/close 22210 (time 1788588166.17 total 1028.68 last 77.26) - open/close 22965 (time 1788588176.19 total 1038.70 last 75.35) - open/close 23765 (time 1788588186.19 total 1048.71 last 79.98) - open/close 24647 (time 1788588196.19 total 1058.71 last 88.20) - open/close 25457 (time 1788588206.20 total 1068.71 last 80.97) - open/close 26381 (time 1788588216.20 total 1078.71 last 92.39) - open/close 27354 (time 1788588226.21 total 1088.72 last 97.22) - open/close 28296 (time 1788588236.21 total 1098.72 last 94.16) - open/close 29183 (time 1788588246.22 total 1108.73 last 88.62) - open/close 30000 (time 1788588254.85 total 1117.37 last 94.62) - open/close 30915 (time 1788588264.85 total 1127.37 last 91.49) - open/close 31872 (time 1788588274.85 total 1137.37 last 95.69) - open/close 32796 (time 1788588284.86 total 1147.37 last 92.34) - open/close 33771 (time 1788588294.87 total 1157.38 last 97.41) - open/close 34702 (time 1788588304.87 total 1167.38 last 93.09) - open/close 35596 (time 1788588314.88 total 1177.39 last 89.35) - open/close 36510 (time 1788588324.88 total 1187.39 last 91.36) - open/close 37467 (time 1788588334.89 total 1197.40 last 95.60) - open/close 38491 (time 1788588344.90 total 1207.42 last 102.28) - open/close 39293 (time 1788588354.91 total 1217.42 last 80.13) - open/close 40000 (time 1788588363.34 total 1225.85 last 83.87) - open/close 41046 (time 1788588373.35 total 1235.86 last 104.54) - open/close 42142 (time 1788588383.35 total 1245.86 last 109.54) - open/close 43025 (time 1788588393.35 total 1255.87 last 88.29) - open/close 44002 (time 1788588403.36 total 1265.87 last 97.65) - open/close 44920 (time 1788588413.36 total 1275.88 last 91.74) - open/close 45803 (time 1788588423.37 total 1285.88 last 88.24) - open/close 46608 (time 1788588433.38 total 1295.89 last 80.44) - open/close 47636 (time 1788588443.38 total 1305.90 last 102.76) - open/close 48489 (time 1788588453.39 total 1315.90 last 85.23) - open/close 49292 (time 1788588463.40 total 1325.91 last 80.24) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1330.31 seconds: 37.31 ops/second - unlinked 0 (time 1788588473 ; total 0 ; last 0) - unlinked 10000 (time 1788588595 ; total 122 ; last 122) - unlinked 20000 (time 1788588714 ; total 241 ; last 119) - unlinked 30000 (time 1788588779 ; total 306 ; last 65) - unlinked 40000 (time 1788588840 ; total 367 ; last 61) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 431 seconds: 115.153130 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg251-server unloading modules via unload_modules_local on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing unload_modules_local oleg251-server: kmemleak disabled modules unloaded. pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2