== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:21:12 (1790778072) start mds service on oleg329-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1952992 Sep 29 14:05 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing load_modules_local oleg329-server: kmemleak disabled oleg329-server: Loading modules from /home/green/git/lustre-release/lustre oleg329-server: detected 4 online CPUs by sysfs oleg329-server: MODOPTS_LIBCFS= oleg329-server: Force libcfs to create 2 CPU partitions oleg329-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg329-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg329-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg329-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg329-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg329-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 271 (time 1790778178.11 total 10.00 last 27.09) - open/close 544 (time 1790778188.78 total 20.67 last 25.58) - open/close 788 (time 1790778201.26 total 33.16 last 19.55) - open/close 1075 (time 1790778211.85 total 43.75 last 27.10) - open/close 1304 (time 1790778222.17 total 54.07 last 22.18) - open/close 1543 (time 1790778232.21 total 64.11 last 23.81) - open/close 1791 (time 1790778242.87 total 74.77 last 23.26) - open/close 2027 (time 1790778252.88 total 84.78 last 23.57) - open/close 2227 (time 1790778262.89 total 94.78 last 19.99) - open/close 2492 (time 1790778272.89 total 104.79 last 26.49) - open/close 2674 (time 1790778282.90 total 114.79 last 18.19) - open/close 2927 (time 1790778292.90 total 124.80 last 25.28) - open/close 3169 (time 1790778303.91 total 135.81 last 21.98) - open/close 3407 (time 1790778314.76 total 146.65 last 21.94) - open/close 3590 (time 1790778326.63 total 158.53 last 15.41) - open/close 3807 (time 1790778337.33 total 169.23 last 20.28) - open/close 3907 (time 1790778347.63 total 179.53 last 9.71) - open/close 3996 (time 1790778358.94 total 190.83 last 7.87) - open/close 4123 (time 1790778368.95 total 200.84 last 12.69) - open/close 4194 (time 1790778379.95 total 211.85 last 6.45) - open/close 4383 (time 1790778389.96 total 221.85 last 18.89) - open/close 4598 (time 1790778400.25 total 232.15 last 20.89) - open/close 4741 (time 1790778410.47 total 242.36 last 14.00) - open/close 4859 (time 1790778421.17 total 253.07 last 11.02) - open/close 5020 (time 1790778431.19 total 263.09 last 16.07) - open/close 5213 (time 1790778441.72 total 273.62 last 18.33) - open/close 5382 (time 1790778451.72 total 283.62 last 16.90) - open/close 5603 (time 1790778462.97 total 294.86 last 19.65) - open/close 5874 (time 1790778473.38 total 305.28 last 26.02) - open/close 5909 (time 1790778483.39 total 315.28 last 3.50) - open/close 6104 (time 1790778493.41 total 325.30 last 19.46) - open/close 6350 (time 1790778503.42 total 335.31 last 24.57) - open/close 6511 (time 1790778513.64 total 345.54 last 15.75) - open/close 6792 (time 1790778524.67 total 356.56 last 25.48) - open/close 7022 (time 1790778535.15 total 367.04 last 21.95) - open/close 7247 (time 1790778545.15 total 377.05 last 22.49) - open/close 7471 (time 1790778555.57 total 387.46 last 21.50) - open/close 7715 (time 1790778566.40 total 398.29 last 22.53) - open/close 7974 (time 1790778576.90 total 408.79 last 24.67) - open/close 8220 (time 1790778587.30 total 419.20 last 23.65) - open/close 8532 (time 1790778598.32 total 430.21 last 28.32) - open/close 8773 (time 1790778608.89 total 440.78 last 22.80) - open/close 9050 (time 1790778618.90 total 450.80 last 27.66) - open/close 9274 (time 1790778629.88 total 461.77 last 20.41) - open/close 9504 (time 1790778639.88 total 471.78 last 22.99) - open/close 9769 (time 1790778650.48 total 482.38 last 25.00) - open/close 10000 (time 1790778658.91 total 490.81 last 27.39) - open/close 10168 (time 1790778668.92 total 500.81 last 16.79) - open/close 10309 (time 1790778679.65 total 511.54 last 13.14) - open/close 10395 (time 1790778689.77 total 521.67 last 8.50) - open/close 10510 (time 1790778700.40 total 532.30 last 10.82) - open/close 10645 (time 1790778710.41 total 542.30 last 13.49) - open/close 10827 (time 1790778722.39 total 554.28 last 15.19) - open/close 11032 (time 1790778732.39 total 564.29 last 20.49) - open/close 11179 (time 1790778743.38 total 575.28 last 13.38) - open/close 11324 (time 1790778753.71 total 585.61 last 14.04) - open/close 11583 (time 1790778763.72 total 595.61 last 25.88) - open/close 11828 (time 1790778773.73 total 605.63 last 24.47) - open/close 12116 (time 1790778783.84 total 615.74 last 28.47) - open/close 12341 (time 1790778793.85 total 625.75 last 22.48) - open/close 12492 (time 1790778804.84 total 636.74 last 13.74) - open/close 12750 (time 1790778815.34 total 647.24 last 24.58) - open/close 12958 (time 1790778825.51 total 657.40 last 20.46) - open/close 13070 (time 1790778835.52 total 667.41 last 11.19) - open/close 13189 (time 1790778846.51 total 678.40 last 10.83) - open/close 13339 (time 1790778856.88 total 688.78 last 14.46) - open/close 13463 (time 1790778866.92 total 698.82 last 12.35) - open/close 13649 (time 1790778876.94 total 708.84 last 18.55) - open/close 13817 (time 1790778887.80 total 719.69 last 15.48) - open/close 14039 (time 1790778898.74 total 730.64 last 20.28) - open/close 14200 (time 1790778909.10 total 740.99 last 15.54) - open/close 14298 (time 1790778919.11 total 751.01 last 9.79) - open/close 14530 (time 1790778929.25 total 761.15 last 22.88) - open/close 14755 (time 1790778940.09 total 771.99 last 20.76) - open/close 15028 (time 1790778950.79 total 782.68 last 25.53) - open/close 15289 (time 1790778961.51 total 793.40 last 24.35) - open/close 15533 (time 1790778972.53 total 804.43 last 22.13) - open/close 15758 (time 1790778982.96 total 814.85 last 21.58) - open/close 16061 (time 1790778992.96 total 824.86 last 30.29) - open/close 16290 (time 1790779004.01 total 835.91 last 20.72) - open/close 16568 (time 1790779014.01 total 845.91 last 27.80) - open/close 16800 (time 1790779024.37 total 856.26 last 22.41) - open/close 17030 (time 1790779035.70 total 867.59 last 20.30) - open/close 17235 (time 1790779045.70 total 877.60 last 20.49) - open/close 17463 (time 1790779056.50 total 888.40 last 21.12) - open/close 17669 (time 1790779066.69 total 898.59 last 20.22) - open/close 17884 (time 1790779077.61 total 909.51 last 19.69) - open/close 18134 (time 1790779087.64 total 919.53 last 24.93) - open/close 18286 (time 1790779097.71 total 929.61 last 15.08) - open/close 18453 (time 1790779108.53 total 940.43 last 15.44) - open/close 18644 (time 1790779118.53 total 950.43 last 19.09) - open/close 18856 (time 1790779128.82 total 960.71 last 20.62) - open/close 19059 (time 1790779139.70 total 971.59 last 18.66) - open/close 19336 (time 1790779150.65 total 982.55 last 25.28) - open/close 20000 (time 1790779160.18 total 992.08 last 69.66) - open/close 20654 (time 1790779170.20 total 1002.10 last 65.28) - open/close 21306 (time 1790779180.21 total 1012.11 last 65.17) - open/close 21991 (time 1790779190.22 total 1022.11 last 68.45) - open/close 22765 (time 1790779200.22 total 1032.11 last 77.39) - open/close 23520 (time 1790779210.23 total 1042.13 last 75.38) - open/close 24298 (time 1790779220.24 total 1052.13 last 77.78) - open/close 24967 (time 1790779230.24 total 1062.14 last 66.88) - open/close 25721 (time 1790779240.25 total 1072.14 last 75.35) - open/close 26542 (time 1790779250.26 total 1082.15 last 82.02) - open/close 27255 (time 1790779260.29 total 1092.18 last 71.09) - open/close 27980 (time 1790779270.29 total 1102.19 last 72.44) - open/close 28662 (time 1790779280.29 total 1112.19 last 68.20) - open/close 29381 (time 1790779290.30 total 1122.20 last 71.86) - open/close 30000 (time 1790779298.75 total 1130.64 last 73.29) - open/close 30722 (time 1790779308.75 total 1140.64 last 72.19) - open/close 31543 (time 1790779318.75 total 1150.65 last 82.05) - open/close 32243 (time 1790779328.77 total 1160.66 last 69.89) - open/close 32951 (time 1790779338.78 total 1170.67 last 70.74) - open/close 33682 (time 1790779348.78 total 1180.68 last 73.04) - open/close 34401 (time 1790779358.79 total 1190.68 last 71.87) - open/close 35092 (time 1790779368.79 total 1200.68 last 69.10) - open/close 35830 (time 1790779378.82 total 1210.71 last 73.59) - open/close 36579 (time 1790779388.83 total 1220.73 last 74.80) - open/close 37281 (time 1790779398.84 total 1230.73 last 70.14) - open/close 38031 (time 1790779408.85 total 1240.74 last 74.95) - open/close 38711 (time 1790779418.87 total 1250.77 last 67.83) - open/close 39446 (time 1790779428.88 total 1260.77 last 73.44) - open/close 40000 (time 1790779436.29 total 1268.19 last 74.71) - open/close 40736 (time 1790779446.29 total 1278.19 last 73.59) - open/close 41511 (time 1790779456.31 total 1288.21 last 77.37) - open/close 42254 (time 1790779466.32 total 1298.21 last 74.27) - open/close 42966 (time 1790779476.34 total 1308.23 last 71.05) - open/close 43681 (time 1790779486.34 total 1318.24 last 71.48) - open/close 44435 (time 1790779496.34 total 1328.24 last 75.38) - open/close 45147 (time 1790779506.35 total 1338.25 last 71.14) - open/close 45895 (time 1790779516.36 total 1348.26 last 74.72) - open/close 46603 (time 1790779526.38 total 1358.28 last 70.65) - open/close 47347 (time 1790779536.40 total 1368.29 last 74.30) - open/close 48130 (time 1790779546.40 total 1378.30 last 78.28) - open/close 48842 (time 1790779556.41 total 1388.30 last 71.15) - open/close 49437 (time 1790779566.41 total 1398.30 last 59.50) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1401.35 seconds: 35.42 ops/second - unlinked 0 (time 1790779574 ; total 0 ; last 0) - unlinked 10000 (time 1790779717 ; total 143 ; last 143) - unlinked 20000 (time 1790779857 ; total 283 ; last 140) - unlinked 30000 (time 1790780001 ; total 427 ; last 144) - unlinked 40000 (time 1790780157 ; total 583 ; last 156) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 726 seconds: 68.362259 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg329-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg329-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg329-server stop mds service on oleg329-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg329-server stop mds service on oleg329-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg329-server unloading modules via unload_modules_local on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing unload_modules_local oleg329-server: kmemleak disabled modules unloaded. pdsh@oleg329-client: oleg329-server: ssh exited with exit code 2 pdsh@oleg329-client: oleg329-client: ssh exited with exit code 2 pdsh@oleg329-client: oleg329-server: ssh exited with exit code 2 pdsh@oleg329-client: oleg329-client: ssh exited with exit code 2