== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:37:47 (1790901467) start mds service on oleg131-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1940568 Oct 1 14:24 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: kmemleak disabled oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions oleg131-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg131-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg131-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg131-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg131-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg131-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 221 (time 1790901597.22 total 11.03 last 20.04) - open/close 468 (time 1790901607.85 total 21.66 last 23.24) - open/close 670 (time 1790901618.68 total 32.49 last 18.65) - open/close 802 (time 1790901629.47 total 43.28 last 12.23) - open/close 1059 (time 1790901639.48 total 53.29 last 25.67) - open/close 1241 (time 1790901649.50 total 63.31 last 18.18) - open/close 1370 (time 1790901659.72 total 73.53 last 12.62) - open/close 1547 (time 1790901670.32 total 84.13 last 16.70) - open/close 1653 (time 1790901680.56 total 94.37 last 10.35) - open/close 1800 (time 1790901691.42 total 105.23 last 13.54) - open/close 1934 (time 1790901701.43 total 115.25 last 13.38) - open/close 2102 (time 1790901712.24 total 126.05 last 15.55) - open/close 2224 (time 1790901723.88 total 137.69 last 10.48) - open/close 2341 (time 1790901734.90 total 148.71 last 10.62) - open/close 2486 (time 1790901745.07 total 158.88 last 14.25) - open/close 2601 (time 1790901755.49 total 169.30 last 11.04) - open/close 2764 (time 1790901765.87 total 179.68 last 15.71) - open/close 2827 (time 1790901775.99 total 189.80 last 6.22) - open/close 3048 (time 1790901786.00 total 199.81 last 22.08) - open/close 3255 (time 1790901798.18 total 211.99 last 16.99) - open/close 3383 (time 1790901809.21 total 223.02 last 11.61) - open/close 3517 (time 1790901819.46 total 233.27 last 13.06) - open/close 3679 (time 1790901830.51 total 244.32 last 14.67) - open/close 3763 (time 1790901840.84 total 254.65 last 8.13) - open/close 3968 (time 1790901850.92 total 264.73 last 20.33) - open/close 4155 (time 1790901860.94 total 274.75 last 18.66) - open/close 4292 (time 1790901871.98 total 285.79 last 12.41) - open/close 4444 (time 1790901882.47 total 296.28 last 14.50) - open/close 4610 (time 1790901893.30 total 307.11 last 15.32) - open/close 4669 (time 1790901903.33 total 317.14 last 5.88) - open/close 4853 (time 1790901913.34 total 327.15 last 18.37) - open/close 4973 (time 1790901923.35 total 337.16 last 11.99) - open/close 5239 (time 1790901933.43 total 347.24 last 26.38) - open/close 5486 (time 1790901944.23 total 358.04 last 22.88) - open/close 5768 (time 1790901954.24 total 368.05 last 28.17) - open/close 5972 (time 1790901964.59 total 378.40 last 19.71) - open/close 6215 (time 1790901975.21 total 389.02 last 22.88) - open/close 6495 (time 1790901985.44 total 399.25 last 27.36) - open/close 6731 (time 1790902000.32 total 414.13 last 15.87) - open/close 6979 (time 1790902011.83 total 425.64 last 21.54) - open/close 7290 (time 1790902021.84 total 435.65 last 31.08) - open/close 7480 (time 1790902031.86 total 445.67 last 18.97) - open/close 7739 (time 1790902041.87 total 455.68 last 25.86) - open/close 7997 (time 1790902052.52 total 466.33 last 24.22) - open/close 8246 (time 1790902063.37 total 477.18 last 22.95) - open/close 8343 (time 1790902073.93 total 487.74 last 9.19) - open/close 8550 (time 1790902084.34 total 498.15 last 19.87) - open/close 8735 (time 1790902095.80 total 509.61 last 16.14) - open/close 8986 (time 1790902105.81 total 519.62 last 25.09) - open/close 9117 (time 1790902116.99 total 530.80 last 11.71) - open/close 9306 (time 1790902127.00 total 540.81 last 18.88) - open/close 9442 (time 1790902137.98 total 551.79 last 12.38) - open/close 9567 (time 1790902150.78 total 564.59 last 9.77) - open/close 9831 (time 1790902161.67 total 575.48 last 24.24) - open/close 10000 (time 1790902168.51 total 582.32 last 24.71) - open/close 10147 (time 1790902178.51 total 592.32 last 14.70) - open/close 10433 (time 1790902188.52 total 602.33 last 28.56) - open/close 10706 (time 1790902198.74 total 612.55 last 26.73) - open/close 10997 (time 1790902209.71 total 623.52 last 26.52) - open/close 11184 (time 1790902220.21 total 634.02 last 17.81) - open/close 11293 (time 1790902230.57 total 644.38 last 10.52) - open/close 11467 (time 1790902241.20 total 655.02 last 16.37) - open/close 11546 (time 1790902251.84 total 665.65 last 7.43) - open/close 11695 (time 1790902261.85 total 675.66 last 14.88) - open/close 11831 (time 1790902273.84 total 687.65 last 11.35) - open/close 12146 (time 1790902284.34 total 698.15 last 30.01) - open/close 12378 (time 1790902294.34 total 708.15 last 23.20) - open/close 12508 (time 1790902305.21 total 719.02 last 11.96) - open/close 12758 (time 1790902315.22 total 729.03 last 24.98) - open/close 12854 (time 1790902325.55 total 739.36 last 9.29) - open/close 13040 (time 1790902336.13 total 749.94 last 17.58) - open/close 13309 (time 1790902346.14 total 759.95 last 26.87) - open/close 13541 (time 1790902356.16 total 769.97 last 23.17) - open/close 13765 (time 1790902366.44 total 780.25 last 21.78) - open/close 13950 (time 1790902377.28 total 791.09 last 17.06) - open/close 14076 (time 1790902387.96 total 801.77 last 11.80) - open/close 14228 (time 1790902397.97 total 811.78 last 15.18) - open/close 14475 (time 1790902407.98 total 821.80 last 24.66) - open/close 14630 (time 1790902419.08 total 832.89 last 13.97) - open/close 14883 (time 1790902430.10 total 843.91 last 22.95) - open/close 15000 (time 1790902440.89 total 854.70 last 10.85) - open/close 15162 (time 1790902451.59 total 865.40 last 15.14) - open/close 15331 (time 1790902462.06 total 875.88 last 16.14) - open/close 15543 (time 1790902472.08 total 885.89 last 21.16) - open/close 15763 (time 1790902482.60 total 896.41 last 20.92) - open/close 15961 (time 1790902493.57 total 907.38 last 18.05) - open/close 16043 (time 1790902503.59 total 917.40 last 8.18) - open/close 16286 (time 1790902513.61 total 927.42 last 24.26) - open/close 16561 (time 1790902523.61 total 937.42 last 27.50) - open/close 16733 (time 1790902534.58 total 948.39 last 15.67) - open/close 16976 (time 1790902544.97 total 958.78 last 23.40) - open/close 17126 (time 1790902554.97 total 968.78 last 14.99) - open/close 17353 (time 1790902565.36 total 979.18 last 21.84) - open/close 17653 (time 1790902575.38 total 989.19 last 29.96) - open/close 17859 (time 1790902585.39 total 999.20 last 20.58) - open/close 18109 (time 1790902596.72 total 1010.53 last 22.06) - open/close 18372 (time 1790902607.74 total 1021.55 last 23.87) - open/close 18645 (time 1790902618.06 total 1031.87 last 26.44) - open/close 19343 (time 1790902628.08 total 1041.89 last 69.68) - open/close 20000 (time 1790902634.92 total 1048.73 last 96.09) - open/close 20902 (time 1790902644.92 total 1058.73 last 90.14) - open/close 21800 (time 1790902654.93 total 1068.74 last 89.71) - open/close 22655 (time 1790902664.93 total 1078.75 last 85.49) - open/close 23578 (time 1790902674.94 total 1088.75 last 92.23) - open/close 24464 (time 1790902684.96 total 1098.77 last 88.41) - open/close 25158 (time 1790902694.97 total 1108.78 last 69.35) - open/close 25935 (time 1790902704.98 total 1118.79 last 77.65) - open/close 26607 (time 1790902715.01 total 1128.82 last 66.99) - open/close 27526 (time 1790902725.04 total 1138.85 last 91.59) - open/close 28280 (time 1790902735.05 total 1148.86 last 75.35) - open/close 28846 (time 1790902745.09 total 1158.90 last 56.36) - open/close 29546 (time 1790902755.12 total 1168.93 last 69.79) - open/close 30000 (time 1790902759.91 total 1173.72 last 94.82) - open/close 30742 (time 1790902769.92 total 1183.73 last 74.13) - open/close 31556 (time 1790902779.92 total 1193.73 last 81.36) - open/close 32294 (time 1790902789.93 total 1203.74 last 73.77) - open/close 33016 (time 1790902799.93 total 1213.74 last 72.17) - open/close 33658 (time 1790902809.94 total 1223.75 last 64.16) - open/close 34454 (time 1790902819.95 total 1233.76 last 79.54) - open/close 35047 (time 1790902829.95 total 1243.76 last 59.29) - open/close 35829 (time 1790902839.95 total 1253.76 last 78.17) - open/close 36606 (time 1790902849.96 total 1263.77 last 77.66) - open/close 37133 (time 1790902859.96 total 1273.77 last 52.68) - open/close 37777 (time 1790902869.97 total 1283.79 last 64.32) - open/close 38446 (time 1790902879.98 total 1293.79 last 66.85) - open/close 39280 (time 1790902889.99 total 1303.80 last 83.34) - open/close 39988 (time 1790902900.01 total 1313.83 last 70.62) - open/close 40785 (time 1790902910.02 total 1323.83 last 79.67) - open/close 41689 (time 1790902920.02 total 1333.83 last 90.38) - open/close 42354 (time 1790902930.03 total 1343.84 last 66.45) - open/close 43048 (time 1790902940.03 total 1353.84 last 69.36) - open/close 43738 (time 1790902950.06 total 1363.87 last 68.82) - open/close 44341 (time 1790902960.09 total 1373.90 last 60.11) - open/close 44980 (time 1790902970.10 total 1383.91 last 63.87) - open/close 45578 (time 1790902980.10 total 1393.91 last 59.79) - open/close 46279 (time 1790902990.10 total 1403.92 last 70.05) - open/close 46911 (time 1790903000.11 total 1413.92 last 63.15) - open/close 47516 (time 1790903010.11 total 1423.93 last 60.48) - open/close 48309 (time 1790903020.12 total 1433.93 last 79.30) - open/close 48733 (time 1790903030.16 total 1443.97 last 42.23) - open/close 49144 (time 1790903040.18 total 1453.99 last 41.01) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1461.63 seconds: 33.96 ops/second - unlinked 0 (time 1790903052 ; total 0 ; last 0) - unlinked 10000 (time 1790903189 ; total 137 ; last 137) - unlinked 20000 (time 1790903327 ; total 275 ; last 138) - unlinked 30000 (time 1790903458 ; total 406 ; last 131) - unlinked 40000 (time 1790903589 ; total 537 ; last 131) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 665 seconds: 74.633080 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg131-server unloading modules via unload_modules_local on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local oleg131-server: kmemleak disabled modules unloaded. pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2