== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:48:04 (1788443284) start mds service on oleg643-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg643-server' oleg643-server: oleg643-server.virtnet: executing load_modules_local oleg643-server: kmemleak disabled oleg643-server: Loading modules from /home/green/git/lustre-release/lustre oleg643-server: detected 4 online CPUs by sysfs oleg643-server: MODOPTS_LIBCFS= oleg643-server: Force libcfs to create 2 CPU partitions oleg643-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg643-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg643-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg643-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg643-server: oleg643-server.virtnet: executing set_default_debug -1 all pdsh@oleg643-client: oleg643-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg643-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg643-server: oleg643-server.virtnet: executing set_default_debug -1 all pdsh@oleg643-client: oleg643-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg643-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg643-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg643-server: oleg643-server.virtnet: executing set_default_debug -1 all pdsh@oleg643-client: oleg643-server: ssh exited with exit code 1 Started lustre-OST0000 oleg643-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg643-client.virtnet: -o user_xattr,flock 192.168.206.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 264 (time 1788443392.79 total 10.01 last 26.37) - open/close 529 (time 1788443402.81 total 20.04 last 26.43) - open/close 801 (time 1788443413.30 total 30.53 last 25.93) - open/close 1080 (time 1788443423.68 total 40.91 last 26.89) - open/close 1307 (time 1788443433.73 total 50.96 last 22.58) - open/close 1571 (time 1788443445.56 total 62.79 last 22.32) - open/close 1780 (time 1788443456.24 total 73.47 last 19.57) - open/close 2021 (time 1788443468.34 total 85.57 last 19.91) - open/close 2300 (time 1788443478.36 total 95.59 last 27.85) - open/close 2586 (time 1788443489.22 total 106.44 last 26.34) - open/close 2827 (time 1788443499.22 total 116.44 last 24.10) - open/close 3080 (time 1788443509.24 total 126.47 last 25.24) - open/close 3361 (time 1788443520.12 total 137.34 last 25.83) - open/close 3606 (time 1788443530.12 total 147.35 last 24.49) - open/close 3653 (time 1788443540.15 total 157.37 last 4.69) - open/close 3931 (time 1788443550.15 total 167.38 last 27.79) - open/close 4092 (time 1788443560.16 total 177.39 last 16.08) - open/close 4250 (time 1788443571.14 total 188.37 last 14.39) - open/close 4395 (time 1788443582.00 total 199.23 last 13.35) - open/close 4611 (time 1788443605.64 total 222.86 last 9.14) - open/close 4839 (time 1788443615.94 total 233.16 last 22.13) - open/close 4927 (time 1788443625.95 total 243.17 last 8.79) - open/close 4980 (time 1788443636.15 total 253.38 last 5.19) - open/close 5247 (time 1788443646.15 total 263.38 last 26.69) - open/close 5499 (time 1788443656.67 total 273.90 last 23.96) - open/close 5832 (time 1788443666.68 total 283.90 last 33.28) - open/close 6017 (time 1788443677.69 total 294.91 last 16.80) - open/close 6102 (time 1788443688.12 total 305.34 last 8.15) - open/close 6292 (time 1788443700.04 total 317.27 last 15.93) - open/close 6525 (time 1788443713.08 total 330.31 last 17.87) - open/close 6797 (time 1788443723.10 total 340.32 last 27.16) - open/close 7060 (time 1788443733.75 total 350.98 last 24.68) - open/close 7292 (time 1788443743.76 total 360.99 last 23.18) - open/close 7529 (time 1788443754.29 total 371.52 last 22.51) - open/close 7795 (time 1788443765.07 total 382.29 last 24.68) - open/close 8067 (time 1788443776.00 total 393.23 last 24.88) - open/close 8346 (time 1788443786.51 total 403.74 last 26.54) - open/close 8599 (time 1788443796.68 total 413.91 last 24.88) - open/close 8814 (time 1788443806.71 total 423.93 last 21.44) - open/close 8980 (time 1788443816.82 total 434.05 last 16.41) - open/close 9192 (time 1788443826.83 total 444.05 last 21.18) - open/close 9488 (time 1788443837.76 total 454.99 last 27.08) - open/close 9747 (time 1788443847.77 total 464.99 last 25.88) - open/close 9985 (time 1788443858.61 total 475.83 last 21.96) - open/close 10244 (time 1788443869.44 total 486.66 last 23.92) - open/close 10455 (time 1788443879.44 total 496.66 last 21.09) - open/close 10709 (time 1788443889.85 total 507.08 last 24.39) - open/close 11018 (time 1788443900.28 total 517.51 last 29.63) - open/close 11349 (time 1788443910.31 total 527.53 last 33.02) - open/close 11577 (time 1788443921.20 total 538.42 last 20.93) - open/close 11881 (time 1788443931.24 total 548.47 last 30.27) - open/close 12172 (time 1788443941.26 total 558.48 last 29.06) - open/close 12362 (time 1788443951.80 total 569.02 last 18.03) - open/close 12517 (time 1788443962.10 total 579.32 last 15.04) - open/close 12636 (time 1788443972.61 total 589.83 last 11.32) - open/close 12843 (time 1788443983.49 total 600.72 last 19.02) - open/close 12938 (time 1788443993.50 total 610.73 last 9.49) - open/close 13063 (time 1788444003.71 total 620.93 last 12.25) - open/close 13326 (time 1788444013.72 total 630.94 last 26.28) - open/close 13506 (time 1788444023.73 total 640.96 last 17.97) - open/close 13752 (time 1788444034.31 total 651.54 last 23.24) - open/close 13937 (time 1788444045.95 total 663.17 last 15.90) - open/close 14136 (time 1788444056.03 total 673.26 last 19.73) - open/close 14339 (time 1788444066.04 total 683.27 last 20.27) - open/close 14621 (time 1788444076.05 total 693.28 last 28.18) - open/close 14878 (time 1788444086.79 total 704.02 last 23.93) - open/close 15118 (time 1788444097.99 total 715.22 last 21.43) - open/close 15288 (time 1788444108.00 total 725.23 last 16.98) - open/close 15381 (time 1788444118.52 total 735.75 last 8.84) - open/close 15572 (time 1788444129.09 total 746.31 last 18.08) - open/close 15837 (time 1788444139.59 total 756.81 last 25.23) - open/close 16090 (time 1788444150.10 total 767.33 last 24.06) - open/close 16382 (time 1788444160.11 total 777.34 last 29.18) - open/close 16616 (time 1788444170.23 total 787.46 last 23.12) - open/close 16804 (time 1788444181.82 total 799.05 last 16.22) - open/close 17057 (time 1788444192.72 total 809.94 last 23.21) - open/close 17353 (time 1788444202.73 total 819.95 last 29.57) - open/close 17600 (time 1788444213.19 total 830.41 last 23.62) - open/close 17721 (time 1788444223.74 total 840.96 last 11.47) - open/close 17922 (time 1788444234.26 total 851.49 last 19.10) - open/close 18084 (time 1788444244.35 total 861.58 last 16.05) - open/close 18242 (time 1788444254.73 total 871.95 last 15.23) - open/close 18461 (time 1788444264.80 total 882.03 last 21.74) - open/close 18635 (time 1788444280.47 total 897.69 last 11.11) - open/close 18862 (time 1788444291.47 total 908.70 last 20.62) - open/close 19123 (time 1788444301.49 total 918.72 last 26.05) - open/close 19364 (time 1788444311.93 total 929.16 last 23.08) - open/close 19566 (time 1788444323.00 total 940.23 last 18.25) - open/close 19748 (time 1788444334.02 total 951.24 last 16.52) - open/close 20000 (time 1788444341.85 total 959.08 last 32.16) - open/close 20763 (time 1788444351.87 total 969.10 last 76.15) - open/close 21387 (time 1788444361.88 total 979.10 last 62.36) - open/close 22109 (time 1788444371.89 total 989.11 last 72.15) - open/close 22831 (time 1788444381.89 total 999.12 last 72.16) - open/close 23561 (time 1788444391.89 total 1009.12 last 72.99) - open/close 24323 (time 1788444401.90 total 1019.13 last 76.12) - open/close 25021 (time 1788444411.91 total 1029.13 last 69.77) - open/close 25701 (time 1788444421.91 total 1039.13 last 68.00) - open/close 26388 (time 1788444431.92 total 1049.15 last 68.60) - open/close 27199 (time 1788444441.93 total 1059.15 last 81.05) - open/close 28006 (time 1788444451.94 total 1069.16 last 80.65) - open/close 28789 (time 1788444461.94 total 1079.16 last 78.29) - open/close 29557 (time 1788444471.94 total 1089.16 last 76.79) - open/close 30000 (time 1788444477.80 total 1095.03 last 75.50) - open/close 30790 (time 1788444487.82 total 1105.05 last 78.87) - open/close 31591 (time 1788444497.83 total 1115.05 last 80.03) - open/close 32342 (time 1788444507.84 total 1125.06 last 75.02) - open/close 33092 (time 1788444517.85 total 1135.07 last 74.93) - open/close 33846 (time 1788444527.85 total 1145.07 last 75.40) - open/close 34561 (time 1788444537.85 total 1155.08 last 71.46) - open/close 35368 (time 1788444547.86 total 1165.08 last 80.68) - open/close 36199 (time 1788444557.86 total 1175.09 last 83.07) - open/close 36993 (time 1788444567.86 total 1185.09 last 79.37) - open/close 37774 (time 1788444577.87 total 1195.10 last 78.05) - open/close 38285 (time 1788444587.88 total 1205.11 last 51.04) - open/close 38943 (time 1788444597.90 total 1215.13 last 65.68) - open/close 39598 (time 1788444607.90 total 1225.13 last 65.47) - open/close 40000 (time 1788444612.94 total 1230.17 last 79.82) - open/close 40824 (time 1788444622.94 total 1240.17 last 82.40) - open/close 41537 (time 1788444632.95 total 1250.17 last 71.27) - open/close 42168 (time 1788444642.95 total 1260.18 last 63.07) - open/close 42909 (time 1788444652.96 total 1270.18 last 74.06) - open/close 43798 (time 1788444662.96 total 1280.18 last 88.89) - open/close 44622 (time 1788444672.96 total 1290.18 last 82.39) - open/close 45389 (time 1788444682.97 total 1300.20 last 76.59) - open/close 46195 (time 1788444692.97 total 1310.20 last 80.59) - open/close 46979 (time 1788444702.98 total 1320.20 last 78.39) - open/close 47740 (time 1788444712.99 total 1330.21 last 75.99) - open/close 48592 (time 1788444723.01 total 1340.23 last 85.06) - open/close 49350 (time 1788444733.01 total 1350.23 last 75.79) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1353.84 seconds: 36.66 ops/second - unlinked 0 (time 1788444740 ; total 0 ; last 0) - unlinked 10000 (time 1788444872 ; total 132 ; last 132) - unlinked 20000 (time 1788444998 ; total 258 ; last 126) - unlinked 30000 (time 1788445137 ; total 397 ; last 139) - unlinked 40000 (time 1788445278 ; total 538 ; last 141) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 655 seconds: 75.772522 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg643-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg643-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg643-server stop mds service on oleg643-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg643-server stop mds service on oleg643-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg643-server unloading modules via unload_modules_local on: 'oleg643-server' oleg643-server: oleg643-server.virtnet: executing unload_modules_local oleg643-server: kmemleak disabled modules unloaded. pdsh@oleg643-client: oleg643-client: ssh exited with exit code 2 pdsh@oleg643-client: oleg643-server: ssh exited with exit code 2 pdsh@oleg643-client: oleg643-client: ssh exited with exit code 2 pdsh@oleg643-client: oleg643-server: ssh exited with exit code 2