== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:32:23 (1788741143) start mds service on oleg329-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921104 Sep 6 19:17 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing load_modules_local oleg329-server: kmemleak disabled oleg329-server: Loading modules from /home/green/git/lustre-release/lustre oleg329-server: detected 4 online CPUs by sysfs oleg329-server: MODOPTS_LIBCFS= oleg329-server: Force libcfs to create 2 CPU partitions oleg329-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg329-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg329-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg329-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg329-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg329-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg329-server: oleg329-server.virtnet: executing set_default_debug -1 all pdsh@oleg329-client: oleg329-server: ssh exited with exit code 1 Started lustre-OST0000 oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg329-client.virtnet: -o user_xattr,flock 192.168.203.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 244 (time 1788741248.00 total 10.26 last 23.78) - open/close 447 (time 1788741258.02 total 20.28 last 20.27) - open/close 679 (time 1788741268.02 total 30.28 last 23.19) - open/close 876 (time 1788741279.43 total 41.69 last 17.27) - open/close 1126 (time 1788741291.77 total 54.03 last 20.26) - open/close 1384 (time 1788741301.77 total 64.03 last 25.79) - open/close 1644 (time 1788741312.58 total 74.84 last 24.05) - open/close 1906 (time 1788741322.59 total 84.85 last 26.18) - open/close 2155 (time 1788741332.60 total 94.86 last 24.87) - open/close 2376 (time 1788741344.24 total 106.50 last 18.99) - open/close 2612 (time 1788741354.85 total 117.11 last 22.24) - open/close 2828 (time 1788741366.54 total 128.80 last 18.48) - open/close 3104 (time 1788741376.55 total 138.81 last 27.56) - open/close 3278 (time 1788741386.56 total 148.82 last 17.38) - open/close 3384 (time 1788741397.01 total 159.27 last 10.14) - open/close 3534 (time 1788741407.69 total 169.95 last 14.05) - open/close 3643 (time 1788741418.56 total 180.82 last 10.03) - open/close 3882 (time 1788741429.57 total 191.83 last 21.71) - open/close 4030 (time 1788741439.60 total 201.86 last 14.75) - open/close 4174 (time 1788741449.61 total 211.87 last 14.39) - open/close 4344 (time 1788741459.98 total 222.24 last 16.39) - open/close 4472 (time 1788741470.02 total 232.28 last 12.75) - open/close 4655 (time 1788741480.13 total 242.39 last 18.10) - open/close 4908 (time 1788741490.30 total 252.56 last 24.89) - open/close 5132 (time 1788741500.57 total 262.83 last 21.82) - open/close 5320 (time 1788741510.57 total 272.83 last 18.78) - open/close 5472 (time 1788741520.58 total 282.84 last 15.19) - open/close 5591 (time 1788741530.76 total 293.02 last 11.69) - open/close 5867 (time 1788741541.24 total 303.50 last 26.33) - open/close 6010 (time 1788741551.25 total 313.51 last 14.29) - open/close 6099 (time 1788741562.69 total 324.95 last 7.78) - open/close 6253 (time 1788741572.91 total 335.17 last 15.08) - open/close 6440 (time 1788741583.37 total 345.63 last 17.87) - open/close 6576 (time 1788741594.18 total 356.44 last 12.58) - open/close 6731 (time 1788741604.20 total 366.46 last 15.47) - open/close 6849 (time 1788741614.20 total 376.46 last 11.80) - open/close 7069 (time 1788741624.32 total 386.58 last 21.74) - open/close 7296 (time 1788741634.33 total 396.59 last 22.68) - open/close 7539 (time 1788741645.07 total 407.33 last 22.62) - open/close 7795 (time 1788741657.34 total 419.60 last 20.85) - open/close 8026 (time 1788741668.42 total 430.68 last 20.86) - open/close 8309 (time 1788741679.41 total 441.67 last 25.73) - open/close 8571 (time 1788741689.43 total 451.69 last 26.15) - open/close 8816 (time 1788741699.79 total 462.05 last 23.66) - open/close 9077 (time 1788741710.84 total 473.10 last 23.61) - open/close 9356 (time 1788741721.04 total 483.30 last 27.36) - open/close 9616 (time 1788741731.31 total 493.57 last 25.30) - open/close 9888 (time 1788741741.82 total 504.08 last 25.88) - open/close 10000 (time 1788741745.39 total 507.65 last 31.39) - open/close 10287 (time 1788741756.22 total 518.48 last 26.49) - open/close 10547 (time 1788741766.57 total 528.83 last 25.12) - open/close 10804 (time 1788741776.58 total 538.84 last 25.68) - open/close 11042 (time 1788741786.59 total 548.85 last 23.78) - open/close 11318 (time 1788741796.75 total 559.01 last 27.15) - open/close 11553 (time 1788741806.98 total 569.24 last 22.99) - open/close 11790 (time 1788741817.99 total 580.25 last 21.51) - open/close 12023 (time 1788741828.00 total 590.26 last 23.29) - open/close 12328 (time 1788741838.01 total 600.27 last 30.47) - open/close 12589 (time 1788741848.13 total 610.39 last 25.79) - open/close 12855 (time 1788741858.13 total 620.39 last 26.58) - open/close 13134 (time 1788741868.54 total 630.80 last 26.80) - open/close 13413 (time 1788741879.42 total 641.68 last 25.65) - open/close 13653 (time 1788741890.11 total 652.37 last 22.46) - open/close 13923 (time 1788741900.12 total 662.38 last 26.98) - open/close 14127 (time 1788741910.20 total 672.46 last 20.23) - open/close 14326 (time 1788741920.21 total 682.47 last 19.88) - open/close 14580 (time 1788741930.23 total 692.49 last 25.37) - open/close 14808 (time 1788741940.48 total 702.74 last 22.23) - open/close 14963 (time 1788741950.72 total 712.98 last 15.13) - open/close 15181 (time 1788741961.73 total 723.99 last 19.81) - open/close 15428 (time 1788741972.63 total 734.89 last 22.66) - open/close 15666 (time 1788741983.05 total 745.31 last 22.84) - open/close 15813 (time 1788741993.29 total 755.55 last 14.35) - open/close 16017 (time 1788742004.10 total 766.36 last 18.88) - open/close 16272 (time 1788742014.81 total 777.07 last 23.80) - open/close 16525 (time 1788742025.34 total 787.60 last 24.03) - open/close 16643 (time 1788742035.90 total 798.16 last 11.17) - open/close 16921 (time 1788742046.84 total 809.10 last 25.41) - open/close 17184 (time 1788742057.11 total 819.37 last 25.61) - open/close 17410 (time 1788742067.97 total 830.23 last 20.80) - open/close 17682 (time 1788742078.41 total 840.67 last 26.05) - open/close 17860 (time 1788742088.41 total 850.67 last 17.80) - open/close 17984 (time 1788742098.77 total 861.03 last 11.97) - open/close 18216 (time 1788742109.31 total 871.57 last 22.00) - open/close 18457 (time 1788742120.14 total 882.40 last 22.26) - open/close 18706 (time 1788742130.24 total 892.50 last 24.66) - open/close 18873 (time 1788742140.80 total 903.06 last 15.82) - open/close 19458 (time 1788742150.80 total 913.06 last 58.49) - open/close 20000 (time 1788742156.65 total 918.91 last 92.59) - open/close 20672 (time 1788742166.66 total 928.92 last 67.16) - open/close 21289 (time 1788742176.67 total 938.93 last 61.63) - open/close 21783 (time 1788742186.67 total 948.93 last 49.38) - open/close 22527 (time 1788742196.68 total 958.94 last 74.36) - open/close 23201 (time 1788742206.69 total 968.95 last 67.35) - open/close 24049 (time 1788742216.69 total 978.95 last 84.75) - open/close 24844 (time 1788742226.70 total 988.96 last 79.44) - open/close 25646 (time 1788742236.71 total 998.97 last 80.15) - open/close 26345 (time 1788742246.71 total 1008.97 last 69.88) - open/close 27090 (time 1788742256.71 total 1018.97 last 74.46) - open/close 27862 (time 1788742266.72 total 1028.98 last 77.16) - open/close 28705 (time 1788742276.72 total 1038.98 last 84.26) - open/close 29495 (time 1788742286.73 total 1048.99 last 78.99) - open/close 30000 (time 1788742294.40 total 1056.66 last 65.82) - open/close 30669 (time 1788742304.41 total 1066.67 last 66.82) - open/close 31360 (time 1788742314.41 total 1076.67 last 69.09) - open/close 32140 (time 1788742324.41 total 1086.67 last 78.00) - open/close 32909 (time 1788742334.41 total 1096.67 last 76.89) - open/close 33800 (time 1788742344.43 total 1106.69 last 88.96) - open/close 34564 (time 1788742354.43 total 1116.69 last 76.38) - open/close 35411 (time 1788742364.43 total 1126.69 last 84.67) - open/close 36286 (time 1788742374.44 total 1136.70 last 87.46) - open/close 37104 (time 1788742384.45 total 1146.71 last 81.69) - open/close 37993 (time 1788742394.46 total 1156.72 last 88.80) - open/close 38798 (time 1788742404.47 total 1166.73 last 80.44) - open/close 39661 (time 1788742414.49 total 1176.75 last 86.13) - open/close 40000 (time 1788742419.51 total 1181.77 last 67.51) - open/close 40820 (time 1788742429.52 total 1191.78 last 81.92) - open/close 41684 (time 1788742439.53 total 1201.78 last 86.38) - open/close 42573 (time 1788742449.54 total 1211.80 last 88.77) - open/close 43479 (time 1788742459.55 total 1221.81 last 90.55) - open/close 44344 (time 1788742469.56 total 1231.82 last 86.40) - open/close 45061 (time 1788742479.56 total 1241.82 last 71.67) - open/close 45864 (time 1788742489.56 total 1251.82 last 80.27) - open/close 46824 (time 1788742499.58 total 1261.84 last 95.89) - open/close 47645 (time 1788742509.58 total 1271.84 last 82.06) - open/close 48497 (time 1788742519.58 total 1281.84 last 85.17) - open/close 49405 (time 1788742529.60 total 1291.86 last 90.65) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1294.19 seconds: 38.35 ops/second - unlinked 0 (time 1788742535 ; total 0 ; last 0) - unlinked 10000 (time 1788742637 ; total 102 ; last 102) - unlinked 20000 (time 1788742735 ; total 200 ; last 98) - unlinked 30000 (time 1788742796 ; total 261 ; last 61) - unlinked 40000 (time 1788742852 ; total 317 ; last 56) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 371 seconds: 133.776276 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg329-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg329-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg329-server stop mds service on oleg329-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg329-server stop mds service on oleg329-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg329-server unloading modules via unload_modules_local on: 'oleg329-server' oleg329-server: oleg329-server.virtnet: executing unload_modules_local oleg329-server: kmemleak disabled modules unloaded. pdsh@oleg329-client: oleg329-client: ssh exited with exit code 2 pdsh@oleg329-client: oleg329-server: ssh exited with exit code 2 pdsh@oleg329-client: oleg329-server: ssh exited with exit code 2 pdsh@oleg329-client: oleg329-client: ssh exited with exit code 2