== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:21:24 (1788535284) start mds service on oleg419-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921072 Sep 3 15:55 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: kmemleak disabled oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: detected 4 online CPUs by sysfs oleg419-server: MODOPTS_LIBCFS= oleg419-server: Force libcfs to create 2 CPU partitions oleg419-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg419-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg419-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg419-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg419-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg419-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 151 (time 1788535420.54 total 10.42 last 14.50) - open/close 301 (time 1788535430.55 total 20.43 last 14.98) - open/close 372 (time 1788535442.81 total 32.69 last 5.79) - open/close 508 (time 1788535453.57 total 43.45 last 12.64) - open/close 714 (time 1788535463.65 total 53.53 last 20.43) - open/close 764 (time 1788535475.26 total 65.14 last 4.31) - open/close 999 (time 1788535486.14 total 76.02 last 21.60) - open/close 1123 (time 1788535496.32 total 86.20 last 12.18) - open/close 1192 (time 1788535506.33 total 96.21 last 6.89) - open/close 1350 (time 1788535516.35 total 106.23 last 15.77) - open/close 1459 (time 1788535526.38 total 116.25 last 10.87) - open/close 1565 (time 1788535536.44 total 126.32 last 10.53) - open/close 1713 (time 1788535546.77 total 136.64 last 14.34) - open/close 1787 (time 1788535556.93 total 146.80 last 7.28) - open/close 1835 (time 1788535567.42 total 157.29 last 4.58) - open/close 1943 (time 1788535577.61 total 167.49 last 10.59) - open/close 2016 (time 1788535587.83 total 177.71 last 7.14) - open/close 2116 (time 1788535598.27 total 188.15 last 9.58) - open/close 2262 (time 1788535608.28 total 198.15 last 14.59) - open/close 2357 (time 1788535618.29 total 208.17 last 9.48) - open/close 2458 (time 1788535628.30 total 218.18 last 10.09) - open/close 2619 (time 1788535639.12 total 229.00 last 14.87) - open/close 2745 (time 1788535649.48 total 239.36 last 12.16) - open/close 2892 (time 1788535659.57 total 249.45 last 14.57) - open/close 3091 (time 1788535670.48 total 260.36 last 18.25) - open/close 3195 (time 1788535681.35 total 271.23 last 9.57) - open/close 3313 (time 1788535691.67 total 281.55 last 11.43) - open/close 3431 (time 1788535701.82 total 291.70 last 11.62) - open/close 3633 (time 1788535712.06 total 301.94 last 19.73) - open/close 3742 (time 1788535723.00 total 312.88 last 9.97) - open/close 3917 (time 1788535733.02 total 322.90 last 17.47) - open/close 4050 (time 1788535743.56 total 333.44 last 12.62) - open/close 4317 (time 1788535754.63 total 344.51 last 24.12) - open/close 4572 (time 1788535764.84 total 354.72 last 24.97) - open/close 4757 (time 1788535775.79 total 365.67 last 16.89) - open/close 4976 (time 1788535785.80 total 375.67 last 21.90) - open/close 5089 (time 1788535796.42 total 386.30 last 10.63) - open/close 5293 (time 1788535807.50 total 397.38 last 18.41) - open/close 5547 (time 1788535818.50 total 408.38 last 23.09) - open/close 5762 (time 1788535830.59 total 420.47 last 17.78) - open/close 5901 (time 1788535841.09 total 430.97 last 13.24) - open/close 6084 (time 1788535851.72 total 441.59 last 17.22) - open/close 6246 (time 1788535862.09 total 451.97 last 15.62) - open/close 6441 (time 1788535872.11 total 461.98 last 19.46) - open/close 6682 (time 1788535882.30 total 472.18 last 23.65) - open/close 6898 (time 1788535895.16 total 485.03 last 16.80) - open/close 7102 (time 1788535906.18 total 496.06 last 18.50) - open/close 7256 (time 1788535918.40 total 508.27 last 12.61) - open/close 7484 (time 1788535928.41 total 518.29 last 22.77) - open/close 7692 (time 1788535938.41 total 528.29 last 20.80) - open/close 7915 (time 1788535948.41 total 538.29 last 22.29) - open/close 8097 (time 1788535958.47 total 548.35 last 18.10) - open/close 8280 (time 1788535968.48 total 558.35 last 18.29) - open/close 8468 (time 1788535981.38 total 571.26 last 14.57) - open/close 8735 (time 1788535992.12 total 582.00 last 24.85) - open/close 8894 (time 1788536002.13 total 592.01 last 15.89) - open/close 9039 (time 1788536013.39 total 603.27 last 12.88) - open/close 9140 (time 1788536023.47 total 613.35 last 10.02) - open/close 9300 (time 1788536036.22 total 626.10 last 12.55) - open/close 9523 (time 1788536047.23 total 637.10 last 20.27) - open/close 9688 (time 1788536057.46 total 647.34 last 16.12) - open/close 9821 (time 1788536067.94 total 657.81 last 12.70) - open/close 9888 (time 1788536078.79 total 668.67 last 6.17) - open/close 10000 (time 1788536083.78 total 673.65 last 22.47) - open/close 10123 (time 1788536093.80 total 683.67 last 12.27) - open/close 10301 (time 1788536104.37 total 694.25 last 16.84) - open/close 10449 (time 1788536114.81 total 704.69 last 14.18) - open/close 10567 (time 1788536124.81 total 714.69 last 11.80) - open/close 10657 (time 1788536135.42 total 725.29 last 8.48) - open/close 10785 (time 1788536146.04 total 735.92 last 12.05) - open/close 10908 (time 1788536156.05 total 745.93 last 12.29) - open/close 11100 (time 1788536166.07 total 755.94 last 19.17) - open/close 11277 (time 1788536176.80 total 766.68 last 16.49) - open/close 11470 (time 1788536187.84 total 777.71 last 17.49) - open/close 11589 (time 1788536197.91 total 787.78 last 11.82) - open/close 11717 (time 1788536208.38 total 798.26 last 12.22) - open/close 11862 (time 1788536218.43 total 808.31 last 14.42) - open/close 12007 (time 1788536228.81 total 818.69 last 13.97) - open/close 12246 (time 1788536239.48 total 829.36 last 22.40) - open/close 12506 (time 1788536249.85 total 839.72 last 25.09) - open/close 12718 (time 1788536260.11 total 849.99 last 20.66) - open/close 12921 (time 1788536270.72 total 860.60 last 19.13) - open/close 13084 (time 1788536280.98 total 870.85 last 15.89) - open/close 13299 (time 1788536290.99 total 880.87 last 21.47) - open/close 13371 (time 1788536301.00 total 890.88 last 7.19) - open/close 13488 (time 1788536314.12 total 903.99 last 8.92) - open/close 13683 (time 1788536324.13 total 914.00 last 19.48) - open/close 13876 (time 1788536334.49 total 924.37 last 18.62) - open/close 14081 (time 1788536345.30 total 935.18 last 18.96) - open/close 14249 (time 1788536356.03 total 945.91 last 15.65) - open/close 14370 (time 1788536366.04 total 955.92 last 12.09) - open/close 14530 (time 1788536376.40 total 966.28 last 15.45) - open/close 14644 (time 1788536388.69 total 978.56 last 9.28) - open/close 14846 (time 1788536398.72 total 988.60 last 20.13) - open/close 15003 (time 1788536411.25 total 1001.13 last 12.53) - open/close 15118 (time 1788536421.58 total 1011.45 last 11.14) - open/close 15287 (time 1788536431.69 total 1021.57 last 16.70) - open/close 15402 (time 1788536441.86 total 1031.73 last 11.32) - open/close 15565 (time 1788536452.62 total 1042.49 last 15.15) - open/close 15691 (time 1788536462.62 total 1052.50 last 12.59) - open/close 15909 (time 1788536472.64 total 1062.52 last 21.76) - open/close 16106 (time 1788536482.81 total 1072.69 last 19.37) - open/close 16199 (time 1788536493.45 total 1083.33 last 8.74) - open/close 16423 (time 1788536503.45 total 1093.33 last 22.40) - open/close 16617 (time 1788536513.68 total 1103.56 last 18.96) - open/close 16782 (time 1788536523.91 total 1113.79 last 16.14) - open/close 17021 (time 1788536534.85 total 1124.73 last 21.85) - open/close 17210 (time 1788536544.87 total 1134.75 last 18.86) - open/close 17795 (time 1788536554.87 total 1144.75 last 58.49) - open/close 18463 (time 1788536564.90 total 1154.78 last 66.61) - open/close 19095 (time 1788536574.92 total 1164.80 last 63.06) - open/close 19767 (time 1788536584.93 total 1174.80 last 67.17) - open/close 20000 (time 1788536589.29 total 1179.17 last 53.37) - open/close 20633 (time 1788536599.32 total 1189.19 last 63.14) - open/close 21288 (time 1788536609.34 total 1199.21 last 65.38) - open/close 21893 (time 1788536619.35 total 1209.22 last 60.43) - open/close 22506 (time 1788536629.35 total 1219.22 last 61.29) - open/close 23091 (time 1788536639.36 total 1229.24 last 58.41) - open/close 23652 (time 1788536649.36 total 1239.24 last 56.09) - open/close 24213 (time 1788536659.37 total 1249.25 last 56.05) - open/close 24774 (time 1788536669.38 total 1259.26 last 56.03) - open/close 25467 (time 1788536679.39 total 1269.27 last 69.27) - open/close 26065 (time 1788536689.40 total 1279.28 last 59.71) - open/close 26742 (time 1788536699.42 total 1289.29 last 67.61) - open/close 27430 (time 1788536709.43 total 1299.31 last 68.69) - open/close 28033 (time 1788536719.46 total 1309.33 last 60.15) - open/close 28652 (time 1788536729.47 total 1319.34 last 61.84) - open/close 29283 (time 1788536739.47 total 1329.35 last 63.07) - open/close 29910 (time 1788536749.48 total 1339.35 last 62.66) - open/close 30512 (time 1788536759.49 total 1349.36 last 60.14) - open/close 31079 (time 1788536769.49 total 1359.37 last 56.65) - open/close 31752 (time 1788536779.51 total 1369.39 last 67.19) - open/close 32280 (time 1788536789.53 total 1379.40 last 52.72) - open/close 32909 (time 1788536799.53 total 1389.41 last 62.87) - open/close 33450 (time 1788536809.53 total 1399.41 last 54.09) - open/close 34084 (time 1788536819.55 total 1409.42 last 63.32) - open/close 34683 (time 1788536829.57 total 1419.45 last 59.77) - open/close 35281 (time 1788536839.61 total 1429.49 last 59.56) - open/close 35916 (time 1788536849.61 total 1439.49 last 63.47) - open/close 36568 (time 1788536859.62 total 1449.50 last 65.17) - open/close 37229 (time 1788536869.62 total 1459.50 last 66.07) - open/close 37854 (time 1788536879.63 total 1469.51 last 62.46) - open/close 38509 (time 1788536889.63 total 1479.51 last 65.50) - open/close 39115 (time 1788536899.63 total 1489.51 last 60.58) - open/close 39737 (time 1788536909.64 total 1499.52 last 62.16) - open/close 40000 (time 1788536913.61 total 1503.49 last 66.23) - open/close 40537 (time 1788536923.62 total 1513.50 last 53.64) - open/close 41170 (time 1788536933.64 total 1523.52 last 63.16) - open/close 41746 (time 1788536943.65 total 1533.53 last 57.57) - open/close 42445 (time 1788536953.65 total 1543.53 last 69.89) - open/close 43098 (time 1788536963.67 total 1553.54 last 65.18) - open/close 43779 (time 1788536973.67 total 1563.55 last 68.09) - open/close 44424 (time 1788536983.68 total 1573.55 last 64.46) - open/close 45067 (time 1788536993.68 total 1583.55 last 64.29) - open/close 45765 (time 1788537003.68 total 1593.56 last 69.76) - open/close 46327 (time 1788537013.70 total 1603.58 last 56.09) - open/close 47018 (time 1788537023.72 total 1613.60 last 68.98) - open/close 47615 (time 1788537033.75 total 1623.62 last 59.54) - open/close 48216 (time 1788537043.75 total 1633.63 last 60.07) - open/close 48877 (time 1788537053.75 total 1643.63 last 66.10) - open/close 49466 (time 1788537063.75 total 1653.63 last 58.90) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1656.34 seconds: 29.96 ops/second - unlinked 0 (time 1788537072 ; total 0 ; last 0) - unlinked 10000 (time 1788537219 ; total 147 ; last 147) - unlinked 20000 (time 1788537386 ; total 314 ; last 167) - unlinked 30000 (time 1788537531 ; total 459 ; last 145) - unlinked 40000 (time 1788537679 ; total 607 ; last 148) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 745 seconds: 66.618790 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg419-server unloading modules via unload_modules_local on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local oleg419-server: kmemleak disabled modules unloaded. pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2