== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:29:10 (1788974950) start mds service on oleg318-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 9 02:32 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: kmemleak disabled oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions oleg318-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg318-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg318-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg318-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg318-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg318-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 250 (time 1788975055.97 total 10.02 last 24.95) - open/close 474 (time 1788975065.98 total 20.03 last 22.37) - open/close 701 (time 1788975076.81 total 30.86 last 20.96) - open/close 916 (time 1788975086.96 total 41.01 last 21.18) - open/close 1123 (time 1788975097.43 total 51.48 last 19.77) - open/close 1389 (time 1788975107.73 total 61.78 last 25.83) - open/close 1639 (time 1788975117.74 total 71.78 last 24.99) - open/close 1797 (time 1788975127.74 total 81.79 last 15.80) - open/close 2053 (time 1788975138.46 total 92.51 last 23.88) - open/close 2266 (time 1788975148.93 total 102.98 last 20.33) - open/close 2517 (time 1788975159.18 total 113.23 last 24.49) - open/close 2771 (time 1788975169.24 total 123.29 last 25.24) - open/close 2914 (time 1788975179.31 total 133.36 last 14.21) - open/close 3019 (time 1788975189.78 total 143.82 last 10.03) - open/close 3181 (time 1788975199.78 total 153.83 last 16.19) - open/close 3342 (time 1788975210.20 total 164.25 last 15.45) - open/close 3444 (time 1788975223.12 total 177.16 last 7.90) - open/close 3601 (time 1788975233.42 total 187.47 last 15.23) - open/close 3849 (time 1788975243.99 total 198.04 last 23.46) - open/close 4060 (time 1788975254.62 total 208.67 last 19.85) - open/close 4271 (time 1788975265.52 total 219.57 last 19.36) - open/close 4358 (time 1788975275.79 total 229.84 last 8.47) - open/close 4602 (time 1788975286.43 total 240.48 last 22.93) - open/close 4826 (time 1788975296.44 total 250.48 last 22.39) - open/close 5095 (time 1788975306.46 total 260.51 last 26.84) - open/close 5294 (time 1788975316.70 total 270.75 last 19.43) - open/close 5570 (time 1788975327.71 total 281.75 last 25.08) - open/close 5790 (time 1788975337.73 total 291.78 last 21.94) - open/close 6066 (time 1788975348.37 total 302.41 last 25.96) - open/close 6328 (time 1788975359.08 total 313.13 last 24.45) - open/close 6619 (time 1788975369.50 total 323.54 last 27.94) - open/close 6925 (time 1788975380.50 total 334.55 last 27.81) - open/close 7214 (time 1788975391.31 total 345.36 last 26.72) - open/close 7487 (time 1788975402.12 total 356.17 last 25.26) - open/close 7656 (time 1788975412.13 total 366.17 last 16.89) - open/close 7813 (time 1788975422.14 total 376.19 last 15.67) - open/close 8069 (time 1788975433.12 total 387.16 last 23.33) - open/close 8356 (time 1788975443.67 total 397.72 last 27.19) - open/close 8620 (time 1788975455.31 total 409.36 last 22.68) - open/close 8881 (time 1788975466.14 total 420.19 last 24.10) - open/close 9094 (time 1788975476.16 total 430.21 last 21.25) - open/close 9250 (time 1788975487.12 total 441.17 last 14.24) - open/close 9494 (time 1788975497.14 total 451.18 last 24.36) - open/close 9688 (time 1788975507.49 total 461.53 last 18.74) - open/close 9960 (time 1788975518.59 total 472.64 last 24.48) - open/close 10180 (time 1788975529.43 total 483.48 last 20.30) - open/close 10425 (time 1788975540.05 total 494.10 last 23.08) - open/close 10704 (time 1788975550.09 total 504.13 last 27.80) - open/close 10910 (time 1788975560.27 total 514.32 last 20.23) - open/close 11119 (time 1788975571.10 total 525.15 last 19.30) - open/close 11335 (time 1788975581.10 total 535.15 last 21.60) - open/close 11640 (time 1788975591.11 total 545.15 last 30.49) - open/close 11884 (time 1788975601.58 total 555.62 last 23.31) - open/close 12108 (time 1788975611.58 total 565.62 last 22.40) - open/close 12340 (time 1788975621.59 total 575.63 last 23.17) - open/close 12605 (time 1788975631.83 total 585.87 last 25.88) - open/close 12874 (time 1788975641.94 total 595.99 last 26.59) - open/close 13184 (time 1788975651.95 total 605.99 last 30.99) - open/close 13442 (time 1788975662.68 total 616.73 last 24.04) - open/close 13614 (time 1788975673.35 total 627.40 last 16.11) - open/close 13813 (time 1788975683.40 total 637.45 last 19.80) - open/close 13932 (time 1788975693.42 total 647.47 last 11.88) - open/close 14100 (time 1788975704.33 total 658.38 last 15.40) - open/close 14348 (time 1788975714.51 total 668.56 last 24.36) - open/close 14590 (time 1788975724.53 total 678.58 last 24.15) - open/close 14852 (time 1788975735.38 total 689.43 last 24.15) - open/close 15112 (time 1788975745.82 total 699.87 last 24.90) - open/close 15386 (time 1788975756.18 total 710.23 last 26.44) - open/close 15611 (time 1788975766.66 total 720.71 last 21.47) - open/close 15853 (time 1788975776.98 total 731.02 last 23.46) - open/close 16044 (time 1788975786.98 total 741.03 last 19.09) - open/close 16290 (time 1788975796.99 total 751.04 last 24.59) - open/close 16587 (time 1788975807.72 total 761.77 last 27.68) - open/close 16859 (time 1788975818.76 total 772.81 last 24.63) - open/close 17101 (time 1788975828.76 total 782.81 last 24.19) - open/close 17347 (time 1788975839.77 total 793.82 last 22.36) - open/close 17609 (time 1788975850.21 total 804.26 last 25.09) - open/close 17868 (time 1788975860.22 total 814.27 last 25.88) - open/close 18091 (time 1788975870.29 total 824.34 last 22.14) - open/close 18357 (time 1788975880.30 total 834.35 last 26.58) - open/close 18439 (time 1788975890.53 total 844.57 last 8.02) - open/close 18560 (time 1788975901.07 total 855.11 last 11.48) - open/close 18651 (time 1788975911.07 total 865.11 last 9.10) - open/close 18916 (time 1788975921.07 total 875.12 last 26.48) - open/close 19172 (time 1788975931.79 total 885.84 last 23.88) - open/close 19395 (time 1788975942.35 total 896.39 last 21.13) - open/close 19644 (time 1788975953.36 total 907.41 last 22.61) - open/close 19907 (time 1788975964.32 total 918.36 last 24.00) - open/close 20000 (time 1788975966.75 total 920.79 last 38.28) - open/close 20741 (time 1788975976.75 total 930.80 last 74.05) - open/close 21672 (time 1788975986.76 total 940.81 last 93.04) - open/close 22432 (time 1788975996.76 total 950.81 last 75.97) - open/close 23368 (time 1788976006.76 total 960.81 last 93.59) - open/close 24297 (time 1788976016.77 total 970.82 last 92.86) - open/close 25192 (time 1788976026.78 total 980.83 last 89.42) - open/close 26104 (time 1788976036.80 total 990.84 last 91.02) - open/close 26979 (time 1788976046.80 total 1000.85 last 87.49) - open/close 27926 (time 1788976056.81 total 1010.86 last 94.61) - open/close 28865 (time 1788976066.81 total 1020.86 last 93.89) - open/close 29826 (time 1788976076.81 total 1030.86 last 96.10) - open/close 30728 (time 1788976086.81 total 1040.86 last 90.17) - open/close 31654 (time 1788976096.82 total 1050.86 last 92.59) - open/close 32559 (time 1788976106.82 total 1060.87 last 90.47) - open/close 33399 (time 1788976116.82 total 1070.87 last 83.97) - open/close 34244 (time 1788976126.83 total 1080.88 last 84.45) - open/close 35235 (time 1788976136.83 total 1090.88 last 99.05) - open/close 36089 (time 1788976146.83 total 1100.88 last 85.40) - open/close 36914 (time 1788976156.83 total 1110.88 last 82.50) - open/close 37708 (time 1788976166.85 total 1120.90 last 79.26) - open/close 38448 (time 1788976176.85 total 1130.90 last 73.98) - open/close 39314 (time 1788976186.85 total 1140.90 last 86.60) - open/close 40000 (time 1788976193.65 total 1147.70 last 100.94) - open/close 40942 (time 1788976203.66 total 1157.70 last 94.13) - open/close 41833 (time 1788976213.66 total 1167.71 last 89.09) - open/close 42764 (time 1788976223.66 total 1177.71 last 93.04) - open/close 43654 (time 1788976233.67 total 1187.71 last 88.97) - open/close 44543 (time 1788976243.67 total 1197.72 last 88.84) - open/close 45522 (time 1788976253.67 total 1207.72 last 97.89) - open/close 46368 (time 1788976263.68 total 1217.73 last 84.53) - open/close 47339 (time 1788976273.69 total 1227.73 last 97.05) - open/close 48267 (time 1788976283.69 total 1237.74 last 92.79) - open/close 49174 (time 1788976293.70 total 1247.74 last 90.63) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1252.54 seconds: 39.62 ops/second - unlinked 0 (time 1788976302 ; total 0 ; last 0) - unlinked 10000 (time 1788976416 ; total 114 ; last 114) - unlinked 20000 (time 1788976530 ; total 228 ; last 114) - unlinked 30000 (time 1788976594 ; total 292 ; last 64) - unlinked 40000 (time 1788976658 ; total 356 ; last 64) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 418 seconds: 118.734451 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg318-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server stop mds service on oleg318-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg318-server stop mds service on oleg318-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg318-server unloading modules via unload_modules_local on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing unload_modules_local oleg318-server: kmemleak disabled modules unloaded. pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2