== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:49:35 (1788842975) start mds service on oleg623-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 7 17:37 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg623-server' oleg623-server: oleg623-server.virtnet: executing load_modules_local oleg623-server: kmemleak disabled oleg623-server: Loading modules from /home/green/git/lustre-release/lustre oleg623-server: detected 4 online CPUs by sysfs oleg623-server: MODOPTS_LIBCFS= oleg623-server: Force libcfs to create 2 CPU partitions oleg623-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg623-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg623-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg623-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg623-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg623-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg623-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg623-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg623-server: oleg623-server.virtnet: executing set_default_debug -1 all pdsh@oleg623-client: oleg623-server: ssh exited with exit code 1 Started lustre-OST0000 oleg623-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg623-client.virtnet: -o user_xattr,flock 192.168.206.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 257 (time 1788843079.71 total 10.48 last 24.51) - open/close 496 (time 1788843089.74 total 20.51 last 23.84) - open/close 721 (time 1788843100.62 total 31.40 last 20.66) - open/close 1016 (time 1788843110.64 total 41.41 last 29.45) - open/close 1302 (time 1788843120.73 total 51.51 last 28.33) - open/close 1580 (time 1788843131.46 total 62.24 last 25.91) - open/close 1821 (time 1788843141.59 total 72.37 last 23.80) - open/close 2053 (time 1788843151.70 total 82.48 last 22.94) - open/close 2348 (time 1788843164.45 total 95.22 last 23.15) - open/close 2546 (time 1788843174.48 total 105.25 last 19.74) - open/close 2802 (time 1788843185.18 total 115.96 last 23.91) - open/close 3058 (time 1788843195.88 total 126.66 last 23.92) - open/close 3324 (time 1788843206.23 total 137.00 last 25.72) - open/close 3622 (time 1788843216.24 total 147.01 last 29.77) - open/close 3864 (time 1788843226.24 total 157.01 last 24.19) - open/close 4106 (time 1788843236.24 total 167.02 last 24.19) - open/close 4360 (time 1788843246.75 total 177.53 last 24.16) - open/close 4617 (time 1788843256.98 total 187.75 last 25.14) - open/close 4761 (time 1788843266.99 total 197.76 last 14.39) - open/close 4967 (time 1788843277.39 total 208.17 last 19.80) - open/close 5146 (time 1788843288.03 total 218.80 last 16.83) - open/close 5366 (time 1788843298.04 total 228.82 last 21.97) - open/close 5515 (time 1788843308.07 total 238.84 last 14.87) - open/close 5683 (time 1788843319.84 total 250.61 last 14.27) - open/close 5979 (time 1788843330.71 total 261.49 last 27.22) - open/close 6195 (time 1788843340.72 total 271.50 last 21.58) - open/close 6477 (time 1788843350.73 total 281.50 last 28.19) - open/close 6835 (time 1788843360.80 total 291.57 last 35.56) - open/close 7099 (time 1788843370.80 total 301.58 last 26.38) - open/close 7309 (time 1788843380.82 total 311.59 last 20.97) - open/close 7606 (time 1788843391.57 total 322.35 last 27.61) - open/close 7889 (time 1788843402.52 total 333.30 last 25.85) - open/close 8092 (time 1788843415.33 total 346.11 last 15.85) - open/close 8324 (time 1788843425.49 total 356.26 last 22.85) - open/close 8634 (time 1788843435.79 total 366.56 last 30.09) - open/close 8919 (time 1788843445.79 total 376.57 last 28.49) - open/close 9173 (time 1788843456.15 total 386.92 last 24.53) - open/close 9262 (time 1788843466.16 total 396.93 last 8.89) - open/close 9402 (time 1788843476.57 total 407.34 last 13.45) - open/close 9557 (time 1788843487.01 total 417.79 last 14.84) - open/close 9850 (time 1788843497.04 total 427.81 last 29.23) - open/close 10000 (time 1788843503.97 total 434.74 last 21.65) - open/close 10315 (time 1788843513.98 total 444.75 last 31.46) - open/close 10547 (time 1788843523.98 total 454.76 last 23.19) - open/close 10811 (time 1788843533.99 total 464.77 last 26.38) - open/close 11066 (time 1788843544.60 total 475.38 last 24.03) - open/close 11162 (time 1788843557.89 total 488.66 last 7.23) - open/close 11388 (time 1788843567.91 total 498.69 last 22.54) - open/close 11597 (time 1788843578.00 total 508.77 last 20.72) - open/close 11870 (time 1788843588.12 total 518.90 last 26.97) - open/close 12045 (time 1788843598.13 total 528.90 last 17.49) - open/close 12093 (time 1788843611.68 total 542.45 last 3.54) - open/close 12340 (time 1788843622.30 total 553.07 last 23.25) - open/close 12605 (time 1788843633.32 total 564.09 last 24.05) - open/close 12893 (time 1788843644.25 total 575.02 last 26.34) - open/close 13169 (time 1788843656.97 total 587.74 last 21.70) - open/close 13364 (time 1788843668.51 total 599.29 last 16.89) - open/close 13589 (time 1788843678.53 total 609.31 last 22.46) - open/close 13867 (time 1788843688.55 total 619.32 last 27.76) - open/close 14095 (time 1788843699.02 total 629.79 last 21.77) - open/close 14350 (time 1788843709.35 total 640.13 last 24.67) - open/close 14603 (time 1788843719.36 total 650.13 last 25.29) - open/close 14862 (time 1788843729.37 total 660.15 last 25.87) - open/close 15074 (time 1788843740.06 total 670.83 last 19.83) - open/close 15329 (time 1788843750.68 total 681.45 last 24.01) - open/close 15478 (time 1788843760.72 total 691.50 last 14.84) - open/close 15683 (time 1788843773.02 total 703.79 last 16.67) - open/close 15939 (time 1788843784.02 total 714.79 last 23.28) - open/close 16142 (time 1788843794.02 total 724.79 last 20.30) - open/close 16399 (time 1788843804.46 total 735.23 last 24.61) - open/close 16651 (time 1788843815.37 total 746.14 last 23.10) - open/close 16887 (time 1788843825.39 total 756.17 last 23.54) - open/close 17128 (time 1788843836.12 total 766.90 last 22.46) - open/close 17314 (time 1788843846.13 total 776.90 last 18.59) - open/close 17572 (time 1788843856.92 total 787.70 last 23.91) - open/close 17824 (time 1788843867.73 total 798.51 last 23.31) - open/close 18113 (time 1788843878.10 total 808.87 last 27.88) - open/close 18328 (time 1788843888.78 total 819.56 last 20.13) - open/close 18544 (time 1788843898.80 total 829.58 last 21.56) - open/close 18841 (time 1788843908.81 total 839.58 last 29.69) - open/close 19084 (time 1788843918.81 total 849.58 last 24.30) - open/close 19319 (time 1788843928.81 total 859.59 last 23.49) - open/close 19550 (time 1788843938.90 total 869.67 last 22.90) - open/close 19811 (time 1788843948.91 total 879.68 last 26.08) - open/close 20000 (time 1788843955.95 total 886.72 last 26.86) - open/close 20246 (time 1788843966.94 total 897.71 last 22.38) - open/close 20757 (time 1788843976.94 total 907.71 last 51.08) - open/close 21593 (time 1788843986.94 total 917.72 last 83.59) - open/close 22415 (time 1788843996.96 total 927.73 last 82.05) - open/close 23248 (time 1788844006.97 total 937.74 last 83.26) - open/close 24062 (time 1788844016.97 total 947.74 last 81.36) - open/close 24905 (time 1788844026.97 total 957.74 last 84.30) - open/close 25697 (time 1788844036.97 total 967.75 last 79.16) - open/close 26529 (time 1788844046.98 total 977.75 last 83.18) - open/close 27346 (time 1788844056.98 total 987.76 last 81.65) - open/close 28180 (time 1788844066.99 total 997.76 last 83.35) - open/close 28953 (time 1788844076.99 total 1007.77 last 77.26) - open/close 29795 (time 1788844087.00 total 1017.77 last 84.18) - open/close 30000 (time 1788844089.40 total 1020.17 last 85.38) - open/close 30795 (time 1788844099.40 total 1030.17 last 79.50) - open/close 31670 (time 1788844109.40 total 1040.17 last 87.47) - open/close 32494 (time 1788844119.41 total 1050.19 last 82.31) - open/close 33249 (time 1788844129.42 total 1060.19 last 75.45) - open/close 34060 (time 1788844139.44 total 1070.21 last 80.94) - open/close 34911 (time 1788844149.45 total 1080.22 last 85.00) - open/close 35716 (time 1788844159.46 total 1090.23 last 80.45) - open/close 36527 (time 1788844169.46 total 1100.23 last 81.08) - open/close 37358 (time 1788844179.46 total 1110.23 last 83.08) - open/close 38175 (time 1788844189.47 total 1120.25 last 81.61) - open/close 38994 (time 1788844199.48 total 1130.25 last 81.83) - open/close 39860 (time 1788844209.48 total 1140.25 last 86.60) - open/close 40673 (time 1788844219.48 total 1150.26 last 81.28) - open/close 41502 (time 1788844229.49 total 1160.26 last 82.87) - open/close 42389 (time 1788844239.49 total 1170.27 last 88.63) - open/close 43219 (time 1788844249.50 total 1180.28 last 82.94) - open/close 44011 (time 1788844259.51 total 1190.28 last 79.15) - open/close 44824 (time 1788844269.51 total 1200.29 last 81.27) - open/close 45535 (time 1788844279.53 total 1210.30 last 70.98) - open/close 46417 (time 1788844289.54 total 1220.32 last 88.09) - open/close 47162 (time 1788844299.55 total 1230.32 last 74.45) - open/close 48047 (time 1788844309.55 total 1240.33 last 88.48) - open/close 48870 (time 1788844319.55 total 1250.33 last 82.28) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1260.00 seconds: 39.39 ops/second - unlinked 0 (time 1788844333 ; total 0 ; last 0) - unlinked 10000 (time 1788844460 ; total 127 ; last 127) - unlinked 20000 (time 1788844592 ; total 259 ; last 132) - unlinked 30000 (time 1788844720 ; total 387 ; last 128) - unlinked 40000 (time 1788844847 ; total 514 ; last 127) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 634 seconds: 78.282333 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg623-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg623-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg623-server stop mds service on oleg623-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg623-server stop mds service on oleg623-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg623-server unloading modules via unload_modules_local on: 'oleg623-server' oleg623-server: oleg623-server.virtnet: executing unload_modules_local oleg623-server: kmemleak disabled modules unloaded. pdsh@oleg623-client: oleg623-client: ssh exited with exit code 2 pdsh@oleg623-client: oleg623-server: ssh exited with exit code 2 pdsh@oleg623-client: oleg623-server: ssh exited with exit code 2 pdsh@oleg623-client: oleg623-client: ssh exited with exit code 2