== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:14:55 (1787796895) start mds service on oleg616-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1755968 Aug 26 20:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg616-server' oleg616-server: oleg616-server.virtnet: executing load_modules_local oleg616-server: kmemleak disabled oleg616-server: Loading modules from /home/green/git/lustre-release/lustre oleg616-server: detected 4 online CPUs by sysfs oleg616-server: MODOPTS_LIBCFS= oleg616-server: Force libcfs to create 2 CPU partitions oleg616-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg616-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg616-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg616-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg616-server: oleg616-server.virtnet: executing set_default_debug -1 all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg616-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg616-server: oleg616-server.virtnet: executing set_default_debug -1 all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg616-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg616-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg616-server: oleg616-server.virtnet: executing set_default_debug -1 all pdsh@oleg616-client: oleg616-server: ssh exited with exit code 1 Started lustre-OST0000 oleg616-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg616-client.virtnet: -o user_xattr,flock 192.168.206.116@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 211 (time 1787797006.20 total 10.92 last 19.32) - open/close 474 (time 1787797016.81 total 21.53 last 24.79) - open/close 661 (time 1787797027.57 total 32.29 last 17.38) - open/close 897 (time 1787797038.47 total 43.19 last 21.65) - open/close 974 (time 1787797048.47 total 53.19 last 7.70) - open/close 1159 (time 1787797058.97 total 63.69 last 17.62) - open/close 1341 (time 1787797069.80 total 74.53 last 16.80) - open/close 1467 (time 1787797080.56 total 85.28 last 11.71) - open/close 1568 (time 1787797091.19 total 95.91 last 9.50) - open/close 1773 (time 1787797102.12 total 106.84 last 18.75) - open/close 1904 (time 1787797112.12 total 116.84 last 13.10) - open/close 2113 (time 1787797123.19 total 127.91 last 18.89) - open/close 2385 (time 1787797133.19 total 137.91 last 27.20) - open/close 2613 (time 1787797144.11 total 148.83 last 20.87) - open/close 2892 (time 1787797155.11 total 159.84 last 25.36) - open/close 3115 (time 1787797165.36 total 170.08 last 21.78) - open/close 3425 (time 1787797175.78 total 180.50 last 29.73) - open/close 3636 (time 1787797186.09 total 190.81 last 20.47) - open/close 3856 (time 1787797196.09 total 200.81 last 21.99) - open/close 4040 (time 1787797206.44 total 211.16 last 17.78) - open/close 4327 (time 1787797216.44 total 221.17 last 28.69) - open/close 4435 (time 1787797226.86 total 231.58 last 10.37) - open/close 4683 (time 1787797236.86 total 241.59 last 24.78) - open/close 4970 (time 1787797246.88 total 251.60 last 28.66) - open/close 5212 (time 1787797257.91 total 262.63 last 21.93) - open/close 5438 (time 1787797268.42 total 273.14 last 21.51) - open/close 5645 (time 1787797279.14 total 283.86 last 19.31) - open/close 5738 (time 1787797289.34 total 294.06 last 9.12) - open/close 6007 (time 1787797299.34 total 304.06 last 26.89) - open/close 6190 (time 1787797310.22 total 314.94 last 16.83) - open/close 6428 (time 1787797320.24 total 324.96 last 23.75) - open/close 6586 (time 1787797330.42 total 335.14 last 15.51) - open/close 6799 (time 1787797340.71 total 345.43 last 20.70) - open/close 6870 (time 1787797350.75 total 355.47 last 7.07) - open/close 7098 (time 1787797361.33 total 366.05 last 21.54) - open/close 7310 (time 1787797371.65 total 376.37 last 20.56) - open/close 7572 (time 1787797382.30 total 387.02 last 24.59) - open/close 7800 (time 1787797392.32 total 397.04 last 22.76) - open/close 7943 (time 1787797402.34 total 407.06 last 14.27) - open/close 8075 (time 1787797412.61 total 417.33 last 12.86) - open/close 8211 (time 1787797422.61 total 427.33 last 13.59) - open/close 8453 (time 1787797435.24 total 439.96 last 19.16) - open/close 8646 (time 1787797445.24 total 449.97 last 19.29) - open/close 8844 (time 1787797455.74 total 460.46 last 18.87) - open/close 9014 (time 1787797465.85 total 470.57 last 16.81) - open/close 9134 (time 1787797475.85 total 480.57 last 12.00) - open/close 9258 (time 1787797488.41 total 493.13 last 9.87) - open/close 9397 (time 1787797499.37 total 504.09 last 12.68) - open/close 9574 (time 1787797509.46 total 514.18 last 17.54) - open/close 9764 (time 1787797519.94 total 524.66 last 18.13) - open/close 9929 (time 1787797530.79 total 535.51 last 15.20) - open/close 10000 (time 1787797535.40 total 540.12 last 15.42) - open/close 10265 (time 1787797545.40 total 550.13 last 26.49) - open/close 10464 (time 1787797555.41 total 560.13 last 19.89) - open/close 10632 (time 1787797565.92 total 570.64 last 15.99) - open/close 10771 (time 1787797575.94 total 580.66 last 13.87) - open/close 10978 (time 1787797585.96 total 590.68 last 20.66) - open/close 11151 (time 1787797596.64 total 601.36 last 16.21) - open/close 11333 (time 1787797606.66 total 611.38 last 18.16) - open/close 11609 (time 1787797616.66 total 621.38 last 27.60) - open/close 11803 (time 1787797626.66 total 631.39 last 19.39) - open/close 12131 (time 1787797636.70 total 641.42 last 32.67) - open/close 12348 (time 1787797647.65 total 652.37 last 19.82) - open/close 12589 (time 1787797658.16 total 662.88 last 22.94) - open/close 12922 (time 1787797671.02 total 675.75 last 25.88) - open/close 13229 (time 1787797681.32 total 686.04 last 29.81) - open/close 13512 (time 1787797691.33 total 696.05 last 28.27) - open/close 13756 (time 1787797702.24 total 706.96 last 22.37) - open/close 14021 (time 1787797712.24 total 716.96 last 26.49) - open/close 14238 (time 1787797722.67 total 727.39 last 20.81) - open/close 14340 (time 1787797733.18 total 737.91 last 9.70) - open/close 14482 (time 1787797743.19 total 747.91 last 14.20) - open/close 14633 (time 1787797753.33 total 758.05 last 14.89) - open/close 14918 (time 1787797763.33 total 768.05 last 28.50) - open/close 15132 (time 1787797776.39 total 781.11 last 16.38) - open/close 15459 (time 1787797787.06 total 791.78 last 30.64) - open/close 15725 (time 1787797797.08 total 801.80 last 26.56) - open/close 16007 (time 1787797807.09 total 811.81 last 28.17) - open/close 16333 (time 1787797817.11 total 821.83 last 32.53) - open/close 16501 (time 1787797828.38 total 833.10 last 14.91) - open/close 16717 (time 1787797839.41 total 844.13 last 19.58) - open/close 16935 (time 1787797849.51 total 854.23 last 21.58) - open/close 17099 (time 1787797859.87 total 864.59 last 15.83) - open/close 17241 (time 1787797869.89 total 874.61 last 14.18) - open/close 17478 (time 1787797879.98 total 884.70 last 23.48) - open/close 17768 (time 1787797889.99 total 894.71 last 28.98) - open/close 17930 (time 1787797900.28 total 905.01 last 15.73) - open/close 18182 (time 1787797913.14 total 917.86 last 19.60) - open/close 18322 (time 1787797924.15 total 928.87 last 12.72) - open/close 18572 (time 1787797937.01 total 941.74 last 19.43) - open/close 18781 (time 1787797947.62 total 952.34 last 19.70) - open/close 18941 (time 1787797960.20 total 964.93 last 12.72) - open/close 19193 (time 1787797970.91 total 975.64 last 23.53) - open/close 19418 (time 1787797982.25 total 986.97 last 19.84) - open/close 19623 (time 1787797993.19 total 997.91 last 18.75) - open/close 20000 (time 1787798002.89 total 1007.61 last 38.84) - open/close 20654 (time 1787798012.91 total 1017.63 last 65.30) - open/close 21444 (time 1787798022.91 total 1027.63 last 79.00) - open/close 22135 (time 1787798032.91 total 1037.63 last 69.09) - open/close 22831 (time 1787798042.92 total 1047.64 last 69.51) - open/close 23521 (time 1787798052.92 total 1057.64 last 68.99) - open/close 24222 (time 1787798062.93 total 1067.65 last 70.08) - open/close 24914 (time 1787798072.93 total 1077.65 last 69.15) - open/close 25672 (time 1787798082.94 total 1087.66 last 75.73) - open/close 26489 (time 1787798092.94 total 1097.67 last 81.68) - open/close 27290 (time 1787798102.95 total 1107.67 last 80.03) - open/close 28056 (time 1787798112.95 total 1117.68 last 76.59) - open/close 28844 (time 1787798122.96 total 1127.68 last 78.78) - open/close 29635 (time 1787798132.96 total 1137.68 last 79.09) - open/close 30000 (time 1787798137.99 total 1142.71 last 72.48) - open/close 30749 (time 1787798148.01 total 1152.73 last 74.81) - open/close 31503 (time 1787798158.01 total 1162.73 last 75.36) - open/close 32228 (time 1787798168.02 total 1172.75 last 72.41) - open/close 32990 (time 1787798178.03 total 1182.75 last 76.19) - open/close 33741 (time 1787798188.04 total 1192.76 last 75.02) - open/close 34510 (time 1787798198.04 total 1202.76 last 76.89) - open/close 35279 (time 1787798208.04 total 1212.76 last 76.86) - open/close 36041 (time 1787798218.04 total 1222.76 last 76.20) - open/close 36814 (time 1787798228.06 total 1232.78 last 77.19) - open/close 37571 (time 1787798238.06 total 1242.78 last 75.65) - open/close 38347 (time 1787798248.06 total 1252.79 last 77.59) - open/close 39081 (time 1787798258.07 total 1262.79 last 73.36) - open/close 39719 (time 1787798268.08 total 1272.80 last 63.74) - open/close 40000 (time 1787798272.35 total 1277.07 last 65.74) - open/close 40704 (time 1787798282.36 total 1287.08 last 70.37) - open/close 41407 (time 1787798292.36 total 1297.08 last 70.28) - open/close 42121 (time 1787798302.37 total 1307.09 last 71.31) - open/close 42875 (time 1787798312.37 total 1317.10 last 75.37) - open/close 43649 (time 1787798322.38 total 1327.10 last 77.34) - open/close 44455 (time 1787798332.39 total 1337.11 last 80.52) - open/close 45292 (time 1787798342.40 total 1347.13 last 83.60) - open/close 45951 (time 1787798352.41 total 1357.13 last 65.86) - open/close 46563 (time 1787798362.42 total 1367.14 last 61.14) - open/close 47362 (time 1787798372.43 total 1377.15 last 79.83) - open/close 48147 (time 1787798382.43 total 1387.15 last 78.47) - open/close 48820 (time 1787798392.45 total 1397.17 last 67.18) - open/close 49460 (time 1787798402.45 total 1407.17 last 63.98) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1409.33 seconds: 35.22 ops/second - unlinked 0 (time 1787798410 ; total 0 ; last 0) - unlinked 10000 (time 1787798555 ; total 145 ; last 145) - unlinked 20000 (time 1787798698 ; total 288 ; last 143) - unlinked 30000 (time 1787798839 ; total 429 ; last 141) - unlinked 40000 (time 1787798978 ; total 568 ; last 139) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 710 seconds: 69.902817 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg616-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg616-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg616-server stop mds service on oleg616-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg616-server stop mds service on oleg616-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg616-server unloading modules via unload_modules_local on: 'oleg616-server' oleg616-server: oleg616-server.virtnet: executing unload_modules_local oleg616-server: kmemleak disabled modules unloaded. pdsh@oleg616-client: oleg616-client: ssh exited with exit code 2 pdsh@oleg616-client: oleg616-server: ssh exited with exit code 2 pdsh@oleg616-client: oleg616-client: ssh exited with exit code 2 pdsh@oleg616-client: oleg616-server: ssh exited with exit code 2