== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:08:50 (1789607330) start mds service on oleg105-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 16 12:13 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing load_modules_local oleg105-server: kmemleak disabled oleg105-server: Loading modules from /home/green/git/lustre-release/lustre oleg105-server: detected 4 online CPUs by sysfs oleg105-server: MODOPTS_LIBCFS= oleg105-server: Force libcfs to create 2 CPU partitions oleg105-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg105-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg105-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg105-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg105-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg105-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Started lustre-OST0000 oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg105-client.virtnet: -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 163 (time 1789607460.12 total 10.52 last 15.49) - open/close 413 (time 1789607470.12 total 20.52 last 24.99) - open/close 619 (time 1789607480.13 total 30.53 last 20.59) - open/close 807 (time 1789607491.16 total 41.56 last 17.04) - open/close 956 (time 1789607501.45 total 51.85 last 14.48) - open/close 1085 (time 1789607511.50 total 61.90 last 12.84) - open/close 1165 (time 1789607521.63 total 72.03 last 7.90) - open/close 1264 (time 1789607534.49 total 84.90 last 7.69) - open/close 1413 (time 1789607546.99 total 97.40 last 11.92) - open/close 1672 (time 1789607557.79 total 108.19 last 24.00) - open/close 1799 (time 1789607567.80 total 118.21 last 12.68) - open/close 1973 (time 1789607577.84 total 128.24 last 17.34) - open/close 2105 (time 1789607588.77 total 139.17 last 12.07) - open/close 2247 (time 1789607598.80 total 149.20 last 14.16) - open/close 2496 (time 1789607609.72 total 160.12 last 22.80) - open/close 2677 (time 1789607620.55 total 170.96 last 16.71) - open/close 2876 (time 1789607632.42 total 182.82 last 16.77) - open/close 3002 (time 1789607642.43 total 192.84 last 12.58) - open/close 3173 (time 1789607652.44 total 202.84 last 17.10) - open/close 3308 (time 1789607663.66 total 214.06 last 12.03) - open/close 3434 (time 1789607673.88 total 224.29 last 12.32) - open/close 3447 (time 1789607684.48 total 234.88 last 1.23) - open/close 3691 (time 1789607695.34 total 245.74 last 22.47) - open/close 3941 (time 1789607705.34 total 255.74 last 24.99) - open/close 4147 (time 1789607715.37 total 265.78 last 20.53) - open/close 4404 (time 1789607725.39 total 275.80 last 25.65) - open/close 4584 (time 1789607736.79 total 287.19 last 15.80) - open/close 4764 (time 1789607746.79 total 297.19 last 18.00) - open/close 4915 (time 1789607757.22 total 307.62 last 14.48) - open/close 4974 (time 1789607768.05 total 318.45 last 5.45) - open/close 5125 (time 1789607778.15 total 328.55 last 14.94) - open/close 5194 (time 1789607788.83 total 339.23 last 6.46) - open/close 5389 (time 1789607799.44 total 349.85 last 18.37) - open/close 5600 (time 1789607810.46 total 360.86 last 19.16) - open/close 5852 (time 1789607820.48 total 370.88 last 25.16) - open/close 6008 (time 1789607831.11 total 381.52 last 14.66) - open/close 6065 (time 1789607842.09 total 392.49 last 5.20) - open/close 6369 (time 1789607852.09 total 402.50 last 30.38) - open/close 6562 (time 1789607862.10 total 412.50 last 19.30) - open/close 6782 (time 1789607873.76 total 424.16 last 18.87) - open/close 7029 (time 1789607883.78 total 434.18 last 24.65) - open/close 7196 (time 1789607894.24 total 444.64 last 15.96) - open/close 7270 (time 1789607904.24 total 454.65 last 7.40) - open/close 7536 (time 1789607915.29 total 465.69 last 24.08) - open/close 7690 (time 1789607925.34 total 475.74 last 15.33) - open/close 7858 (time 1789607935.34 total 485.75 last 16.79) - open/close 8047 (time 1789607945.36 total 495.76 last 18.88) - open/close 8174 (time 1789607955.36 total 505.76 last 12.70) - open/close 8222 (time 1789607968.03 total 518.43 last 3.79) - open/close 8355 (time 1789607978.03 total 528.44 last 13.30) - open/close 8409 (time 1789607988.79 total 539.19 last 5.02) - open/close 8477 (time 1789607998.79 total 549.20 last 6.80) - open/close 8631 (time 1789608009.68 total 560.09 last 14.14) - open/close 8807 (time 1789608020.49 total 570.90 last 16.28) - open/close 8872 (time 1789608030.55 total 580.95 last 6.47) - open/close 8982 (time 1789608041.30 total 591.71 last 10.22) - open/close 9090 (time 1789608051.32 total 601.72 last 10.78) - open/close 9344 (time 1789608061.78 total 612.19 last 24.27) - open/close 9416 (time 1789608071.81 total 622.21 last 7.18) - open/close 9510 (time 1789608082.42 total 632.82 last 8.86) - open/close 9676 (time 1789608093.16 total 643.57 last 15.45) - open/close 9934 (time 1789608103.27 total 653.67 last 25.53) - open/close 10000 (time 1789608106.61 total 657.01 last 19.79) - open/close 10189 (time 1789608116.63 total 667.03 last 18.85) - open/close 10336 (time 1789608126.82 total 677.23 last 14.42) - open/close 10570 (time 1789608136.93 total 687.34 last 23.15) - open/close 10634 (time 1789608147.05 total 697.46 last 6.32) - open/close 10739 (time 1789608157.98 total 708.39 last 9.61) - open/close 10845 (time 1789608168.22 total 718.62 last 10.36) - open/close 11054 (time 1789608178.98 total 729.38 last 19.42) - open/close 11225 (time 1789608190.06 total 740.46 last 15.43) - open/close 11372 (time 1789608200.06 total 750.46 last 14.70) - open/close 11502 (time 1789608210.12 total 760.53 last 12.92) - open/close 11589 (time 1789608220.52 total 770.92 last 8.37) - open/close 11847 (time 1789608230.52 total 780.93 last 25.79) - open/close 11943 (time 1789608240.94 total 791.35 last 9.21) - open/close 12075 (time 1789608251.36 total 801.76 last 12.67) - open/close 12253 (time 1789608261.37 total 811.77 last 17.79) - open/close 12384 (time 1789608272.93 total 823.33 last 11.33) - open/close 12534 (time 1789608282.93 total 833.33 last 15.00) - open/close 12684 (time 1789608293.40 total 843.80 last 14.33) - open/close 12788 (time 1789608304.24 total 854.64 last 9.59) - open/close 12879 (time 1789608316.98 total 867.38 last 7.14) - open/close 12950 (time 1789608327.08 total 877.48 last 7.03) - open/close 13086 (time 1789608337.25 total 887.66 last 13.37) - open/close 13332 (time 1789608347.26 total 897.66 last 24.60) - open/close 13502 (time 1789608357.93 total 908.33 last 15.93) - open/close 13754 (time 1789608367.94 total 918.35 last 25.16) - open/close 13902 (time 1789608378.98 total 929.38 last 13.41) - open/close 14029 (time 1789608389.10 total 939.50 last 12.55) - open/close 14287 (time 1789608399.11 total 949.51 last 25.79) - open/close 14403 (time 1789608409.31 total 959.72 last 11.36) - open/close 14584 (time 1789608419.33 total 969.73 last 18.07) - open/close 14719 (time 1789608429.34 total 979.75 last 13.48) - open/close 14931 (time 1789608439.36 total 989.76 last 21.17) - open/close 15252 (time 1789608450.40 total 1000.81 last 29.07) - open/close 15385 (time 1789608460.62 total 1011.03 last 13.01) - open/close 15521 (time 1789608471.13 total 1021.53 last 12.94) - open/close 15723 (time 1789608481.26 total 1031.66 last 19.95) - open/close 15982 (time 1789608491.83 total 1042.23 last 24.49) - open/close 16174 (time 1789608502.04 total 1052.44 last 18.81) - open/close 16298 (time 1789608513.13 total 1063.53 last 11.18) - open/close 16525 (time 1789608523.54 total 1073.95 last 21.79) - open/close 16687 (time 1789608534.48 total 1084.88 last 14.82) - open/close 16878 (time 1789608544.80 total 1095.21 last 18.49) - open/close 17681 (time 1789608554.81 total 1105.21 last 80.26) - open/close 18507 (time 1789608564.82 total 1115.22 last 82.51) - open/close 19298 (time 1789608574.82 total 1125.22 last 79.07) - open/close 20000 (time 1789608584.12 total 1134.52 last 75.52) - open/close 20790 (time 1789608594.12 total 1144.53 last 78.95) - open/close 21495 (time 1789608604.12 total 1154.53 last 70.49) - open/close 22282 (time 1789608614.13 total 1164.54 last 78.62) - open/close 23083 (time 1789608624.13 total 1174.54 last 80.10) - open/close 23827 (time 1789608634.14 total 1184.55 last 74.34) - open/close 24462 (time 1789608644.17 total 1194.57 last 63.35) - open/close 25230 (time 1789608654.17 total 1204.57 last 76.77) - open/close 26033 (time 1789608664.17 total 1214.58 last 80.26) - open/close 26588 (time 1789608674.18 total 1224.58 last 55.49) - open/close 27289 (time 1789608684.19 total 1234.59 last 69.99) - open/close 28046 (time 1789608694.20 total 1244.60 last 75.63) - open/close 28693 (time 1789608704.21 total 1254.61 last 64.66) - open/close 29500 (time 1789608714.23 total 1264.63 last 80.55) - open/close 30000 (time 1789608722.54 total 1272.95 last 60.12) - open/close 30748 (time 1789608732.55 total 1282.95 last 74.74) - open/close 31291 (time 1789608742.56 total 1292.96 last 54.28) - open/close 31950 (time 1789608752.56 total 1302.96 last 65.87) - open/close 32610 (time 1789608762.57 total 1312.97 last 65.97) - open/close 33306 (time 1789608772.57 total 1322.97 last 69.56) - open/close 34077 (time 1789608782.58 total 1332.98 last 77.03) - open/close 34919 (time 1789608792.59 total 1342.99 last 84.12) - open/close 35645 (time 1789608802.59 total 1353.00 last 72.58) - open/close 36340 (time 1789608812.60 total 1363.00 last 69.47) - open/close 37122 (time 1789608822.60 total 1373.00 last 78.19) - open/close 37888 (time 1789608832.60 total 1383.00 last 76.57) - open/close 38754 (time 1789608842.61 total 1393.01 last 86.53) - open/close 39549 (time 1789608852.61 total 1403.02 last 79.47) - open/close 40000 (time 1789608858.62 total 1409.02 last 75.07) - open/close 40759 (time 1789608868.64 total 1419.04 last 75.77) - open/close 41545 (time 1789608878.66 total 1429.06 last 78.43) - open/close 42288 (time 1789608888.67 total 1439.08 last 74.21) - open/close 42970 (time 1789608898.68 total 1449.08 last 68.16) - open/close 43634 (time 1789608908.69 total 1459.09 last 66.35) - open/close 44380 (time 1789608918.69 total 1469.09 last 74.58) - open/close 45234 (time 1789608928.72 total 1479.12 last 85.13) - open/close 46009 (time 1789608938.73 total 1489.13 last 77.47) - open/close 46802 (time 1789608948.74 total 1499.14 last 79.19) - open/close 47582 (time 1789608958.75 total 1509.15 last 77.92) - open/close 48353 (time 1789608968.75 total 1519.16 last 77.08) - open/close 49196 (time 1789608978.75 total 1529.16 last 84.29) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1534.21 seconds: 32.35 ops/second - unlinked 0 (time 1789608989 ; total 0 ; last 0) - unlinked 10000 (time 1789609119 ; total 130 ; last 130) - unlinked 20000 (time 1789609245 ; total 256 ; last 126) - unlinked 30000 (time 1789609374 ; total 385 ; last 129) - unlinked 40000 (time 1789609516 ; total 527 ; last 142) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 647 seconds: 76.709427 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg105-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg105-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg105-server unloading modules via unload_modules_local on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing unload_modules_local oleg105-server: kmemleak disabled modules unloaded. pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2