== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:35:34 (1788669334) start mds service on oleg337-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1922328 Sep 5 15:39 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg337-server' oleg337-server: oleg337-server.virtnet: executing load_modules_local oleg337-server: kmemleak disabled oleg337-server: Loading modules from /home/green/git/lustre-release/lustre oleg337-server: detected 4 online CPUs by sysfs oleg337-server: MODOPTS_LIBCFS= oleg337-server: Force libcfs to create 2 CPU partitions oleg337-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg337-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg337-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg337-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg337-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg337-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-OST0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 275 (time 1788669444.64 total 10.80 last 25.45) - open/close 422 (time 1788669454.65 total 20.81 last 14.70) - open/close 658 (time 1788669464.74 total 30.90 last 23.39) - open/close 926 (time 1788669475.63 total 41.79 last 24.61) - open/close 1091 (time 1788669485.66 total 51.82 last 16.44) - open/close 1233 (time 1788669496.68 total 62.84 last 12.89) - open/close 1307 (time 1788669506.68 total 72.84 last 7.40) - open/close 1492 (time 1788669516.69 total 82.85 last 18.49) - open/close 1768 (time 1788669526.89 total 93.05 last 27.06) - open/close 1920 (time 1788669536.89 total 103.05 last 15.20) - open/close 2090 (time 1788669547.20 total 113.36 last 16.48) - open/close 2332 (time 1788669561.71 total 127.87 last 16.68) - open/close 2598 (time 1788669571.73 total 137.89 last 26.55) - open/close 2830 (time 1788669581.99 total 148.15 last 22.62) - open/close 3111 (time 1788669592.00 total 158.16 last 28.08) - open/close 3350 (time 1788669602.00 total 168.16 last 23.90) - open/close 3589 (time 1788669612.60 total 178.76 last 22.55) - open/close 3853 (time 1788669622.83 total 188.99 last 25.79) - open/close 3945 (time 1788669635.56 total 201.72 last 7.23) - open/close 4133 (time 1788669646.45 total 212.61 last 17.27) - open/close 4324 (time 1788669656.45 total 222.61 last 19.09) - open/close 4511 (time 1788669667.00 total 233.16 last 17.72) - open/close 4733 (time 1788669677.00 total 243.16 last 22.19) - open/close 4939 (time 1788669687.85 total 254.01 last 19.00) - open/close 5202 (time 1788669698.16 total 264.32 last 25.51) - open/close 5489 (time 1788669709.05 total 275.21 last 26.35) - open/close 5722 (time 1788669719.36 total 285.52 last 22.61) - open/close 5950 (time 1788669729.98 total 296.14 last 21.45) - open/close 6170 (time 1788669740.73 total 306.89 last 20.48) - open/close 6391 (time 1788669751.03 total 317.19 last 21.45) - open/close 6497 (time 1788669761.32 total 327.48 last 10.30) - open/close 6741 (time 1788669772.39 total 338.55 last 22.05) - open/close 6999 (time 1788669782.40 total 348.56 last 25.77) - open/close 7268 (time 1788669792.41 total 358.57 last 26.88) - open/close 7541 (time 1788669803.30 total 369.46 last 25.06) - open/close 7807 (time 1788669813.30 total 379.46 last 26.59) - open/close 8002 (time 1788669824.19 total 390.35 last 17.91) - open/close 8199 (time 1788669836.01 total 402.17 last 16.68) - open/close 8429 (time 1788669846.67 total 412.83 last 21.58) - open/close 8633 (time 1788669857.34 total 423.50 last 19.11) - open/close 8773 (time 1788669867.35 total 433.51 last 13.98) - open/close 8975 (time 1788669877.37 total 443.53 last 20.17) - open/close 9179 (time 1788669887.79 total 453.95 last 19.57) - open/close 9357 (time 1788669898.61 total 464.77 last 16.45) - open/close 9508 (time 1788669908.73 total 474.89 last 14.92) - open/close 9752 (time 1788669918.74 total 484.90 last 24.38) - open/close 9966 (time 1788669931.12 total 497.28 last 17.28) - open/close 10000 (time 1788669933.98 total 500.14 last 11.91) - open/close 10259 (time 1788669943.99 total 510.15 last 25.88) - open/close 10398 (time 1788669955.78 total 521.94 last 11.79) - open/close 10612 (time 1788669966.08 total 532.24 last 20.77) - open/close 10834 (time 1788669976.81 total 542.97 last 20.69) - open/close 11004 (time 1788669987.36 total 553.52 last 16.11) - open/close 11171 (time 1788669997.39 total 563.55 last 16.66) - open/close 11308 (time 1788670008.23 total 574.39 last 12.63) - open/close 11443 (time 1788670018.57 total 584.73 last 13.06) - open/close 11564 (time 1788670029.18 total 595.34 last 11.40) - open/close 11659 (time 1788670039.37 total 605.53 last 9.33) - open/close 11811 (time 1788670049.40 total 615.56 last 15.15) - open/close 12004 (time 1788670059.42 total 625.58 last 19.27) - open/close 12077 (time 1788670072.18 total 638.34 last 5.72) - open/close 12160 (time 1788670083.19 total 649.35 last 7.54) - open/close 12279 (time 1788670093.20 total 659.36 last 11.89) - open/close 12388 (time 1788670103.22 total 669.38 last 10.87) - open/close 12548 (time 1788670113.97 total 680.13 last 14.89) - open/close 12789 (time 1788670124.67 total 690.83 last 22.53) - open/close 12890 (time 1788670134.84 total 701.00 last 9.93) - open/close 13078 (time 1788670145.58 total 711.74 last 17.50) - open/close 13234 (time 1788670157.86 total 724.02 last 12.71) - open/close 13351 (time 1788670167.87 total 734.03 last 11.69) - open/close 13412 (time 1788670178.28 total 744.44 last 5.86) - open/close 13624 (time 1788670188.29 total 754.45 last 21.20) - open/close 13898 (time 1788670199.02 total 765.18 last 25.53) - open/close 14157 (time 1788670209.70 total 775.86 last 24.24) - open/close 14407 (time 1788670220.19 total 786.35 last 23.83) - open/close 14678 (time 1788670231.20 total 797.36 last 24.62) - open/close 14903 (time 1788670241.41 total 807.57 last 22.05) - open/close 15175 (time 1788670252.34 total 818.50 last 24.87) - open/close 15360 (time 1788670262.57 total 828.73 last 18.08) - open/close 15629 (time 1788670272.89 total 839.05 last 26.07) - open/close 15857 (time 1788670282.90 total 849.06 last 22.78) - open/close 16105 (time 1788670292.92 total 859.08 last 24.76) - open/close 16345 (time 1788670302.92 total 869.08 last 23.98) - open/close 16502 (time 1788670313.34 total 879.50 last 15.07) - open/close 16664 (time 1788670323.76 total 889.92 last 15.55) - open/close 16879 (time 1788670333.98 total 900.14 last 21.04) - open/close 16952 (time 1788670344.81 total 910.97 last 6.74) - open/close 17119 (time 1788670355.33 total 921.49 last 15.87) - open/close 17329 (time 1788670365.61 total 931.77 last 20.43) - open/close 17489 (time 1788670378.54 total 944.70 last 12.37) - open/close 17685 (time 1788670389.04 total 955.20 last 18.68) - open/close 17841 (time 1788670399.66 total 965.82 last 14.69) - open/close 17920 (time 1788670409.76 total 975.92 last 7.82) - open/close 18071 (time 1788670420.26 total 986.42 last 14.39) - open/close 18288 (time 1788670430.90 total 997.06 last 20.40) - open/close 18448 (time 1788670440.91 total 1007.07 last 15.98) - open/close 19192 (time 1788670450.91 total 1017.08 last 74.37) - open/close 19938 (time 1788670460.92 total 1027.08 last 74.59) - open/close 20719 (time 1788670470.92 total 1037.08 last 78.04) - open/close 21319 (time 1788670480.95 total 1047.11 last 59.86) - open/close 21902 (time 1788670490.97 total 1057.13 last 58.19) - open/close 22506 (time 1788670500.98 total 1067.14 last 60.30) - open/close 23164 (time 1788670510.99 total 1077.15 last 65.76) - open/close 23926 (time 1788670521.01 total 1087.17 last 76.07) - open/close 24532 (time 1788670531.02 total 1097.18 last 60.49) - open/close 25163 (time 1788670541.03 total 1107.19 last 63.06) - open/close 25812 (time 1788670551.06 total 1117.22 last 64.74) - open/close 26448 (time 1788670561.06 total 1127.22 last 63.56) - open/close 27135 (time 1788670571.07 total 1137.23 last 68.63) - open/close 27924 (time 1788670581.08 total 1147.24 last 78.87) - open/close 28579 (time 1788670591.08 total 1157.24 last 65.46) - open/close 29365 (time 1788670601.08 total 1167.24 last 78.59) - open/close 30000 (time 1788670610.36 total 1176.52 last 68.42) - open/close 30716 (time 1788670620.38 total 1186.54 last 71.50) - open/close 31309 (time 1788670630.42 total 1196.58 last 59.08) - open/close 31920 (time 1788670640.42 total 1206.58 last 61.06) - open/close 32558 (time 1788670650.46 total 1216.62 last 63.57) - open/close 33143 (time 1788670660.47 total 1226.63 last 58.42) - open/close 33654 (time 1788670670.50 total 1236.66 last 50.98) - open/close 34172 (time 1788670680.50 total 1246.66 last 51.76) - open/close 34748 (time 1788670690.51 total 1256.67 last 57.56) - open/close 35488 (time 1788670700.52 total 1266.68 last 73.96) - open/close 36199 (time 1788670710.54 total 1276.70 last 70.94) - open/close 36935 (time 1788670720.54 total 1286.70 last 73.60) - open/close 37630 (time 1788670730.54 total 1296.70 last 69.46) - open/close 38403 (time 1788670740.55 total 1306.71 last 77.29) - open/close 39101 (time 1788670750.56 total 1316.72 last 69.70) - open/close 39870 (time 1788670760.56 total 1326.72 last 76.87) - open/close 40638 (time 1788670770.57 total 1336.73 last 76.76) - open/close 41356 (time 1788670780.58 total 1346.74 last 71.75) - open/close 42036 (time 1788670790.60 total 1356.76 last 67.86) - open/close 42640 (time 1788670800.61 total 1366.77 last 60.34) - open/close 43208 (time 1788670810.61 total 1376.77 last 56.77) - open/close 43825 (time 1788670820.61 total 1386.78 last 61.69) - open/close 44457 (time 1788670830.62 total 1396.78 last 63.19) - open/close 45115 (time 1788670840.63 total 1406.79 last 65.73) - open/close 45894 (time 1788670850.63 total 1416.79 last 77.89) - open/close 46580 (time 1788670860.65 total 1426.81 last 68.47) - open/close 47149 (time 1788670870.66 total 1436.82 last 56.81) - open/close 47839 (time 1788670880.67 total 1446.84 last 68.92) - open/close 48567 (time 1788670890.68 total 1456.84 last 72.77) - open/close 49270 (time 1788670900.69 total 1466.85 last 70.21) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1472.05 seconds: 33.72 ops/second - unlinked 0 (time 1788670910 ; total 0 ; last 0) - unlinked 10000 (time 1788671057 ; total 147 ; last 147) - unlinked 20000 (time 1788671201 ; total 291 ; last 144) - unlinked 30000 (time 1788671353 ; total 443 ; last 152) - unlinked 40000 (time 1788671501 ; total 591 ; last 148) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 726 seconds: 68.362259 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg337-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg337-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg337-server stop mds service on oleg337-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg337-server stop mds service on oleg337-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg337-server unloading modules via unload_modules_local on: 'oleg337-server' oleg337-server: oleg337-server.virtnet: executing unload_modules_local oleg337-server: kmemleak disabled modules unloaded. pdsh@oleg337-client: oleg337-client: ssh exited with exit code 2 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 2 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 2 pdsh@oleg337-client: oleg337-client: ssh exited with exit code 2