== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:02:55 (1789408975) start mds service on oleg107-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 14 05:02 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing load_modules_local oleg107-server: kmemleak disabled oleg107-server: Loading modules from /home/green/git/lustre-release/lustre oleg107-server: detected 4 online CPUs by sysfs oleg107-server: MODOPTS_LIBCFS= oleg107-server: Force libcfs to create 2 CPU partitions oleg107-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg107-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg107-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg107-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg107-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg107-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg107-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg107-server: oleg107-server.virtnet: executing set_default_debug -1 all pdsh@oleg107-client: oleg107-server: ssh exited with exit code 1 Started lustre-OST0000 oleg107-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg107-client.virtnet: -o user_xattr,flock 192.168.201.107@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 165 (time 1789409090.10 total 10.00 last 16.50) - open/close 304 (time 1789409100.59 total 20.49 last 13.25) - open/close 495 (time 1789409111.46 total 31.36 last 17.57) - open/close 668 (time 1789409122.29 total 42.20 last 15.97) - open/close 845 (time 1789409132.31 total 52.22 last 17.67) - open/close 1068 (time 1789409143.00 total 62.90 last 20.87) - open/close 1168 (time 1789409153.02 total 72.93 last 9.97) - open/close 1435 (time 1789409163.03 total 82.93 last 26.69) - open/close 1653 (time 1789409173.28 total 93.18 last 21.27) - open/close 1786 (time 1789409183.42 total 103.33 last 13.11) - open/close 1905 (time 1789409194.25 total 114.15 last 10.99) - open/close 2043 (time 1789409204.25 total 124.15 last 13.80) - open/close 2174 (time 1789409214.34 total 134.25 last 12.98) - open/close 2320 (time 1789409224.90 total 144.80 last 13.84) - open/close 2504 (time 1789409235.53 total 155.43 last 17.31) - open/close 2642 (time 1789409246.35 total 166.25 last 12.76) - open/close 2818 (time 1789409256.64 total 176.55 last 17.09) - open/close 2948 (time 1789409266.64 total 186.55 last 13.00) - open/close 3109 (time 1789409276.67 total 196.58 last 16.06) - open/close 3227 (time 1789409286.68 total 206.58 last 11.79) - open/close 3423 (time 1789409296.71 total 216.61 last 19.55) - open/close 3573 (time 1789409307.46 total 227.36 last 13.95) - open/close 3747 (time 1789409317.48 total 237.38 last 17.37) - open/close 3883 (time 1789409327.61 total 247.52 last 13.42) - open/close 4023 (time 1789409338.45 total 258.36 last 12.92) - open/close 4150 (time 1789409348.47 total 268.38 last 12.67) - open/close 4350 (time 1789409359.20 total 279.11 last 18.64) - open/close 4567 (time 1789409369.23 total 289.13 last 21.64) - open/close 4669 (time 1789409382.94 total 302.85 last 7.44) - open/close 4782 (time 1789409392.95 total 312.85 last 11.29) - open/close 4882 (time 1789409403.65 total 323.55 last 9.34) - open/close 5010 (time 1789409413.67 total 333.58 last 12.77) - open/close 5113 (time 1789409424.70 total 344.61 last 9.34) - open/close 5185 (time 1789409434.71 total 354.62 last 7.19) - open/close 5392 (time 1789409444.87 total 364.77 last 20.39) - open/close 5626 (time 1789409454.87 total 374.78 last 23.38) - open/close 5827 (time 1789409465.15 total 385.06 last 19.56) - open/close 5996 (time 1789409475.16 total 395.07 last 16.88) - open/close 6138 (time 1789409485.18 total 405.09 last 14.17) - open/close 6284 (time 1789409495.32 total 415.23 last 14.40) - open/close 6359 (time 1789409505.36 total 425.26 last 7.47) - open/close 6550 (time 1789409515.36 total 435.26 last 19.09) - open/close 6677 (time 1789409525.42 total 445.32 last 12.62) - open/close 6828 (time 1789409536.18 total 456.08 last 14.04) - open/close 6984 (time 1789409546.82 total 466.72 last 14.66) - open/close 7106 (time 1789409557.77 total 477.68 last 11.14) - open/close 7297 (time 1789409568.10 total 488.00 last 18.50) - open/close 7454 (time 1789409578.15 total 498.06 last 15.62) - open/close 7622 (time 1789409589.15 total 509.05 last 15.28) - open/close 7701 (time 1789409599.82 total 519.73 last 7.40) - open/close 7846 (time 1789409609.87 total 529.77 last 14.44) - open/close 7946 (time 1789409620.83 total 540.74 last 9.12) - open/close 8000 (time 1789409631.69 total 551.59 last 4.98) - open/close 8115 (time 1789409642.11 total 562.01 last 11.04) - open/close 8229 (time 1789409652.74 total 572.65 last 10.72) - open/close 8433 (time 1789409663.81 total 583.71 last 18.44) - open/close 8593 (time 1789409674.63 total 594.53 last 14.79) - open/close 8657 (time 1789409684.69 total 604.60 last 6.36) - open/close 8868 (time 1789409695.48 total 615.39 last 19.55) - open/close 9000 (time 1789409705.51 total 625.41 last 13.17) - open/close 9135 (time 1789409715.52 total 635.42 last 13.49) - open/close 9248 (time 1789409726.99 total 646.89 last 9.85) - open/close 9333 (time 1789409738.85 total 658.76 last 7.17) - open/close 9427 (time 1789409748.93 total 668.84 last 9.33) - open/close 9543 (time 1789409758.94 total 678.84 last 11.59) - open/close 9656 (time 1789409769.43 total 689.33 last 10.77) - open/close 9798 (time 1789409779.52 total 699.43 last 14.07) - open/close 9877 (time 1789409789.83 total 709.73 last 7.66) - open/close 10000 (time 1789409798.68 total 718.58 last 13.90) - open/close 10216 (time 1789409809.41 total 729.32 last 20.13) - open/close 10345 (time 1789409819.43 total 739.34 last 12.87) - open/close 10482 (time 1789409829.90 total 749.80 last 13.09) - open/close 10540 (time 1789409840.64 total 760.55 last 5.40) - open/close 10679 (time 1789409850.65 total 770.56 last 13.89) - open/close 10808 (time 1789409860.66 total 780.56 last 12.90) - open/close 11053 (time 1789409870.66 total 790.56 last 24.49) - open/close 11278 (time 1789409881.38 total 801.28 last 20.99) - open/close 11393 (time 1789409892.08 total 811.99 last 10.74) - open/close 11495 (time 1789409903.00 total 822.91 last 9.34) - open/close 11650 (time 1789409913.73 total 833.63 last 14.45) - open/close 11773 (time 1789409923.97 total 843.87 last 12.01) - open/close 11931 (time 1789409934.62 total 854.53 last 14.83) - open/close 11996 (time 1789409944.79 total 864.70 last 6.39) - open/close 12141 (time 1789409954.81 total 874.72 last 14.47) - open/close 12257 (time 1789409965.08 total 884.99 last 11.29) - open/close 12375 (time 1789409975.38 total 895.29 last 11.46) - open/close 12525 (time 1789409985.71 total 905.61 last 14.53) - open/close 12706 (time 1789409996.38 total 916.29 last 16.96) - open/close 12880 (time 1789410006.68 total 926.59 last 16.89) - open/close 12993 (time 1789410017.73 total 937.63 last 10.23) - open/close 13196 (time 1789410028.80 total 948.70 last 18.34) - open/close 13279 (time 1789410039.11 total 959.01 last 8.05) - open/close 13485 (time 1789410049.77 total 969.67 last 19.32) - open/close 13633 (time 1789410060.70 total 980.61 last 13.53) - open/close 13709 (time 1789410070.88 total 990.78 last 7.47) - open/close 13963 (time 1789410080.88 total 1000.79 last 25.40) - open/close 14087 (time 1789410091.75 total 1011.65 last 11.41) - open/close 14229 (time 1789410103.54 total 1023.44 last 12.05) - open/close 14354 (time 1789410113.67 total 1033.57 last 12.34) - open/close 14465 (time 1789410124.37 total 1044.27 last 10.38) - open/close 14629 (time 1789410135.25 total 1055.16 last 15.07) - open/close 14763 (time 1789410145.27 total 1065.18 last 13.37) - open/close 14865 (time 1789410155.28 total 1075.19 last 10.19) - open/close 14922 (time 1789410166.23 total 1086.13 last 5.21) - open/close 15092 (time 1789410182.05 total 1101.95 last 10.74) - open/close 15246 (time 1789410192.91 total 1112.81 last 14.18) - open/close 15338 (time 1789410202.96 total 1122.86 last 9.15) - open/close 15404 (time 1789410212.96 total 1132.87 last 6.60) - open/close 15567 (time 1789410222.97 total 1142.87 last 16.29) - open/close 15791 (time 1789410232.97 total 1152.88 last 22.38) - open/close 15898 (time 1789410243.35 total 1163.26 last 10.31) - open/close 16037 (time 1789410253.64 total 1173.55 last 13.51) - open/close 16332 (time 1789410263.65 total 1183.56 last 29.47) - open/close 16982 (time 1789410273.66 total 1193.56 last 64.97) - open/close 17848 (time 1789410283.67 total 1203.58 last 86.48) - open/close 18643 (time 1789410293.67 total 1213.58 last 79.50) - open/close 19316 (time 1789410303.68 total 1223.58 last 67.28) - open/close 20000 (time 1789410313.61 total 1233.51 last 68.89) - open/close 20753 (time 1789410323.63 total 1243.54 last 75.08) - open/close 21303 (time 1789410333.66 total 1253.56 last 54.87) - open/close 22110 (time 1789410343.69 total 1263.59 last 80.45) - open/close 22835 (time 1789410353.69 total 1273.60 last 72.46) - open/close 23722 (time 1789410363.70 total 1283.60 last 88.68) - open/close 24555 (time 1789410373.71 total 1293.61 last 83.21) - open/close 25214 (time 1789410383.73 total 1303.64 last 65.72) - open/close 26002 (time 1789410393.75 total 1313.65 last 78.69) - open/close 26728 (time 1789410403.76 total 1323.66 last 72.54) - open/close 27340 (time 1789410413.76 total 1333.67 last 61.15) - open/close 27909 (time 1789410423.78 total 1343.68 last 56.83) - open/close 28492 (time 1789410433.80 total 1353.70 last 58.16) - open/close 29144 (time 1789410443.82 total 1363.73 last 65.05) - open/close 29773 (time 1789410453.83 total 1373.73 last 62.87) - open/close 30000 (time 1789410457.33 total 1377.24 last 64.78) - open/close 30728 (time 1789410467.34 total 1387.25 last 72.73) - open/close 31368 (time 1789410477.35 total 1397.26 last 63.92) - open/close 31875 (time 1789410487.36 total 1407.26 last 50.68) - open/close 32510 (time 1789410497.37 total 1417.28 last 63.39) - open/close 33080 (time 1789410507.39 total 1427.29 last 56.91) - open/close 33841 (time 1789410517.40 total 1437.30 last 76.06) - open/close 34576 (time 1789410527.41 total 1447.31 last 73.41) - open/close 35332 (time 1789410537.41 total 1457.32 last 75.55) - open/close 36074 (time 1789410547.42 total 1467.32 last 74.17) - open/close 36763 (time 1789410557.42 total 1477.33 last 68.88) - open/close 37467 (time 1789410567.43 total 1487.34 last 70.33) - open/close 38223 (time 1789410577.44 total 1497.34 last 75.55) - open/close 38998 (time 1789410587.44 total 1507.35 last 77.44) - open/close 39617 (time 1789410597.45 total 1517.36 last 61.86) - open/close 40000 (time 1789410603.34 total 1523.24 last 65.04) - open/close 40580 (time 1789410613.34 total 1533.25 last 57.97) - open/close 41271 (time 1789410623.36 total 1543.26 last 69.02) - open/close 41816 (time 1789410633.36 total 1553.27 last 54.48) - open/close 42539 (time 1789410643.36 total 1563.27 last 72.28) - open/close 43237 (time 1789410653.36 total 1573.27 last 69.79) - open/close 43918 (time 1789410663.37 total 1583.28 last 68.04) - open/close 44562 (time 1789410673.38 total 1593.29 last 64.33) - open/close 45254 (time 1789410683.39 total 1603.29 last 69.17) - open/close 45752 (time 1789410693.40 total 1613.30 last 49.77) - open/close 46252 (time 1789410703.41 total 1623.32 last 49.92) - open/close 46716 (time 1789410713.42 total 1633.33 last 46.35) - open/close 47209 (time 1789410723.44 total 1643.35 last 49.21) - open/close 47639 (time 1789410733.47 total 1653.37 last 42.88) - open/close 48090 (time 1789410743.48 total 1663.38 last 45.06) - open/close 48701 (time 1789410753.49 total 1673.40 last 61.01) - open/close 49307 (time 1789410763.50 total 1683.41 last 60.56) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1687.51 seconds: 29.41 ops/second - unlinked 0 (time 1789410772 ; total 0 ; last 0) - unlinked 10000 (time 1789410933 ; total 161 ; last 161) - unlinked 20000 (time 1789411073 ; total 301 ; last 140) - unlinked 30000 (time 1789411213 ; total 441 ; last 140) - unlinked 40000 (time 1789411367 ; total 595 ; last 154) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 718 seconds: 69.123955 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg107-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg107-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg107-server stop mds service on oleg107-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg107-server unloading modules via unload_modules_local on: 'oleg107-server' oleg107-server: oleg107-server.virtnet: executing unload_modules_local oleg107-server: kmemleak disabled modules unloaded. pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-client: ssh exited with exit code 2 pdsh@oleg107-client: oleg107-server: ssh exited with exit code 2