== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:37:37 (1790923057) start mds service on oleg232-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1946968 Oct 1 19:15 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: kmemleak disabled oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: MODOPTS_LIBCFS= oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg232-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg232-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg232-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg232-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg232-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 175 (time 1790923191.07 total 13.03 last 13.43) - open/close 263 (time 1790923202.13 total 24.09 last 7.96) - open/close 409 (time 1790923213.45 total 35.41 last 12.89) - open/close 646 (time 1790923223.49 total 45.45 last 23.62) - open/close 784 (time 1790923233.50 total 55.46 last 13.78) - open/close 930 (time 1790923244.81 total 66.77 last 12.91) - open/close 1125 (time 1790923254.82 total 76.77 last 19.49) - open/close 1299 (time 1790923265.81 total 87.77 last 15.83) - open/close 1526 (time 1790923275.81 total 97.77 last 22.69) - open/close 1769 (time 1790923286.05 total 108.00 last 23.75) - open/close 1863 (time 1790923296.48 total 118.44 last 9.00) - open/close 1963 (time 1790923306.49 total 128.45 last 9.99) - open/close 2125 (time 1790923317.19 total 139.14 last 15.15) - open/close 2155 (time 1790923327.38 total 149.33 last 2.94) - open/close 2331 (time 1790923337.40 total 159.35 last 17.57) - open/close 2551 (time 1790923347.40 total 169.36 last 21.98) - open/close 2712 (time 1790923357.82 total 179.77 last 15.46) - open/close 2886 (time 1790923367.82 total 189.78 last 17.40) - open/close 3087 (time 1790923378.88 total 200.84 last 18.17) - open/close 3238 (time 1790923389.83 total 211.78 last 13.80) - open/close 3326 (time 1790923400.60 total 222.56 last 8.17) - open/close 3430 (time 1790923410.60 total 232.56 last 10.39) - open/close 3568 (time 1790923420.90 total 242.85 last 13.41) - open/close 3826 (time 1790923430.90 total 252.85 last 25.80) - open/close 4037 (time 1790923441.54 total 263.50 last 19.83) - open/close 4300 (time 1790923451.80 total 273.76 last 25.63) - open/close 4493 (time 1790923462.05 total 284.00 last 18.84) - open/close 4627 (time 1790923473.04 total 295.00 last 12.19) - open/close 4809 (time 1790923483.05 total 305.00 last 18.19) - open/close 4936 (time 1790923493.83 total 315.79 last 11.78) - open/close 5026 (time 1790923504.20 total 326.16 last 8.67) - open/close 5192 (time 1790923514.91 total 336.86 last 15.51) - open/close 5355 (time 1790923525.98 total 347.94 last 14.71) - open/close 5534 (time 1790923536.00 total 357.96 last 17.86) - open/close 5673 (time 1790923546.52 total 368.48 last 13.21) - open/close 5846 (time 1790923556.54 total 378.49 last 17.28) - open/close 5931 (time 1790923566.69 total 388.65 last 8.37) - open/close 6168 (time 1790923576.79 total 398.74 last 23.48) - open/close 6351 (time 1790923586.80 total 408.75 last 18.28) - open/close 6489 (time 1790923596.80 total 418.76 last 13.79) - open/close 6679 (time 1790923606.81 total 428.77 last 18.98) - open/close 6899 (time 1790923617.62 total 439.57 last 20.36) - open/close 6969 (time 1790923628.92 total 450.88 last 6.19) - open/close 7102 (time 1790923639.56 total 461.52 last 12.50) - open/close 7199 (time 1790923650.38 total 472.34 last 8.97) - open/close 7281 (time 1790923660.90 total 482.86 last 7.80) - open/close 7372 (time 1790923671.05 total 493.01 last 8.96) - open/close 7486 (time 1790923681.05 total 503.01 last 11.40) - open/close 7680 (time 1790923691.22 total 513.17 last 19.09) - open/close 7755 (time 1790923701.22 total 523.18 last 7.50) - open/close 7961 (time 1790923711.45 total 533.40 last 20.14) - open/close 8062 (time 1790923722.34 total 544.30 last 9.27) - open/close 8204 (time 1790923732.76 total 554.71 last 13.63) - open/close 8437 (time 1790923743.18 total 565.14 last 22.35) - open/close 8603 (time 1790923753.98 total 575.94 last 15.37) - open/close 8761 (time 1790923764.00 total 585.96 last 15.77) - open/close 8862 (time 1790923774.18 total 596.13 last 9.93) - open/close 9069 (time 1790923784.18 total 606.14 last 20.69) - open/close 9310 (time 1790923795.00 total 616.95 last 22.28) - open/close 9521 (time 1790923805.01 total 626.97 last 21.08) - open/close 9673 (time 1790923815.03 total 636.99 last 15.17) - open/close 9805 (time 1790923825.42 total 647.37 last 12.71) - open/close 9870 (time 1790923835.42 total 657.38 last 6.50) - open/close 10000 (time 1790923841.17 total 663.13 last 22.60) - open/close 10212 (time 1790923851.19 total 673.15 last 21.16) - open/close 10347 (time 1790923861.85 total 683.81 last 12.66) - open/close 10519 (time 1790923872.23 total 694.19 last 16.56) - open/close 10646 (time 1790923882.25 total 704.21 last 12.68) - open/close 10747 (time 1790923894.87 total 716.83 last 8.00) - open/close 11006 (time 1790923905.95 total 727.91 last 23.38) - open/close 11073 (time 1790923915.96 total 737.92 last 6.70) - open/close 11303 (time 1790923926.86 total 748.81 last 21.10) - open/close 11442 (time 1790923937.82 total 759.77 last 12.68) - open/close 11506 (time 1790923947.93 total 769.88 last 6.33) - open/close 11698 (time 1790923957.95 total 779.91 last 19.15) - open/close 11842 (time 1790923968.99 total 790.94 last 13.05) - open/close 11920 (time 1790923979.21 total 801.17 last 7.63) - open/close 12036 (time 1790923990.28 total 812.24 last 10.48) - open/close 12256 (time 1790924000.29 total 822.24 last 21.99) - open/close 12475 (time 1790924011.01 total 832.97 last 20.41) - open/close 12618 (time 1790924021.62 total 843.58 last 13.48) - open/close 12779 (time 1790924031.63 total 853.59 last 16.09) - open/close 12943 (time 1790924042.38 total 864.34 last 15.26) - open/close 13113 (time 1790924052.39 total 874.35 last 16.98) - open/close 13234 (time 1790924062.41 total 884.36 last 12.08) - open/close 13498 (time 1790924072.41 total 894.36 last 26.40) - open/close 13646 (time 1790924082.66 total 904.61 last 14.44) - open/close 13749 (time 1790924092.88 total 914.83 last 10.08) - open/close 13897 (time 1790924103.47 total 925.43 last 13.97) - open/close 14023 (time 1790924113.47 total 935.43 last 12.60) - open/close 14183 (time 1790924124.18 total 946.14 last 14.94) - open/close 14406 (time 1790924134.73 total 956.69 last 21.14) - open/close 14647 (time 1790924145.03 total 966.99 last 23.38) - open/close 14902 (time 1790924155.04 total 976.99 last 25.50) - open/close 15094 (time 1790924165.71 total 987.67 last 17.99) - open/close 15258 (time 1790924176.45 total 998.41 last 15.27) - open/close 15432 (time 1790924186.46 total 1008.41 last 17.39) - open/close 15590 (time 1790924197.22 total 1019.17 last 14.69) - open/close 15678 (time 1790924207.76 total 1029.72 last 8.34) - open/close 15794 (time 1790924217.77 total 1039.72 last 11.60) - open/close 16000 (time 1790924227.77 total 1049.73 last 20.59) - open/close 16132 (time 1790924238.82 total 1060.77 last 11.95) - open/close 16292 (time 1790924250.02 total 1071.98 last 14.28) - open/close 16452 (time 1790924260.75 total 1082.70 last 14.91) - open/close 16628 (time 1790924271.76 total 1093.71 last 15.99) - open/close 17164 (time 1790924281.76 total 1103.72 last 53.58) - open/close 17818 (time 1790924291.77 total 1113.72 last 65.36) - open/close 18548 (time 1790924301.77 total 1123.73 last 72.98) - open/close 19338 (time 1790924311.77 total 1133.73 last 78.97) - open/close 20000 (time 1790924320.73 total 1142.69 last 73.88) - open/close 20736 (time 1790924330.74 total 1152.70 last 73.53) - open/close 21483 (time 1790924340.75 total 1162.71 last 74.63) - open/close 22076 (time 1790924350.77 total 1172.72 last 59.21) - open/close 22561 (time 1790924360.79 total 1182.74 last 48.40) - open/close 22992 (time 1790924370.80 total 1192.76 last 43.04) - open/close 23634 (time 1790924380.80 total 1202.76 last 64.20) - open/close 24256 (time 1790924390.82 total 1212.78 last 62.08) - open/close 24945 (time 1790924400.82 total 1222.78 last 68.89) - open/close 25548 (time 1790924410.82 total 1232.78 last 60.28) - open/close 26113 (time 1790924420.84 total 1242.80 last 56.41) - open/close 26732 (time 1790924430.85 total 1252.81 last 61.85) - open/close 27346 (time 1790924440.86 total 1262.82 last 61.35) - open/close 27918 (time 1790924450.86 total 1272.82 last 57.18) - open/close 28408 (time 1790924460.87 total 1282.82 last 48.98) - open/close 28955 (time 1790924470.87 total 1292.83 last 54.66) - open/close 29590 (time 1790924480.89 total 1302.84 last 63.43) - open/close 30000 (time 1790924487.51 total 1309.46 last 61.93) - open/close 30551 (time 1790924497.51 total 1319.46 last 55.10) - open/close 31207 (time 1790924507.51 total 1329.47 last 65.59) - open/close 31938 (time 1790924517.51 total 1339.47 last 73.07) - open/close 32657 (time 1790924527.51 total 1349.47 last 71.90) - open/close 33364 (time 1790924537.52 total 1359.47 last 70.66) - open/close 34075 (time 1790924547.53 total 1369.49 last 71.02) - open/close 34716 (time 1790924557.53 total 1379.49 last 64.06) - open/close 35311 (time 1790924567.55 total 1389.51 last 59.42) - open/close 35926 (time 1790924577.55 total 1399.51 last 61.48) - open/close 36512 (time 1790924587.58 total 1409.54 last 58.44) - open/close 37112 (time 1790924597.59 total 1419.55 last 59.93) - open/close 37688 (time 1790924607.60 total 1429.56 last 57.52) - open/close 38068 (time 1790924617.61 total 1439.57 last 37.96) - open/close 38694 (time 1790924627.62 total 1449.58 last 62.55) - open/close 39289 (time 1790924637.62 total 1459.58 last 59.49) - open/close 39969 (time 1790924647.63 total 1469.59 last 67.96) - open/close 40496 (time 1790924657.64 total 1479.60 last 52.62) - open/close 41109 (time 1790924667.65 total 1489.61 last 61.25) - open/close 41746 (time 1790924677.67 total 1499.63 last 63.60) - open/close 42374 (time 1790924687.67 total 1509.63 last 62.79) - open/close 42916 (time 1790924697.68 total 1519.64 last 54.13) - open/close 43495 (time 1790924707.69 total 1529.64 last 57.88) - open/close 44145 (time 1790924717.70 total 1539.66 last 64.89) - open/close 44689 (time 1790924727.72 total 1549.67 last 54.33) - open/close 45389 (time 1790924737.72 total 1559.68 last 69.97) - open/close 46061 (time 1790924747.74 total 1569.70 last 67.06) - open/close 46824 (time 1790924757.75 total 1579.71 last 76.22) - open/close 47472 (time 1790924767.77 total 1589.73 last 64.66) - open/close 48147 (time 1790924777.78 total 1599.74 last 67.46) - open/close 48729 (time 1790924787.79 total 1609.75 last 58.12) - open/close 49172 (time 1790924797.82 total 1619.78 last 44.18) - open/close 49693 (time 1790924807.87 total 1629.83 last 51.83) total: 50000 open/close in 1635.84 seconds: 30.57 ops/second - unlinked 0 (time 1790924819 ; total 0 ; last 0) - unlinked 10000 (time 1790924963 ; total 144 ; last 144) - unlinked 20000 (time 1790925111 ; total 292 ; last 148) - unlinked 30000 (time 1790925251 ; total 432 ; last 140) - unlinked 40000 (time 1790925403 ; total 584 ; last 152) total: 50000 unlinks in 724 seconds: 69.060776 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg232-server unloading modules via unload_modules_local on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local oleg232-server: kmemleak disabled modules unloaded. pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2