== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:01:38 (1790100098) start mds service on oleg421-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1940568 Sep 22 12:15 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing load_modules_local oleg421-server: kmemleak disabled oleg421-server: Loading modules from /home/green/git/lustre-release/lustre oleg421-server: detected 4 online CPUs by sysfs oleg421-server: MODOPTS_LIBCFS= oleg421-server: Force libcfs to create 2 CPU partitions oleg421-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg421-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg421-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg421-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg421-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg421-client.virtnet: -o user_xattr,flock 192.168.204.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 194 (time 1790100184.60 total 10.01 last 19.39) - open/close 357 (time 1790100196.14 total 21.55 last 14.12) - open/close 521 (time 1790100206.19 total 31.60 last 16.33) - open/close 667 (time 1790100216.49 total 41.89 last 14.18) - open/close 854 (time 1790100226.50 total 51.90 last 18.68) - open/close 1038 (time 1790100236.52 total 61.93 last 18.35) - open/close 1191 (time 1790100247.02 total 72.43 last 14.57) - open/close 1285 (time 1790100257.59 total 83.00 last 8.90) - open/close 1420 (time 1790100267.59 total 93.00 last 13.49) - open/close 1625 (time 1790100278.63 total 104.04 last 18.57) - open/close 1824 (time 1790100288.64 total 114.05 last 19.88) - open/close 1999 (time 1790100299.05 total 124.46 last 16.82) - open/close 2133 (time 1790100310.06 total 135.47 last 12.17) - open/close 2324 (time 1790100320.94 total 146.35 last 17.56) - open/close 2574 (time 1790100331.38 total 156.78 last 23.95) - open/close 2727 (time 1790100341.49 total 166.90 last 15.12) - open/close 2797 (time 1790100351.66 total 177.06 last 6.89) - open/close 2933 (time 1790100361.72 total 187.13 last 13.52) - open/close 3031 (time 1790100372.52 total 197.93 last 9.07) - open/close 3147 (time 1790100382.53 total 207.93 last 11.60) - open/close 3275 (time 1790100392.54 total 217.95 last 12.78) - open/close 3417 (time 1790100402.55 total 227.96 last 14.19) - open/close 3523 (time 1790100413.23 total 238.64 last 9.93) - open/close 3668 (time 1790100423.24 total 248.65 last 14.48) - open/close 3851 (time 1790100433.26 total 258.67 last 18.27) - open/close 3996 (time 1790100443.27 total 268.68 last 14.49) - open/close 4132 (time 1790100454.16 total 279.57 last 12.49) - open/close 4277 (time 1790100464.17 total 289.58 last 14.48) - open/close 4407 (time 1790100474.18 total 299.59 last 12.99) - open/close 4541 (time 1790100484.19 total 309.60 last 13.39) - open/close 4685 (time 1790100494.71 total 320.12 last 13.69) - open/close 4807 (time 1790100505.04 total 330.45 last 11.81) - open/close 4961 (time 1790100515.07 total 340.48 last 15.36) - open/close 5127 (time 1790100525.09 total 350.50 last 16.56) - open/close 5299 (time 1790100535.10 total 360.51 last 17.18) - open/close 5384 (time 1790100546.18 total 371.58 last 7.68) - open/close 5538 (time 1790100557.54 total 382.95 last 13.55) - open/close 5723 (time 1790100568.25 total 393.66 last 17.28) - open/close 5791 (time 1790100579.26 total 404.67 last 6.18) - open/close 5902 (time 1790100589.36 total 414.77 last 10.99) - open/close 6045 (time 1790100599.37 total 424.77 last 14.30) - open/close 6119 (time 1790100609.82 total 435.23 last 7.08) - open/close 6278 (time 1790100619.83 total 445.23 last 15.89) - open/close 6422 (time 1790100630.27 total 455.68 last 13.78) - open/close 6643 (time 1790100640.28 total 465.69 last 22.09) - open/close 6780 (time 1790100652.63 total 478.04 last 11.09) - open/close 7025 (time 1790100663.10 total 488.51 last 23.40) - open/close 7064 (time 1790100673.45 total 498.86 last 3.77) - open/close 7160 (time 1790100684.91 total 510.32 last 8.38) - open/close 7359 (time 1790100695.18 total 520.59 last 19.38) - open/close 7576 (time 1790100705.21 total 530.62 last 21.63) - open/close 7736 (time 1790100715.21 total 540.62 last 16.00) - open/close 7912 (time 1790100725.23 total 550.64 last 17.57) - open/close 7991 (time 1790100735.34 total 560.75 last 7.81) - open/close 8212 (time 1790100746.11 total 571.52 last 20.52) - open/close 8398 (time 1790100756.14 total 581.55 last 18.54) - open/close 8625 (time 1790100766.57 total 591.98 last 21.77) - open/close 8728 (time 1790100776.81 total 602.22 last 10.05) - open/close 8884 (time 1790100786.85 total 612.26 last 15.55) - open/close 9028 (time 1790100796.86 total 622.27 last 14.38) - open/close 9069 (time 1790100807.25 total 632.66 last 3.95) - open/close 9166 (time 1790100819.38 total 644.79 last 7.99) - open/close 9310 (time 1790100830.07 total 655.48 last 13.47) - open/close 9396 (time 1790100840.62 total 666.03 last 8.15) - open/close 9496 (time 1790100851.08 total 676.49 last 9.56) - open/close 9720 (time 1790100863.96 total 689.36 last 17.40) - open/close 9895 (time 1790100873.96 total 699.37 last 17.49) - open/close 10000 (time 1790100878.55 total 703.96 last 22.88) - open/close 10237 (time 1790100888.57 total 713.98 last 23.66) - open/close 10426 (time 1790100898.82 total 724.23 last 18.43) - open/close 10664 (time 1790100909.63 total 735.03 last 22.03) - open/close 10865 (time 1790100920.51 total 745.92 last 18.47) - open/close 11041 (time 1790100930.51 total 755.92 last 17.60) - open/close 11208 (time 1790100940.52 total 765.93 last 16.69) - open/close 11374 (time 1790100951.19 total 776.60 last 15.56) - open/close 11538 (time 1790100961.79 total 787.20 last 15.47) - open/close 11666 (time 1790100972.22 total 797.63 last 12.28) - open/close 11882 (time 1790100982.22 total 807.63 last 21.58) - open/close 12010 (time 1790100994.32 total 819.73 last 10.58) - open/close 12231 (time 1790101004.32 total 829.73 last 22.10) - open/close 12358 (time 1790101014.33 total 839.74 last 12.69) - open/close 12588 (time 1790101024.33 total 849.74 last 22.99) - open/close 12851 (time 1790101034.34 total 859.75 last 26.28) - open/close 13049 (time 1790101045.36 total 870.77 last 17.97) - open/close 13261 (time 1790101055.37 total 880.77 last 21.19) - open/close 13474 (time 1790101065.70 total 891.11 last 20.60) - open/close 13720 (time 1790101075.90 total 901.31 last 24.14) - open/close 14261 (time 1790101085.90 total 911.31 last 54.09) - open/close 14873 (time 1790101095.91 total 921.32 last 61.13) - open/close 15520 (time 1790101105.92 total 931.33 last 64.62) - open/close 16180 (time 1790101115.92 total 941.33 last 66.00) - open/close 16833 (time 1790101125.92 total 951.33 last 65.30) - open/close 17461 (time 1790101135.94 total 961.34 last 62.72) - open/close 18097 (time 1790101145.95 total 971.35 last 63.54) - open/close 18726 (time 1790101155.95 total 981.35 last 62.90) - open/close 19384 (time 1790101165.96 total 991.37 last 65.73) - open/close 20000 (time 1790101175.74 total 1001.15 last 62.96) - open/close 20647 (time 1790101185.83 total 1011.24 last 64.10) - open/close 21264 (time 1790101195.84 total 1021.24 last 61.69) - open/close 21920 (time 1790101205.84 total 1031.25 last 65.56) - open/close 22597 (time 1790101215.86 total 1041.27 last 67.56) - open/close 23299 (time 1790101225.87 total 1051.28 last 70.14) - open/close 23969 (time 1790101235.89 total 1061.30 last 66.88) - open/close 24678 (time 1790101245.90 total 1071.31 last 70.80) - open/close 25345 (time 1790101255.91 total 1081.32 last 66.67) - open/close 26008 (time 1790101265.91 total 1091.32 last 66.30) - open/close 26644 (time 1790101275.91 total 1101.32 last 63.59) - open/close 27347 (time 1790101285.91 total 1111.32 last 70.28) - open/close 28013 (time 1790101295.94 total 1121.34 last 66.44) - open/close 28655 (time 1790101305.95 total 1131.35 last 64.13) - open/close 29267 (time 1790101315.95 total 1141.36 last 61.18) - open/close 29899 (time 1790101325.95 total 1151.36 last 63.20) - open/close 30508 (time 1790101335.96 total 1161.37 last 60.82) - open/close 31158 (time 1790101345.97 total 1171.38 last 64.98) - open/close 31796 (time 1790101355.98 total 1181.39 last 63.74) - open/close 32434 (time 1790101365.98 total 1191.39 last 63.78) - open/close 33043 (time 1790101376.00 total 1201.41 last 60.75) - open/close 33653 (time 1790101386.01 total 1211.42 last 60.96) - open/close 34237 (time 1790101396.02 total 1221.43 last 58.35) - open/close 34842 (time 1790101406.03 total 1231.44 last 60.44) - open/close 35433 (time 1790101416.04 total 1241.44 last 59.06) - open/close 36095 (time 1790101426.04 total 1251.45 last 66.19) - open/close 36679 (time 1790101436.04 total 1261.45 last 58.38) - open/close 37293 (time 1790101446.05 total 1271.46 last 61.34) - open/close 37914 (time 1790101456.06 total 1281.47 last 62.04) - open/close 38584 (time 1790101466.09 total 1291.50 last 66.81) - open/close 39252 (time 1790101476.10 total 1301.50 last 66.75) - open/close 39840 (time 1790101486.10 total 1311.51 last 58.75) - open/close 40000 (time 1790101488.76 total 1314.17 last 60.28) - open/close 40618 (time 1790101498.77 total 1324.18 last 61.72) - open/close 41269 (time 1790101508.78 total 1334.18 last 65.08) - open/close 41876 (time 1790101518.78 total 1344.19 last 60.69) - open/close 42447 (time 1790101528.78 total 1354.19 last 57.06) - open/close 43048 (time 1790101538.80 total 1364.21 last 59.98) - open/close 43674 (time 1790101548.81 total 1374.22 last 62.53) - open/close 44328 (time 1790101558.82 total 1384.23 last 65.35) - open/close 44949 (time 1790101568.85 total 1394.26 last 61.90) - open/close 45564 (time 1790101578.86 total 1404.27 last 61.45) - open/close 46206 (time 1790101588.87 total 1414.28 last 64.15) - open/close 46875 (time 1790101598.87 total 1424.28 last 66.86) - open/close 47475 (time 1790101608.90 total 1434.31 last 59.85) - open/close 48120 (time 1790101618.91 total 1444.32 last 64.42) - open/close 48739 (time 1790101628.92 total 1454.33 last 61.84) - open/close 49356 (time 1790101638.93 total 1464.34 last 61.66) total: 50000 open/close in 1473.85 seconds: 33.92 ops/second - unlinked 0 (time 1790101651 ; total 0 ; last 0) - unlinked 10000 (time 1790101801 ; total 150 ; last 150) - unlinked 20000 (time 1790101964 ; total 313 ; last 163) - unlinked 30000 (time 1790102130 ; total 479 ; last 166) - unlinked 40000 (time 1790102298 ; total 647 ; last 168) total: 50000 unlinks in 816 seconds: 61.274509 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg421-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg421-server unloading modules via unload_modules_local on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing unload_modules_local oleg421-server: kmemleak disabled modules unloaded. pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2