== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:59:29 (1790737169) start mds service on oleg611-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1947032 Sep 29 13:13 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: kmemleak disabled oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions oleg611-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg611-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg611-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg611-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg611-server: oleg611-server.virtnet: executing set_default_debug -1 all pdsh@oleg611-client: oleg611-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg611-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg611-server: oleg611-server.virtnet: executing set_default_debug -1 all pdsh@oleg611-client: oleg611-server: ssh exited with exit code 1 Started lustre-OST0000 oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 128 (time 1790737257.69 total 10.01 last 12.78) - open/close 298 (time 1790737268.69 total 21.02 last 15.44) - open/close 437 (time 1790737278.70 total 31.02 last 13.90) - open/close 675 (time 1790737294.55 total 46.88 last 15.01) - open/close 896 (time 1790737304.89 total 57.22 last 21.36) - open/close 1116 (time 1790737315.03 total 67.36 last 21.71) - open/close 1377 (time 1790737327.64 total 79.97 last 20.69) - open/close 1589 (time 1790737337.65 total 89.98 last 21.18) - open/close 1853 (time 1790737347.65 total 99.98 last 26.40) - open/close 2061 (time 1790737357.66 total 109.99 last 20.78) - open/close 2206 (time 1790737367.66 total 119.99 last 14.50) - open/close 2408 (time 1790737380.51 total 132.84 last 15.73) - open/close 2646 (time 1790737390.74 total 143.07 last 23.25) - open/close 2866 (time 1790737400.75 total 153.08 last 21.98) - open/close 3148 (time 1790737410.77 total 163.10 last 28.14) - open/close 3183 (time 1790737420.90 total 173.23 last 3.46) - open/close 3414 (time 1790737430.92 total 183.24 last 23.06) - open/close 3593 (time 1790737443.44 total 195.77 last 14.29) - open/close 3855 (time 1790737453.45 total 205.78 last 26.17) - open/close 4096 (time 1790737463.96 total 216.29 last 22.94) - open/close 4351 (time 1790737473.97 total 226.29 last 25.48) - open/close 4575 (time 1790737484.39 total 236.72 last 21.49) - open/close 4851 (time 1790737495.46 total 247.79 last 24.93) - open/close 5040 (time 1790737506.07 total 258.40 last 17.81) - open/close 5317 (time 1790737516.77 total 269.09 last 25.90) - open/close 5430 (time 1790737527.32 total 279.64 last 10.71) - open/close 5631 (time 1790737537.88 total 290.21 last 19.03) - open/close 5875 (time 1790737547.89 total 300.22 last 24.37) - open/close 6101 (time 1790737557.90 total 310.23 last 22.58) - open/close 6296 (time 1790737567.92 total 320.25 last 19.46) - open/close 6576 (time 1790737577.93 total 330.26 last 27.97) - open/close 6843 (time 1790737588.89 total 341.22 last 24.36) - open/close 7151 (time 1790737599.14 total 351.46 last 30.07) - open/close 7364 (time 1790737609.15 total 361.48 last 21.27) - open/close 7528 (time 1790737619.16 total 371.48 last 16.39) - open/close 7776 (time 1790737630.18 total 382.50 last 22.50) - open/close 7951 (time 1790737640.60 total 392.93 last 16.78) - open/close 8238 (time 1790737650.62 total 402.95 last 28.66) - open/close 8390 (time 1790737660.64 total 412.97 last 15.16) - open/close 8578 (time 1790737670.66 total 422.98 last 18.77) - open/close 8731 (time 1790737681.11 total 433.44 last 14.64) - open/close 8926 (time 1790737692.18 total 444.51 last 17.61) - open/close 9134 (time 1790737702.19 total 454.52 last 20.77) - open/close 9361 (time 1790737712.23 total 464.56 last 22.60) - open/close 9504 (time 1790737722.47 total 474.80 last 13.97) - open/close 9765 (time 1790737733.11 total 485.44 last 24.53) - open/close 10000 (time 1790737740.34 total 492.67 last 32.48) - open/close 10246 (time 1790737750.35 total 502.68 last 24.58) - open/close 10512 (time 1790737760.36 total 512.69 last 26.57) - open/close 10741 (time 1790737770.78 total 523.11 last 21.98) - open/close 11015 (time 1790737780.80 total 533.13 last 27.35) - open/close 11218 (time 1790737791.71 total 544.04 last 18.60) - open/close 11452 (time 1790737802.40 total 554.73 last 21.90) - open/close 11701 (time 1790737812.96 total 565.29 last 23.58) - open/close 11899 (time 1790737823.17 total 575.50 last 19.40) - open/close 12030 (time 1790737833.17 total 585.50 last 13.10) - open/close 12267 (time 1790737843.18 total 595.51 last 23.68) - open/close 12446 (time 1790737854.06 total 606.39 last 16.45) - open/close 12672 (time 1790737864.98 total 617.31 last 20.71) - open/close 12899 (time 1790737877.72 total 630.05 last 17.81) - open/close 13096 (time 1790737888.41 total 640.74 last 18.42) - open/close 13358 (time 1790737899.52 total 651.85 last 23.60) - open/close 13561 (time 1790737911.29 total 663.62 last 17.24) - open/close 13757 (time 1790737921.38 total 673.71 last 19.43) - open/close 14029 (time 1790737932.45 total 684.77 last 24.58) - open/close 14246 (time 1790737942.46 total 694.79 last 21.67) - open/close 14462 (time 1790737952.47 total 704.80 last 21.57) - open/close 14619 (time 1790737962.48 total 714.81 last 15.69) - open/close 14848 (time 1790737972.48 total 724.81 last 22.90) - open/close 15091 (time 1790737983.00 total 735.33 last 23.10) - open/close 15331 (time 1790737993.01 total 745.33 last 23.98) - open/close 15568 (time 1790738003.01 total 755.34 last 23.68) - open/close 15866 (time 1790738013.01 total 765.34 last 29.80) - open/close 16107 (time 1790738023.34 total 775.67 last 23.34) - open/close 16337 (time 1790738033.83 total 786.16 last 21.93) - open/close 16628 (time 1790738044.53 total 796.86 last 27.18) - open/close 16810 (time 1790738055.13 total 807.46 last 17.18) - open/close 17071 (time 1790738065.14 total 817.46 last 26.08) - open/close 17186 (time 1790738075.94 total 828.27 last 10.64) - open/close 17456 (time 1790738085.96 total 838.29 last 26.94) - open/close 17700 (time 1790738096.61 total 848.94 last 22.92) - open/close 17864 (time 1790738107.62 total 859.95 last 14.89) - open/close 18098 (time 1790738119.71 total 872.04 last 19.35) - open/close 18305 (time 1790738129.73 total 882.06 last 20.65) - open/close 18431 (time 1790738140.92 total 893.25 last 11.26) - open/close 18614 (time 1790738150.93 total 903.26 last 18.29) - open/close 18852 (time 1790738161.02 total 913.34 last 23.59) - open/close 18974 (time 1790738171.02 total 923.35 last 12.19) - open/close 19200 (time 1790738183.45 total 935.78 last 18.18) - open/close 19691 (time 1790738193.46 total 945.79 last 49.08) - open/close 20000 (time 1790738197.28 total 949.61 last 80.77) - open/close 20691 (time 1790738207.29 total 959.62 last 69.06) - open/close 21407 (time 1790738217.30 total 969.63 last 71.49) - open/close 22125 (time 1790738227.31 total 979.64 last 71.76) - open/close 22826 (time 1790738237.34 total 989.67 last 69.88) - open/close 23576 (time 1790738247.36 total 999.68 last 74.88) - open/close 24220 (time 1790738257.36 total 1009.69 last 64.39) - open/close 24866 (time 1790738267.36 total 1019.69 last 64.58) - open/close 25517 (time 1790738277.36 total 1029.69 last 65.10) - open/close 26251 (time 1790738287.39 total 1039.72 last 73.21) - open/close 26958 (time 1790738297.39 total 1049.72 last 70.65) - open/close 27693 (time 1790738307.40 total 1059.73 last 73.48) - open/close 28363 (time 1790738317.40 total 1069.73 last 66.95) - open/close 28986 (time 1790738327.42 total 1079.75 last 62.22) - open/close 29655 (time 1790738337.42 total 1089.75 last 66.87) - open/close 30000 (time 1790738342.20 total 1094.53 last 72.15) - open/close 30722 (time 1790738352.21 total 1104.53 last 72.18) - open/close 31353 (time 1790738362.21 total 1114.54 last 63.07) - open/close 32029 (time 1790738372.21 total 1124.54 last 67.59) - open/close 32728 (time 1790738382.22 total 1134.54 last 69.87) - open/close 33413 (time 1790738392.22 total 1144.55 last 68.48) - open/close 34055 (time 1790738402.22 total 1154.55 last 64.17) - open/close 34792 (time 1790738412.22 total 1164.55 last 73.70) - open/close 35390 (time 1790738422.23 total 1174.56 last 59.76) - open/close 36082 (time 1790738432.24 total 1184.56 last 69.16) - open/close 36800 (time 1790738442.25 total 1194.58 last 71.68) - open/close 37489 (time 1790738452.30 total 1204.62 last 68.61) - open/close 38169 (time 1790738462.30 total 1214.63 last 67.97) - open/close 38837 (time 1790738472.30 total 1224.63 last 66.79) - open/close 39498 (time 1790738482.31 total 1234.64 last 66.05) - open/close 40000 (time 1790738489.17 total 1241.49 last 73.20) - open/close 40683 (time 1790738499.17 total 1251.50 last 68.29) - open/close 41422 (time 1790738509.18 total 1261.51 last 73.82) - open/close 42105 (time 1790738519.18 total 1271.51 last 68.27) - open/close 42725 (time 1790738529.21 total 1281.54 last 61.82) - open/close 43432 (time 1790738539.22 total 1291.54 last 70.67) - open/close 44138 (time 1790738549.22 total 1301.55 last 70.55) - open/close 44802 (time 1790738559.23 total 1311.56 last 66.38) - open/close 45494 (time 1790738569.23 total 1321.56 last 69.17) - open/close 46201 (time 1790738579.24 total 1331.56 last 70.67) - open/close 46874 (time 1790738589.24 total 1341.57 last 67.27) - open/close 47567 (time 1790738599.25 total 1351.58 last 69.25) - open/close 48348 (time 1790738609.26 total 1361.58 last 78.03) - open/close 49016 (time 1790738619.26 total 1371.59 last 66.77) - open/close 49625 (time 1790738629.27 total 1381.60 last 60.85) total: 50000 open/close in 1388.03 seconds: 36.02 ops/second - unlinked 0 (time 1790738640 ; total 0 ; last 0) - unlinked 10000 (time 1790738785 ; total 145 ; last 145) - unlinked 20000 (time 1790738934 ; total 294 ; last 149) - unlinked 30000 (time 1790739080 ; total 440 ; last 146) - unlinked 40000 (time 1790739236 ; total 596 ; last 156) total: 50000 unlinks in 740 seconds: 67.567566 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg611-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg611-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg611-server stop mds service on oleg611-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg611-server unloading modules via unload_modules_local on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing unload_modules_local oleg611-server: kmemleak disabled modules unloaded. pdsh@oleg611-client: oleg611-client: ssh exited with exit code 2 pdsh@oleg611-client: oleg611-server: ssh exited with exit code 2 pdsh@oleg611-client: oleg611-client: ssh exited with exit code 2 pdsh@oleg611-client: oleg611-server: ssh exited with exit code 2