== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:59:11 (1788627551) start mds service on oleg460-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1921056 Sep 4 17:56 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing load_modules_local oleg460-server: kmemleak disabled oleg460-server: Loading modules from /home/green/git/lustre-release/lustre oleg460-server: detected 4 online CPUs by sysfs oleg460-server: MODOPTS_LIBCFS= oleg460-server: Force libcfs to create 2 CPU partitions oleg460-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg460-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg460-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg460-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg460-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg460-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg460-server: oleg460-server.virtnet: executing set_default_debug -1 all pdsh@oleg460-client: oleg460-server: ssh exited with exit code 1 Started lustre-OST0000 oleg460-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg460-client.virtnet: -o user_xattr,flock 192.168.204.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 248 (time 1788627624.89 total 10.87 last 22.81) - open/close 428 (time 1788627635.40 total 21.38 last 17.14) - open/close 610 (time 1788627645.57 total 31.55 last 17.89) - open/close 769 (time 1788627655.80 total 41.79 last 15.54) - open/close 939 (time 1788627666.49 total 52.48 last 15.90) - open/close 1160 (time 1788627676.81 total 62.80 last 21.41) - open/close 1376 (time 1788627687.18 total 73.16 last 20.84) - open/close 1638 (time 1788627697.19 total 83.18 last 26.16) - open/close 1831 (time 1788627707.20 total 93.18 last 19.29) - open/close 2050 (time 1788627717.77 total 103.75 last 20.72) - open/close 2289 (time 1788627727.80 total 113.79 last 23.82) - open/close 2541 (time 1788627737.81 total 123.79 last 25.18) - open/close 2772 (time 1788627747.82 total 133.81 last 23.07) - open/close 2959 (time 1788627757.83 total 143.81 last 18.69) - open/close 3165 (time 1788627768.20 total 154.18 last 19.87) - open/close 3428 (time 1788627778.20 total 164.19 last 26.28) - open/close 3657 (time 1788627788.23 total 174.21 last 22.84) - open/close 3937 (time 1788627798.47 total 184.45 last 27.34) - open/close 4130 (time 1788627808.64 total 194.63 last 18.97) - open/close 4326 (time 1788627818.65 total 204.64 last 19.58) - open/close 4624 (time 1788627828.68 total 214.67 last 29.71) - open/close 4857 (time 1788627839.18 total 225.16 last 22.20) - open/close 5107 (time 1788627849.19 total 235.18 last 24.96) - open/close 5345 (time 1788627860.24 total 246.22 last 21.55) - open/close 5560 (time 1788627870.26 total 256.24 last 21.47) - open/close 5806 (time 1788627880.26 total 266.24 last 24.58) - open/close 5940 (time 1788627890.27 total 276.26 last 13.38) - open/close 6117 (time 1788627900.94 total 286.92 last 16.60) - open/close 6284 (time 1788627911.69 total 297.67 last 15.54) - open/close 6522 (time 1788627921.71 total 307.69 last 23.75) - open/close 6781 (time 1788627932.38 total 318.36 last 24.27) - open/close 7046 (time 1788627943.06 total 329.05 last 24.80) - open/close 7236 (time 1788627953.07 total 339.05 last 18.98) - open/close 7425 (time 1788627963.55 total 349.53 last 18.04) - open/close 7592 (time 1788627973.70 total 359.68 last 16.45) - open/close 7776 (time 1788627984.41 total 370.39 last 17.19) - open/close 8079 (time 1788627994.42 total 380.40 last 30.26) - open/close 8317 (time 1788628005.06 total 391.04 last 22.36) - open/close 8544 (time 1788628015.64 total 401.62 last 21.46) - open/close 8675 (time 1788628025.65 total 411.63 last 13.09) - open/close 8895 (time 1788628036.43 total 422.41 last 20.41) - open/close 9153 (time 1788628046.45 total 432.43 last 25.74) - open/close 9391 (time 1788628057.34 total 443.32 last 21.85) - open/close 9671 (time 1788628067.34 total 453.33 last 27.99) - open/close 9898 (time 1788628077.44 total 463.42 last 22.49) - open/close 10000 (time 1788628083.46 total 469.44 last 16.94) - open/close 10153 (time 1788628094.15 total 480.13 last 14.31) - open/close 10317 (time 1788628104.16 total 490.15 last 16.38) - open/close 10566 (time 1788628114.17 total 500.15 last 24.89) - open/close 10730 (time 1788628124.65 total 510.63 last 15.64) - open/close 10967 (time 1788628134.66 total 520.64 last 23.68) - open/close 11169 (time 1788628145.03 total 531.01 last 19.48) - open/close 11334 (time 1788628155.49 total 541.47 last 15.78) - open/close 11487 (time 1788628166.47 total 552.45 last 13.94) - open/close 11594 (time 1788628177.93 total 563.91 last 9.33) - open/close 11810 (time 1788628188.17 total 574.15 last 21.10) - open/close 11955 (time 1788628198.54 total 584.52 last 13.97) - open/close 12068 (time 1788628208.55 total 594.53 last 11.29) - open/close 12312 (time 1788628218.90 total 604.88 last 23.56) - open/close 12552 (time 1788628229.46 total 615.44 last 22.73) - open/close 12774 (time 1788628239.62 total 625.61 last 21.84) - open/close 13010 (time 1788628249.64 total 635.62 last 23.56) - open/close 13246 (time 1788628260.29 total 646.27 last 22.16) - open/close 13497 (time 1788628270.37 total 656.35 last 24.90) - open/close 13766 (time 1788628280.41 total 666.39 last 26.81) - open/close 14025 (time 1788628291.01 total 677.00 last 24.41) - open/close 14239 (time 1788628301.25 total 687.23 last 20.91) - open/close 14461 (time 1788628311.25 total 697.23 last 22.20) - open/close 14674 (time 1788628321.80 total 707.78 last 20.19) - open/close 14921 (time 1788628331.81 total 717.79 last 24.68) - open/close 15120 (time 1788628341.83 total 727.81 last 19.87) - open/close 15366 (time 1788628352.14 total 738.12 last 23.85) - open/close 15650 (time 1788628362.96 total 748.94 last 26.25) - open/close 15866 (time 1788628373.25 total 759.24 last 20.99) - open/close 16119 (time 1788628383.27 total 769.26 last 25.25) - open/close 16341 (time 1788628393.35 total 779.34 last 22.03) - open/close 16611 (time 1788628403.36 total 789.34 last 26.98) - open/close 16889 (time 1788628413.40 total 799.38 last 27.69) - open/close 17104 (time 1788628423.43 total 809.41 last 21.44) - open/close 17353 (time 1788628433.45 total 819.43 last 24.84) - open/close 17588 (time 1788628444.23 total 830.21 last 21.81) - open/close 17778 (time 1788628454.73 total 840.71 last 18.09) - open/close 18049 (time 1788628464.73 total 850.71 last 27.10) - open/close 18284 (time 1788628475.53 total 861.51 last 21.76) - open/close 18504 (time 1788628485.95 total 871.93 last 21.12) - open/close 18750 (time 1788628495.96 total 881.94 last 24.57) - open/close 19479 (time 1788628505.97 total 891.95 last 72.85) - open/close 20000 (time 1788628513.14 total 899.12 last 72.63) - open/close 20693 (time 1788628523.17 total 909.15 last 69.08) - open/close 21375 (time 1788628533.18 total 919.16 last 68.14) - open/close 22116 (time 1788628543.19 total 929.17 last 74.03) - open/close 22846 (time 1788628553.20 total 939.19 last 72.89) - open/close 23562 (time 1788628563.21 total 949.19 last 71.59) - open/close 24344 (time 1788628573.23 total 959.21 last 78.04) - open/close 25118 (time 1788628583.24 total 969.22 last 77.32) - open/close 25844 (time 1788628593.24 total 979.22 last 72.57) - open/close 26616 (time 1788628603.25 total 989.23 last 77.16) - open/close 27347 (time 1788628613.26 total 999.24 last 73.02) - open/close 28101 (time 1788628623.26 total 1009.24 last 75.40) - open/close 28833 (time 1788628633.27 total 1019.25 last 73.13) - open/close 29600 (time 1788628643.27 total 1029.25 last 76.70) - open/close 30000 (time 1788628649.23 total 1035.21 last 67.12) - open/close 30650 (time 1788628659.24 total 1045.22 last 64.94) - open/close 31377 (time 1788628669.24 total 1055.22 last 72.66) - open/close 32148 (time 1788628679.27 total 1065.25 last 76.91) - open/close 32834 (time 1788628689.28 total 1075.26 last 68.52) - open/close 33529 (time 1788628699.29 total 1085.27 last 69.44) - open/close 34252 (time 1788628709.29 total 1095.28 last 72.24) - open/close 35008 (time 1788628719.30 total 1105.28 last 75.58) - open/close 35713 (time 1788628729.33 total 1115.31 last 70.30) - open/close 36440 (time 1788628739.33 total 1125.31 last 72.66) - open/close 37195 (time 1788628749.34 total 1135.32 last 75.47) - open/close 37891 (time 1788628759.34 total 1145.32 last 69.55) - open/close 38567 (time 1788628769.35 total 1155.34 last 67.52) - open/close 39250 (time 1788628779.37 total 1165.36 last 68.17) - open/close 39934 (time 1788628789.38 total 1175.37 last 68.33) - open/close 40601 (time 1788628799.39 total 1185.37 last 66.67) - open/close 41256 (time 1788628809.39 total 1195.38 last 65.46) - open/close 41836 (time 1788628819.40 total 1205.39 last 57.94) - open/close 42354 (time 1788628829.44 total 1215.42 last 51.63) - open/close 42846 (time 1788628839.45 total 1225.43 last 49.16) - open/close 43635 (time 1788628849.45 total 1235.43 last 78.85) - open/close 44286 (time 1788628859.46 total 1245.44 last 65.05) - open/close 45063 (time 1788628869.47 total 1255.45 last 77.64) - open/close 45691 (time 1788628879.47 total 1265.46 last 62.76) - open/close 46183 (time 1788628889.49 total 1275.47 last 49.13) - open/close 46756 (time 1788628899.50 total 1285.48 last 57.25) - open/close 47438 (time 1788628909.51 total 1295.49 last 68.12) - open/close 47986 (time 1788628919.51 total 1305.49 last 54.78) - open/close 48509 (time 1788628929.53 total 1315.51 last 52.23) - open/close 49201 (time 1788628939.54 total 1325.52 last 69.13) - open/close 49830 (time 1788628949.54 total 1335.52 last 62.88) total: 50000 open/close in 1338.30 seconds: 37.36 ops/second - unlinked 0 (time 1788628957 ; total 0 ; last 0) - unlinked 10000 (time 1788629103 ; total 146 ; last 146) - unlinked 20000 (time 1788629244 ; total 287 ; last 141) - unlinked 30000 (time 1788629399 ; total 442 ; last 155) - unlinked 40000 (time 1788629539 ; total 582 ; last 140) total: 50000 unlinks in 727 seconds: 68.775787 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg460-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg460-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg460-server stop mds service on oleg460-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg460-server unloading modules via unload_modules_local on: 'oleg460-server' oleg460-server: oleg460-server.virtnet: executing unload_modules_local oleg460-server: kmemleak disabled modules unloaded. pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-client: ssh exited with exit code 2 pdsh@oleg460-client: oleg460-server: ssh exited with exit code 2