== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:37:34 (1789450654) start mds service on oleg611-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1933472 Sep 14 13:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing load_modules_local oleg611-server: kmemleak disabled oleg611-server: Loading modules from /home/green/git/lustre-release/lustre oleg611-server: detected 4 online CPUs by sysfs oleg611-server: MODOPTS_LIBCFS= oleg611-server: Force libcfs to create 2 CPU partitions oleg611-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg611-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg611-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg611-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg611-server: oleg611-server.virtnet: executing set_default_debug -1 all pdsh@oleg611-client: oleg611-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg611-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg611-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg611-server: oleg611-server.virtnet: executing set_default_debug -1 all pdsh@oleg611-client: oleg611-server: ssh exited with exit code 1 Started lustre-OST0000 oleg611-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg611-client.virtnet: -o user_xattr,flock 192.168.206.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 212 (time 1789450739.73 total 11.15 last 19.01) - open/close 424 (time 1789450750.17 total 21.60 last 20.30) - open/close 542 (time 1789450760.18 total 31.60 last 11.80) - open/close 736 (time 1789450770.19 total 41.62 last 19.37) - open/close 956 (time 1789450780.58 total 52.00 last 21.18) - open/close 1223 (time 1789450790.64 total 62.06 last 26.55) - open/close 1451 (time 1789450801.12 total 72.55 last 21.74) - open/close 1695 (time 1789450811.73 total 83.16 last 22.99) - open/close 1900 (time 1789450822.55 total 93.98 last 18.96) - open/close 2161 (time 1789450832.55 total 103.98 last 26.09) - open/close 2331 (time 1789450843.15 total 114.58 last 16.04) - open/close 2516 (time 1789450853.64 total 125.07 last 17.63) - open/close 2740 (time 1789450863.69 total 135.12 last 22.29) - open/close 3012 (time 1789450874.73 total 146.15 last 24.65) - open/close 3249 (time 1789450884.73 total 156.16 last 23.68) - open/close 3458 (time 1789450895.25 total 166.67 last 19.88) - open/close 3715 (time 1789450906.21 total 177.64 last 23.43) - open/close 3930 (time 1789450916.22 total 187.64 last 21.49) - open/close 4150 (time 1789450926.35 total 197.78 last 21.71) - open/close 4362 (time 1789450937.38 total 208.81 last 19.22) - open/close 4570 (time 1789450947.39 total 218.82 last 20.79) - open/close 4781 (time 1789450957.40 total 228.82 last 21.08) - open/close 5033 (time 1789450967.64 total 239.07 last 24.61) - open/close 5250 (time 1789450977.65 total 249.08 last 21.67) - open/close 5477 (time 1789450988.64 total 260.07 last 20.65) - open/close 5716 (time 1789450999.19 total 270.61 last 22.67) - open/close 5952 (time 1789451009.43 total 280.85 last 23.05) - open/close 6201 (time 1789451020.30 total 291.73 last 22.89) - open/close 6388 (time 1789451030.47 total 301.90 last 18.39) - open/close 6607 (time 1789451040.49 total 311.92 last 21.86) - open/close 6837 (time 1789451050.51 total 321.94 last 22.96) - open/close 7029 (time 1789451061.22 total 332.65 last 17.93) - open/close 7295 (time 1789451071.22 total 342.65 last 26.59) - open/close 7500 (time 1789451082.14 total 353.57 last 18.78) - open/close 7750 (time 1789451092.49 total 363.92 last 24.15) - open/close 7957 (time 1789451102.50 total 373.92 last 20.69) - open/close 8177 (time 1789451113.04 total 384.47 last 20.86) - open/close 8420 (time 1789451123.30 total 394.73 last 23.70) - open/close 8664 (time 1789451133.30 total 404.73 last 24.39) - open/close 8911 (time 1789451143.32 total 414.75 last 24.66) - open/close 9167 (time 1789451154.00 total 425.43 last 23.96) - open/close 9261 (time 1789451164.03 total 435.45 last 9.38) - open/close 9479 (time 1789451175.06 total 446.49 last 19.76) - open/close 9621 (time 1789451185.12 total 456.54 last 14.12) - open/close 9882 (time 1789451195.79 total 467.21 last 24.46) - open/close 10000 (time 1789451200.75 total 472.18 last 23.76) - open/close 10229 (time 1789451210.93 total 482.35 last 22.51) - open/close 10446 (time 1789451220.94 total 492.37 last 21.66) - open/close 10670 (time 1789451231.63 total 503.06 last 20.96) - open/close 10930 (time 1789451242.26 total 513.69 last 24.46) - open/close 11098 (time 1789451252.38 total 523.80 last 16.61) - open/close 11265 (time 1789451262.39 total 533.81 last 16.68) - open/close 11449 (time 1789451273.10 total 544.53 last 17.18) - open/close 11626 (time 1789451283.13 total 554.56 last 17.64) - open/close 11766 (time 1789451293.66 total 565.09 last 13.30) - open/close 11908 (time 1789451304.32 total 575.75 last 13.32) - open/close 12034 (time 1789451315.06 total 586.48 last 11.74) - open/close 12241 (time 1789451327.82 total 599.25 last 16.21) - open/close 12417 (time 1789451337.83 total 609.25 last 17.59) - open/close 12704 (time 1789451347.85 total 619.27 last 28.64) - open/close 12961 (time 1789451357.85 total 629.28 last 25.69) - open/close 13193 (time 1789451367.86 total 639.29 last 23.17) - open/close 13360 (time 1789451377.95 total 649.38 last 16.55) - open/close 13579 (time 1789451387.97 total 659.39 last 21.87) - open/close 13807 (time 1789451398.41 total 669.84 last 21.83) - open/close 14055 (time 1789451409.18 total 680.60 last 23.04) - open/close 14315 (time 1789451420.10 total 691.53 last 23.80) - open/close 14528 (time 1789451430.93 total 702.36 last 19.67) - open/close 14730 (time 1789451440.97 total 712.40 last 20.12) - open/close 14976 (time 1789451451.29 total 722.72 last 23.84) - open/close 15205 (time 1789451461.31 total 732.73 last 22.87) - open/close 15428 (time 1789451472.15 total 743.58 last 20.56) - open/close 15677 (time 1789451482.15 total 753.58 last 24.89) - open/close 15892 (time 1789451492.55 total 763.98 last 20.68) - open/close 16250 (time 1789451502.80 total 774.22 last 34.94) - open/close 16513 (time 1789451512.80 total 784.23 last 26.29) - open/close 16752 (time 1789451523.40 total 794.83 last 22.54) - open/close 16949 (time 1789451533.45 total 804.87 last 19.62) - open/close 17109 (time 1789451543.75 total 815.17 last 15.53) - open/close 17239 (time 1789451554.14 total 825.57 last 12.51) - open/close 17454 (time 1789451564.25 total 835.68 last 21.26) - open/close 17703 (time 1789451574.49 total 845.91 last 24.33) - open/close 17926 (time 1789451584.78 total 856.21 last 21.66) - open/close 18169 (time 1789451594.80 total 866.22 last 24.27) - open/close 18432 (time 1789451604.80 total 876.23 last 26.29) - open/close 18704 (time 1789451614.80 total 886.23 last 27.19) - open/close 19326 (time 1789451624.81 total 896.24 last 62.16) - open/close 19939 (time 1789451634.82 total 906.25 last 61.22) - open/close 20610 (time 1789451644.84 total 916.26 last 67.00) - open/close 21290 (time 1789451654.85 total 926.27 last 67.93) - open/close 21944 (time 1789451664.85 total 936.27 last 65.40) - open/close 22565 (time 1789451674.86 total 946.28 last 62.05) - open/close 23177 (time 1789451684.86 total 956.29 last 61.15) - open/close 23806 (time 1789451694.87 total 966.30 last 62.87) - open/close 24395 (time 1789451704.88 total 976.31 last 58.81) - open/close 25021 (time 1789451714.90 total 986.32 last 62.53) - open/close 25633 (time 1789451724.90 total 996.33 last 61.16) - open/close 26321 (time 1789451734.93 total 1006.36 last 68.60) - open/close 26900 (time 1789451744.95 total 1016.37 last 57.82) - open/close 27522 (time 1789451754.96 total 1026.38 last 62.13) - open/close 28153 (time 1789451764.98 total 1036.40 last 62.97) - open/close 28797 (time 1789451775.00 total 1046.43 last 64.23) - open/close 29403 (time 1789451785.01 total 1056.43 last 60.59) - open/close 29929 (time 1789451795.01 total 1066.43 last 52.59) - open/close 30511 (time 1789451805.02 total 1076.44 last 58.15) - open/close 31078 (time 1789451815.03 total 1086.46 last 56.60) - open/close 31666 (time 1789451825.04 total 1096.46 last 58.79) - open/close 32272 (time 1789451835.05 total 1106.47 last 60.53) - open/close 32916 (time 1789451845.05 total 1116.48 last 64.36) - open/close 33576 (time 1789451855.06 total 1126.49 last 65.95) - open/close 34239 (time 1789451865.07 total 1136.50 last 66.24) - open/close 34908 (time 1789451875.08 total 1146.50 last 66.86) - open/close 35488 (time 1789451885.08 total 1156.50 last 57.99) - open/close 36133 (time 1789451895.08 total 1166.51 last 64.50) - open/close 36796 (time 1789451905.08 total 1176.51 last 66.27) - open/close 37433 (time 1789451915.10 total 1186.53 last 63.59) - open/close 38122 (time 1789451925.11 total 1196.54 last 68.83) - open/close 38787 (time 1789451935.12 total 1206.54 last 66.46) - open/close 39483 (time 1789451945.12 total 1216.55 last 69.55) - open/close 40000 (time 1789451952.67 total 1224.10 last 68.48) - open/close 40580 (time 1789451962.69 total 1234.12 last 57.88) - open/close 41177 (time 1789451972.70 total 1244.13 last 59.66) - open/close 41777 (time 1789451982.70 total 1254.13 last 59.99) - open/close 42377 (time 1789451992.72 total 1264.14 last 59.92) - open/close 42932 (time 1789452002.73 total 1274.16 last 55.41) - open/close 43531 (time 1789452012.74 total 1284.16 last 59.86) - open/close 44181 (time 1789452022.75 total 1294.18 last 64.93) - open/close 44782 (time 1789452032.76 total 1304.18 last 60.07) - open/close 45458 (time 1789452042.76 total 1314.19 last 67.56) - open/close 46150 (time 1789452052.77 total 1324.20 last 69.13) - open/close 46798 (time 1789452062.79 total 1334.21 last 64.71) - open/close 47493 (time 1789452072.83 total 1344.25 last 69.22) - open/close 48173 (time 1789452082.83 total 1354.26 last 67.96) - open/close 48833 (time 1789452092.84 total 1364.27 last 65.94) - open/close 49515 (time 1789452102.85 total 1374.28 last 68.15) total: 50000 open/close in 1381.30 seconds: 36.20 ops/second - unlinked 0 (time 1789452114 ; total 0 ; last 0) - unlinked 10000 (time 1789452259 ; total 145 ; last 145) - unlinked 20000 (time 1789452413 ; total 299 ; last 154) - unlinked 30000 (time 1789452564 ; total 450 ; last 151) - unlinked 40000 (time 1789452719 ; total 605 ; last 155) total: 50000 unlinks in 760 seconds: 65.789474 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg611-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg611-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg611-server stop mds service on oleg611-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg611-server unloading modules via unload_modules_local on: 'oleg611-server' oleg611-server: oleg611-server.virtnet: executing unload_modules_local oleg611-server: kmemleak disabled modules unloaded. pdsh@oleg611-client: oleg611-server: ssh exited with exit code 2 pdsh@oleg611-client: oleg611-client: ssh exited with exit code 2 pdsh@oleg611-client: oleg611-server: ssh exited with exit code 2 pdsh@oleg611-client: oleg611-client: ssh exited with exit code 2