== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:02:10 (1788652930) start mds service on oleg648-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 4 18:31 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg648-server' oleg648-server: oleg648-server.virtnet: executing load_modules_local oleg648-server: kmemleak disabled oleg648-server: Loading modules from /home/green/git/lustre-release/lustre oleg648-server: detected 4 online CPUs by sysfs oleg648-server: MODOPTS_LIBCFS= oleg648-server: Force libcfs to create 2 CPU partitions oleg648-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg648-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg648-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg648-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg648-server: oleg648-server.virtnet: executing set_default_debug -1 all pdsh@oleg648-client: oleg648-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg648-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg648-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg648-server: oleg648-server.virtnet: executing set_default_debug -1 all pdsh@oleg648-client: oleg648-server: ssh exited with exit code 1 Started lustre-OST0000 oleg648-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg648-client.virtnet: -o user_xattr,flock 192.168.206.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 278 (time 1788653007.41 total 10.72 last 25.92) - open/close 526 (time 1788653017.42 total 20.73 last 24.78) - open/close 759 (time 1788653027.43 total 30.74 last 23.27) - open/close 1016 (time 1788653038.15 total 41.46 last 23.97) - open/close 1262 (time 1788653048.76 total 52.08 last 23.18) - open/close 1452 (time 1788653059.45 total 62.77 last 17.77) - open/close 1679 (time 1788653069.91 total 73.22 last 21.71) - open/close 1821 (time 1788653079.91 total 83.22 last 14.20) - open/close 2065 (time 1788653090.42 total 93.73 last 23.22) - open/close 2290 (time 1788653100.53 total 103.85 last 22.25) - open/close 2507 (time 1788653110.54 total 113.86 last 21.68) - open/close 2784 (time 1788653121.53 total 124.85 last 25.21) - open/close 2967 (time 1788653131.85 total 135.16 last 17.74) - open/close 3184 (time 1788653141.95 total 145.27 last 21.48) - open/close 3418 (time 1788653152.89 total 156.20 last 21.40) - open/close 3653 (time 1788653162.89 total 166.20 last 23.49) - open/close 3830 (time 1788653174.45 total 177.76 last 15.32) - open/close 4011 (time 1788653185.23 total 188.54 last 16.79) - open/close 4225 (time 1788653195.38 total 198.69 last 21.08) - open/close 4460 (time 1788653205.50 total 208.81 last 23.23) - open/close 4721 (time 1788653215.50 total 218.81 last 26.09) - open/close 4946 (time 1788653225.51 total 228.82 last 22.48) - open/close 5196 (time 1788653236.34 total 239.65 last 23.09) - open/close 5441 (time 1788653246.59 total 249.91 last 23.89) - open/close 5678 (time 1788653257.40 total 260.72 last 21.93) - open/close 5872 (time 1788653267.44 total 270.75 last 19.33) - open/close 6137 (time 1788653277.45 total 280.76 last 26.47) - open/close 6380 (time 1788653288.18 total 291.49 last 22.64) - open/close 6607 (time 1788653298.75 total 302.06 last 21.48) - open/close 6865 (time 1788653309.17 total 312.48 last 24.75) - open/close 7074 (time 1788653319.19 total 322.50 last 20.86) - open/close 7297 (time 1788653329.19 total 332.51 last 22.29) - open/close 7578 (time 1788653339.20 total 342.51 last 28.09) - open/close 7788 (time 1788653349.75 total 353.07 last 19.90) - open/close 8042 (time 1788653360.82 total 364.14 last 22.94) - open/close 8183 (time 1788653370.94 total 374.26 last 13.93) - open/close 8371 (time 1788653380.96 total 384.28 last 18.76) - open/close 8615 (time 1788653391.35 total 394.66 last 23.49) - open/close 8818 (time 1788653401.36 total 404.68 last 20.27) - open/close 9022 (time 1788653411.76 total 415.07 last 19.62) - open/close 9204 (time 1788653421.87 total 425.18 last 18.00) - open/close 9349 (time 1788653432.77 total 436.09 last 13.30) - open/close 9509 (time 1788653442.78 total 446.09 last 15.99) - open/close 9664 (time 1788653452.93 total 456.24 last 15.27) - open/close 9971 (time 1788653463.36 total 466.68 last 29.42) - open/close 10220 (time 1788653474.37 total 477.68 last 22.63) - open/close 10399 (time 1788653486.52 total 489.84 last 14.73) - open/close 10667 (time 1788653497.56 total 500.87 last 24.28) - open/close 10903 (time 1788653507.97 total 511.29 last 22.66) - open/close 11165 (time 1788653518.78 total 522.10 last 24.23) - open/close 11403 (time 1788653529.41 total 532.72 last 22.40) - open/close 11712 (time 1788653540.14 total 543.46 last 28.79) - open/close 11928 (time 1788653550.22 total 553.53 last 21.44) - open/close 12022 (time 1788653560.39 total 563.70 last 9.24) - open/close 12266 (time 1788653571.01 total 574.32 last 22.98) - open/close 12515 (time 1788653581.87 total 585.19 last 22.91) - open/close 12787 (time 1788653592.50 total 595.82 last 25.59) - open/close 13079 (time 1788653603.51 total 606.83 last 26.51) - open/close 13374 (time 1788653613.52 total 616.84 last 29.47) - open/close 13600 (time 1788653623.68 total 627.00 last 22.24) - open/close 13851 (time 1788653633.97 total 637.29 last 24.39) - open/close 14098 (time 1788653644.15 total 647.47 last 24.27) - open/close 14350 (time 1788653654.59 total 657.90 last 24.14) - open/close 14586 (time 1788653664.60 total 667.91 last 23.58) - open/close 14841 (time 1788653674.88 total 678.19 last 24.80) - open/close 15103 (time 1788653685.63 total 688.94 last 24.37) - open/close 15366 (time 1788653696.44 total 699.76 last 24.33) - open/close 15613 (time 1788653706.45 total 709.76 last 24.69) - open/close 15883 (time 1788653716.45 total 719.76 last 27.00) - open/close 16120 (time 1788653727.28 total 730.59 last 21.88) - open/close 16316 (time 1788653737.45 total 740.77 last 19.26) - open/close 16540 (time 1788653748.29 total 751.61 last 20.67) - open/close 16798 (time 1788653758.53 total 761.84 last 25.20) - open/close 17018 (time 1788653768.54 total 771.85 last 21.98) - open/close 17232 (time 1788653778.57 total 781.89 last 21.33) - open/close 17491 (time 1788653789.56 total 792.87 last 23.58) - open/close 17732 (time 1788653799.58 total 802.89 last 24.05) - open/close 17941 (time 1788653809.61 total 812.92 last 20.85) - open/close 18117 (time 1788653819.62 total 822.93 last 17.57) - open/close 18364 (time 1788653829.65 total 832.97 last 24.62) - open/close 18555 (time 1788653839.67 total 842.98 last 19.07) - open/close 18719 (time 1788653850.81 total 854.12 last 14.72) - open/close 18922 (time 1788653860.81 total 864.13 last 20.29) - open/close 19162 (time 1788653870.82 total 874.13 last 23.98) - open/close 19812 (time 1788653880.83 total 884.14 last 64.93) - open/close 20000 (time 1788653883.59 total 886.91 last 67.98) - open/close 20754 (time 1788653893.60 total 896.91 last 75.36) - open/close 21480 (time 1788653903.60 total 906.92 last 72.57) - open/close 22188 (time 1788653913.60 total 916.92 last 70.80) - open/close 22865 (time 1788653923.60 total 926.92 last 67.70) - open/close 23589 (time 1788653933.61 total 936.92 last 72.40) - open/close 24270 (time 1788653943.61 total 946.92 last 68.09) - open/close 24953 (time 1788653953.62 total 956.94 last 68.20) - open/close 25612 (time 1788653963.63 total 966.95 last 65.84) - open/close 26281 (time 1788653973.64 total 976.95 last 66.86) - open/close 26985 (time 1788653983.64 total 986.96 last 70.36) - open/close 27647 (time 1788653993.65 total 996.96 last 66.16) - open/close 28344 (time 1788654003.65 total 1006.96 last 69.69) - open/close 29001 (time 1788654013.65 total 1016.97 last 65.67) - open/close 29743 (time 1788654023.66 total 1026.97 last 74.15) - open/close 30000 (time 1788654027.31 total 1030.62 last 70.43) - open/close 30731 (time 1788654037.32 total 1040.63 last 73.03) - open/close 31455 (time 1788654047.33 total 1050.65 last 72.30) - open/close 32149 (time 1788654057.35 total 1060.67 last 69.25) - open/close 32816 (time 1788654067.37 total 1070.68 last 66.59) - open/close 33533 (time 1788654077.37 total 1080.69 last 71.67) - open/close 34269 (time 1788654087.38 total 1090.69 last 73.58) - open/close 34959 (time 1788654097.42 total 1100.73 last 68.71) - open/close 35672 (time 1788654107.43 total 1110.75 last 71.21) - open/close 36390 (time 1788654117.44 total 1120.75 last 71.77) - open/close 37083 (time 1788654127.44 total 1130.75 last 69.30) - open/close 37760 (time 1788654137.46 total 1140.77 last 67.57) - open/close 38481 (time 1788654147.47 total 1150.78 last 72.01) - open/close 39148 (time 1788654157.49 total 1160.80 last 66.59) - open/close 39817 (time 1788654167.49 total 1170.80 last 66.90) - open/close 40000 (time 1788654170.15 total 1173.46 last 68.82) - open/close 40752 (time 1788654180.16 total 1183.47 last 75.13) - open/close 41474 (time 1788654190.17 total 1193.48 last 72.13) - open/close 42181 (time 1788654200.18 total 1203.50 last 70.59) - open/close 42829 (time 1788654210.19 total 1213.50 last 64.76) - open/close 43504 (time 1788654220.19 total 1223.51 last 67.46) - open/close 44273 (time 1788654230.20 total 1233.52 last 76.81) - open/close 44898 (time 1788654240.21 total 1243.52 last 62.49) - open/close 45492 (time 1788654250.21 total 1253.53 last 59.36) - open/close 46191 (time 1788654260.24 total 1263.55 last 69.72) - open/close 46814 (time 1788654270.25 total 1273.56 last 62.25) - open/close 47371 (time 1788654280.25 total 1283.56 last 55.68) - open/close 48042 (time 1788654290.25 total 1293.56 last 67.10) - open/close 48757 (time 1788654300.25 total 1303.57 last 71.47) - open/close 49404 (time 1788654310.27 total 1313.58 last 64.63) total: 50000 open/close in 1322.54 seconds: 37.81 ops/second - unlinked 0 (time 1788654323 ; total 0 ; last 0) - unlinked 10000 (time 1788654463 ; total 140 ; last 140) - unlinked 20000 (time 1788654607 ; total 284 ; last 144) - unlinked 30000 (time 1788654740 ; total 417 ; last 133) - unlinked 40000 (time 1788654883 ; total 560 ; last 143) total: 50000 unlinks in 706 seconds: 70.821533 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg648-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg648-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg648-server stop mds service on oleg648-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg648-server unloading modules via unload_modules_local on: 'oleg648-server' oleg648-server: oleg648-server.virtnet: executing unload_modules_local oleg648-server: kmemleak disabled modules unloaded. pdsh@oleg648-client: oleg648-client: ssh exited with exit code 2 pdsh@oleg648-client: oleg648-server: ssh exited with exit code 2 pdsh@oleg648-client: oleg648-client: ssh exited with exit code 2 pdsh@oleg648-client: oleg648-server: ssh exited with exit code 2