== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:52:07 (1788623527) start mds service on oleg651-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1755168 Sep 4 17:38 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg651-server' oleg651-server: oleg651-server.virtnet: executing load_modules_local oleg651-server: kmemleak disabled oleg651-server: Loading modules from /home/green/git/lustre-release/lustre oleg651-server: detected 4 online CPUs by sysfs oleg651-server: MODOPTS_LIBCFS= oleg651-server: Force libcfs to create 2 CPU partitions oleg651-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg651-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg651-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg651-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg651-server: oleg651-server.virtnet: executing set_default_debug -1 all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg651-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg651-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg651-server: oleg651-server.virtnet: executing set_default_debug -1 all pdsh@oleg651-client: oleg651-server: ssh exited with exit code 1 Started lustre-OST0000 oleg651-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg651-client.virtnet: -o user_xattr,flock 192.168.206.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 291 (time 1788623609.24 total 10.97 last 26.52) - open/close 574 (time 1788623620.07 total 21.80 last 26.13) - open/close 704 (time 1788623630.18 total 31.92 last 12.85) - open/close 959 (time 1788623640.54 total 42.28 last 24.61) - open/close 1156 (time 1788623650.55 total 52.28 last 19.69) - open/close 1397 (time 1788623660.76 total 62.49 last 23.61) - open/close 1666 (time 1788623670.77 total 72.51 last 26.85) - open/close 1899 (time 1788623681.63 total 83.37 last 21.46) - open/close 2114 (time 1788623692.33 total 94.06 last 20.10) - open/close 2349 (time 1788623702.75 total 104.48 last 22.55) - open/close 2576 (time 1788623712.76 total 114.49 last 22.67) - open/close 2843 (time 1788623723.35 total 125.09 last 25.21) - open/close 3143 (time 1788623733.36 total 135.10 last 29.97) - open/close 3299 (time 1788623743.58 total 145.32 last 15.26) - open/close 3446 (time 1788623755.93 total 157.67 last 11.91) - open/close 3668 (time 1788623767.01 total 168.74 last 20.04) - open/close 3864 (time 1788623777.31 total 179.04 last 19.03) - open/close 4036 (time 1788623788.33 total 190.07 last 15.60) - open/close 4180 (time 1788623798.34 total 200.07 last 14.39) - open/close 4400 (time 1788623808.35 total 210.09 last 21.97) - open/close 4621 (time 1788623818.36 total 220.10 last 22.08) - open/close 4817 (time 1788623829.09 total 230.83 last 18.27) - open/close 5055 (time 1788623839.26 total 241.00 last 23.40) - open/close 5320 (time 1788623849.77 total 251.51 last 25.21) - open/close 5425 (time 1788623859.78 total 261.52 last 10.49) - open/close 5690 (time 1788623870.62 total 272.35 last 24.47) - open/close 5818 (time 1788623880.63 total 282.36 last 12.79) - open/close 5958 (time 1788623891.57 total 293.31 last 12.79) - open/close 6158 (time 1788623901.60 total 303.33 last 19.95) - open/close 6432 (time 1788623911.61 total 313.34 last 27.38) - open/close 6611 (time 1788623922.08 total 323.82 last 17.09) - open/close 6889 (time 1788623932.08 total 333.82 last 27.79) - open/close 7090 (time 1788623942.30 total 344.03 last 19.68) - open/close 7340 (time 1788623952.30 total 354.04 last 24.99) - open/close 7621 (time 1788623963.30 total 365.03 last 25.55) - open/close 7877 (time 1788623973.31 total 375.04 last 25.58) - open/close 8077 (time 1788623983.71 total 385.44 last 19.23) - open/close 8344 (time 1788623994.18 total 395.92 last 25.49) - open/close 8542 (time 1788624004.83 total 406.56 last 18.60) - open/close 8791 (time 1788624014.85 total 416.58 last 24.85) - open/close 9069 (time 1788624025.76 total 427.50 last 25.46) - open/close 9288 (time 1788624035.80 total 437.54 last 21.82) - open/close 9516 (time 1788624045.80 total 447.54 last 22.79) - open/close 9729 (time 1788624055.81 total 457.55 last 21.29) - open/close 9956 (time 1788624066.27 total 468.00 last 21.70) - open/close 10204 (time 1788624077.15 total 478.88 last 22.80) - open/close 10508 (time 1788624088.03 total 489.76 last 27.94) - open/close 10744 (time 1788624098.04 total 499.78 last 23.57) - open/close 10988 (time 1788624108.06 total 509.79 last 24.36) - open/close 11242 (time 1788624118.11 total 519.84 last 25.27) - open/close 11512 (time 1788624128.57 total 530.31 last 25.81) - open/close 11749 (time 1788624138.79 total 540.53 last 23.19) - open/close 12024 (time 1788624149.10 total 550.84 last 26.67) - open/close 12273 (time 1788624159.11 total 560.85 last 24.88) - open/close 12547 (time 1788624169.90 total 571.64 last 25.39) - open/close 12784 (time 1788624180.39 total 582.13 last 22.60) - open/close 13009 (time 1788624190.41 total 592.15 last 22.46) - open/close 13276 (time 1788624200.62 total 602.36 last 26.14) - open/close 13520 (time 1788624210.64 total 612.37 last 24.37) - open/close 13734 (time 1788624221.03 total 622.76 last 20.60) - open/close 13979 (time 1788624231.03 total 632.77 last 24.48) - open/close 14237 (time 1788624241.41 total 643.14 last 24.88) - open/close 14503 (time 1788624252.07 total 653.80 last 24.95) - open/close 14712 (time 1788624262.08 total 663.82 last 20.86) - open/close 14950 (time 1788624272.09 total 673.82 last 23.79) - open/close 15193 (time 1788624283.16 total 684.90 last 21.94) - open/close 15431 (time 1788624293.17 total 694.91 last 23.78) - open/close 15581 (time 1788624303.27 total 705.00 last 14.86) - open/close 15855 (time 1788624314.11 total 715.84 last 25.27) - open/close 16121 (time 1788624324.15 total 725.89 last 26.48) - open/close 16281 (time 1788624334.17 total 735.90 last 15.98) - open/close 16564 (time 1788624344.19 total 745.92 last 28.25) - open/close 16781 (time 1788624354.85 total 756.58 last 20.35) - open/close 17017 (time 1788624364.89 total 766.62 last 23.51) - open/close 17275 (time 1788624375.71 total 777.44 last 23.84) - open/close 17516 (time 1788624386.27 total 788.01 last 22.81) - open/close 17743 (time 1788624396.64 total 798.37 last 21.90) - open/close 17866 (time 1788624407.72 total 809.45 last 11.10) - open/close 18075 (time 1788624417.73 total 819.46 last 20.88) - open/close 18301 (time 1788624427.74 total 829.48 last 22.57) - open/close 18543 (time 1788624437.79 total 839.53 last 24.08) - open/close 18791 (time 1788624448.79 total 850.53 last 22.54) - open/close 19024 (time 1788624459.42 total 861.16 last 21.92) - open/close 19269 (time 1788624469.43 total 871.17 last 24.49) - open/close 19848 (time 1788624479.44 total 881.17 last 57.86) - open/close 20711 (time 1788624489.44 total 891.18 last 86.28) - open/close 21509 (time 1788624499.45 total 901.19 last 79.70) - open/close 22373 (time 1788624509.46 total 911.19 last 86.35) - open/close 23278 (time 1788624519.47 total 921.20 last 90.40) - open/close 24152 (time 1788624529.47 total 931.21 last 87.36) - open/close 25012 (time 1788624539.49 total 941.23 last 85.87) - open/close 25858 (time 1788624549.49 total 951.23 last 84.57) - open/close 26767 (time 1788624559.50 total 961.23 last 90.87) - open/close 27586 (time 1788624569.50 total 971.23 last 81.90) - open/close 28335 (time 1788624579.50 total 981.23 last 74.88) - open/close 29124 (time 1788624589.53 total 991.26 last 78.68) - open/close 29874 (time 1788624599.53 total 1001.27 last 74.96) - open/close 30650 (time 1788624609.55 total 1011.28 last 77.49) - open/close 31432 (time 1788624619.55 total 1021.28 last 78.19) - open/close 32116 (time 1788624629.56 total 1031.29 last 68.33) - open/close 32860 (time 1788624639.57 total 1041.30 last 74.34) - open/close 33701 (time 1788624649.57 total 1051.30 last 84.09) - open/close 34314 (time 1788624659.57 total 1061.31 last 61.28) - open/close 35118 (time 1788624669.59 total 1071.32 last 80.27) - open/close 35829 (time 1788624679.60 total 1081.34 last 71.00) - open/close 36599 (time 1788624689.61 total 1091.35 last 76.94) - open/close 37441 (time 1788624699.62 total 1101.36 last 84.10) - open/close 38281 (time 1788624709.63 total 1111.36 last 83.97) - open/close 39085 (time 1788624719.63 total 1121.37 last 80.36) - open/close 39884 (time 1788624729.63 total 1131.37 last 79.87) - open/close 40702 (time 1788624739.65 total 1141.39 last 81.64) - open/close 41536 (time 1788624749.66 total 1151.39 last 83.37) - open/close 42324 (time 1788624759.67 total 1161.41 last 78.67) - open/close 43181 (time 1788624769.68 total 1171.41 last 85.65) - open/close 44047 (time 1788624779.68 total 1181.42 last 86.58) - open/close 44933 (time 1788624789.69 total 1191.43 last 88.50) - open/close 45707 (time 1788624799.70 total 1201.44 last 77.33) - open/close 46516 (time 1788624809.70 total 1211.44 last 80.90) - open/close 47353 (time 1788624819.71 total 1221.45 last 83.59) - open/close 48162 (time 1788624829.73 total 1231.47 last 80.76) - open/close 49023 (time 1788624839.74 total 1241.47 last 86.06) - open/close 49868 (time 1788624849.74 total 1251.48 last 84.43) total: 50000 open/close in 1253.13 seconds: 39.90 ops/second - unlinked 0 (time 1788624855 ; total 0 ; last 0) - unlinked 10000 (time 1788624984 ; total 129 ; last 129) - unlinked 20000 (time 1788625114 ; total 259 ; last 130) - unlinked 30000 (time 1788625242 ; total 387 ; last 128) - unlinked 40000 (time 1788625372 ; total 517 ; last 130) total: 50000 unlinks in 645 seconds: 77.519379 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg651-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg651-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg651-server stop mds service on oleg651-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg651-server unloading modules via unload_modules_local on: 'oleg651-server' oleg651-server: oleg651-server.virtnet: executing unload_modules_local oleg651-server: kmemleak disabled modules unloaded. pdsh@oleg651-client: oleg651-client: ssh exited with exit code 2 pdsh@oleg651-client: oleg651-server: ssh exited with exit code 2 pdsh@oleg651-client: oleg651-client: ssh exited with exit code 2 pdsh@oleg651-client: oleg651-server: ssh exited with exit code 2