== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:43:00 (1788442980) start mds service on oleg621-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1921056 Sep 3 07:51 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg621-server' oleg621-server: oleg621-server.virtnet: executing load_modules_local oleg621-server: kmemleak disabled oleg621-server: Loading modules from /home/green/git/lustre-release/lustre oleg621-server: detected 4 online CPUs by sysfs oleg621-server: MODOPTS_LIBCFS= oleg621-server: Force libcfs to create 2 CPU partitions oleg621-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg621-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg621-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg621-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg621-server: oleg621-server.virtnet: executing set_default_debug -1 all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg621-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg621-server: oleg621-server.virtnet: executing set_default_debug -1 all pdsh@oleg621-client: oleg621-server: ssh exited with exit code 1 Started lustre-OST0000 oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg621-client.virtnet: -o user_xattr,flock 192.168.206.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 143 (time 1788443065.70 total 10.12 last 14.13) - open/close 356 (time 1788443075.72 total 20.13 last 21.27) - open/close 566 (time 1788443086.39 total 30.81 last 19.67) - open/close 804 (time 1788443097.32 total 41.74 last 21.78) - open/close 998 (time 1788443107.67 total 52.09 last 18.74) - open/close 1166 (time 1788443117.72 total 62.13 last 16.72) - open/close 1291 (time 1788443127.84 total 72.26 last 12.35) - open/close 1520 (time 1788443137.88 total 82.29 last 22.82) - open/close 1759 (time 1788443147.89 total 92.30 last 23.88) - open/close 1960 (time 1788443157.89 total 102.31 last 20.09) - open/close 2166 (time 1788443167.92 total 112.34 last 20.54) - open/close 2381 (time 1788443178.59 total 123.01 last 20.15) - open/close 2653 (time 1788443188.59 total 133.01 last 27.19) - open/close 2900 (time 1788443198.61 total 143.03 last 24.65) - open/close 3177 (time 1788443208.62 total 153.04 last 27.69) - open/close 3376 (time 1788443218.65 total 163.07 last 19.83) - open/close 3612 (time 1788443229.21 total 173.63 last 22.36) - open/close 3839 (time 1788443239.33 total 183.75 last 22.42) - open/close 4074 (time 1788443249.33 total 193.75 last 23.50) - open/close 4317 (time 1788443259.35 total 203.77 last 24.26) - open/close 4552 (time 1788443270.17 total 214.59 last 21.72) - open/close 4764 (time 1788443280.17 total 224.59 last 21.19) - open/close 4986 (time 1788443290.72 total 235.14 last 21.05) - open/close 5245 (time 1788443301.35 total 245.77 last 24.36) - open/close 5494 (time 1788443311.65 total 256.07 last 24.17) - open/close 5730 (time 1788443322.21 total 266.63 last 22.35) - open/close 6000 (time 1788443332.22 total 276.64 last 26.98) - open/close 6214 (time 1788443342.60 total 287.02 last 20.62) - open/close 6470 (time 1788443352.64 total 297.06 last 25.50) - open/close 6651 (time 1788443365.53 total 309.95 last 14.05) - open/close 6810 (time 1788443375.91 total 320.33 last 15.32) - open/close 7074 (time 1788443386.22 total 330.64 last 25.61) - open/close 7310 (time 1788443396.98 total 341.40 last 21.93) - open/close 7607 (time 1788443407.54 total 351.96 last 28.11) - open/close 7840 (time 1788443417.84 total 362.26 last 22.62) - open/close 8070 (time 1788443430.30 total 374.72 last 18.46) - open/close 8322 (time 1788443441.32 total 385.74 last 22.87) - open/close 8590 (time 1788443451.33 total 395.75 last 26.77) - open/close 8801 (time 1788443461.35 total 405.77 last 21.05) - open/close 9065 (time 1788443471.36 total 415.78 last 26.37) - open/close 9275 (time 1788443481.49 total 425.91 last 20.73) - open/close 9500 (time 1788443491.50 total 435.92 last 22.49) - open/close 9728 (time 1788443502.46 total 446.88 last 20.80) - open/close 9931 (time 1788443512.88 total 457.29 last 19.48) - open/close 10000 (time 1788443515.07 total 459.49 last 31.47) - open/close 10205 (time 1788443525.08 total 469.50 last 20.47) - open/close 10388 (time 1788443535.10 total 479.51 last 18.28) - open/close 10586 (time 1788443545.10 total 489.51 last 19.80) - open/close 10778 (time 1788443557.34 total 501.76 last 15.68) - open/close 10993 (time 1788443568.10 total 512.52 last 19.99) - open/close 11180 (time 1788443578.81 total 523.22 last 17.47) - open/close 11408 (time 1788443589.54 total 533.96 last 21.24) - open/close 11601 (time 1788443599.71 total 544.13 last 18.98) - open/close 11786 (time 1788443609.73 total 554.15 last 18.47) - open/close 12022 (time 1788443621.48 total 565.90 last 20.08) - open/close 12184 (time 1788443631.50 total 575.92 last 16.18) - open/close 12394 (time 1788443641.50 total 585.92 last 20.99) - open/close 12620 (time 1788443652.26 total 596.68 last 21.01) - open/close 12827 (time 1788443663.16 total 607.57 last 18.99) - open/close 12959 (time 1788443676.10 total 620.51 last 10.20) - open/close 13076 (time 1788443686.46 total 630.87 last 11.29) - open/close 13188 (time 1788443698.21 total 642.63 last 9.53) - open/close 13377 (time 1788443708.99 total 653.41 last 17.53) - open/close 13631 (time 1788443719.00 total 663.42 last 25.36) - open/close 13856 (time 1788443729.01 total 673.43 last 22.48) - open/close 14018 (time 1788443739.35 total 683.77 last 15.67) - open/close 14264 (time 1788443750.17 total 694.59 last 22.74) - open/close 14449 (time 1788443760.55 total 704.97 last 17.82) - open/close 14694 (time 1788443771.18 total 715.60 last 23.05) - open/close 14876 (time 1788443782.18 total 726.60 last 16.54) - open/close 15095 (time 1788443792.19 total 736.61 last 21.88) - open/close 15348 (time 1788443802.22 total 746.64 last 25.23) - open/close 15566 (time 1788443812.32 total 756.74 last 21.59) - open/close 15810 (time 1788443822.77 total 767.19 last 23.34) - open/close 16062 (time 1788443832.78 total 777.20 last 25.18) - open/close 16342 (time 1788443843.58 total 787.99 last 25.93) - open/close 16567 (time 1788443854.23 total 798.65 last 21.11) - open/close 16798 (time 1788443864.55 total 808.97 last 22.38) - open/close 17076 (time 1788443874.92 total 819.34 last 26.80) - open/close 17351 (time 1788443884.93 total 829.35 last 27.47) - open/close 17564 (time 1788443894.95 total 839.37 last 21.26) - open/close 17799 (time 1788443905.26 total 849.68 last 22.80) - open/close 18053 (time 1788443915.89 total 860.31 last 23.89) - open/close 18297 (time 1788443925.92 total 870.33 last 24.34) - open/close 18551 (time 1788443936.10 total 880.52 last 24.94) - open/close 18814 (time 1788443946.49 total 890.91 last 25.32) - open/close 19058 (time 1788443956.90 total 901.31 last 23.45) - open/close 19266 (time 1788443966.92 total 911.34 last 20.75) - open/close 19485 (time 1788443976.94 total 921.35 last 21.87) - open/close 19970 (time 1788443986.94 total 931.36 last 48.46) - open/close 20399 (time 1788443996.96 total 941.38 last 42.83) - open/close 21025 (time 1788444006.97 total 951.38 last 62.56) - open/close 21764 (time 1788444016.97 total 961.39 last 73.89) - open/close 22492 (time 1788444026.98 total 971.40 last 72.73) - open/close 23196 (time 1788444036.98 total 981.40 last 70.37) - open/close 23865 (time 1788444047.00 total 991.42 last 66.76) - open/close 24500 (time 1788444057.01 total 1001.43 last 63.45) - open/close 25164 (time 1788444067.03 total 1011.45 last 66.25) - open/close 25924 (time 1788444077.04 total 1021.46 last 75.97) - open/close 26613 (time 1788444087.04 total 1031.46 last 68.88) - open/close 27377 (time 1788444097.05 total 1041.46 last 76.36) - open/close 28081 (time 1788444107.07 total 1051.49 last 70.25) - open/close 28715 (time 1788444117.08 total 1061.50 last 63.33) - open/close 29407 (time 1788444127.08 total 1071.50 last 69.17) - open/close 30000 (time 1788444135.55 total 1079.96 last 70.08) - open/close 30695 (time 1788444145.56 total 1089.97 last 69.43) - open/close 31386 (time 1788444155.57 total 1099.99 last 69.00) - open/close 32086 (time 1788444165.58 total 1110.00 last 69.95) - open/close 32741 (time 1788444175.58 total 1120.00 last 65.50) - open/close 33463 (time 1788444185.59 total 1130.01 last 72.09) - open/close 34204 (time 1788444195.60 total 1140.02 last 74.05) - open/close 34911 (time 1788444205.61 total 1150.02 last 70.66) - open/close 35600 (time 1788444215.61 total 1160.03 last 68.86) - open/close 36304 (time 1788444225.62 total 1170.03 last 70.38) - open/close 36970 (time 1788444235.64 total 1180.06 last 66.41) - open/close 37608 (time 1788444245.66 total 1190.08 last 63.69) - open/close 38324 (time 1788444255.70 total 1200.12 last 71.30) - open/close 38940 (time 1788444265.71 total 1210.13 last 61.54) - open/close 39554 (time 1788444275.71 total 1220.13 last 61.40) - open/close 40000 (time 1788444282.95 total 1227.37 last 61.63) - open/close 40614 (time 1788444292.96 total 1237.37 last 61.38) - open/close 41323 (time 1788444302.96 total 1247.38 last 70.84) - open/close 41998 (time 1788444312.98 total 1257.39 last 67.42) - open/close 42716 (time 1788444322.99 total 1267.41 last 71.67) - open/close 43280 (time 1788444333.02 total 1277.44 last 56.27) - open/close 43907 (time 1788444343.03 total 1287.44 last 62.65) - open/close 44579 (time 1788444353.03 total 1297.44 last 67.20) - open/close 45245 (time 1788444363.04 total 1307.46 last 66.50) - open/close 45935 (time 1788444373.06 total 1317.48 last 68.83) - open/close 46611 (time 1788444383.07 total 1327.49 last 67.57) - open/close 47253 (time 1788444393.08 total 1337.50 last 64.15) - open/close 48000 (time 1788444403.09 total 1347.51 last 74.60) - open/close 48757 (time 1788444413.09 total 1357.51 last 75.67) - open/close 49382 (time 1788444423.10 total 1367.52 last 62.46) total: 50000 open/close in 1375.97 seconds: 36.34 ops/second - unlinked 0 (time 1788444435 ; total 0 ; last 0) - unlinked 10000 (time 1788444578 ; total 143 ; last 143) - unlinked 20000 (time 1788444730 ; total 295 ; last 152) - unlinked 30000 (time 1788444881 ; total 446 ; last 151) - unlinked 40000 (time 1788445031 ; total 596 ; last 150) total: 50000 unlinks in 754 seconds: 66.312996 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg621-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg621-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg621-server stop mds service on oleg621-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg621-server unloading modules via unload_modules_local on: 'oleg621-server' oleg621-server: oleg621-server.virtnet: executing unload_modules_local oleg621-server: kmemleak disabled modules unloaded. pdsh@oleg621-client: oleg621-client: ssh exited with exit code 2 pdsh@oleg621-client: oleg621-server: ssh exited with exit code 2 pdsh@oleg621-client: oleg621-client: ssh exited with exit code 2 pdsh@oleg621-client: oleg621-server: ssh exited with exit code 2