== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:08:22 (1788512902) start mds service on oleg241-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1933656 Sep 3 13:58 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing load_modules_local oleg241-server: kmemleak disabled oleg241-server: Loading modules from /home/green/git/lustre-release/lustre oleg241-server: detected 4 online CPUs by sysfs oleg241-server: MODOPTS_LIBCFS= oleg241-server: Force libcfs to create 2 CPU partitions oleg241-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg241-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg241-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg241-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg241-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg241-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg241-server: oleg241-server.virtnet: executing set_default_debug -1 all pdsh@oleg241-client: oleg241-server: ssh exited with exit code 1 Started lustre-OST0000 oleg241-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg241-client.virtnet: -o user_xattr,flock 192.168.202.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 293 (time 1788513007.03 total 10.01 last 29.28) - open/close 412 (time 1788513017.08 total 20.06 last 11.84) - open/close 520 (time 1788513027.24 total 30.22 last 10.63) - open/close 581 (time 1788513037.25 total 40.22 last 6.10) - open/close 685 (time 1788513047.29 total 50.27 last 10.36) - open/close 802 (time 1788513057.84 total 60.82 last 11.09) - open/close 921 (time 1788513067.88 total 70.85 last 11.86) - open/close 1167 (time 1788513078.66 total 81.64 last 22.81) - open/close 1340 (time 1788513088.67 total 91.65 last 17.28) - open/close 1527 (time 1788513098.67 total 101.65 last 18.70) - open/close 1751 (time 1788513109.05 total 112.02 last 21.60) - open/close 1931 (time 1788513119.56 total 122.54 last 17.12) - open/close 2084 (time 1788513130.11 total 133.09 last 14.50) - open/close 2320 (time 1788513140.71 total 143.69 last 22.26) - open/close 2532 (time 1788513150.73 total 153.70 last 21.17) - open/close 2683 (time 1788513161.52 total 164.50 last 13.98) - open/close 2845 (time 1788513171.54 total 174.51 last 16.18) - open/close 2948 (time 1788513181.55 total 184.53 last 10.28) - open/close 3228 (time 1788513191.58 total 194.55 last 27.93) - open/close 3442 (time 1788513201.59 total 204.56 last 21.38) - open/close 3653 (time 1788513211.83 total 214.81 last 20.60) - open/close 3744 (time 1788513222.13 total 225.10 last 8.84) - open/close 3897 (time 1788513232.49 total 235.47 last 14.76) - open/close 4113 (time 1788513242.50 total 245.48 last 21.58) - open/close 4274 (time 1788513252.98 total 255.96 last 15.36) - open/close 4363 (time 1788513263.41 total 266.38 last 8.54) - open/close 4515 (time 1788513273.44 total 276.42 last 15.14) - open/close 4688 (time 1788513283.47 total 286.45 last 17.25) - open/close 4830 (time 1788513294.39 total 297.36 last 13.01) - open/close 4953 (time 1788513304.82 total 307.79 last 11.79) - open/close 5057 (time 1788513315.00 total 317.98 last 10.21) - open/close 5190 (time 1788513325.17 total 328.14 last 13.08) - open/close 5304 (time 1788513335.18 total 338.16 last 11.39) - open/close 5438 (time 1788513345.18 total 348.16 last 13.40) - open/close 5542 (time 1788513355.88 total 358.86 last 9.72) - open/close 5707 (time 1788513366.62 total 369.60 last 15.36) - open/close 5860 (time 1788513376.92 total 379.90 last 14.86) - open/close 5955 (time 1788513389.91 total 392.89 last 7.32) - open/close 6062 (time 1788513401.27 total 404.24 last 9.42) - open/close 6195 (time 1788513411.56 total 414.54 last 12.92) - open/close 6321 (time 1788513421.93 total 424.91 last 12.15) - open/close 6601 (time 1788513432.88 total 435.86 last 25.57) - open/close 6837 (time 1788513442.88 total 445.86 last 23.60) - open/close 6990 (time 1788513453.30 total 456.28 last 14.68) - open/close 7162 (time 1788513463.93 total 466.91 last 16.18) - open/close 7330 (time 1788513474.92 total 477.90 last 15.29) - open/close 7522 (time 1788513484.93 total 487.90 last 19.19) - open/close 7627 (time 1788513494.93 total 497.90 last 10.50) - open/close 7743 (time 1788513505.83 total 508.81 last 10.64) - open/close 7890 (time 1788513515.90 total 518.88 last 14.60) - open/close 8064 (time 1788513526.15 total 529.13 last 16.97) - open/close 8234 (time 1788513536.81 total 539.78 last 15.96) - open/close 8458 (time 1788513546.81 total 549.79 last 22.38) - open/close 8700 (time 1788513557.50 total 560.47 last 22.66) - open/close 8882 (time 1788513568.20 total 571.18 last 17.00) - open/close 8928 (time 1788513580.86 total 583.83 last 3.63) - open/close 9070 (time 1788513591.68 total 594.66 last 13.12) - open/close 9238 (time 1788513602.50 total 605.48 last 15.52) - open/close 9343 (time 1788513612.51 total 615.49 last 10.49) - open/close 9472 (time 1788513623.26 total 626.24 last 12.00) - open/close 9655 (time 1788513634.26 total 637.24 last 16.63) - open/close 9757 (time 1788513645.04 total 648.02 last 9.46) - open/close 9853 (time 1788513655.09 total 658.07 last 9.55) - open/close 10000 (time 1788513664.37 total 667.35 last 15.84) - open/close 10194 (time 1788513675.19 total 678.17 last 17.93) - open/close 10398 (time 1788513685.19 total 688.17 last 20.39) - open/close 10651 (time 1788513696.21 total 699.19 last 22.97) - open/close 10842 (time 1788513707.00 total 709.98 last 17.70) - open/close 11019 (time 1788513717.13 total 720.10 last 17.47) - open/close 11186 (time 1788513727.26 total 730.24 last 16.48) - open/close 11301 (time 1788513737.26 total 740.24 last 11.50) - open/close 11405 (time 1788513748.19 total 751.17 last 9.52) - open/close 11608 (time 1788513758.23 total 761.21 last 20.23) - open/close 11690 (time 1788513768.71 total 771.69 last 7.82) - open/close 11747 (time 1788513778.86 total 781.84 last 5.62) - open/close 11957 (time 1788513789.82 total 792.79 last 19.17) - open/close 12186 (time 1788513799.83 total 802.81 last 22.87) - open/close 12438 (time 1788513812.17 total 815.15 last 20.42) - open/close 12627 (time 1788513822.22 total 825.20 last 18.80) - open/close 12875 (time 1788513832.23 total 835.21 last 24.79) - open/close 13118 (time 1788513842.24 total 845.21 last 24.28) - open/close 13356 (time 1788513852.25 total 855.22 last 23.78) - open/close 13628 (time 1788513863.29 total 866.27 last 24.63) - open/close 13849 (time 1788513873.30 total 876.28 last 22.08) - open/close 14075 (time 1788513884.34 total 887.31 last 20.48) - open/close 14228 (time 1788513894.35 total 897.33 last 15.28) - open/close 14444 (time 1788513904.57 total 907.55 last 21.13) - open/close 14637 (time 1788513915.57 total 918.54 last 17.55) - open/close 14918 (time 1788513926.14 total 929.11 last 26.59) - open/close 15067 (time 1788513936.90 total 939.87 last 13.85) - open/close 15102 (time 1788513947.07 total 950.05 last 3.44) - open/close 15329 (time 1788513957.80 total 960.78 last 21.16) - open/close 15498 (time 1788513967.82 total 970.80 last 16.86) - open/close 15538 (time 1788513978.16 total 981.14 last 3.87) - open/close 15771 (time 1788513989.10 total 992.08 last 21.31) - open/close 16003 (time 1788513999.56 total 1002.53 last 22.18) - open/close 16215 (time 1788514009.56 total 1012.53 last 21.20) - open/close 16386 (time 1788514019.81 total 1022.79 last 16.67) - open/close 16649 (time 1788514030.19 total 1033.17 last 25.34) - open/close 16913 (time 1788514040.20 total 1043.18 last 26.37) - open/close 17019 (time 1788514050.21 total 1053.19 last 10.59) - open/close 17208 (time 1788514060.23 total 1063.21 last 18.86) - open/close 17308 (time 1788514070.52 total 1073.50 last 9.71) - open/close 17494 (time 1788514080.54 total 1083.51 last 18.58) - open/close 17642 (time 1788514091.59 total 1094.57 last 13.39) - open/close 17825 (time 1788514101.59 total 1104.57 last 18.29) - open/close 18053 (time 1788514111.60 total 1114.57 last 22.80) - open/close 18809 (time 1788514121.60 total 1124.58 last 75.56) - open/close 19714 (time 1788514131.61 total 1134.59 last 90.43) - open/close 20000 (time 1788514134.89 total 1137.87 last 87.04) - open/close 20860 (time 1788514144.90 total 1147.87 last 85.98) - open/close 21766 (time 1788514154.90 total 1157.88 last 90.55) - open/close 22677 (time 1788514164.91 total 1167.89 last 91.04) - open/close 23583 (time 1788514174.91 total 1177.89 last 90.56) - open/close 24425 (time 1788514184.92 total 1187.90 last 84.11) - open/close 25233 (time 1788514194.93 total 1197.91 last 80.73) - open/close 26132 (time 1788514204.94 total 1207.92 last 89.84) - open/close 27009 (time 1788514214.95 total 1217.93 last 87.60) - open/close 27742 (time 1788514224.95 total 1227.93 last 73.27) - open/close 28499 (time 1788514234.96 total 1237.94 last 75.66) - open/close 29268 (time 1788514244.97 total 1247.95 last 76.81) - open/close 29901 (time 1788514254.98 total 1257.95 last 63.27) - open/close 30810 (time 1788514264.98 total 1267.96 last 90.88) - open/close 31589 (time 1788514274.99 total 1277.97 last 77.81) - open/close 32446 (time 1788514285.02 total 1287.99 last 85.48) - open/close 33330 (time 1788514295.02 total 1298.00 last 88.38) - open/close 34107 (time 1788514305.03 total 1308.01 last 77.59) - open/close 34998 (time 1788514315.05 total 1318.02 last 88.99) - open/close 35856 (time 1788514325.05 total 1328.02 last 85.79) - open/close 36606 (time 1788514335.05 total 1338.03 last 74.97) - open/close 37537 (time 1788514345.06 total 1348.03 last 93.04) - open/close 38439 (time 1788514355.06 total 1358.04 last 90.19) - open/close 39322 (time 1788514365.07 total 1368.04 last 88.24) - open/close 40000 (time 1788514372.45 total 1375.43 last 91.77) - open/close 40725 (time 1788514382.47 total 1385.44 last 72.41) - open/close 41452 (time 1788514392.51 total 1395.49 last 72.39) - open/close 42282 (time 1788514402.51 total 1405.49 last 82.96) - open/close 43085 (time 1788514412.51 total 1415.49 last 80.29) - open/close 43903 (time 1788514422.52 total 1425.50 last 81.74) - open/close 44811 (time 1788514432.52 total 1435.50 last 90.77) - open/close 45674 (time 1788514442.53 total 1445.51 last 86.25) - open/close 46521 (time 1788514452.54 total 1455.51 last 84.65) - open/close 47303 (time 1788514462.54 total 1465.52 last 78.17) - open/close 48050 (time 1788514472.54 total 1475.52 last 74.68) - open/close 48816 (time 1788514482.55 total 1485.53 last 76.54) - open/close 49619 (time 1788514492.56 total 1495.53 last 80.27) total: 50000 open/close in 1500.16 seconds: 33.33 ops/second - unlinked 0 (time 1788514502 ; total 0 ; last 0) - unlinked 10000 (time 1788514633 ; total 131 ; last 131) - unlinked 20000 (time 1788514760 ; total 258 ; last 127) - unlinked 30000 (time 1788514878 ; total 376 ; last 118) - unlinked 40000 (time 1788515002 ; total 500 ; last 124) total: 50000 unlinks in 620 seconds: 80.645164 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg241-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg241-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg241-server stop mds service on oleg241-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg241-server unloading modules via unload_modules_local on: 'oleg241-server' oleg241-server: oleg241-server.virtnet: executing unload_modules_local oleg241-server: kmemleak disabled modules unloaded. pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-server: ssh exited with exit code 2 pdsh@oleg241-client: oleg241-client: ssh exited with exit code 2