== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:54:12 (1788854052) start mds service on oleg435-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931560 Sep 7 20:56 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: kmemleak disabled oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: detected 4 online CPUs by sysfs oleg435-server: MODOPTS_LIBCFS= oleg435-server: Force libcfs to create 2 CPU partitions oleg435-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg435-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg435-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg435-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg435-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 105 (time 1788854160.15 total 10.31 last 10.18) - open/close 226 (time 1788854170.17 total 20.32 last 12.09) - open/close 327 (time 1788854180.17 total 30.33 last 10.09) - open/close 500 (time 1788854191.11 total 41.27 last 15.81) - open/close 685 (time 1788854201.12 total 51.28 last 18.49) - open/close 892 (time 1788854211.17 total 61.33 last 20.59) - open/close 1066 (time 1788854223.78 total 73.94 last 13.80) - open/close 1189 (time 1788854234.25 total 84.41 last 11.75) - open/close 1385 (time 1788854244.28 total 94.44 last 19.55) - open/close 1594 (time 1788854254.82 total 104.98 last 19.82) - open/close 1745 (time 1788854265.12 total 115.28 last 14.67) - open/close 1912 (time 1788854275.12 total 125.28 last 16.70) - open/close 2014 (time 1788854285.86 total 136.02 last 9.49) - open/close 2138 (time 1788854296.37 total 146.53 last 11.80) - open/close 2237 (time 1788854306.38 total 156.54 last 9.89) - open/close 2380 (time 1788854316.47 total 166.63 last 14.17) - open/close 2489 (time 1788854326.50 total 176.66 last 10.87) - open/close 2632 (time 1788854337.06 total 187.22 last 13.54) - open/close 2761 (time 1788854347.63 total 197.79 last 12.20) - open/close 2899 (time 1788854357.66 total 207.81 last 13.77) - open/close 3035 (time 1788854367.95 total 218.11 last 13.21) - open/close 3161 (time 1788854378.29 total 228.45 last 12.18) - open/close 3301 (time 1788854388.30 total 238.46 last 14.00) - open/close 3467 (time 1788854398.32 total 248.48 last 16.56) - open/close 3634 (time 1788854408.55 total 258.71 last 16.32) - open/close 3802 (time 1788854419.44 total 269.60 last 15.43) - open/close 4053 (time 1788854429.45 total 279.61 last 25.08) - open/close 4240 (time 1788854439.63 total 289.79 last 18.36) - open/close 4386 (time 1788854449.66 total 299.82 last 14.56) - open/close 4539 (time 1788854460.64 total 310.80 last 13.93) - open/close 4677 (time 1788854470.87 total 321.03 last 13.49) - open/close 4841 (time 1788854481.56 total 331.72 last 15.34) - open/close 4900 (time 1788854491.56 total 341.72 last 5.90) - open/close 5019 (time 1788854501.58 total 351.74 last 11.88) - open/close 5191 (time 1788854511.59 total 361.75 last 17.19) - open/close 5389 (time 1788854522.91 total 373.07 last 17.48) - open/close 5641 (time 1788854533.46 total 383.62 last 23.88) - open/close 5776 (time 1788854544.48 total 394.64 last 12.26) - open/close 6001 (time 1788854556.21 total 406.37 last 19.18) - open/close 6229 (time 1788854567.07 total 417.23 last 20.99) - open/close 6384 (time 1788854577.08 total 427.24 last 15.47) - open/close 6469 (time 1788854587.12 total 437.28 last 8.47) - open/close 6563 (time 1788854597.99 total 448.15 last 8.65) - open/close 6750 (time 1788854608.68 total 458.84 last 17.50) - open/close 7004 (time 1788854618.69 total 468.85 last 25.36) - open/close 7192 (time 1788854628.78 total 478.94 last 18.64) - open/close 7283 (time 1788854639.34 total 489.50 last 8.61) - open/close 7372 (time 1788854649.36 total 499.52 last 8.88) - open/close 7547 (time 1788854659.74 total 509.89 last 16.87) - open/close 7783 (time 1788854669.75 total 519.91 last 23.57) - open/close 7999 (time 1788854680.34 total 530.50 last 20.39) - open/close 8186 (time 1788854690.35 total 540.51 last 18.68) - open/close 8395 (time 1788854701.02 total 551.18 last 19.60) - open/close 8624 (time 1788854713.95 total 564.11 last 17.71) - open/close 8870 (time 1788854724.70 total 574.86 last 22.88) - open/close 9070 (time 1788854735.00 total 585.16 last 19.42) - open/close 9207 (time 1788854745.63 total 595.79 last 12.89) - open/close 9347 (time 1788854756.65 total 606.81 last 12.71) - open/close 9481 (time 1788854767.46 total 617.62 last 12.39) - open/close 9596 (time 1788854777.47 total 627.63 last 11.49) - open/close 9692 (time 1788854787.48 total 637.64 last 9.60) - open/close 9812 (time 1788854799.06 total 649.22 last 10.36) - open/close 10000 (time 1788854808.73 total 658.89 last 19.45) - open/close 10151 (time 1788854819.56 total 669.72 last 13.94) - open/close 10268 (time 1788854830.18 total 680.34 last 11.02) - open/close 10377 (time 1788854840.34 total 690.50 last 10.72) - open/close 10544 (time 1788854851.44 total 701.60 last 15.05) - open/close 10643 (time 1788854862.61 total 712.77 last 8.86) - open/close 10804 (time 1788854872.64 total 722.80 last 16.06) - open/close 10975 (time 1788854884.19 total 734.35 last 14.80) - open/close 11096 (time 1788854894.88 total 745.04 last 11.32) - open/close 11142 (time 1788854905.23 total 755.39 last 4.44) - open/close 11278 (time 1788854915.43 total 765.59 last 13.34) - open/close 11390 (time 1788854925.43 total 775.59 last 11.20) - open/close 11580 (time 1788854937.22 total 787.38 last 16.11) - open/close 11829 (time 1788854947.23 total 797.39 last 24.89) - open/close 11992 (time 1788854957.60 total 807.76 last 15.71) - open/close 12251 (time 1788854967.78 total 817.94 last 25.44) - open/close 12478 (time 1788854977.80 total 827.96 last 22.66) - open/close 12618 (time 1788854988.83 total 838.99 last 12.69) - open/close 12866 (time 1788854999.41 total 849.57 last 23.44) - open/close 13143 (time 1788855010.02 total 860.18 last 26.11) - open/close 13331 (time 1788855023.02 total 873.18 last 14.46) - open/close 13494 (time 1788855033.05 total 883.21 last 16.24) - open/close 13714 (time 1788855045.52 total 895.68 last 17.65) - open/close 13922 (time 1788855055.54 total 905.70 last 20.76) - open/close 14150 (time 1788855065.70 total 915.86 last 22.45) - open/close 14376 (time 1788855076.54 total 926.70 last 20.84) - open/close 14537 (time 1788855086.98 total 937.14 last 15.43) - open/close 14729 (time 1788855098.16 total 948.32 last 17.17) - open/close 14890 (time 1788855109.23 total 959.39 last 14.55) - open/close 15029 (time 1788855119.23 total 969.39 last 13.89) - open/close 15188 (time 1788855129.25 total 979.41 last 15.87) - open/close 15272 (time 1788855139.29 total 989.45 last 8.37) - open/close 15358 (time 1788855149.93 total 1000.09 last 8.08) - open/close 15594 (time 1788855162.18 total 1012.34 last 19.27) - open/close 15778 (time 1788855173.03 total 1023.19 last 16.97) - open/close 15938 (time 1788855183.78 total 1033.94 last 14.88) - open/close 16046 (time 1788855194.72 total 1044.88 last 9.87) - open/close 16176 (time 1788855205.54 total 1055.70 last 12.02) - open/close 16300 (time 1788855216.52 total 1066.68 last 11.30) - open/close 16458 (time 1788855227.34 total 1077.50 last 14.60) - open/close 16588 (time 1788855237.63 total 1087.79 last 12.63) - open/close 16678 (time 1788855247.64 total 1097.80 last 9.00) - open/close 17131 (time 1788855257.66 total 1107.82 last 45.21) - open/close 17789 (time 1788855267.66 total 1117.82 last 65.77) - open/close 18527 (time 1788855277.68 total 1127.84 last 73.68) - open/close 19282 (time 1788855287.68 total 1137.84 last 75.47) - open/close 20000 (time 1788855296.85 total 1147.01 last 78.34) - open/close 20760 (time 1788855306.86 total 1157.02 last 75.91) - open/close 21367 (time 1788855316.86 total 1167.02 last 60.70) - open/close 22097 (time 1788855326.87 total 1177.03 last 72.93) - open/close 22800 (time 1788855336.89 total 1187.05 last 70.17) - open/close 23545 (time 1788855346.90 total 1197.06 last 74.45) - open/close 24314 (time 1788855356.90 total 1207.06 last 76.85) - open/close 25024 (time 1788855366.91 total 1217.07 last 70.94) - open/close 25706 (time 1788855376.93 total 1227.09 last 68.06) - open/close 26236 (time 1788855386.95 total 1237.11 last 52.93) - open/close 26829 (time 1788855396.96 total 1247.12 last 59.22) - open/close 27523 (time 1788855406.96 total 1257.12 last 69.37) - open/close 28383 (time 1788855416.97 total 1267.12 last 85.99) - open/close 29113 (time 1788855426.97 total 1277.13 last 72.97) - open/close 29812 (time 1788855436.98 total 1287.14 last 69.83) - open/close 30000 (time 1788855439.79 total 1289.95 last 66.97) - open/close 30650 (time 1788855449.80 total 1299.96 last 64.93) - open/close 31385 (time 1788855459.80 total 1309.96 last 73.49) - open/close 31982 (time 1788855469.81 total 1319.96 last 59.66) - open/close 32697 (time 1788855479.81 total 1329.97 last 71.49) - open/close 33373 (time 1788855489.82 total 1339.98 last 67.51) - open/close 34005 (time 1788855499.86 total 1350.02 last 62.97) - open/close 34714 (time 1788855509.87 total 1360.03 last 70.83) - open/close 35464 (time 1788855519.87 total 1370.03 last 75.00) - open/close 36258 (time 1788855529.87 total 1380.03 last 79.36) - open/close 37046 (time 1788855539.87 total 1390.03 last 78.78) - open/close 37622 (time 1788855549.89 total 1400.05 last 57.52) - open/close 38312 (time 1788855559.90 total 1410.06 last 68.95) - open/close 39003 (time 1788855569.90 total 1420.06 last 69.05) - open/close 39694 (time 1788855579.91 total 1430.07 last 69.04) - open/close 40000 (time 1788855584.11 total 1434.27 last 72.90) - open/close 40697 (time 1788855594.12 total 1444.28 last 69.64) - open/close 41292 (time 1788855604.13 total 1454.29 last 59.44) - open/close 41882 (time 1788855614.14 total 1464.30 last 58.93) - open/close 42502 (time 1788855624.14 total 1474.30 last 61.99) - open/close 43181 (time 1788855634.15 total 1484.31 last 67.88) - open/close 43916 (time 1788855644.16 total 1494.32 last 73.43) - open/close 44592 (time 1788855654.16 total 1504.32 last 67.59) - open/close 45354 (time 1788855664.17 total 1514.33 last 76.14) - open/close 45949 (time 1788855674.17 total 1524.33 last 59.46) - open/close 46591 (time 1788855684.17 total 1534.33 last 64.19) - open/close 47171 (time 1788855694.18 total 1544.34 last 57.97) - open/close 47869 (time 1788855704.19 total 1554.35 last 69.76) - open/close 48457 (time 1788855714.19 total 1564.35 last 58.78) - open/close 49130 (time 1788855724.21 total 1574.37 last 67.15) - open/close 49745 (time 1788855734.21 total 1584.37 last 61.48) total: 50000 open/close in 1588.08 seconds: 31.48 ops/second - unlinked 0 (time 1788855742 ; total 0 ; last 0) - unlinked 10000 (time 1788855902 ; total 160 ; last 160) - unlinked 20000 (time 1788856065 ; total 323 ; last 163) - unlinked 30000 (time 1788856221 ; total 479 ; last 156) - unlinked 40000 (time 1788856375 ; total 633 ; last 154) total: 50000 unlinks in 787 seconds: 63.532402 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server unloading modules via unload_modules_local on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing unload_modules_local oleg435-server: kmemleak disabled modules unloaded. pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2