== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:45:34 (1790775934) start mds service on oleg233-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1952992 Sep 29 14:05 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg233-server' oleg233-server: oleg233-server.virtnet: executing load_modules_local oleg233-server: kmemleak disabled oleg233-server: Loading modules from /home/green/git/lustre-release/lustre oleg233-server: detected 4 online CPUs by sysfs oleg233-server: MODOPTS_LIBCFS= oleg233-server: Force libcfs to create 2 CPU partitions oleg233-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg233-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg233-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg233-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg233-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg233-server: oleg233-server.virtnet: executing set_default_debug -1 all pdsh@oleg233-client: oleg233-server: ssh exited with exit code 1 Started lustre-OST0000 oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg233-client.virtnet: -o user_xattr,flock 192.168.202.133@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 153 (time 1790776019.73 total 10.46 last 14.63) - open/close 300 (time 1790776030.03 total 20.75 last 14.28) - open/close 488 (time 1790776040.71 total 31.43 last 17.60) - open/close 666 (time 1790776051.48 total 42.20 last 16.53) - open/close 915 (time 1790776061.50 total 52.23 last 24.83) - open/close 1042 (time 1790776071.91 total 62.64 last 12.20) - open/close 1236 (time 1790776081.93 total 72.65 last 19.37) - open/close 1423 (time 1790776092.29 total 83.02 last 18.05) - open/close 1681 (time 1790776102.31 total 93.04 last 25.75) - open/close 1755 (time 1790776112.92 total 103.64 last 6.98) - open/close 1973 (time 1790776122.93 total 113.66 last 21.76) - open/close 2222 (time 1790776133.84 total 124.56 last 22.83) - open/close 2477 (time 1790776143.85 total 134.58 last 25.46) - open/close 2617 (time 1790776153.87 total 144.59 last 13.98) - open/close 2841 (time 1790776164.56 total 155.29 last 20.94) - open/close 3073 (time 1790776174.58 total 165.31 last 23.16) - open/close 3248 (time 1790776184.60 total 175.33 last 17.46) - open/close 3340 (time 1790776197.70 total 188.43 last 7.02) - open/close 3582 (time 1790776208.14 total 198.86 last 23.19) - open/close 3832 (time 1790776218.76 total 209.48 last 23.54) - open/close 4085 (time 1790776229.01 total 219.74 last 24.67) - open/close 4381 (time 1790776239.30 total 230.03 last 28.78) - open/close 4579 (time 1790776249.80 total 240.52 last 18.86) - open/close 4769 (time 1790776260.86 total 251.59 last 17.17) - open/close 4977 (time 1790776271.91 total 262.63 last 18.83) - open/close 5211 (time 1790776281.92 total 272.64 last 23.37) - open/close 5443 (time 1790776292.93 total 283.66 last 21.07) - open/close 5624 (time 1790776303.56 total 294.29 last 17.03) - open/close 5865 (time 1790776316.10 total 306.82 last 19.23) - open/close 6055 (time 1790776326.10 total 316.83 last 18.99) - open/close 6133 (time 1790776336.14 total 326.86 last 7.77) - open/close 6279 (time 1790776346.69 total 337.41 last 13.84) - open/close 6393 (time 1790776357.11 total 347.84 last 10.93) - open/close 6624 (time 1790776367.95 total 358.68 last 21.31) - open/close 6855 (time 1790776378.48 total 369.21 last 21.94) - open/close 7100 (time 1790776388.50 total 379.22 last 24.46) - open/close 7268 (time 1790776399.24 total 389.97 last 15.63) - open/close 7472 (time 1790776409.25 total 399.98 last 20.39) - open/close 7646 (time 1790776419.25 total 409.98 last 17.39) - open/close 7875 (time 1790776430.09 total 420.82 last 21.13) - open/close 8072 (time 1790776440.26 total 430.98 last 19.38) - open/close 8290 (time 1790776450.27 total 441.00 last 21.77) - open/close 8486 (time 1790776460.93 total 451.66 last 18.38) - open/close 8756 (time 1790776470.94 total 461.67 last 26.98) - open/close 8949 (time 1790776480.95 total 471.68 last 19.28) - open/close 9158 (time 1790776491.86 total 482.59 last 19.15) - open/close 9319 (time 1790776502.23 total 492.96 last 15.53) - open/close 9421 (time 1790776512.28 total 503.00 last 10.15) - open/close 9520 (time 1790776522.37 total 513.10 last 9.81) - open/close 9692 (time 1790776533.26 total 523.98 last 15.80) - open/close 9908 (time 1790776543.75 total 534.47 last 20.60) - open/close 10000 (time 1790776550.79 total 541.51 last 13.07) - open/close 10211 (time 1790776560.80 total 551.53 last 21.07) - open/close 10441 (time 1790776571.27 total 562.00 last 21.96) - open/close 10663 (time 1790776582.92 total 573.65 last 19.06) - open/close 10903 (time 1790776592.93 total 583.65 last 23.98) - open/close 11161 (time 1790776602.93 total 593.66 last 25.78) - open/close 11415 (time 1790776613.66 total 604.39 last 23.68) - open/close 11656 (time 1790776623.67 total 614.40 last 24.08) - open/close 11911 (time 1790776634.32 total 625.05 last 23.94) - open/close 12141 (time 1790776644.61 total 635.34 last 22.36) - open/close 12374 (time 1790776654.63 total 645.35 last 23.26) - open/close 12625 (time 1790776665.71 total 656.44 last 22.65) - open/close 12777 (time 1790776676.49 total 667.22 last 14.09) - open/close 12858 (time 1790776687.31 total 678.03 last 7.49) - open/close 13049 (time 1790776697.41 total 688.13 last 18.91) - open/close 13164 (time 1790776707.72 total 698.44 last 11.15) - open/close 13383 (time 1790776717.73 total 708.45 last 21.88) - open/close 13592 (time 1790776728.60 total 719.32 last 19.23) - open/close 13831 (time 1790776738.62 total 729.34 last 23.86) - open/close 14024 (time 1790776748.62 total 739.34 last 19.30) - open/close 14254 (time 1790776759.01 total 749.73 last 22.14) - open/close 14445 (time 1790776769.03 total 759.75 last 19.06) - open/close 14544 (time 1790776779.94 total 770.67 last 9.07) - open/close 14634 (time 1790776789.98 total 780.71 last 8.96) - open/close 14838 (time 1790776800.80 total 791.53 last 18.86) - open/close 15069 (time 1790776810.81 total 801.53 last 23.08) - open/close 15236 (time 1790776821.78 total 812.50 last 15.22) - open/close 15325 (time 1790776832.18 total 822.90 last 8.56) - open/close 15436 (time 1790776842.58 total 833.31 last 10.67) - open/close 15608 (time 1790776853.25 total 843.98 last 16.12) - open/close 15764 (time 1790776863.81 total 854.54 last 14.78) - open/close 15949 (time 1790776874.14 total 864.87 last 17.91) - open/close 16133 (time 1790776884.16 total 874.89 last 18.37) - open/close 16307 (time 1790776895.19 total 885.91 last 15.78) - open/close 16442 (time 1790776905.19 total 895.92 last 13.49) - open/close 16668 (time 1790776915.22 total 905.94 last 22.55) - open/close 16871 (time 1790776926.57 total 917.29 last 17.89) - open/close 17109 (time 1790776936.58 total 927.30 last 23.77) - open/close 17214 (time 1790776946.62 total 937.34 last 10.46) - open/close 17339 (time 1790776957.19 total 947.91 last 11.82) - open/close 17595 (time 1790776967.21 total 957.94 last 25.54) - open/close 17804 (time 1790776977.67 total 968.39 last 19.99) - open/close 17912 (time 1790776988.61 total 979.34 last 9.87) - open/close 18003 (time 1790776999.11 total 989.83 last 8.67) - open/close 18132 (time 1790777009.11 total 999.84 last 12.90) - open/close 18647 (time 1790777019.13 total 1009.85 last 51.42) - open/close 19405 (time 1790777029.13 total 1019.86 last 75.73) - open/close 20000 (time 1790777038.07 total 1028.79 last 66.62) - open/close 20773 (time 1790777048.07 total 1038.80 last 77.24) - open/close 21580 (time 1790777058.10 total 1048.82 last 80.50) - open/close 22337 (time 1790777068.10 total 1058.83 last 75.68) - open/close 23001 (time 1790777078.11 total 1068.83 last 66.36) - open/close 23823 (time 1790777088.11 total 1078.83 last 82.19) - open/close 24465 (time 1790777098.11 total 1088.84 last 64.17) - open/close 25240 (time 1790777108.12 total 1098.85 last 77.41) - open/close 26093 (time 1790777118.12 total 1108.85 last 85.29) - open/close 26891 (time 1790777128.13 total 1118.85 last 79.78) - open/close 27704 (time 1790777138.13 total 1128.86 last 81.26) - open/close 28521 (time 1790777148.14 total 1138.86 last 81.67) - open/close 29393 (time 1790777158.14 total 1148.87 last 87.17) - open/close 30000 (time 1790777164.94 total 1155.67 last 89.26) - open/close 30835 (time 1790777174.96 total 1165.68 last 83.36) - open/close 31626 (time 1790777184.96 total 1175.69 last 79.05) - open/close 32455 (time 1790777194.96 total 1185.69 last 82.90) - open/close 33172 (time 1790777204.97 total 1195.69 last 71.66) - open/close 33977 (time 1790777214.98 total 1205.70 last 80.44) - open/close 34695 (time 1790777225.00 total 1215.73 last 71.62) - open/close 35532 (time 1790777235.00 total 1225.73 last 83.67) - open/close 36352 (time 1790777245.00 total 1235.73 last 82.00) - open/close 37228 (time 1790777255.01 total 1245.73 last 87.58) - open/close 38033 (time 1790777265.01 total 1255.74 last 80.46) - open/close 38716 (time 1790777275.02 total 1265.74 last 68.25) - open/close 39363 (time 1790777285.02 total 1275.75 last 64.68) - open/close 40000 (time 1790777292.76 total 1283.48 last 82.34) - open/close 40942 (time 1790777302.76 total 1293.48 last 94.19) - open/close 41767 (time 1790777312.77 total 1303.49 last 82.43) - open/close 42479 (time 1790777322.78 total 1313.50 last 71.11) - open/close 43294 (time 1790777332.79 total 1323.51 last 81.44) - open/close 43987 (time 1790777342.80 total 1333.52 last 69.23) - open/close 44797 (time 1790777352.80 total 1343.53 last 80.95) - open/close 45631 (time 1790777362.81 total 1353.54 last 83.34) - open/close 46503 (time 1790777372.81 total 1363.54 last 87.18) - open/close 47310 (time 1790777382.83 total 1373.55 last 80.58) - open/close 48105 (time 1790777392.83 total 1383.56 last 79.46) - open/close 48964 (time 1790777402.84 total 1393.56 last 85.84) - open/close 49700 (time 1790777412.85 total 1403.58 last 73.52) total: 50000 open/close in 1408.20 seconds: 35.51 ops/second - unlinked 0 (time 1790777421 ; total 0 ; last 0) - unlinked 10000 (time 1790777548 ; total 127 ; last 127) - unlinked 20000 (time 1790777671 ; total 250 ; last 123) - unlinked 30000 (time 1790777794 ; total 373 ; last 123) - unlinked 40000 (time 1790777920 ; total 499 ; last 126) total: 50000 unlinks in 627 seconds: 79.744820 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg233-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg233-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg233-server stop mds service on oleg233-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg233-server unloading modules via unload_modules_local on: 'oleg233-server' oleg233-server: oleg233-server.virtnet: executing unload_modules_local oleg233-server: kmemleak disabled modules unloaded. pdsh@oleg233-client: oleg233-server: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-client: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-client: ssh exited with exit code 2 pdsh@oleg233-client: oleg233-server: ssh exited with exit code 2