== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:02:27 (1789052547) start mds service on oleg219-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1931568 Sep 10 01:50 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing load_modules_local oleg219-server: kmemleak disabled oleg219-server: Loading modules from /home/green/git/lustre-release/lustre oleg219-server: detected 4 online CPUs by sysfs oleg219-server: MODOPTS_LIBCFS= oleg219-server: Force libcfs to create 2 CPU partitions oleg219-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg219-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg219-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg219-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg219-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg219-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg219-server: oleg219-server.virtnet: executing set_default_debug -1 all pdsh@oleg219-client: oleg219-server: ssh exited with exit code 1 Started lustre-OST0000 oleg219-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg219-client.virtnet: -o user_xattr,flock 192.168.202.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 199 (time 1789052631.86 total 10.31 last 19.30) - open/close 369 (time 1789052641.87 total 20.31 last 17.00) - open/close 627 (time 1789052652.34 total 30.79 last 24.62) - open/close 891 (time 1789052662.72 total 41.17 last 25.44) - open/close 1086 (time 1789052672.82 total 51.27 last 19.30) - open/close 1227 (time 1789052683.19 total 61.63 last 13.60) - open/close 1317 (time 1789052693.74 total 72.19 last 8.52) - open/close 1412 (time 1789052706.36 total 84.80 last 7.53) - open/close 1568 (time 1789052717.12 total 95.56 last 14.50) - open/close 1792 (time 1789052727.16 total 105.61 last 22.30) - open/close 1917 (time 1789052737.17 total 115.62 last 12.49) - open/close 2037 (time 1789052747.36 total 125.80 last 11.78) - open/close 2176 (time 1789052757.76 total 136.20 last 13.36) - open/close 2388 (time 1789052767.76 total 146.20 last 21.20) - open/close 2626 (time 1789052778.04 total 156.48 last 23.16) - open/close 2843 (time 1789052789.91 total 168.36 last 18.27) - open/close 3050 (time 1789052799.92 total 178.37 last 20.69) - open/close 3176 (time 1789052809.99 total 188.43 last 12.52) - open/close 3369 (time 1789052820.37 total 198.82 last 18.59) - open/close 3446 (time 1789052830.45 total 208.90 last 7.64) - open/close 3688 (time 1789052840.47 total 218.91 last 24.16) - open/close 3931 (time 1789052850.49 total 228.94 last 24.24) - open/close 4157 (time 1789052860.92 total 239.37 last 21.67) - open/close 4358 (time 1789052871.04 total 249.48 last 19.87) - open/close 4513 (time 1789052882.05 total 260.50 last 14.08) - open/close 4635 (time 1789052892.05 total 270.50 last 12.19) - open/close 4769 (time 1789052902.60 total 281.05 last 12.70) - open/close 5045 (time 1789052913.52 total 291.96 last 25.29) - open/close 5229 (time 1789052923.53 total 301.98 last 18.38) - open/close 5406 (time 1789052933.56 total 312.01 last 17.65) - open/close 5515 (time 1789052944.49 total 322.93 last 9.97) - open/close 5716 (time 1789052956.53 total 334.98 last 16.69) - open/close 5971 (time 1789052967.44 total 345.88 last 23.38) - open/close 6140 (time 1789052977.44 total 355.89 last 16.89) - open/close 6364 (time 1789052987.75 total 366.20 last 21.72) - open/close 6561 (time 1789052997.79 total 376.23 last 19.64) - open/close 6684 (time 1789053008.37 total 386.82 last 11.62) - open/close 6854 (time 1789053018.74 total 397.18 last 16.40) - open/close 7101 (time 1789053029.46 total 407.91 last 23.03) - open/close 7335 (time 1789053039.49 total 417.93 last 23.34) - open/close 7462 (time 1789053049.59 total 428.04 last 12.57) - open/close 7600 (time 1789053059.63 total 438.07 last 13.75) - open/close 7762 (time 1789053069.75 total 448.19 last 16.01) - open/close 7954 (time 1789053080.13 total 458.57 last 18.50) - open/close 8061 (time 1789053090.38 total 468.83 last 10.44) - open/close 8145 (time 1789053100.41 total 478.86 last 8.38) - open/close 8232 (time 1789053110.42 total 488.87 last 8.69) - open/close 8426 (time 1789053121.31 total 499.76 last 17.82) - open/close 8666 (time 1789053131.97 total 510.42 last 22.52) - open/close 8915 (time 1789053141.97 total 520.42 last 24.89) - open/close 9114 (time 1789053151.98 total 530.43 last 19.88) - open/close 9305 (time 1789053161.99 total 540.44 last 19.09) - open/close 9494 (time 1789053172.59 total 551.03 last 17.84) - open/close 9691 (time 1789053182.66 total 561.11 last 19.56) - open/close 9834 (time 1789053192.67 total 571.12 last 14.28) - open/close 10000 (time 1789053202.35 total 580.79 last 17.16) - open/close 10203 (time 1789053212.73 total 591.18 last 19.54) - open/close 10369 (time 1789053222.76 total 601.21 last 16.56) - open/close 10601 (time 1789053232.78 total 611.23 last 23.15) - open/close 10789 (time 1789053242.81 total 621.26 last 18.74) - open/close 11022 (time 1789053253.14 total 631.59 last 22.55) - open/close 11135 (time 1789053263.16 total 641.61 last 11.28) - open/close 11276 (time 1789053273.41 total 651.86 last 13.76) - open/close 11444 (time 1789053283.46 total 661.91 last 16.71) - open/close 11670 (time 1789053293.46 total 671.91 last 22.60) - open/close 11852 (time 1789053303.67 total 682.11 last 17.84) - open/close 12114 (time 1789053313.85 total 692.30 last 25.72) - open/close 12341 (time 1789053323.98 total 702.43 last 22.41) - open/close 12596 (time 1789053333.99 total 712.43 last 25.50) - open/close 12816 (time 1789053343.99 total 722.44 last 21.99) - open/close 12933 (time 1789053354.10 total 732.54 last 11.58) - open/close 13047 (time 1789053365.58 total 744.03 last 9.92) - open/close 13244 (time 1789053376.33 total 754.78 last 18.33) - open/close 13475 (time 1789053386.34 total 764.79 last 23.07) - open/close 13662 (time 1789053396.53 total 774.98 last 18.36) - open/close 13823 (time 1789053406.54 total 784.99 last 16.08) - open/close 13956 (time 1789053416.57 total 795.02 last 13.26) - open/close 14098 (time 1789053426.58 total 805.03 last 14.18) - open/close 14268 (time 1789053436.84 total 815.29 last 16.57) - open/close 14311 (time 1789053447.56 total 826.01 last 4.01) - open/close 14465 (time 1789053458.25 total 836.70 last 14.41) - open/close 14572 (time 1789053468.81 total 847.25 last 10.13) - open/close 14770 (time 1789053479.49 total 857.94 last 18.54) - open/close 15016 (time 1789053489.50 total 867.94 last 24.58) - open/close 15257 (time 1789053499.51 total 877.96 last 24.06) - open/close 15403 (time 1789053509.76 total 888.21 last 14.25) - open/close 15624 (time 1789053519.93 total 898.38 last 21.73) - open/close 15769 (time 1789053529.94 total 908.39 last 14.49) - open/close 15937 (time 1789053540.22 total 918.67 last 16.34) - open/close 16101 (time 1789053550.42 total 928.87 last 16.08) - open/close 16305 (time 1789053560.43 total 938.87 last 20.39) - open/close 16498 (time 1789053570.45 total 948.89 last 19.26) - open/close 16609 (time 1789053581.32 total 959.76 last 10.21) - open/close 16772 (time 1789053591.87 total 970.32 last 15.45) - open/close 16881 (time 1789053601.88 total 980.33 last 10.89) - open/close 17034 (time 1789053612.87 total 991.32 last 13.92) - open/close 17234 (time 1789053623.62 total 1002.06 last 18.62) - open/close 17392 (time 1789053636.09 total 1014.53 last 12.67) - open/close 17616 (time 1789053646.10 total 1024.54 last 22.38) - open/close 17967 (time 1789053656.11 total 1034.56 last 35.04) - open/close 18574 (time 1789053666.13 total 1044.57 last 60.62) - open/close 19116 (time 1789053676.13 total 1054.57 last 54.19) - open/close 19775 (time 1789053686.14 total 1064.58 last 65.84) - open/close 20000 (time 1789053688.90 total 1067.34 last 81.53) - open/close 20771 (time 1789053698.90 total 1077.35 last 77.07) - open/close 21431 (time 1789053708.90 total 1087.35 last 66.00) - open/close 22147 (time 1789053718.91 total 1097.36 last 71.52) - open/close 22807 (time 1789053728.93 total 1107.37 last 65.88) - open/close 23375 (time 1789053738.93 total 1117.38 last 56.79) - open/close 24079 (time 1789053748.94 total 1127.38 last 70.35) - open/close 24862 (time 1789053758.95 total 1137.40 last 78.18) - open/close 25631 (time 1789053768.96 total 1147.40 last 76.85) - open/close 26426 (time 1789053778.97 total 1157.42 last 79.37) - open/close 27182 (time 1789053788.98 total 1167.43 last 75.55) - open/close 27875 (time 1789053798.98 total 1177.43 last 69.29) - open/close 28587 (time 1789053808.99 total 1187.43 last 71.16) - open/close 29267 (time 1789053818.99 total 1197.44 last 67.97) - open/close 29918 (time 1789053829.01 total 1207.45 last 65.00) - open/close 30455 (time 1789053839.01 total 1217.46 last 53.66) - open/close 30898 (time 1789053849.04 total 1227.48 last 44.20) - open/close 31494 (time 1789053859.04 total 1237.49 last 59.58) - open/close 32174 (time 1789053869.06 total 1247.51 last 67.87) - open/close 32920 (time 1789053879.10 total 1257.55 last 74.27) - open/close 33586 (time 1789053889.11 total 1267.55 last 66.58) - open/close 34258 (time 1789053899.11 total 1277.55 last 67.19) - open/close 34916 (time 1789053909.11 total 1287.56 last 65.76) - open/close 35631 (time 1789053919.13 total 1297.58 last 71.38) - open/close 36386 (time 1789053929.14 total 1307.59 last 75.39) - open/close 37058 (time 1789053939.15 total 1317.60 last 67.15) - open/close 37706 (time 1789053949.15 total 1327.60 last 64.80) - open/close 38281 (time 1789053959.17 total 1337.62 last 57.39) - open/close 38889 (time 1789053969.17 total 1347.62 last 60.78) - open/close 39718 (time 1789053979.18 total 1357.63 last 82.86) - open/close 40000 (time 1789053983.03 total 1361.48 last 73.20) - open/close 40591 (time 1789053993.03 total 1371.48 last 59.09) - open/close 41196 (time 1789054003.04 total 1381.49 last 60.43) - open/close 41866 (time 1789054013.06 total 1391.50 last 66.92) - open/close 42515 (time 1789054023.06 total 1401.51 last 64.89) - open/close 43201 (time 1789054033.06 total 1411.51 last 68.59) - open/close 43866 (time 1789054043.06 total 1421.51 last 66.48) - open/close 44413 (time 1789054053.13 total 1431.57 last 54.36) - open/close 44876 (time 1789054063.14 total 1441.59 last 46.24) - open/close 45481 (time 1789054073.16 total 1451.60 last 60.40) - open/close 46033 (time 1789054083.16 total 1461.61 last 55.17) - open/close 46630 (time 1789054093.17 total 1471.61 last 59.67) - open/close 47302 (time 1789054103.18 total 1481.62 last 67.14) - open/close 47946 (time 1789054113.18 total 1491.63 last 64.38) - open/close 48665 (time 1789054123.19 total 1501.64 last 71.80) - open/close 49392 (time 1789054133.20 total 1511.64 last 72.66) total: 50000 open/close in 1520.08 seconds: 32.89 ops/second - unlinked 0 (time 1789054146 ; total 0 ; last 0) - unlinked 10000 (time 1789054301 ; total 155 ; last 155) - unlinked 20000 (time 1789054458 ; total 312 ; last 157) - unlinked 30000 (time 1789054613 ; total 467 ; last 155) - unlinked 40000 (time 1789054769 ; total 623 ; last 156) total: 50000 unlinks in 772 seconds: 64.766838 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg219-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg219-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg219-server stop mds service on oleg219-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg219-server unloading modules via unload_modules_local on: 'oleg219-server' oleg219-server: oleg219-server.virtnet: executing unload_modules_local oleg219-server: kmemleak disabled modules unloaded. pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-client: ssh exited with exit code 2 pdsh@oleg219-client: oleg219-server: ssh exited with exit code 2