== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:52:03 (1789447923) start mds service on oleg134-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1933448 Sep 14 13:54 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing load_modules_local oleg134-server: kmemleak disabled oleg134-server: Loading modules from /home/green/git/lustre-release/lustre oleg134-server: detected 4 online CPUs by sysfs oleg134-server: MODOPTS_LIBCFS= oleg134-server: Force libcfs to create 2 CPU partitions oleg134-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg134-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg134-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg134-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg134-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg134-server: oleg134-server.virtnet: executing set_default_debug -1 all pdsh@oleg134-client: oleg134-server: ssh exited with exit code 1 Started lustre-OST0000 oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg134-client.virtnet: -o user_xattr,flock 192.168.201.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 152 (time 1789448017.58 total 10.01 last 15.19) - open/close 288 (time 1789448027.58 total 20.01 last 13.60) - open/close 387 (time 1789448042.96 total 35.39 last 6.44) - open/close 560 (time 1789448055.34 total 47.78 last 13.97) - open/close 761 (time 1789448065.37 total 57.80 last 20.04) - open/close 962 (time 1789448076.48 total 68.91 last 18.10) - open/close 1149 (time 1789448086.49 total 78.92 last 18.68) - open/close 1244 (time 1789448096.50 total 88.94 last 9.49) - open/close 1453 (time 1789448107.08 total 99.52 last 19.75) - open/close 1702 (time 1789448117.08 total 109.52 last 24.90) - open/close 1840 (time 1789448127.23 total 119.67 last 13.60) - open/close 1981 (time 1789448138.37 total 130.80 last 12.66) - open/close 2120 (time 1789448148.43 total 140.86 last 13.82) - open/close 2388 (time 1789448158.43 total 150.86 last 26.80) - open/close 2668 (time 1789448168.45 total 160.89 last 27.93) - open/close 2939 (time 1789448178.46 total 170.90 last 27.08) - open/close 3142 (time 1789448188.46 total 180.90 last 20.30) - open/close 3404 (time 1789448201.54 total 193.98 last 20.03) - open/close 3433 (time 1789448211.55 total 203.99 last 2.90) - open/close 3638 (time 1789448222.32 total 214.76 last 19.03) - open/close 3815 (time 1789448232.35 total 224.78 last 17.66) - open/close 3935 (time 1789448243.40 total 235.83 last 10.86) - open/close 4145 (time 1789448253.44 total 245.87 last 20.92) - open/close 4348 (time 1789448263.80 total 256.24 last 19.59) - open/close 4522 (time 1789448273.81 total 266.24 last 17.39) - open/close 4724 (time 1789448283.90 total 276.33 last 20.02) - open/close 4990 (time 1789448293.92 total 286.35 last 26.55) - open/close 5234 (time 1789448303.93 total 296.36 last 24.37) - open/close 5459 (time 1789448313.94 total 306.37 last 22.49) - open/close 5709 (time 1789448325.01 total 317.44 last 22.58) - open/close 5930 (time 1789448336.05 total 328.49 last 20.01) - open/close 6186 (time 1789448346.08 total 338.51 last 25.53) - open/close 6429 (time 1789448356.09 total 348.52 last 24.28) - open/close 6635 (time 1789448366.43 total 358.87 last 19.92) - open/close 6746 (time 1789448376.44 total 368.87 last 11.09) - open/close 6936 (time 1789448386.96 total 379.39 last 18.07) - open/close 7198 (time 1789448396.97 total 389.40 last 26.17) - open/close 7379 (time 1789448407.03 total 399.47 last 17.98) - open/close 7571 (time 1789448417.91 total 410.34 last 17.65) - open/close 7702 (time 1789448427.91 total 420.35 last 13.10) - open/close 7900 (time 1789448437.92 total 430.35 last 19.79) - open/close 8034 (time 1789448447.95 total 440.38 last 13.36) - open/close 8222 (time 1789448459.59 total 452.02 last 16.15) - open/close 8368 (time 1789448470.41 total 462.84 last 13.49) - open/close 8552 (time 1789448481.27 total 473.70 last 16.94) - open/close 8623 (time 1789448492.32 total 484.75 last 6.43) - open/close 8739 (time 1789448503.36 total 495.80 last 10.50) - open/close 8836 (time 1789448513.37 total 505.81 last 9.69) - open/close 9069 (time 1789448524.29 total 516.72 last 21.35) - open/close 9264 (time 1789448534.33 total 526.76 last 19.42) - open/close 9459 (time 1789448544.33 total 536.76 last 19.50) - open/close 9676 (time 1789448556.99 total 549.42 last 17.14) - open/close 9879 (time 1789448568.33 total 560.76 last 17.90) - open/close 10000 (time 1789448575.05 total 567.48 last 18.02) - open/close 10141 (time 1789448585.34 total 577.77 last 13.69) - open/close 10263 (time 1789448595.57 total 588.00 last 11.93) - open/close 10448 (time 1789448606.64 total 599.07 last 16.71) - open/close 10669 (time 1789448616.64 total 609.07 last 22.10) - open/close 10815 (time 1789448626.76 total 619.19 last 14.43) - open/close 11041 (time 1789448637.42 total 629.86 last 21.20) - open/close 11330 (time 1789448649.61 total 642.04 last 23.72) - open/close 11581 (time 1789448660.55 total 652.99 last 22.93) - open/close 11789 (time 1789448671.27 total 663.70 last 19.41) - open/close 11971 (time 1789448681.28 total 673.72 last 18.18) - open/close 12212 (time 1789448692.03 total 684.46 last 22.42) - open/close 12439 (time 1789448702.04 total 694.47 last 22.67) - open/close 12608 (time 1789448712.07 total 704.50 last 16.86) - open/close 12755 (time 1789448722.81 total 715.25 last 13.68) - open/close 13012 (time 1789448733.58 total 726.01 last 23.87) - open/close 13231 (time 1789448744.19 total 736.62 last 20.64) - open/close 13512 (time 1789448754.21 total 746.65 last 28.03) - open/close 13658 (time 1789448766.02 total 758.45 last 12.37) - open/close 13894 (time 1789448776.31 total 768.75 last 22.93) - open/close 14069 (time 1789448786.32 total 778.75 last 17.49) - open/close 14297 (time 1789448798.34 total 790.77 last 18.97) - open/close 14464 (time 1789448808.84 total 801.28 last 15.90) - open/close 14740 (time 1789448819.59 total 812.02 last 25.69) - open/close 14989 (time 1789448830.20 total 822.64 last 23.45) - open/close 15170 (time 1789448840.96 total 833.39 last 16.83) - open/close 15316 (time 1789448850.97 total 843.40 last 14.58) - open/close 15499 (time 1789448861.56 total 853.99 last 17.28) - open/close 15734 (time 1789448871.58 total 864.01 last 23.45) - open/close 15958 (time 1789448882.50 total 874.93 last 20.51) - open/close 16027 (time 1789448893.31 total 885.74 last 6.39) - open/close 16298 (time 1789448904.38 total 896.81 last 24.47) - open/close 16485 (time 1789448915.18 total 907.61 last 17.32) - open/close 16732 (time 1789448926.09 total 918.52 last 22.65) - open/close 16916 (time 1789448936.37 total 928.81 last 17.89) - open/close 17111 (time 1789448946.38 total 938.81 last 19.49) - open/close 17350 (time 1789448956.38 total 948.82 last 23.88) - open/close 17504 (time 1789448966.96 total 959.40 last 14.56) - open/close 17716 (time 1789448978.01 total 970.44 last 19.19) - open/close 17995 (time 1789448988.50 total 980.93 last 26.61) - open/close 18144 (time 1789448999.00 total 991.44 last 14.18) - open/close 18341 (time 1789449009.12 total 1001.56 last 19.47) - open/close 19108 (time 1789449019.12 total 1011.56 last 76.69) - open/close 19867 (time 1789449029.14 total 1021.57 last 75.80) - open/close 20576 (time 1789449039.15 total 1031.58 last 70.83) - open/close 21356 (time 1789449049.16 total 1041.59 last 77.94) - open/close 22016 (time 1789449059.18 total 1051.62 last 65.81) - open/close 22589 (time 1789449069.19 total 1061.63 last 57.25) - open/close 23498 (time 1789449079.20 total 1071.63 last 90.82) - open/close 24306 (time 1789449089.21 total 1081.64 last 80.74) - open/close 25093 (time 1789449099.22 total 1091.66 last 78.58) - open/close 25787 (time 1789449109.23 total 1101.67 last 69.32) - open/close 26416 (time 1789449119.24 total 1111.68 last 62.85) - open/close 27454 (time 1789449129.24 total 1121.68 last 103.78) - open/close 28310 (time 1789449139.25 total 1131.68 last 85.56) - open/close 29117 (time 1789449149.26 total 1141.70 last 80.57) - open/close 30000 (time 1789449158.47 total 1150.90 last 95.92) - open/close 30910 (time 1789449168.48 total 1160.91 last 90.95) - open/close 31717 (time 1789449178.48 total 1170.91 last 80.70) - open/close 32574 (time 1789449188.49 total 1180.92 last 85.61) - open/close 33413 (time 1789449198.49 total 1190.92 last 83.87) - open/close 34208 (time 1789449208.49 total 1200.92 last 79.49) - open/close 35075 (time 1789449218.55 total 1210.98 last 86.22) - open/close 35955 (time 1789449228.55 total 1220.98 last 88.00) - open/close 36765 (time 1789449238.55 total 1230.98 last 80.99) - open/close 37583 (time 1789449248.56 total 1240.99 last 81.74) - open/close 38226 (time 1789449258.56 total 1251.00 last 64.26) - open/close 38970 (time 1789449268.56 total 1261.00 last 74.38) - open/close 39699 (time 1789449278.58 total 1271.01 last 72.82) - open/close 40000 (time 1789449282.10 total 1274.53 last 85.49) - open/close 40846 (time 1789449292.10 total 1284.54 last 84.55) - open/close 41552 (time 1789449302.12 total 1294.55 last 70.49) - open/close 42322 (time 1789449312.13 total 1304.56 last 76.93) - open/close 43147 (time 1789449322.14 total 1314.57 last 82.39) - open/close 43988 (time 1789449332.16 total 1324.60 last 83.92) - open/close 44737 (time 1789449342.17 total 1334.60 last 74.85) - open/close 45544 (time 1789449352.18 total 1344.61 last 80.61) - open/close 46320 (time 1789449362.19 total 1354.62 last 77.56) - open/close 47194 (time 1789449372.20 total 1364.63 last 87.27) - open/close 48019 (time 1789449382.21 total 1374.64 last 82.43) - open/close 48763 (time 1789449392.21 total 1384.65 last 74.38) - open/close 49451 (time 1789449402.22 total 1394.65 last 68.77) total: 50000 open/close in 1401.52 seconds: 35.68 ops/second - unlinked 0 (time 1789449413 ; total 0 ; last 0) - unlinked 10000 (time 1789449552 ; total 139 ; last 139) - unlinked 20000 (time 1789449690 ; total 277 ; last 138) - unlinked 30000 (time 1789449826 ; total 413 ; last 136) - unlinked 40000 (time 1789449965 ; total 552 ; last 139) total: 50000 unlinks in 690 seconds: 72.463768 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg134-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg134-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg134-server stop mds service on oleg134-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg134-server unloading modules via unload_modules_local on: 'oleg134-server' oleg134-server: oleg134-server.virtnet: executing unload_modules_local oleg134-server: kmemleak disabled modules unloaded. pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-client: ssh exited with exit code 2 pdsh@oleg134-client: oleg134-server: ssh exited with exit code 2