== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:13:50 (1788815630) start mds service on oleg120-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1931936 Sep 7 16:07 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing load_modules_local oleg120-server: kmemleak disabled oleg120-server: Loading modules from /home/green/git/lustre-release/lustre oleg120-server: detected 4 online CPUs by sysfs oleg120-server: MODOPTS_LIBCFS= oleg120-server: Force libcfs to create 2 CPU partitions oleg120-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg120-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg120-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg120-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg120-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg120-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg120-server: oleg120-server.virtnet: executing set_default_debug -1 all pdsh@oleg120-client: oleg120-server: ssh exited with exit code 1 Started lustre-OST0000 oleg120-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg120-client.virtnet: -o user_xattr,flock 192.168.201.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 335 (time 1788815678.58 total 11.06 last 30.28) - open/close 737 (time 1788815689.45 total 21.93 last 36.99) - open/close 1060 (time 1788815699.62 total 32.10 last 31.75) - open/close 1375 (time 1788815709.62 total 42.11 last 31.50) - open/close 1657 (time 1788815720.42 total 52.91 last 26.10) - open/close 1925 (time 1788815731.24 total 63.73 last 24.77) - open/close 2108 (time 1788815741.29 total 73.77 last 18.22) - open/close 2423 (time 1788815751.34 total 83.82 last 31.35) - open/close 2773 (time 1788815761.70 total 94.18 last 33.79) - open/close 3065 (time 1788815772.71 total 105.19 last 26.51) - open/close 3234 (time 1788815782.90 total 115.38 last 16.59) - open/close 3323 (time 1788815793.32 total 125.81 last 8.53) - open/close 3460 (time 1788815804.00 total 136.49 last 12.83) - open/close 3759 (time 1788815814.75 total 147.24 last 27.81) - open/close 4082 (time 1788815824.76 total 157.25 last 32.27) - open/close 4272 (time 1788815835.05 total 167.53 last 18.47) - open/close 4626 (time 1788815845.41 total 177.90 last 34.15) - open/close 4936 (time 1788815855.42 total 187.90 last 30.99) - open/close 5233 (time 1788815866.36 total 198.84 last 27.16) - open/close 5500 (time 1788815877.22 total 209.71 last 24.57) - open/close 5799 (time 1788815888.10 total 220.59 last 27.48) - open/close 6100 (time 1788815898.16 total 230.64 last 29.94) - open/close 6240 (time 1788815909.00 total 241.48 last 12.92) - open/close 6550 (time 1788815919.93 total 252.41 last 28.36) - open/close 6882 (time 1788815930.28 total 262.76 last 32.07) - open/close 7111 (time 1788815940.72 total 273.20 last 21.93) - open/close 7362 (time 1788815951.08 total 283.56 last 24.23) - open/close 7524 (time 1788815962.68 total 295.16 last 13.97) - open/close 7617 (time 1788815973.35 total 305.84 last 8.71) - open/close 7802 (time 1788815984.00 total 316.48 last 17.38) - open/close 7901 (time 1788815994.00 total 326.48 last 9.90) - open/close 8081 (time 1788816004.95 total 337.43 last 16.44) - open/close 8222 (time 1788816015.36 total 347.85 last 13.54) - open/close 8384 (time 1788816025.37 total 357.85 last 16.19) - open/close 8509 (time 1788816037.49 total 369.97 last 10.31) - open/close 8680 (time 1788816048.78 total 381.26 last 15.15) - open/close 8917 (time 1788816059.76 total 392.24 last 21.58) - open/close 9170 (time 1788816070.63 total 403.11 last 23.27) - open/close 9427 (time 1788816081.26 total 413.74 last 24.18) - open/close 9691 (time 1788816091.58 total 424.06 last 25.58) - open/close 9890 (time 1788816102.12 total 434.60 last 18.88) - open/close 10000 (time 1788816105.86 total 438.34 last 29.43) - open/close 10236 (time 1788816115.86 total 448.35 last 23.58) - open/close 10513 (time 1788816126.52 total 459.00 last 26.00) - open/close 10776 (time 1788816136.52 total 469.01 last 26.28) - open/close 10968 (time 1788816146.53 total 479.01 last 19.19) - open/close 11201 (time 1788816156.54 total 489.02 last 23.28) - open/close 11410 (time 1788816167.05 total 499.53 last 19.88) - open/close 11664 (time 1788816177.06 total 509.54 last 25.38) - open/close 11883 (time 1788816187.07 total 519.55 last 21.88) - open/close 12098 (time 1788816197.09 total 529.57 last 21.46) - open/close 12327 (time 1788816207.19 total 539.68 last 22.66) - open/close 12558 (time 1788816217.20 total 549.68 last 23.09) - open/close 12783 (time 1788816227.63 total 560.11 last 21.58) - open/close 12986 (time 1788816238.12 total 570.60 last 19.35) - open/close 13237 (time 1788816248.43 total 580.91 last 24.35) - open/close 13496 (time 1788816258.87 total 591.36 last 24.79) - open/close 13713 (time 1788816269.75 total 602.24 last 19.94) - open/close 14022 (time 1788816280.10 total 612.59 last 29.86) - open/close 14275 (time 1788816290.92 total 623.41 last 23.38) - open/close 14556 (time 1788816301.68 total 634.16 last 26.12) - open/close 14837 (time 1788816311.68 total 644.17 last 28.10) - open/close 14940 (time 1788816321.69 total 654.17 last 10.29) - open/close 15231 (time 1788816332.75 total 665.23 last 26.31) - open/close 15504 (time 1788816343.66 total 676.14 last 25.03) - open/close 15714 (time 1788816353.84 total 686.32 last 20.63) - open/close 16090 (time 1788816363.84 total 696.32 last 37.59) - open/close 16871 (time 1788816373.84 total 706.32 last 78.09) - open/close 17649 (time 1788816383.85 total 716.33 last 77.73) - open/close 18516 (time 1788816393.86 total 726.34 last 86.62) - open/close 19347 (time 1788816403.87 total 736.35 last 83.03) - open/close 20000 (time 1788816411.35 total 743.83 last 87.29) - open/close 20781 (time 1788816421.35 total 753.83 last 78.08) - open/close 21574 (time 1788816431.35 total 763.83 last 79.30) - open/close 22314 (time 1788816441.35 total 773.83 last 73.98) - open/close 23229 (time 1788816451.40 total 783.88 last 91.07) - open/close 23937 (time 1788816461.40 total 793.89 last 70.77) - open/close 24729 (time 1788816471.41 total 803.89 last 79.18) - open/close 25581 (time 1788816481.41 total 813.89 last 85.15) - open/close 26357 (time 1788816491.42 total 823.90 last 77.52) - open/close 27129 (time 1788816501.43 total 833.91 last 77.14) - open/close 27932 (time 1788816511.43 total 843.91 last 80.30) - open/close 28777 (time 1788816521.44 total 853.92 last 84.39) - open/close 29510 (time 1788816531.45 total 863.93 last 73.24) - open/close 30000 (time 1788816536.60 total 869.08 last 95.16) - open/close 30914 (time 1788816546.60 total 879.09 last 91.37) - open/close 31778 (time 1788816556.61 total 889.09 last 86.37) - open/close 32642 (time 1788816566.62 total 899.10 last 86.31) - open/close 33473 (time 1788816576.62 total 909.11 last 83.05) - open/close 34380 (time 1788816586.63 total 919.11 last 90.64) - open/close 35226 (time 1788816596.64 total 929.12 last 84.55) - open/close 36106 (time 1788816606.64 total 939.12 last 87.99) - open/close 36820 (time 1788816616.64 total 949.12 last 71.39) - open/close 37475 (time 1788816626.65 total 959.13 last 65.44) - open/close 38301 (time 1788816636.65 total 969.13 last 82.58) - open/close 39233 (time 1788816646.67 total 979.15 last 93.05) - open/close 40000 (time 1788816655.32 total 987.80 last 88.62) - open/close 40944 (time 1788816665.33 total 997.81 last 94.31) - open/close 41940 (time 1788816675.33 total 1007.82 last 99.57) - open/close 42979 (time 1788816685.33 total 1017.82 last 103.88) - open/close 44130 (time 1788816695.34 total 1027.82 last 115.01) - open/close 45137 (time 1788816705.35 total 1037.83 last 100.65) - open/close 46092 (time 1788816715.35 total 1047.83 last 95.47) - open/close 47056 (time 1788816725.36 total 1057.84 last 96.29) - open/close 48034 (time 1788816735.37 total 1067.85 last 97.74) - open/close 48972 (time 1788816745.37 total 1077.85 last 93.79) - open/close 49913 (time 1788816755.37 total 1087.85 last 94.10) total: 50000 open/close in 1088.89 seconds: 45.92 ops/second - unlinked 0 (time 1788816759 ; total 0 ; last 0) - unlinked 10000 (time 1788816869 ; total 110 ; last 110) - unlinked 20000 (time 1788816951 ; total 192 ; last 82) - unlinked 30000 (time 1788817007 ; total 248 ; last 56) - unlinked 40000 (time 1788817063 ; total 304 ; last 56) total: 50000 unlinks in 357 seconds: 140.056015 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg120-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg120-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg120-server stop mds service on oleg120-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg120-server unloading modules via unload_modules_local on: 'oleg120-server' oleg120-server: oleg120-server.virtnet: executing unload_modules_local oleg120-server: kmemleak disabled modules unloaded. pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-client: ssh exited with exit code 2 pdsh@oleg120-client: oleg120-server: ssh exited with exit code 2