== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:31:47 (1788910307) start mds service on oleg355-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1757936 Sep 8 18:39 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg355-server' oleg355-server: oleg355-server.virtnet: executing load_modules_local oleg355-server: kmemleak disabled oleg355-server: Loading modules from /home/green/git/lustre-release/lustre oleg355-server: detected 4 online CPUs by sysfs oleg355-server: MODOPTS_LIBCFS= oleg355-server: Force libcfs to create 2 CPU partitions oleg355-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg355-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg355-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg355-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg355-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg355-server: oleg355-server.virtnet: executing set_default_debug -1 all pdsh@oleg355-client: oleg355-server: ssh exited with exit code 1 Started lustre-OST0000 oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg355-client.virtnet: -o user_xattr,flock 192.168.203.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 421 (time 1788910348.70 total 10.92 last 38.56) - open/close 977 (time 1788910360.03 total 22.25 last 49.08) - open/close 1424 (time 1788910372.01 total 34.22 last 37.33) - open/close 1558 (time 1788910384.16 total 46.37 last 11.03) - open/close 1947 (time 1788910395.04 total 57.25 last 35.75) - open/close 2393 (time 1788910405.04 total 67.26 last 44.59) - open/close 2825 (time 1788910415.78 total 77.99 last 40.24) - open/close 3312 (time 1788910426.79 total 89.00 last 44.22) - open/close 3429 (time 1788910436.79 total 99.00 last 11.70) - open/close 3627 (time 1788910447.20 total 109.42 last 19.02) - open/close 3734 (time 1788910458.15 total 120.36 last 9.78) - open/close 4005 (time 1788910468.96 total 131.17 last 25.06) - open/close 4417 (time 1788910478.97 total 141.18 last 41.18) - open/close 4666 (time 1788910490.21 total 152.43 last 22.14) - open/close 5237 (time 1788910500.45 total 162.66 last 55.78) - open/close 5918 (time 1788910510.62 total 172.84 last 66.93) - open/close 6634 (time 1788910521.12 total 183.34 last 68.21) - open/close 7299 (time 1788910531.49 total 193.71 last 64.13) - open/close 7781 (time 1788910542.38 total 204.59 last 44.27) - open/close 8416 (time 1788910552.48 total 214.69 last 62.87) - open/close 9057 (time 1788910562.72 total 224.93 last 62.59) - open/close 9642 (time 1788910572.83 total 235.05 last 57.86) - open/close 10000 (time 1788910577.95 total 240.16 last 69.95) - open/close 10692 (time 1788910588.19 total 250.41 last 67.56) - open/close 11411 (time 1788910598.50 total 260.71 last 69.76) - open/close 11568 (time 1788910608.55 total 270.76 last 15.63) - open/close 11811 (time 1788910619.43 total 281.64 last 22.34) - open/close 12148 (time 1788910630.43 total 292.65 last 30.61) - open/close 12509 (time 1788910641.38 total 303.60 last 32.97) - open/close 12848 (time 1788910651.62 total 313.83 last 33.11) - open/close 13381 (time 1788910661.62 total 323.83 last 53.30) - open/close 13709 (time 1788910672.17 total 334.38 last 31.10) - open/close 14980 (time 1788910682.17 total 344.38 last 127.06) - open/close 17268 (time 1788910692.17 total 354.39 last 228.70) - open/close 19670 (time 1788910702.18 total 364.40 last 239.96) - open/close 20000 (time 1788910704.23 total 366.44 last 161.40) - open/close 22314 (time 1788910714.23 total 376.44 last 231.34) - open/close 24004 (time 1788910724.23 total 386.45 last 168.97) - open/close 24800 (time 1788910734.24 total 396.46 last 79.52) - open/close 25768 (time 1788910744.25 total 406.46 last 96.77) - open/close 26546 (time 1788910754.40 total 416.61 last 76.64) - open/close 27407 (time 1788910764.40 total 426.61 last 86.08) - open/close 28097 (time 1788910774.43 total 436.65 last 68.77) - open/close 28774 (time 1788910784.43 total 446.65 last 67.69) - open/close 29494 (time 1788910794.44 total 456.65 last 71.96) - open/close 30000 (time 1788910800.94 total 463.16 last 77.83) - open/close 30846 (time 1788910810.95 total 473.16 last 84.53) - open/close 31845 (time 1788910820.95 total 483.17 last 99.86) - open/close 32903 (time 1788910830.96 total 493.17 last 105.77) - open/close 33904 (time 1788910840.96 total 503.17 last 100.07) - open/close 34678 (time 1788910850.97 total 513.19 last 77.27) - open/close 35488 (time 1788910860.98 total 523.19 last 80.97) - open/close 36275 (time 1788910870.99 total 533.20 last 78.64) - open/close 37111 (time 1788910880.99 total 543.21 last 83.54) - open/close 38093 (time 1788910890.99 total 553.21 last 98.20) - open/close 39175 (time 1788910901.00 total 563.22 last 108.11) - open/close 40000 (time 1788910909.14 total 571.36 last 101.31) - open/close 40950 (time 1788910919.15 total 581.36 last 94.99) - open/close 41846 (time 1788910929.15 total 591.37 last 89.53) - open/close 42768 (time 1788910939.16 total 601.38 last 92.12) - open/close 43564 (time 1788910949.16 total 611.38 last 79.59) - open/close 44430 (time 1788910959.17 total 621.38 last 86.54) - open/close 45333 (time 1788910969.18 total 631.39 last 90.24) - open/close 46329 (time 1788910979.18 total 641.39 last 99.58) - open/close 47342 (time 1788910989.18 total 651.39 last 101.29) - open/close 48355 (time 1788910999.18 total 661.40 last 101.28) - open/close 49425 (time 1788911009.19 total 671.40 last 106.96) total: 50000 open/close in 675.38 seconds: 74.03 ops/second - unlinked 0 (time 1788911015 ; total 0 ; last 0) - unlinked 10000 (time 1788911062 ; total 47 ; last 47) - unlinked 20000 (time 1788911096 ; total 81 ; last 34) - unlinked 30000 (time 1788911133 ; total 118 ; last 37) - unlinked 40000 (time 1788911166 ; total 151 ; last 33) total: 50000 unlinks in 182 seconds: 274.725281 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg355-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg355-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg355-server stop mds service on oleg355-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg355-server unloading modules via unload_modules_local on: 'oleg355-server' oleg355-server: oleg355-server.virtnet: executing unload_modules_local oleg355-server: kmemleak disabled modules unloaded. pdsh@oleg355-client: oleg355-client: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-client: ssh exited with exit code 2 pdsh@oleg355-client: oleg355-server: ssh exited with exit code 2