== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:35:05 (1788392105) start mds service on oleg239-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1754936 Sep 2 18:43 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing load_modules_local oleg239-server: kmemleak disabled oleg239-server: Loading modules from /home/green/git/lustre-release/lustre oleg239-server: detected 4 online CPUs by sysfs oleg239-server: MODOPTS_LIBCFS= oleg239-server: Force libcfs to create 2 CPU partitions oleg239-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg239-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg239-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg239-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg239-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 177 (time 1788392148.53 total 10.42 last 16.98) - open/close 325 (time 1788392158.76 total 20.66 last 14.47) - open/close 547 (time 1788392168.77 total 30.67 last 22.18) - open/close 800 (time 1788392179.11 total 41.01 last 24.46) - open/close 1016 (time 1788392189.11 total 51.01 last 21.59) - open/close 1266 (time 1788392199.59 total 61.49 last 23.85) - open/close 1604 (time 1788392209.96 total 71.86 last 32.61) - open/close 1863 (time 1788392220.99 total 82.89 last 23.48) - open/close 2171 (time 1788392231.84 total 93.74 last 28.37) - open/close 2325 (time 1788392242.86 total 104.76 last 13.98) - open/close 2513 (time 1788392253.35 total 115.25 last 17.92) - open/close 2640 (time 1788392264.37 total 126.27 last 11.53) - open/close 2859 (time 1788392275.17 total 137.07 last 20.28) - open/close 3009 (time 1788392285.22 total 147.12 last 14.93) - open/close 3330 (time 1788392295.23 total 157.13 last 32.07) - open/close 3696 (time 1788392305.24 total 167.14 last 36.57) - open/close 3930 (time 1788392316.52 total 178.42 last 20.74) - open/close 4306 (time 1788392327.33 total 189.23 last 34.78) - open/close 4635 (time 1788392337.51 total 199.41 last 32.33) - open/close 4978 (time 1788392347.52 total 209.42 last 34.28) - open/close 5294 (time 1788392357.52 total 219.42 last 31.57) - open/close 5591 (time 1788392367.91 total 229.81 last 28.61) - open/close 5790 (time 1788392378.79 total 240.69 last 18.29) - open/close 6024 (time 1788392388.96 total 250.86 last 23.00) - open/close 6316 (time 1788392399.39 total 261.29 last 28.00) - open/close 6554 (time 1788392409.58 total 271.48 last 23.37) - open/close 6856 (time 1788392420.39 total 282.29 last 27.94) - open/close 7091 (time 1788392430.39 total 292.29 last 23.49) - open/close 7344 (time 1788392440.56 total 302.46 last 24.87) - open/close 7559 (time 1788392451.12 total 313.02 last 20.37) - open/close 7843 (time 1788392461.73 total 323.63 last 26.75) - open/close 8080 (time 1788392471.98 total 333.88 last 23.14) - open/close 8400 (time 1788392481.98 total 343.88 last 31.99) - open/close 8668 (time 1788392491.98 total 353.88 last 26.78) - open/close 8854 (time 1788392502.37 total 364.27 last 17.91) - open/close 9094 (time 1788392513.25 total 375.15 last 22.06) - open/close 9427 (time 1788392524.00 total 385.90 last 30.97) - open/close 9775 (time 1788392534.50 total 396.40 last 33.16) - open/close 10000 (time 1788392542.38 total 404.28 last 28.53) - open/close 10235 (time 1788392552.48 total 414.38 last 23.28) - open/close 10573 (time 1788392563.50 total 425.40 last 30.66) - open/close 10843 (time 1788392573.85 total 435.75 last 26.09) - open/close 11120 (time 1788392584.16 total 446.06 last 26.87) - open/close 11472 (time 1788392594.17 total 456.07 last 35.18) - open/close 12959 (time 1788392604.17 total 466.07 last 148.66) - open/close 14406 (time 1788392614.17 total 476.07 last 144.70) - open/close 16072 (time 1788392624.18 total 486.08 last 166.51) - open/close 17907 (time 1788392634.18 total 496.08 last 183.48) - open/close 19621 (time 1788392644.18 total 506.08 last 171.37) - open/close 20000 (time 1788392646.60 total 508.50 last 156.70) - open/close 22243 (time 1788392656.60 total 518.50 last 224.29) - open/close 24042 (time 1788392666.60 total 528.50 last 179.84) - open/close 25622 (time 1788392676.61 total 538.51 last 157.94) - open/close 27372 (time 1788392686.61 total 548.51 last 174.94) - open/close 29041 (time 1788392696.62 total 558.51 last 166.81) - open/close 30000 (time 1788392702.09 total 563.99 last 175.09) - open/close 31659 (time 1788392712.10 total 574.00 last 165.83) - open/close 33045 (time 1788392722.10 total 584.00 last 138.53) - open/close 34425 (time 1788392732.11 total 594.01 last 137.94) - open/close 35904 (time 1788392742.11 total 604.01 last 147.84) - open/close 37289 (time 1788392752.11 total 614.01 last 138.49) - open/close 38892 (time 1788392762.11 total 624.01 last 160.28) - open/close 40000 (time 1788392768.39 total 630.29 last 176.42) - open/close 41423 (time 1788392778.40 total 640.30 last 142.25) - open/close 42066 (time 1788392788.40 total 650.30 last 64.29) - open/close 42907 (time 1788392798.40 total 660.30 last 84.07) - open/close 43777 (time 1788392808.41 total 670.31 last 86.92) - open/close 44656 (time 1788392818.41 total 680.31 last 87.88) - open/close 45481 (time 1788392828.42 total 690.32 last 82.43) - open/close 46341 (time 1788392838.42 total 700.32 last 85.99) - open/close 47020 (time 1788392848.43 total 710.33 last 67.85) - open/close 48003 (time 1788392858.43 total 720.33 last 98.25) - open/close 49016 (time 1788392868.45 total 730.35 last 101.16) - open/close 49961 (time 1788392878.45 total 740.35 last 94.45) total: 50000 open/close in 740.72 seconds: 67.50 ops/second - unlinked 0 (time 1788392882 ; total 0 ; last 0) - unlinked 10000 (time 1788392952 ; total 70 ; last 70) - unlinked 20000 (time 1788393007 ; total 125 ; last 55) - unlinked 30000 (time 1788393061 ; total 179 ; last 54) - unlinked 40000 (time 1788393172 ; total 290 ; last 111) total: 50000 unlinks in 370 seconds: 135.135132 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg239-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server stop mds service on oleg239-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg239-server unloading modules via unload_modules_local on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing unload_modules_local oleg239-server: kmemleak disabled modules unloaded. pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-client: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-client: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2