== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:20:34 (1790774434) start mds service on oleg213-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x. 1 green green 1952976 Sep 29 14:03 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: kmemleak disabled oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions oleg213-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg213-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg213-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg213-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg213-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 228 (time 1790774512.97 total 10.00 last 22.79) - open/close 475 (time 1790774523.07 total 20.10 last 24.46) - open/close 703 (time 1790774533.39 total 30.42 last 22.10) - open/close 879 (time 1790774543.39 total 40.42 last 17.60) - open/close 1057 (time 1790774553.40 total 50.43 last 17.78) - open/close 1241 (time 1790774563.42 total 60.45 last 18.36) - open/close 1457 (time 1790774573.70 total 70.73 last 21.02) - open/close 1564 (time 1790774583.71 total 80.74 last 10.69) - open/close 1731 (time 1790774593.99 total 91.02 last 16.25) - open/close 2022 (time 1790774604.87 total 101.90 last 26.75) - open/close 2185 (time 1790774615.62 total 112.65 last 15.15) - open/close 2419 (time 1790774625.64 total 122.67 last 23.35) - open/close 2608 (time 1790774635.65 total 132.67 last 18.90) - open/close 2763 (time 1790774648.46 total 145.49 last 12.09) - open/close 2955 (time 1790774658.47 total 155.50 last 19.20) - open/close 3148 (time 1790774669.31 total 166.34 last 17.80) - open/close 3410 (time 1790774679.31 total 176.34 last 26.19) - open/close 3651 (time 1790774689.32 total 186.35 last 24.09) - open/close 3840 (time 1790774699.65 total 196.68 last 18.29) - open/close 4089 (time 1790774710.28 total 207.31 last 23.42) - open/close 4356 (time 1790774720.29 total 217.32 last 26.69) - open/close 4606 (time 1790774730.29 total 227.32 last 25.00) - open/close 4871 (time 1790774741.12 total 238.15 last 24.46) - open/close 5147 (time 1790774751.57 total 248.60 last 26.42) - open/close 5277 (time 1790774761.58 total 258.61 last 12.99) - open/close 5515 (time 1790774772.14 total 269.17 last 22.54) - open/close 5623 (time 1790774782.17 total 279.20 last 10.77) - open/close 5898 (time 1790774793.11 total 290.14 last 25.14) - open/close 6109 (time 1790774804.06 total 301.09 last 19.27) - open/close 6328 (time 1790774814.08 total 311.10 last 21.86) - open/close 6602 (time 1790774825.81 total 322.84 last 23.35) - open/close 6905 (time 1790774836.80 total 333.83 last 27.57) - open/close 7163 (time 1790774847.29 total 344.32 last 24.59) - open/close 7386 (time 1790774857.30 total 354.33 last 22.29) - open/close 7575 (time 1790774869.05 total 366.08 last 16.08) - open/close 7789 (time 1790774879.05 total 376.08 last 21.39) - open/close 8058 (time 1790774889.06 total 386.09 last 26.89) - open/close 8356 (time 1790774899.45 total 396.48 last 28.67) - open/close 8635 (time 1790774909.71 total 406.74 last 27.20) - open/close 8782 (time 1790774920.22 total 417.25 last 13.99) - open/close 8964 (time 1790774930.22 total 427.25 last 18.20) - open/close 9056 (time 1790774940.75 total 437.77 last 8.74) - open/close 9279 (time 1790774951.17 total 448.20 last 21.40) - open/close 9540 (time 1790774962.04 total 459.07 last 24.01) - open/close 9803 (time 1790774972.04 total 469.07 last 26.29) - open/close 9934 (time 1790774982.14 total 479.17 last 12.97) - open/close 10000 (time 1790774989.93 total 486.96 last 8.47) - open/close 10128 (time 1790775000.46 total 497.49 last 12.16) - open/close 10312 (time 1790775010.49 total 507.52 last 18.33) - open/close 10504 (time 1790775021.05 total 518.08 last 18.18) - open/close 10699 (time 1790775032.16 total 529.19 last 17.55) - open/close 10932 (time 1790775047.38 total 544.41 last 15.32) - open/close 11122 (time 1790775057.39 total 554.42 last 18.98) - open/close 11344 (time 1790775067.41 total 564.44 last 22.14) - open/close 11539 (time 1790775077.96 total 574.99 last 18.49) - open/close 11703 (time 1790775088.84 total 585.87 last 15.07) - open/close 11914 (time 1790775098.86 total 595.89 last 21.07) - open/close 12185 (time 1790775109.76 total 606.79 last 24.85) - open/close 12394 (time 1790775119.78 total 616.81 last 20.86) - open/close 12549 (time 1790775129.81 total 626.84 last 15.46) - open/close 12675 (time 1790775140.81 total 637.84 last 11.46) - open/close 12920 (time 1790775151.20 total 648.23 last 23.56) - open/close 13178 (time 1790775161.21 total 658.24 last 25.79) - open/close 13378 (time 1790775171.48 total 668.51 last 19.48) - open/close 13554 (time 1790775185.64 total 682.67 last 12.42) - open/close 13738 (time 1790775195.66 total 692.69 last 18.37) - open/close 13797 (time 1790775205.71 total 702.74 last 5.87) - open/close 14026 (time 1790775216.72 total 713.75 last 20.80) - open/close 14264 (time 1790775226.72 total 723.75 last 23.79) - open/close 14382 (time 1790775236.74 total 733.77 last 11.79) - open/close 14615 (time 1790775246.74 total 743.77 last 23.29) - open/close 14750 (time 1790775257.06 total 754.09 last 13.08) - open/close 14972 (time 1790775267.18 total 764.21 last 21.95) - open/close 15235 (time 1790775278.12 total 775.15 last 24.04) - open/close 15506 (time 1790775289.09 total 786.12 last 24.70) - open/close 15781 (time 1790775299.10 total 796.13 last 27.47) - open/close 16024 (time 1790775309.25 total 806.28 last 23.94) - open/close 16325 (time 1790775319.97 total 817.00 last 28.09) - open/close 16565 (time 1790775330.69 total 827.72 last 22.40) - open/close 16722 (time 1790775341.05 total 838.08 last 15.15) - open/close 16967 (time 1790775351.69 total 848.72 last 23.01) - open/close 17242 (time 1790775361.70 total 858.73 last 27.48) - open/close 17505 (time 1790775371.70 total 868.73 last 26.30) - open/close 17621 (time 1790775382.28 total 879.31 last 10.97) - open/close 17731 (time 1790775392.29 total 889.32 last 10.99) - open/close 17858 (time 1790775402.50 total 899.53 last 12.44) - open/close 18091 (time 1790775412.50 total 909.53 last 23.28) - open/close 18283 (time 1790775422.85 total 919.88 last 18.56) - open/close 18525 (time 1790775433.03 total 930.06 last 23.76) - open/close 18792 (time 1790775443.73 total 940.76 last 24.96) - open/close 18949 (time 1790775453.73 total 950.76 last 15.70) - open/close 19043 (time 1790775463.75 total 960.78 last 9.39) - open/close 19193 (time 1790775473.77 total 970.80 last 14.96) - open/close 19559 (time 1790775483.80 total 980.83 last 36.51) - open/close 20000 (time 1790775491.65 total 988.68 last 56.15) - open/close 20591 (time 1790775501.65 total 998.68 last 59.08) - open/close 21188 (time 1790775511.66 total 1008.69 last 59.67) - open/close 22006 (time 1790775521.68 total 1018.71 last 81.65) - open/close 22815 (time 1790775531.68 total 1028.71 last 80.85) - open/close 23677 (time 1790775541.70 total 1038.73 last 86.09) - open/close 24459 (time 1790775551.70 total 1048.73 last 78.17) - open/close 25204 (time 1790775561.71 total 1058.74 last 74.45) - open/close 26011 (time 1790775571.72 total 1068.75 last 80.62) - open/close 26783 (time 1790775581.72 total 1078.75 last 77.15) - open/close 27653 (time 1790775591.73 total 1088.76 last 86.97) - open/close 28492 (time 1790775601.73 total 1098.76 last 83.88) - open/close 29198 (time 1790775611.75 total 1108.78 last 70.43) - open/close 29949 (time 1790775621.76 total 1118.79 last 75.06) - open/close 30598 (time 1790775631.76 total 1128.79 last 64.87) - open/close 31264 (time 1790775641.79 total 1138.82 last 66.44) - open/close 32091 (time 1790775651.80 total 1148.83 last 82.62) - open/close 32928 (time 1790775661.80 total 1158.83 last 83.68) - open/close 33801 (time 1790775671.81 total 1168.84 last 87.23) - open/close 34626 (time 1790775681.81 total 1178.84 last 82.45) - open/close 35418 (time 1790775691.81 total 1188.84 last 79.18) - open/close 36275 (time 1790775701.82 total 1198.85 last 85.64) - open/close 37064 (time 1790775711.83 total 1208.86 last 78.85) - open/close 37881 (time 1790775721.83 total 1218.86 last 81.69) - open/close 38735 (time 1790775731.84 total 1228.87 last 85.32) - open/close 39509 (time 1790775741.85 total 1238.88 last 77.29) - open/close 40000 (time 1790775748.11 total 1245.14 last 78.43) - open/close 40830 (time 1790775758.12 total 1255.15 last 82.93) - open/close 41642 (time 1790775768.12 total 1265.15 last 81.20) - open/close 42559 (time 1790775778.13 total 1275.16 last 91.64) - open/close 43392 (time 1790775788.13 total 1285.16 last 83.28) - open/close 44149 (time 1790775798.14 total 1295.17 last 75.64) - open/close 44900 (time 1790775808.14 total 1305.17 last 75.08) - open/close 45671 (time 1790775818.15 total 1315.18 last 77.08) - open/close 46500 (time 1790775828.15 total 1325.18 last 82.86) - open/close 47332 (time 1790775838.17 total 1335.20 last 83.02) - open/close 48151 (time 1790775848.17 total 1345.20 last 81.89) - open/close 48928 (time 1790775858.18 total 1355.21 last 77.64) - open/close 49694 (time 1790775868.18 total 1365.21 last 76.59) total: 50000 open/close in 1368.63 seconds: 36.53 ops/second - unlinked 0 (time 1790775876 ; total 0 ; last 0) - unlinked 10000 (time 1790775998 ; total 122 ; last 122) - unlinked 20000 (time 1790776129 ; total 253 ; last 131) - unlinked 30000 (time 1790776257 ; total 381 ; last 128) - unlinked 40000 (time 1790776388 ; total 512 ; last 131) total: 50000 unlinks in 647 seconds: 77.279755 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg213-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg213-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg213-server stop mds service on oleg213-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg213-server unloading modules via unload_modules_local on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing unload_modules_local oleg213-server: kmemleak disabled modules unloaded. pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2