== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:55:37 (1788778537) start mds service on oleg125-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' -rwxr-xr-x 1 green green 1754936 Sep 6 20:06 /home/green/git/lustre-release/lustre/utils/mount.lustre -rwxr-xr-x. 1 root root 12 Jan 31 2019 /usr/sbin/mount.lustre mount --bind /home/green/git/lustre-release/lustre/utils/mount.lustre /usr/sbin/mount.lustre loading modules on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing load_modules_local oleg125-server: kmemleak disabled oleg125-server: Loading modules from /home/green/git/lustre-release/lustre oleg125-server: detected 4 online CPUs by sysfs oleg125-server: MODOPTS_LIBCFS= oleg125-server: Force libcfs to create 2 CPU partitions oleg125-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg125-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg125-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg125-server: /usr/sbin/ not writable for /usr/sbin/mount.lustre_tgt Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg125-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg125-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg125-server: oleg125-server.virtnet: executing set_default_debug -1 all pdsh@oleg125-client: oleg125-server: ssh exited with exit code 1 Started lustre-OST0000 oleg125-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg125-client.virtnet: -o user_xattr,flock 192.168.201.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 142 (time 1788778638.19 total 10.01 last 14.18) - open/close 249 (time 1788778648.96 total 20.78 last 9.94) - open/close 378 (time 1788778660.33 total 32.15 last 11.35) - open/close 571 (time 1788778670.84 total 42.66 last 18.36) - open/close 778 (time 1788778681.58 total 53.41 last 19.26) - open/close 952 (time 1788778692.08 total 63.90 last 16.58) - open/close 1120 (time 1788778705.10 total 76.92 last 12.90) - open/close 1309 (time 1788778717.62 total 89.44 last 15.09) - open/close 1475 (time 1788778728.18 total 100.01 last 15.71) - open/close 1642 (time 1788778738.23 total 110.06 last 16.62) - open/close 1851 (time 1788778748.61 total 120.44 last 20.14) - open/close 1944 (time 1788778758.96 total 130.78 last 8.99) - open/close 2094 (time 1788778769.35 total 141.17 last 14.43) - open/close 2289 (time 1788778779.35 total 151.18 last 19.49) - open/close 2438 (time 1788778790.17 total 162.00 last 13.77) - open/close 2626 (time 1788778800.17 total 172.00 last 18.80) - open/close 2944 (time 1788778810.20 total 182.02 last 31.72) - open/close 3175 (time 1788778820.21 total 192.04 last 23.07) - open/close 3347 (time 1788778830.39 total 202.21 last 16.90) - open/close 3497 (time 1788778841.41 total 213.23 last 13.61) - open/close 3670 (time 1788778852.27 total 224.10 last 15.92) - open/close 3885 (time 1788778862.32 total 234.14 last 21.40) - open/close 4025 (time 1788778872.80 total 244.63 last 13.36) - open/close 4190 (time 1788778884.83 total 256.66 last 13.71) - open/close 4381 (time 1788778894.88 total 266.70 last 19.02) - open/close 4539 (time 1788778904.90 total 276.72 last 15.77) - open/close 4719 (time 1788778914.92 total 286.74 last 17.97) - open/close 4829 (time 1788778924.92 total 296.74 last 11.00) - open/close 4989 (time 1788778935.28 total 307.10 last 15.44) - open/close 5165 (time 1788778945.72 total 317.54 last 16.86) - open/close 5293 (time 1788778955.73 total 327.56 last 12.78) - open/close 5468 (time 1788778965.86 total 337.68 last 17.29) - open/close 5641 (time 1788778976.13 total 347.95 last 16.84) - open/close 5898 (time 1788778986.14 total 357.96 last 25.68) - open/close 6103 (time 1788778996.98 total 368.80 last 18.91) - open/close 6176 (time 1788779006.98 total 378.80 last 7.30) - open/close 6338 (time 1788779017.00 total 388.82 last 16.17) - open/close 6480 (time 1788779027.10 total 398.92 last 14.06) - open/close 6696 (time 1788779037.29 total 409.12 last 21.19) - open/close 6918 (time 1788779049.26 total 421.09 last 18.54) - open/close 7062 (time 1788779062.66 total 434.48 last 10.75) - open/close 7207 (time 1788779073.60 total 445.43 last 13.25) - open/close 7392 (time 1788779083.61 total 455.43 last 18.50) - open/close 7566 (time 1788779094.54 total 466.37 last 15.91) - open/close 7665 (time 1788779105.97 total 477.79 last 8.67) - open/close 7889 (time 1788779115.99 total 487.81 last 22.35) - open/close 7980 (time 1788779126.51 total 498.33 last 8.65) - open/close 8011 (time 1788779136.76 total 508.58 last 3.02) - open/close 8155 (time 1788779147.13 total 518.95 last 13.89) - open/close 8297 (time 1788779157.88 total 529.71 last 13.20) - open/close 8529 (time 1788779167.90 total 539.72 last 23.17) - open/close 8577 (time 1788779178.30 total 550.12 last 4.62) - open/close 8772 (time 1788779188.38 total 560.21 last 19.33) - open/close 8916 (time 1788779198.39 total 570.21 last 14.40) - open/close 9121 (time 1788779209.90 total 581.72 last 17.81) - open/close 9318 (time 1788779219.92 total 591.74 last 19.66) - open/close 9518 (time 1788779229.92 total 601.75 last 19.99) - open/close 9786 (time 1788779239.93 total 611.75 last 26.79) - open/close 9960 (time 1788779249.95 total 621.77 last 17.36) - open/close 10184 (time 1788779260.65 total 632.48 last 20.93) - open/close 10261 (time 1788779270.78 total 642.61 last 7.60) - open/close 10516 (time 1788779283.34 total 655.16 last 20.31) - open/close 10710 (time 1788779293.35 total 665.17 last 19.38) - open/close 10821 (time 1788779306.29 total 678.11 last 8.58) - open/close 10966 (time 1788779316.30 total 688.12 last 14.48) - open/close 11036 (time 1788779326.33 total 698.16 last 6.97) - open/close 11127 (time 1788779336.50 total 708.33 last 8.95) - open/close 11413 (time 1788779346.76 total 718.59 last 27.88) - open/close 11669 (time 1788779356.77 total 728.59 last 25.59) - open/close 11912 (time 1788779367.14 total 738.97 last 23.42) - open/close 12144 (time 1788779377.14 total 748.97 last 23.20) - open/close 12343 (time 1788779387.64 total 759.46 last 18.97) - open/close 12586 (time 1788779398.06 total 769.89 last 23.30) - open/close 12769 (time 1788779408.49 total 780.32 last 17.54) - open/close 13019 (time 1788779418.51 total 790.34 last 24.95) - open/close 13206 (time 1788779428.52 total 800.34 last 18.69) - open/close 13393 (time 1788779438.97 total 810.79 last 17.90) - open/close 13602 (time 1788779449.34 total 821.17 last 20.14) - open/close 13841 (time 1788779460.08 total 831.90 last 22.27) - open/close 13983 (time 1788779470.09 total 841.91 last 14.18) - open/close 14101 (time 1788779480.88 total 852.70 last 10.93) - open/close 14184 (time 1788779493.24 total 865.06 last 6.72) - open/close 14241 (time 1788779504.24 total 876.06 last 5.18) - open/close 14340 (time 1788779514.26 total 886.08 last 9.88) - open/close 14550 (time 1788779524.92 total 896.74 last 19.70) - open/close 14752 (time 1788779535.22 total 907.04 last 19.61) - open/close 14977 (time 1788779546.16 total 917.98 last 20.57) - open/close 15163 (time 1788779557.04 total 928.86 last 17.09) - open/close 15348 (time 1788779567.46 total 939.28 last 17.76) - open/close 15525 (time 1788779578.28 total 950.10 last 16.36) - open/close 15804 (time 1788779589.30 total 961.13 last 25.31) - open/close 15954 (time 1788779599.38 total 971.21 last 14.88) - open/close 16117 (time 1788779612.21 total 984.03 last 12.71) - open/close 16335 (time 1788779622.34 total 994.16 last 21.53) - open/close 16587 (time 1788779632.35 total 1004.17 last 25.17) - open/close 16826 (time 1788779642.35 total 1014.18 last 23.90) - open/close 17070 (time 1788779652.38 total 1024.21 last 24.32) - open/close 17284 (time 1788779663.23 total 1035.05 last 19.74) - open/close 17495 (time 1788779673.23 total 1045.06 last 21.09) - open/close 18312 (time 1788779683.24 total 1055.06 last 81.62) - open/close 19233 (time 1788779693.25 total 1065.07 last 92.06) - open/close 20000 (time 1788779701.63 total 1073.45 last 91.51) - open/close 20889 (time 1788779711.63 total 1083.45 last 88.88) - open/close 21585 (time 1788779721.65 total 1093.48 last 69.43) - open/close 22267 (time 1788779731.66 total 1103.49 last 68.12) - open/close 23023 (time 1788779741.68 total 1113.50 last 75.50) - open/close 23851 (time 1788779751.68 total 1123.51 last 82.74) - open/close 24635 (time 1788779761.70 total 1133.52 last 78.28) - open/close 25347 (time 1788779771.70 total 1143.53 last 71.18) - open/close 26079 (time 1788779781.72 total 1153.54 last 73.09) - open/close 26798 (time 1788779791.72 total 1163.54 last 71.88) - open/close 27587 (time 1788779801.73 total 1173.55 last 78.85) - open/close 28332 (time 1788779811.73 total 1183.56 last 74.44) - open/close 29123 (time 1788779821.74 total 1193.56 last 79.07) - open/close 29813 (time 1788779831.74 total 1203.56 last 68.98) - open/close 30000 (time 1788779834.27 total 1206.09 last 74.01) - open/close 30649 (time 1788779844.28 total 1216.10 last 64.85) - open/close 31377 (time 1788779854.30 total 1226.12 last 72.61) - open/close 32032 (time 1788779864.30 total 1236.13 last 65.49) - open/close 32720 (time 1788779874.32 total 1246.15 last 68.65) - open/close 33234 (time 1788779884.35 total 1256.17 last 51.28) - open/close 34007 (time 1788779894.38 total 1266.21 last 77.02) - open/close 34599 (time 1788779904.39 total 1276.21 last 59.18) - open/close 35372 (time 1788779914.39 total 1286.21 last 77.27) - open/close 36247 (time 1788779924.41 total 1296.23 last 87.34) - open/close 36871 (time 1788779934.42 total 1306.24 last 62.33) - open/close 37633 (time 1788779944.43 total 1316.26 last 76.11) - open/close 38509 (time 1788779954.44 total 1326.27 last 87.51) - open/close 39210 (time 1788779964.45 total 1336.27 last 70.04) - open/close 39888 (time 1788779974.47 total 1346.30 last 67.65) - open/close 40660 (time 1788779984.48 total 1356.31 last 77.14) - open/close 41492 (time 1788779994.49 total 1366.31 last 83.17) - open/close 42221 (time 1788780004.49 total 1376.32 last 72.85) - open/close 42872 (time 1788780014.50 total 1386.32 last 65.06) - open/close 43593 (time 1788780024.51 total 1396.33 last 72.02) - open/close 44350 (time 1788780034.51 total 1406.34 last 75.68) - open/close 45180 (time 1788780044.52 total 1416.34 last 82.97) - open/close 45954 (time 1788780054.53 total 1426.35 last 77.33) - open/close 46800 (time 1788780064.53 total 1436.35 last 84.57) - open/close 47562 (time 1788780074.55 total 1446.37 last 76.04) - open/close 48354 (time 1788780084.56 total 1456.38 last 79.15) - open/close 49039 (time 1788780094.59 total 1466.41 last 68.27) - open/close 49860 (time 1788780104.59 total 1476.42 last 82.07) total: 50000 open/close in 1478.05 seconds: 33.83 ops/second - unlinked 0 (time 1788780111 ; total 0 ; last 0) - unlinked 10000 (time 1788780249 ; total 138 ; last 138) - unlinked 20000 (time 1788780385 ; total 274 ; last 136) - unlinked 30000 (time 1788780520 ; total 409 ; last 135) - unlinked 40000 (time 1788780656 ; total 545 ; last 136) total: 50000 unlinks in 679 seconds: 73.637703 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg125-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg125-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg125-server stop mds service on oleg125-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg125-server unloading modules via unload_modules_local on: 'oleg125-server' oleg125-server: oleg125-server.virtnet: executing unload_modules_local oleg125-server: kmemleak disabled modules unloaded. pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-client: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-server: ssh exited with exit code 2 pdsh@oleg125-client: oleg125-client: ssh exited with exit code 2