== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:32:19 (1783967539) start mds service on oleg635-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg635-server' oleg635-server: oleg635-server.virtnet: executing load_modules_local oleg635-server: Loading modules from /home/green/git/lustre-release/lustre oleg635-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg635-server: detected 4 online CPUs by sysfs oleg635-server: MODOPTS_LIBCFS= oleg635-server: Force libcfs to create 2 CPU partitions oleg635-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg635-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg635-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg635-server: oleg635-server.virtnet: executing set_default_debug -1 all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg635-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg635-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg635-server: oleg635-server.virtnet: executing set_default_debug -1 all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-OST0000 oleg635-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg635-client.virtnet: -o user_xattr,flock 192.168.206.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 229 (time 1783967617.25 total 10.62 last 21.56) - open/close 465 (time 1783967627.27 total 20.64 last 23.56) - open/close 691 (time 1783967637.28 total 30.65 last 22.57) - open/close 919 (time 1783967647.28 total 40.65 last 22.79) - open/close 1173 (time 1783967657.58 total 50.95 last 24.66) - open/close 1396 (time 1783967667.70 total 61.07 last 22.04) - open/close 1621 (time 1783967678.12 total 71.49 last 21.60) - open/close 1856 (time 1783967688.12 total 81.50 last 23.48) - open/close 2065 (time 1783967698.72 total 92.09 last 19.72) - open/close 2289 (time 1783967709.54 total 102.91 last 20.71) - open/close 2529 (time 1783967719.55 total 112.92 last 23.98) - open/close 2741 (time 1783967729.98 total 123.35 last 20.32) - open/close 3011 (time 1783967740.57 total 133.95 last 25.48) - open/close 3249 (time 1783967751.59 total 144.96 last 21.61) - open/close 3494 (time 1783967762.60 total 155.97 last 22.26) - open/close 3745 (time 1783967772.61 total 165.98 last 25.07) - open/close 3979 (time 1783967782.62 total 175.99 last 23.39) - open/close 4108 (time 1783967792.62 total 186.00 last 12.89) - open/close 4330 (time 1783967802.99 total 196.36 last 21.41) - open/close 4564 (time 1783967813.00 total 206.38 last 23.37) - open/close 4725 (time 1783967824.12 total 217.49 last 14.49) - open/close 4946 (time 1783967834.60 total 227.97 last 21.08) - open/close 5151 (time 1783967845.36 total 238.73 last 19.05) - open/close 5390 (time 1783967856.03 total 249.40 last 22.40) - open/close 5588 (time 1783967866.91 total 260.28 last 18.19) - open/close 5784 (time 1783967876.95 total 270.32 last 19.53) - open/close 5995 (time 1783967887.30 total 280.67 last 20.39) - open/close 6301 (time 1783967897.89 total 291.26 last 28.88) - open/close 6506 (time 1783967908.02 total 301.39 last 20.24) - open/close 6741 (time 1783967918.37 total 311.74 last 22.70) - open/close 7009 (time 1783967928.80 total 322.18 last 25.68) - open/close 7226 (time 1783967939.45 total 332.83 last 20.38) - open/close 7443 (time 1783967950.44 total 343.81 last 19.75) - open/close 7688 (time 1783967961.00 total 354.37 last 23.20) - open/close 7900 (time 1783967971.56 total 364.93 last 20.08) - open/close 8116 (time 1783967981.56 total 374.93 last 21.59) - open/close 8347 (time 1783967993.89 total 387.26 last 18.74) - open/close 8595 (time 1783968004.64 total 398.01 last 23.06) - open/close 8807 (time 1783968015.26 total 408.63 last 19.96) - open/close 9050 (time 1783968025.72 total 419.09 last 23.24) - open/close 9267 (time 1783968035.73 total 429.11 last 21.66) - open/close 9367 (time 1783968046.52 total 439.89 last 9.27) - open/close 9548 (time 1783968056.53 total 449.90 last 18.08) - open/close 9691 (time 1783968066.81 total 460.19 last 13.90) - open/close 9845 (time 1783968076.82 total 470.20 last 15.38) - open/close 10000 (time 1783968081.91 total 475.28 last 30.50) - open/close 10230 (time 1783968092.26 total 485.64 last 22.21) - open/close 10452 (time 1783968103.99 total 497.36 last 18.93) - open/close 10665 (time 1783968113.99 total 507.37 last 21.29) - open/close 10857 (time 1783968124.85 total 518.23 last 17.68) - open/close 11097 (time 1783968135.27 total 528.64 last 23.05) - open/close 11314 (time 1783968145.29 total 538.66 last 21.66) - open/close 11487 (time 1783968155.88 total 549.26 last 16.33) - open/close 11693 (time 1783968166.24 total 559.61 last 19.89) - open/close 11898 (time 1783968176.87 total 570.24 last 19.28) - open/close 12069 (time 1783968186.87 total 580.25 last 17.10) - open/close 12315 (time 1783968197.03 total 590.40 last 24.23) - open/close 12507 (time 1783968207.26 total 600.64 last 18.75) - open/close 12737 (time 1783968217.33 total 610.70 last 22.85) - open/close 12984 (time 1783968227.34 total 620.71 last 24.68) - open/close 13198 (time 1783968237.34 total 630.72 last 21.38) - open/close 13398 (time 1783968247.36 total 640.74 last 19.96) - open/close 13645 (time 1783968257.37 total 650.75 last 24.68) - open/close 13841 (time 1783968267.39 total 660.76 last 19.57) - open/close 14119 (time 1783968277.40 total 670.77 last 27.78) - open/close 14330 (time 1783968287.72 total 681.10 last 20.43) - open/close 14549 (time 1783968298.61 total 691.98 last 20.13) - open/close 14814 (time 1783968309.54 total 702.92 last 24.23) - open/close 15042 (time 1783968319.91 total 713.28 last 21.99) - open/close 15263 (time 1783968329.91 total 723.29 last 22.09) - open/close 15480 (time 1783968340.52 total 733.90 last 20.45) - open/close 15724 (time 1783968351.59 total 744.96 last 22.05) - open/close 15968 (time 1783968361.73 total 755.10 last 24.07) - open/close 16182 (time 1783968372.78 total 766.15 last 19.36) - open/close 16395 (time 1783968383.13 total 776.50 last 20.58) - open/close 16622 (time 1783968393.77 total 787.14 last 21.34) - open/close 16840 (time 1783968403.77 total 797.14 last 21.80) - open/close 17076 (time 1783968413.79 total 807.16 last 23.55) - open/close 17288 (time 1783968423.80 total 817.18 last 21.17) - open/close 17526 (time 1783968434.72 total 828.09 last 21.81) - open/close 17817 (time 1783968445.60 total 838.97 last 26.74) - open/close 18049 (time 1783968456.12 total 849.49 last 22.05) - open/close 18214 (time 1783968467.07 total 860.44 last 15.07) - open/close 18369 (time 1783968477.89 total 871.26 last 14.33) - open/close 18961 (time 1783968487.90 total 881.27 last 59.15) - open/close 19651 (time 1783968497.90 total 891.28 last 68.95) - open/close 20000 (time 1783968503.25 total 896.63 last 65.25) - open/close 20608 (time 1783968513.27 total 906.64 last 60.69) - open/close 21279 (time 1783968523.28 total 916.65 last 67.04) - open/close 21945 (time 1783968533.28 total 926.66 last 66.58) - open/close 22592 (time 1783968543.29 total 936.66 last 64.66) - open/close 23274 (time 1783968553.30 total 946.67 last 68.11) - open/close 24037 (time 1783968563.30 total 956.67 last 76.30) - open/close 24743 (time 1783968573.31 total 966.68 last 70.53) - open/close 25428 (time 1783968583.31 total 976.69 last 68.48) - open/close 26150 (time 1783968593.32 total 986.70 last 72.13) - open/close 26763 (time 1783968603.33 total 996.70 last 61.29) - open/close 27431 (time 1783968613.34 total 1006.71 last 66.71) - open/close 28080 (time 1783968623.36 total 1016.73 last 64.80) - open/close 28723 (time 1783968633.36 total 1026.73 last 64.28) - open/close 29378 (time 1783968643.37 total 1036.74 last 65.45) - open/close 30000 (time 1783968652.28 total 1045.65 last 69.77) - open/close 30732 (time 1783968662.30 total 1055.67 last 73.06) - open/close 31435 (time 1783968672.30 total 1065.68 last 70.27) - open/close 32098 (time 1783968682.31 total 1075.69 last 66.25) - open/close 32741 (time 1783968692.32 total 1085.69 last 64.24) - open/close 33416 (time 1783968702.32 total 1095.70 last 67.49) - open/close 34104 (time 1783968712.34 total 1105.71 last 68.71) - open/close 34800 (time 1783968722.34 total 1115.72 last 69.55) - open/close 35530 (time 1783968732.36 total 1125.74 last 72.84) - open/close 36304 (time 1783968742.37 total 1135.74 last 77.35) - open/close 37010 (time 1783968752.39 total 1145.76 last 70.50) - open/close 37700 (time 1783968762.39 total 1155.76 last 68.98) - open/close 38411 (time 1783968772.39 total 1165.77 last 71.07) - open/close 39123 (time 1783968782.40 total 1175.77 last 71.17) - open/close 39797 (time 1783968792.41 total 1185.78 last 67.34) - open/close 40000 (time 1783968795.47 total 1188.84 last 66.22) - open/close 40685 (time 1783968805.48 total 1198.85 last 68.45) - open/close 41364 (time 1783968815.48 total 1208.85 last 67.88) - open/close 42049 (time 1783968825.49 total 1218.86 last 68.44) - open/close 42713 (time 1783968835.49 total 1228.87 last 66.37) - open/close 43375 (time 1783968845.50 total 1238.88 last 66.13) - open/close 44021 (time 1783968855.52 total 1248.90 last 64.46) - open/close 44651 (time 1783968865.53 total 1258.90 last 62.99) - open/close 45267 (time 1783968875.53 total 1268.90 last 61.60) - open/close 45889 (time 1783968885.53 total 1278.91 last 62.17) - open/close 46544 (time 1783968895.54 total 1288.91 last 65.45) - open/close 47232 (time 1783968905.54 total 1298.92 last 68.79) - open/close 47909 (time 1783968915.55 total 1308.93 last 67.63) - open/close 48633 (time 1783968925.56 total 1318.94 last 72.33) - open/close 49371 (time 1783968935.57 total 1328.94 last 73.73) total: 50000 open/close in 1337.54 seconds: 37.38 ops/second - unlinked 0 (time 1783968947 ; total 0 ; last 0) - unlinked 10000 (time 1783969110 ; total 163 ; last 163) - unlinked 20000 (time 1783969271 ; total 324 ; last 161) - unlinked 30000 (time 1783969443 ; total 496 ; last 172) - unlinked 40000 (time 1783969614 ; total 667 ; last 171) total: 50000 unlinks in 842 seconds: 59.382423 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg635-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg635-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg635-server stop mds service on oleg635-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg635-server unloading modules via unload_modules_local on: 'oleg635-server' oleg635-server: oleg635-server.virtnet: executing unload_modules_local oleg635-server: modules unloaded. pdsh@oleg635-client: oleg635-client: ssh exited with exit code 2 pdsh@oleg635-client: oleg635-server: ssh exited with exit code 2 pdsh@oleg635-client: oleg635-client: ssh exited with exit code 2 pdsh@oleg635-client: oleg635-server: ssh exited with exit code 2