== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:24:22 (1787736262) start mds service on oleg124-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing load_modules_local oleg124-server: Loading modules from /home/green/git/lustre-release/lustre oleg124-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg124-server: detected 4 online CPUs by sysfs oleg124-server: MODOPTS_LIBCFS= oleg124-server: Force libcfs to create 2 CPU partitions oleg124-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg124-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg124-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg124-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg124-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg124-server: oleg124-server.virtnet: executing set_default_debug -1 all pdsh@oleg124-client: oleg124-server: ssh exited with exit code 1 Started lustre-OST0000 oleg124-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg124-client.virtnet: -o user_xattr,flock 192.168.201.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 162 (time 1787736341.84 total 10.50 last 15.42) - open/close 410 (time 1787736352.23 total 20.90 last 23.85) - open/close 661 (time 1787736363.12 total 31.78 last 23.07) - open/close 887 (time 1787736373.12 total 41.79 last 22.59) - open/close 1084 (time 1787736383.86 total 52.53 last 18.35) - open/close 1325 (time 1787736394.16 total 62.83 last 23.39) - open/close 1575 (time 1787736404.54 total 73.21 last 24.10) - open/close 1808 (time 1787736415.56 total 84.22 last 21.15) - open/close 1997 (time 1787736425.56 total 94.22 last 18.90) - open/close 2206 (time 1787736435.72 total 104.38 last 20.57) - open/close 2463 (time 1787736446.19 total 114.86 last 24.54) - open/close 2628 (time 1787736456.89 total 125.56 last 15.42) - open/close 2849 (time 1787736467.64 total 136.31 last 20.56) - open/close 3071 (time 1787736478.64 total 147.31 last 20.19) - open/close 3300 (time 1787736488.90 total 157.57 last 22.32) - open/close 3522 (time 1787736499.26 total 167.93 last 21.43) - open/close 3747 (time 1787736509.27 total 177.93 last 22.49) - open/close 3978 (time 1787736520.17 total 188.84 last 21.18) - open/close 4232 (time 1787736530.17 total 198.84 last 25.40) - open/close 4411 (time 1787736540.33 total 209.00 last 17.62) - open/close 4609 (time 1787736550.47 total 219.14 last 19.52) - open/close 4839 (time 1787736560.48 total 229.14 last 22.99) - open/close 5040 (time 1787736570.80 total 239.47 last 19.47) - open/close 5269 (time 1787736581.04 total 249.71 last 22.37) - open/close 5497 (time 1787736591.04 total 259.71 last 22.80) - open/close 5723 (time 1787736601.07 total 269.74 last 22.53) - open/close 5980 (time 1787736611.84 total 280.51 last 23.85) - open/close 6240 (time 1787736622.77 total 291.44 last 23.79) - open/close 6457 (time 1787736632.77 total 301.44 last 21.69) - open/close 6678 (time 1787736643.69 total 312.35 last 20.25) - open/close 6823 (time 1787736653.76 total 322.43 last 14.39) - open/close 7055 (time 1787736664.74 total 333.41 last 21.13) - open/close 7201 (time 1787736675.26 total 343.93 last 13.88) - open/close 7418 (time 1787736685.51 total 354.18 last 21.19) - open/close 7688 (time 1787736695.92 total 364.59 last 25.92) - open/close 7933 (time 1787736706.86 total 375.53 last 22.40) - open/close 8188 (time 1787736717.43 total 386.10 last 24.12) - open/close 8399 (time 1787736727.43 total 396.10 last 21.10) - open/close 8641 (time 1787736737.96 total 406.63 last 22.98) - open/close 8877 (time 1787736748.28 total 416.95 last 22.87) - open/close 9090 (time 1787736759.10 total 427.76 last 19.70) - open/close 9310 (time 1787736769.14 total 437.81 last 21.90) - open/close 9540 (time 1787736779.32 total 447.99 last 22.59) - open/close 9787 (time 1787736790.25 total 458.92 last 22.60) - open/close 10000 (time 1787736799.34 total 468.01 last 23.44) - open/close 10227 (time 1787736809.51 total 478.18 last 22.32) - open/close 10461 (time 1787736819.52 total 488.19 last 23.38) - open/close 10703 (time 1787736830.45 total 499.12 last 22.13) - open/close 10839 (time 1787736840.89 total 509.56 last 13.03) - open/close 11025 (time 1787736850.90 total 519.57 last 18.58) - open/close 11251 (time 1787736861.16 total 529.83 last 22.03) - open/close 11410 (time 1787736871.86 total 540.53 last 14.85) - open/close 11533 (time 1787736881.94 total 550.61 last 12.21) - open/close 11577 (time 1787736892.62 total 561.29 last 4.12) - open/close 11753 (time 1787736903.16 total 571.83 last 16.70) - open/close 11930 (time 1787736914.04 total 582.71 last 16.26) - open/close 12139 (time 1787736924.04 total 592.71 last 20.90) - open/close 12245 (time 1787736934.77 total 603.44 last 9.88) - open/close 12409 (time 1787736945.48 total 614.15 last 15.31) - open/close 12502 (time 1787736955.51 total 624.18 last 9.27) - open/close 12621 (time 1787736966.53 total 635.20 last 10.80) - open/close 12827 (time 1787736976.89 total 645.56 last 19.88) - open/close 12987 (time 1787736986.90 total 655.56 last 15.99) - open/close 13049 (time 1787736997.00 total 665.67 last 6.14) - open/close 13158 (time 1787737007.43 total 676.09 last 10.45) - open/close 13346 (time 1787737017.70 total 686.36 last 18.31) - open/close 13600 (time 1787737027.71 total 696.37 last 25.38) - open/close 13802 (time 1787737038.11 total 706.78 last 19.41) - open/close 13954 (time 1787737048.35 total 717.02 last 14.84) - open/close 14093 (time 1787737058.37 total 727.04 last 13.88) - open/close 14264 (time 1787737068.38 total 737.05 last 17.08) - open/close 14302 (time 1787737078.71 total 747.38 last 3.68) - open/close 14546 (time 1787737088.74 total 757.41 last 24.34) - open/close 14652 (time 1787737098.77 total 767.44 last 10.57) - open/close 14865 (time 1787737109.37 total 778.04 last 20.09) - open/close 15095 (time 1787737119.39 total 788.06 last 22.96) - open/close 15297 (time 1787737129.62 total 798.29 last 19.75) - open/close 15551 (time 1787737139.62 total 808.29 last 25.39) - open/close 15770 (time 1787737149.93 total 818.60 last 21.25) - open/close 15986 (time 1787737159.93 total 828.60 last 21.60) - open/close 16212 (time 1787737170.60 total 839.27 last 21.18) - open/close 16419 (time 1787737180.60 total 849.27 last 20.69) - open/close 16599 (time 1787737191.55 total 860.22 last 16.45) - open/close 16837 (time 1787737202.06 total 870.72 last 22.65) - open/close 17079 (time 1787737212.08 total 880.74 last 24.15) - open/close 17394 (time 1787737224.87 total 893.54 last 24.62) - open/close 17569 (time 1787737234.87 total 903.54 last 17.50) - open/close 17780 (time 1787737245.23 total 913.89 last 20.38) - open/close 17993 (time 1787737256.16 total 924.83 last 19.48) - open/close 18460 (time 1787737266.17 total 934.84 last 46.66) - open/close 19090 (time 1787737276.18 total 944.85 last 62.95) - open/close 19864 (time 1787737286.18 total 954.85 last 77.36) - open/close 20581 (time 1787737296.19 total 964.86 last 71.65) - open/close 21287 (time 1787737306.19 total 974.86 last 70.59) - open/close 21942 (time 1787737316.20 total 984.87 last 65.42) - open/close 22622 (time 1787737326.22 total 994.89 last 67.90) - open/close 23255 (time 1787737336.23 total 1004.90 last 63.24) - open/close 23923 (time 1787737346.24 total 1014.91 last 66.73) - open/close 24634 (time 1787737356.24 total 1024.91 last 71.08) - open/close 25328 (time 1787737366.28 total 1034.95 last 69.14) - open/close 26060 (time 1787737376.29 total 1044.96 last 73.12) - open/close 26707 (time 1787737386.30 total 1054.96 last 64.67) - open/close 27502 (time 1787737396.30 total 1064.97 last 79.49) - open/close 28240 (time 1787737406.30 total 1074.97 last 73.75) - open/close 28954 (time 1787737416.31 total 1084.97 last 71.39) - open/close 29697 (time 1787737426.31 total 1094.98 last 74.29) - open/close 30000 (time 1787737430.28 total 1098.94 last 76.36) - open/close 30766 (time 1787737440.28 total 1108.95 last 76.57) - open/close 31311 (time 1787737450.30 total 1118.97 last 54.39) - open/close 31979 (time 1787737460.30 total 1128.97 last 66.77) - open/close 32717 (time 1787737470.33 total 1139.00 last 73.61) - open/close 33377 (time 1787737480.33 total 1149.00 last 65.98) - open/close 34096 (time 1787737490.35 total 1159.02 last 71.77) - open/close 34728 (time 1787737500.35 total 1169.02 last 63.18) - open/close 35441 (time 1787737510.36 total 1179.03 last 71.24) - open/close 36165 (time 1787737520.36 total 1189.03 last 72.39) - open/close 36802 (time 1787737530.37 total 1199.04 last 63.65) - open/close 37518 (time 1787737540.37 total 1209.04 last 71.57) - open/close 38305 (time 1787737550.39 total 1219.06 last 78.60) - open/close 38931 (time 1787737560.40 total 1229.07 last 62.54) - open/close 39590 (time 1787737570.41 total 1239.08 last 65.80) - open/close 40000 (time 1787737576.62 total 1245.29 last 66.01) - open/close 40701 (time 1787737586.63 total 1255.30 last 70.06) - open/close 41453 (time 1787737596.65 total 1265.32 last 75.02) - open/close 42190 (time 1787737606.66 total 1275.33 last 73.62) - open/close 43048 (time 1787737616.67 total 1285.34 last 85.70) - open/close 43740 (time 1787737626.69 total 1295.36 last 69.11) - open/close 44339 (time 1787737636.72 total 1305.39 last 59.70) - open/close 44975 (time 1787737646.73 total 1315.40 last 63.53) - open/close 45648 (time 1787737656.74 total 1325.41 last 67.25) - open/close 46265 (time 1787737666.74 total 1335.41 last 61.69) - open/close 46848 (time 1787737676.75 total 1345.42 last 58.26) - open/close 47509 (time 1787737686.75 total 1355.42 last 66.08) - open/close 48078 (time 1787737696.75 total 1365.42 last 56.89) - open/close 48706 (time 1787737706.79 total 1375.46 last 62.54) - open/close 49344 (time 1787737716.80 total 1385.47 last 63.74) total: 50000 open/close in 1394.68 seconds: 35.85 ops/second - unlinked 0 (time 1787737731 ; total 0 ; last 0) - unlinked 10000 (time 1787737889 ; total 158 ; last 158) - unlinked 20000 (time 1787738044 ; total 313 ; last 155) - unlinked 30000 (time 1787738193 ; total 462 ; last 149) - unlinked 40000 (time 1787738334 ; total 603 ; last 141) total: 50000 unlinks in 755 seconds: 66.225166 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg124-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg124-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg124-server stop mds service on oleg124-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg124-server unloading modules via unload_modules_local on: 'oleg124-server' oleg124-server: oleg124-server.virtnet: executing unload_modules_local oleg124-server: modules unloaded. pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-client: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-client: ssh exited with exit code 2 pdsh@oleg124-client: oleg124-server: ssh exited with exit code 2