== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:03:35 (1788156215) start mds service on oleg336-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing load_modules_local oleg336-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg336-server: Loading modules from /home/green/git/lustre-release/lustre oleg336-server: detected 4 online CPUs by sysfs oleg336-server: MODOPTS_LIBCFS= oleg336-server: Force libcfs to create 2 CPU partitions oleg336-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg336-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg336-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg336-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg336-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg336-server: oleg336-server.virtnet: executing set_default_debug -1 all pdsh@oleg336-client: oleg336-server: ssh exited with exit code 1 Started lustre-OST0000 oleg336-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg336-client.virtnet: -o user_xattr,flock 192.168.203.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 245 (time 1788156290.10 total 11.06 last 22.15) - open/close 494 (time 1788156300.10 total 21.07 last 24.88) - open/close 668 (time 1788156310.11 total 31.08 last 17.39) - open/close 839 (time 1788156320.12 total 41.09 last 17.08) - open/close 1027 (time 1788156330.13 total 51.10 last 18.79) - open/close 1258 (time 1788156340.14 total 61.10 last 23.09) - open/close 1474 (time 1788156351.02 total 71.98 last 19.85) - open/close 1727 (time 1788156361.04 total 82.00 last 25.25) - open/close 1922 (time 1788156371.31 total 92.27 last 18.99) - open/close 2115 (time 1788156381.80 total 102.76 last 18.39) - open/close 2364 (time 1788156391.81 total 112.78 last 24.86) - open/close 2608 (time 1788156402.56 total 123.53 last 22.70) - open/close 2851 (time 1788156413.02 total 133.99 last 23.23) - open/close 3105 (time 1788156423.03 total 144.00 last 25.36) - open/close 3334 (time 1788156433.47 total 154.43 last 21.95) - open/close 3572 (time 1788156443.48 total 164.44 last 23.78) - open/close 3681 (time 1788156454.50 total 175.47 last 9.89) - open/close 3807 (time 1788156464.51 total 185.48 last 12.59) - open/close 3962 (time 1788156474.52 total 195.48 last 15.49) - open/close 4170 (time 1788156484.52 total 205.49 last 20.78) - open/close 4299 (time 1788156494.54 total 215.50 last 12.89) - open/close 4546 (time 1788156505.44 total 226.41 last 22.64) - open/close 4788 (time 1788156515.98 total 236.94 last 22.98) - open/close 5019 (time 1788156526.32 total 247.28 last 22.34) - open/close 5239 (time 1788156537.00 total 257.96 last 20.60) - open/close 5464 (time 1788156547.25 total 268.21 last 21.96) - open/close 5701 (time 1788156557.66 total 278.63 last 22.75) - open/close 5917 (time 1788156567.68 total 288.64 last 21.57) - open/close 6113 (time 1788156577.68 total 298.65 last 19.58) - open/close 6282 (time 1788156587.70 total 308.66 last 16.88) - open/close 6470 (time 1788156597.78 total 318.74 last 18.65) - open/close 6678 (time 1788156609.34 total 330.31 last 17.99) - open/close 6844 (time 1788156619.34 total 340.31 last 16.60) - open/close 7064 (time 1788156629.35 total 350.32 last 21.98) - open/close 7258 (time 1788156639.58 total 360.55 last 18.96) - open/close 7558 (time 1788156650.14 total 371.11 last 28.42) - open/close 7824 (time 1788156660.15 total 381.11 last 26.58) - open/close 8053 (time 1788156670.65 total 391.62 last 21.81) - open/close 8225 (time 1788156681.00 total 401.96 last 16.63) - open/close 8462 (time 1788156691.87 total 412.83 last 21.80) - open/close 8685 (time 1788156702.24 total 423.20 last 21.51) - open/close 8914 (time 1788156712.82 total 433.78 last 21.64) - open/close 9105 (time 1788156723.06 total 444.02 last 18.66) - open/close 9323 (time 1788156733.06 total 454.02 last 21.79) - open/close 9545 (time 1788156743.72 total 464.68 last 20.83) - open/close 9769 (time 1788156754.38 total 475.34 last 21.01) - open/close 9999 (time 1788156765.11 total 486.07 last 21.43) - open/close 10235 (time 1788156776.16 total 497.12 last 21.36) - open/close 10411 (time 1788156788.10 total 509.07 last 14.73) - open/close 10465 (time 1788156799.09 total 520.06 last 4.91) - open/close 10683 (time 1788156809.96 total 530.93 last 20.06) - open/close 10879 (time 1788156822.45 total 543.42 last 15.70) - open/close 11097 (time 1788156833.37 total 554.34 last 19.96) - open/close 11346 (time 1788156843.38 total 564.35 last 24.88) - open/close 11540 (time 1788156853.81 total 574.78 last 18.60) - open/close 11653 (time 1788156864.61 total 585.58 last 10.46) - open/close 11835 (time 1788156874.62 total 595.58 last 18.18) - open/close 12049 (time 1788156885.03 total 606.00 last 20.55) - open/close 12213 (time 1788156895.03 total 616.00 last 16.40) - open/close 12436 (time 1788156905.06 total 626.02 last 22.25) - open/close 12630 (time 1788156915.18 total 636.14 last 19.17) - open/close 12812 (time 1788156925.19 total 646.15 last 18.18) - open/close 13024 (time 1788156935.92 total 656.88 last 19.76) - open/close 13261 (time 1788156945.92 total 666.89 last 23.69) - open/close 13503 (time 1788156955.93 total 676.89 last 24.20) - open/close 13691 (time 1788156965.93 total 686.90 last 18.79) - open/close 13879 (time 1788156976.94 total 697.90 last 17.08) - open/close 13954 (time 1788156986.94 total 707.90 last 7.50) - open/close 14187 (time 1788156996.95 total 717.92 last 23.27) - open/close 14266 (time 1788157006.96 total 727.92 last 7.90) - open/close 14438 (time 1788157017.51 total 738.48 last 16.30) - open/close 14577 (time 1788157027.53 total 748.49 last 13.88) - open/close 14692 (time 1788157038.00 total 758.97 last 10.97) - open/close 14823 (time 1788157048.37 total 769.34 last 12.64) - open/close 15012 (time 1788157058.61 total 779.57 last 18.46) - open/close 15240 (time 1788157068.62 total 789.58 last 22.78) - open/close 15338 (time 1788157079.29 total 800.26 last 9.18) - open/close 15488 (time 1788157089.30 total 810.26 last 14.99) - open/close 15558 (time 1788157100.47 total 821.43 last 6.27) - open/close 15801 (time 1788157110.47 total 831.44 last 24.28) - open/close 15895 (time 1788157121.26 total 842.23 last 8.71) - open/close 16094 (time 1788157131.75 total 852.72 last 18.97) - open/close 16272 (time 1788157142.63 total 863.60 last 16.36) - open/close 16317 (time 1788157155.06 total 876.02 last 3.62) - open/close 16382 (time 1788157165.25 total 886.21 last 6.38) - open/close 16465 (time 1788157177.85 total 898.81 last 6.59) - open/close 16643 (time 1788157188.81 total 909.78 last 16.23) - open/close 16844 (time 1788157199.41 total 920.37 last 18.97) - open/close 17028 (time 1788157210.69 total 931.66 last 16.30) - open/close 17138 (time 1788157221.47 total 942.43 last 10.21) - open/close 17263 (time 1788157232.38 total 953.34 last 11.46) - open/close 17439 (time 1788157243.73 total 964.69 last 15.51) - open/close 17565 (time 1788157253.80 total 974.77 last 12.50) - open/close 17802 (time 1788157263.85 total 984.82 last 23.58) - open/close 17990 (time 1788157273.86 total 994.83 last 18.78) - open/close 18539 (time 1788157283.87 total 1004.84 last 54.86) - open/close 19140 (time 1788157293.88 total 1014.85 last 60.03) - open/close 19729 (time 1788157303.90 total 1024.86 last 58.82) - open/close 20000 (time 1788157308.55 total 1029.51 last 58.27) - open/close 20630 (time 1788157318.56 total 1039.53 last 62.90) - open/close 21159 (time 1788157328.58 total 1049.55 last 52.81) - open/close 21784 (time 1788157338.58 total 1059.55 last 62.48) - open/close 22383 (time 1788157348.60 total 1069.57 last 59.79) - open/close 23042 (time 1788157358.61 total 1079.57 last 65.87) - open/close 23665 (time 1788157368.62 total 1089.58 last 62.22) - open/close 24247 (time 1788157378.63 total 1099.60 last 58.11) - open/close 24838 (time 1788157388.64 total 1109.60 last 59.09) - open/close 25429 (time 1788157398.64 total 1119.60 last 59.10) - open/close 26004 (time 1788157408.64 total 1129.60 last 57.48) - open/close 26626 (time 1788157418.65 total 1139.61 last 62.15) - open/close 27220 (time 1788157428.67 total 1149.63 last 59.28) - open/close 27852 (time 1788157438.68 total 1159.65 last 63.10) - open/close 28502 (time 1788157448.69 total 1169.65 last 64.96) - open/close 29145 (time 1788157458.72 total 1179.68 last 64.11) - open/close 29763 (time 1788157468.72 total 1189.68 last 61.79) - open/close 30000 (time 1788157471.95 total 1192.91 last 73.41) - open/close 30695 (time 1788157481.95 total 1202.92 last 69.45) - open/close 31280 (time 1788157491.97 total 1212.94 last 58.40) - open/close 31853 (time 1788157501.98 total 1222.95 last 57.23) - open/close 32390 (time 1788157511.99 total 1232.96 last 53.67) - open/close 32992 (time 1788157522.01 total 1242.98 last 60.06) - open/close 33537 (time 1788157532.04 total 1253.00 last 54.36) - open/close 34142 (time 1788157542.04 total 1263.01 last 60.47) - open/close 34739 (time 1788157552.05 total 1273.01 last 59.67) - open/close 35374 (time 1788157562.06 total 1283.02 last 63.45) - open/close 35994 (time 1788157572.06 total 1293.03 last 61.98) - open/close 36498 (time 1788157582.07 total 1303.03 last 50.38) - open/close 36927 (time 1788157592.08 total 1313.04 last 42.85) - open/close 37535 (time 1788157602.09 total 1323.05 last 60.75) - open/close 38152 (time 1788157612.11 total 1333.07 last 61.57) - open/close 38737 (time 1788157622.11 total 1343.07 last 58.48) - open/close 39341 (time 1788157632.11 total 1353.08 last 60.37) - open/close 39949 (time 1788157642.12 total 1363.08 last 60.77) - open/close 40519 (time 1788157652.13 total 1373.09 last 56.96) - open/close 41130 (time 1788157662.14 total 1383.10 last 61.04) - open/close 41785 (time 1788157672.15 total 1393.11 last 65.43) - open/close 42428 (time 1788157682.15 total 1403.11 last 64.28) - open/close 43055 (time 1788157692.16 total 1413.12 last 62.64) - open/close 43699 (time 1788157702.17 total 1423.14 last 64.32) - open/close 44335 (time 1788157712.20 total 1433.17 last 63.40) - open/close 44988 (time 1788157722.23 total 1443.20 last 65.11) - open/close 45594 (time 1788157732.24 total 1453.21 last 60.53) - open/close 46203 (time 1788157742.25 total 1463.22 last 60.84) - open/close 46828 (time 1788157752.26 total 1473.23 last 62.46) - open/close 47395 (time 1788157762.29 total 1483.25 last 56.56) - open/close 47950 (time 1788157772.32 total 1493.28 last 55.33) - open/close 48526 (time 1788157782.32 total 1503.29 last 57.57) - open/close 49110 (time 1788157792.33 total 1513.29 last 58.37) - open/close 49659 (time 1788157802.33 total 1523.30 last 54.87) total: 50000 open/close in 1529.63 seconds: 32.69 ops/second - unlinked 0 (time 1788157813 ; total 0 ; last 0) - unlinked 10000 (time 1788157986 ; total 173 ; last 173) - unlinked 20000 (time 1788158154 ; total 341 ; last 168) - unlinked 30000 (time 1788158312 ; total 499 ; last 158) - unlinked 40000 (time 1788158468 ; total 655 ; last 156) total: 50000 unlinks in 815 seconds: 61.349693 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg336-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg336-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg336-server stop mds service on oleg336-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg336-server unloading modules via unload_modules_local on: 'oleg336-server' oleg336-server: oleg336-server.virtnet: executing unload_modules_local oleg336-server: modules unloaded. pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-client: ssh exited with exit code 2 pdsh@oleg336-client: oleg336-server: ssh exited with exit code 2