== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:47:37 (1787867257) start mds service on oleg303-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing load_modules_local oleg303-server: Loading modules from /home/green/git/lustre-release/lustre oleg303-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg303-server: detected 4 online CPUs by sysfs oleg303-server: MODOPTS_LIBCFS= oleg303-server: Force libcfs to create 2 CPU partitions oleg303-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg303-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg303-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg303-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg303-client.virtnet: -o user_xattr,flock 192.168.203.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 261 (time 1787867340.22 total 10.53 last 24.79) - open/close 479 (time 1787867350.29 total 20.60 last 21.65) - open/close 716 (time 1787867360.58 total 30.89 last 23.04) - open/close 903 (time 1787867370.59 total 40.90 last 18.68) - open/close 1080 (time 1787867381.12 total 51.43 last 16.81) - open/close 1209 (time 1787867391.75 total 62.06 last 12.13) - open/close 1474 (time 1787867402.83 total 73.13 last 23.93) - open/close 1703 (time 1787867412.85 total 83.16 last 22.84) - open/close 1901 (time 1787867422.85 total 93.16 last 19.80) - open/close 2133 (time 1787867433.67 total 103.98 last 21.45) - open/close 2330 (time 1787867443.67 total 113.98 last 19.69) - open/close 2583 (time 1787867453.68 total 123.99 last 25.28) - open/close 2741 (time 1787867464.71 total 135.02 last 14.32) - open/close 3013 (time 1787867475.46 total 145.77 last 25.30) - open/close 3260 (time 1787867486.45 total 156.76 last 22.48) - open/close 3495 (time 1787867497.09 total 167.40 last 22.08) - open/close 3666 (time 1787867507.94 total 178.25 last 15.77) - open/close 3902 (time 1787867519.76 total 190.07 last 19.96) - open/close 4059 (time 1787867533.92 total 204.22 last 11.09) - open/close 4280 (time 1787867544.77 total 215.08 last 20.36) - open/close 4548 (time 1787867554.78 total 225.09 last 26.78) - open/close 4776 (time 1787867564.79 total 235.10 last 22.77) - open/close 5002 (time 1787867574.81 total 245.11 last 22.56) - open/close 5277 (time 1787867584.82 total 255.13 last 27.46) - open/close 5504 (time 1787867595.54 total 265.85 last 21.17) - open/close 5705 (time 1787867606.30 total 276.61 last 18.68) - open/close 5781 (time 1787867617.19 total 287.50 last 6.98) - open/close 5872 (time 1787867627.81 total 298.12 last 8.57) - open/close 5979 (time 1787867638.80 total 309.11 last 9.74) - open/close 6101 (time 1787867648.93 total 319.24 last 12.05) - open/close 6361 (time 1787867659.91 total 330.22 last 23.68) - open/close 6594 (time 1787867669.94 total 340.24 last 23.24) - open/close 6729 (time 1787867680.02 total 350.32 last 13.39) - open/close 6946 (time 1787867691.11 total 361.42 last 19.55) - open/close 7180 (time 1787867701.13 total 371.44 last 23.37) - open/close 7285 (time 1787867711.58 total 381.89 last 10.04) - open/close 7500 (time 1787867722.26 total 392.57 last 20.13) - open/close 7761 (time 1787867735.34 total 405.65 last 19.96) - open/close 8020 (time 1787867746.26 total 416.57 last 23.73) - open/close 8247 (time 1787867756.26 total 426.57 last 22.69) - open/close 8496 (time 1787867766.67 total 436.98 last 23.92) - open/close 8731 (time 1787867776.68 total 446.99 last 23.49) - open/close 8966 (time 1787867787.01 total 457.31 last 22.76) - open/close 9201 (time 1787867797.02 total 467.33 last 23.47) - open/close 9468 (time 1787867807.02 total 477.33 last 26.69) - open/close 9704 (time 1787867817.03 total 487.34 last 23.58) - open/close 9912 (time 1787867827.15 total 497.45 last 20.56) - open/close 10000 (time 1787867832.31 total 502.62 last 17.04) - open/close 10246 (time 1787867843.04 total 513.34 last 22.94) - open/close 10500 (time 1787867856.08 total 526.39 last 19.47) - open/close 10749 (time 1787867867.09 total 537.40 last 22.62) - open/close 11026 (time 1787867877.10 total 547.41 last 27.68) - open/close 11273 (time 1787867887.90 total 558.21 last 22.87) - open/close 11501 (time 1787867897.93 total 568.24 last 22.72) - open/close 11779 (time 1787867908.83 total 579.14 last 25.52) - open/close 11973 (time 1787867919.42 total 589.73 last 18.31) - open/close 12187 (time 1787867929.43 total 599.74 last 21.38) - open/close 12422 (time 1787867939.87 total 610.18 last 22.51) - open/close 12616 (time 1787867950.06 total 620.37 last 19.04) - open/close 12847 (time 1787867960.65 total 630.96 last 21.80) - open/close 13104 (time 1787867970.67 total 640.97 last 25.66) - open/close 13364 (time 1787867981.26 total 651.57 last 24.54) - open/close 13611 (time 1787867991.96 total 662.27 last 23.09) - open/close 13870 (time 1787868002.88 total 673.19 last 23.71) - open/close 14079 (time 1787868012.91 total 683.22 last 20.85) - open/close 14328 (time 1787868023.09 total 693.40 last 24.46) - open/close 14567 (time 1787868033.09 total 703.40 last 23.90) - open/close 14755 (time 1787868043.10 total 713.41 last 18.78) - open/close 15008 (time 1787868056.01 total 726.32 last 19.59) - open/close 15216 (time 1787868066.05 total 736.36 last 20.72) - open/close 15447 (time 1787868076.88 total 747.18 last 21.34) - open/close 15636 (time 1787868087.87 total 758.18 last 17.19) - open/close 15805 (time 1787868098.05 total 768.36 last 16.60) - open/close 15901 (time 1787868108.07 total 778.38 last 9.58) - open/close 15975 (time 1787868118.75 total 789.06 last 6.93) - open/close 16151 (time 1787868129.55 total 799.85 last 16.30) - open/close 16371 (time 1787868140.04 total 810.34 last 20.97) - open/close 16604 (time 1787868150.09 total 820.40 last 23.18) - open/close 16851 (time 1787868160.35 total 830.66 last 24.07) - open/close 17089 (time 1787868171.06 total 841.37 last 22.22) - open/close 17331 (time 1787868181.07 total 851.38 last 24.17) - open/close 17404 (time 1787868191.58 total 861.89 last 6.95) - open/close 17656 (time 1787868201.75 total 872.05 last 24.79) - open/close 17855 (time 1787868213.58 total 883.89 last 16.81) - open/close 18073 (time 1787868223.60 total 893.91 last 21.77) - open/close 18271 (time 1787868234.47 total 904.78 last 18.20) - open/close 18456 (time 1787868244.81 total 915.12 last 17.89) - open/close 18717 (time 1787868255.57 total 925.88 last 24.26) - open/close 18917 (time 1787868268.62 total 938.93 last 15.33) - open/close 19571 (time 1787868278.62 total 948.93 last 65.40) - open/close 20000 (time 1787868284.28 total 954.59 last 75.82) - open/close 20704 (time 1787868294.29 total 964.60 last 70.33) - open/close 21363 (time 1787868304.30 total 974.61 last 65.82) - open/close 22081 (time 1787868314.30 total 984.61 last 71.78) - open/close 22675 (time 1787868324.31 total 994.62 last 59.36) - open/close 23484 (time 1787868334.32 total 1004.63 last 80.82) - open/close 24281 (time 1787868344.33 total 1014.63 last 79.64) - open/close 25049 (time 1787868354.33 total 1024.64 last 76.79) - open/close 25855 (time 1787868364.33 total 1034.64 last 80.59) - open/close 26634 (time 1787868374.33 total 1044.64 last 77.87) - open/close 27401 (time 1787868384.37 total 1054.68 last 76.41) - open/close 28088 (time 1787868394.40 total 1064.71 last 68.52) - open/close 28890 (time 1787868404.40 total 1074.71 last 80.17) - open/close 29663 (time 1787868414.40 total 1084.71 last 77.27) - open/close 30000 (time 1787868418.56 total 1088.87 last 81.15) - open/close 30774 (time 1787868428.56 total 1098.87 last 77.36) - open/close 31549 (time 1787868438.59 total 1108.90 last 77.27) - open/close 32237 (time 1787868448.59 total 1118.90 last 68.79) - open/close 33051 (time 1787868458.61 total 1128.92 last 81.29) - open/close 33828 (time 1787868468.61 total 1138.92 last 77.67) - open/close 34550 (time 1787868478.62 total 1148.93 last 72.14) - open/close 35230 (time 1787868488.63 total 1158.94 last 67.92) - open/close 35952 (time 1787868498.64 total 1168.95 last 72.12) - open/close 36734 (time 1787868508.65 total 1178.95 last 78.19) - open/close 37487 (time 1787868518.66 total 1188.97 last 75.15) - open/close 38137 (time 1787868528.68 total 1198.98 last 64.92) - open/close 38788 (time 1787868538.68 total 1208.99 last 65.07) - open/close 39397 (time 1787868548.68 total 1218.99 last 60.89) - open/close 40000 (time 1787868557.85 total 1228.16 last 65.77) - open/close 40791 (time 1787868567.87 total 1238.17 last 78.98) - open/close 41521 (time 1787868577.87 total 1248.18 last 72.98) - open/close 42286 (time 1787868587.88 total 1258.19 last 76.42) - open/close 43078 (time 1787868597.88 total 1268.19 last 79.20) - open/close 43805 (time 1787868607.90 total 1278.21 last 72.53) - open/close 44558 (time 1787868617.91 total 1288.22 last 75.25) - open/close 45284 (time 1787868627.91 total 1298.22 last 72.59) - open/close 45907 (time 1787868637.92 total 1308.22 last 62.27) - open/close 46679 (time 1787868647.92 total 1318.23 last 77.18) - open/close 47470 (time 1787868657.93 total 1328.23 last 79.04) - open/close 48212 (time 1787868667.93 total 1338.24 last 74.18) - open/close 49036 (time 1787868677.94 total 1348.24 last 82.35) - open/close 49778 (time 1787868687.94 total 1358.25 last 74.18) total: 50000 open/close in 1361.19 seconds: 36.73 ops/second - unlinked 0 (time 1787868696 ; total 0 ; last 0) - unlinked 10000 (time 1787868834 ; total 138 ; last 138) - unlinked 20000 (time 1787868968 ; total 272 ; last 134) - unlinked 30000 (time 1787869103 ; total 407 ; last 135) - unlinked 40000 (time 1787869244 ; total 548 ; last 141) total: 50000 unlinks in 688 seconds: 72.674416 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg303-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg303-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg303-server stop mds service on oleg303-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg303-server unloading modules via unload_modules_local on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing unload_modules_local oleg303-server: modules unloaded. pdsh@oleg303-client: oleg303-server: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-client: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-client: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 2