== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:14:03 (1788156843) start mds service on oleg121-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing load_modules_local oleg121-server: Loading modules from /home/green/git/lustre-release/lustre oleg121-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg121-server: detected 4 online CPUs by sysfs oleg121-server: MODOPTS_LIBCFS= oleg121-server: Force libcfs to create 2 CPU partitions oleg121-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg121-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg121-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg121-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg121-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg121-server: oleg121-server.virtnet: executing set_default_debug -1 all pdsh@oleg121-client: oleg121-server: ssh exited with exit code 1 Started lustre-OST0000 oleg121-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg121-client.virtnet: -o user_xattr,flock 192.168.201.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 170 (time 1788156935.70 total 10.89 last 15.61) - open/close 287 (time 1788156945.81 total 21.00 last 11.57) - open/close 439 (time 1788156955.83 total 31.03 last 15.17) - open/close 637 (time 1788156965.84 total 41.03 last 19.79) - open/close 803 (time 1788156976.91 total 52.10 last 14.99) - open/close 1062 (time 1788156987.33 total 62.52 last 24.87) - open/close 1143 (time 1788156997.83 total 73.02 last 7.71) - open/close 1338 (time 1788157008.08 total 83.28 last 19.01) - open/close 1462 (time 1788157018.20 total 93.39 last 12.26) - open/close 1721 (time 1788157028.21 total 103.40 last 25.88) - open/close 1944 (time 1788157038.54 total 113.73 last 21.58) - open/close 2147 (time 1788157051.58 total 126.77 last 15.57) - open/close 2388 (time 1788157062.36 total 137.55 last 22.35) - open/close 2567 (time 1788157072.57 total 147.76 last 17.53) - open/close 2704 (time 1788157082.58 total 157.77 last 13.69) - open/close 2843 (time 1788157093.34 total 168.54 last 12.91) - open/close 3022 (time 1788157103.56 total 178.76 last 17.51) - open/close 3145 (time 1788157113.59 total 188.78 last 12.27) - open/close 3235 (time 1788157123.72 total 198.92 last 8.88) - open/close 3375 (time 1788157134.23 total 209.42 last 13.33) - open/close 3521 (time 1788157144.24 total 219.43 last 14.58) - open/close 3770 (time 1788157155.28 total 230.47 last 22.55) - open/close 4037 (time 1788157165.29 total 240.48 last 26.68) - open/close 4174 (time 1788157176.13 total 251.32 last 12.64) - open/close 4253 (time 1788157186.32 total 261.51 last 7.75) - open/close 4432 (time 1788157197.26 total 272.45 last 16.37) - open/close 4605 (time 1788157207.27 total 282.46 last 17.28) - open/close 4756 (time 1788157217.73 total 292.92 last 14.44) - open/close 4996 (time 1788157228.29 total 303.48 last 22.72) - open/close 5201 (time 1788157238.81 total 314.00 last 19.48) - open/close 5399 (time 1788157248.82 total 324.01 last 19.78) - open/close 5541 (time 1788157259.33 total 334.52 last 13.52) - open/close 5692 (time 1788157269.74 total 344.93 last 14.51) - open/close 5899 (time 1788157280.57 total 355.77 last 19.10) - open/close 6043 (time 1788157290.94 total 366.13 last 13.89) - open/close 6302 (time 1788157300.96 total 376.15 last 25.85) - open/close 6451 (time 1788157311.03 total 386.22 last 14.79) - open/close 6669 (time 1788157321.04 total 396.23 last 21.79) - open/close 6901 (time 1788157331.78 total 406.97 last 21.60) - open/close 7163 (time 1788157341.94 total 417.13 last 25.79) - open/close 7409 (time 1788157351.96 total 427.15 last 24.57) - open/close 7666 (time 1788157362.00 total 437.19 last 25.59) - open/close 7856 (time 1788157372.96 total 448.16 last 17.33) - open/close 8052 (time 1788157382.98 total 458.17 last 19.57) - open/close 8149 (time 1788157393.16 total 468.35 last 9.53) - open/close 8334 (time 1788157403.20 total 478.39 last 18.42) - open/close 8398 (time 1788157413.21 total 488.40 last 6.40) - open/close 8603 (time 1788157425.54 total 500.73 last 16.63) - open/close 8718 (time 1788157436.34 total 511.54 last 10.64) - open/close 8913 (time 1788157446.62 total 521.81 last 18.98) - open/close 9109 (time 1788157457.53 total 532.72 last 17.97) - open/close 9248 (time 1788157468.24 total 543.43 last 12.97) - open/close 9445 (time 1788157478.42 total 553.62 last 19.35) - open/close 9694 (time 1788157488.44 total 563.63 last 24.87) - open/close 9827 (time 1788157498.46 total 573.65 last 13.27) - open/close 10000 (time 1788157505.02 total 580.21 last 26.36) - open/close 10238 (time 1788157515.02 total 590.21 last 23.80) - open/close 10472 (time 1788157525.59 total 600.78 last 22.14) - open/close 10688 (time 1788157535.87 total 611.06 last 21.02) - open/close 11028 (time 1788157546.81 total 622.00 last 31.07) - open/close 11306 (time 1788157556.82 total 632.02 last 27.76) - open/close 11536 (time 1788157568.07 total 643.26 last 20.45) - open/close 11783 (time 1788157578.71 total 653.90 last 23.21) - open/close 12050 (time 1788157588.83 total 664.02 last 26.39) - open/close 12293 (time 1788157599.35 total 674.54 last 23.09) - open/close 12518 (time 1788157609.92 total 685.11 last 21.28) - open/close 12755 (time 1788157621.18 total 696.37 last 21.06) - open/close 12832 (time 1788157631.18 total 706.37 last 7.70) - open/close 12994 (time 1788157641.85 total 717.04 last 15.19) - open/close 13115 (time 1788157651.85 total 727.04 last 12.09) - open/close 13316 (time 1788157661.85 total 737.04 last 20.10) - open/close 13519 (time 1788157671.85 total 747.05 last 20.29) - open/close 13789 (time 1788157682.88 total 758.07 last 24.50) - open/close 14024 (time 1788157692.90 total 768.09 last 23.45) - open/close 14180 (time 1788157704.98 total 780.17 last 12.92) - open/close 14305 (time 1788157714.98 total 790.17 last 12.49) - open/close 14554 (time 1788157724.99 total 800.18 last 24.87) - open/close 14753 (time 1788157735.29 total 810.49 last 19.32) - open/close 14967 (time 1788157745.94 total 821.13 last 20.11) - open/close 15171 (time 1788157756.30 total 831.49 last 19.68) - open/close 15408 (time 1788157766.32 total 841.51 last 23.66) - open/close 15584 (time 1788157777.23 total 852.42 last 16.13) - open/close 15765 (time 1788157789.76 total 864.95 last 14.45) - open/close 15997 (time 1788157799.77 total 874.96 last 23.18) - open/close 16169 (time 1788157809.78 total 884.98 last 17.17) - open/close 16241 (time 1788157819.81 total 895.00 last 7.18) - open/close 16433 (time 1788157829.85 total 905.04 last 19.13) - open/close 16661 (time 1788157840.50 total 915.70 last 21.40) - open/close 16905 (time 1788157851.46 total 926.65 last 22.28) - open/close 17002 (time 1788157862.07 total 937.27 last 9.14) - open/close 17215 (time 1788157872.08 total 947.27 last 21.28) - open/close 17426 (time 1788157882.25 total 957.44 last 20.75) - open/close 17709 (time 1788157894.67 total 969.86 last 22.79) - open/close 17956 (time 1788157904.67 total 979.87 last 24.70) - open/close 18202 (time 1788157915.41 total 990.60 last 22.92) - open/close 18441 (time 1788157925.41 total 1000.60 last 23.90) - open/close 18623 (time 1788157935.51 total 1010.71 last 18.01) - open/close 19345 (time 1788157945.52 total 1020.71 last 72.17) - open/close 20000 (time 1788157954.43 total 1029.62 last 73.53) - open/close 20714 (time 1788157964.43 total 1039.62 last 71.36) - open/close 21466 (time 1788157974.43 total 1049.62 last 75.19) - open/close 22250 (time 1788157984.44 total 1059.63 last 78.35) - open/close 23015 (time 1788157994.45 total 1069.65 last 76.38) - open/close 23941 (time 1788158004.46 total 1079.65 last 92.54) - open/close 24703 (time 1788158014.46 total 1089.65 last 76.20) - open/close 25517 (time 1788158024.46 total 1099.66 last 81.37) - open/close 26373 (time 1788158034.47 total 1109.66 last 85.57) - open/close 27208 (time 1788158044.49 total 1119.68 last 83.36) - open/close 27937 (time 1788158054.50 total 1129.69 last 72.82) - open/close 28726 (time 1788158064.50 total 1139.69 last 78.87) - open/close 29514 (time 1788158074.50 total 1149.69 last 78.79) - open/close 30000 (time 1788158079.15 total 1154.34 last 104.51) - open/close 30717 (time 1788158089.16 total 1164.35 last 71.67) - open/close 31474 (time 1788158099.16 total 1174.35 last 75.68) - open/close 32304 (time 1788158109.16 total 1184.35 last 83.00) - open/close 33093 (time 1788158119.16 total 1194.35 last 78.88) - open/close 33834 (time 1788158129.16 total 1204.36 last 74.07) - open/close 34599 (time 1788158139.17 total 1214.36 last 76.45) - open/close 35416 (time 1788158149.17 total 1224.37 last 81.68) - open/close 36235 (time 1788158159.18 total 1234.37 last 81.88) - open/close 36907 (time 1788158169.18 total 1244.37 last 67.17) - open/close 37629 (time 1788158179.21 total 1254.40 last 71.99) - open/close 38372 (time 1788158189.21 total 1264.41 last 74.28) - open/close 39206 (time 1788158199.22 total 1274.41 last 83.37) - open/close 39986 (time 1788158209.22 total 1284.41 last 77.97) - open/close 40834 (time 1788158219.23 total 1294.42 last 84.75) - open/close 41428 (time 1788158229.23 total 1304.42 last 59.39) - open/close 42100 (time 1788158239.24 total 1314.43 last 67.14) - open/close 42810 (time 1788158249.24 total 1324.43 last 71.00) - open/close 43497 (time 1788158259.25 total 1334.44 last 68.61) - open/close 44412 (time 1788158269.26 total 1344.45 last 91.41) - open/close 45209 (time 1788158279.26 total 1354.46 last 79.68) - open/close 46084 (time 1788158289.27 total 1364.46 last 87.49) - open/close 46723 (time 1788158299.27 total 1374.46 last 63.89) - open/close 47535 (time 1788158309.28 total 1384.47 last 81.12) - open/close 48285 (time 1788158319.28 total 1394.47 last 74.99) - open/close 49012 (time 1788158329.29 total 1404.48 last 72.61) - open/close 49786 (time 1788158339.29 total 1414.48 last 77.40) total: 50000 open/close in 1417.54 seconds: 35.27 ops/second - unlinked 0 (time 1788158347 ; total 1 ; last 1) - unlinked 10000 (time 1788158487 ; total 141 ; last 140) - unlinked 20000 (time 1788158632 ; total 286 ; last 145) - unlinked 30000 (time 1788158775 ; total 429 ; last 143) - unlinked 40000 (time 1788158922 ; total 576 ; last 147) total: 50000 unlinks in 710 seconds: 70.422539 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg121-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg121-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg121-server stop mds service on oleg121-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg121-server unloading modules via unload_modules_local on: 'oleg121-server' oleg121-server: oleg121-server.virtnet: executing unload_modules_local oleg121-server: modules unloaded. pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-client: ssh exited with exit code 2 pdsh@oleg121-client: oleg121-server: ssh exited with exit code 2