== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:05:08 (1787382308) start mds service on oleg445-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing load_modules_local oleg445-server: Loading modules from /home/green/git/lustre-release/lustre oleg445-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg445-server: detected 4 online CPUs by sysfs oleg445-server: MODOPTS_LIBCFS= oleg445-server: Force libcfs to create 2 CPU partitions oleg445-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg445-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg445-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg445-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg445-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg445-server: oleg445-server.virtnet: executing set_default_debug -1 all pdsh@oleg445-client: oleg445-server: ssh exited with exit code 1 Started lustre-OST0000 oleg445-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg445-client.virtnet: -o user_xattr,flock 192.168.204.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 92 (time 1787382399.68 total 11.72 last 7.85) - open/close 215 (time 1787382410.18 total 22.22 last 11.72) - open/close 445 (time 1787382422.28 total 34.32 last 19.01) - open/close 719 (time 1787382432.30 total 44.34 last 27.35) - open/close 830 (time 1787382442.31 total 54.35 last 11.09) - open/close 967 (time 1787382452.32 total 64.35 last 13.70) - open/close 1116 (time 1787382463.23 total 75.27 last 13.65) - open/close 1252 (time 1787382473.94 total 85.98 last 12.70) - open/close 1432 (time 1787382484.55 total 96.58 last 16.97) - open/close 1596 (time 1787382494.57 total 106.61 last 16.36) - open/close 1765 (time 1787382504.59 total 116.62 last 16.88) - open/close 1930 (time 1787382515.20 total 127.24 last 15.54) - open/close 2135 (time 1787382525.21 total 137.24 last 20.49) - open/close 2190 (time 1787382535.21 total 147.25 last 5.50) - open/close 2292 (time 1787382546.20 total 158.24 last 9.28) - open/close 2398 (time 1787382556.37 total 168.41 last 10.42) - open/close 2575 (time 1787382567.36 total 179.39 last 16.11) - open/close 2732 (time 1787382577.53 total 189.56 last 15.44) - open/close 2857 (time 1787382587.55 total 199.58 last 12.48) - open/close 3023 (time 1787382597.56 total 209.59 last 16.59) - open/close 3180 (time 1787382607.58 total 219.62 last 15.66) - open/close 3241 (time 1787382617.59 total 229.62 last 6.10) - open/close 3373 (time 1787382628.10 total 240.13 last 12.56) - open/close 3535 (time 1787382639.18 total 251.21 last 14.62) - open/close 3740 (time 1787382649.73 total 261.76 last 19.43) - open/close 3899 (time 1787382660.28 total 272.32 last 15.06) - open/close 4049 (time 1787382670.52 total 282.56 last 14.65) - open/close 4157 (time 1787382680.54 total 292.57 last 10.78) - open/close 4205 (time 1787382691.21 total 303.25 last 4.50) - open/close 4360 (time 1787382701.44 total 313.47 last 15.16) - open/close 4553 (time 1787382714.18 total 326.21 last 15.15) - open/close 4763 (time 1787382724.87 total 336.90 last 19.65) - open/close 5012 (time 1787382734.87 total 346.91 last 24.89) - open/close 5207 (time 1787382745.18 total 357.21 last 18.92) - open/close 5353 (time 1787382756.75 total 368.79 last 12.62) - open/close 5525 (time 1787382766.76 total 378.79 last 17.20) - open/close 5761 (time 1787382777.60 total 389.64 last 21.76) - open/close 6033 (time 1787382788.56 total 400.60 last 24.82) - open/close 6170 (time 1787382798.87 total 410.90 last 13.30) - open/close 6324 (time 1787382808.90 total 420.93 last 15.35) - open/close 6520 (time 1787382820.04 total 432.07 last 17.60) - open/close 6784 (time 1787382830.05 total 442.08 last 26.36) - open/close 6992 (time 1787382840.53 total 452.56 last 19.85) - open/close 7181 (time 1787382851.56 total 463.59 last 17.13) - open/close 7271 (time 1787382861.77 total 473.80 last 8.82) - open/close 7472 (time 1787382871.77 total 483.80 last 20.09) - open/close 7706 (time 1787382882.71 total 494.74 last 21.39) - open/close 7945 (time 1787382892.97 total 505.00 last 23.30) - open/close 8133 (time 1787382904.64 total 516.68 last 16.10) - open/close 8306 (time 1787382915.53 total 527.57 last 15.88) - open/close 8480 (time 1787382925.55 total 537.58 last 17.38) - open/close 8718 (time 1787382935.56 total 547.59 last 23.78) - open/close 8904 (time 1787382945.60 total 557.63 last 18.53) - open/close 9014 (time 1787382955.68 total 567.71 last 10.91) - open/close 9233 (time 1787382966.36 total 578.39 last 20.50) - open/close 9418 (time 1787382976.37 total 588.40 last 18.48) - open/close 9672 (time 1787382986.38 total 598.42 last 25.37) - open/close 9888 (time 1787382997.33 total 609.36 last 19.74) - open/close 10000 (time 1787383005.05 total 617.08 last 14.51) - open/close 10223 (time 1787383015.06 total 627.09 last 22.27) - open/close 10400 (time 1787383025.29 total 637.32 last 17.30) - open/close 10534 (time 1787383035.84 total 647.87 last 12.70) - open/close 10725 (time 1787383046.56 total 658.60 last 17.82) - open/close 10860 (time 1787383056.61 total 668.64 last 13.44) - open/close 11049 (time 1787383066.95 total 678.99 last 18.26) - open/close 11306 (time 1787383077.33 total 689.37 last 24.76) - open/close 11440 (time 1787383087.81 total 699.84 last 12.80) - open/close 11713 (time 1787383098.20 total 710.23 last 26.27) - open/close 11973 (time 1787383108.21 total 720.25 last 25.96) - open/close 12133 (time 1787383120.26 total 732.29 last 13.28) - open/close 12383 (time 1787383130.27 total 742.30 last 24.98) - open/close 12573 (time 1787383140.48 total 752.51 last 18.60) - open/close 12804 (time 1787383150.50 total 762.53 last 23.06) - open/close 12966 (time 1787383160.51 total 772.54 last 16.19) - open/close 13185 (time 1787383173.20 total 785.24 last 17.25) - open/close 13258 (time 1787383183.21 total 795.24 last 7.29) - open/close 13497 (time 1787383193.66 total 805.70 last 22.87) - open/close 13735 (time 1787383203.68 total 815.71 last 23.77) - open/close 13941 (time 1787383214.03 total 826.07 last 19.89) - open/close 14128 (time 1787383224.96 total 836.99 last 17.12) - open/close 14306 (time 1787383235.27 total 847.30 last 17.27) - open/close 14564 (time 1787383245.70 total 857.73 last 24.73) - open/close 14757 (time 1787383255.71 total 867.74 last 19.29) - open/close 15001 (time 1787383266.25 total 878.29 last 23.14) - open/close 15215 (time 1787383276.54 total 888.57 last 20.81) - open/close 15442 (time 1787383287.12 total 899.16 last 21.44) - open/close 15649 (time 1787383297.14 total 909.17 last 20.67) - open/close 15871 (time 1787383307.73 total 919.76 last 20.96) - open/close 16064 (time 1787383318.41 total 930.45 last 18.07) - open/close 16339 (time 1787383329.43 total 941.46 last 24.97) - open/close 16521 (time 1787383339.43 total 951.46 last 18.20) - open/close 16726 (time 1787383350.73 total 962.76 last 18.15) - open/close 16866 (time 1787383362.04 total 974.07 last 12.37) - open/close 17084 (time 1787383372.04 total 984.07 last 21.80) - open/close 17160 (time 1787383382.48 total 994.51 last 7.28) - open/close 17315 (time 1787383392.48 total 1004.51 last 15.50) - open/close 17459 (time 1787383402.49 total 1014.52 last 14.39) - open/close 17584 (time 1787383412.63 total 1024.66 last 12.33) - open/close 17700 (time 1787383422.64 total 1034.67 last 11.59) - open/close 18357 (time 1787383432.64 total 1044.68 last 65.67) - open/close 19021 (time 1787383442.65 total 1054.69 last 66.32) - open/close 19682 (time 1787383452.67 total 1064.70 last 66.02) - open/close 20000 (time 1787383457.75 total 1069.78 last 62.53) - open/close 20703 (time 1787383467.76 total 1079.80 last 70.22) - open/close 21351 (time 1787383477.77 total 1089.81 last 64.72) - open/close 22010 (time 1787383487.78 total 1099.81 last 65.86) - open/close 22643 (time 1787383497.78 total 1109.82 last 63.27) - open/close 23271 (time 1787383507.79 total 1119.83 last 62.75) - open/close 23924 (time 1787383517.80 total 1129.83 last 65.28) - open/close 24542 (time 1787383527.83 total 1139.86 last 61.60) - open/close 25061 (time 1787383537.84 total 1149.87 last 51.86) - open/close 25642 (time 1787383547.84 total 1159.87 last 58.09) - open/close 26227 (time 1787383557.85 total 1169.89 last 58.41) - open/close 26798 (time 1787383567.86 total 1179.89 last 57.08) - open/close 27464 (time 1787383577.87 total 1189.90 last 66.54) - open/close 28080 (time 1787383587.88 total 1199.91 last 61.52) - open/close 28732 (time 1787383597.89 total 1209.92 last 65.15) - open/close 29393 (time 1787383607.89 total 1219.92 last 66.09) - open/close 29927 (time 1787383617.91 total 1229.94 last 53.31) - open/close 30488 (time 1787383627.92 total 1239.95 last 56.02) - open/close 31022 (time 1787383637.98 total 1250.01 last 53.10) - open/close 31567 (time 1787383647.98 total 1260.02 last 54.47) - open/close 32110 (time 1787383658.00 total 1270.03 last 54.23) - open/close 32698 (time 1787383668.01 total 1280.04 last 58.73) - open/close 33347 (time 1787383678.01 total 1290.04 last 64.89) - open/close 34029 (time 1787383688.01 total 1300.05 last 68.19) - open/close 34698 (time 1787383698.03 total 1310.07 last 66.76) - open/close 35265 (time 1787383708.04 total 1320.07 last 56.68) - open/close 35893 (time 1787383718.05 total 1330.08 last 62.74) - open/close 36496 (time 1787383728.05 total 1340.08 last 60.30) - open/close 37210 (time 1787383738.05 total 1350.09 last 71.36) - open/close 37866 (time 1787383748.06 total 1360.10 last 65.54) - open/close 38509 (time 1787383758.09 total 1370.12 last 64.15) - open/close 39146 (time 1787383768.11 total 1380.14 last 63.53) - open/close 39797 (time 1787383778.12 total 1390.15 last 65.04) - open/close 40000 (time 1787383781.84 total 1393.88 last 54.53) - open/close 40552 (time 1787383791.86 total 1403.90 last 55.09) - open/close 41190 (time 1787383801.86 total 1413.90 last 63.78) - open/close 41815 (time 1787383811.90 total 1423.93 last 62.29) - open/close 42490 (time 1787383821.91 total 1433.94 last 67.44) - open/close 43129 (time 1787383831.92 total 1443.95 last 63.85) - open/close 43791 (time 1787383841.92 total 1453.95 last 66.17) - open/close 44417 (time 1787383851.93 total 1463.97 last 62.53) - open/close 45023 (time 1787383861.96 total 1473.99 last 60.43) - open/close 45665 (time 1787383871.97 total 1484.00 last 64.16) - open/close 46362 (time 1787383881.97 total 1494.00 last 69.68) - open/close 46989 (time 1787383891.98 total 1504.01 last 62.67) - open/close 47646 (time 1787383901.98 total 1514.01 last 65.67) - open/close 48298 (time 1787383911.99 total 1524.02 last 65.15) - open/close 48938 (time 1787383922.00 total 1534.03 last 63.91) - open/close 49546 (time 1787383932.01 total 1544.05 last 60.72) total: 50000 open/close in 1553.04 seconds: 32.19 ops/second - unlinked 0 (time 1787383946 ; total 0 ; last 0) - unlinked 10000 (time 1787384118 ; total 172 ; last 172) - unlinked 20000 (time 1787384282 ; total 336 ; last 164) - unlinked 30000 (time 1787384455 ; total 509 ; last 173) - unlinked 40000 (time 1787384631 ; total 685 ; last 176) total: 50000 unlinks in 851 seconds: 58.754406 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg445-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg445-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg445-server stop mds service on oleg445-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg445-server unloading modules via unload_modules_local on: 'oleg445-server' oleg445-server: oleg445-server.virtnet: executing unload_modules_local oleg445-server: modules unloaded. pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-client: ssh exited with exit code 2 pdsh@oleg445-client: oleg445-server: ssh exited with exit code 2