== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:52:27 (1782759147) start mds service on oleg246-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing load_modules_local oleg246-server: Loading modules from /home/green/git/lustre-release/lustre oleg246-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg246-server: detected 4 online CPUs by sysfs oleg246-server: MODOPTS_LIBCFS= oleg246-server: Force libcfs to create 2 CPU partitions oleg246-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg246-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg246-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg246-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg246-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg246-server: oleg246-server.virtnet: executing set_default_debug -1 all pdsh@oleg246-client: oleg246-server: ssh exited with exit code 1 Started lustre-OST0000 oleg246-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg246-client.virtnet: -o user_xattr,flock 192.168.202.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 202 (time 1782759214.60 total 10.41 last 19.40) - open/close 385 (time 1782759225.36 total 21.17 last 17.01) - open/close 588 (time 1782759235.40 total 31.21 last 20.22) - open/close 733 (time 1782759245.56 total 41.37 last 14.26) - open/close 927 (time 1782759256.07 total 51.88 last 18.46) - open/close 1075 (time 1782759266.43 total 62.24 last 14.28) - open/close 1236 (time 1782759277.26 total 73.07 last 14.87) - open/close 1408 (time 1782759287.26 total 83.07 last 17.20) - open/close 1716 (time 1782759297.34 total 93.15 last 30.55) - open/close 1947 (time 1782759308.93 total 104.74 last 19.94) - open/close 2177 (time 1782759319.29 total 115.10 last 22.20) - open/close 2338 (time 1782759329.33 total 125.15 last 16.03) - open/close 2579 (time 1782759340.16 total 135.97 last 22.27) - open/close 2782 (time 1782759350.53 total 146.34 last 19.57) - open/close 3004 (time 1782759360.70 total 156.52 last 21.82) - open/close 3175 (time 1782759372.54 total 168.35 last 14.45) - open/close 3420 (time 1782759382.78 total 178.59 last 23.93) - open/close 3620 (time 1782759392.78 total 188.59 last 20.00) - open/close 3991 (time 1782759403.22 total 199.03 last 35.53) - open/close 4334 (time 1782759413.23 total 209.04 last 34.27) - open/close 4561 (time 1782759423.25 total 219.06 last 22.66) - open/close 4878 (time 1782759433.34 total 229.16 last 31.40) - open/close 5204 (time 1782759443.39 total 239.21 last 32.44) - open/close 5515 (time 1782759453.40 total 249.21 last 31.08) - open/close 5713 (time 1782759464.32 total 260.13 last 18.13) - open/close 5994 (time 1782759474.81 total 270.62 last 26.79) - open/close 6357 (time 1782759484.82 total 280.63 last 36.28) - open/close 6585 (time 1782759494.82 total 290.63 last 22.79) - open/close 6754 (time 1782759505.21 total 301.03 last 16.26) - open/close 6942 (time 1782759515.26 total 311.07 last 18.72) - open/close 7178 (time 1782759525.85 total 321.67 last 22.27) - open/close 7354 (time 1782759536.38 total 332.20 last 16.71) - open/close 7478 (time 1782759546.40 total 342.21 last 12.38) - open/close 7649 (time 1782759556.49 total 352.30 last 16.96) - open/close 7833 (time 1782759567.51 total 363.32 last 16.70) - open/close 8004 (time 1782759577.53 total 373.34 last 17.06) - open/close 8156 (time 1782759587.71 total 383.52 last 14.93) - open/close 8260 (time 1782759599.40 total 395.21 last 8.90) - open/close 8408 (time 1782759609.92 total 405.73 last 14.07) - open/close 8519 (time 1782759620.04 total 415.86 last 10.96) - open/close 8659 (time 1782759630.64 total 426.45 last 13.21) - open/close 8879 (time 1782759640.66 total 436.47 last 21.96) - open/close 9008 (time 1782759650.66 total 446.47 last 12.90) - open/close 9264 (time 1782759660.94 total 456.75 last 24.92) - open/close 9373 (time 1782759670.94 total 466.75 last 10.90) - open/close 9449 (time 1782759683.04 total 478.86 last 6.28) - open/close 9649 (time 1782759697.41 total 493.22 last 13.92) - open/close 9820 (time 1782759707.43 total 503.25 last 17.06) - open/close 10000 (time 1782759717.09 total 512.90 last 18.65) - open/close 10191 (time 1782759727.10 total 522.91 last 19.08) - open/close 10411 (time 1782759737.99 total 533.80 last 20.21) - open/close 10582 (time 1782759748.59 total 544.40 last 16.13) - open/close 10659 (time 1782759759.49 total 555.31 last 7.06) - open/close 10757 (time 1782759770.63 total 566.44 last 8.80) - open/close 10926 (time 1782759781.01 total 576.83 last 16.27) - open/close 11133 (time 1782759791.02 total 586.83 last 20.69) - open/close 11375 (time 1782759801.06 total 596.88 last 24.09) - open/close 11616 (time 1782759811.79 total 607.60 last 22.48) - open/close 11811 (time 1782759821.89 total 617.70 last 19.30) - open/close 12058 (time 1782759831.90 total 627.71 last 24.67) - open/close 12291 (time 1782759842.70 total 638.51 last 21.58) - open/close 12430 (time 1782759852.72 total 648.53 last 13.87) - open/close 12641 (time 1782759863.31 total 659.12 last 19.92) - open/close 12880 (time 1782759873.32 total 669.13 last 23.87) - open/close 13133 (time 1782759886.21 total 682.02 last 19.63) - open/close 13327 (time 1782759896.59 total 692.40 last 18.69) - open/close 13560 (time 1782759906.61 total 702.42 last 23.25) - open/close 13657 (time 1782759917.08 total 712.89 last 9.27) - open/close 13804 (time 1782759927.76 total 723.57 last 13.76) - open/close 14035 (time 1782759937.80 total 733.62 last 22.99) - open/close 14732 (time 1782759947.81 total 743.62 last 69.66) - open/close 15527 (time 1782759957.82 total 753.63 last 79.45) - open/close 16420 (time 1782759967.83 total 763.64 last 89.24) - open/close 17132 (time 1782759977.84 total 773.65 last 71.12) - open/close 17870 (time 1782759987.84 total 783.65 last 73.78) - open/close 18689 (time 1782759997.85 total 793.66 last 81.85) - open/close 19415 (time 1782760007.86 total 803.67 last 72.53) - open/close 20000 (time 1782760015.25 total 811.06 last 79.12) - open/close 20765 (time 1782760025.25 total 821.07 last 76.47) - open/close 21549 (time 1782760035.26 total 831.07 last 78.34) - open/close 22288 (time 1782760045.28 total 841.09 last 73.80) - open/close 23107 (time 1782760055.28 total 851.09 last 81.86) - open/close 23987 (time 1782760065.29 total 861.10 last 87.94) - open/close 24841 (time 1782760075.29 total 871.10 last 85.38) - open/close 25623 (time 1782760085.29 total 881.10 last 78.19) - open/close 26371 (time 1782760095.29 total 891.11 last 74.77) - open/close 27219 (time 1782760105.30 total 901.11 last 84.80) - open/close 28026 (time 1782760115.30 total 911.11 last 80.65) - open/close 28797 (time 1782760125.32 total 921.14 last 76.92) - open/close 29590 (time 1782760135.33 total 931.15 last 79.22) - open/close 30000 (time 1782760141.07 total 936.88 last 71.53) - open/close 30744 (time 1782760151.10 total 946.92 last 74.12) - open/close 31391 (time 1782760161.14 total 956.95 last 64.49) - open/close 32080 (time 1782760171.14 total 966.95 last 68.87) - open/close 32830 (time 1782760181.15 total 976.96 last 74.93) - open/close 33696 (time 1782760191.16 total 986.97 last 86.53) - open/close 34481 (time 1782760201.16 total 996.97 last 78.48) - open/close 35289 (time 1782760211.17 total 1006.98 last 80.73) - open/close 35968 (time 1782760221.17 total 1016.98 last 67.89) - open/close 36571 (time 1782760231.18 total 1026.99 last 60.24) - open/close 37272 (time 1782760241.20 total 1037.01 last 69.97) - open/close 38027 (time 1782760251.21 total 1047.02 last 75.42) - open/close 38775 (time 1782760261.24 total 1057.05 last 74.59) - open/close 39566 (time 1782760271.25 total 1067.06 last 79.03) - open/close 40000 (time 1782760277.07 total 1072.89 last 74.51) - open/close 40713 (time 1782760287.08 total 1082.90 last 71.23) - open/close 41434 (time 1782760297.10 total 1092.92 last 71.95) - open/close 42153 (time 1782760307.11 total 1102.92 last 71.85) - open/close 42873 (time 1782760317.11 total 1112.92 last 72.00) - open/close 43625 (time 1782760327.12 total 1122.93 last 75.15) - open/close 44409 (time 1782760337.17 total 1132.98 last 77.99) - open/close 45245 (time 1782760347.18 total 1142.99 last 83.56) - open/close 45984 (time 1782760357.18 total 1152.99 last 73.88) - open/close 46707 (time 1782760367.18 total 1162.99 last 72.28) - open/close 47420 (time 1782760377.18 total 1172.99 last 71.29) - open/close 48196 (time 1782760387.19 total 1183.00 last 77.57) - open/close 49000 (time 1782760397.20 total 1193.01 last 80.31) - open/close 49744 (time 1782760407.21 total 1203.02 last 74.31) total: 50000 open/close in 1206.65 seconds: 41.44 ops/second - unlinked 0 (time 1782760415 ; total 0 ; last 0) - unlinked 10000 (time 1782760557 ; total 142 ; last 142) - unlinked 20000 (time 1782760700 ; total 285 ; last 143) - unlinked 30000 (time 1782760837 ; total 422 ; last 137) - unlinked 40000 (time 1782760974 ; total 559 ; last 137) total: 50000 unlinks in 693 seconds: 72.150070 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg246-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg246-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg246-server stop mds service on oleg246-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg246-server unloading modules via unload_modules_local on: 'oleg246-server' oleg246-server: oleg246-server.virtnet: executing unload_modules_local oleg246-server: modules unloaded. pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-client: ssh exited with exit code 2 pdsh@oleg246-client: oleg246-server: ssh exited with exit code 2