== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:06:02 (1787601962) start mds service on oleg119-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing load_modules_local oleg119-server: Loading modules from /home/green/git/lustre-release/lustre oleg119-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg119-server: detected 4 online CPUs by sysfs oleg119-server: MODOPTS_LIBCFS= oleg119-server: Force libcfs to create 2 CPU partitions oleg119-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg119-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg119-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg119-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg119-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg119-server: oleg119-server.virtnet: executing set_default_debug -1 all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-OST0000 oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 86 (time 1787602047.50 total 10.01 last 8.59) - open/close 283 (time 1787602057.52 total 20.03 last 19.67) - open/close 520 (time 1787602067.52 total 30.03 last 23.68) - open/close 696 (time 1787602077.54 total 40.05 last 17.58) - open/close 886 (time 1787602087.55 total 50.06 last 18.98) - open/close 1104 (time 1787602097.57 total 60.08 last 21.76) - open/close 1326 (time 1787602108.16 total 70.67 last 20.96) - open/close 1543 (time 1787602118.66 total 81.17 last 20.67) - open/close 1804 (time 1787602128.77 total 91.28 last 25.81) - open/close 2062 (time 1787602139.37 total 101.88 last 24.33) - open/close 2273 (time 1787602149.38 total 111.89 last 21.08) - open/close 2460 (time 1787602160.65 total 123.16 last 16.59) - open/close 2670 (time 1787602171.09 total 133.59 last 20.13) - open/close 2820 (time 1787602181.69 total 144.19 last 14.15) - open/close 2883 (time 1787602191.70 total 154.21 last 6.29) - open/close 3082 (time 1787602201.71 total 164.22 last 19.87) - open/close 3227 (time 1787602212.08 total 174.59 last 13.99) - open/close 3295 (time 1787602223.09 total 185.60 last 6.18) - open/close 3390 (time 1787602233.14 total 195.65 last 9.45) - open/close 3519 (time 1787602244.18 total 206.68 last 11.69) - open/close 3582 (time 1787602255.02 total 217.53 last 5.81) - open/close 3719 (time 1787602265.90 total 228.41 last 12.59) - open/close 3940 (time 1787602275.90 total 238.41 last 22.10) - open/close 4176 (time 1787602285.92 total 248.42 last 23.57) - open/close 4461 (time 1787602298.14 total 260.65 last 23.31) - open/close 4752 (time 1787602308.18 total 270.68 last 29.00) - open/close 4986 (time 1787602318.20 total 280.71 last 23.35) - open/close 5251 (time 1787602329.04 total 291.55 last 24.45) - open/close 5428 (time 1787602339.77 total 302.27 last 16.50) - open/close 5657 (time 1787602350.33 total 312.84 last 21.67) - open/close 5872 (time 1787602360.35 total 322.86 last 21.47) - open/close 6153 (time 1787602370.36 total 332.86 last 28.08) - open/close 6383 (time 1787602380.79 total 343.30 last 22.04) - open/close 6568 (time 1787602391.05 total 353.55 last 18.04) - open/close 6838 (time 1787602402.12 total 364.63 last 24.38) - open/close 6879 (time 1787602412.82 total 375.32 last 3.83) - open/close 7127 (time 1787602423.15 total 385.66 last 23.99) - open/close 7379 (time 1787602433.86 total 396.37 last 23.55) - open/close 7634 (time 1787602443.86 total 406.37 last 25.49) - open/close 7835 (time 1787602453.88 total 416.39 last 20.07) - open/close 8066 (time 1787602463.88 total 426.39 last 23.10) - open/close 8317 (time 1787602474.55 total 437.05 last 23.53) - open/close 8548 (time 1787602484.55 total 447.06 last 23.08) - open/close 8669 (time 1787602494.56 total 457.07 last 12.10) - open/close 8919 (time 1787602505.26 total 467.77 last 23.35) - open/close 9191 (time 1787602515.28 total 477.79 last 27.15) - open/close 9326 (time 1787602525.95 total 488.46 last 12.66) - open/close 9575 (time 1787602536.97 total 499.48 last 22.58) - open/close 9790 (time 1787602546.99 total 509.49 last 21.48) - open/close 9974 (time 1787602556.99 total 519.50 last 18.40) - open/close 10189 (time 1787602567.04 total 529.55 last 21.39) - open/close 10446 (time 1787602578.11 total 540.62 last 23.21) - open/close 10664 (time 1787602588.12 total 550.62 last 21.79) - open/close 10911 (time 1787602599.16 total 561.67 last 22.36) - open/close 11066 (time 1787602609.27 total 571.78 last 15.33) - open/close 11290 (time 1787602619.27 total 581.78 last 22.40) - open/close 11545 (time 1787602629.28 total 591.79 last 25.48) - open/close 11784 (time 1787602639.29 total 601.79 last 23.89) - open/close 11870 (time 1787602649.67 total 612.18 last 8.28) - open/close 12008 (time 1787602659.68 total 622.19 last 13.79) - open/close 12149 (time 1787602671.23 total 633.74 last 12.21) - open/close 12342 (time 1787602681.23 total 643.74 last 19.30) - open/close 12532 (time 1787602691.24 total 653.75 last 18.98) - open/close 12702 (time 1787602702.29 total 664.80 last 15.39) - open/close 12989 (time 1787602712.30 total 674.81 last 28.66) - open/close 13239 (time 1787602723.13 total 685.64 last 23.09) - open/close 13470 (time 1787602733.70 total 696.21 last 21.85) - open/close 13663 (time 1787602743.71 total 706.22 last 19.29) - open/close 13871 (time 1787602755.45 total 717.96 last 17.71) - open/close 14017 (time 1787602766.42 total 728.93 last 13.31) - open/close 14198 (time 1787602776.42 total 738.93 last 18.10) - open/close 14508 (time 1787602787.34 total 749.85 last 28.39) - open/close 14626 (time 1787602805.38 total 767.88 last 6.54) - open/close 14725 (time 1787602815.43 total 777.93 last 9.85) - open/close 14904 (time 1787602826.30 total 788.81 last 16.46) - open/close 15061 (time 1787602836.33 total 798.84 last 15.65) - open/close 15209 (time 1787602846.35 total 808.86 last 14.78) - open/close 15376 (time 1787602856.96 total 819.47 last 15.74) - open/close 15637 (time 1787602866.96 total 829.47 last 26.09) - open/close 15892 (time 1787602876.97 total 839.48 last 25.48) - open/close 16063 (time 1787602886.98 total 849.49 last 17.09) - open/close 16281 (time 1787602896.98 total 859.49 last 21.80) - open/close 16512 (time 1787602906.98 total 869.49 last 23.09) - open/close 16729 (time 1787602918.01 total 880.52 last 19.68) - open/close 16879 (time 1787602928.70 total 891.21 last 14.03) - open/close 17147 (time 1787602938.71 total 901.22 last 26.77) - open/close 17300 (time 1787602948.98 total 911.49 last 14.89) - open/close 17562 (time 1787602959.99 total 922.50 last 23.80) - open/close 17684 (time 1787602970.18 total 932.68 last 11.98) - open/close 17832 (time 1787602980.62 total 943.13 last 14.17) - open/close 17981 (time 1787602990.90 total 953.41 last 14.50) - open/close 18376 (time 1787603000.90 total 963.41 last 39.50) - open/close 19186 (time 1787603010.90 total 973.41 last 80.99) - open/close 20000 (time 1787603019.68 total 982.19 last 92.67) - open/close 20946 (time 1787603029.68 total 992.19 last 94.59) - open/close 21699 (time 1787603039.69 total 1002.20 last 75.27) - open/close 22514 (time 1787603049.70 total 1012.20 last 81.45) - open/close 23313 (time 1787603059.70 total 1022.21 last 79.87) - open/close 24105 (time 1787603069.71 total 1032.22 last 79.14) - open/close 24924 (time 1787603079.74 total 1042.25 last 81.63) - open/close 25534 (time 1787603089.77 total 1052.28 last 60.82) - open/close 26100 (time 1787603099.78 total 1062.28 last 56.56) - open/close 26863 (time 1787603109.78 total 1072.29 last 76.28) - open/close 27705 (time 1787603119.79 total 1082.29 last 84.14) - open/close 28633 (time 1787603129.79 total 1092.30 last 92.75) - open/close 29475 (time 1787603139.80 total 1102.30 last 84.16) - open/close 30000 (time 1787603145.63 total 1108.14 last 89.92) - open/close 30864 (time 1787603155.65 total 1118.16 last 86.28) - open/close 31530 (time 1787603165.67 total 1128.18 last 66.47) - open/close 32318 (time 1787603175.68 total 1138.19 last 78.72) - open/close 33112 (time 1787603185.69 total 1148.20 last 79.34) - open/close 33995 (time 1787603195.70 total 1158.21 last 88.21) - open/close 34758 (time 1787603205.71 total 1168.22 last 76.20) - open/close 35492 (time 1787603215.74 total 1178.24 last 73.21) - open/close 36192 (time 1787603225.74 total 1188.25 last 69.95) - open/close 36777 (time 1787603235.76 total 1198.27 last 58.40) - open/close 37492 (time 1787603245.76 total 1208.27 last 71.49) - open/close 38301 (time 1787603255.77 total 1218.28 last 80.84) - open/close 39109 (time 1787603265.77 total 1228.28 last 80.78) - open/close 39845 (time 1787603275.78 total 1238.29 last 73.54) - open/close 40676 (time 1787603285.78 total 1248.29 last 83.08) - open/close 41403 (time 1787603295.80 total 1258.31 last 72.54) - open/close 42195 (time 1787603305.81 total 1268.32 last 79.18) - open/close 42967 (time 1787603315.81 total 1278.32 last 77.15) - open/close 43825 (time 1787603325.83 total 1288.33 last 85.70) - open/close 44680 (time 1787603335.83 total 1298.34 last 85.42) - open/close 45530 (time 1787603345.87 total 1308.38 last 84.71) - open/close 46114 (time 1787603355.87 total 1318.38 last 58.38) - open/close 46742 (time 1787603365.87 total 1328.38 last 62.80) - open/close 47600 (time 1787603375.88 total 1338.39 last 85.74) - open/close 48392 (time 1787603385.88 total 1348.39 last 79.19) - open/close 49323 (time 1787603395.90 total 1358.40 last 92.97) total: 50000 open/close in 1366.88 seconds: 36.58 ops/second - unlinked 0 (time 1787603408 ; total 0 ; last 0) - unlinked 10000 (time 1787603539 ; total 131 ; last 131) - unlinked 20000 (time 1787603664 ; total 256 ; last 125) - unlinked 30000 (time 1787603782 ; total 374 ; last 118) - unlinked 40000 (time 1787603903 ; total 495 ; last 121) total: 50000 unlinks in 616 seconds: 81.168831 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg119-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg119-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg119-server stop mds service on oleg119-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg119-server unloading modules via unload_modules_local on: 'oleg119-server' oleg119-server: oleg119-server.virtnet: executing unload_modules_local oleg119-server: modules unloaded. pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-server: ssh exited with exit code 2 pdsh@oleg119-client: oleg119-client: ssh exited with exit code 2