== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:43:59 (1787582639) start mds service on oleg631-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg631-server' oleg631-server: oleg631-server.virtnet: executing load_modules_local oleg631-server: Loading modules from /home/green/git/lustre-release/lustre oleg631-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg631-server: detected 4 online CPUs by sysfs oleg631-server: MODOPTS_LIBCFS= oleg631-server: Force libcfs to create 2 CPU partitions oleg631-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg631-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg631-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg631-server: oleg631-server.virtnet: executing set_default_debug -1 all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg631-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg631-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg631-server: oleg631-server.virtnet: executing set_default_debug -1 all pdsh@oleg631-client: oleg631-server: ssh exited with exit code 1 Started lustre-OST0000 oleg631-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg631-client.virtnet: -o user_xattr,flock 192.168.206.131@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 105 (time 1787582727.54 total 10.02 last 10.48) - open/close 168 (time 1787582737.61 total 20.09 last 6.26) - open/close 294 (time 1787582749.76 total 32.23 last 10.38) - open/close 491 (time 1787582760.04 total 42.52 last 19.16) - open/close 665 (time 1787582770.61 total 53.09 last 16.46) - open/close 924 (time 1787582780.63 total 63.10 last 25.86) - open/close 1160 (time 1787582791.49 total 73.97 last 21.73) - open/close 1378 (time 1787582802.17 total 84.65 last 20.41) - open/close 1628 (time 1787582812.84 total 95.32 last 23.43) - open/close 1857 (time 1787582823.35 total 105.83 last 21.79) - open/close 2105 (time 1787582833.36 total 115.83 last 24.79) - open/close 2270 (time 1787582844.42 total 126.90 last 14.91) - open/close 2401 (time 1787582854.43 total 136.91 last 13.09) - open/close 2561 (time 1787582864.45 total 146.93 last 15.97) - open/close 2678 (time 1787582874.47 total 156.94 last 11.68) - open/close 2858 (time 1787582884.48 total 166.96 last 17.97) - open/close 3073 (time 1787582894.49 total 176.96 last 21.48) - open/close 3252 (time 1787582904.49 total 186.97 last 17.89) - open/close 3435 (time 1787582914.82 total 197.30 last 17.71) - open/close 3509 (time 1787582924.84 total 207.31 last 7.39) - open/close 3597 (time 1787582935.16 total 217.64 last 8.53) - open/close 3793 (time 1787582945.80 total 228.28 last 18.42) - open/close 3974 (time 1787582955.89 total 238.37 last 17.94) - open/close 4183 (time 1787582966.28 total 248.75 last 20.13) - open/close 4454 (time 1787582976.43 total 258.91 last 26.68) - open/close 4671 (time 1787582986.68 total 269.15 last 21.19) - open/close 4918 (time 1787582996.68 total 279.16 last 24.69) - open/close 5189 (time 1787583006.68 total 289.16 last 27.09) - open/close 5395 (time 1787583017.26 total 299.74 last 19.48) - open/close 5672 (time 1787583027.26 total 309.74 last 27.69) - open/close 5955 (time 1787583039.80 total 322.28 last 22.57) - open/close 6155 (time 1787583049.81 total 332.29 last 19.98) - open/close 6367 (time 1787583060.16 total 342.64 last 20.49) - open/close 6575 (time 1787583071.09 total 353.57 last 19.03) - open/close 6653 (time 1787583081.41 total 363.89 last 7.56) - open/close 6817 (time 1787583091.46 total 373.94 last 16.32) - open/close 6997 (time 1787583101.48 total 383.96 last 17.96) - open/close 7078 (time 1787583111.54 total 394.01 last 8.06) - open/close 7324 (time 1787583122.62 total 405.10 last 22.19) - open/close 7592 (time 1787583133.42 total 415.90 last 24.82) - open/close 7824 (time 1787583143.99 total 426.47 last 21.95) - open/close 8074 (time 1787583154.68 total 437.16 last 23.38) - open/close 8272 (time 1787583165.49 total 447.97 last 18.32) - open/close 8475 (time 1787583175.50 total 457.98 last 20.27) - open/close 8707 (time 1787583186.50 total 468.98 last 21.10) - open/close 8840 (time 1787583196.81 total 479.29 last 12.89) - open/close 9042 (time 1787583206.82 total 489.30 last 20.18) - open/close 9239 (time 1787583216.85 total 499.32 last 19.66) - open/close 9483 (time 1787583227.07 total 509.54 last 23.87) - open/close 9666 (time 1787583237.68 total 520.16 last 17.24) - open/close 9860 (time 1787583248.62 total 531.10 last 17.74) - open/close 10000 (time 1787583254.40 total 536.88 last 24.21) - open/close 10193 (time 1787583264.41 total 546.89 last 19.29) - open/close 10412 (time 1787583274.41 total 556.89 last 21.89) - open/close 10519 (time 1787583284.73 total 567.20 last 10.37) - open/close 10755 (time 1787583295.62 total 578.10 last 21.66) - open/close 10987 (time 1787583308.21 total 590.69 last 18.43) - open/close 11182 (time 1787583320.32 total 602.80 last 16.10) - open/close 11402 (time 1787583330.35 total 612.82 last 21.95) - open/close 11644 (time 1787583340.36 total 622.84 last 24.16) - open/close 11800 (time 1787583350.37 total 632.85 last 15.59) - open/close 12055 (time 1787583361.15 total 643.62 last 23.66) - open/close 12258 (time 1787583371.86 total 654.34 last 18.94) - open/close 12509 (time 1787583381.87 total 664.35 last 25.08) - open/close 12714 (time 1787583392.89 total 675.37 last 18.60) - open/close 12889 (time 1787583402.90 total 685.38 last 17.48) - open/close 13137 (time 1787583413.89 total 696.36 last 22.57) - open/close 13399 (time 1787583423.89 total 706.37 last 26.19) - open/close 13667 (time 1787583434.40 total 716.88 last 25.49) - open/close 13907 (time 1787583444.42 total 726.90 last 23.94) - open/close 13980 (time 1787583454.46 total 736.94 last 7.27) - open/close 14164 (time 1787583464.47 total 746.95 last 18.38) - open/close 14430 (time 1787583477.32 total 759.79 last 20.71) - open/close 14562 (time 1787583488.17 total 770.65 last 12.16) - open/close 14786 (time 1787583499.19 total 781.67 last 20.33) - open/close 15049 (time 1787583510.02 total 792.49 last 24.29) - open/close 15295 (time 1787583520.02 total 802.50 last 24.58) - open/close 15358 (time 1787583530.03 total 812.51 last 6.30) - open/close 15423 (time 1787583540.08 total 822.56 last 6.47) - open/close 15544 (time 1787583550.19 total 832.67 last 11.97) - open/close 15801 (time 1787583560.44 total 842.91 last 25.09) - open/close 16027 (time 1787583570.63 total 853.11 last 22.16) - open/close 16281 (time 1787583581.55 total 864.03 last 23.26) - open/close 16507 (time 1787583592.11 total 874.58 last 21.42) - open/close 16763 (time 1787583602.17 total 884.64 last 25.44) - open/close 16994 (time 1787583612.17 total 894.65 last 23.09) - open/close 17179 (time 1787583622.19 total 904.67 last 18.46) - open/close 17402 (time 1787583632.77 total 915.24 last 21.09) - open/close 17626 (time 1787583643.33 total 925.81 last 21.21) - open/close 17763 (time 1787583654.27 total 936.74 last 12.52) - open/close 18023 (time 1787583667.39 total 949.87 last 19.81) - open/close 18199 (time 1787583677.40 total 959.88 last 17.58) - open/close 18383 (time 1787583688.17 total 970.65 last 17.08) - open/close 18606 (time 1787583698.18 total 980.66 last 22.28) - open/close 19327 (time 1787583708.19 total 990.66 last 72.05) - open/close 20000 (time 1787583718.17 total 1000.65 last 67.39) - open/close 20607 (time 1787583728.19 total 1010.66 last 60.63) - open/close 21263 (time 1787583738.19 total 1020.66 last 65.59) - open/close 21946 (time 1787583748.19 total 1030.66 last 68.30) - open/close 22577 (time 1787583758.20 total 1040.67 last 63.05) - open/close 23270 (time 1787583768.20 total 1050.68 last 69.27) - open/close 23948 (time 1787583778.23 total 1060.71 last 67.60) - open/close 24630 (time 1787583788.23 total 1070.71 last 68.18) - open/close 25360 (time 1787583798.24 total 1080.72 last 72.96) - open/close 26042 (time 1787583808.24 total 1090.72 last 68.19) - open/close 26613 (time 1787583818.24 total 1100.72 last 57.09) - open/close 27200 (time 1787583828.25 total 1110.73 last 58.65) - open/close 27857 (time 1787583838.26 total 1120.74 last 65.62) - open/close 28433 (time 1787583848.27 total 1130.74 last 57.57) - open/close 29014 (time 1787583858.28 total 1140.75 last 58.05) - open/close 29753 (time 1787583868.28 total 1150.75 last 73.89) - open/close 30000 (time 1787583871.67 total 1154.14 last 72.85) - open/close 30764 (time 1787583881.67 total 1164.15 last 76.37) - open/close 31487 (time 1787583891.67 total 1174.15 last 72.30) - open/close 32284 (time 1787583901.69 total 1184.16 last 79.58) - open/close 33022 (time 1787583911.69 total 1194.17 last 73.75) - open/close 33760 (time 1787583921.70 total 1204.18 last 73.72) - open/close 34463 (time 1787583931.71 total 1214.18 last 70.28) - open/close 35214 (time 1787583941.72 total 1224.19 last 75.04) - open/close 35919 (time 1787583951.72 total 1234.20 last 70.48) - open/close 36633 (time 1787583961.72 total 1244.20 last 71.39) - open/close 37394 (time 1787583971.73 total 1254.21 last 76.03) - open/close 38145 (time 1787583981.74 total 1264.21 last 75.04) - open/close 38875 (time 1787583991.75 total 1274.22 last 72.94) - open/close 39556 (time 1787584001.76 total 1284.23 last 68.02) - open/close 40000 (time 1787584007.87 total 1290.34 last 72.67) - open/close 40717 (time 1787584017.87 total 1300.35 last 71.66) - open/close 41400 (time 1787584027.88 total 1310.36 last 68.25) - open/close 42125 (time 1787584037.89 total 1320.36 last 72.45) - open/close 42835 (time 1787584047.90 total 1330.37 last 70.93) - open/close 43557 (time 1787584057.91 total 1340.38 last 72.14) - open/close 44280 (time 1787584067.91 total 1350.39 last 72.25) - open/close 45046 (time 1787584077.92 total 1360.40 last 76.53) - open/close 45760 (time 1787584087.93 total 1370.41 last 71.34) - open/close 46404 (time 1787584097.93 total 1380.41 last 64.39) - open/close 47039 (time 1787584107.94 total 1390.41 last 63.46) - open/close 47618 (time 1787584117.95 total 1400.42 last 57.85) - open/close 48205 (time 1787584127.95 total 1410.43 last 58.68) - open/close 48825 (time 1787584137.96 total 1420.44 last 61.95) - open/close 49490 (time 1787584147.96 total 1430.44 last 66.49) total: 50000 open/close in 1437.65 seconds: 34.78 ops/second - unlinked 0 (time 1787584159 ; total 0 ; last 0) - unlinked 10000 (time 1787584306 ; total 147 ; last 147) - unlinked 20000 (time 1787584463 ; total 304 ; last 157) - unlinked 30000 (time 1787584615 ; total 456 ; last 152) - unlinked 40000 (time 1787584767 ; total 608 ; last 152) total: 50000 unlinks in 759 seconds: 65.876152 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg631-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg631-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg631-server stop mds service on oleg631-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg631-server unloading modules via unload_modules_local on: 'oleg631-server' oleg631-server: oleg631-server.virtnet: executing unload_modules_local oleg631-server: modules unloaded. pdsh@oleg631-client: oleg631-server: ssh exited with exit code 2 pdsh@oleg631-client: oleg631-client: ssh exited with exit code 2 pdsh@oleg631-client: oleg631-server: ssh exited with exit code 2 pdsh@oleg631-client: oleg631-client: ssh exited with exit code 2