== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:02:55 (1786651375) start mds service on oleg130-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' llite/lustre options: 'enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg130-server' oleg130-server: oleg130-server.virtnet: executing load_modules_local oleg130-server: Loading modules from /home/green/git/lustre-release/lustre oleg130-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg130-server: detected 4 online CPUs by sysfs oleg130-server: MODOPTS_LIBCFS= oleg130-server: Force libcfs to create 2 CPU partitions oleg130-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg130-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg130-server: mdt/mdt options: 'mdt_enable_flr_ec=1' oleg130-server: llite/lustre options: 'enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg130-server: oleg130-server.virtnet: executing set_default_debug -1 all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg130-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg130-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg130-server: oleg130-server.virtnet: executing set_default_debug -1 all pdsh@oleg130-client: oleg130-server: ssh exited with exit code 1 Started lustre-OST0000 oleg130-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg130-client.virtnet: -o user_xattr,flock 192.168.201.130@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 130 (time 1786651480.65 total 10.70 last 12.15) - open/close 356 (time 1786651490.66 total 20.70 last 22.59) - open/close 505 (time 1786651500.90 total 30.94 last 14.56) - open/close 617 (time 1786651511.90 total 41.94 last 10.18) - open/close 848 (time 1786651521.91 total 51.95 last 23.08) - open/close 1055 (time 1786651532.20 total 62.24 last 20.11) - open/close 1259 (time 1786651542.86 total 72.91 last 19.12) - open/close 1521 (time 1786651553.74 total 83.79 last 24.08) - open/close 1683 (time 1786651563.76 total 93.80 last 16.18) - open/close 1874 (time 1786651574.45 total 104.50 last 17.86) - open/close 2053 (time 1786651584.47 total 114.51 last 17.87) - open/close 2231 (time 1786651594.96 total 125.00 last 16.97) - open/close 2464 (time 1786651604.97 total 135.02 last 23.28) - open/close 2606 (time 1786651615.58 total 145.62 last 13.39) - open/close 2720 (time 1786651625.59 total 155.63 last 11.39) - open/close 2839 (time 1786651635.60 total 165.65 last 11.88) - open/close 3064 (time 1786651645.61 total 175.66 last 22.48) - open/close 3310 (time 1786651655.62 total 185.66 last 24.58) - open/close 3421 (time 1786651666.64 total 196.69 last 10.07) - open/close 3504 (time 1786651676.88 total 206.92 last 8.11) - open/close 3701 (time 1786651689.76 total 219.80 last 15.30) - open/close 3764 (time 1786651699.76 total 229.81 last 6.29) - open/close 4029 (time 1786651710.68 total 240.72 last 24.29) - open/close 4186 (time 1786651720.68 total 250.72 last 15.70) - open/close 4438 (time 1786651730.91 total 260.95 last 24.64) - open/close 4744 (time 1786651741.65 total 271.69 last 28.49) - open/close 4956 (time 1786651751.65 total 281.69 last 21.19) - open/close 5047 (time 1786651761.93 total 291.97 last 8.85) - open/close 5163 (time 1786651773.99 total 304.03 last 9.62) - open/close 5244 (time 1786651783.99 total 314.04 last 8.10) - open/close 5493 (time 1786651794.71 total 324.76 last 23.23) - open/close 5636 (time 1786651804.71 total 334.76 last 14.30) - open/close 5707 (time 1786651814.74 total 344.79 last 7.08) - open/close 5851 (time 1786651824.74 total 354.79 last 14.40) - open/close 6076 (time 1786651834.75 total 364.79 last 22.49) - open/close 6234 (time 1786651845.39 total 375.43 last 14.85) - open/close 6478 (time 1786651856.28 total 386.32 last 22.40) - open/close 6658 (time 1786651866.30 total 396.34 last 17.97) - open/close 6853 (time 1786651876.30 total 406.35 last 19.49) - open/close 7034 (time 1786651887.07 total 417.12 last 16.81) - open/close 7212 (time 1786651897.07 total 427.12 last 17.80) - open/close 7457 (time 1786651907.09 total 437.13 last 24.46) - open/close 7626 (time 1786651917.46 total 447.50 last 16.29) - open/close 7880 (time 1786651927.46 total 457.51 last 25.39) - open/close 8091 (time 1786651937.68 total 467.72 last 20.66) - open/close 8320 (time 1786651947.68 total 477.73 last 22.89) - open/close 8550 (time 1786651958.75 total 488.80 last 20.78) - open/close 8694 (time 1786651969.10 total 499.14 last 13.92) - open/close 8816 (time 1786651979.12 total 509.16 last 12.17) - open/close 8912 (time 1786651990.68 total 520.73 last 8.30) - open/close 9061 (time 1786652000.91 total 530.95 last 14.57) - open/close 9224 (time 1786652010.92 total 540.97 last 16.27) - open/close 9453 (time 1786652020.95 total 550.99 last 22.85) - open/close 9633 (time 1786652030.95 total 561.00 last 17.98) - open/close 9779 (time 1786652041.43 total 571.47 last 13.94) - open/close 9917 (time 1786652051.43 total 581.48 last 13.79) - open/close 9978 (time 1786652062.24 total 592.28 last 5.65) - open/close 10222 (time 1786652072.27 total 602.31 last 24.33) - open/close 10411 (time 1786652083.28 total 613.32 last 17.17) - open/close 10644 (time 1786652093.86 total 623.90 last 22.02) - open/close 10748 (time 1786652104.93 total 634.98 last 9.39) - open/close 10895 (time 1786652115.73 total 645.77 last 13.62) - open/close 11115 (time 1786652126.35 total 656.40 last 20.70) - open/close 11285 (time 1786652138.85 total 668.90 last 13.60) - open/close 11498 (time 1786652149.98 total 680.02 last 19.15) - open/close 11735 (time 1786652159.99 total 690.03 last 23.68) - open/close 11950 (time 1786652171.03 total 701.08 last 19.47) - open/close 12195 (time 1786652181.03 total 711.08 last 24.50) - open/close 12429 (time 1786652191.63 total 721.67 last 22.09) - open/close 12616 (time 1786652201.67 total 731.72 last 18.61) - open/close 12825 (time 1786652212.88 total 742.93 last 18.64) - open/close 13055 (time 1786652222.90 total 752.95 last 22.95) - open/close 13124 (time 1786652234.07 total 764.12 last 6.18) - open/close 13226 (time 1786652244.11 total 774.16 last 10.16) - open/close 13423 (time 1786652254.12 total 784.17 last 19.68) - open/close 13617 (time 1786652264.14 total 794.18 last 19.37) - open/close 13855 (time 1786652274.77 total 804.81 last 22.39) - open/close 14087 (time 1786652285.45 total 815.50 last 21.72) - open/close 14279 (time 1786652295.60 total 825.64 last 18.93) - open/close 14574 (time 1786652306.45 total 836.50 last 27.17) - open/close 14737 (time 1786652316.56 total 846.60 last 16.13) - open/close 14913 (time 1786652326.57 total 856.62 last 17.58) - open/close 15083 (time 1786652337.20 total 867.24 last 16.00) - open/close 15278 (time 1786652350.12 total 880.17 last 15.08) - open/close 15395 (time 1786652360.13 total 890.18 last 11.69) - open/close 15606 (time 1786652370.25 total 900.29 last 20.86) - open/close 15845 (time 1786652380.26 total 910.31 last 23.86) - open/close 16105 (time 1786652390.99 total 921.03 last 24.24) - open/close 16399 (time 1786652401.01 total 931.05 last 29.34) - open/close 16656 (time 1786652411.02 total 941.07 last 25.66) - open/close 16921 (time 1786652423.31 total 953.36 last 21.56) - open/close 17113 (time 1786652433.36 total 963.40 last 19.12) - open/close 17344 (time 1786652443.95 total 973.99 last 21.81) - open/close 17500 (time 1786652454.43 total 984.47 last 14.89) - open/close 17744 (time 1786652464.45 total 994.49 last 24.35) - open/close 17998 (time 1786652474.46 total 1004.51 last 25.36) - open/close 18373 (time 1786652484.48 total 1014.53 last 37.41) - open/close 19243 (time 1786652494.49 total 1024.53 last 86.95) - open/close 19953 (time 1786652504.52 total 1034.57 last 70.75) - open/close 20888 (time 1786652514.53 total 1044.57 last 93.46) - open/close 21658 (time 1786652524.53 total 1054.58 last 76.96) - open/close 22602 (time 1786652534.53 total 1064.58 last 94.40) - open/close 23545 (time 1786652544.54 total 1074.58 last 94.25) - open/close 24065 (time 1786652554.54 total 1084.59 last 51.98) - open/close 24777 (time 1786652564.56 total 1094.61 last 71.06) - open/close 25545 (time 1786652574.58 total 1104.62 last 76.71) - open/close 26364 (time 1786652584.58 total 1114.63 last 81.86) - open/close 27153 (time 1786652594.59 total 1124.63 last 78.85) - open/close 27850 (time 1786652604.59 total 1134.64 last 69.66) - open/close 28525 (time 1786652614.61 total 1144.66 last 67.37) - open/close 29410 (time 1786652624.62 total 1154.66 last 88.47) - open/close 30000 (time 1786652631.73 total 1161.78 last 82.91) - open/close 30649 (time 1786652641.74 total 1171.78 last 64.85) - open/close 31498 (time 1786652651.74 total 1181.79 last 84.86) - open/close 32419 (time 1786652661.75 total 1191.80 last 92.03) - open/close 33091 (time 1786652671.76 total 1201.80 last 67.15) - open/close 33995 (time 1786652681.77 total 1211.81 last 90.32) - open/close 34827 (time 1786652691.77 total 1221.82 last 83.16) - open/close 35676 (time 1786652701.78 total 1231.82 last 84.88) - open/close 36552 (time 1786652711.79 total 1241.83 last 87.48) - open/close 37451 (time 1786652721.80 total 1251.84 last 89.81) - open/close 38286 (time 1786652731.81 total 1261.85 last 83.45) - open/close 38987 (time 1786652741.82 total 1271.86 last 70.02) - open/close 39874 (time 1786652751.82 total 1281.86 last 88.69) - open/close 40789 (time 1786652761.82 total 1291.86 last 91.49) - open/close 41578 (time 1786652771.82 total 1301.87 last 78.86) - open/close 42273 (time 1786652781.83 total 1311.87 last 69.49) - open/close 42957 (time 1786652791.83 total 1321.87 last 68.38) - open/close 43879 (time 1786652801.83 total 1331.87 last 92.18) - open/close 44803 (time 1786652811.84 total 1341.89 last 92.30) - open/close 45688 (time 1786652821.84 total 1351.89 last 88.49) - open/close 46576 (time 1786652831.85 total 1361.89 last 88.74) - open/close 47345 (time 1786652841.85 total 1371.90 last 76.88) - open/close 48200 (time 1786652851.85 total 1381.90 last 85.48) - open/close 49070 (time 1786652861.86 total 1391.90 last 86.98) - open/close 49911 (time 1786652871.86 total 1401.90 last 84.09) total: 50000 open/close in 1402.75 seconds: 35.64 ops/second - unlinked 0 (time 1786652876 ; total 0 ; last 0) - unlinked 10000 (time 1786652999 ; total 123 ; last 123) - unlinked 20000 (time 1786653125 ; total 249 ; last 126) - unlinked 30000 (time 1786653245 ; total 369 ; last 120) - unlinked 40000 (time 1786653361 ; total 485 ; last 116) total: 50000 unlinks in 611 seconds: 81.833061 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg130-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg130-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg130-server stop mds service on oleg130-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg130-server unloading modules via unload_modules_local on: 'oleg130-server' oleg130-server: oleg130-server.virtnet: executing unload_modules_local oleg130-server: modules unloaded. pdsh@oleg130-client: oleg130-client: ssh exited with exit code 2 pdsh@oleg130-client: oleg130-server: ssh exited with exit code 2 pdsh@oleg130-client: oleg130-client: ssh exited with exit code 2 pdsh@oleg130-client: oleg130-server: ssh exited with exit code 2