== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 12:47:08 (1786726028) start mds service on oleg234-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing load_modules_local oleg234-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg234-server: Loading modules from /home/green/git/lustre-release/lustre oleg234-server: detected 4 online CPUs by sysfs oleg234-server: MODOPTS_LIBCFS= oleg234-server: Force libcfs to create 2 CPU partitions oleg234-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg234-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg234-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg234-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg234-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 138 (time 1786726144.69 total 10.20 last 13.53) - open/close 353 (time 1786726154.70 total 20.21 last 21.47) - open/close 468 (time 1786726166.42 total 31.93 last 9.81) - open/close 584 (time 1786726176.47 total 41.98 last 11.54) - open/close 718 (time 1786726186.53 total 52.05 last 13.32) - open/close 898 (time 1786726196.75 total 62.26 last 17.62) - open/close 1122 (time 1786726209.69 total 75.21 last 17.30) - open/close 1333 (time 1786726220.15 total 85.66 last 20.18) - open/close 1572 (time 1786726230.15 total 95.67 last 23.89) - open/close 1781 (time 1786726240.16 total 105.67 last 20.89) - open/close 1927 (time 1786726250.31 total 115.83 last 14.37) - open/close 2186 (time 1786726261.24 total 126.75 last 23.70) - open/close 2354 (time 1786726271.82 total 137.33 last 15.88) - open/close 2482 (time 1786726282.62 total 148.13 last 11.85) - open/close 2566 (time 1786726293.40 total 158.92 last 7.79) - open/close 2772 (time 1786726303.42 total 168.93 last 20.58) - open/close 2916 (time 1786726313.42 total 178.93 last 14.40) - open/close 3087 (time 1786726323.43 total 188.94 last 17.08) - open/close 3286 (time 1786726333.68 total 199.19 last 19.42) - open/close 3460 (time 1786726344.65 total 210.16 last 15.86) - open/close 3549 (time 1786726355.38 total 220.89 last 8.30) - open/close 3698 (time 1786726365.96 total 231.47 last 14.09) - open/close 3924 (time 1786726375.97 total 241.48 last 22.57) - open/close 4174 (time 1786726385.98 total 251.49 last 24.98) - open/close 4361 (time 1786726395.99 total 261.51 last 18.67) - open/close 4545 (time 1786726406.03 total 271.54 last 18.33) - open/close 4818 (time 1786726416.69 total 282.20 last 25.60) - open/close 5038 (time 1786726427.69 total 293.21 last 20.00) - open/close 5194 (time 1786726438.07 total 303.58 last 15.03) - open/close 5314 (time 1786726448.13 total 313.64 last 11.93) - open/close 5383 (time 1786726458.42 total 323.93 last 6.70) - open/close 5594 (time 1786726469.25 total 334.76 last 19.49) - open/close 5875 (time 1786726480.34 total 345.85 last 25.34) - open/close 6086 (time 1786726491.19 total 356.71 last 19.43) - open/close 6315 (time 1786726501.20 total 366.71 last 22.89) - open/close 6571 (time 1786726511.81 total 377.32 last 24.12) - open/close 6777 (time 1786726522.68 total 388.20 last 18.95) - open/close 6907 (time 1786726532.68 total 398.20 last 13.00) - open/close 7095 (time 1786726542.88 total 408.39 last 18.44) - open/close 7266 (time 1786726553.13 total 418.64 last 16.69) - open/close 7481 (time 1786726564.03 total 429.55 last 19.72) - open/close 7574 (time 1786726574.78 total 440.29 last 8.65) - open/close 7661 (time 1786726585.38 total 450.89 last 8.21) - open/close 7870 (time 1786726595.86 total 461.37 last 19.93) - open/close 8089 (time 1786726606.52 total 472.03 last 20.54) - open/close 8201 (time 1786726616.96 total 482.48 last 10.73) - open/close 8351 (time 1786726627.28 total 492.79 last 14.54) - open/close 8607 (time 1786726637.95 total 503.46 last 23.99) - open/close 8787 (time 1786726647.95 total 513.47 last 17.99) - open/close 9000 (time 1786726657.96 total 523.47 last 21.29) - open/close 9247 (time 1786726667.97 total 533.48 last 24.68) - open/close 9343 (time 1786726679.03 total 544.54 last 8.68) - open/close 9551 (time 1786726689.04 total 554.55 last 20.78) - open/close 9800 (time 1786726699.41 total 564.92 last 24.00) - open/close 9991 (time 1786726709.70 total 575.21 last 18.57) - open/close 10081 (time 1786726720.59 total 586.10 last 8.27) - open/close 10206 (time 1786726731.06 total 596.57 last 11.93) - open/close 10329 (time 1786726741.31 total 606.82 last 12.00) - open/close 10529 (time 1786726751.75 total 617.26 last 19.16) - open/close 10704 (time 1786726761.80 total 627.31 last 17.41) - open/close 10953 (time 1786726771.81 total 637.32 last 24.88) - open/close 11265 (time 1786726782.68 total 648.19 last 28.70) - open/close 11577 (time 1786726792.70 total 658.21 last 31.14) - open/close 11774 (time 1786726802.72 total 668.23 last 19.67) - open/close 11983 (time 1786726817.95 total 683.46 last 13.72) - open/close 12171 (time 1786726828.47 total 693.98 last 17.87) - open/close 12364 (time 1786726839.32 total 704.83 last 17.79) - open/close 12559 (time 1786726849.33 total 714.84 last 19.49) - open/close 12807 (time 1786726859.67 total 725.18 last 23.98) - open/close 12899 (time 1786726870.14 total 735.66 last 8.78) - open/close 13031 (time 1786726880.21 total 745.73 last 13.11) - open/close 13174 (time 1786726890.23 total 755.75 last 14.27) - open/close 13373 (time 1786726900.87 total 766.38 last 18.71) - open/close 13585 (time 1786726911.67 total 777.19 last 19.62) - open/close 13771 (time 1786726921.69 total 787.20 last 18.57) - open/close 13908 (time 1786726931.78 total 797.29 last 13.58) - open/close 14137 (time 1786726941.89 total 807.40 last 22.66) - open/close 14384 (time 1786726952.27 total 817.79 last 23.78) - open/close 14621 (time 1786726962.28 total 827.79 last 23.69) - open/close 14884 (time 1786726972.99 total 838.51 last 24.55) - open/close 15127 (time 1786726983.36 total 848.87 last 23.44) - open/close 15407 (time 1786726993.38 total 858.89 last 27.95) - open/close 15667 (time 1786727004.45 total 869.97 last 23.48) - open/close 15928 (time 1786727015.30 total 880.81 last 24.08) - open/close 16007 (time 1786727025.34 total 890.86 last 7.86) - open/close 16262 (time 1786727035.70 total 901.21 last 24.62) - open/close 16520 (time 1786727046.33 total 911.85 last 24.27) - open/close 16741 (time 1786727056.84 total 922.35 last 21.04) - open/close 16963 (time 1786727067.25 total 932.76 last 21.33) - open/close 17217 (time 1786727077.27 total 942.78 last 25.35) - open/close 17420 (time 1786727087.43 total 952.94 last 19.98) - open/close 17624 (time 1786727097.53 total 963.04 last 20.20) - open/close 17884 (time 1786727108.41 total 973.92 last 23.90) - open/close 18125 (time 1786727119.12 total 984.63 last 22.51) - open/close 18372 (time 1786727129.33 total 994.84 last 24.18) - open/close 18675 (time 1786727139.84 total 1005.35 last 28.83) - open/close 18945 (time 1786727149.88 total 1015.39 last 26.89) - open/close 19188 (time 1786727160.06 total 1025.57 last 23.87) - open/close 19944 (time 1786727170.07 total 1035.59 last 75.51) - open/close 20713 (time 1786727180.09 total 1045.60 last 76.80) - open/close 21416 (time 1786727190.11 total 1055.62 last 70.13) - open/close 22133 (time 1786727200.12 total 1065.63 last 71.65) - open/close 22795 (time 1786727210.13 total 1075.64 last 66.11) - open/close 23512 (time 1786727220.14 total 1085.65 last 71.68) - open/close 24253 (time 1786727230.15 total 1095.66 last 74.02) - open/close 25099 (time 1786727240.15 total 1105.66 last 84.58) - open/close 25927 (time 1786727250.16 total 1115.68 last 82.68) - open/close 26739 (time 1786727260.18 total 1125.69 last 81.06) - open/close 27676 (time 1786727270.19 total 1135.70 last 93.62) - open/close 28550 (time 1786727280.19 total 1145.71 last 87.36) - open/close 29446 (time 1786727290.20 total 1155.71 last 89.53) - open/close 30000 (time 1786727296.24 total 1161.76 last 91.70) - open/close 30909 (time 1786727306.25 total 1171.76 last 90.84) - open/close 31789 (time 1786727316.26 total 1181.77 last 87.94) - open/close 32624 (time 1786727326.27 total 1191.78 last 83.41) - open/close 33380 (time 1786727336.28 total 1201.79 last 75.53) - open/close 34009 (time 1786727346.28 total 1211.80 last 62.86) - open/close 34773 (time 1786727356.29 total 1221.81 last 76.32) - open/close 35603 (time 1786727366.30 total 1231.81 last 82.96) - open/close 36360 (time 1786727376.31 total 1241.82 last 75.62) - open/close 37157 (time 1786727386.31 total 1251.82 last 79.69) - open/close 37844 (time 1786727396.33 total 1261.84 last 68.57) - open/close 38605 (time 1786727406.34 total 1271.85 last 76.06) - open/close 39366 (time 1786727416.34 total 1281.86 last 76.03) - open/close 40000 (time 1786727424.83 total 1290.34 last 74.74) - open/close 40820 (time 1786727434.85 total 1300.36 last 81.82) - open/close 41711 (time 1786727444.86 total 1310.37 last 89.03) - open/close 42513 (time 1786727454.87 total 1320.38 last 80.13) - open/close 43144 (time 1786727464.89 total 1330.40 last 62.96) - open/close 43836 (time 1786727474.89 total 1340.40 last 69.16) - open/close 44601 (time 1786727484.91 total 1350.42 last 76.38) - open/close 45268 (time 1786727494.92 total 1360.43 last 66.65) - open/close 46000 (time 1786727504.92 total 1370.44 last 73.15) - open/close 46820 (time 1786727514.93 total 1380.44 last 81.96) - open/close 47532 (time 1786727524.93 total 1390.44 last 71.18) - open/close 48130 (time 1786727534.93 total 1400.45 last 59.79) - open/close 48917 (time 1786727544.93 total 1410.45 last 78.70) - open/close 49510 (time 1786727554.94 total 1420.45 last 59.28) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1421.92 seconds: 34.90 ops/second - unlinked 0 (time 1786727561 ; total 0 ; last 0) - unlinked 10000 (time 1786727693 ; total 132 ; last 132) - unlinked 20000 (time 1786727826 ; total 265 ; last 133) - unlinked 30000 (time 1786727972 ; total 411 ; last 146) - unlinked 40000 (time 1786728102 ; total 541 ; last 130) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 679 seconds: 73.094254 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg234-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg234-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg234-server stop mds service on oleg234-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg234-server unloading modules via unload_modules_local on: 'oleg234-server' oleg234-server: oleg234-server.virtnet: executing unload_modules_local oleg234-server: modules unloaded. pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-server: ssh exited with exit code 2 pdsh@oleg234-client: oleg234-client: ssh exited with exit code 2