== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:43:20 (1786405400) start mds service on oleg435-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg435-server: detected 4 online CPUs by sysfs oleg435-server: MODOPTS_LIBCFS= oleg435-server: Force libcfs to create 2 CPU partitions oleg435-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg435-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg435-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg435-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg435-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 115 (time 1786405527.94 total 10.75 last 10.69) - open/close 255 (time 1786405537.95 total 20.76 last 13.99) - open/close 500 (time 1786405547.95 total 30.77 last 24.50) - open/close 772 (time 1786405558.92 total 41.74 last 24.79) - open/close 1001 (time 1786405568.99 total 51.80 last 22.75) - open/close 1245 (time 1786405579.00 total 61.81 last 24.38) - open/close 1407 (time 1786405589.85 total 72.66 last 14.93) - open/close 1603 (time 1786405599.86 total 82.68 last 19.58) - open/close 1862 (time 1786405610.03 total 92.85 last 25.47) - open/close 2133 (time 1786405620.51 total 103.32 last 25.87) - open/close 2284 (time 1786405631.04 total 113.85 last 14.34) - open/close 2420 (time 1786405641.38 total 124.19 last 13.15) - open/close 2585 (time 1786405651.43 total 134.24 last 16.42) - open/close 2782 (time 1786405662.39 total 145.21 last 17.96) - open/close 2903 (time 1786405672.41 total 155.23 last 12.08) - open/close 3063 (time 1786405683.45 total 166.27 last 14.49) - open/close 3237 (time 1786405693.52 total 176.33 last 17.29) - open/close 3356 (time 1786405703.93 total 186.74 last 11.43) - open/close 3453 (time 1786405713.93 total 196.75 last 9.69) - open/close 3616 (time 1786405723.94 total 206.75 last 16.29) - open/close 3666 (time 1786405734.64 total 217.46 last 4.67) - open/close 3798 (time 1786405745.01 total 227.82 last 12.73) - open/close 3951 (time 1786405755.76 total 238.58 last 14.22) - open/close 4030 (time 1786405766.26 total 249.07 last 7.53) - open/close 4159 (time 1786405776.27 total 259.08 last 12.89) - open/close 4282 (time 1786405786.99 total 269.80 last 11.48) - open/close 4445 (time 1786405797.33 total 280.14 last 15.76) - open/close 4609 (time 1786405807.35 total 290.16 last 16.37) - open/close 4853 (time 1786405817.41 total 300.23 last 24.25) - open/close 5046 (time 1786405827.42 total 310.23 last 19.29) - open/close 5296 (time 1786405837.44 total 320.25 last 24.95) - open/close 5515 (time 1786405847.81 total 330.63 last 21.11) - open/close 5675 (time 1786405858.40 total 341.21 last 15.11) - open/close 5776 (time 1786405869.16 total 351.98 last 9.38) - open/close 5884 (time 1786405880.24 total 363.05 last 9.75) - open/close 6079 (time 1786405891.05 total 373.86 last 18.04) - open/close 6189 (time 1786405903.29 total 386.10 last 8.99) - open/close 6293 (time 1786405914.31 total 397.13 last 9.43) - open/close 6485 (time 1786405924.93 total 407.75 last 18.08) - open/close 6615 (time 1786405936.77 total 419.58 last 10.99) - open/close 6791 (time 1786405946.87 total 429.68 last 17.43) - open/close 6985 (time 1786405957.19 total 440.00 last 18.80) - open/close 7152 (time 1786405967.22 total 450.03 last 16.64) - open/close 7261 (time 1786405977.73 total 460.54 last 10.37) - open/close 7408 (time 1786405988.36 total 471.17 last 13.83) - open/close 7513 (time 1786405998.51 total 481.33 last 10.34) - open/close 7601 (time 1786406009.16 total 491.97 last 8.27) - open/close 7756 (time 1786406020.52 total 503.33 last 13.64) - open/close 7941 (time 1786406033.58 total 516.39 last 14.17) - open/close 8163 (time 1786406043.95 total 526.77 last 21.40) - open/close 8344 (time 1786406053.96 total 536.78 last 18.08) - open/close 8509 (time 1786406064.94 total 547.76 last 15.03) - open/close 8619 (time 1786406074.96 total 557.77 last 10.99) - open/close 8796 (time 1786406085.12 total 567.93 last 17.42) - open/close 9017 (time 1786406096.09 total 578.91 last 20.14) - open/close 9129 (time 1786406106.33 total 589.14 last 10.94) - open/close 9324 (time 1786406116.94 total 599.76 last 18.37) - open/close 9515 (time 1786406127.01 total 609.82 last 18.98) - open/close 9665 (time 1786406137.20 total 620.02 last 14.72) - open/close 9825 (time 1786406147.25 total 630.07 last 15.92) - open/close 10000 (time 1786406157.09 total 639.91 last 17.78) - open/close 10125 (time 1786406167.85 total 650.67 last 11.62) - open/close 10297 (time 1786406177.86 total 660.67 last 17.19) - open/close 10386 (time 1786406188.15 total 670.97 last 8.64) - open/close 10583 (time 1786406201.07 total 683.88 last 15.26) - open/close 10733 (time 1786406211.09 total 693.91 last 14.96) - open/close 10846 (time 1786406221.30 total 704.12 last 11.07) - open/close 10932 (time 1786406231.30 total 714.12 last 8.60) - open/close 11063 (time 1786406241.76 total 724.58 last 12.52) - open/close 11183 (time 1786406251.78 total 734.59 last 11.98) - open/close 11413 (time 1786406262.84 total 745.66 last 20.78) - open/close 11562 (time 1786406273.72 total 756.53 last 13.71) - open/close 11683 (time 1786406283.74 total 766.56 last 12.07) - open/close 11781 (time 1786406293.88 total 776.69 last 9.67) - open/close 11870 (time 1786406304.69 total 787.50 last 8.23) - open/close 12037 (time 1786406314.75 total 797.57 last 16.59) - open/close 12131 (time 1786406327.72 total 810.54 last 7.25) - open/close 12321 (time 1786406338.67 total 821.49 last 17.35) - open/close 12432 (time 1786406351.67 total 834.48 last 8.54) - open/close 12473 (time 1786406361.91 total 844.72 last 4.00) - open/close 12595 (time 1786406372.78 total 855.59 last 11.22) - open/close 12811 (time 1786406383.21 total 866.03 last 20.69) - open/close 12915 (time 1786406393.33 total 876.14 last 10.28) - open/close 13021 (time 1786406403.37 total 886.18 last 10.56) - open/close 13156 (time 1786406413.95 total 896.77 last 12.75) - open/close 13299 (time 1786406424.48 total 907.30 last 13.58) - open/close 13513 (time 1786406437.32 total 920.13 last 16.67) - open/close 13756 (time 1786406448.83 total 931.65 last 21.10) - open/close 13933 (time 1786406459.38 total 942.19 last 16.79) - open/close 14093 (time 1786406470.14 total 952.95 last 14.87) - open/close 14250 (time 1786406480.85 total 963.67 last 14.65) - open/close 14374 (time 1786406491.75 total 974.56 last 11.38) - open/close 14533 (time 1786406502.57 total 985.38 last 14.70) - open/close 14645 (time 1786406513.01 total 995.83 last 10.73) - open/close 14752 (time 1786406523.27 total 1006.08 last 10.43) - open/close 14878 (time 1786406533.32 total 1016.14 last 12.53) - open/close 15037 (time 1786406543.33 total 1026.15 last 15.89) - open/close 15220 (time 1786406554.16 total 1036.97 last 16.90) - open/close 15327 (time 1786406564.19 total 1047.00 last 10.67) - open/close 15495 (time 1786406574.22 total 1057.03 last 16.75) - open/close 15556 (time 1786406584.24 total 1067.06 last 6.09) - open/close 15752 (time 1786406596.84 total 1079.66 last 15.55) - open/close 15886 (time 1786406607.16 total 1089.97 last 12.99) - open/close 16020 (time 1786406617.18 total 1100.00 last 13.37) - open/close 16152 (time 1786406627.56 total 1110.38 last 12.72) - open/close 16259 (time 1786406637.58 total 1120.39 last 10.68) - open/close 16390 (time 1786406647.60 total 1130.41 last 13.08) - open/close 16549 (time 1786406657.63 total 1140.44 last 15.85) - open/close 16719 (time 1786406667.64 total 1150.46 last 16.98) - open/close 16859 (time 1786406678.39 total 1161.21 last 13.02) - open/close 17091 (time 1786406688.41 total 1171.22 last 23.17) - open/close 17400 (time 1786406698.43 total 1181.25 last 30.82) - open/close 18157 (time 1786406708.44 total 1191.26 last 75.65) - open/close 18916 (time 1786406718.45 total 1201.27 last 75.79) - open/close 19734 (time 1786406728.46 total 1211.27 last 81.77) - open/close 20000 (time 1786406731.65 total 1214.46 last 83.46) - open/close 20855 (time 1786406741.66 total 1224.47 last 85.39) - open/close 21695 (time 1786406751.67 total 1234.49 last 83.90) - open/close 22463 (time 1786406761.68 total 1244.49 last 76.76) - open/close 23251 (time 1786406771.69 total 1254.50 last 78.73) - open/close 24050 (time 1786406781.70 total 1264.51 last 79.79) - open/close 24892 (time 1786406791.71 total 1274.52 last 84.12) - open/close 25771 (time 1786406801.71 total 1284.53 last 87.88) - open/close 26586 (time 1786406811.71 total 1294.53 last 81.49) - open/close 27458 (time 1786406821.71 total 1304.53 last 87.19) - open/close 28324 (time 1786406831.72 total 1314.53 last 86.58) - open/close 29066 (time 1786406841.72 total 1324.53 last 74.18) - open/close 29884 (time 1786406851.72 total 1334.54 last 81.75) - open/close 30809 (time 1786406861.73 total 1344.55 last 92.45) - open/close 31719 (time 1786406871.73 total 1354.55 last 90.98) - open/close 32604 (time 1786406881.74 total 1364.55 last 88.44) - open/close 33495 (time 1786406891.74 total 1374.56 last 89.08) - open/close 34340 (time 1786406901.76 total 1384.57 last 84.35) - open/close 35076 (time 1786406911.76 total 1394.58 last 73.57) - open/close 35876 (time 1786406921.77 total 1404.59 last 79.95) - open/close 36758 (time 1786406931.78 total 1414.59 last 88.11) - open/close 37501 (time 1786406941.79 total 1424.60 last 74.25) - open/close 38251 (time 1786406951.79 total 1434.60 last 74.97) - open/close 39107 (time 1786406961.80 total 1444.61 last 85.53) - open/close 40000 (time 1786406971.57 total 1454.39 last 91.35) - open/close 40833 (time 1786406981.57 total 1464.39 last 83.30) - open/close 41788 (time 1786406991.58 total 1474.39 last 95.46) - open/close 42632 (time 1786407001.59 total 1484.40 last 84.30) - open/close 43427 (time 1786407011.60 total 1494.41 last 79.43) - open/close 44131 (time 1786407021.60 total 1504.42 last 70.38) - open/close 44781 (time 1786407031.61 total 1514.43 last 64.94) - open/close 45531 (time 1786407041.63 total 1524.45 last 74.84) - open/close 46336 (time 1786407051.64 total 1534.45 last 80.48) - open/close 47179 (time 1786407061.64 total 1544.45 last 84.27) - open/close 47985 (time 1786407071.64 total 1554.46 last 80.55) - open/close 48814 (time 1786407081.65 total 1564.47 last 82.85) - open/close 49457 (time 1786407091.66 total 1574.47 last 64.24) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1576.40 seconds: 31.48 ops/second - unlinked 0 (time 1786407098 ; total 0 ; last 0) - unlinked 10000 (time 1786407246 ; total 148 ; last 148) - unlinked 20000 (time 1786407380 ; total 282 ; last 134) - unlinked 30000 (time 1786407518 ; total 420 ; last 138) - unlinked 40000 (time 1786407652 ; total 554 ; last 134) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 674 seconds: 73.636497 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg435-server unloading modules via unload_modules_local on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing unload_modules_local oleg435-server: modules unloaded. pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2