== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:00:44 (1787796044) start mds service on oleg327-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing load_modules_local oleg327-server: Loading modules from /home/green/git/lustre-release/lustre oleg327-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg327-server: detected 4 online CPUs by sysfs oleg327-server: MODOPTS_LIBCFS= oleg327-server: Force libcfs to create 2 CPU partitions oleg327-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg327-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg327-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg327-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg327-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg327-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg327-server: oleg327-server.virtnet: executing set_default_debug -1 all pdsh@oleg327-client: oleg327-server: ssh exited with exit code 1 Started lustre-OST0000 oleg327-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg327-client.virtnet: -o user_xattr,flock 192.168.203.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 202 (time 1787796159.45 total 11.01 last 18.35) - open/close 478 (time 1787796170.07 total 21.63 last 25.99) - open/close 739 (time 1787796180.68 total 32.25 last 24.58) - open/close 1014 (time 1787796191.77 total 43.33 last 24.81) - open/close 1253 (time 1787796202.05 total 53.61 last 23.25) - open/close 1444 (time 1787796212.87 total 64.43 last 17.65) - open/close 1706 (time 1787796222.87 total 74.44 last 26.20) - open/close 1930 (time 1787796233.04 total 84.60 last 22.03) - open/close 2077 (time 1787796243.05 total 94.62 last 14.68) - open/close 2324 (time 1787796253.33 total 104.89 last 24.03) - open/close 2479 (time 1787796263.44 total 115.00 last 15.34) - open/close 2717 (time 1787796273.44 total 125.00 last 23.80) - open/close 2938 (time 1787796283.87 total 135.44 last 21.18) - open/close 3173 (time 1787796294.35 total 145.92 last 22.42) - open/close 3393 (time 1787796304.45 total 156.01 last 21.79) - open/close 3628 (time 1787796314.45 total 166.01 last 23.50) - open/close 3859 (time 1787796325.51 total 177.08 last 20.88) - open/close 4071 (time 1787796335.53 total 187.10 last 21.16) - open/close 4317 (time 1787796345.53 total 197.10 last 24.60) - open/close 4572 (time 1787796356.30 total 207.86 last 23.69) - open/close 4808 (time 1787796366.30 total 217.86 last 23.60) - open/close 5111 (time 1787796377.29 total 228.86 last 27.56) - open/close 5369 (time 1787796387.71 total 239.27 last 24.77) - open/close 5601 (time 1787796397.74 total 249.31 last 23.13) - open/close 5794 (time 1787796407.75 total 259.31 last 19.28) - open/close 6076 (time 1787796418.83 total 270.39 last 25.46) - open/close 6293 (time 1787796429.71 total 281.27 last 19.94) - open/close 6534 (time 1787796439.72 total 291.29 last 24.07) - open/close 6791 (time 1787796450.41 total 301.97 last 24.05) - open/close 7036 (time 1787796461.06 total 312.62 last 23.00) - open/close 7284 (time 1787796471.06 total 322.63 last 24.79) - open/close 7495 (time 1787796481.19 total 332.75 last 20.84) - open/close 7767 (time 1787796491.78 total 343.34 last 25.68) - open/close 8036 (time 1787796502.23 total 353.79 last 25.74) - open/close 8206 (time 1787796514.86 total 366.42 last 13.46) - open/close 8371 (time 1787796524.86 total 376.42 last 16.49) - open/close 8617 (time 1787796535.19 total 386.75 last 23.81) - open/close 8714 (time 1787796546.32 total 397.88 last 8.72) - open/close 8955 (time 1787796557.26 total 408.83 last 22.02) - open/close 9193 (time 1787796572.82 total 424.38 last 15.30) - open/close 9345 (time 1787796583.31 total 434.87 last 14.49) - open/close 9567 (time 1787796593.32 total 444.88 last 22.18) - open/close 9784 (time 1787796603.33 total 454.90 last 21.66) - open/close 10000 (time 1787796612.79 total 464.35 last 22.85) - open/close 10087 (time 1787796622.89 total 474.45 last 8.61) - open/close 10255 (time 1787796633.36 total 484.92 last 16.05) - open/close 10458 (time 1787796644.24 total 495.81 last 18.65) - open/close 10757 (time 1787796654.25 total 505.82 last 29.86) - open/close 10985 (time 1787796664.26 total 515.83 last 22.78) - open/close 11171 (time 1787796674.56 total 526.13 last 18.06) - open/close 11371 (time 1787796684.93 total 536.49 last 19.29) - open/close 11498 (time 1787796695.63 total 547.20 last 11.87) - open/close 11624 (time 1787796705.64 total 557.20 last 12.59) - open/close 11831 (time 1787796716.48 total 568.04 last 19.09) - open/close 12107 (time 1787796726.53 total 578.10 last 27.45) - open/close 12344 (time 1787796736.53 total 588.10 last 23.70) - open/close 12578 (time 1787796746.75 total 598.32 last 22.89) - open/close 12829 (time 1787796757.63 total 609.20 last 23.07) - open/close 13067 (time 1787796767.64 total 619.21 last 23.78) - open/close 13280 (time 1787796777.93 total 629.49 last 20.71) - open/close 13421 (time 1787796787.94 total 639.50 last 14.09) - open/close 13588 (time 1787796798.29 total 649.86 last 16.13) - open/close 13766 (time 1787796808.30 total 659.86 last 17.79) - open/close 13901 (time 1787796818.89 total 670.46 last 12.75) - open/close 14033 (time 1787796829.85 total 681.41 last 12.05) - open/close 14301 (time 1787796840.77 total 692.33 last 24.53) - open/close 14485 (time 1787796851.29 total 702.85 last 17.50) - open/close 14727 (time 1787796861.78 total 713.34 last 23.07) - open/close 14881 (time 1787796872.07 total 723.64 last 14.96) - open/close 15009 (time 1787796882.09 total 733.66 last 12.77) - open/close 15225 (time 1787796892.33 total 743.89 last 21.10) - open/close 15391 (time 1787796902.66 total 754.22 last 16.07) - open/close 15586 (time 1787796912.72 total 764.29 last 19.38) - open/close 15765 (time 1787796923.43 total 775.00 last 16.71) - open/close 15926 (time 1787796934.23 total 785.79 last 14.91) - open/close 16095 (time 1787796944.91 total 796.47 last 15.82) - open/close 16341 (time 1787796954.95 total 806.52 last 24.49) - open/close 16583 (time 1787796964.96 total 816.53 last 24.18) - open/close 16869 (time 1787796974.98 total 826.54 last 28.55) - open/close 17022 (time 1787796985.01 total 836.57 last 15.25) - open/close 17293 (time 1787796995.02 total 846.58 last 27.07) - open/close 17519 (time 1787797005.05 total 856.62 last 22.53) - open/close 17823 (time 1787797015.50 total 867.06 last 29.11) - open/close 18083 (time 1787797025.51 total 877.07 last 25.97) - open/close 18341 (time 1787797035.60 total 887.16 last 25.58) - open/close 18531 (time 1787797045.62 total 897.18 last 18.96) - open/close 18682 (time 1787797055.88 total 907.45 last 14.71) - open/close 18744 (time 1787797065.92 total 917.48 last 6.18) - open/close 18897 (time 1787797076.30 total 927.87 last 14.73) - open/close 19138 (time 1787797086.74 total 938.30 last 23.10) - open/close 19389 (time 1787797096.74 total 948.30 last 25.10) - open/close 20000 (time 1787797105.72 total 957.28 last 68.05) - open/close 20727 (time 1787797115.72 total 967.28 last 72.68) - open/close 21498 (time 1787797125.73 total 977.29 last 77.04) - open/close 22285 (time 1787797135.73 total 987.29 last 78.67) - open/close 23029 (time 1787797145.73 total 997.30 last 74.38) - open/close 23676 (time 1787797155.74 total 1007.30 last 64.69) - open/close 24336 (time 1787797165.76 total 1017.33 last 65.82) - open/close 24905 (time 1787797175.77 total 1027.33 last 56.88) - open/close 25616 (time 1787797185.77 total 1037.33 last 71.06) - open/close 26338 (time 1787797195.78 total 1047.34 last 72.14) - open/close 27149 (time 1787797205.79 total 1057.35 last 81.04) - open/close 27924 (time 1787797215.80 total 1067.36 last 77.42) - open/close 28724 (time 1787797225.80 total 1077.36 last 79.98) - open/close 29414 (time 1787797235.81 total 1087.37 last 68.95) - open/close 30000 (time 1787797243.88 total 1095.45 last 72.57) - open/close 30685 (time 1787797253.89 total 1105.45 last 68.46) - open/close 31457 (time 1787797263.89 total 1115.45 last 77.19) - open/close 32217 (time 1787797273.90 total 1125.46 last 75.93) - open/close 32860 (time 1787797283.91 total 1135.48 last 64.22) - open/close 33614 (time 1787797293.92 total 1145.48 last 75.37) - open/close 34299 (time 1787797303.93 total 1155.49 last 68.42) - open/close 34991 (time 1787797313.94 total 1165.50 last 69.14) - open/close 35754 (time 1787797323.94 total 1175.51 last 76.27) - open/close 36476 (time 1787797333.95 total 1185.51 last 72.16) - open/close 37132 (time 1787797343.95 total 1195.52 last 65.56) - open/close 37856 (time 1787797353.96 total 1205.52 last 72.37) - open/close 38508 (time 1787797363.97 total 1215.53 last 65.13) - open/close 39214 (time 1787797373.97 total 1225.53 last 70.59) - open/close 39839 (time 1787797383.98 total 1235.54 last 62.44) - open/close 40000 (time 1787797386.07 total 1237.64 last 76.93) - open/close 40721 (time 1787797396.07 total 1247.64 last 72.10) - open/close 41450 (time 1787797406.09 total 1257.65 last 72.79) - open/close 42153 (time 1787797416.10 total 1267.66 last 70.23) - open/close 42814 (time 1787797426.11 total 1277.68 last 66.00) - open/close 43488 (time 1787797436.12 total 1287.68 last 67.37) - open/close 44191 (time 1787797446.13 total 1297.69 last 70.21) - open/close 44910 (time 1787797456.13 total 1307.70 last 71.86) - open/close 45617 (time 1787797466.14 total 1317.70 last 70.67) - open/close 46338 (time 1787797476.15 total 1327.72 last 72.00) - open/close 47089 (time 1787797486.15 total 1337.72 last 75.10) - open/close 47802 (time 1787797496.16 total 1347.72 last 71.27) - open/close 48543 (time 1787797506.16 total 1357.72 last 74.07) - open/close 49288 (time 1787797516.17 total 1367.74 last 74.42) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1372.42 seconds: 36.16 ops/second - unlinked 0 (time 1787797525 ; total 0 ; last 0) - unlinked 10000 (time 1787797664 ; total 139 ; last 139) - unlinked 20000 (time 1787797806 ; total 281 ; last 142) - unlinked 30000 (time 1787797939 ; total 414 ; last 133) - unlinked 40000 (time 1787798077 ; total 552 ; last 138) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 681 seconds: 72.879585 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg327-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg327-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg327-server stop mds service on oleg327-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg327-server stop mds service on oleg327-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg327-server unloading modules via unload_modules_local on: 'oleg327-server' oleg327-server: oleg327-server.virtnet: executing unload_modules_local oleg327-server: modules unloaded. pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-client: ssh exited with exit code 2 pdsh@oleg327-client: oleg327-server: ssh exited with exit code 2