== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:09:00 (1787386140) start mds service on oleg456-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing load_modules_local oleg456-server: Loading modules from /home/green/git/lustre-release/lustre oleg456-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg456-server: detected 4 online CPUs by sysfs oleg456-server: MODOPTS_LIBCFS= oleg456-server: Force libcfs to create 2 CPU partitions oleg456-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg456-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg456-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg456-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg456-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Started lustre-OST0000 oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg456-client.virtnet: -o user_xattr,flock 192.168.204.156@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 194 (time 1787386248.06 total 10.03 last 19.34) - open/close 380 (time 1787386259.02 total 20.99 last 16.98) - open/close 481 (time 1787386269.30 total 31.27 last 9.82) - open/close 682 (time 1787386281.68 total 43.65 last 16.24) - open/close 797 (time 1787386292.37 total 54.34 last 10.75) - open/close 1053 (time 1787386303.31 total 65.28 last 23.40) - open/close 1306 (time 1787386313.87 total 75.84 last 23.96) - open/close 1535 (time 1787386324.14 total 86.11 last 22.29) - open/close 1666 (time 1787386335.17 total 97.14 last 11.88) - open/close 1838 (time 1787386345.89 total 107.86 last 16.05) - open/close 1978 (time 1787386355.90 total 117.87 last 13.99) - open/close 2168 (time 1787386366.86 total 128.83 last 17.32) - open/close 2366 (time 1787386376.89 total 138.86 last 19.76) - open/close 2610 (time 1787386387.04 total 149.01 last 24.04) - open/close 2874 (time 1787386397.04 total 159.01 last 26.38) - open/close 3069 (time 1787386407.06 total 169.03 last 19.47) - open/close 3253 (time 1787386417.44 total 179.41 last 17.73) - open/close 3370 (time 1787386427.49 total 189.46 last 11.64) - open/close 3549 (time 1787386437.99 total 199.96 last 17.06) - open/close 3793 (time 1787386447.99 total 209.96 last 24.39) - open/close 3966 (time 1787386458.34 total 220.31 last 16.72) - open/close 4243 (time 1787386468.34 total 230.31 last 27.69) - open/close 4514 (time 1787386478.34 total 240.31 last 27.10) - open/close 4754 (time 1787386488.90 total 250.87 last 22.73) - open/close 4976 (time 1787386499.40 total 261.38 last 21.13) - open/close 5159 (time 1787386511.64 total 273.61 last 14.96) - open/close 5406 (time 1787386521.65 total 283.62 last 24.68) - open/close 5635 (time 1787386532.36 total 294.33 last 21.37) - open/close 5705 (time 1787386542.98 total 304.95 last 6.59) - open/close 5875 (time 1787386553.64 total 315.62 last 15.94) - open/close 6121 (time 1787386563.73 total 325.70 last 24.39) - open/close 6261 (time 1787386574.02 total 335.99 last 13.60) - open/close 6453 (time 1787386584.15 total 346.12 last 18.97) - open/close 6665 (time 1787386596.64 total 358.61 last 16.97) - open/close 6866 (time 1787386607.39 total 369.36 last 18.71) - open/close 7013 (time 1787386620.81 total 382.78 last 10.95) - open/close 7171 (time 1787386631.46 total 393.43 last 14.84) - open/close 7438 (time 1787386642.45 total 404.42 last 24.29) - open/close 7648 (time 1787386652.45 total 414.42 last 20.99) - open/close 7749 (time 1787386662.67 total 424.64 last 9.89) - open/close 7992 (time 1787386674.32 total 436.29 last 20.86) - open/close 8184 (time 1787386684.32 total 446.29 last 19.19) - open/close 8394 (time 1787386694.43 total 456.40 last 20.79) - open/close 8622 (time 1787386705.37 total 467.34 last 20.84) - open/close 8755 (time 1787386715.38 total 477.35 last 13.28) - open/close 8848 (time 1787386725.91 total 487.88 last 8.83) - open/close 8993 (time 1787386736.26 total 498.23 last 14.01) - open/close 9187 (time 1787386746.26 total 508.24 last 19.40) - open/close 9439 (time 1787386756.49 total 518.46 last 24.65) - open/close 9652 (time 1787386767.88 total 529.85 last 18.70) - open/close 9832 (time 1787386780.13 total 542.10 last 14.70) - open/close 9987 (time 1787386790.75 total 552.72 last 14.60) - open/close 10220 (time 1787386800.75 total 562.72 last 23.29) - open/close 10366 (time 1787386810.91 total 572.88 last 14.37) - open/close 10605 (time 1787386823.15 total 585.12 last 19.53) - open/close 10799 (time 1787386833.17 total 595.14 last 19.37) - open/close 10970 (time 1787386843.21 total 605.18 last 17.02) - open/close 11132 (time 1787386853.39 total 615.36 last 15.91) - open/close 11310 (time 1787386863.95 total 625.92 last 16.87) - open/close 11583 (time 1787386875.01 total 636.98 last 24.68) - open/close 11706 (time 1787386885.85 total 647.82 last 11.35) - open/close 11879 (time 1787386895.86 total 657.83 last 17.28) - open/close 12003 (time 1787386905.88 total 667.85 last 12.38) - open/close 12079 (time 1787386916.87 total 678.84 last 6.91) - open/close 12198 (time 1787386927.89 total 689.86 last 10.80) - open/close 12381 (time 1787386938.83 total 700.80 last 16.74) - open/close 12565 (time 1787386948.85 total 710.82 last 18.37) - open/close 12695 (time 1787386959.00 total 720.97 last 12.80) - open/close 12888 (time 1787386970.06 total 732.03 last 17.46) - open/close 13066 (time 1787386980.69 total 742.66 last 16.74) - open/close 13318 (time 1787386990.70 total 752.67 last 25.17) - open/close 13546 (time 1787387001.18 total 763.15 last 21.76) - open/close 13742 (time 1787387013.15 total 775.12 last 16.37) - open/close 13913 (time 1787387023.18 total 785.15 last 17.05) - open/close 14157 (time 1787387034.00 total 795.98 last 22.54) - open/close 14323 (time 1787387045.59 total 807.56 last 14.33) - open/close 14565 (time 1787387058.46 total 820.43 last 18.79) - open/close 14797 (time 1787387068.56 total 830.53 last 22.97) - open/close 14933 (time 1787387078.59 total 840.57 last 13.56) - open/close 15315 (time 1787387088.97 total 850.94 last 36.83) - open/close 15586 (time 1787387098.98 total 860.95 last 27.07) - open/close 15732 (time 1787387109.72 total 871.69 last 13.59) - open/close 15908 (time 1787387119.78 total 881.75 last 17.50) - open/close 16142 (time 1787387129.80 total 891.77 last 23.35) - open/close 16319 (time 1787387140.55 total 902.52 last 16.47) - open/close 16504 (time 1787387151.07 total 913.04 last 17.58) - open/close 16662 (time 1787387161.87 total 923.84 last 14.64) - open/close 16900 (time 1787387172.44 total 934.41 last 22.52) - open/close 17108 (time 1787387182.47 total 944.44 last 20.73) - open/close 17259 (time 1787387192.77 total 954.74 last 14.66) - open/close 17393 (time 1787387202.91 total 964.88 last 13.22) - open/close 17571 (time 1787387212.92 total 974.89 last 17.77) - open/close 17705 (time 1787387223.82 total 985.79 last 12.29) - open/close 17852 (time 1787387234.65 total 996.62 last 13.58) - open/close 17944 (time 1787387245.07 total 1007.05 last 8.82) - open/close 18057 (time 1787387255.11 total 1017.08 last 11.27) - open/close 18192 (time 1787387265.12 total 1027.09 last 13.48) - open/close 18823 (time 1787387275.13 total 1037.10 last 63.06) - open/close 19480 (time 1787387285.13 total 1047.10 last 65.69) - open/close 20000 (time 1787387293.02 total 1054.99 last 65.89) - open/close 20713 (time 1787387303.03 total 1065.00 last 71.27) - open/close 21408 (time 1787387313.04 total 1075.01 last 69.44) - open/close 22186 (time 1787387323.04 total 1085.01 last 77.80) - open/close 22858 (time 1787387333.05 total 1095.02 last 67.12) - open/close 23612 (time 1787387343.06 total 1105.03 last 75.28) - open/close 24309 (time 1787387353.07 total 1115.05 last 69.62) - open/close 25010 (time 1787387363.08 total 1125.05 last 70.07) - open/close 25659 (time 1787387373.09 total 1135.06 last 64.85) - open/close 26203 (time 1787387383.10 total 1145.07 last 54.32) - open/close 26695 (time 1787387393.12 total 1155.09 last 49.11) - open/close 27241 (time 1787387403.12 total 1165.09 last 54.59) - open/close 27841 (time 1787387413.12 total 1175.09 last 59.99) - open/close 28427 (time 1787387423.14 total 1185.11 last 58.53) - open/close 29136 (time 1787387433.14 total 1195.11 last 70.86) - open/close 29833 (time 1787387443.16 total 1205.13 last 69.58) - open/close 30000 (time 1787387445.77 total 1207.74 last 63.97) - open/close 30609 (time 1787387455.78 total 1217.75 last 60.84) - open/close 31315 (time 1787387465.78 total 1227.75 last 70.58) - open/close 32037 (time 1787387475.81 total 1237.78 last 72.01) - open/close 32729 (time 1787387485.81 total 1247.79 last 69.15) - open/close 33252 (time 1787387495.84 total 1257.81 last 52.17) - open/close 33778 (time 1787387505.86 total 1267.83 last 52.52) - open/close 34262 (time 1787387515.86 total 1277.83 last 48.38) - open/close 34826 (time 1787387525.89 total 1287.86 last 56.25) - open/close 35389 (time 1787387535.89 total 1297.86 last 56.26) - open/close 36112 (time 1787387545.90 total 1307.87 last 72.29) - open/close 36796 (time 1787387555.91 total 1317.88 last 68.32) - open/close 37464 (time 1787387565.93 total 1327.90 last 66.63) - open/close 38102 (time 1787387575.93 total 1337.90 last 63.79) - open/close 38715 (time 1787387585.94 total 1347.91 last 61.24) - open/close 39304 (time 1787387595.96 total 1357.93 last 58.82) - open/close 39989 (time 1787387605.96 total 1367.93 last 68.46) - open/close 40649 (time 1787387615.97 total 1377.94 last 65.94) - open/close 41351 (time 1787387625.98 total 1387.95 last 70.16) - open/close 42083 (time 1787387635.98 total 1397.95 last 73.16) - open/close 42686 (time 1787387645.99 total 1407.96 last 60.25) - open/close 43295 (time 1787387656.00 total 1417.97 last 60.85) - open/close 43925 (time 1787387666.00 total 1427.97 last 62.98) - open/close 44546 (time 1787387676.01 total 1437.98 last 62.03) - open/close 45101 (time 1787387686.03 total 1448.00 last 55.38) - open/close 45711 (time 1787387696.05 total 1458.02 last 60.90) - open/close 46240 (time 1787387706.06 total 1468.03 last 52.88) - open/close 46796 (time 1787387716.08 total 1478.05 last 55.48) - open/close 47375 (time 1787387726.09 total 1488.06 last 57.85) - open/close 47914 (time 1787387736.09 total 1498.06 last 53.89) - open/close 48480 (time 1787387746.09 total 1508.06 last 56.59) - open/close 49113 (time 1787387756.09 total 1518.06 last 63.28) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1527.36 seconds: 32.49 ops/second - unlinked 0 (time 1787387770 ; total 0 ; last 0) - unlinked 10000 (time 1787387927 ; total 157 ; last 157) - unlinked 20000 (time 1787388074 ; total 304 ; last 147) - unlinked 30000 (time 1787388223 ; total 453 ; last 149) - unlinked 40000 (time 1787388368 ; total 598 ; last 145) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 737 seconds: 67.341927 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg456-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server stop mds service on oleg456-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server stop mds service on oleg456-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg456-server unloading modules via unload_modules_local on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing unload_modules_local oleg456-server: modules unloaded. pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-client: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-client: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2