== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:05:27 (1785395127) start mds service on oleg319-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: detected 4 online CPUs by sysfs oleg319-server: MODOPTS_LIBCFS= oleg319-server: Force libcfs to create 2 CPU partitions oleg319-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg319-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg319-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg319-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg319-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 93 (time 1785395264.98 total 10.02 last 9.28) - open/close 212 (time 1785395275.93 total 20.97 last 10.87) - open/close 329 (time 1785395285.93 total 30.98 last 11.69) - open/close 392 (time 1785395295.95 total 41.00 last 6.29) - open/close 469 (time 1785395306.10 total 51.14 last 7.59) - open/close 594 (time 1785395317.10 total 62.14 last 11.36) - open/close 713 (time 1785395327.32 total 72.36 last 11.64) - open/close 835 (time 1785395337.33 total 82.37 last 12.19) - open/close 897 (time 1785395348.11 total 93.15 last 5.75) - open/close 1028 (time 1785395358.42 total 103.46 last 12.71) - open/close 1120 (time 1785395368.98 total 114.03 last 8.71) - open/close 1277 (time 1785395379.50 total 124.54 last 14.93) - open/close 1384 (time 1785395390.57 total 135.61 last 9.67) - open/close 1477 (time 1785395400.73 total 145.77 last 9.15) - open/close 1649 (time 1785395410.73 total 155.78 last 17.19) - open/close 1813 (time 1785395421.65 total 166.69 last 15.02) - open/close 1926 (time 1785395431.67 total 176.72 last 11.27) - open/close 2034 (time 1785395441.68 total 186.72 last 10.79) - open/close 2116 (time 1785395452.08 total 197.12 last 7.89) - open/close 2217 (time 1785395462.32 total 207.36 last 9.87) - open/close 2335 (time 1785395472.68 total 217.72 last 11.38) - open/close 2451 (time 1785395482.73 total 227.77 last 11.55) - open/close 2547 (time 1785395492.74 total 237.78 last 9.59) - open/close 2628 (time 1785395503.24 total 248.28 last 7.71) - open/close 2749 (time 1785395514.21 total 259.25 last 11.03) - open/close 2816 (time 1785395524.50 total 269.54 last 6.51) - open/close 2891 (time 1785395534.56 total 279.60 last 7.46) - open/close 3029 (time 1785395544.82 total 289.86 last 13.45) - open/close 3165 (time 1785395554.84 total 299.88 last 13.57) - open/close 3277 (time 1785395565.30 total 310.35 last 10.70) - open/close 3391 (time 1785395575.40 total 320.44 last 11.29) - open/close 3522 (time 1785395585.42 total 330.46 last 13.07) - open/close 3687 (time 1785395596.01 total 341.05 last 15.58) - open/close 3776 (time 1785395606.04 total 351.08 last 8.87) - open/close 3939 (time 1785395616.99 total 362.03 last 14.89) - open/close 4064 (time 1785395628.07 total 373.11 last 11.28) - open/close 4157 (time 1785395638.08 total 383.12 last 9.29) - open/close 4258 (time 1785395648.93 total 393.97 last 9.31) - open/close 4352 (time 1785395659.87 total 404.91 last 8.60) - open/close 4462 (time 1785395670.07 total 415.11 last 10.78) - open/close 4635 (time 1785395680.29 total 425.33 last 16.92) - open/close 4768 (time 1785395690.53 total 435.57 last 12.99) - open/close 4873 (time 1785395703.45 total 448.49 last 8.13) - open/close 4993 (time 1785395713.47 total 458.51 last 11.97) - open/close 5100 (time 1785395724.11 total 469.15 last 10.06) - open/close 5256 (time 1785395735.08 total 480.12 last 14.22) - open/close 5327 (time 1785395745.10 total 490.14 last 7.09) - open/close 5424 (time 1785395755.55 total 500.59 last 9.28) - open/close 5543 (time 1785395765.75 total 510.79 last 11.67) - open/close 5662 (time 1785395776.49 total 521.53 last 11.08) - open/close 5740 (time 1785395786.86 total 531.90 last 7.52) - open/close 5802 (time 1785395797.67 total 542.71 last 5.74) - open/close 5960 (time 1785395807.67 total 552.71 last 15.80) - open/close 6077 (time 1785395818.32 total 563.37 last 10.98) - open/close 6211 (time 1785395829.24 total 574.29 last 12.27) - open/close 6350 (time 1785395839.57 total 584.62 last 13.46) - open/close 6507 (time 1785395849.68 total 594.73 last 15.53) - open/close 6617 (time 1785395860.01 total 605.06 last 10.65) - open/close 6796 (time 1785395870.01 total 615.06 last 17.90) - open/close 6879 (time 1785395880.48 total 625.52 last 7.93) - open/close 6995 (time 1785395891.16 total 636.20 last 10.86) - open/close 7068 (time 1785395902.19 total 647.23 last 6.62) - open/close 7236 (time 1785395912.36 total 657.40 last 16.52) - open/close 7382 (time 1785395922.77 total 667.81 last 14.02) - open/close 7462 (time 1785395933.08 total 678.12 last 7.76) - open/close 7536 (time 1785395943.34 total 688.38 last 7.21) - open/close 7577 (time 1785395953.69 total 698.73 last 3.96) - open/close 7704 (time 1785395964.38 total 709.43 last 11.88) - open/close 7888 (time 1785395975.42 total 720.47 last 16.66) - open/close 8040 (time 1785395987.31 total 732.35 last 12.79) - open/close 8201 (time 1785395998.36 total 743.40 last 14.57) - open/close 8279 (time 1785396008.82 total 753.86 last 7.46) - open/close 8474 (time 1785396019.30 total 764.34 last 18.61) - open/close 8573 (time 1785396029.32 total 774.36 last 9.88) - open/close 8645 (time 1785396039.79 total 784.83 last 6.88) - open/close 8722 (time 1785396049.96 total 795.01 last 7.57) - open/close 8785 (time 1785396060.91 total 805.95 last 5.75) - open/close 8945 (time 1785396072.47 total 817.51 last 13.84) - open/close 9039 (time 1785396082.79 total 827.83 last 9.11) - open/close 9162 (time 1785396093.48 total 838.52 last 11.51) - open/close 9333 (time 1785396103.71 total 848.75 last 16.71) - open/close 9498 (time 1785396114.15 total 859.19 last 15.80) - open/close 9631 (time 1785396125.02 total 870.06 last 12.23) - open/close 9750 (time 1785396135.25 total 880.30 last 11.63) - open/close 9843 (time 1785396145.82 total 890.86 last 8.80) - open/close 9967 (time 1785396155.87 total 900.92 last 12.34) - open/close 10000 (time 1785396158.56 total 903.60 last 12.27) - open/close 10067 (time 1785396168.61 total 913.66 last 6.66) - open/close 10183 (time 1785396178.72 total 923.77 last 11.47) - open/close 10303 (time 1785396188.90 total 933.94 last 11.79) - open/close 10403 (time 1785396199.38 total 944.43 last 9.54) - open/close 10498 (time 1785396210.08 total 955.12 last 8.88) - open/close 10609 (time 1785396220.33 total 965.37 last 10.83) - open/close 10727 (time 1785396230.50 total 975.55 last 11.59) - open/close 10831 (time 1785396240.51 total 985.55 last 10.40) - open/close 10978 (time 1785396251.04 total 996.09 last 13.95) - open/close 11096 (time 1785396262.01 total 1007.05 last 10.76) - open/close 11282 (time 1785396272.41 total 1017.45 last 17.89) - open/close 11432 (time 1785396282.41 total 1027.45 last 15.00) - open/close 11576 (time 1785396292.42 total 1037.46 last 14.38) - open/close 11733 (time 1785396303.38 total 1048.42 last 14.33) - open/close 11873 (time 1785396313.38 total 1058.42 last 14.00) - open/close 11998 (time 1785396324.39 total 1069.43 last 11.36) - open/close 12123 (time 1785396334.45 total 1079.49 last 12.42) - open/close 12272 (time 1785396344.47 total 1089.51 last 14.87) - open/close 12337 (time 1785396354.49 total 1099.54 last 6.49) - open/close 12487 (time 1785396364.87 total 1109.91 last 14.45) - open/close 12572 (time 1785396375.16 total 1120.20 last 8.27) - open/close 12611 (time 1785396385.33 total 1130.37 last 3.83) - open/close 12710 (time 1785396395.89 total 1140.93 last 9.38) - open/close 12802 (time 1785396406.50 total 1151.54 last 8.67) - open/close 12859 (time 1785396417.34 total 1162.38 last 5.26) - open/close 12936 (time 1785396428.37 total 1173.42 last 6.98) - open/close 13057 (time 1785396438.55 total 1183.60 last 11.89) - open/close 13156 (time 1785396448.56 total 1193.60 last 9.90) - open/close 13253 (time 1785396458.59 total 1203.63 last 9.67) - open/close 13406 (time 1785396469.34 total 1214.39 last 14.23) - open/close 13502 (time 1785396479.43 total 1224.48 last 9.51) - open/close 13619 (time 1785396489.49 total 1234.53 last 11.64) - open/close 13775 (time 1785396500.25 total 1245.29 last 14.49) - open/close 13898 (time 1785396510.29 total 1255.34 last 12.25) - open/close 14035 (time 1785396520.99 total 1266.03 last 12.81) - open/close 14140 (time 1785396531.03 total 1276.08 last 10.46) - open/close 14216 (time 1785396541.36 total 1286.40 last 7.36) - open/close 14307 (time 1785396551.39 total 1296.43 last 9.07) - open/close 14437 (time 1785396561.76 total 1306.80 last 12.54) - open/close 14515 (time 1785396574.25 total 1319.29 last 6.25) - open/close 14880 (time 1785396584.27 total 1329.31 last 36.43) - open/close 15488 (time 1785396594.28 total 1339.32 last 60.73) - open/close 16061 (time 1785396604.28 total 1349.32 last 57.29) - open/close 16546 (time 1785396614.30 total 1359.34 last 48.40) - open/close 17205 (time 1785396624.31 total 1369.35 last 65.83) - open/close 17827 (time 1785396634.32 total 1379.36 last 62.16) - open/close 18386 (time 1785396644.32 total 1389.36 last 55.90) - open/close 18894 (time 1785396654.32 total 1399.37 last 50.76) - open/close 19467 (time 1785396664.33 total 1409.37 last 57.28) - open/close 20000 (time 1785396672.65 total 1417.69 last 64.09) - open/close 20677 (time 1785396682.65 total 1427.69 last 67.67) - open/close 21128 (time 1785396692.67 total 1437.71 last 45.03) - open/close 21656 (time 1785396702.67 total 1447.71 last 52.77) - open/close 22172 (time 1785396712.67 total 1457.71 last 51.60) - open/close 22651 (time 1785396722.68 total 1467.72 last 47.87) - open/close 23091 (time 1785396732.68 total 1477.72 last 43.99) - open/close 23886 (time 1785396742.69 total 1487.74 last 79.39) - open/close 24465 (time 1785396752.71 total 1497.75 last 57.81) - open/close 25003 (time 1785396762.72 total 1507.77 last 53.71) - open/close 25680 (time 1785396772.75 total 1517.79 last 67.52) - open/close 26077 (time 1785396782.78 total 1527.82 last 39.60) - open/close 26487 (time 1785396792.78 total 1537.82 last 40.98) - open/close 27056 (time 1785396802.78 total 1547.83 last 56.87) - open/close 27470 (time 1785396812.80 total 1557.84 last 41.34) - open/close 28001 (time 1785396822.80 total 1567.84 last 53.09) - open/close 28469 (time 1785396832.81 total 1577.85 last 46.77) - open/close 28949 (time 1785396842.81 total 1587.85 last 48.00) - open/close 29340 (time 1785396852.81 total 1597.85 last 39.08) - open/close 29821 (time 1785396862.85 total 1607.89 last 47.93) - open/close 30000 (time 1785396865.95 total 1610.99 last 57.72) - open/close 30495 (time 1785396875.95 total 1621.00 last 49.47) - open/close 30999 (time 1785396885.98 total 1631.02 last 50.27) - open/close 31503 (time 1785396895.99 total 1641.03 last 50.34) - open/close 32040 (time 1785396906.00 total 1651.04 last 53.66) - open/close 32565 (time 1785396916.03 total 1661.07 last 52.34) - open/close 33187 (time 1785396926.05 total 1671.09 last 62.10) - open/close 33750 (time 1785396936.06 total 1681.10 last 56.23) - open/close 34189 (time 1785396946.06 total 1691.11 last 43.88) - open/close 34713 (time 1785396956.08 total 1701.13 last 52.30) - open/close 35215 (time 1785396966.09 total 1711.14 last 50.15) - open/close 35653 (time 1785396976.10 total 1721.14 last 43.77) - open/close 36243 (time 1785396986.11 total 1731.16 last 58.93) - open/close 36930 (time 1785396996.14 total 1741.18 last 68.53) - open/close 37450 (time 1785397006.18 total 1751.22 last 51.78) - open/close 37963 (time 1785397016.20 total 1761.24 last 51.22) - open/close 38370 (time 1785397026.22 total 1771.26 last 40.61) - open/close 38770 (time 1785397036.23 total 1781.27 last 39.97) - open/close 39434 (time 1785397046.23 total 1791.28 last 66.35) - open/close 39921 (time 1785397056.26 total 1801.30 last 48.59) - open/close 40379 (time 1785397066.27 total 1811.31 last 45.73) - open/close 40859 (time 1785397076.28 total 1821.32 last 47.98) - open/close 41361 (time 1785397086.33 total 1831.37 last 49.93) - open/close 41904 (time 1785397096.33 total 1841.38 last 54.28) - open/close 42541 (time 1785397106.36 total 1851.40 last 63.56) - open/close 43144 (time 1785397116.37 total 1861.41 last 60.22) - open/close 43668 (time 1785397126.37 total 1871.41 last 52.39) - open/close 44255 (time 1785397136.37 total 1881.42 last 58.68) - open/close 45005 (time 1785397146.39 total 1891.43 last 74.88) - open/close 45427 (time 1785397156.39 total 1901.43 last 42.20) - open/close 45853 (time 1785397166.43 total 1911.47 last 42.43) - open/close 46236 (time 1785397176.45 total 1921.49 last 38.23) - open/close 46671 (time 1785397186.46 total 1931.50 last 43.45) - open/close 47110 (time 1785397196.46 total 1941.51 last 43.89) - open/close 47564 (time 1785397206.47 total 1951.51 last 45.36) - open/close 48120 (time 1785397216.48 total 1961.52 last 55.57) - open/close 48552 (time 1785397226.49 total 1971.53 last 43.15) - open/close 49079 (time 1785397236.51 total 1981.55 last 52.60) - open/close 49467 (time 1785397246.52 total 1991.56 last 38.74) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1996.03 seconds: 24.86 ops/second - unlinked 0 (time 1785397257 ; total 0 ; last 0) - unlinked 10000 (time 1785397439 ; total 182 ; last 182) - unlinked 20000 (time 1785397641 ; total 384 ; last 202) - unlinked 30000 (time 1785397828 ; total 571 ; last 187) - unlinked 40000 (time 1785398034 ; total 777 ; last 206) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 962 seconds: 51.591476 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg319-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg319-server unloading modules via unload_modules_local on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing unload_modules_local oleg319-server: modules unloaded. pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2