== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:51:11 (1709787071) start mds service on oleg110-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg110-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg110-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Started lustre-OST0000 oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg110-client.virtnet: -o user_xattr,flock oleg110-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 183 (time 1709787122.51 total 10.01 last 18.28) - open/close 346 (time 1709787133.34 total 20.84 last 15.05) - open/close 523 (time 1709787143.59 total 31.08 last 17.28) - open/close 733 (time 1709787153.60 total 41.10 last 20.97) - open/close 944 (time 1709787163.60 total 51.10 last 21.09) - open/close 1107 (time 1709787174.52 total 62.02 last 14.93) - open/close 1299 (time 1709787185.27 total 72.76 last 17.87) - open/close 1469 (time 1709787195.77 total 83.27 last 16.18) - open/close 1715 (time 1709787206.63 total 94.13 last 22.65) - open/close 2104 (time 1709787217.39 total 104.88 last 36.16) - open/close 2323 (time 1709787227.40 total 114.90 last 21.88) - open/close 2501 (time 1709787237.41 total 124.91 last 17.78) - open/close 2736 (time 1709787248.11 total 135.61 last 21.96) - open/close 2952 (time 1709787258.13 total 145.63 last 21.55) - open/close 3095 (time 1709787268.13 total 155.63 last 14.30) - open/close 3270 (time 1709787278.16 total 165.66 last 17.46) - open/close 3478 (time 1709787288.42 total 175.91 last 20.28) - open/close 3640 (time 1709787298.94 total 186.43 last 15.40) - open/close 3757 (time 1709787309.26 total 196.76 last 11.33) - open/close 4001 (time 1709787320.20 total 207.70 last 22.31) - open/close 4130 (time 1709787330.21 total 217.71 last 12.89) - open/close 4256 (time 1709787341.23 total 228.72 last 11.44) - open/close 4477 (time 1709787351.29 total 238.78 last 21.97) - open/close 4641 (time 1709787362.31 total 249.81 last 14.87) - open/close 4749 (time 1709787372.37 total 259.87 last 10.74) - open/close 4926 (time 1709787382.86 total 270.35 last 16.89) - open/close 5160 (time 1709787393.60 total 281.10 last 21.77) - open/close 5314 (time 1709787404.09 total 291.58 last 14.69) - open/close 5525 (time 1709787414.93 total 302.43 last 19.45) - open/close 5648 (time 1709787425.13 total 312.63 last 12.06) - open/close 5761 (time 1709787435.70 total 323.20 last 10.70) - open/close 5946 (time 1709787445.70 total 333.20 last 18.50) - open/close 6165 (time 1709787456.28 total 343.78 last 20.70) - open/close 6428 (time 1709787466.53 total 354.02 last 25.67) - open/close 6618 (time 1709787476.86 total 364.36 last 18.39) - open/close 6848 (time 1709787486.87 total 374.37 last 22.96) - open/close 6953 (time 1709787497.67 total 385.17 last 9.72) - open/close 7104 (time 1709787507.68 total 395.18 last 15.09) - open/close 7282 (time 1709787517.69 total 405.19 last 17.79) - open/close 7473 (time 1709787527.97 total 415.47 last 18.57) - open/close 7630 (time 1709787537.99 total 425.49 last 15.67) - open/close 7733 (time 1709787548.00 total 435.50 last 10.29) - open/close 7912 (time 1709787558.56 total 446.05 last 16.96) - open/close 8118 (time 1709787568.69 total 456.19 last 20.32) - open/close 8246 (time 1709787578.88 total 466.38 last 12.56) - open/close 8458 (time 1709787588.91 total 476.41 last 21.14) - open/close 8638 (time 1709787598.92 total 486.42 last 17.98) - open/close 8727 (time 1709787609.90 total 497.40 last 8.11) - open/close 8930 (time 1709787620.01 total 507.51 last 20.08) - open/close 9088 (time 1709787631.07 total 518.56 last 14.29) - open/close 9283 (time 1709787642.04 total 529.54 last 17.77) - open/close 9414 (time 1709787652.16 total 539.66 last 12.94) - open/close 9633 (time 1709787662.17 total 549.67 last 21.88) - open/close 9835 (time 1709787672.17 total 559.67 last 20.19) - open/close 10000 (time 1709787679.05 total 566.55 last 24.00) - open/close 10147 (time 1709787689.88 total 577.38 last 13.57) - open/close 10416 (time 1709787699.89 total 587.39 last 26.88) - open/close 11020 (time 1709787709.90 total 597.40 last 60.33) - open/close 11601 (time 1709787719.90 total 607.40 last 58.09) - open/close 12198 (time 1709787729.90 total 617.40 last 59.70) - open/close 12716 (time 1709787739.92 total 627.42 last 51.72) - open/close 13205 (time 1709787749.94 total 637.43 last 48.82) - open/close 13660 (time 1709787759.94 total 647.44 last 45.46) - open/close 14098 (time 1709787769.95 total 657.45 last 43.76) - open/close 14563 (time 1709787779.96 total 667.46 last 46.45) - open/close 15004 (time 1709787789.98 total 677.48 last 44.04) - open/close 15377 (time 1709787800.00 total 687.49 last 37.23) - open/close 15811 (time 1709787810.01 total 697.51 last 43.33) - open/close 16262 (time 1709787820.02 total 707.51 last 45.08) - open/close 16676 (time 1709787830.07 total 717.57 last 41.18) - open/close 17020 (time 1709787840.07 total 727.57 last 34.39) - open/close 17401 (time 1709787850.10 total 737.60 last 38.00) - open/close 17782 (time 1709787860.11 total 747.61 last 38.07) - open/close 18175 (time 1709787870.13 total 757.63 last 39.21) - open/close 18562 (time 1709787880.16 total 767.66 last 38.59) - open/close 18919 (time 1709787890.18 total 777.68 last 35.61) - open/close 19283 (time 1709787900.20 total 787.70 last 36.35) - open/close 19653 (time 1709787910.21 total 797.71 last 36.96) - open/close 20000 (time 1709787919.81 total 807.31 last 36.14) - open/close 20299 (time 1709787929.82 total 817.32 last 29.88) - open/close 20627 (time 1709787939.85 total 827.35 last 32.70) - open/close 20984 (time 1709787949.94 total 837.43 last 35.40) - open/close 21376 (time 1709787959.95 total 847.45 last 39.14) - open/close 21727 (time 1709787969.97 total 857.46 last 35.05) - open/close 22118 (time 1709787979.97 total 867.47 last 39.08) - open/close 22505 (time 1709787989.98 total 877.47 last 38.68) - open/close 22869 (time 1709787999.98 total 887.48 last 36.39) - open/close 23219 (time 1709788009.98 total 897.48 last 34.99) - open/close 23564 (time 1709788020.00 total 907.50 last 34.44) - open/close 23891 (time 1709788030.02 total 917.52 last 32.64) - open/close 24248 (time 1709788040.04 total 927.54 last 35.62) - open/close 24595 (time 1709788050.07 total 937.57 last 34.61) - open/close 24943 (time 1709788060.10 total 947.59 last 34.70) - open/close 25267 (time 1709788070.12 total 957.61 last 32.34) - open/close 25619 (time 1709788080.12 total 967.62 last 35.17) - open/close 25965 (time 1709788090.16 total 977.65 last 34.49) - open/close 26296 (time 1709788100.16 total 987.66 last 33.07) - open/close 26616 (time 1709788110.18 total 997.68 last 31.94) - open/close 26939 (time 1709788120.22 total 1007.72 last 32.18) - open/close 27243 (time 1709788130.22 total 1017.72 last 30.39) - open/close 27582 (time 1709788140.25 total 1027.75 last 33.82) - open/close 27886 (time 1709788150.28 total 1037.77 last 30.32) - open/close 28168 (time 1709788160.31 total 1047.80 last 28.12) - open/close 28455 (time 1709788170.31 total 1057.80 last 28.70) - open/close 28763 (time 1709788180.33 total 1067.82 last 30.74) - open/close 29061 (time 1709788190.35 total 1077.85 last 29.74) - open/close 29217 (time 1709788206.68 total 1094.18 last 9.55) - open/close 29555 (time 1709788216.70 total 1104.20 last 33.73) - open/close 29842 (time 1709788226.72 total 1114.22 last 28.65) - open/close 30000 (time 1709788231.28 total 1118.78 last 34.63) - open/close 30318 (time 1709788241.30 total 1128.79 last 31.75) - open/close 30606 (time 1709788251.30 total 1138.80 last 28.79) - open/close 30875 (time 1709788261.30 total 1148.80 last 26.90) - open/close 31156 (time 1709788271.31 total 1158.81 last 28.07) - open/close 31430 (time 1709788281.34 total 1168.84 last 27.32) - open/close 31749 (time 1709788291.36 total 1178.85 last 31.84) - open/close 32051 (time 1709788301.37 total 1188.87 last 30.16) - open/close 32353 (time 1709788311.38 total 1198.88 last 30.16) - open/close 32650 (time 1709788321.39 total 1208.89 last 29.69) - open/close 32948 (time 1709788331.40 total 1218.90 last 29.76) - open/close 33238 (time 1709788341.43 total 1228.92 last 28.93) - open/close 33522 (time 1709788351.46 total 1238.96 last 28.30) - open/close 33824 (time 1709788361.47 total 1248.97 last 30.17) - open/close 34100 (time 1709788371.47 total 1258.97 last 27.60) - open/close 34395 (time 1709788381.49 total 1268.99 last 29.45) - open/close 34683 (time 1709788391.50 total 1278.99 last 28.78) - open/close 34953 (time 1709788401.51 total 1289.00 last 26.97) - open/close 35228 (time 1709788411.53 total 1299.03 last 27.43) - open/close 35501 (time 1709788421.55 total 1309.05 last 27.25) - open/close 35776 (time 1709788431.58 total 1319.08 last 27.41) - open/close 36041 (time 1709788441.62 total 1329.12 last 26.41) - open/close 36279 (time 1709788451.64 total 1339.14 last 23.75) - open/close 36529 (time 1709788461.64 total 1349.14 last 25.00) - open/close 36791 (time 1709788471.64 total 1359.14 last 26.19) - open/close 37063 (time 1709788481.65 total 1369.15 last 27.18) - open/close 37313 (time 1709788491.71 total 1379.20 last 24.86) - open/close 37575 (time 1709788501.72 total 1389.22 last 26.17) - open/close 37852 (time 1709788511.74 total 1399.23 last 27.65) - open/close 38120 (time 1709788521.74 total 1409.24 last 26.79) - open/close 38388 (time 1709788531.78 total 1419.28 last 26.70) - open/close 38670 (time 1709788541.78 total 1429.28 last 28.19) - open/close 38944 (time 1709788551.78 total 1439.28 last 27.39) - open/close 39212 (time 1709788561.79 total 1449.29 last 26.79) - open/close 39487 (time 1709788571.79 total 1459.29 last 27.49) - open/close 39734 (time 1709788581.83 total 1469.33 last 24.61) - open/close 39995 (time 1709788591.85 total 1479.34 last 26.06) - open/close 40254 (time 1709788601.85 total 1489.34 last 25.90) - open/close 40519 (time 1709788611.88 total 1499.38 last 26.40) - open/close 40783 (time 1709788621.91 total 1509.41 last 26.33) - open/close 41051 (time 1709788631.94 total 1519.43 last 26.73) - open/close 41291 (time 1709788641.96 total 1529.46 last 23.93) - open/close 41521 (time 1709788651.98 total 1539.47 last 22.97) - open/close 41746 (time 1709788662.00 total 1549.50 last 22.44) - open/close 41987 (time 1709788672.01 total 1559.51 last 24.08) - open/close 42206 (time 1709788682.02 total 1569.51 last 21.89) - open/close 42425 (time 1709788692.07 total 1579.57 last 21.78) - open/close 42657 (time 1709788702.10 total 1589.60 last 23.13) - open/close 42905 (time 1709788712.14 total 1599.63 last 24.72) - open/close 43140 (time 1709788722.18 total 1609.68 last 23.39) - open/close 43375 (time 1709788732.19 total 1619.69 last 23.49) - open/close 43604 (time 1709788742.22 total 1629.72 last 22.82) - open/close 43827 (time 1709788752.23 total 1639.73 last 22.29) - open/close 44037 (time 1709788762.24 total 1649.74 last 20.98) - open/close 44249 (time 1709788772.26 total 1659.76 last 21.15) - open/close 44466 (time 1709788782.28 total 1669.77 last 21.67) - open/close 44702 (time 1709788792.30 total 1679.80 last 23.53) - open/close 44922 (time 1709788802.31 total 1689.81 last 21.98) - open/close 45149 (time 1709788812.32 total 1699.82 last 22.68) - open/close 45376 (time 1709788822.37 total 1709.87 last 22.60) - open/close 45604 (time 1709788832.38 total 1719.88 last 22.77) - open/close 45827 (time 1709788842.42 total 1729.92 last 22.21) - open/close 46050 (time 1709788852.48 total 1739.97 last 22.18) - open/close 46258 (time 1709788862.50 total 1749.99 last 20.76) - open/close 46468 (time 1709788872.50 total 1760.00 last 20.99) - open/close 46677 (time 1709788882.52 total 1770.02 last 20.87) - open/close 46873 (time 1709788892.52 total 1780.02 last 19.59) - open/close 47085 (time 1709788902.53 total 1790.03 last 21.18) - open/close 47290 (time 1709788912.57 total 1800.07 last 20.43) - open/close 47497 (time 1709788922.58 total 1810.07 last 20.69) - open/close 47708 (time 1709788932.62 total 1820.11 last 21.02) - open/close 47920 (time 1709788942.63 total 1830.12 last 21.18) - open/close 48124 (time 1709788952.65 total 1840.14 last 20.36) - open/close 48324 (time 1709788962.65 total 1850.15 last 19.98) - open/close 48527 (time 1709788972.71 total 1860.21 last 20.18) - open/close 48730 (time 1709788982.72 total 1870.22 last 20.29) - open/close 48926 (time 1709788992.72 total 1880.22 last 19.59) - open/close 49123 (time 1709789002.75 total 1890.25 last 19.64) - open/close 49323 (time 1709789012.77 total 1900.27 last 19.97) - open/close 49533 (time 1709789022.78 total 1910.28 last 20.97) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1914.91 seconds: 25.92 ops/second - unlinked 0 (time 1709789028 ; total 0 ; last 0) - unlinked 10000 (time 1709789606 ; total 578 ; last 578) - unlinked 20000 (time 1709790078 ; total 1050 ; last 472) - unlinked 30000 (time 1709790383 ; total 1355 ; last 305) - unlinked 40000 (time 1709790513 ; total 1485 ; last 130) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 1536 seconds: 32.312500 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg110-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg110-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server stop mds service on oleg110-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg110-server stop mds service on oleg110-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg110-server unloading modules on: 'oleg110-server' oleg110-server: oleg110-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg110-client: oleg110-client: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-client: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2