== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:56:22 (1709787382) start mds service on oleg202-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg202-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg202-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg202-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg202-server: oleg202-server.virtnet: executing set_default_debug -1 all pdsh@oleg202-client: oleg202-server: ssh exited with exit code 1 Started lustre-OST0000 oleg202-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg202-client.virtnet: -o user_xattr,flock oleg202-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 204 (time 1709787434.96 total 10.12 last 20.16) - open/close 468 (time 1709787445.59 total 20.75 last 24.82) - open/close 660 (time 1709787455.61 total 30.77 last 19.17) - open/close 834 (time 1709787466.55 total 41.71 last 15.90) - open/close 1018 (time 1709787476.63 total 51.79 last 18.25) - open/close 1305 (time 1709787486.63 total 61.79 last 28.69) - open/close 1455 (time 1709787497.29 total 72.45 last 14.07) - open/close 1644 (time 1709787507.96 total 83.12 last 17.71) - open/close 1838 (time 1709787517.97 total 93.13 last 19.39) - open/close 1993 (time 1709787528.78 total 103.94 last 14.34) - open/close 2108 (time 1709787539.04 total 114.20 last 11.20) - open/close 2216 (time 1709787549.79 total 124.95 last 10.05) - open/close 2384 (time 1709787559.80 total 134.96 last 16.78) - open/close 2684 (time 1709787569.82 total 144.98 last 29.94) - open/close 2845 (time 1709787580.72 total 155.88 last 14.76) - open/close 3062 (time 1709787590.74 total 165.90 last 21.67) - open/close 3273 (time 1709787601.61 total 176.77 last 19.41) - open/close 3529 (time 1709787612.61 total 187.77 last 23.27) - open/close 3853 (time 1709787622.78 total 197.94 last 31.85) - open/close 4159 (time 1709787633.77 total 208.93 last 27.85) - open/close 4358 (time 1709787644.78 total 219.94 last 18.07) - open/close 4509 (time 1709787656.83 total 231.99 last 12.53) - open/close 4725 (time 1709787666.85 total 242.01 last 21.56) - open/close 4855 (time 1709787676.85 total 252.01 last 13.00) - open/close 5022 (time 1709787687.11 total 262.27 last 16.28) - open/close 5209 (time 1709787697.73 total 272.89 last 17.61) - open/close 5420 (time 1709787708.56 total 283.72 last 19.49) - open/close 5611 (time 1709787718.94 total 294.10 last 18.40) - open/close 5826 (time 1709787728.96 total 304.12 last 21.46) - open/close 6046 (time 1709787739.60 total 314.76 last 20.67) - open/close 6213 (time 1709787750.55 total 325.71 last 15.25) - open/close 6390 (time 1709787760.58 total 335.74 last 17.65) - open/close 6538 (time 1709787772.62 total 347.78 last 12.29) - open/close 6665 (time 1709787783.02 total 358.18 last 12.21) - open/close 6815 (time 1709787793.03 total 368.19 last 14.98) - open/close 6976 (time 1709787803.04 total 378.20 last 16.09) - open/close 7067 (time 1709787813.06 total 388.22 last 9.08) - open/close 7167 (time 1709787830.52 total 405.68 last 5.73) - open/close 7408 (time 1709787840.53 total 415.69 last 24.10) - open/close 7524 (time 1709787851.23 total 426.39 last 10.84) - open/close 7757 (time 1709787862.05 total 437.21 last 21.53) - open/close 7894 (time 1709787872.06 total 447.22 last 13.68) - open/close 8014 (time 1709787883.06 total 458.22 last 10.91) - open/close 8145 (time 1709787893.08 total 468.24 last 13.08) - open/close 8278 (time 1709787903.63 total 478.79 last 12.61) - open/close 8488 (time 1709787913.65 total 488.81 last 20.95) - open/close 8620 (time 1709787923.72 total 498.88 last 13.11) - open/close 8740 (time 1709787933.85 total 509.01 last 11.84) - open/close 8892 (time 1709787943.87 total 519.03 last 15.18) - open/close 9164 (time 1709787953.87 total 529.03 last 27.20) - open/close 9347 (time 1709787963.90 total 539.06 last 18.24) - open/close 9667 (time 1709787974.46 total 549.62 last 30.31) - open/close 9853 (time 1709787984.81 total 559.97 last 17.96) - open/close 10000 (time 1709787992.33 total 567.49 last 19.55) - open/close 10257 (time 1709788002.34 total 577.50 last 25.69) - open/close 10753 (time 1709788012.34 total 587.50 last 49.56) - open/close 11490 (time 1709788022.35 total 597.51 last 73.69) - open/close 12220 (time 1709788032.35 total 607.51 last 72.99) - open/close 12940 (time 1709788042.36 total 617.52 last 71.92) - open/close 13618 (time 1709788052.37 total 627.53 last 67.71) - open/close 14278 (time 1709788062.38 total 637.54 last 65.98) - open/close 14878 (time 1709788072.39 total 647.55 last 59.91) - open/close 15484 (time 1709788082.39 total 657.55 last 60.60) - open/close 16091 (time 1709788092.39 total 667.55 last 60.69) - open/close 16657 (time 1709788102.39 total 677.55 last 56.60) - open/close 17207 (time 1709788112.40 total 687.56 last 54.99) - open/close 17768 (time 1709788122.40 total 697.56 last 56.09) - open/close 18308 (time 1709788132.41 total 707.57 last 53.91) - open/close 18835 (time 1709788142.44 total 717.60 last 52.58) - open/close 19351 (time 1709788152.45 total 727.61 last 51.54) - open/close 19881 (time 1709788162.47 total 737.63 last 52.89) - open/close 20000 (time 1709788164.86 total 740.02 last 49.79) - open/close 20522 (time 1709788174.87 total 750.03 last 52.16) - open/close 21017 (time 1709788184.87 total 760.03 last 49.49) - open/close 21500 (time 1709788194.88 total 770.04 last 48.25) - open/close 21974 (time 1709788204.89 total 780.05 last 47.35) - open/close 22468 (time 1709788214.89 total 790.05 last 49.40) - open/close 22959 (time 1709788224.91 total 800.07 last 48.99) - open/close 23425 (time 1709788234.91 total 810.07 last 46.60) - open/close 23885 (time 1709788244.93 total 820.09 last 45.92) - open/close 24341 (time 1709788254.95 total 830.11 last 45.50) - open/close 24782 (time 1709788264.97 total 840.13 last 44.04) - open/close 25224 (time 1709788274.99 total 850.15 last 44.12) - open/close 25633 (time 1709788284.99 total 860.15 last 40.87) - open/close 26027 (time 1709788295.02 total 870.18 last 39.31) - open/close 26413 (time 1709788305.02 total 880.18 last 38.59) - open/close 26805 (time 1709788315.04 total 890.20 last 39.11) - open/close 27183 (time 1709788325.06 total 900.22 last 37.72) - open/close 27551 (time 1709788335.07 total 910.23 last 36.76) - open/close 27920 (time 1709788345.08 total 920.24 last 36.88) - open/close 28283 (time 1709788355.09 total 930.25 last 36.23) - open/close 28655 (time 1709788365.11 total 940.27 last 37.13) - open/close 29002 (time 1709788375.13 total 950.29 last 34.64) - open/close 29357 (time 1709788385.15 total 960.31 last 35.44) - open/close 29699 (time 1709788395.16 total 970.32 last 34.16) - open/close 30000 (time 1709788403.48 total 978.64 last 36.17) - open/close 30373 (time 1709788413.50 total 988.66 last 37.23) - open/close 30732 (time 1709788423.53 total 998.69 last 35.79) - open/close 31107 (time 1709788433.55 total 1008.71 last 37.41) - open/close 31474 (time 1709788443.57 total 1018.73 last 36.65) - open/close 31828 (time 1709788453.59 total 1028.75 last 35.32) - open/close 32154 (time 1709788463.62 total 1038.78 last 32.50) - open/close 32495 (time 1709788473.63 total 1048.79 last 34.06) - open/close 32843 (time 1709788483.67 total 1058.83 last 34.67) - open/close 33185 (time 1709788493.67 total 1068.83 last 34.19) - open/close 33545 (time 1709788503.67 total 1078.83 last 36.00) - open/close 33904 (time 1709788513.68 total 1088.84 last 35.88) - open/close 34248 (time 1709788523.71 total 1098.87 last 34.30) - open/close 34576 (time 1709788533.73 total 1108.89 last 32.73) - open/close 34913 (time 1709788543.74 total 1118.90 last 33.63) - open/close 35231 (time 1709788553.76 total 1128.92 last 31.74) - open/close 35550 (time 1709788563.77 total 1138.93 last 31.89) - open/close 35871 (time 1709788573.78 total 1148.94 last 32.06) - open/close 36193 (time 1709788583.81 total 1158.97 last 32.11) - open/close 36524 (time 1709788593.83 total 1168.99 last 33.01) - open/close 36851 (time 1709788603.84 total 1179.00 last 32.67) - open/close 37165 (time 1709788613.87 total 1189.03 last 31.33) - open/close 37498 (time 1709788623.87 total 1199.03 last 33.28) - open/close 37818 (time 1709788633.88 total 1209.04 last 31.98) - open/close 38113 (time 1709788643.89 total 1219.05 last 29.46) - open/close 38418 (time 1709788653.91 total 1229.07 last 30.45) - open/close 38722 (time 1709788663.93 total 1239.09 last 30.33) - open/close 39015 (time 1709788673.94 total 1249.10 last 29.26) - open/close 39308 (time 1709788683.95 total 1259.11 last 29.28) - open/close 39603 (time 1709788693.97 total 1269.13 last 29.43) - open/close 39894 (time 1709788703.99 total 1279.15 last 29.06) - open/close 40000 (time 1709788707.54 total 1282.70 last 29.89) - open/close 40299 (time 1709788717.54 total 1292.70 last 29.88) - open/close 40593 (time 1709788727.54 total 1302.70 last 29.40) - open/close 40881 (time 1709788737.57 total 1312.73 last 28.73) - open/close 41153 (time 1709788747.57 total 1322.73 last 27.18) - open/close 41425 (time 1709788757.60 total 1332.76 last 27.14) - open/close 41719 (time 1709788767.61 total 1342.77 last 29.38) - open/close 41990 (time 1709788777.62 total 1352.78 last 27.05) - open/close 42252 (time 1709788787.66 total 1362.82 last 26.11) - open/close 42519 (time 1709788797.67 total 1372.83 last 26.67) - open/close 42790 (time 1709788807.68 total 1382.84 last 27.06) - open/close 43056 (time 1709788817.72 total 1392.88 last 26.50) - open/close 43314 (time 1709788827.72 total 1402.88 last 25.79) - open/close 43573 (time 1709788837.76 total 1412.92 last 25.81) - open/close 43831 (time 1709788847.79 total 1422.95 last 25.72) - open/close 44084 (time 1709788857.82 total 1432.98 last 25.22) - open/close 44338 (time 1709788867.82 total 1442.98 last 25.40) - open/close 44605 (time 1709788877.84 total 1453.00 last 26.66) - open/close 44884 (time 1709788887.86 total 1463.02 last 27.83) - open/close 45157 (time 1709788897.89 total 1473.05 last 27.22) - open/close 45434 (time 1709788907.92 total 1483.08 last 27.62) - open/close 45704 (time 1709788917.92 total 1493.08 last 26.99) - open/close 45977 (time 1709788927.95 total 1503.11 last 27.23) - open/close 46246 (time 1709788937.97 total 1513.13 last 26.84) - open/close 46516 (time 1709788947.99 total 1523.15 last 26.95) - open/close 46787 (time 1709788958.01 total 1533.17 last 27.04) - open/close 47055 (time 1709788968.04 total 1543.20 last 26.70) - open/close 47319 (time 1709788978.07 total 1553.23 last 26.34) - open/close 47576 (time 1709788988.07 total 1563.23 last 25.69) - open/close 47830 (time 1709788998.10 total 1573.26 last 25.33) - open/close 48081 (time 1709789008.10 total 1583.27 last 25.09) - open/close 48328 (time 1709789018.12 total 1593.28 last 24.67) - open/close 48581 (time 1709789028.13 total 1603.29 last 25.26) - open/close 48830 (time 1709789038.17 total 1613.33 last 24.81) - open/close 49080 (time 1709789048.20 total 1623.36 last 24.93) - open/close 49326 (time 1709789058.21 total 1633.37 last 24.56) - open/close 49571 (time 1709789068.23 total 1643.39 last 24.46) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1644.55 seconds: 30.16 ops/second - unlinked 0 (time 1709789070 ; total 0 ; last 0) - unlinked 10000 (time 1709789549 ; total 479 ; last 479) - unlinked 20000 (time 1709789911 ; total 841 ; last 362) - unlinked 30000 (time 1709790094 ; total 1024 ; last 183) - unlinked 40000 (time 1709790157 ; total 1087 ; last 63) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1111 seconds: 44.644463 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg202-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg202-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg202-server stop mds service on oleg202-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg202-server stop mds service on oleg202-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg202-server unloading modules on: 'oleg202-server' oleg202-server: oleg202-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2 pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2 pdsh@oleg202-client: oleg202-client: ssh exited with exit code 2 pdsh@oleg202-client: oleg202-server: ssh exited with exit code 2