== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:56:28 (1709787388) start mds service on oleg403-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg403-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg403-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock oleg403-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 154 (time 1709787443.49 total 10.42 last 14.78) - open/close 339 (time 1709787454.24 total 21.17 last 17.21) - open/close 539 (time 1709787464.26 total 31.19 last 19.96) - open/close 674 (time 1709787474.34 total 41.26 last 13.40) - open/close 897 (time 1709787484.47 total 51.40 last 22.01) - open/close 1083 (time 1709787494.47 total 61.40 last 18.60) - open/close 1253 (time 1709787504.49 total 71.42 last 16.96) - open/close 1419 (time 1709787515.25 total 82.18 last 15.43) - open/close 1594 (time 1709787526.12 total 93.05 last 16.09) - open/close 1693 (time 1709787536.94 total 103.87 last 9.15) - open/close 1820 (time 1709787546.95 total 113.87 last 12.70) - open/close 1891 (time 1709787557.03 total 123.96 last 7.04) - open/close 2005 (time 1709787567.44 total 134.36 last 10.96) - open/close 2238 (time 1709787577.74 total 144.67 last 22.61) - open/close 2565 (time 1709787588.03 total 154.96 last 31.79) - open/close 2880 (time 1709787598.03 total 164.96 last 31.49) - open/close 3070 (time 1709787608.51 total 175.43 last 18.14) - open/close 3271 (time 1709787618.84 total 185.77 last 19.44) - open/close 3469 (time 1709787629.60 total 196.53 last 18.41) - open/close 3745 (time 1709787639.61 total 206.54 last 27.57) - open/close 3902 (time 1709787649.97 total 216.90 last 15.16) - open/close 4063 (time 1709787659.98 total 226.90 last 16.09) - open/close 4288 (time 1709787669.99 total 236.92 last 22.46) - open/close 4483 (time 1709787680.01 total 246.94 last 19.46) - open/close 4674 (time 1709787690.85 total 257.78 last 17.62) - open/close 4834 (time 1709787701.31 total 268.24 last 15.30) - open/close 5050 (time 1709787711.36 total 278.28 last 21.50) - open/close 5209 (time 1709787722.23 total 289.16 last 14.62) - open/close 5502 (time 1709787732.55 total 299.47 last 28.40) - open/close 5763 (time 1709787742.97 total 309.90 last 25.03) - open/close 6002 (time 1709787754.45 total 321.38 last 20.82) - open/close 6272 (time 1709787764.46 total 331.39 last 26.97) - open/close 6396 (time 1709787774.80 total 341.72 last 12.00) - open/close 6653 (time 1709787784.85 total 351.78 last 25.56) - open/close 6848 (time 1709787794.87 total 361.79 last 19.47) - open/close 7135 (time 1709787805.79 total 372.72 last 26.27) - open/close 7433 (time 1709787816.72 total 383.65 last 27.26) - open/close 7707 (time 1709787826.73 total 393.66 last 27.39) - open/close 7947 (time 1709787836.93 total 403.86 last 23.52) - open/close 8225 (time 1709787847.48 total 414.40 last 26.37) - open/close 8434 (time 1709787857.61 total 424.54 last 20.62) - open/close 8660 (time 1709787867.83 total 434.76 last 22.11) - open/close 8849 (time 1709787877.84 total 444.76 last 18.89) - open/close 9130 (time 1709787888.54 total 455.47 last 26.26) - open/close 9438 (time 1709787899.40 total 466.33 last 28.35) - open/close 9615 (time 1709787909.46 total 476.38 last 17.61) - open/close 10000 (time 1709787918.44 total 485.37 last 42.86) - open/close 10644 (time 1709787928.45 total 495.38 last 64.30) - open/close 11251 (time 1709787938.46 total 505.38 last 60.70) - open/close 11833 (time 1709787948.46 total 515.39 last 58.15) - open/close 12399 (time 1709787958.48 total 525.41 last 56.52) - open/close 12914 (time 1709787968.49 total 535.42 last 51.44) - open/close 13409 (time 1709787978.50 total 545.42 last 49.48) - open/close 13919 (time 1709787988.50 total 555.43 last 50.97) - open/close 14449 (time 1709787998.52 total 565.44 last 52.92) - open/close 14901 (time 1709788008.53 total 575.46 last 45.14) - open/close 15348 (time 1709788018.54 total 585.47 last 44.63) - open/close 15815 (time 1709788028.56 total 595.49 last 46.61) - open/close 16275 (time 1709788038.56 total 605.49 last 46.00) - open/close 16712 (time 1709788048.58 total 615.51 last 43.62) - open/close 17117 (time 1709788058.60 total 625.53 last 40.42) - open/close 17541 (time 1709788068.62 total 635.54 last 42.34) - open/close 18004 (time 1709788078.64 total 645.57 last 46.19) - open/close 18464 (time 1709788088.64 total 655.57 last 45.98) - open/close 18907 (time 1709788098.65 total 665.58 last 44.28) - open/close 19330 (time 1709788108.65 total 675.58 last 42.28) - open/close 19751 (time 1709788118.66 total 685.58 last 42.08) - open/close 20000 (time 1709788124.22 total 691.15 last 44.77) - open/close 20380 (time 1709788134.22 total 701.15 last 37.99) - open/close 20763 (time 1709788144.24 total 711.16 last 38.24) - open/close 21126 (time 1709788154.25 total 721.17 last 36.26) - open/close 21473 (time 1709788164.28 total 731.21 last 34.58) - open/close 21836 (time 1709788174.29 total 741.22 last 36.25) - open/close 22231 (time 1709788184.30 total 751.23 last 39.47) - open/close 22626 (time 1709788194.31 total 761.24 last 39.46) - open/close 22945 (time 1709788204.33 total 771.26 last 31.84) - open/close 23310 (time 1709788214.35 total 781.27 last 36.44) - open/close 23665 (time 1709788224.37 total 791.30 last 35.43) - open/close 24010 (time 1709788234.38 total 801.31 last 34.46) - open/close 24319 (time 1709788244.39 total 811.32 last 30.86) - open/close 24674 (time 1709788254.41 total 821.34 last 35.42) - open/close 25068 (time 1709788264.42 total 831.35 last 39.35) - open/close 25398 (time 1709788274.44 total 841.36 last 32.96) - open/close 25734 (time 1709788284.45 total 851.38 last 33.55) - open/close 26074 (time 1709788294.46 total 861.39 last 33.96) - open/close 26417 (time 1709788304.48 total 871.40 last 34.24) - open/close 26754 (time 1709788314.49 total 881.41 last 33.67) - open/close 27072 (time 1709788324.49 total 891.42 last 31.78) - open/close 27414 (time 1709788334.51 total 901.44 last 34.14) - open/close 27733 (time 1709788344.53 total 911.46 last 31.84) - open/close 28023 (time 1709788354.56 total 921.48 last 28.93) - open/close 28300 (time 1709788364.56 total 931.49 last 27.68) - open/close 28611 (time 1709788374.58 total 941.51 last 31.04) - open/close 28927 (time 1709788384.59 total 951.52 last 31.56) - open/close 29234 (time 1709788394.60 total 961.52 last 30.69) - open/close 29546 (time 1709788404.61 total 971.54 last 31.16) - open/close 29842 (time 1709788414.62 total 981.55 last 29.56) - open/close 30000 (time 1709788420.50 total 987.43 last 26.88) - open/close 30286 (time 1709788430.51 total 997.43 last 28.59) - open/close 30564 (time 1709788440.52 total 1007.45 last 27.77) - open/close 30827 (time 1709788450.54 total 1017.47 last 26.24) - open/close 31102 (time 1709788460.57 total 1027.50 last 27.41) - open/close 31375 (time 1709788470.58 total 1037.51 last 27.27) - open/close 31661 (time 1709788480.60 total 1047.53 last 28.54) - open/close 31960 (time 1709788490.61 total 1057.53 last 29.89) - open/close 32253 (time 1709788500.61 total 1067.54 last 29.28) - open/close 32580 (time 1709788510.64 total 1077.56 last 32.63) - open/close 32881 (time 1709788520.66 total 1087.59 last 30.01) - open/close 33135 (time 1709788530.67 total 1097.60 last 25.38) - open/close 33421 (time 1709788540.68 total 1107.61 last 28.58) - open/close 33725 (time 1709788550.70 total 1117.63 last 30.32) - open/close 34005 (time 1709788560.73 total 1127.66 last 27.93) - open/close 34291 (time 1709788570.76 total 1137.69 last 28.51) - open/close 34589 (time 1709788580.77 total 1147.70 last 29.78) - open/close 34879 (time 1709788590.77 total 1157.70 last 28.99) - open/close 35199 (time 1709788600.80 total 1167.73 last 31.92) - open/close 35506 (time 1709788610.80 total 1177.73 last 30.70) - open/close 35800 (time 1709788620.82 total 1187.75 last 29.33) - open/close 36141 (time 1709788630.85 total 1197.78 last 34.00) - open/close 36475 (time 1709788640.86 total 1207.78 last 33.39) - open/close 36779 (time 1709788650.86 total 1217.79 last 30.38) - open/close 37087 (time 1709788660.89 total 1227.81 last 30.72) - open/close 37363 (time 1709788670.92 total 1237.85 last 27.50) - open/close 37569 (time 1709788680.95 total 1247.88 last 20.54) - open/close 37785 (time 1709788690.99 total 1257.92 last 21.52) - open/close 38023 (time 1709788700.99 total 1267.92 last 23.79) - open/close 38324 (time 1709788711.02 total 1277.94 last 30.02) - open/close 38582 (time 1709788721.02 total 1287.95 last 25.78) - open/close 38884 (time 1709788731.05 total 1297.98 last 30.12) - open/close 39166 (time 1709788741.05 total 1307.98 last 28.20) - open/close 39416 (time 1709788751.08 total 1318.01 last 24.93) - open/close 39688 (time 1709788761.08 total 1328.01 last 27.19) - open/close 40000 (time 1709788771.07 total 1337.99 last 31.25) - open/close 40299 (time 1709788781.08 total 1348.01 last 29.86) - open/close 40566 (time 1709788791.10 total 1358.03 last 26.65) - open/close 40849 (time 1709788801.13 total 1368.06 last 28.20) - open/close 41135 (time 1709788811.15 total 1378.08 last 28.55) - open/close 41429 (time 1709788821.16 total 1388.09 last 29.37) - open/close 41722 (time 1709788831.19 total 1398.12 last 29.21) - open/close 42000 (time 1709788841.20 total 1408.12 last 27.79) - open/close 42251 (time 1709788851.20 total 1418.13 last 25.09) - open/close 42539 (time 1709788861.23 total 1428.16 last 28.72) - open/close 42850 (time 1709788871.26 total 1438.18 last 31.01) - open/close 43150 (time 1709788881.28 total 1448.21 last 29.93) - open/close 43441 (time 1709788891.29 total 1458.22 last 29.08) - open/close 43733 (time 1709788901.30 total 1468.23 last 29.17) - open/close 44020 (time 1709788911.32 total 1478.25 last 28.64) - open/close 44307 (time 1709788921.34 total 1488.26 last 28.66) - open/close 44589 (time 1709788931.36 total 1498.29 last 28.13) - open/close 44879 (time 1709788941.39 total 1508.31 last 28.93) - open/close 45157 (time 1709788951.40 total 1518.33 last 27.76) - open/close 45439 (time 1709788961.41 total 1528.34 last 28.16) - open/close 45720 (time 1709788971.42 total 1538.35 last 28.08) - open/close 45992 (time 1709788981.45 total 1548.38 last 27.12) - open/close 46246 (time 1709788991.47 total 1558.40 last 25.34) - open/close 46519 (time 1709789001.50 total 1568.43 last 27.23) - open/close 46771 (time 1709789011.53 total 1578.46 last 25.13) - open/close 47018 (time 1709789021.53 total 1588.46 last 24.69) - open/close 47263 (time 1709789031.56 total 1598.49 last 24.43) - open/close 47503 (time 1709789041.60 total 1608.52 last 23.92) - open/close 47766 (time 1709789051.60 total 1618.53 last 26.29) - open/close 48022 (time 1709789061.62 total 1628.55 last 25.54) - open/close 48285 (time 1709789071.66 total 1638.58 last 26.21) - open/close 48544 (time 1709789081.67 total 1648.59 last 25.87) - open/close 48796 (time 1709789091.67 total 1658.60 last 25.18) - open/close 49056 (time 1709789101.70 total 1668.63 last 25.93) - open/close 49317 (time 1709789111.72 total 1678.64 last 26.07) - open/close 49575 (time 1709789121.73 total 1688.66 last 25.77) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1689.64 seconds: 29.36 ops/second - unlinked 0 (time 1709789123 ; total 0 ; last 0) - unlinked 10000 (time 1709789559 ; total 436 ; last 436) - unlinked 20000 (time 1709789859 ; total 736 ; last 300) - unlinked 30000 (time 1709790016 ; total 893 ; last 157) - unlinked 40000 (time 1709790088 ; total 965 ; last 72) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1007 seconds: 49.255215 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server unloading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2