== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:17:32 (1709785052) start mds service on oleg406-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg406-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock oleg406-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 768 (time 1709785079.25 total 10.55 last 72.82) - open/close 1271 (time 1709785089.25 total 20.55 last 50.30) - open/close 1623 (time 1709785100.00 total 31.30 last 32.74) - open/close 2206 (time 1709785110.22 total 41.52 last 57.04) - open/close 2867 (time 1709785120.59 total 51.88 last 63.79) - open/close 3434 (time 1709785130.87 total 62.16 last 55.16) - open/close 4067 (time 1709785141.37 total 72.66 last 60.29) - open/close 4656 (time 1709785151.90 total 83.19 last 55.93) - open/close 5280 (time 1709785162.67 total 93.97 last 57.90) - open/close 5783 (time 1709785173.26 total 104.55 last 47.51) - open/close 6295 (time 1709785183.91 total 115.20 last 48.09) - open/close 6783 (time 1709785194.85 total 126.14 last 44.59) - open/close 7116 (time 1709785204.85 total 136.15 last 33.29) - open/close 7693 (time 1709785215.78 total 147.08 last 52.79) - open/close 8139 (time 1709785226.26 total 157.56 last 42.56) - open/close 8573 (time 1709785236.77 total 168.06 last 41.30) - open/close 9016 (time 1709785247.28 total 178.57 last 42.16) - open/close 9454 (time 1709785257.89 total 189.19 last 41.27) - open/close 9893 (time 1709785268.60 total 199.90 last 40.99) - open/close 10332 (time 1709785278.62 total 209.91 last 43.85) - open/close 10667 (time 1709785289.11 total 220.40 last 31.93) - open/close 11090 (time 1709785299.11 total 230.40 last 42.28) - open/close 12215 (time 1709785309.11 total 240.41 last 112.44) - open/close 13993 (time 1709785319.12 total 250.41 last 177.72) - open/close 15760 (time 1709785329.12 total 260.41 last 176.65) - open/close 17301 (time 1709785339.13 total 270.42 last 154.05) - open/close 18769 (time 1709785349.13 total 280.42 last 146.72) - open/close 20000 (time 1709785358.14 total 289.43 last 136.70) - open/close 21286 (time 1709785368.14 total 299.43 last 128.53) - open/close 22351 (time 1709785378.15 total 309.44 last 106.46) - open/close 23321 (time 1709785388.16 total 319.45 last 96.91) - open/close 24270 (time 1709785398.16 total 329.45 last 94.85) - open/close 25237 (time 1709785408.17 total 339.46 last 96.62) - open/close 26148 (time 1709785418.17 total 349.46 last 91.08) - open/close 27102 (time 1709785428.17 total 359.46 last 95.40) - open/close 27935 (time 1709785438.18 total 369.47 last 83.25) - open/close 28771 (time 1709785448.18 total 379.47 last 83.58) - open/close 29658 (time 1709785458.18 total 389.47 last 88.68) - open/close 30000 (time 1709785462.25 total 393.54 last 84.05) - open/close 30794 (time 1709785472.25 total 403.55 last 79.38) - open/close 31510 (time 1709785482.26 total 413.55 last 71.56) - open/close 32218 (time 1709785492.27 total 423.56 last 70.72) - open/close 32925 (time 1709785502.29 total 433.58 last 70.56) - open/close 33588 (time 1709785512.30 total 443.59 last 66.22) - open/close 34264 (time 1709785522.31 total 453.60 last 67.58) - open/close 34935 (time 1709785532.32 total 463.61 last 67.04) - open/close 35588 (time 1709785542.32 total 473.61 last 65.28) - open/close 36276 (time 1709785552.33 total 483.63 last 68.69) - open/close 36984 (time 1709785562.33 total 493.63 last 70.79) - open/close 37688 (time 1709785572.35 total 503.64 last 70.30) - open/close 38326 (time 1709785582.36 total 513.65 last 63.74) - open/close 38868 (time 1709785592.37 total 523.66 last 54.15) - open/close 39465 (time 1709785602.37 total 533.66 last 59.67) - open/close 40000 (time 1709785611.26 total 542.55 last 60.21) - open/close 40568 (time 1709785621.27 total 552.56 last 56.71) - open/close 41099 (time 1709785631.28 total 562.57 last 53.08) - open/close 41689 (time 1709785641.29 total 572.59 last 58.90) - open/close 42246 (time 1709785651.30 total 582.59 last 55.69) - open/close 42754 (time 1709785661.31 total 592.60 last 50.72) - open/close 43281 (time 1709785671.32 total 602.62 last 52.63) - open/close 43805 (time 1709785681.34 total 612.63 last 52.31) - open/close 44315 (time 1709785691.36 total 622.65 last 50.91) - open/close 44850 (time 1709785701.36 total 632.65 last 53.48) - open/close 45333 (time 1709785711.38 total 642.67 last 48.21) - open/close 45824 (time 1709785721.39 total 652.68 last 49.06) - open/close 46320 (time 1709785731.39 total 662.68 last 49.58) - open/close 46814 (time 1709785741.40 total 672.69 last 49.36) - open/close 47293 (time 1709785751.40 total 682.69 last 47.89) - open/close 47776 (time 1709785761.42 total 692.72 last 48.19) - open/close 48249 (time 1709785771.43 total 702.72 last 47.27) - open/close 48753 (time 1709785781.44 total 712.73 last 50.34) - open/close 49227 (time 1709785791.45 total 722.74 last 47.37) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 732.49 seconds: 67.85 ops/second - unlinked 0 (time 1709785802 ; total 0 ; last 0) - unlinked 10000 (time 1709786051 ; total 249 ; last 249) - unlinked 20000 (time 1709786212 ; total 410 ; last 161) - unlinked 30000 (time 1709786325 ; total 523 ; last 113) - unlinked 40000 (time 1709786391 ; total 589 ; last 66) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 612 seconds: 81.205879 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server unloading modules on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2