== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:48:12 (1709786892) start mds service on oleg209-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg209-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock oleg209-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 240 (time 1709786924.00 total 10.01 last 23.98) - open/close 370 (time 1709786934.58 total 20.59 last 12.29) - open/close 567 (time 1709786944.91 total 30.92 last 19.07) - open/close 772 (time 1709786955.43 total 41.44 last 19.49) - open/close 1089 (time 1709786966.24 total 52.24 last 29.34) - open/close 1339 (time 1709786976.38 total 62.39 last 24.63) - open/close 1650 (time 1709786986.40 total 72.41 last 31.05) - open/close 1859 (time 1709786997.23 total 83.24 last 19.29) - open/close 2051 (time 1709787007.24 total 93.25 last 19.18) - open/close 2216 (time 1709787017.99 total 104.00 last 15.35) - open/close 2400 (time 1709787030.06 total 116.07 last 15.24) - open/close 2548 (time 1709787040.07 total 126.08 last 14.78) - open/close 2800 (time 1709787050.61 total 136.61 last 23.93) - open/close 3010 (time 1709787060.62 total 146.62 last 20.98) - open/close 3126 (time 1709787070.62 total 156.63 last 11.59) - open/close 3306 (time 1709787080.62 total 166.63 last 18.00) - open/close 3519 (time 1709787091.34 total 177.35 last 19.87) - open/close 3743 (time 1709787101.97 total 187.98 last 21.07) - open/close 3921 (time 1709787112.86 total 198.87 last 16.34) - open/close 3971 (time 1709787125.91 total 211.91 last 3.83) - open/close 4148 (time 1709787135.91 total 221.92 last 17.69) - open/close 4353 (time 1709787145.92 total 231.92 last 20.49) - open/close 4396 (time 1709787157.87 total 243.88 last 3.60) - open/close 4554 (time 1709787167.89 total 253.89 last 15.78) - open/close 4767 (time 1709787178.54 total 264.54 last 20.00) - open/close 4939 (time 1709787188.65 total 274.65 last 17.01) - open/close 5072 (time 1709787199.36 total 285.36 last 12.42) - open/close 5246 (time 1709787209.84 total 295.85 last 16.59) - open/close 5414 (time 1709787219.92 total 305.93 last 16.67) - open/close 5519 (time 1709787229.94 total 315.94 last 10.48) - open/close 5662 (time 1709787239.97 total 325.97 last 14.26) - open/close 5819 (time 1709787250.76 total 336.77 last 14.54) - open/close 5953 (time 1709787261.01 total 347.02 last 13.07) - open/close 6129 (time 1709787271.02 total 357.03 last 17.59) - open/close 6355 (time 1709787282.58 total 368.59 last 19.54) - open/close 6543 (time 1709787292.84 total 378.85 last 18.33) - open/close 6730 (time 1709787302.85 total 388.86 last 18.67) - open/close 6889 (time 1709787313.48 total 399.49 last 14.96) - open/close 7129 (time 1709787324.03 total 410.03 last 22.77) - open/close 7255 (time 1709787334.04 total 420.05 last 12.58) - open/close 7496 (time 1709787344.79 total 430.80 last 22.41) - open/close 7722 (time 1709787354.80 total 440.81 last 22.57) - open/close 7900 (time 1709787364.99 total 450.99 last 17.48) - open/close 8051 (time 1709787375.72 total 461.72 last 14.07) - open/close 8218 (time 1709787385.72 total 471.73 last 16.69) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-8305) error: No space left on device total: 8305 open/close in 474.09 seconds: 17.52 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8267: kill: (28692) - No such process - unlinked 0 (time 1709787396 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-8305) error: No such file or directory total: 8305 unlinks in 79 seconds: 105.126579 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server unloading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2