== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:01:03 (1708840863) start mds service on oleg201-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing load_modules_local oleg201-server: Loading modules from /home/green/git/lustre-release/lustre oleg201-server: detected 4 online CPUs by sysfs oleg201-server: Force libcfs to create 2 CPU partitions oleg201-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg201-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg201-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg201-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock oleg201-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 336 (time 1708840904.49 total 10.34 last 32.49) - open/close 870 (time 1708840915.24 total 21.09 last 49.69) - open/close 1297 (time 1708840925.47 total 31.32 last 41.71) - open/close 1825 (time 1708840936.19 total 42.05 last 49.25) - open/close 2246 (time 1708840946.61 total 52.46 last 40.43) - open/close 2494 (time 1708840957.46 total 63.31 last 22.84) - open/close 2792 (time 1708840967.55 total 73.40 last 29.55) - open/close 3077 (time 1708840978.52 total 84.37 last 25.97) - open/close 3494 (time 1708840989.25 total 95.10 last 38.87) - open/close 3991 (time 1708840999.25 total 105.10 last 49.70) - open/close 4417 (time 1708841009.25 total 115.11 last 42.58) - open/close 4656 (time 1708841019.26 total 125.12 last 23.88) - open/close 5073 (time 1708841030.25 total 136.10 last 37.95) - open/close 5274 (time 1708841041.31 total 147.16 last 18.19) - open/close 5692 (time 1708841052.26 total 158.11 last 38.16) - open/close 6135 (time 1708841063.16 total 169.01 last 40.63) - open/close 6595 (time 1708841073.17 total 179.02 last 45.98) - open/close 6969 (time 1708841083.97 total 189.82 last 34.63) - open/close 7444 (time 1708841094.35 total 200.20 last 45.75) - open/close 7833 (time 1708841105.10 total 210.95 last 36.20) - open/close 8251 (time 1708841115.10 total 220.95 last 41.79) - open/close 8645 (time 1708841125.10 total 230.95 last 39.40) - open/close 9003 (time 1708841135.10 total 240.95 last 35.80) - open/close 9402 (time 1708841145.26 total 251.11 last 39.28) - open/close 9644 (time 1708841155.61 total 261.46 last 23.38) - open/close 9965 (time 1708841167.51 total 273.36 last 26.98) - open/close 10302 (time 1708841179.41 total 285.26 last 28.31) - open/close 10524 (time 1708841189.76 total 295.62 last 21.44) - open/close 10813 (time 1708841200.56 total 306.41 last 26.78) - open/close 12139 (time 1708841210.56 total 316.41 last 132.58) - open/close 13748 (time 1708841220.56 total 326.42 last 160.80) - open/close 15342 (time 1708841230.57 total 336.42 last 159.34) - open/close 16867 (time 1708841240.57 total 346.43 last 152.40) - open/close 18476 (time 1708841250.58 total 356.43 last 160.87) - open/close 19982 (time 1708841260.58 total 366.43 last 150.57) - open/close 21467 (time 1708841270.58 total 376.43 last 148.49) - open/close 22733 (time 1708841280.58 total 386.43 last 126.56) - open/close 24207 (time 1708841290.58 total 396.43 last 147.39) - open/close 25738 (time 1708841300.59 total 406.44 last 153.05) - open/close 27093 (time 1708841310.59 total 416.44 last 135.49) - open/close 28437 (time 1708841320.59 total 426.44 last 134.37) - open/close 29778 (time 1708841330.59 total 436.44 last 134.10) - open/close 31206 (time 1708841340.60 total 446.45 last 142.69) - open/close 32321 (time 1708841350.61 total 456.46 last 111.42) - open/close 33429 (time 1708841360.61 total 466.46 last 110.71) - open/close 34566 (time 1708841370.61 total 476.47 last 113.69) - open/close 35702 (time 1708841380.62 total 486.47 last 113.58) - open/close 36798 (time 1708841390.62 total 496.47 last 109.59) - open/close 37983 (time 1708841400.62 total 506.47 last 118.47) - open/close 39001 (time 1708841410.63 total 516.48 last 101.72) - open/close 39858 (time 1708841420.63 total 526.49 last 85.65) - open/close 40703 (time 1708841430.64 total 536.49 last 84.45) - open/close 41499 (time 1708841440.65 total 546.50 last 79.54) - open/close 42265 (time 1708841450.66 total 556.51 last 76.54) - open/close 43011 (time 1708841460.66 total 566.51 last 74.55) - open/close 43758 (time 1708841470.66 total 576.51 last 74.69) - open/close 44490 (time 1708841480.67 total 586.52 last 73.18) - open/close 45201 (time 1708841490.68 total 596.53 last 71.03) - open/close 45888 (time 1708841500.68 total 606.53 last 68.69) - open/close 46559 (time 1708841510.68 total 616.53 last 67.07) - open/close 47212 (time 1708841520.70 total 626.55 last 65.20) - open/close 47899 (time 1708841530.70 total 636.55 last 68.69) - open/close 48555 (time 1708841540.70 total 646.55 last 65.59) - open/close 49247 (time 1708841550.71 total 656.56 last 69.15) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 660.40 seconds: 75.15 ops/second - unlinked 0 (time 1708841555 ; total 0 ; last 0) - unlinked 10000 (time 1708841662 ; total 107 ; last 107) - unlinked 20000 (time 1708841756 ; total 201 ; last 94) - unlinked 30000 (time 1708841833 ; total 278 ; last 77) - unlinked 40000 (time 1708841888 ; total 333 ; last 55) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 364 seconds: 136.351654 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg201-server unloading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2