== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:22:48 (1746073368) start mds service on oleg332-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: detected 4 online CPUs by sysfs oleg332-server: MODOPTS_LIBCFS= oleg332-server: Force libcfs to create 2 CPU partitions oleg332-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg332-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg332-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg332-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 251 (time 1746073434.72 total 10.21 last 24.58) - open/close 444 (time 1746073445.66 total 21.15 last 17.64) - open/close 645 (time 1746073455.74 total 31.24 last 19.93) - open/close 896 (time 1746073466.28 total 41.77 last 23.82) - open/close 1133 (time 1746073476.73 total 52.22 last 22.69) - open/close 1302 (time 1746073486.93 total 62.42 last 16.56) - open/close 1550 (time 1746073497.06 total 72.55 last 24.49) - open/close 1699 (time 1746073508.60 total 84.09 last 12.91) - open/close 1829 (time 1746073519.37 total 94.86 last 12.07) - open/close 2031 (time 1746073529.66 total 105.15 last 19.63) - open/close 2262 (time 1746073539.90 total 115.39 last 22.56) - open/close 2485 (time 1746073549.94 total 125.43 last 22.21) - open/close 2716 (time 1746073560.78 total 136.27 last 21.31) - open/close 2904 (time 1746073570.85 total 146.34 last 18.66) - open/close 3102 (time 1746073580.86 total 156.35 last 19.78) - open/close 3188 (time 1746073591.10 total 166.60 last 8.40) - open/close 3347 (time 1746073601.55 total 177.04 last 15.22) - open/close 3651 (time 1746073611.55 total 187.05 last 30.38) - open/close 3809 (time 1746073622.20 total 197.70 last 14.84) - open/close 4016 (time 1746073633.19 total 208.69 last 18.83) - open/close 4286 (time 1746073643.85 total 219.34 last 25.34) - open/close 4488 (time 1746073654.12 total 229.61 last 19.67) - open/close 4616 (time 1746073664.32 total 239.82 last 12.54) - open/close 4891 (time 1746073674.81 total 250.30 last 26.23) - open/close 5186 (time 1746073685.72 total 261.22 last 27.03) - open/close 5438 (time 1746073696.50 total 272.00 last 23.37) - open/close 5662 (time 1746073707.24 total 282.73 last 20.86) - open/close 5857 (time 1746073717.43 total 292.92 last 19.14) - open/close 5939 (time 1746073728.61 total 304.11 last 7.33) - open/close 6130 (time 1746073739.37 total 314.87 last 17.75) - open/close 6304 (time 1746073749.55 total 325.04 last 17.11) - open/close 6498 (time 1746073760.89 total 336.39 last 17.10) - open/close 6691 (time 1746073770.95 total 346.44 last 19.19) - open/close 6872 (time 1746073781.58 total 357.07 last 17.04) - open/close 7049 (time 1746073792.09 total 367.58 last 16.83) - open/close 7300 (time 1746073802.33 total 377.82 last 24.51) - open/close 7519 (time 1746073813.23 total 388.72 last 20.09) - open/close 7780 (time 1746073825.06 total 400.56 last 22.06) - open/close 7959 (time 1746073835.57 total 411.06 last 17.04) - open/close 8202 (time 1746073845.68 total 421.17 last 24.04) - open/close 8534 (time 1746073855.68 total 431.18 last 33.18) - open/close 8768 (time 1746073866.61 total 442.10 last 21.42) - open/close 9005 (time 1746073876.80 total 452.29 last 23.25) - open/close 9273 (time 1746073887.58 total 463.08 last 24.85) - open/close 10000 (time 1746073897.35 total 472.85 last 74.42) - open/close 11696 (time 1746073907.36 total 482.85 last 169.55) - open/close 13106 (time 1746073917.36 total 492.85 last 140.95) - open/close 14441 (time 1746073927.36 total 502.86 last 133.46) - open/close 15641 (time 1746073937.37 total 512.86 last 119.94) - open/close 16842 (time 1746073947.38 total 522.87 last 119.97) - open/close 17885 (time 1746073957.38 total 532.87 last 104.26) - open/close 19084 (time 1746073967.38 total 542.88 last 119.85) - open/close 20000 (time 1746073973.06 total 548.56 last 161.30) - open/close 21306 (time 1746073983.07 total 558.56 last 130.56) - open/close 22550 (time 1746073993.07 total 568.56 last 124.38) - open/close 23707 (time 1746074003.07 total 578.57 last 115.63) - open/close 24990 (time 1746074013.08 total 588.57 last 128.24) - open/close 26195 (time 1746074023.08 total 598.58 last 120.44) - open/close 27393 (time 1746074033.09 total 608.58 last 119.73) - open/close 28601 (time 1746074043.10 total 618.59 last 120.74) - open/close 29805 (time 1746074053.10 total 628.59 last 120.40) - open/close 31144 (time 1746074063.10 total 638.59 last 133.84) - open/close 32253 (time 1746074073.11 total 648.60 last 110.84) - open/close 33442 (time 1746074083.11 total 658.60 last 118.89) - open/close 34596 (time 1746074093.11 total 668.61 last 115.31) - open/close 35716 (time 1746074103.12 total 678.61 last 111.95) - open/close 36983 (time 1746074113.12 total 688.62 last 126.64) - open/close 38200 (time 1746074123.13 total 698.62 last 121.61) - open/close 39535 (time 1746074133.13 total 708.62 last 133.48) - open/close 40000 (time 1746074136.55 total 712.05 last 135.85) - open/close 41256 (time 1746074146.56 total 722.05 last 125.52) - open/close 42470 (time 1746074156.56 total 732.06 last 121.37) - open/close 43666 (time 1746074166.57 total 742.06 last 119.54) - open/close 44745 (time 1746074176.57 total 752.06 last 107.88) - open/close 45838 (time 1746074186.57 total 762.06 last 109.29) - open/close 46954 (time 1746074196.57 total 772.07 last 111.56) - open/close 48158 (time 1746074206.58 total 782.07 last 120.37) - open/close 49189 (time 1746074216.59 total 792.08 last 102.94) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 796.30 seconds: 62.33 ops/second - unlinked 0 (time 1746074222 ; total 0 ; last 0) - unlinked 10000 (time 1746074294 ; total 72 ; last 72) - unlinked 20000 (time 1746074351 ; total 129 ; last 57) - unlinked 30000 (time 1746074400 ; total 178 ; last 49) - unlinked 40000 (time 1746074450 ; total 228 ; last 50) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 272 seconds: 182.470581 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg332-server unloading modules via unload_modules_local on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2