== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:20:02 (1746141602) start mds service on oleg151-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing load_modules_local oleg151-server: Loading modules from /home/green/git/lustre-release/lustre oleg151-server: detected 4 online CPUs by sysfs oleg151-server: MODOPTS_LIBCFS= oleg151-server: Force libcfs to create 2 CPU partitions oleg151-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg151-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg151-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg151-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 207 (time 1746141668.79 total 10.53 last 19.65) - open/close 379 (time 1746141679.75 total 21.49 last 15.69) - open/close 479 (time 1746141689.79 total 31.54 last 9.95) - open/close 623 (time 1746141700.67 total 42.42 last 13.23) - open/close 772 (time 1746141712.45 total 54.19 last 12.66) - open/close 967 (time 1746141723.44 total 65.18 last 17.74) - open/close 1146 (time 1746141734.15 total 75.89 last 16.71) - open/close 1313 (time 1746141744.35 total 86.10 last 16.37) - open/close 1434 (time 1746141755.20 total 96.95 last 11.15) - open/close 1618 (time 1746141765.44 total 107.18 last 17.98) - open/close 1793 (time 1746141775.65 total 117.40 last 17.13) - open/close 1905 (time 1746141785.98 total 127.72 last 10.85) - open/close 2066 (time 1746141796.30 total 138.05 last 15.59) - open/close 2205 (time 1746141806.87 total 148.61 last 13.15) - open/close 2336 (time 1746141817.41 total 159.15 last 12.43) - open/close 2434 (time 1746141827.72 total 169.46 last 9.51) - open/close 2677 (time 1746141838.48 total 180.22 last 22.58) - open/close 3020 (time 1746141849.27 total 191.02 last 31.78) - open/close 3241 (time 1746141859.36 total 201.11 last 21.90) - open/close 3300 (time 1746141870.19 total 211.94 last 5.45) - open/close 3430 (time 1746141880.55 total 222.30 last 12.55) - open/close 3611 (time 1746141890.89 total 232.63 last 17.51) - open/close 3820 (time 1746141901.48 total 243.23 last 19.73) - open/close 3963 (time 1746141912.16 total 253.90 last 13.40) - open/close 4062 (time 1746141922.39 total 264.14 last 9.67) - open/close 4175 (time 1746141932.61 total 274.35 last 11.06) - open/close 4343 (time 1746141942.62 total 284.36 last 16.79) - open/close 4718 (time 1746141952.62 total 294.36 last 37.50) - open/close 4923 (time 1746141963.42 total 305.17 last 18.97) - open/close 5100 (time 1746141974.16 total 315.91 last 16.48) - open/close 5221 (time 1746141984.50 total 326.25 last 11.71) - open/close 5528 (time 1746141995.22 total 336.97 last 28.64) - open/close 5705 (time 1746142005.27 total 347.02 last 17.61) - open/close 5981 (time 1746142015.89 total 357.64 last 25.98) - open/close 6201 (time 1746142026.21 total 367.96 last 21.32) - open/close 6455 (time 1746142036.21 total 377.96 last 25.39) - open/close 6624 (time 1746142046.31 total 388.06 last 16.74) - open/close 6795 (time 1746142056.60 total 398.35 last 16.62) - open/close 6957 (time 1746142066.63 total 408.37 last 16.16) - open/close 7126 (time 1746142076.79 total 418.54 last 16.63) - open/close 7308 (time 1746142087.51 total 429.25 last 16.98) - open/close 7529 (time 1746142097.52 total 439.26 last 22.08) - open/close 7628 (time 1746142107.61 total 449.36 last 9.81) - open/close 7798 (time 1746142118.13 total 459.88 last 16.16) - open/close 7960 (time 1746142130.90 total 472.65 last 12.68) - open/close 8101 (time 1746142141.49 total 483.24 last 13.31) - open/close 8261 (time 1746142151.98 total 493.73 last 15.26) - open/close 8440 (time 1746142162.83 total 504.57 last 16.50) - open/close 8644 (time 1746142172.83 total 514.58 last 20.40) - open/close 8829 (time 1746142183.54 total 525.29 last 17.27) - open/close 8985 (time 1746142193.83 total 535.57 last 15.17) - open/close 9116 (time 1746142204.39 total 546.14 last 12.40) - open/close 9383 (time 1746142215.05 total 556.80 last 25.05) - open/close 9589 (time 1746142226.04 total 567.78 last 18.75) - open/close 10000 (time 1746142229.19 total 570.93 last 130.45) - open/close 11220 (time 1746142239.19 total 580.94 last 121.94) - open/close 12451 (time 1746142249.20 total 590.94 last 123.08) - open/close 13605 (time 1746142259.20 total 600.95 last 115.30) - open/close 14632 (time 1746142269.21 total 610.96 last 102.64) - open/close 15602 (time 1746142279.22 total 620.96 last 96.94) - open/close 16487 (time 1746142289.23 total 630.97 last 88.40) - open/close 17381 (time 1746142299.23 total 640.98 last 89.33) - open/close 18460 (time 1746142309.24 total 650.99 last 107.84) - open/close 19617 (time 1746142319.24 total 660.99 last 115.68) - open/close 20000 (time 1746142322.70 total 664.45 last 110.73) - open/close 21054 (time 1746142332.70 total 674.45 last 105.37) - open/close 22394 (time 1746142342.72 total 684.46 last 133.83) - open/close 23521 (time 1746142352.72 total 694.46 last 112.67) - open/close 24736 (time 1746142362.72 total 704.47 last 121.45) - open/close 25845 (time 1746142372.73 total 714.47 last 110.84) - open/close 26909 (time 1746142382.73 total 724.48 last 106.36) - open/close 27987 (time 1746142392.73 total 734.48 last 107.77) - open/close 29078 (time 1746142402.74 total 744.49 last 109.00) - open/close 30000 (time 1746142410.75 total 752.49 last 115.20) - open/close 31024 (time 1746142420.76 total 762.51 last 102.26) - open/close 32017 (time 1746142430.76 total 772.51 last 99.28) - open/close 33025 (time 1746142440.76 total 782.51 last 100.80) - open/close 34001 (time 1746142450.76 total 792.51 last 97.57) - open/close 34952 (time 1746142460.77 total 802.52 last 95.02) - open/close 35905 (time 1746142470.79 total 812.53 last 95.16) - open/close 36985 (time 1746142480.79 total 822.54 last 107.92) - open/close 38018 (time 1746142490.80 total 832.54 last 103.29) - open/close 39117 (time 1746142500.80 total 842.55 last 109.82) - open/close 40000 (time 1746142510.06 total 851.81 last 95.34) - open/close 41090 (time 1746142520.07 total 861.82 last 108.94) - open/close 42149 (time 1746142530.08 total 871.83 last 105.78) - open/close 43357 (time 1746142540.08 total 881.83 last 120.79) - open/close 44488 (time 1746142550.09 total 891.83 last 113.05) - open/close 45657 (time 1746142560.09 total 901.84 last 116.80) - open/close 46891 (time 1746142570.10 total 911.85 last 123.29) - open/close 47925 (time 1746142580.11 total 921.85 last 103.36) - open/close 48934 (time 1746142590.11 total 931.86 last 100.85) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 938.59 seconds: 52.88 ops/second - unlinked 0 (time 1746142599 ; total 1 ; last 1) - unlinked 10000 (time 1746142685 ; total 87 ; last 86) - unlinked 20000 (time 1746142742 ; total 144 ; last 57) - unlinked 30000 (time 1746142796 ; total 198 ; last 54) - unlinked 40000 (time 1746142858 ; total 260 ; last 62) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 319 seconds: 155.586212 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg151-server unloading modules via unload_modules_local on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2