== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:55:39 (1746140139) start mds service on oleg227-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing load_modules_local oleg227-server: Loading modules from /home/green/git/lustre-release/lustre oleg227-server: detected 4 online CPUs by sysfs oleg227-server: MODOPTS_LIBCFS= oleg227-server: Force libcfs to create 2 CPU partitions oleg227-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg227-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg227-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg227-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 196 (time 1746140203.78 total 10.01 last 19.58) - open/close 336 (time 1746140214.23 total 20.47 last 13.39) - open/close 467 (time 1746140225.11 total 31.35 last 12.04) - open/close 632 (time 1746140235.81 total 42.04 last 15.43) - open/close 811 (time 1746140246.49 total 52.73 last 16.75) - open/close 1004 (time 1746140256.50 total 62.74 last 19.28) - open/close 1143 (time 1746140266.51 total 72.75 last 13.89) - open/close 1293 (time 1746140276.71 total 82.95 last 14.70) - open/close 1400 (time 1746140287.38 total 93.61 last 10.04) - open/close 1529 (time 1746140298.34 total 104.58 last 11.77) - open/close 1662 (time 1746140308.43 total 114.66 last 13.19) - open/close 1836 (time 1746140318.84 total 125.07 last 16.71) - open/close 1988 (time 1746140329.03 total 135.26 last 14.92) - open/close 2121 (time 1746140340.00 total 146.24 last 12.12) - open/close 2265 (time 1746140350.34 total 156.58 last 13.93) - open/close 2348 (time 1746140360.66 total 166.89 last 8.05) - open/close 2543 (time 1746140370.93 total 177.17 last 18.98) - open/close 2762 (time 1746140381.49 total 187.73 last 20.73) - open/close 2968 (time 1746140391.72 total 197.96 last 20.14) - open/close 3051 (time 1746140402.20 total 208.43 last 7.93) - open/close 3206 (time 1746140412.89 total 219.13 last 14.49) - open/close 3333 (time 1746140423.62 total 229.85 last 11.84) - open/close 3467 (time 1746140433.68 total 239.92 last 13.32) - open/close 3548 (time 1746140443.91 total 250.15 last 7.92) - open/close 3620 (time 1746140454.01 total 260.24 last 7.13) - open/close 3841 (time 1746140465.01 total 271.25 last 20.09) - open/close 3977 (time 1746140475.02 total 281.25 last 13.59) - open/close 4153 (time 1746140485.82 total 292.05 last 16.29) - open/close 4210 (time 1746140496.77 total 303.00 last 5.21) - open/close 4374 (time 1746140506.87 total 313.11 last 16.23) - open/close 4504 (time 1746140517.80 total 324.04 last 11.89) - open/close 4591 (time 1746140528.01 total 334.25 last 8.52) - open/close 4747 (time 1746140538.09 total 344.33 last 15.48) - open/close 4836 (time 1746140548.33 total 354.56 last 8.69) - open/close 4954 (time 1746140558.99 total 365.23 last 11.06) - open/close 5185 (time 1746140569.63 total 375.87 last 21.71) - open/close 5367 (time 1746140580.48 total 386.72 last 16.78) - open/close 5535 (time 1746140591.47 total 397.71 last 15.28) - open/close 5607 (time 1746140601.53 total 407.76 last 7.16) - open/close 5685 (time 1746140611.55 total 417.78 last 7.78) - open/close 5955 (time 1746140621.55 total 427.79 last 26.99) - open/close 6032 (time 1746140631.55 total 437.79 last 7.70) - open/close 6328 (time 1746140641.56 total 447.80 last 29.58) - open/close 6416 (time 1746140651.74 total 457.98 last 8.64) - open/close 6637 (time 1746140663.09 total 469.32 last 19.48) - open/close 6733 (time 1746140673.27 total 479.51 last 9.42) - open/close 6813 (time 1746140684.30 total 490.54 last 7.25) - open/close 7009 (time 1746140695.22 total 501.45 last 17.96) - open/close 7162 (time 1746140705.84 total 512.07 last 14.40) - open/close 7316 (time 1746140717.96 total 524.20 last 12.70) - open/close 7477 (time 1746140727.97 total 534.20 last 16.09) - open/close 7621 (time 1746140738.35 total 544.58 last 13.87) - open/close 7825 (time 1746140748.56 total 554.80 last 19.97) - open/close 8812 (time 1746140758.57 total 564.80 last 98.63) - open/close 9826 (time 1746140768.57 total 574.81 last 101.34) - open/close 11049 (time 1746140778.58 total 584.81 last 122.26) - open/close 12043 (time 1746140788.58 total 594.82 last 99.34) - open/close 13091 (time 1746140798.59 total 604.82 last 104.76) - open/close 14157 (time 1746140808.59 total 614.83 last 106.56) - open/close 15377 (time 1746140818.59 total 624.83 last 121.98) - open/close 16412 (time 1746140828.59 total 634.83 last 103.49) - open/close 17450 (time 1746140838.60 total 644.84 last 103.71) - open/close 18480 (time 1746140848.60 total 654.84 last 102.96) - open/close 19615 (time 1746140858.61 total 664.85 last 113.44) - open/close 20000 (time 1746140861.86 total 668.10 last 118.31) - open/close 21004 (time 1746140871.87 total 678.11 last 100.32) - open/close 21933 (time 1746140881.88 total 688.12 last 92.81) - open/close 22874 (time 1746140891.89 total 698.13 last 94.00) - open/close 23940 (time 1746140901.89 total 708.13 last 106.58) - open/close 25011 (time 1746140911.90 total 718.14 last 107.00) - open/close 26025 (time 1746140921.91 total 728.15 last 101.30) - open/close 27015 (time 1746140931.92 total 738.15 last 98.96) - open/close 28022 (time 1746140941.92 total 748.15 last 100.69) - open/close 29209 (time 1746140951.93 total 758.17 last 118.52) - open/close 30000 (time 1746140958.47 total 764.70 last 121.08) - open/close 31080 (time 1746140968.47 total 774.71 last 107.94) - open/close 32170 (time 1746140978.47 total 784.71 last 108.98) - open/close 33322 (time 1746140988.48 total 794.71 last 115.15) - open/close 34246 (time 1746140998.48 total 804.72 last 92.35) - open/close 35172 (time 1746141008.49 total 814.72 last 92.57) - open/close 36068 (time 1746141018.49 total 824.72 last 89.58) - open/close 37064 (time 1746141028.49 total 834.73 last 99.54) - open/close 38030 (time 1746141038.51 total 844.74 last 96.48) - open/close 39020 (time 1746141048.51 total 854.75 last 98.92) - open/close 40000 (time 1746141057.97 total 864.21 last 103.65) - open/close 41064 (time 1746141067.97 total 874.21 last 106.36) - open/close 42155 (time 1746141077.98 total 884.22 last 109.02) - open/close 43110 (time 1746141087.98 total 894.22 last 95.48) - open/close 44085 (time 1746141097.99 total 904.22 last 97.45) - open/close 45172 (time 1746141107.99 total 914.23 last 108.67) - open/close 46310 (time 1746141118.00 total 924.23 last 113.72) - open/close 47263 (time 1746141128.01 total 934.25 last 95.17) - open/close 48151 (time 1746141138.02 total 944.25 last 88.74) - open/close 49119 (time 1746141148.02 total 954.26 last 96.74) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 958.95 seconds: 51.76 ops/second - unlinked 0 (time 1746141154 ; total 0 ; last 0) - unlinked 10000 (time 1746141245 ; total 91 ; last 91) - unlinked 20000 (time 1746141298 ; total 144 ; last 53) - unlinked 30000 (time 1746141347 ; total 193 ; last 49) - unlinked 40000 (time 1746141392 ; total 238 ; last 45) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 284 seconds: 174.760559 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg227-server unloading modules via unload_modules_local on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2