== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:45:55 (1746132355) start mds service on oleg319-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing load_modules_local oleg319-server: Loading modules from /home/green/git/lustre-release/lustre oleg319-server: detected 4 online CPUs by sysfs oleg319-server: MODOPTS_LIBCFS= oleg319-server: Force libcfs to create 2 CPU partitions oleg319-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg319-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg319-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg319-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg319-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg319-server: oleg319-server.virtnet: executing set_default_debug -1 all pdsh@oleg319-client: oleg319-server: ssh exited with exit code 1 Started lustre-OST0000 oleg319-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg319-client.virtnet: -o user_xattr,flock 192.168.203.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 159 (time 1746132410.58 total 10.94 last 14.54) - open/close 379 (time 1746132420.64 total 21.00 last 21.87) - open/close 516 (time 1746132430.78 total 31.14 last 13.50) - open/close 837 (time 1746132440.90 total 41.26 last 31.74) - open/close 1084 (time 1746132451.29 total 51.65 last 23.76) - open/close 1349 (time 1746132461.55 total 61.91 last 25.84) - open/close 1616 (time 1746132471.56 total 71.92 last 26.68) - open/close 1759 (time 1746132482.50 total 82.87 last 13.07) - open/close 1916 (time 1746132494.51 total 94.87 last 13.08) - open/close 2158 (time 1746132504.88 total 105.24 last 23.33) - open/close 2441 (time 1746132515.35 total 115.71 last 27.03) - open/close 2709 (time 1746132526.09 total 126.46 last 24.95) - open/close 2938 (time 1746132536.10 total 136.46 last 22.89) - open/close 3094 (time 1746132546.80 total 147.16 last 14.58) - open/close 3308 (time 1746132558.97 total 159.33 last 17.58) - open/close 3484 (time 1746132568.98 total 169.34 last 17.59) - open/close 3703 (time 1746132579.26 total 179.62 last 21.31) - open/close 3980 (time 1746132589.66 total 190.02 last 26.63) - open/close 4276 (time 1746132600.39 total 200.75 last 27.58) - open/close 4578 (time 1746132611.19 total 211.55 last 27.95) - open/close 4878 (time 1746132621.20 total 221.56 last 29.98) - open/close 5098 (time 1746132631.20 total 231.56 last 22.00) - open/close 5333 (time 1746132641.93 total 242.29 last 21.90) - open/close 5599 (time 1746132652.91 total 253.28 last 24.22) - open/close 5818 (time 1746132663.73 total 264.10 last 20.24) - open/close 6109 (time 1746132673.74 total 274.10 last 29.08) - open/close 6307 (time 1746132683.74 total 284.10 last 19.80) - open/close 6504 (time 1746132693.94 total 294.30 last 19.32) - open/close 6681 (time 1746132704.10 total 304.46 last 17.42) - open/close 6958 (time 1746132714.92 total 315.28 last 25.59) - open/close 7240 (time 1746132724.92 total 325.28 last 28.19) - open/close 7424 (time 1746132737.08 total 337.45 last 15.13) - open/close 7594 (time 1746132747.09 total 347.45 last 16.99) - open/close 7790 (time 1746132757.11 total 357.47 last 19.56) - open/close 8068 (time 1746132767.51 total 367.87 last 26.72) - open/close 8274 (time 1746132777.52 total 377.88 last 20.59) - open/close 8456 (time 1746132787.53 total 387.89 last 18.19) - open/close 8743 (time 1746132797.93 total 398.29 last 27.59) - open/close 8932 (time 1746132808.06 total 408.43 last 18.64) - open/close 9217 (time 1746132818.99 total 419.35 last 26.08) - open/close 9497 (time 1746132829.00 total 429.36 last 27.97) - open/close 9761 (time 1746132839.32 total 439.68 last 25.59) - open/close 10000 (time 1746132847.79 total 448.15 last 28.21) - open/close 10202 (time 1746132858.57 total 458.93 last 18.74) - open/close 11559 (time 1746132868.57 total 468.93 last 135.65) - open/close 13082 (time 1746132878.59 total 478.95 last 152.10) - open/close 14516 (time 1746132888.59 total 488.95 last 143.30) - open/close 15895 (time 1746132898.59 total 498.95 last 137.89) - open/close 17539 (time 1746132908.60 total 508.96 last 164.32) - open/close 19227 (time 1746132918.60 total 518.96 last 168.74) - open/close 20000 (time 1746132923.05 total 523.41 last 173.76) - open/close 21836 (time 1746132933.05 total 533.41 last 183.59) - open/close 23414 (time 1746132943.05 total 543.42 last 157.73) - open/close 24909 (time 1746132953.06 total 553.42 last 149.48) - open/close 26542 (time 1746132963.06 total 563.42 last 163.26) - open/close 28286 (time 1746132973.06 total 573.42 last 174.33) - open/close 29812 (time 1746132983.07 total 583.43 last 152.51) - open/close 31408 (time 1746132993.07 total 593.43 last 159.53) - open/close 33029 (time 1746133003.08 total 603.44 last 162.05) - open/close 34865 (time 1746133013.08 total 613.44 last 183.54) - open/close 36900 (time 1746133023.08 total 623.44 last 203.47) - open/close 38592 (time 1746133033.09 total 633.45 last 169.06) - open/close 39873 (time 1746133043.09 total 643.45 last 128.07) - open/close 41364 (time 1746133053.10 total 653.46 last 148.96) - open/close 42947 (time 1746133063.10 total 663.46 last 158.29) - open/close 44295 (time 1746133073.10 total 673.47 last 134.75) - open/close 46042 (time 1746133083.11 total 683.47 last 174.64) - open/close 47848 (time 1746133093.11 total 693.47 last 180.53) - open/close 49590 (time 1746133103.12 total 703.48 last 174.12) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 703.71 seconds: 70.53 ops/second - unlinked 0 (time 1746133104 ; total 0 ; last 0) - unlinked 10000 (time 1746133157 ; total 53 ; last 53) - unlinked 20000 (time 1746133198 ; total 94 ; last 41) - unlinked 30000 (time 1746133237 ; total 133 ; last 39) - unlinked 40000 (time 1746133274 ; total 170 ; last 37) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 211 seconds: 235.222748 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg319-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg319-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg319-server stop mds service on oleg319-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg319-server unloading modules via unload_modules_local on: 'oleg319-server' oleg319-server: oleg319-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-client: ssh exited with exit code 2 pdsh@oleg319-client: oleg319-server: ssh exited with exit code 2