== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:04:18 (1745622258) start mds service on oleg221-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing load_modules_local oleg221-server: Loading modules from /home/green/git/lustre-release/lustre oleg221-server: detected 4 online CPUs by sysfs oleg221-server: MODOPTS_LIBCFS= oleg221-server: Force libcfs to create 2 CPU partitions oleg221-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg221-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg221-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg221-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg221-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg221-server: oleg221-server.virtnet: executing set_default_debug -1 all pdsh@oleg221-client: oleg221-server: ssh exited with exit code 1 Started lustre-OST0000 oleg221-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg221-client.virtnet: -o user_xattr,flock 192.168.202.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 241 (time 1745622312.94 total 10.10 last 23.85) - open/close 438 (time 1745622323.30 total 20.47 last 19.00) - open/close 750 (time 1745622334.01 total 31.18 last 29.13) - open/close 1025 (time 1745622344.77 total 41.94 last 25.57) - open/close 1347 (time 1745622355.37 total 52.54 last 30.37) - open/close 1578 (time 1745622365.96 total 63.12 last 21.83) - open/close 1797 (time 1745622378.44 total 75.61 last 17.54) - open/close 2062 (time 1745622388.71 total 85.88 last 25.81) - open/close 2229 (time 1745622399.18 total 96.35 last 15.94) - open/close 2469 (time 1745622409.18 total 106.35 last 23.99) - open/close 2669 (time 1745622419.85 total 117.02 last 18.75) - open/close 2873 (time 1745622430.64 total 127.80 last 18.91) - open/close 3047 (time 1745622440.75 total 137.92 last 17.21) - open/close 3260 (time 1745622451.29 total 148.46 last 20.20) - open/close 3392 (time 1745622461.30 total 158.46 last 13.20) - open/close 3649 (time 1745622471.40 total 168.57 last 25.44) - open/close 3900 (time 1745622481.41 total 178.58 last 25.07) - open/close 4051 (time 1745622491.82 total 188.99 last 14.51) - open/close 4368 (time 1745622502.45 total 199.62 last 29.81) - open/close 4668 (time 1745622513.11 total 210.28 last 28.15) - open/close 5093 (time 1745622523.34 total 220.51 last 41.53) - open/close 5229 (time 1745622536.25 total 233.42 last 10.54) - open/close 5587 (time 1745622547.20 total 244.37 last 32.67) - open/close 5723 (time 1745622558.13 total 255.30 last 12.45) - open/close 5939 (time 1745622568.54 total 265.71 last 20.74) - open/close 6140 (time 1745622578.76 total 275.93 last 19.67) - open/close 6309 (time 1745622588.76 total 285.93 last 16.90) - open/close 6574 (time 1745622599.60 total 296.77 last 24.46) - open/close 6786 (time 1745622609.61 total 306.77 last 21.18) - open/close 7002 (time 1745622620.44 total 317.60 last 19.94) - open/close 7145 (time 1745622630.44 total 327.61 last 14.29) - open/close 7348 (time 1745622641.08 total 338.24 last 19.10) - open/close 7511 (time 1745622652.07 total 349.24 last 14.82) - open/close 7711 (time 1745622663.23 total 360.40 last 17.93) - open/close 8563 (time 1745622673.23 total 370.40 last 85.17) - open/close 10000 (time 1745622682.73 total 379.90 last 151.28) - open/close 11671 (time 1745622692.74 total 389.90 last 167.04) - open/close 13937 (time 1745622702.75 total 399.92 last 226.37) - open/close 16213 (time 1745622712.75 total 409.92 last 227.56) - open/close 17793 (time 1745622722.75 total 419.92 last 157.94) - open/close 19357 (time 1745622732.76 total 429.92 last 156.35) - open/close 20000 (time 1745622737.78 total 434.95 last 127.93) - open/close 21952 (time 1745622747.78 total 444.95 last 195.19) - open/close 24576 (time 1745622757.78 total 454.95 last 262.34) - open/close 26714 (time 1745622767.79 total 464.95 last 213.78) - open/close 28953 (time 1745622777.79 total 474.96 last 223.87) - open/close 30000 (time 1745622781.90 total 479.07 last 254.37) - open/close 32327 (time 1745622791.91 total 489.07 last 232.63) - open/close 34653 (time 1745622801.91 total 499.08 last 232.49) - open/close 36601 (time 1745622811.91 total 509.08 last 194.73) - open/close 38881 (time 1745622821.91 total 519.08 last 227.99) - open/close 40000 (time 1745622825.58 total 522.75 last 305.42) - open/close 42949 (time 1745622835.58 total 532.75 last 294.89) - open/close 46595 (time 1745622845.58 total 542.75 last 364.59) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 551.13 seconds: 90.05 ops/second - unlinked 0 (time 1745622854 ; total 0 ; last 0) - unlinked 10000 (time 1745622867 ; total 13 ; last 13) - unlinked 20000 (time 1745622891 ; total 37 ; last 24) - unlinked 30000 (time 1745622911 ; total 57 ; last 20) - unlinked 40000 (time 1745622924 ; total 70 ; last 13) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 81 seconds: 612.740723 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg221-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg221-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg221-server stop mds service on oleg221-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg221-server unloading modules via unload_modules_local on: 'oleg221-server' oleg221-server: oleg221-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-client: ssh exited with exit code 2 pdsh@oleg221-client: oleg221-server: ssh exited with exit code 2