== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 03:22:16 (1746170536) start mds service on oleg259-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing load_modules_local oleg259-server: Loading modules from /home/green/git/lustre-release/lustre oleg259-server: detected 4 online CPUs by sysfs oleg259-server: MODOPTS_LIBCFS= oleg259-server: Force libcfs to create 2 CPU partitions oleg259-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg259-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg259-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg259-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg259-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg259-server: oleg259-server.virtnet: executing set_default_debug -1 all pdsh@oleg259-client: oleg259-server: ssh exited with exit code 1 Started lustre-OST0000 oleg259-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg259-client.virtnet: -o user_xattr,flock 192.168.202.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 866 (time 1746170575.48 total 10.61 last 81.58) - open/close 1626 (time 1746170585.90 total 21.04 last 72.93) - open/close 2606 (time 1746170596.77 total 31.91 last 90.10) - open/close 3329 (time 1746170607.11 total 42.25 last 69.94) - open/close 3925 (time 1746170617.26 total 52.40 last 58.72) - open/close 4552 (time 1746170627.44 total 62.57 last 61.63) - open/close 4961 (time 1746170637.44 total 72.57 last 40.90) - open/close 5589 (time 1746170647.82 total 82.96 last 60.47) - open/close 6118 (time 1746170657.87 total 93.01 last 52.65) - open/close 6888 (time 1746170668.30 total 103.43 last 73.85) - open/close 7402 (time 1746170678.47 total 113.61 last 50.52) - open/close 7979 (time 1746170688.61 total 123.75 last 56.91) - open/close 8432 (time 1746170698.66 total 133.80 last 45.07) - open/close 9497 (time 1746170709.64 total 144.78 last 96.96) - open/close 10000 (time 1746170715.63 total 150.76 last 84.07) - open/close 10528 (time 1746170725.63 total 160.77 last 52.80) - open/close 11287 (time 1746170736.06 total 171.20 last 72.73) - open/close 12066 (time 1746170746.60 total 181.74 last 73.93) - open/close 12918 (time 1746170757.27 total 192.41 last 79.83) - open/close 13794 (time 1746170767.95 total 203.09 last 82.04) - open/close 14660 (time 1746170778.58 total 213.72 last 81.49) - open/close 15561 (time 1746170789.34 total 224.48 last 83.73) - open/close 16275 (time 1746170799.80 total 234.94 last 68.21) - open/close 16726 (time 1746170809.91 total 245.05 last 44.62) - open/close 19543 (time 1746170819.91 total 255.05 last 281.69) - open/close 24150 (time 1746170829.91 total 265.05 last 460.69) - open/close 29204 (time 1746170839.91 total 275.05 last 505.39) - open/close 30000 (time 1746170842.01 total 277.15 last 378.76) - open/close 35208 (time 1746170852.02 total 287.15 last 520.72) - open/close 40000 (time 1746170861.51 total 296.65 last 504.70) - open/close 45218 (time 1746170871.51 total 306.65 last 521.71) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 315.42 seconds: 157.35 ops/second - unlinked 0 (time 1746170881 ; total 0 ; last 0) - unlinked 10000 (time 1746170892 ; total 11 ; last 11) - unlinked 20000 (time 1746170903 ; total 22 ; last 11) - unlinked 30000 (time 1746170914 ; total 33 ; last 11) - unlinked 40000 (time 1746170925 ; total 44 ; last 11) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 55 seconds: 902.400024 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg259-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg259-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg259-server stop mds service on oleg259-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg259-server stop mds service on oleg259-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg259-server unloading modules via unload_modules_local on: 'oleg259-server' oleg259-server: oleg259-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg259-client: oleg259-server: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-client: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-client: ssh exited with exit code 2 pdsh@oleg259-client: oleg259-server: ssh exited with exit code 2