== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 05:19:21 (1726305561) start mds service on oleg422-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg422-server' oleg422-server: oleg422-server.virtnet: executing load_modules_local oleg422-server: Loading modules from /home/green/git/lustre-release/lustre oleg422-server: detected 4 online CPUs by sysfs oleg422-server: Force libcfs to create 2 CPU partitions oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg422-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg422-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Started lustre-OST0000 oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 813 (time 1726305600.50 total 10.55 last 77.09) - open/close 1634 (time 1726305611.05 total 21.10 last 77.81) - open/close 2443 (time 1726305621.56 total 31.60 last 77.00) - open/close 3302 (time 1726305632.17 total 42.22 last 80.89) - open/close 4092 (time 1726305642.67 total 52.71 last 75.29) - open/close 4683 (time 1726305652.79 total 62.84 last 58.35) - open/close 5344 (time 1726305664.05 total 74.10 last 58.73) - open/close 5678 (time 1726305674.72 total 84.77 last 31.30) - open/close 6404 (time 1726305685.10 total 95.15 last 69.93) - open/close 7148 (time 1726305695.48 total 105.53 last 71.68) - open/close 8038 (time 1726305706.15 total 116.20 last 83.41) - open/close 8809 (time 1726305717.64 total 127.69 last 67.09) - open/close 9417 (time 1726305727.84 total 137.89 last 59.64) - open/close 10000 (time 1726305733.97 total 144.02 last 95.09) - open/close 10870 (time 1726305744.58 total 154.63 last 81.96) - open/close 11618 (time 1726305756.00 total 166.05 last 65.51) - open/close 12130 (time 1726305766.98 total 177.03 last 46.64) - open/close 12917 (time 1726305777.50 total 187.54 last 74.83) - open/close 13703 (time 1726305787.98 total 198.03 last 74.96) - open/close 14481 (time 1726305800.47 total 210.52 last 62.30) - open/close 15081 (time 1726305810.61 total 220.66 last 59.18) - open/close 15619 (time 1726305820.67 total 230.72 last 53.48) - open/close 17131 (time 1726305830.67 total 240.72 last 151.19) - open/close 20000 (time 1726305836.13 total 246.18 last 524.96) - open/close 24866 (time 1726305846.13 total 256.18 last 486.57) - open/close 30000 (time 1726305856.11 total 266.15 last 514.83) - open/close 35345 (time 1726305866.11 total 276.16 last 534.40) - open/close 40000 (time 1726305874.81 total 284.85 last 535.17) - open/close 45421 (time 1726305884.81 total 294.86 last 542.03) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 302.99 seconds: 163.81 ops/second - unlinked 0 (time 1726305893 ; total 0 ; last 0) - unlinked 10000 (time 1726305908 ; total 15 ; last 15) - unlinked 20000 (time 1726305919 ; total 26 ; last 11) - unlinked 30000 (time 1726305929 ; total 36 ; last 10) - unlinked 40000 (time 1726305939 ; total 46 ; last 10) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 56 seconds: 886.285706 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg422-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg422-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server stop mds service on oleg422-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server stop mds service on oleg422-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server unloading modules on: 'oleg422-server' oleg422-server: oleg422-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2 pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2