== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:37:27 (1746074247) start mds service on oleg247-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing load_modules_local oleg247-server: Loading modules from /home/green/git/lustre-release/lustre oleg247-server: detected 4 online CPUs by sysfs oleg247-server: MODOPTS_LIBCFS= oleg247-server: Force libcfs to create 2 CPU partitions oleg247-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg247-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg247-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg247-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg247-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg247-server: oleg247-server.virtnet: executing set_default_debug -1 all pdsh@oleg247-client: oleg247-server: ssh exited with exit code 1 Started lustre-OST0000 oleg247-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg247-client.virtnet: -o user_xattr,flock 192.168.202.147@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 188 (time 1746074311.99 total 10.13 last 18.56) - open/close 367 (time 1746074322.52 total 20.66 last 17.00) - open/close 532 (time 1746074332.89 total 31.03 last 15.91) - open/close 689 (time 1746074342.90 total 41.03 last 15.70) - open/close 833 (time 1746074353.88 total 52.02 last 13.11) - open/close 942 (time 1746074364.42 total 62.56 last 10.34) - open/close 1090 (time 1746074375.42 total 73.55 last 13.46) - open/close 1246 (time 1746074386.13 total 84.26 last 14.56) - open/close 1547 (time 1746074396.14 total 94.28 last 30.06) - open/close 1782 (time 1746074406.67 total 104.81 last 22.32) - open/close 1994 (time 1746074416.89 total 115.02 last 20.75) - open/close 2207 (time 1746074426.89 total 125.03 last 21.30) - open/close 2425 (time 1746074437.81 total 135.94 last 19.97) - open/close 2538 (time 1746074448.52 total 146.66 last 10.55) - open/close 2663 (time 1746074459.22 total 157.36 last 11.68) - open/close 2829 (time 1746074470.17 total 168.30 last 15.16) - open/close 3050 (time 1746074480.53 total 178.67 last 21.32) - open/close 3329 (time 1746074490.97 total 189.11 last 26.73) - open/close 3549 (time 1746074501.43 total 199.57 last 21.04) - open/close 3796 (time 1746074512.03 total 210.16 last 23.30) - open/close 4043 (time 1746074522.49 total 220.63 last 23.60) - open/close 4308 (time 1746074533.50 total 231.64 last 24.08) - open/close 4461 (time 1746074544.19 total 242.32 last 14.32) - open/close 4742 (time 1746074555.03 total 253.17 last 25.91) - open/close 5012 (time 1746074565.30 total 263.43 last 26.30) - open/close 5276 (time 1746074575.66 total 273.79 last 25.48) - open/close 5566 (time 1746074586.50 total 284.64 last 26.75) - open/close 5889 (time 1746074596.50 total 294.64 last 32.30) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-6097) error: No space left on device total: 6097 open/close in 303.48 seconds: 20.09 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8972: kill: (12963) - No such process - unlinked 0 (time 1746074607 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-6097) error: No such file or directory total: 6097 unlinks in 30 seconds: 203.233337 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg247-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg247-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg247-server stop mds service on oleg247-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg247-server unloading modules via unload_modules_local on: 'oleg247-server' oleg247-server: oleg247-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-server: ssh exited with exit code 2 pdsh@oleg247-client: oleg247-client: ssh exited with exit code 2