== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 19:04:40 (1746140680) start mds service on oleg448-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing load_modules_local oleg448-server: Loading modules from /home/green/git/lustre-release/lustre oleg448-server: detected 4 online CPUs by sysfs oleg448-server: MODOPTS_LIBCFS= oleg448-server: Force libcfs to create 2 CPU partitions oleg448-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg448-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg448-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg448-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg448-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg448-server: oleg448-server.virtnet: executing set_default_debug -1 all pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 oleg448-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg448-client.virtnet: -o user_xattr,flock 192.168.204.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 137 (time 1746140736.41 total 10.71 last 12.79) - open/close 352 (time 1746140746.95 total 21.26 last 20.39) - open/close 582 (time 1746140757.31 total 31.62 last 22.19) - open/close 844 (time 1746140767.89 total 42.20 last 24.76) - open/close 1064 (time 1746140778.23 total 52.54 last 21.28) - open/close 1333 (time 1746140789.21 total 63.52 last 24.51) - open/close 1595 (time 1746140799.79 total 74.10 last 24.76) - open/close 1785 (time 1746140810.34 total 84.64 last 18.02) - open/close 1977 (time 1746140820.71 total 95.02 last 18.51) - open/close 2187 (time 1746140830.84 total 105.15 last 20.73) - open/close 2433 (time 1746140841.11 total 115.41 last 23.96) - open/close 2648 (time 1746140851.99 total 126.30 last 19.75) - open/close 2883 (time 1746140862.20 total 136.51 last 23.01) - open/close 3092 (time 1746140872.23 total 146.53 last 20.85) - open/close 3309 (time 1746140882.86 total 157.17 last 20.40) - open/close 3437 (time 1746140892.88 total 167.19 last 12.77) - open/close 3701 (time 1746140903.71 total 178.02 last 24.38) - open/close 3899 (time 1746140914.12 total 188.43 last 19.03) - open/close 4163 (time 1746140924.68 total 198.99 last 25.00) - open/close 4405 (time 1746140934.68 total 208.99 last 24.20) - open/close 4641 (time 1746140944.92 total 219.23 last 23.04) - open/close 4873 (time 1746140955.70 total 230.01 last 21.52) - open/close 5095 (time 1746140966.65 total 240.96 last 20.27) - open/close 5323 (time 1746140976.77 total 251.08 last 22.53) - open/close 5546 (time 1746140987.38 total 261.69 last 21.01) - open/close 5688 (time 1746140997.63 total 271.93 last 13.86) - open/close 5812 (time 1746141009.46 total 283.76 last 10.48) - open/close 5902 (time 1746141020.12 total 294.43 last 8.44) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-6161) error: No space left on device total: 6161 open/close in 303.97 seconds: 20.27 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8972: kill: (13155) - No such process - unlinked 0 (time 1746141031 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-6161) error: No such file or directory total: 6161 unlinks in 25 seconds: 246.440002 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg448-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg448-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg448-server stop mds service on oleg448-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg448-server stop mds service on oleg448-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg448-server unloading modules via unload_modules_local on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg448-client: oleg448-client: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-client: ssh exited with exit code 2 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 2