== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:21:19 (1743510079) start mds service on oleg206-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing load_modules_local oleg206-server: Loading modules from /home/green/git/lustre-release/lustre oleg206-server: detected 4 online CPUs by sysfs oleg206-server: MODOPTS_LIBCFS= oleg206-server: Force libcfs to create 2 CPU partitions oleg206-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg206-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg206-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg206-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-OST0000 oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 419 (time 1743510117.55 total 10.00 last 41.90) - open/close 964 (time 1743510127.55 total 20.00 last 54.50) - open/close 1516 (time 1743510137.55 total 30.00 last 55.19) - open/close 1940 (time 1743510148.55 total 41.00 last 38.54) - open/close 2299 (time 1743510158.64 total 51.09 last 35.60) - open/close 2985 (time 1743510169.08 total 61.53 last 65.67) - open/close 3787 (time 1743510179.72 total 72.17 last 75.40) - open/close 4335 (time 1743510189.73 total 82.18 last 54.76) - open/close 4657 (time 1743510199.78 total 92.23 last 32.03) - open/close 5237 (time 1743510210.06 total 102.51 last 56.40) - open/close 5695 (time 1743510220.09 total 112.54 last 45.67) - open/close 6116 (time 1743510233.05 total 125.50 last 32.47) - open/close 6551 (time 1743510243.43 total 135.88 last 41.94) - open/close 6778 (time 1743510254.90 total 147.35 last 19.79) - open/close 7241 (time 1743510265.10 total 157.55 last 45.40) - open/close 7981 (time 1743510275.85 total 168.30 last 68.79) - open/close 8489 (time 1743510285.88 total 178.33 last 50.66) - open/close 8945 (time 1743510295.89 total 188.34 last 45.55) - open/close 9372 (time 1743510306.29 total 198.74 last 41.07) - open/close 9800 (time 1743510316.59 total 209.04 last 41.54) - open/close 10000 (time 1743510320.05 total 212.50 last 57.82) - open/close 10617 (time 1743510330.43 total 222.88 last 59.42) - open/close 11245 (time 1743510340.91 total 233.36 last 59.96) - open/close 11920 (time 1743510351.52 total 243.97 last 63.63) - open/close 15257 (time 1743510361.52 total 253.97 last 333.69) - open/close 19391 (time 1743510371.52 total 263.97 last 413.24) - open/close 23704 (time 1743510381.52 total 273.97 last 431.17) - open/close 27632 (time 1743510391.53 total 283.98 last 392.69) - open/close 30000 (time 1743510397.29 total 289.74 last 411.06) - open/close 33772 (time 1743510407.29 total 299.74 last 377.18) - open/close 37334 (time 1743510417.29 total 309.74 last 356.17) - open/close 40000 (time 1743510423.05 total 315.50 last 462.60) - open/close 44376 (time 1743510433.05 total 325.50 last 437.47) - open/close 47686 (time 1743510443.06 total 335.51 last 330.98) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 340.18 seconds: 145.90 ops/second - unlinked 0 (time 1743510448 ; total 0 ; last 0) - unlinked 10000 (time 1743510466 ; total 18 ; last 18) - unlinked 20000 (time 1743510484 ; total 36 ; last 18) - unlinked 30000 (time 1743510498 ; total 50 ; last 14) - unlinked 40000 (time 1743510512 ; total 64 ; last 14) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 80 seconds: 620.400024 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg206-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg206-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg206-server unloading modules via unload_modules_local on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg206-client: oleg206-client: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-server: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-client: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-server: ssh exited with exit code 2