== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:19:00 (1744885140) start mds service on oleg152-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg152-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg152-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg152-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg152-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 587 (time 1744885178.73 total 10.29 last 57.05) - open/close 944 (time 1744885189.55 total 21.11 last 33.00) - open/close 1383 (time 1744885199.55 total 31.11 last 43.89) - open/close 2136 (time 1744885210.24 total 41.80 last 70.40) - open/close 2506 (time 1744885221.07 total 52.63 last 34.17) - open/close 2991 (time 1744885231.18 total 62.74 last 48.00) - open/close 3411 (time 1744885241.32 total 72.88 last 41.40) - open/close 3627 (time 1744885252.14 total 83.70 last 19.97) - open/close 3828 (time 1744885265.11 total 96.67 last 15.49) - open/close 4153 (time 1744885275.26 total 106.82 last 32.02) - open/close 4360 (time 1744885285.34 total 116.90 last 20.53) - open/close 4767 (time 1744885295.74 total 127.30 last 39.14) - open/close 5200 (time 1744885305.74 total 137.30 last 43.30) - open/close 5821 (time 1744885316.13 total 147.69 last 59.78) - open/close 6212 (time 1744885326.21 total 157.77 last 38.77) - open/close 6607 (time 1744885336.21 total 167.77 last 39.50) - open/close 6922 (time 1744885347.14 total 178.70 last 28.82) - open/close 7194 (time 1744885357.26 total 188.82 last 26.89) - open/close 7585 (time 1744885369.08 total 200.64 last 33.06) - open/close 7840 (time 1744885379.44 total 211.00 last 24.63) - open/close 8100 (time 1744885390.26 total 221.82 last 24.02) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-8177) error: No space left on device total: 8177 open/close in 225.33 seconds: 36.29 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8972: kill: (12726) - No such process - unlinked 0 (time 1744885395 ; total 0 ; last 0) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-8177) error: No such file or directory total: 8177 unlinks in 19 seconds: 430.368408 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg152-server unloading modules via unload_modules_local on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2