== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:59:29 (1748869169) start mds service on oleg127-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg127-server' oleg127-server: oleg127-server.virtnet: executing load_modules_local oleg127-server: Loading modules from /home/green/git/lustre-release/lustre oleg127-server: detected 4 online CPUs by sysfs oleg127-server: MODOPTS_LIBCFS= oleg127-server: Force libcfs to create 2 CPU partitions oleg127-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg127-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg127-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg127-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg127-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg127-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 657 (time 1748869209.05 total 10.31 last 63.75) - open/close 1209 (time 1748869219.18 total 20.44 last 54.46) - open/close 2075 (time 1748869230.00 total 31.26 last 80.06) - open/close 2761 (time 1748869240.45 total 41.71 last 65.62) - open/close 3643 (time 1748869251.14 total 52.40 last 82.49) - open/close 4368 (time 1748869264.61 total 65.87 last 53.84) - open/close 5299 (time 1748869275.51 total 76.78 last 85.38) - open/close 5944 (time 1748869285.84 total 87.10 last 62.46) - open/close 6442 (time 1748869295.91 total 97.17 last 49.48) - open/close 7129 (time 1748869306.30 total 107.56 last 66.11) - open/close 7931 (time 1748869316.93 total 118.19 last 75.40) - open/close 8407 (time 1748869326.96 total 128.22 last 47.49) - open/close 9040 (time 1748869337.22 total 138.48 last 61.67) - open/close 9856 (time 1748869347.91 total 149.17 last 76.35) - open/close 10606 (time 1748869358.38 total 159.64 last 71.61) - open/close 11502 (time 1748869369.14 total 170.40 last 83.32) - open/close 12095 (time 1748869379.30 total 180.56 last 58.34) - open/close 12737 (time 1748869389.59 total 190.86 last 62.36) - open/close 13668 (time 1748869400.49 total 201.75 last 85.47) - open/close 14418 (time 1748869411.04 total 212.31 last 71.04) - open/close 15164 (time 1748869421.56 total 222.83 last 70.91) - open/close 15935 (time 1748869432.14 total 233.40 last 72.88) - open/close 16939 (time 1748869442.14 total 243.41 last 100.39) - open/close 20000 (time 1748869448.17 total 249.43 last 508.28) - open/close 25003 (time 1748869458.17 total 259.43 last 500.30) - open/close 29862 (time 1748869468.17 total 269.43 last 485.88) - open/close 34460 (time 1748869478.17 total 279.43 last 459.75) - open/close 39367 (time 1748869488.17 total 289.43 last 490.59) - open/close 44274 (time 1748869498.17 total 299.43 last 490.66) - open/close 49263 (time 1748869508.17 total 309.43 last 498.89) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 310.18 seconds: 160.01 ops/second - unlinked 0 (time 1748869509 ; total 0 ; last 0) - unlinked 10000 (time 1748869521 ; total 12 ; last 12) - unlinked 20000 (time 1748869532 ; total 23 ; last 11) - unlinked 30000 (time 1748869543 ; total 34 ; last 11) - unlinked 40000 (time 1748869554 ; total 45 ; last 11) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 55 seconds: 902.400024 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg127-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg127-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg127-server stop mds service on oleg127-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg127-server stop mds service on oleg127-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg127-server unloading modules via unload_modules_local on: 'oleg127-server' oleg127-server: oleg127-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg127-client: oleg127-client: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-server: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-client: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-server: ssh exited with exit code 2