== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:21:32 (1748470892) start mds service on oleg128-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing load_modules_local oleg128-server: Loading modules from /home/green/git/lustre-release/lustre oleg128-server: detected 4 online CPUs by sysfs oleg128-server: MODOPTS_LIBCFS= oleg128-server: Force libcfs to create 2 CPU partitions oleg128-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg128-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg128-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg128-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg128-server: oleg128-server.virtnet: executing set_default_debug -1 all pdsh@oleg128-client: oleg128-server: ssh exited with exit code 1 Started lustre-OST0000 oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg128-client.virtnet: -o user_xattr,flock 192.168.201.128@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 353 (time 1748470932.63 total 10.57 last 33.40) - open/close 636 (time 1748470943.16 total 21.09 last 26.89) - open/close 915 (time 1748470953.16 total 31.10 last 27.89) - open/close 1633 (time 1748470963.59 total 41.53 last 68.84) - open/close 2346 (time 1748470974.00 total 51.94 last 68.48) - open/close 2843 (time 1748470984.00 total 61.94 last 49.70) - open/close 3298 (time 1748470994.00 total 71.94 last 45.50) - open/close 3873 (time 1748471004.15 total 82.08 last 56.70) - open/close 4722 (time 1748471014.78 total 92.71 last 79.86) - open/close 5683 (time 1748471025.64 total 103.57 last 88.48) - open/close 6142 (time 1748471035.65 total 113.58 last 45.86) - open/close 6432 (time 1748471048.37 total 126.30 last 22.80) - open/close 7088 (time 1748471058.69 total 136.63 last 63.53) - open/close 8040 (time 1748471069.58 total 147.51 last 87.45) - open/close 8936 (time 1748471080.34 total 158.28 last 83.25) - open/close 9827 (time 1748471091.09 total 169.03 last 82.89) - open/close 10725 (time 1748471101.86 total 179.80 last 83.35) - open/close 11609 (time 1748471112.59 total 190.53 last 82.38) - open/close 12537 (time 1748471123.47 total 201.41 last 85.33) - open/close 13448 (time 1748471134.30 total 212.23 last 84.15) - open/close 14388 (time 1748471145.13 total 223.07 last 86.73) - open/close 15316 (time 1748471155.98 total 233.91 last 85.58) - open/close 16109 (time 1748471166.58 total 244.52 last 74.80) - open/close 19598 (time 1748471176.58 total 254.52 last 348.89) - open/close 24624 (time 1748471186.58 total 264.52 last 502.48) - open/close 29756 (time 1748471196.58 total 274.52 last 513.16) - open/close 34687 (time 1748471206.58 total 284.52 last 493.06) - open/close 39709 (time 1748471216.59 total 294.52 last 502.13) - open/close 44711 (time 1748471226.59 total 304.52 last 500.13) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 314.51 seconds: 157.81 ops/second - unlinked 0 (time 1748471237 ; total 0 ; last 0) - unlinked 10000 (time 1748471249 ; total 12 ; last 12) - unlinked 20000 (time 1748471260 ; total 23 ; last 11) - unlinked 30000 (time 1748471271 ; total 34 ; last 11) - unlinked 40000 (time 1748471281 ; total 44 ; last 10) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 55 seconds: 902.400024 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg128-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg128-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg128-server stop mds service on oleg128-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg128-server stop mds service on oleg128-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg128-server unloading modules via unload_modules_local on: 'oleg128-server' oleg128-server: oleg128-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg128-client: oleg128-client: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-client: ssh exited with exit code 2 pdsh@oleg128-client: oleg128-server: ssh exited with exit code 2