== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:39:29 (1744047569) start mds service on oleg437-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions oleg437-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg437-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg437-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg437-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 345 (time 1744047620.37 total 10.21 last 33.78) - open/close 643 (time 1744047631.43 total 21.27 last 26.94) - open/close 1120 (time 1744047643.63 total 33.47 last 39.10) - open/close 1286 (time 1744047654.30 total 44.15 last 15.55) - open/close 1566 (time 1744047664.48 total 54.32 last 27.51) - open/close 1988 (time 1744047675.05 total 64.89 last 39.92) - open/close 2253 (time 1744047685.97 total 75.81 last 24.27) - open/close 2712 (time 1744047696.45 total 86.30 last 43.78) - open/close 3108 (time 1744047706.96 total 96.80 last 37.70) - open/close 3603 (time 1744047717.77 total 107.61 last 45.79) - open/close 3975 (time 1744047728.02 total 117.87 last 36.27) - open/close 4391 (time 1744047738.69 total 128.53 last 39.00) - open/close 5088 (time 1744047749.38 total 139.22 last 65.22) - open/close 5911 (time 1744047759.97 total 149.81 last 77.68) - open/close 6378 (time 1744047770.95 total 160.79 last 42.55) - open/close 7079 (time 1744047781.38 total 171.22 last 67.21) - open/close 7986 (time 1744047792.14 total 181.98 last 84.27) - open/close 8780 (time 1744047802.77 total 192.61 last 74.71) - open/close 9358 (time 1744047813.05 total 202.89 last 56.22) - open/close 10000 (time 1744047822.35 total 212.19 last 69.04) - open/close 10535 (time 1744047832.57 total 222.41 last 52.32) - open/close 11054 (time 1744047843.79 total 233.64 last 46.25) - open/close 11481 (time 1744047853.98 total 243.83 last 41.90) - open/close 11715 (time 1744047864.05 total 253.89 last 23.24) - open/close 12012 (time 1744047874.05 total 263.90 last 29.70) - open/close 12723 (time 1744047884.44 total 274.29 last 68.42) - open/close 15381 (time 1744047894.45 total 284.29 last 265.76) - open/close 20000 (time 1744047902.76 total 292.60 last 555.58) - open/close 25466 (time 1744047912.76 total 302.60 last 546.60) - open/close 30000 (time 1744047922.41 total 312.26 last 469.68) - open/close 33249 (time 1744047932.42 total 322.26 last 324.78) - open/close 36615 (time 1744047942.42 total 332.26 last 336.55) - open/close 40000 (time 1744047949.44 total 339.28 last 481.89) - open/close 44199 (time 1744047959.44 total 349.29 last 419.87) - open/close 49149 (time 1744047969.44 total 359.29 last 494.94) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 360.23 seconds: 137.78 ops/second - unlinked 0 (time 1744047971 ; total 0 ; last 0) - unlinked 10000 (time 1744047986 ; total 15 ; last 15) - unlinked 20000 (time 1744048008 ; total 37 ; last 22) - unlinked 30000 (time 1744048026 ; total 55 ; last 18) - unlinked 40000 (time 1744048046 ; total 75 ; last 20) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 98 seconds: 506.448975 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg437-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg437-server unloading modules via unload_modules_local on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2