== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:54:45 (1747850085) start mds service on oleg142-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg142-server' oleg142-server: oleg142-server.virtnet: executing load_modules_local oleg142-server: Loading modules from /home/green/git/lustre-release/lustre oleg142-server: detected 4 online CPUs by sysfs oleg142-server: MODOPTS_LIBCFS= oleg142-server: Force libcfs to create 2 CPU partitions oleg142-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg142-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg142-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg142-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg142-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg142-server: oleg142-server.virtnet: executing set_default_debug -1 all pdsh@oleg142-client: oleg142-server: ssh exited with exit code 1 Started lustre-OST0000 oleg142-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg142-client.virtnet: -o user_xattr,flock 192.168.201.142@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 299 (time 1747850124.78 total 10.00 last 29.90) - open/close 496 (time 1747850135.36 total 20.58 last 18.63) - open/close 702 (time 1747850145.95 total 31.17 last 19.45) - open/close 848 (time 1747850156.33 total 41.55 last 14.06) - open/close 1022 (time 1747850166.82 total 52.03 last 16.60) - open/close 1198 (time 1747850177.28 total 62.50 last 16.81) - open/close 1382 (time 1747850188.72 total 73.94 last 16.08) - open/close 1721 (time 1747850199.58 total 84.79 last 31.24) - open/close 1924 (time 1747850210.21 total 95.43 last 19.08) - open/close 2184 (time 1747850221.18 total 106.40 last 23.70) - open/close 2331 (time 1747850231.77 total 116.99 last 13.89) - open/close 2546 (time 1747850242.64 total 127.86 last 19.77) - open/close 2726 (time 1747850253.32 total 138.54 last 16.86) - open/close 3055 (time 1747850263.66 total 148.88 last 31.81) - open/close 3281 (time 1747850274.50 total 159.72 last 20.86) - open/close 3474 (time 1747850285.19 total 170.41 last 18.05) - open/close 3716 (time 1747850296.08 total 181.30 last 22.22) - open/close 4014 (time 1747850306.42 total 191.64 last 28.83) - open/close 4288 (time 1747850316.67 total 201.88 last 26.74) - open/close 4558 (time 1747850326.74 total 211.95 last 26.81) - open/close 4819 (time 1747850336.74 total 221.96 last 26.10) - open/close 5149 (time 1747850347.15 total 232.37 last 31.70) - open/close 5570 (time 1747850357.15 total 242.37 last 42.09) - open/close 5964 (time 1747850367.97 total 253.19 last 36.43) - open/close 6327 (time 1747850378.65 total 263.87 last 33.99) - open/close 6868 (time 1747850388.65 total 273.87 last 54.09) - open/close 9905 (time 1747850398.65 total 283.87 last 303.70) - open/close 13026 (time 1747850408.65 total 293.87 last 312.09) - open/close 16100 (time 1747850418.65 total 303.87 last 307.34) - open/close 19052 (time 1747850428.65 total 313.87 last 295.19) - open/close 20000 (time 1747850432.28 total 317.50 last 261.12) - open/close 22521 (time 1747850442.29 total 327.50 last 252.03) - open/close 25194 (time 1747850452.29 total 337.51 last 267.20) - open/close 27626 (time 1747850462.29 total 347.51 last 243.14) - open/close 30000 (time 1747850470.38 total 355.60 last 293.52) - open/close 32463 (time 1747850480.38 total 365.60 last 246.24) - open/close 35098 (time 1747850490.38 total 375.60 last 263.49) - open/close 37461 (time 1747850500.38 total 385.60 last 236.27) - open/close 40000 (time 1747850508.91 total 394.13 last 297.76) - open/close 44002 (time 1747850518.91 total 404.13 last 400.12) - open/close 47462 (time 1747850528.91 total 414.13 last 345.94) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 419.97 seconds: 118.18 ops/second - unlinked 0 (time 1747850535 ; total 0 ; last 0) - unlinked 10000 (time 1747850548 ; total 13 ; last 13) - unlinked 20000 (time 1747850563 ; total 28 ; last 15) - unlinked 30000 (time 1747850577 ; total 42 ; last 14) - unlinked 40000 (time 1747850595 ; total 60 ; last 18) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 82 seconds: 605.268311 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg142-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg142-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg142-server stop mds service on oleg142-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg142-server stop mds service on oleg142-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg142-server unloading modules via unload_modules_local on: 'oleg142-server' oleg142-server: oleg142-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg142-client: oleg142-client: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-server: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-client: ssh exited with exit code 2 pdsh@oleg142-client: oleg142-server: ssh exited with exit code 2