== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 15:45:46 (1746128746) start mds service on oleg443-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing load_modules_local oleg443-server: Loading modules from /home/green/git/lustre-release/lustre oleg443-server: detected 4 online CPUs by sysfs oleg443-server: MODOPTS_LIBCFS= oleg443-server: Force libcfs to create 2 CPU partitions oleg443-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg443-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg443-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg443-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg443-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg443-server: oleg443-server.virtnet: executing set_default_debug -1 all pdsh@oleg443-client: oleg443-server: ssh exited with exit code 1 Started lustre-OST0000 oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg443-client.virtnet: -o user_xattr,flock 192.168.204.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 199 (time 1746128820.83 total 10.24 last 19.44) - open/close 399 (time 1746128830.83 total 20.24 last 19.99) - open/close 582 (time 1746128840.89 total 30.30 last 18.19) - open/close 770 (time 1746128850.90 total 40.31 last 18.78) - open/close 948 (time 1746128860.91 total 50.31 last 17.80) - open/close 1145 (time 1746128871.00 total 60.41 last 19.52) - open/close 1363 (time 1746128881.00 total 70.41 last 21.79) - open/close 1545 (time 1746128891.50 total 80.91 last 17.33) - open/close 1727 (time 1746128902.11 total 91.52 last 17.16) - open/close 1890 (time 1746128912.60 total 102.01 last 15.54) - open/close 2065 (time 1746128923.01 total 112.42 last 16.81) - open/close 2282 (time 1746128933.75 total 123.16 last 20.20) - open/close 2430 (time 1746128944.84 total 134.25 last 13.35) - open/close 2615 (time 1746128957.47 total 146.88 last 14.64) - open/close 2747 (time 1746128968.25 total 157.65 last 12.25) - open/close 2975 (time 1746128978.84 total 168.25 last 21.51) - open/close 3142 (time 1746128989.24 total 178.65 last 16.06) - open/close 3340 (time 1746128999.24 total 188.65 last 19.80) - open/close 3491 (time 1746129009.56 total 198.97 last 14.63) - open/close 3692 (time 1746129019.57 total 208.97 last 20.09) - open/close 3850 (time 1746129030.03 total 219.44 last 15.10) - open/close 3969 (time 1746129040.52 total 229.93 last 11.35) - open/close 4127 (time 1746129051.67 total 241.07 last 14.18) - open/close 4255 (time 1746129061.69 total 251.09 last 12.77) - open/close 4446 (time 1746129072.59 total 261.99 last 17.52) - open/close 4609 (time 1746129083.52 total 272.92 last 14.91) - open/close 4852 (time 1746129094.27 total 283.67 last 22.61) - open/close 5046 (time 1746129104.42 total 293.83 last 19.10) - open/close 5195 (time 1746129115.24 total 304.65 last 13.77) - open/close 5268 (time 1746129125.24 total 314.65 last 7.30) - open/close 5509 (time 1746129136.16 total 325.56 last 22.08) - open/close 5691 (time 1746129146.17 total 335.57 last 18.19) - open/close 5908 (time 1746129157.25 total 346.66 last 19.58) - open/close 6083 (time 1746129167.26 total 356.67 last 17.48) - open/close 6322 (time 1746129179.84 total 369.24 last 19.01) - open/close 6507 (time 1746129189.86 total 379.26 last 18.46) - open/close 6782 (time 1746129199.86 total 389.27 last 27.49) - open/close 6961 (time 1746129209.87 total 399.27 last 17.89) - open/close 7168 (time 1746129220.05 total 409.45 last 20.34) - open/close 7309 (time 1746129230.47 total 419.88 last 13.52) - open/close 7475 (time 1746129240.54 total 429.94 last 16.49) - open/close 7592 (time 1746129251.22 total 440.63 last 10.95) - open/close 7747 (time 1746129261.23 total 450.64 last 15.49) - open/close 7913 (time 1746129272.21 total 461.62 last 15.12) - open/close 8190 (time 1746129282.22 total 471.63 last 27.68) - open/close 8308 (time 1746129292.76 total 482.16 last 11.20) - open/close 8486 (time 1746129303.49 total 492.90 last 16.58) - open/close 8757 (time 1746129314.26 total 503.67 last 25.17) - open/close 8970 (time 1746129324.57 total 513.98 last 20.66) - open/close 9222 (time 1746129335.00 total 524.40 last 24.17) - open/close 9551 (time 1746129345.86 total 535.27 last 30.29) - open/close 9726 (time 1746129355.93 total 545.34 last 17.38) - open/close 9948 (time 1746129366.08 total 555.49 last 21.87) - open/close 10000 (time 1746129368.90 total 558.31 last 18.44) - open/close 10506 (time 1746129378.91 total 568.31 last 50.56) - open/close 12069 (time 1746129388.91 total 578.31 last 156.27) - open/close 13728 (time 1746129398.91 total 588.32 last 165.82) - open/close 15201 (time 1746129408.91 total 598.32 last 147.29) - open/close 16688 (time 1746129418.93 total 608.33 last 148.53) - open/close 18137 (time 1746129428.93 total 618.33 last 144.87) - open/close 19635 (time 1746129438.93 total 628.33 last 149.78) - open/close 20000 (time 1746129441.04 total 630.44 last 173.25) - open/close 21599 (time 1746129451.04 total 640.45 last 159.83) - open/close 23109 (time 1746129461.04 total 650.45 last 150.99) - open/close 24511 (time 1746129471.05 total 660.45 last 140.12) - open/close 25915 (time 1746129481.05 total 670.46 last 140.32) - open/close 27220 (time 1746129491.05 total 680.46 last 130.48) - open/close 28514 (time 1746129501.06 total 690.47 last 129.32) - open/close 30000 (time 1746129510.61 total 700.01 last 155.68) - open/close 31444 (time 1746129520.62 total 710.03 last 144.18) - open/close 32997 (time 1746129530.63 total 720.03 last 155.22) - open/close 34560 (time 1746129540.63 total 730.03 last 156.28) - open/close 35978 (time 1746129550.63 total 740.03 last 141.79) - open/close 37395 (time 1746129560.63 total 750.04 last 141.66) - open/close 38763 (time 1746129570.63 total 760.04 last 136.80) - open/close 40000 (time 1746129580.19 total 769.60 last 129.40) - open/close 41320 (time 1746129590.20 total 779.60 last 131.92) - open/close 42625 (time 1746129600.20 total 789.61 last 130.45) - open/close 43843 (time 1746129610.20 total 799.61 last 121.78) - open/close 45028 (time 1746129620.21 total 809.61 last 118.47) - open/close 46277 (time 1746129630.21 total 819.62 last 124.85) - open/close 47510 (time 1746129640.21 total 829.62 last 123.23) - open/close 48767 (time 1746129650.22 total 839.62 last 125.68) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 846.89 seconds: 58.60 ops/second - unlinked 0 (time 1746129659 ; total 0 ; last 0) - unlinked 10000 (time 1746129725 ; total 66 ; last 66) - unlinked 20000 (time 1746129774 ; total 115 ; last 49) - unlinked 30000 (time 1746129812 ; total 153 ; last 38) - unlinked 40000 (time 1746129850 ; total 191 ; last 38) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 216 seconds: 229.777771 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg443-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg443-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg443-server stop mds service on oleg443-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg443-server unloading modules via unload_modules_local on: 'oleg443-server' oleg443-server: oleg443-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg443-client: oleg443-client: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-client: ssh exited with exit code 2 pdsh@oleg443-client: oleg443-server: ssh exited with exit code 2