== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:25:21 (1746138321) start mds service on oleg240-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg240-server' oleg240-server: oleg240-server.virtnet: executing load_modules_local oleg240-server: Loading modules from /home/green/git/lustre-release/lustre oleg240-server: detected 4 online CPUs by sysfs oleg240-server: MODOPTS_LIBCFS= oleg240-server: Force libcfs to create 2 CPU partitions oleg240-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg240-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg240-server: oleg240-server.virtnet: executing set_default_debug -1 all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg240-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg240-server: oleg240-server.virtnet: executing set_default_debug -1 all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg240-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug -1 all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 290 (time 1746138379.64 total 10.45 last 27.74) - open/close 619 (time 1746138389.80 total 20.62 last 32.36) - open/close 864 (time 1746138399.81 total 30.63 last 24.48) - open/close 1034 (time 1746138410.81 total 41.63 last 15.45) - open/close 1180 (time 1746138421.46 total 52.28 last 13.71) - open/close 1351 (time 1746138432.45 total 63.27 last 15.56) - open/close 1456 (time 1746138442.86 total 73.68 last 10.09) - open/close 1692 (time 1746138454.13 total 84.94 last 20.95) - open/close 1816 (time 1746138464.36 total 95.18 last 12.12) - open/close 2073 (time 1746138475.37 total 106.18 last 23.35) - open/close 2304 (time 1746138485.83 total 116.64 last 22.08) - open/close 2503 (time 1746138495.95 total 126.76 last 19.67) - open/close 2790 (time 1746138506.59 total 137.41 last 26.96) - open/close 3114 (time 1746138516.60 total 147.42 last 32.36) - open/close 3386 (time 1746138526.61 total 157.43 last 27.18) - open/close 3623 (time 1746138536.92 total 167.73 last 23.00) - open/close 3849 (time 1746138546.92 total 177.74 last 22.58) - open/close 4051 (time 1746138557.47 total 188.29 last 19.16) - open/close 4275 (time 1746138568.40 total 199.22 last 20.49) - open/close 4513 (time 1746138578.40 total 209.22 last 23.79) - open/close 4748 (time 1746138590.49 total 221.30 last 19.45) - open/close 5053 (time 1746138600.49 total 231.31 last 30.49) - open/close 5216 (time 1746138610.72 total 241.54 last 15.93) - open/close 5440 (time 1746138621.55 total 252.37 last 20.69) - open/close 5610 (time 1746138632.16 total 262.97 last 16.03) - open/close 5789 (time 1746138642.16 total 272.98 last 17.89) - open/close 6009 (time 1746138652.20 total 283.01 last 21.92) - open/close 6177 (time 1746138662.64 total 293.46 last 16.08) - open/close 6389 (time 1746138674.01 total 304.83 last 18.64) - open/close 6634 (time 1746138684.40 total 315.21 last 23.60) - open/close 6860 (time 1746138695.17 total 325.99 last 20.98) - open/close 7108 (time 1746138705.83 total 336.65 last 23.27) - open/close 7375 (time 1746138715.83 total 346.65 last 26.70) - open/close 7623 (time 1746138725.93 total 356.74 last 24.57) - open/close 7888 (time 1746138736.25 total 367.07 last 25.66) - open/close 8190 (time 1746138747.02 total 377.84 last 28.04) - open/close 8380 (time 1746138757.11 total 387.92 last 18.84) - open/close 8630 (time 1746138768.43 total 399.25 last 22.07) - open/close 8844 (time 1746138778.59 total 409.41 last 21.07) - open/close 9065 (time 1746138789.31 total 420.13 last 20.61) - open/close 9333 (time 1746138799.32 total 430.13 last 26.78) - open/close 9562 (time 1746138809.34 total 440.16 last 22.85) - open/close 10000 (time 1746138816.97 total 447.78 last 57.44) - open/close 11254 (time 1746138826.98 total 457.79 last 125.26) - open/close 12588 (time 1746138836.98 total 467.80 last 133.34) - open/close 13989 (time 1746138846.99 total 477.80 last 140.05) - open/close 15488 (time 1746138856.99 total 487.81 last 149.79) - open/close 16753 (time 1746138866.99 total 497.81 last 126.49) - open/close 17812 (time 1746138877.00 total 507.82 last 105.82) - open/close 19194 (time 1746138887.00 total 517.82 last 138.19) - open/close 20000 (time 1746138893.19 total 524.01 last 130.26) - open/close 21383 (time 1746138903.19 total 534.01 last 138.27) - open/close 23099 (time 1746138913.19 total 544.01 last 171.59) - open/close 24405 (time 1746138923.20 total 554.01 last 130.56) - open/close 25944 (time 1746138933.20 total 564.02 last 153.82) - open/close 27213 (time 1746138943.20 total 574.02 last 126.89) - open/close 28525 (time 1746138953.21 total 584.02 last 131.13) - open/close 29908 (time 1746138963.21 total 594.03 last 138.25) - open/close 31081 (time 1746138973.22 total 604.03 last 117.22) - open/close 32400 (time 1746138983.22 total 614.03 last 131.90) - open/close 33662 (time 1746138993.23 total 624.04 last 126.08) - open/close 34972 (time 1746139003.23 total 634.04 last 130.97) - open/close 36292 (time 1746139013.23 total 644.05 last 131.93) - open/close 37431 (time 1746139023.25 total 654.06 last 113.77) - open/close 38649 (time 1746139033.25 total 664.06 last 121.80) - open/close 39915 (time 1746139043.25 total 674.06 last 126.60) - open/close 41193 (time 1746139053.25 total 684.06 last 127.79) - open/close 43024 (time 1746139063.25 total 694.07 last 182.96) - open/close 44695 (time 1746139073.26 total 704.07 last 167.04) - open/close 46395 (time 1746139083.26 total 714.08 last 169.91) - open/close 47751 (time 1746139093.27 total 724.08 last 135.55) - open/close 48998 (time 1746139103.28 total 734.10 last 124.55) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 740.39 seconds: 67.04 ops/second - unlinked 0 (time 1746139111 ; total 0 ; last 0) - unlinked 10000 (time 1746139156 ; total 45 ; last 45) - unlinked 20000 (time 1746139194 ; total 83 ; last 38) - unlinked 30000 (time 1746139233 ; total 122 ; last 39) - unlinked 40000 (time 1746139274 ; total 163 ; last 41) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 202 seconds: 245.702972 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg240-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg240-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg240-server stop mds service on oleg240-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg240-server stop mds service on oleg240-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg240-server unloading modules via unload_modules_local on: 'oleg240-server' oleg240-server: oleg240-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg240-client: oleg240-client: ssh exited with exit code 2 pdsh@oleg240-client: oleg240-server: ssh exited with exit code 2 pdsh@oleg240-client: oleg240-client: ssh exited with exit code 2 pdsh@oleg240-client: oleg240-server: ssh exited with exit code 2