== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:40:06 (1746157206) start mds service on oleg334-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing load_modules_local oleg334-server: Loading modules from /home/green/git/lustre-release/lustre oleg334-server: detected 4 online CPUs by sysfs oleg334-server: MODOPTS_LIBCFS= oleg334-server: Force libcfs to create 2 CPU partitions oleg334-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg334-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg334-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg334-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 280 (time 1746157263.33 total 10.52 last 26.62) - open/close 550 (time 1746157274.05 total 21.24 last 25.19) - open/close 760 (time 1746157284.58 total 31.77 last 19.94) - open/close 1010 (time 1746157295.44 total 42.63 last 23.02) - open/close 1232 (time 1746157306.46 total 53.65 last 20.15) - open/close 1516 (time 1746157317.10 total 64.29 last 26.69) - open/close 1773 (time 1746157327.38 total 74.57 last 25.01) - open/close 1952 (time 1746157338.02 total 85.22 last 16.81) - open/close 2164 (time 1746157348.58 total 95.77 last 20.09) - open/close 2468 (time 1746157359.33 total 106.52 last 28.28) - open/close 2654 (time 1746157369.70 total 116.89 last 17.93) - open/close 2912 (time 1746157379.98 total 127.17 last 25.10) - open/close 3181 (time 1746157390.00 total 137.19 last 26.83) - open/close 3425 (time 1746157401.78 total 148.97 last 20.72) - open/close 3641 (time 1746157412.44 total 159.64 last 20.25) - open/close 3909 (time 1746157423.15 total 170.34 last 25.04) - open/close 4191 (time 1746157433.75 total 180.94 last 26.61) - open/close 4429 (time 1746157446.35 total 193.54 last 18.88) - open/close 4765 (time 1746157456.86 total 204.05 last 31.96) - open/close 4998 (time 1746157467.08 total 214.27 last 22.80) - open/close 5281 (time 1746157477.83 total 225.02 last 26.33) - open/close 5483 (time 1746157488.25 total 235.44 last 19.38) - open/close 5912 (time 1746157499.24 total 246.43 last 39.03) - open/close 6346 (time 1746157509.43 total 256.62 last 42.58) - open/close 6652 (time 1746157519.92 total 267.11 last 29.17) - open/close 7026 (time 1746157530.91 total 278.10 last 34.05) - open/close 7225 (time 1746157541.48 total 288.67 last 18.81) - open/close 7423 (time 1746157552.06 total 299.26 last 18.71) - open/close 7566 (time 1746157562.07 total 309.26 last 14.30) - open/close 7819 (time 1746157572.63 total 319.82 last 23.95) - open/close 8082 (time 1746157583.35 total 330.54 last 24.53) - open/close 8343 (time 1746157593.74 total 340.93 last 25.13) - open/close 8598 (time 1746157604.66 total 351.85 last 23.34) - open/close 8724 (time 1746157614.66 total 361.85 last 12.60) - open/close 8969 (time 1746157625.07 total 372.26 last 23.55) - open/close 10000 (time 1746157633.35 total 380.54 last 124.49) - open/close 11519 (time 1746157643.35 total 390.54 last 151.89) - open/close 13504 (time 1746157653.35 total 400.54 last 198.47) - open/close 15444 (time 1746157663.36 total 410.55 last 193.91) - open/close 16966 (time 1746157673.36 total 420.55 last 152.19) - open/close 18862 (time 1746157683.36 total 430.55 last 189.59) - open/close 20000 (time 1746157690.03 total 437.22 last 170.61) - open/close 21705 (time 1746157700.04 total 447.23 last 170.37) - open/close 23420 (time 1746157710.04 total 457.23 last 171.44) - open/close 25311 (time 1746157720.04 total 467.23 last 189.02) - open/close 26778 (time 1746157730.05 total 477.24 last 146.66) - open/close 28399 (time 1746157740.05 total 487.25 last 161.98) - open/close 29840 (time 1746157750.06 total 497.25 last 144.04) - open/close 31138 (time 1746157760.07 total 507.26 last 129.59) - open/close 32571 (time 1746157770.08 total 517.27 last 143.26) - open/close 34205 (time 1746157780.09 total 527.28 last 163.27) - open/close 35566 (time 1746157790.09 total 537.28 last 136.03) - open/close 37070 (time 1746157800.09 total 547.28 last 150.36) - open/close 38489 (time 1746157810.10 total 557.29 last 141.85) - open/close 39744 (time 1746157820.10 total 567.29 last 125.43) - open/close 41030 (time 1746157830.11 total 577.31 last 128.45) - open/close 42329 (time 1746157840.12 total 587.31 last 129.87) - open/close 43601 (time 1746157850.12 total 597.32 last 127.10) - open/close 44810 (time 1746157860.13 total 607.32 last 120.87) - open/close 46128 (time 1746157870.14 total 617.33 last 131.64) - open/close 47320 (time 1746157880.14 total 627.33 last 119.17) - open/close 48824 (time 1746157890.15 total 637.34 last 150.31) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 644.03 seconds: 77.06 ops/second - unlinked 0 (time 1746157899 ; total 0 ; last 0) - unlinked 10000 (time 1746157943 ; total 44 ; last 44) - unlinked 20000 (time 1746157985 ; total 86 ; last 42) - unlinked 30000 (time 1746158023 ; total 124 ; last 38) - unlinked 40000 (time 1746158065 ; total 166 ; last 42) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 201 seconds: 246.925369 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg334-server unloading modules via unload_modules_local on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2