== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:38:11 (1746113891) start mds service on oleg236-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg236-server' oleg236-server: oleg236-server.virtnet: executing load_modules_local oleg236-server: Loading modules from /home/green/git/lustre-release/lustre oleg236-server: detected 4 online CPUs by sysfs oleg236-server: MODOPTS_LIBCFS= oleg236-server: Force libcfs to create 2 CPU partitions oleg236-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg236-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg236-server: oleg236-server.virtnet: executing set_default_debug -1 all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg236-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg236-server: oleg236-server.virtnet: executing set_default_debug -1 all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg236-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg236-server: oleg236-server.virtnet: executing set_default_debug -1 all pdsh@oleg236-client: oleg236-server: ssh exited with exit code 1 Started lustre-OST0000 oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg236-client.virtnet: -o user_xattr,flock 192.168.202.136@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 155 (time 1746113946.24 total 11.69 last 13.26) - open/close 281 (time 1746113956.77 total 22.22 last 11.97) - open/close 473 (time 1746113967.59 total 33.04 last 17.73) - open/close 586 (time 1746113978.09 total 43.54 last 10.76) - open/close 783 (time 1746113988.94 total 54.39 last 18.16) - open/close 924 (time 1746113999.53 total 64.98 last 13.31) - open/close 938 (time 1746114009.62 total 75.07 last 1.39) - open/close 1043 (time 1746114020.07 total 85.52 last 10.04) - open/close 1171 (time 1746114030.63 total 96.08 last 12.12) - open/close 1414 (time 1746114041.62 total 107.07 last 22.10) - open/close 1709 (time 1746114052.62 total 118.07 last 26.82) - open/close 1919 (time 1746114064.47 total 129.92 last 17.72) - open/close 1937 (time 1746114076.58 total 142.03 last 1.49) - open/close 2072 (time 1746114087.10 total 152.55 last 12.82) - open/close 2217 (time 1746114097.71 total 163.17 last 13.66) - open/close 2341 (time 1746114108.14 total 173.59 last 11.89) - open/close 2637 (time 1746114118.31 total 183.76 last 29.12) - open/close 2851 (time 1746114130.21 total 195.66 last 17.98) - open/close 2912 (time 1746114140.51 total 205.96 last 5.92) - open/close 2981 (time 1746114150.83 total 216.28 last 6.69) - open/close 3043 (time 1746114162.11 total 227.56 last 5.50) - open/close 3229 (time 1746114172.86 total 238.31 last 17.30) - open/close 3457 (time 1746114184.70 total 250.15 last 19.26) - open/close 3848 (time 1746114195.29 total 260.74 last 36.92) - open/close 4038 (time 1746114206.06 total 271.51 last 17.63) - open/close 4155 (time 1746114216.57 total 282.02 last 11.13) - open/close 4396 (time 1746114227.42 total 292.87 last 22.23) - open/close 4571 (time 1746114238.18 total 303.63 last 16.26) - open/close 4651 (time 1746114250.56 total 316.01 last 6.46) - open/close 4784 (time 1746114261.14 total 326.59 last 12.57) - open/close 5280 (time 1746114271.14 total 336.59 last 49.58) - open/close 7878 (time 1746114281.14 total 346.60 last 259.70) - open/close 10000 (time 1746114289.21 total 354.66 last 263.00) - open/close 12559 (time 1746114299.22 total 364.67 last 255.80) - open/close 15033 (time 1746114309.22 total 374.67 last 247.37) - open/close 17485 (time 1746114319.22 total 384.67 last 245.14) - open/close 19919 (time 1746114329.22 total 394.67 last 243.34) - open/close 22719 (time 1746114339.22 total 404.68 last 279.95) - open/close 25327 (time 1746114349.23 total 414.68 last 260.77) - open/close 27787 (time 1746114359.23 total 424.68 last 245.90) - open/close 30000 (time 1746114368.41 total 433.86 last 241.00) - open/close 32474 (time 1746114378.41 total 443.86 last 247.37) - open/close 35155 (time 1746114388.41 total 453.87 last 268.08) - open/close 37533 (time 1746114398.42 total 463.87 last 237.77) - open/close 39978 (time 1746114408.42 total 473.87 last 244.42) - open/close 42316 (time 1746114418.42 total 483.87 last 233.71) - open/close 45179 (time 1746114428.42 total 493.87 last 286.27) - open/close 47645 (time 1746114438.43 total 503.88 last 246.56) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 512.37 seconds: 96.87 ops/second - unlinked 0 (time 1746114448 ; total 0 ; last 0) - unlinked 10000 (time 1746114469 ; total 21 ; last 21) - unlinked 20000 (time 1746114491 ; total 43 ; last 22) - unlinked 30000 (time 1746114514 ; total 66 ; last 23) - unlinked 40000 (time 1746114536 ; total 88 ; last 22) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 109 seconds: 455.339447 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg236-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg236-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg236-server stop mds service on oleg236-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg236-server stop mds service on oleg236-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg236-server unloading modules via unload_modules_local on: 'oleg236-server' oleg236-server: oleg236-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg236-client: oleg236-client: ssh exited with exit code 2 pdsh@oleg236-client: oleg236-server: ssh exited with exit code 2 pdsh@oleg236-client: oleg236-client: ssh exited with exit code 2 pdsh@oleg236-client: oleg236-server: ssh exited with exit code 2