== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:07:06 (1746137226) start mds service on oleg315-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing load_modules_local oleg315-server: Loading modules from /home/green/git/lustre-release/lustre oleg315-server: detected 4 online CPUs by sysfs oleg315-server: MODOPTS_LIBCFS= oleg315-server: Force libcfs to create 2 CPU partitions oleg315-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg315-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg315-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg315-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug -1 all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 126 (time 1746137285.58 total 13.61 last 9.26) - open/close 294 (time 1746137295.58 total 23.61 last 16.79) - open/close 446 (time 1746137307.31 total 35.34 last 12.96) - open/close 636 (time 1746137317.48 total 45.52 last 18.68) - open/close 857 (time 1746137328.31 total 56.34 last 20.41) - open/close 1014 (time 1746137341.09 total 69.13 last 12.28) - open/close 1116 (time 1746137351.70 total 79.73 last 9.62) - open/close 1264 (time 1746137361.74 total 89.77 last 14.74) - open/close 1388 (time 1746137372.39 total 100.43 last 11.64) - open/close 1578 (time 1746137382.92 total 110.95 last 18.05) - open/close 1760 (time 1746137393.85 total 121.88 last 16.65) - open/close 1941 (time 1746137406.01 total 134.04 last 14.89) - open/close 2133 (time 1746137416.03 total 144.06 last 19.16) - open/close 2315 (time 1746137426.40 total 154.43 last 17.55) - open/close 2507 (time 1746137436.98 total 165.01 last 18.15) - open/close 2752 (time 1746137447.87 total 175.90 last 22.51) - open/close 3033 (time 1746137458.70 total 186.73 last 25.94) - open/close 3248 (time 1746137469.12 total 197.15 last 20.64) - open/close 3366 (time 1746137479.27 total 207.31 last 11.62) - open/close 3632 (time 1746137489.29 total 217.32 last 26.56) - open/close 3905 (time 1746137499.58 total 227.61 last 26.53) - open/close 4203 (time 1746137510.58 total 238.61 last 27.09) - open/close 4428 (time 1746137520.58 total 248.61 last 22.49) - open/close 4730 (time 1746137531.45 total 259.48 last 27.80) - open/close 5024 (time 1746137541.45 total 269.49 last 29.38) - open/close 5302 (time 1746137552.14 total 280.17 last 26.02) - open/close 5635 (time 1746137564.80 total 292.83 last 26.30) - open/close 5917 (time 1746137574.95 total 302.99 last 27.76) - open/close 6112 (time 1746137585.16 total 313.19 last 19.11) - open/close 6337 (time 1746137596.16 total 324.20 last 20.45) - open/close 6526 (time 1746137606.95 total 334.98 last 17.52) - open/close 6677 (time 1746137617.17 total 345.20 last 14.77) - open/close 6955 (time 1746137627.17 total 355.20 last 27.80) - open/close 7126 (time 1746137637.18 total 365.21 last 17.09) - open/close 7317 (time 1746137647.67 total 375.71 last 18.20) - open/close 7539 (time 1746137658.31 total 386.34 last 20.87) - open/close 7643 (time 1746137669.11 total 397.14 last 9.63) - open/close 7892 (time 1746137680.05 total 408.09 last 22.76) - open/close 8063 (time 1746137690.71 total 418.75 last 16.04) - open/close 8277 (time 1746137702.17 total 430.20 last 18.69) - open/close 9968 (time 1746137712.17 total 440.20 last 169.09) - open/close 11584 (time 1746137722.17 total 450.20 last 161.56) - open/close 12798 (time 1746137732.17 total 460.20 last 121.36) - open/close 14339 (time 1746137742.17 total 470.20 last 154.09) - open/close 15980 (time 1746137752.18 total 480.21 last 164.02) - open/close 17300 (time 1746137762.18 total 490.21 last 131.98) - open/close 19085 (time 1746137772.18 total 500.21 last 178.47) - open/close 20000 (time 1746137776.35 total 504.38 last 219.38) - open/close 21691 (time 1746137786.35 total 514.38 last 169.10) - open/close 23730 (time 1746137796.36 total 524.39 last 203.82) - open/close 25106 (time 1746137806.36 total 534.39 last 137.58) - open/close 27202 (time 1746137816.36 total 544.39 last 209.53) - open/close 28502 (time 1746137826.37 total 554.40 last 129.93) - open/close 30000 (time 1746137835.74 total 563.77 last 159.82) - open/close 31678 (time 1746137845.74 total 573.78 last 167.71) - open/close 32679 (time 1746137855.75 total 583.78 last 100.08) - open/close 34013 (time 1746137865.75 total 593.78 last 133.33) - open/close 35594 (time 1746137875.75 total 603.78 last 158.08) - open/close 37230 (time 1746137885.76 total 613.79 last 163.53) - open/close 38765 (time 1746137895.77 total 623.80 last 153.38) - open/close 39941 (time 1746137905.78 total 633.81 last 117.45) - open/close 41049 (time 1746137915.78 total 643.81 last 110.75) - open/close 42109 (time 1746137925.79 total 653.82 last 105.94) - open/close 43298 (time 1746137935.79 total 663.82 last 118.87) - open/close 44964 (time 1746137945.79 total 673.82 last 166.57) - open/close 46356 (time 1746137955.80 total 683.83 last 139.14) - open/close 47535 (time 1746137965.80 total 693.83 last 117.87) - open/close 49041 (time 1746137975.80 total 703.84 last 150.52) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 707.61 seconds: 70.14 ops/second - unlinked 0 (time 1746137981 ; total 0 ; last 0) - unlinked 10000 (time 1746138032 ; total 51 ; last 51) - unlinked 20000 (time 1746138061 ; total 80 ; last 29) - unlinked 30000 (time 1746138106 ; total 125 ; last 45) - unlinked 40000 (time 1746138145 ; total 164 ; last 39) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 190 seconds: 261.221039 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg315-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server stop mds service on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server unloading modules via unload_modules_local on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg315-client: oleg315-client: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-client: ssh exited with exit code 2 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2