== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:17:32 (1746166652) start mds service on oleg317-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing load_modules_local oleg317-server: Loading modules from /home/green/git/lustre-release/lustre oleg317-server: detected 4 online CPUs by sysfs oleg317-server: MODOPTS_LIBCFS= oleg317-server: Force libcfs to create 2 CPU partitions oleg317-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg317-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg317-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg317-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg317-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg317-server: oleg317-server.virtnet: executing set_default_debug -1 all pdsh@oleg317-client: oleg317-server: ssh exited with exit code 1 Started lustre-OST0000 oleg317-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg317-client.virtnet: -o user_xattr,flock 192.168.203.117@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 295 (time 1746166713.11 total 10.95 last 26.93) - open/close 535 (time 1746166723.33 total 21.17 last 23.49) - open/close 712 (time 1746166733.99 total 31.83 last 16.60) - open/close 957 (time 1746166744.00 total 41.84 last 24.49) - open/close 1231 (time 1746166754.60 total 52.44 last 25.84) - open/close 1530 (time 1746166764.60 total 62.44 last 29.89) - open/close 1756 (time 1746166774.63 total 72.47 last 22.53) - open/close 1926 (time 1746166785.43 total 83.26 last 15.75) - open/close 2220 (time 1746166796.18 total 94.02 last 27.35) - open/close 2548 (time 1746166806.18 total 104.02 last 32.79) - open/close 2766 (time 1746166817.02 total 114.86 last 20.10) - open/close 2967 (time 1746166827.70 total 125.54 last 18.81) - open/close 3216 (time 1746166838.37 total 136.21 last 23.35) - open/close 3373 (time 1746166848.94 total 146.78 last 14.85) - open/close 3539 (time 1746166858.95 total 156.79 last 16.59) - open/close 3748 (time 1746166868.95 total 166.79 last 20.90) - open/close 3875 (time 1746166879.59 total 177.43 last 11.93) - open/close 3944 (time 1746166889.59 total 187.43 last 6.90) - open/close 4071 (time 1746166902.15 total 199.99 last 10.12) - open/close 4239 (time 1746166912.15 total 209.99 last 16.79) - open/close 4435 (time 1746166924.73 total 222.57 last 15.58) - open/close 4611 (time 1746166934.79 total 232.63 last 17.50) - open/close 4694 (time 1746166945.03 total 242.87 last 8.10) - open/close 4768 (time 1746166955.05 total 252.89 last 7.38) - open/close 4909 (time 1746166965.91 total 263.75 last 12.99) - open/close 5025 (time 1746166976.23 total 274.07 last 11.23) - open/close 5196 (time 1746166986.24 total 284.08 last 17.08) - open/close 5474 (time 1746166996.92 total 294.76 last 26.04) - open/close 5660 (time 1746167006.92 total 304.76 last 18.60) - open/close 5871 (time 1746167016.92 total 314.76 last 21.09) - open/close 6136 (time 1746167026.92 total 324.76 last 26.49) - open/close 6379 (time 1746167037.66 total 335.50 last 22.64) - open/close 6611 (time 1746167047.69 total 345.53 last 23.12) - open/close 6927 (time 1746167057.70 total 355.54 last 31.58) - open/close 7141 (time 1746167069.78 total 367.62 last 17.71) - open/close 7225 (time 1746167081.81 total 379.65 last 6.98) - open/close 7476 (time 1746167092.65 total 390.49 last 23.16) - open/close 7703 (time 1746167102.84 total 400.67 last 22.29) - open/close 7882 (time 1746167113.25 total 411.09 last 17.18) - open/close 8108 (time 1746167124.04 total 421.88 last 20.95) - open/close 8292 (time 1746167136.60 total 434.44 last 14.66) - open/close 8462 (time 1746167146.76 total 444.60 last 16.73) - open/close 8577 (time 1746167157.42 total 455.26 last 10.79) - open/close 8712 (time 1746167168.42 total 466.26 last 12.27) - open/close 8835 (time 1746167179.18 total 477.02 last 11.43) - open/close 9200 (time 1746167189.19 total 487.03 last 36.48) - open/close 10000 (time 1746167195.75 total 493.59 last 121.87) - open/close 11225 (time 1746167205.75 total 503.59 last 122.49) - open/close 12303 (time 1746167215.76 total 513.60 last 107.71) - open/close 13438 (time 1746167225.76 total 523.60 last 113.50) - open/close 14483 (time 1746167235.77 total 533.61 last 104.44) - open/close 15702 (time 1746167245.77 total 543.61 last 121.84) - open/close 16934 (time 1746167255.78 total 553.62 last 123.14) - open/close 18142 (time 1746167265.78 total 563.62 last 120.74) - open/close 19417 (time 1746167275.78 total 573.62 last 127.49) - open/close 20000 (time 1746167280.42 total 578.26 last 125.57) - open/close 21142 (time 1746167290.43 total 588.27 last 114.13) - open/close 22289 (time 1746167300.43 total 598.27 last 114.66) - open/close 23518 (time 1746167310.43 total 608.27 last 122.90) - open/close 24843 (time 1746167320.44 total 618.28 last 132.43) - open/close 26075 (time 1746167330.44 total 628.28 last 123.16) - open/close 27268 (time 1746167340.44 total 638.28 last 119.29) - open/close 28502 (time 1746167350.44 total 648.28 last 123.39) - open/close 29727 (time 1746167360.45 total 658.29 last 122.41) - open/close 30000 (time 1746167362.60 total 660.44 last 126.86) - open/close 31188 (time 1746167372.61 total 670.45 last 118.67) - open/close 32392 (time 1746167382.62 total 680.46 last 120.35) - open/close 33597 (time 1746167392.62 total 690.46 last 120.49) - open/close 34961 (time 1746167402.62 total 700.46 last 136.36) - open/close 36205 (time 1746167412.62 total 710.46 last 124.37) - open/close 37409 (time 1746167422.63 total 720.47 last 120.29) - open/close 38573 (time 1746167432.64 total 730.48 last 116.33) - open/close 39778 (time 1746167442.65 total 740.49 last 120.40) - open/close 41001 (time 1746167452.66 total 750.50 last 122.16) - open/close 42085 (time 1746167462.66 total 760.50 last 108.35) - open/close 43148 (time 1746167472.67 total 770.51 last 106.21) - open/close 44250 (time 1746167482.68 total 780.52 last 110.15) - open/close 45371 (time 1746167492.68 total 790.52 last 112.03) - open/close 46551 (time 1746167502.68 total 800.52 last 117.98) - open/close 47495 (time 1746167512.69 total 810.52 last 94.40) - open/close 48515 (time 1746167522.70 total 820.54 last 101.89) - open/close 49478 (time 1746167532.70 total 830.54 last 96.26) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 831.87 seconds: 59.66 ops/second - unlinked 0 (time 1746167536 ; total 0 ; last 0) - unlinked 10000 (time 1746167613 ; total 77 ; last 77) - unlinked 20000 (time 1746167662 ; total 126 ; last 49) - unlinked 30000 (time 1746167711 ; total 175 ; last 49) - unlinked 40000 (time 1746167756 ; total 220 ; last 45) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 231 seconds: 214.857147 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg317-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg317-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg317-server stop mds service on oleg317-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg317-server unloading modules via unload_modules_local on: 'oleg317-server' oleg317-server: oleg317-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-client: ssh exited with exit code 2 pdsh@oleg317-client: oleg317-server: ssh exited with exit code 2