== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:01:10 (1746144070) start mds service on oleg245-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: MODOPTS_LIBCFS= oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg245-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 172 (time 1746144135.31 total 10.14 last 16.97) - open/close 438 (time 1746144145.31 total 20.14 last 26.60) - open/close 683 (time 1746144155.60 total 30.44 last 23.79) - open/close 889 (time 1746144166.08 total 40.91 last 19.66) - open/close 1175 (time 1746144177.07 total 51.90 last 26.04) - open/close 1377 (time 1746144187.07 total 61.90 last 20.20) - open/close 1566 (time 1746144198.03 total 72.86 last 17.24) - open/close 1900 (time 1746144210.41 total 85.24 last 26.97) - open/close 2080 (time 1746144220.62 total 95.45 last 17.63) - open/close 2302 (time 1746144230.99 total 105.82 last 21.41) - open/close 2606 (time 1746144241.06 total 115.90 last 30.18) - open/close 2826 (time 1746144251.07 total 125.90 last 21.99) - open/close 3089 (time 1746144261.51 total 136.35 last 25.17) - open/close 3373 (time 1746144271.60 total 146.44 last 28.15) - open/close 3638 (time 1746144282.05 total 156.88 last 25.37) - open/close 3941 (time 1746144292.06 total 166.90 last 30.26) - open/close 4089 (time 1746144303.62 total 178.45 last 12.81) - open/close 4218 (time 1746144315.34 total 190.17 last 11.01) - open/close 4409 (time 1746144325.52 total 200.35 last 18.77) - open/close 4624 (time 1746144335.52 total 210.35 last 21.49) - open/close 4846 (time 1746144346.33 total 221.16 last 20.55) - open/close 5095 (time 1746144357.27 total 232.10 last 22.75) - open/close 5303 (time 1746144368.09 total 242.92 last 19.22) - open/close 5511 (time 1746144378.18 total 253.01 last 20.62) - open/close 5710 (time 1746144388.27 total 263.10 last 19.72) - open/close 5982 (time 1746144399.14 total 273.98 last 25.01) - open/close 6224 (time 1746144409.79 total 284.62 last 22.74) - open/close 6476 (time 1746144422.08 total 296.91 last 20.50) - open/close 6691 (time 1746144432.56 total 307.39 last 20.50) - open/close 6851 (time 1746144442.57 total 317.40 last 15.99) - open/close 6954 (time 1746144452.93 total 327.76 last 9.94) - open/close 7268 (time 1746144463.76 total 338.59 last 29.00) - open/close 7514 (time 1746144475.88 total 350.71 last 20.29) - open/close 7652 (time 1746144486.07 total 360.90 last 13.54) - open/close 8032 (time 1746144496.07 total 370.90 last 37.99) - open/close 8454 (time 1746144506.33 total 381.16 last 41.14) - open/close 8653 (time 1746144516.67 total 391.50 last 19.25) - open/close 8867 (time 1746144530.55 total 405.39 last 15.41) - open/close 9048 (time 1746144540.81 total 415.64 last 17.66) - open/close 9265 (time 1746144551.55 total 426.38 last 20.20) - open/close 9494 (time 1746144561.62 total 436.45 last 22.73) - open/close 10000 (time 1746144570.03 total 444.86 last 60.16) - open/close 11399 (time 1746144580.04 total 454.87 last 139.80) - open/close 12698 (time 1746144590.04 total 464.88 last 129.84) - open/close 14019 (time 1746144600.05 total 474.88 last 132.00) - open/close 15463 (time 1746144610.06 total 484.89 last 144.33) - open/close 16804 (time 1746144620.06 total 494.89 last 134.02) - open/close 18350 (time 1746144630.06 total 504.90 last 154.58) - open/close 19895 (time 1746144640.07 total 514.90 last 154.47) - open/close 21432 (time 1746144650.07 total 524.90 last 153.66) - open/close 23210 (time 1746144660.07 total 534.90 last 177.79) - open/close 24448 (time 1746144670.08 total 544.92 last 123.60) - open/close 25590 (time 1746144680.11 total 554.94 last 113.95) - open/close 26940 (time 1746144690.11 total 564.94 last 134.99) - open/close 28264 (time 1746144700.11 total 574.94 last 132.37) - open/close 29580 (time 1746144710.13 total 584.96 last 131.33) - open/close 30000 (time 1746144713.61 total 588.44 last 120.57) - open/close 31354 (time 1746144723.61 total 598.45 last 135.39) - open/close 32843 (time 1746144733.62 total 608.45 last 148.83) - open/close 34478 (time 1746144743.63 total 618.46 last 163.37) - open/close 35872 (time 1746144753.63 total 628.47 last 139.28) - open/close 37090 (time 1746144763.64 total 638.47 last 121.73) - open/close 38365 (time 1746144773.65 total 648.48 last 127.43) - open/close 39825 (time 1746144783.65 total 658.48 last 145.96) - open/close 41229 (time 1746144793.65 total 668.48 last 140.39) - open/close 42557 (time 1746144803.65 total 678.48 last 132.76) - open/close 43697 (time 1746144813.66 total 688.49 last 113.93) - open/close 44871 (time 1746144823.66 total 698.49 last 117.39) - open/close 46040 (time 1746144833.66 total 708.49 last 116.86) - open/close 47403 (time 1746144843.67 total 718.50 last 136.21) - open/close 48685 (time 1746144853.67 total 728.50 last 128.18) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 736.88 seconds: 67.35 ops/second - unlinked 0 (time 1746144864 ; total 0 ; last 0) - unlinked 10000 (time 1746144932 ; total 68 ; last 68) - unlinked 20000 (time 1746144978 ; total 114 ; last 46) - unlinked 30000 (time 1746145024 ; total 160 ; last 46) - unlinked 40000 (time 1746145072 ; total 208 ; last 48) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 248 seconds: 200.129028 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2