== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:25:04 (1750181104) start mds service on oleg426-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg426-server' oleg426-server: oleg426-server.virtnet: executing load_modules_local oleg426-server: Loading modules from /home/green/git/lustre-release/lustre oleg426-server: detected 4 online CPUs by sysfs oleg426-server: MODOPTS_LIBCFS= oleg426-server: Force libcfs to create 2 CPU partitions oleg426-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg426-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg426-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg426-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg426-server: oleg426-server.virtnet: executing set_default_debug -1 all pdsh@oleg426-client: oleg426-server: ssh exited with exit code 1 Started lustre-OST0000 oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg426-client.virtnet: -o user_xattr,flock 192.168.204.126@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 179 (time 1750181163.15 total 10.00 last 17.89) - open/close 353 (time 1750181173.15 total 20.01 last 17.39) - open/close 430 (time 1750181183.19 total 30.04 last 7.67) - open/close 676 (time 1750181193.93 total 40.79 last 22.90) - open/close 948 (time 1750181204.72 total 51.58 last 25.19) - open/close 1125 (time 1750181215.73 total 62.59 last 16.09) - open/close 1249 (time 1750181226.43 total 73.29 last 11.58) - open/close 1344 (time 1750181236.75 total 83.61 last 9.21) - open/close 1499 (time 1750181247.75 total 94.61 last 14.09) - open/close 1640 (time 1750181258.52 total 105.38 last 13.09) - open/close 1734 (time 1750181268.71 total 115.57 last 9.23) - open/close 1883 (time 1750181279.39 total 126.25 last 13.94) - open/close 2125 (time 1750181290.02 total 136.88 last 22.77) - open/close 2327 (time 1750181300.03 total 146.89 last 20.18) - open/close 2521 (time 1750181310.40 total 157.26 last 18.71) - open/close 2734 (time 1750181321.06 total 167.92 last 19.97) - open/close 2804 (time 1750181331.97 total 178.83 last 6.42) - open/close 3040 (time 1750181342.72 total 189.58 last 21.95) - open/close 3122 (time 1750181352.80 total 199.66 last 8.13) - open/close 3223 (time 1750181363.07 total 209.93 last 9.84) - open/close 3316 (time 1750181373.94 total 220.80 last 8.55) - open/close 3389 (time 1750181383.96 total 230.82 last 7.29) - open/close 3581 (time 1750181394.15 total 241.01 last 18.84) - open/close 3742 (time 1750181405.80 total 252.66 last 13.82) - open/close 3948 (time 1750181416.60 total 263.46 last 19.07) - open/close 4136 (time 1750181426.61 total 273.47 last 18.79) - open/close 4274 (time 1750181439.01 total 285.87 last 11.13) - open/close 4421 (time 1750181449.55 total 296.41 last 13.95) - open/close 4609 (time 1750181460.14 total 307.00 last 17.75) - open/close 4757 (time 1750181471.01 total 317.87 last 13.62) - open/close 4875 (time 1750181481.52 total 328.38 last 11.23) - open/close 4968 (time 1750181491.66 total 338.52 last 9.17) - open/close 5153 (time 1750181502.08 total 348.94 last 17.75) - open/close 5293 (time 1750181512.94 total 359.80 last 12.90) - open/close 5596 (time 1750181522.94 total 369.80 last 30.29) - open/close 5740 (time 1750181533.59 total 380.45 last 13.52) - open/close 5961 (time 1750181543.96 total 390.81 last 21.32) - open/close 6085 (time 1750181554.32 total 401.18 last 11.96) - open/close 6280 (time 1750181564.32 total 411.18 last 19.50) - open/close 6423 (time 1750181575.87 total 422.73 last 12.38) - open/close 6671 (time 1750181586.40 total 433.26 last 23.55) - open/close 6958 (time 1750181597.03 total 443.89 last 27.02) - open/close 7214 (time 1750181607.26 total 454.12 last 25.02) - open/close 7458 (time 1750181617.30 total 464.15 last 24.31) - open/close 7727 (time 1750181627.58 total 474.44 last 26.17) - open/close 7993 (time 1750181638.07 total 484.93 last 25.34) - open/close 8259 (time 1750181648.87 total 495.73 last 24.63) - open/close 8430 (time 1750181660.76 total 507.62 last 14.38) - open/close 9573 (time 1750181670.77 total 517.63 last 114.18) - open/close 10000 (time 1750181674.07 total 520.92 last 129.68) - open/close 11612 (time 1750181684.07 total 530.93 last 161.10) - open/close 13104 (time 1750181694.07 total 540.93 last 149.16) - open/close 14511 (time 1750181704.08 total 550.94 last 140.65) - open/close 16083 (time 1750181714.08 total 560.94 last 157.13) - open/close 17453 (time 1750181724.08 total 570.94 last 136.98) - open/close 18872 (time 1750181734.09 total 580.95 last 141.81) - open/close 20000 (time 1750181742.57 total 589.43 last 133.05) - open/close 21421 (time 1750181752.57 total 599.43 last 142.04) - open/close 22847 (time 1750181762.58 total 609.44 last 142.47) - open/close 24500 (time 1750181772.59 total 619.44 last 165.21) - open/close 26174 (time 1750181782.59 total 629.45 last 167.34) - open/close 27432 (time 1750181792.60 total 639.46 last 125.69) - open/close 28732 (time 1750181802.60 total 649.46 last 129.99) - open/close 30000 (time 1750181812.54 total 659.39 last 127.61) - open/close 31317 (time 1750181822.54 total 669.40 last 131.68) - open/close 32545 (time 1750181832.55 total 679.40 last 122.71) - open/close 33856 (time 1750181842.55 total 689.41 last 131.00) - open/close 35229 (time 1750181852.56 total 699.42 last 137.20) - open/close 36528 (time 1750181862.56 total 709.42 last 129.89) - open/close 37789 (time 1750181872.57 total 719.42 last 126.04) - open/close 39026 (time 1750181882.57 total 729.43 last 123.64) - open/close 40000 (time 1750181891.18 total 738.03 last 113.20) - open/close 41144 (time 1750181901.18 total 748.04 last 114.35) - open/close 42409 (time 1750181911.18 total 758.04 last 126.48) - open/close 43688 (time 1750181921.19 total 768.04 last 127.84) - open/close 44936 (time 1750181931.19 total 778.05 last 124.72) - open/close 46151 (time 1750181941.21 total 788.07 last 121.23) - open/close 47503 (time 1750181951.23 total 798.09 last 134.95) - open/close 48792 (time 1750181961.25 total 808.11 last 128.66) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 815.05 seconds: 60.89 ops/second - unlinked 0 (time 1750181969 ; total 0 ; last 0) - unlinked 10000 (time 1750182025 ; total 56 ; last 56) - unlinked 20000 (time 1750182071 ; total 102 ; last 46) - unlinked 30000 (time 1750182118 ; total 149 ; last 47) - unlinked 40000 (time 1750182164 ; total 195 ; last 46) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 240 seconds: 206.800003 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg426-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg426-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg426-server stop mds service on oleg426-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg426-server stop mds service on oleg426-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg426-server unloading modules via unload_modules_local on: 'oleg426-server' oleg426-server: oleg426-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg426-client: oleg426-client: ssh exited with exit code 2 pdsh@oleg426-client: oleg426-server: ssh exited with exit code 2 pdsh@oleg426-client: oleg426-client: ssh exited with exit code 2 pdsh@oleg426-client: oleg426-server: ssh exited with exit code 2