== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:51:52 (1746139912) start mds service on oleg148-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing load_modules_local oleg148-server: Loading modules from /home/green/git/lustre-release/lustre oleg148-server: detected 4 online CPUs by sysfs oleg148-server: MODOPTS_LIBCFS= oleg148-server: Force libcfs to create 2 CPU partitions oleg148-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg148-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg148-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg148-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg148-server: oleg148-server.virtnet: executing set_default_debug -1 all pdsh@oleg148-client: oleg148-server: ssh exited with exit code 1 Started lustre-OST0000 oleg148-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg148-client.virtnet: -o user_xattr,flock 192.168.201.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 147 (time 1746139960.12 total 10.02 last 14.67) - open/close 321 (time 1746139970.33 total 20.23 last 17.05) - open/close 548 (time 1746139980.35 total 30.25 last 22.65) - open/close 692 (time 1746139990.77 total 40.67 last 13.82) - open/close 867 (time 1746140001.58 total 51.48 last 16.19) - open/close 1031 (time 1746140012.36 total 62.25 last 15.22) - open/close 1155 (time 1746140022.74 total 72.64 last 11.94) - open/close 1342 (time 1746140033.49 total 83.39 last 17.40) - open/close 1525 (time 1746140044.06 total 93.95 last 17.32) - open/close 1659 (time 1746140054.18 total 104.08 last 13.23) - open/close 1815 (time 1746140064.27 total 114.16 last 15.47) - open/close 1945 (time 1746140074.42 total 124.32 last 12.80) - open/close 2064 (time 1746140085.00 total 134.90 last 11.25) - open/close 2250 (time 1746140095.01 total 144.91 last 18.58) - open/close 2435 (time 1746140105.22 total 155.12 last 18.12) - open/close 2700 (time 1746140116.26 total 166.15 last 24.02) - open/close 2902 (time 1746140127.27 total 177.17 last 18.33) - open/close 3012 (time 1746140137.86 total 187.76 last 10.39) - open/close 3132 (time 1746140148.08 total 197.97 last 11.75) - open/close 3304 (time 1746140158.08 total 207.98 last 17.19) - open/close 3538 (time 1746140168.67 total 218.56 last 22.11) - open/close 3676 (time 1746140179.07 total 228.96 last 13.27) - open/close 3884 (time 1746140189.16 total 239.05 last 20.61) - open/close 4038 (time 1746140199.17 total 249.06 last 15.39) - open/close 4218 (time 1746140209.23 total 259.12 last 17.89) - open/close 4452 (time 1746140219.50 total 269.39 last 22.78) - open/close 4594 (time 1746140230.13 total 280.03 last 13.36) - open/close 4746 (time 1746140240.20 total 290.10 last 15.09) - open/close 4877 (time 1746140251.02 total 300.91 last 12.11) - open/close 5052 (time 1746140261.93 total 311.82 last 16.05) - open/close 5208 (time 1746140272.25 total 322.14 last 15.11) - open/close 5372 (time 1746140283.26 total 333.16 last 14.89) - open/close 5542 (time 1746140293.47 total 343.36 last 16.66) - open/close 5708 (time 1746140303.80 total 353.69 last 16.07) - open/close 5943 (time 1746140314.43 total 364.32 last 22.12) - open/close 6084 (time 1746140324.54 total 374.43 last 13.94) - open/close 6195 (time 1746140334.80 total 384.70 last 10.81) - open/close 6370 (time 1746140345.05 total 394.95 last 17.08) - open/close 6652 (time 1746140355.64 total 405.54 last 26.63) - open/close 6894 (time 1746140367.85 total 417.75 last 19.82) - open/close 7053 (time 1746140377.86 total 427.76 last 15.88) - open/close 7154 (time 1746140388.42 total 438.32 last 9.56) - open/close 7307 (time 1746140398.87 total 448.76 last 14.65) - open/close 7495 (time 1746140409.29 total 459.18 last 18.04) - open/close 7679 (time 1746140419.66 total 469.56 last 17.74) - open/close 7901 (time 1746140429.98 total 479.88 last 21.52) - open/close 8080 (time 1746140439.99 total 489.88 last 17.88) - open/close 8254 (time 1746140450.69 total 500.59 last 16.26) - open/close 8395 (time 1746140461.49 total 511.38 last 13.06) - open/close 8516 (time 1746140472.42 total 522.32 last 11.07) - open/close 8690 (time 1746140483.06 total 532.95 last 16.36) - open/close 8837 (time 1746140493.94 total 543.83 last 13.51) - open/close 8994 (time 1746140504.09 total 553.98 last 15.47) - open/close 9117 (time 1746140514.99 total 564.88 last 11.29) - open/close 9849 (time 1746140524.99 total 574.89 last 73.15) - open/close 10760 (time 1746140535.00 total 584.89 last 91.04) - open/close 11760 (time 1746140545.00 total 594.90 last 99.99) - open/close 12798 (time 1746140555.00 total 604.90 last 103.76) - open/close 13808 (time 1746140565.02 total 614.91 last 100.90) - open/close 14772 (time 1746140575.02 total 624.92 last 96.34) - open/close 15775 (time 1746140585.02 total 634.92 last 100.27) - open/close 16702 (time 1746140595.03 total 644.92 last 92.69) - open/close 17704 (time 1746140605.03 total 654.93 last 100.11) - open/close 18748 (time 1746140615.05 total 664.94 last 104.29) - open/close 19949 (time 1746140625.05 total 674.94 last 120.08) - open/close 21094 (time 1746140635.05 total 684.95 last 114.44) - open/close 22066 (time 1746140645.06 total 694.95 last 97.13) - open/close 23050 (time 1746140655.06 total 704.96 last 98.36) - open/close 24042 (time 1746140665.07 total 714.96 last 99.19) - open/close 25066 (time 1746140675.07 total 724.96 last 102.38) - open/close 25975 (time 1746140685.07 total 734.96 last 90.88) - open/close 27006 (time 1746140695.07 total 744.97 last 103.05) - open/close 28107 (time 1746140705.08 total 754.98 last 109.98) - open/close 29065 (time 1746140715.09 total 764.99 last 95.71) - open/close 30000 (time 1746140724.48 total 774.38 last 99.61) - open/close 31030 (time 1746140734.49 total 784.38 last 102.93) - open/close 31988 (time 1746140744.49 total 794.38 last 95.79) - open/close 32914 (time 1746140754.50 total 804.39 last 92.54) - open/close 33879 (time 1746140764.50 total 814.40 last 96.43) - open/close 34919 (time 1746140774.50 total 824.40 last 103.99) - open/close 35922 (time 1746140784.51 total 834.41 last 100.20) - open/close 36972 (time 1746140794.52 total 844.42 last 104.93) - open/close 37997 (time 1746140804.53 total 854.42 last 102.45) - open/close 39028 (time 1746140814.53 total 864.42 last 103.09) - open/close 39968 (time 1746140824.53 total 874.43 last 93.95) - open/close 40774 (time 1746140834.54 total 884.43 last 80.56) - open/close 41625 (time 1746140844.54 total 894.44 last 85.04) - open/close 42450 (time 1746140854.55 total 904.44 last 82.47) - open/close 43316 (time 1746140864.55 total 914.44 last 86.58) - open/close 44205 (time 1746140874.55 total 924.44 last 88.90) - open/close 45013 (time 1746140884.55 total 934.44 last 80.80) - open/close 45815 (time 1746140894.55 total 944.45 last 80.17) - open/close 46802 (time 1746140904.56 total 954.46 last 98.63) - open/close 47893 (time 1746140914.57 total 964.47 last 108.95) - open/close 48945 (time 1746140924.57 total 974.47 last 105.20) - open/close 49887 (time 1746140934.58 total 984.47 last 94.15) total: 50000 open/close in 985.61 seconds: 50.73 ops/second - unlinked 0 (time 1746140938 ; total 0 ; last 0) - unlinked 10000 (time 1746141023 ; total 85 ; last 85) - unlinked 20000 (time 1746141079 ; total 141 ; last 56) - unlinked 30000 (time 1746141132 ; total 194 ; last 53) - unlinked 40000 (time 1746141186 ; total 248 ; last 54) total: 50000 unlinks in 297 seconds: 168.350174 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg148-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg148-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg148-server stop mds service on oleg148-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg148-server unloading modules via unload_modules_local on: 'oleg148-server' oleg148-server: oleg148-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg148-client: oleg148-client: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-client: ssh exited with exit code 2 pdsh@oleg148-client: oleg148-server: ssh exited with exit code 2