== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 16:29:52 (1746131392) start mds service on oleg406-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing load_modules_local oleg406-server: Loading modules from /home/green/git/lustre-release/lustre oleg406-server: detected 4 online CPUs by sysfs oleg406-server: MODOPTS_LIBCFS= oleg406-server: Force libcfs to create 2 CPU partitions oleg406-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg406-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg406-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg406-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg406-server: oleg406-server.virtnet: executing set_default_debug -1 all pdsh@oleg406-client: oleg406-server: ssh exited with exit code 1 Started lustre-OST0000 oleg406-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg406-client.virtnet: -o user_xattr,flock 192.168.204.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 238 (time 1746131432.24 total 10.53 last 22.60) - open/close 434 (time 1746131442.42 total 20.71 last 19.25) - open/close 783 (time 1746131452.68 total 30.97 last 34.01) - open/close 996 (time 1746131463.48 total 41.78 last 19.72) - open/close 1117 (time 1746131473.49 total 51.78 last 12.10) - open/close 1384 (time 1746131484.23 total 62.52 last 24.86) - open/close 1613 (time 1746131494.23 total 72.52 last 22.89) - open/close 1803 (time 1746131504.23 total 82.52 last 19.00) - open/close 2147 (time 1746131514.28 total 92.57 last 34.23) - open/close 2343 (time 1746131525.15 total 103.44 last 18.04) - open/close 2610 (time 1746131536.15 total 114.44 last 24.26) - open/close 2854 (time 1746131547.17 total 125.46 last 22.15) - open/close 3104 (time 1746131557.95 total 136.24 last 23.18) - open/close 3353 (time 1746131567.97 total 146.26 last 24.86) - open/close 3617 (time 1746131578.47 total 156.76 last 25.14) - open/close 3985 (time 1746131589.25 total 167.54 last 34.15) - open/close 4247 (time 1746131599.25 total 177.54 last 26.20) - open/close 4434 (time 1746131609.99 total 188.28 last 17.41) - open/close 4716 (time 1746131620.47 total 198.76 last 26.90) - open/close 4975 (time 1746131630.47 total 208.76 last 25.89) - open/close 5239 (time 1746131640.77 total 219.06 last 25.63) - open/close 5489 (time 1746131650.77 total 229.06 last 25.00) - open/close 5718 (time 1746131660.77 total 239.07 last 22.90) - open/close 5942 (time 1746131670.78 total 249.07 last 22.39) - open/close 6211 (time 1746131681.34 total 259.63 last 25.47) - open/close 6456 (time 1746131691.48 total 269.78 last 24.14) - open/close 6717 (time 1746131702.20 total 280.49 last 24.36) - open/close 6976 (time 1746131712.48 total 290.77 last 25.20) - open/close 7184 (time 1746131722.73 total 301.02 last 20.29) - open/close 7418 (time 1746131732.90 total 311.20 last 23.00) - open/close 7629 (time 1746131745.02 total 323.31 last 17.42) - open/close 7835 (time 1746131755.72 total 334.01 last 19.25) - open/close 8066 (time 1746131767.11 total 345.41 last 20.28) - open/close 8379 (time 1746131777.38 total 355.67 last 30.48) - open/close 8456 (time 1746131789.37 total 367.66 last 6.42) - open/close 8651 (time 1746131799.57 total 377.86 last 19.11) - open/close 8764 (time 1746131809.94 total 388.23 last 10.89) - open/close 9066 (time 1746131820.58 total 398.88 last 28.38) - open/close 9273 (time 1746131830.61 total 408.90 last 20.65) - open/close 10000 (time 1746131840.45 total 418.74 last 73.86) - open/close 11153 (time 1746131850.46 total 428.75 last 115.21) - open/close 12311 (time 1746131860.46 total 438.76 last 115.74) - open/close 13882 (time 1746131870.46 total 448.76 last 157.08) - open/close 15328 (time 1746131880.47 total 458.76 last 144.57) - open/close 16462 (time 1746131890.48 total 468.77 last 113.30) - open/close 17494 (time 1746131900.49 total 478.78 last 103.09) - open/close 18521 (time 1746131910.49 total 488.78 last 102.66) - open/close 19899 (time 1746131920.50 total 498.79 last 137.71) - open/close 21094 (time 1746131930.50 total 508.79 last 119.45) - open/close 22234 (time 1746131940.50 total 518.80 last 113.97) - open/close 23593 (time 1746131950.51 total 528.80 last 135.84) - open/close 24809 (time 1746131960.51 total 538.80 last 121.58) - open/close 26275 (time 1746131970.52 total 548.81 last 146.53) - open/close 27656 (time 1746131980.52 total 558.81 last 138.07) - open/close 29082 (time 1746131990.52 total 568.81 last 142.59) - open/close 30000 (time 1746131998.26 total 576.56 last 118.53) - open/close 31008 (time 1746132008.27 total 586.56 last 100.74) - open/close 32145 (time 1746132018.27 total 596.56 last 113.68) - open/close 33390 (time 1746132028.28 total 606.57 last 124.36) - open/close 34580 (time 1746132038.29 total 616.58 last 118.95) - open/close 36016 (time 1746132048.29 total 626.59 last 143.50) - open/close 37390 (time 1746132058.30 total 636.59 last 137.34) - open/close 38803 (time 1746132068.30 total 646.59 last 141.27) - open/close 39911 (time 1746132078.31 total 656.60 last 110.73) - open/close 41088 (time 1746132088.31 total 666.60 last 117.67) - open/close 42148 (time 1746132098.31 total 676.60 last 105.98) - open/close 43227 (time 1746132108.31 total 686.60 last 107.90) - open/close 44295 (time 1746132118.31 total 696.61 last 106.77) - open/close 45432 (time 1746132128.32 total 706.61 last 113.63) - open/close 46741 (time 1746132138.33 total 716.63 last 130.70) - open/close 47881 (time 1746132148.34 total 726.64 last 113.90) - open/close 49213 (time 1746132158.34 total 736.64 last 133.18) total: 50000 open/close in 743.46 seconds: 67.25 ops/second - unlinked 0 (time 1746132166 ; total 0 ; last 0) - unlinked 10000 (time 1746132235 ; total 69 ; last 69) - unlinked 20000 (time 1746132291 ; total 125 ; last 56) - unlinked 30000 (time 1746132355 ; total 189 ; last 64) - unlinked 40000 (time 1746132403 ; total 237 ; last 48) total: 50000 unlinks in 279 seconds: 179.211472 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg406-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg406-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg406-server stop mds service on oleg406-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg406-server unloading modules via unload_modules_local on: 'oleg406-server' oleg406-server: oleg406-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-client: ssh exited with exit code 2 pdsh@oleg406-client: oleg406-server: ssh exited with exit code 2