== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:21:56 (1746156116) start mds service on oleg343-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing load_modules_local oleg343-server: Loading modules from /home/green/git/lustre-release/lustre oleg343-server: detected 4 online CPUs by sysfs oleg343-server: MODOPTS_LIBCFS= oleg343-server: Force libcfs to create 2 CPU partitions oleg343-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg343-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg343-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0000 oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 223 (time 1746156160.99 total 10.72 last 20.80) - open/close 472 (time 1746156171.00 total 20.73 last 24.88) - open/close 653 (time 1746156181.29 total 31.02 last 17.58) - open/close 964 (time 1746156192.20 total 41.94 last 28.50) - open/close 1239 (time 1746156202.52 total 52.25 last 26.67) - open/close 1512 (time 1746156213.16 total 62.89 last 25.64) - open/close 1749 (time 1746156223.76 total 73.50 last 22.36) - open/close 1951 (time 1746156234.52 total 84.25 last 18.78) - open/close 2263 (time 1746156244.53 total 94.26 last 31.17) - open/close 2488 (time 1746156254.71 total 104.44 last 22.11) - open/close 2713 (time 1746156265.18 total 114.91 last 21.49) - open/close 2954 (time 1746156275.18 total 124.91 last 24.09) - open/close 3136 (time 1746156285.19 total 134.92 last 18.19) - open/close 3356 (time 1746156295.33 total 145.06 last 21.69) - open/close 3514 (time 1746156305.72 total 155.45 last 15.21) - open/close 3694 (time 1746156316.00 total 165.73 last 17.52) - open/close 3900 (time 1746156326.30 total 176.03 last 19.99) - open/close 4179 (time 1746156336.31 total 186.04 last 27.87) - open/close 4460 (time 1746156347.19 total 196.92 last 25.84) - open/close 4692 (time 1746156357.20 total 206.93 last 23.18) - open/close 4896 (time 1746156367.20 total 216.93 last 20.39) - open/close 5116 (time 1746156377.34 total 227.07 last 21.71) - open/close 5388 (time 1746156387.65 total 237.39 last 26.36) - open/close 5638 (time 1746156397.78 total 247.51 last 24.69) - open/close 5844 (time 1746156407.78 total 257.51 last 20.60) - open/close 6041 (time 1746156417.78 total 267.51 last 19.70) - open/close 6295 (time 1746156428.31 total 278.04 last 24.12) - open/close 6599 (time 1746156439.24 total 288.97 last 27.82) - open/close 6860 (time 1746156449.65 total 299.38 last 25.07) - open/close 7023 (time 1746156459.95 total 309.68 last 15.82) - open/close 7171 (time 1746156470.35 total 320.08 last 14.24) - open/close 7489 (time 1746156480.35 total 330.08 last 31.78) - open/close 7750 (time 1746156491.02 total 340.75 last 24.48) - open/close 8048 (time 1746156501.81 total 351.54 last 27.61) - open/close 8255 (time 1746156512.29 total 362.02 last 19.75) - open/close 8462 (time 1746156522.70 total 372.43 last 19.89) - open/close 8653 (time 1746156533.05 total 382.78 last 18.45) - open/close 8949 (time 1746156543.66 total 393.39 last 27.89) - open/close 9162 (time 1746156553.95 total 403.68 last 20.70) - open/close 9429 (time 1746156564.21 total 413.94 last 26.01) - open/close 10000 (time 1746156571.91 total 421.64 last 74.24) - open/close 11941 (time 1746156581.91 total 431.64 last 194.09) - open/close 13792 (time 1746156591.91 total 441.64 last 184.99) - open/close 15641 (time 1746156601.91 total 451.64 last 184.87) - open/close 17122 (time 1746156611.92 total 461.65 last 148.07) - open/close 18570 (time 1746156621.92 total 471.65 last 144.78) - open/close 20000 (time 1746156630.05 total 479.78 last 175.88) - open/close 21623 (time 1746156640.06 total 489.79 last 162.15) - open/close 23255 (time 1746156650.06 total 499.79 last 163.15) - open/close 24537 (time 1746156660.07 total 509.80 last 128.13) - open/close 25767 (time 1746156670.07 total 519.80 last 123.00) - open/close 26972 (time 1746156680.07 total 529.81 last 120.40) - open/close 28241 (time 1746156690.08 total 539.81 last 126.78) - open/close 29473 (time 1746156700.09 total 549.82 last 123.15) - open/close 30000 (time 1746156703.88 total 553.61 last 139.08) - open/close 31447 (time 1746156713.90 total 563.63 last 144.43) - open/close 32792 (time 1746156723.90 total 573.63 last 134.42) - open/close 34111 (time 1746156733.92 total 583.65 last 131.73) - open/close 35525 (time 1746156743.92 total 593.65 last 141.33) - open/close 36865 (time 1746156753.93 total 603.66 last 133.85) - open/close 38180 (time 1746156763.93 total 613.66 last 131.50) - open/close 39679 (time 1746156773.93 total 623.67 last 149.86) - open/close 40000 (time 1746156776.39 total 626.12 last 130.66) - open/close 41413 (time 1746156786.40 total 636.13 last 141.18) - open/close 42867 (time 1746156796.41 total 646.14 last 145.26) - open/close 44283 (time 1746156806.42 total 656.15 last 141.44) - open/close 45900 (time 1746156816.42 total 666.15 last 161.66) - open/close 47233 (time 1746156826.43 total 676.16 last 133.27) - open/close 48334 (time 1746156836.44 total 686.17 last 109.99) - open/close 49499 (time 1746156846.44 total 696.17 last 116.44) total: 50000 open/close in 700.32 seconds: 71.40 ops/second - unlinked 0 (time 1746156852 ; total 0 ; last 0) - unlinked 10000 (time 1746156913 ; total 61 ; last 61) - unlinked 20000 (time 1746156961 ; total 109 ; last 48) - unlinked 30000 (time 1746157008 ; total 156 ; last 47) - unlinked 40000 (time 1746157059 ; total 207 ; last 51) total: 50000 unlinks in 257 seconds: 194.552536 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg343-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg343-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg343-server unloading modules via unload_modules_local on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2