== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:34:33 (1734676473) start mds service on oleg211-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing load_modules_local oleg211-server: Loading modules from /home/green/git/lustre-release/lustre oleg211-server: detected 4 online CPUs by sysfs oleg211-server: MODOPTS_LIBCFS= oleg211-server: Force libcfs to create 2 CPU partitions oleg211-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg211-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg211-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg211-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg211-server: oleg211-server.virtnet: executing set_default_debug -1 all pdsh@oleg211-client: oleg211-server: ssh exited with exit code 1 Started lustre-OST0000 oleg211-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg211-client.virtnet: -o user_xattr,flock 192.168.202.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 263 (time 1734676514.05 total 10.44 last 25.18) - open/close 445 (time 1734676524.49 total 20.89 last 17.42) - open/close 597 (time 1734676534.87 total 31.26 last 14.65) - open/close 786 (time 1734676546.38 total 42.78 last 16.42) - open/close 1025 (time 1734676557.29 total 53.69 last 21.90) - open/close 1307 (time 1734676567.99 total 64.39 last 26.36) - open/close 1483 (time 1734676578.33 total 74.72 last 17.03) - open/close 1647 (time 1734676588.51 total 84.91 last 16.09) - open/close 1874 (time 1734676599.71 total 96.11 last 20.28) - open/close 2101 (time 1734676610.31 total 106.70 last 21.42) - open/close 2429 (time 1734676620.34 total 116.74 last 32.69) - open/close 2648 (time 1734676630.37 total 126.77 last 21.83) - open/close 2890 (time 1734676640.41 total 136.81 last 24.10) - open/close 3119 (time 1734676650.52 total 146.92 last 22.66) - open/close 3296 (time 1734676660.72 total 157.12 last 17.35) - open/close 3582 (time 1734676671.52 total 167.92 last 26.49) - open/close 3873 (time 1734676681.52 total 177.92 last 29.09) - open/close 4128 (time 1734676691.83 total 188.22 last 24.74) - open/close 4270 (time 1734676704.17 total 200.56 last 11.51) - open/close 4486 (time 1734676714.17 total 210.57 last 21.59) - open/close 4688 (time 1734676724.39 total 220.78 last 19.78) - open/close 4798 (time 1734676735.26 total 231.66 last 10.11) - open/close 4983 (time 1734676747.53 total 243.93 last 15.08) - open/close 5165 (time 1734676757.56 total 253.96 last 18.14) - open/close 5357 (time 1734676768.09 total 264.49 last 18.23) - open/close 5547 (time 1734676778.53 total 274.92 last 18.21) - open/close 5657 (time 1734676788.54 total 284.94 last 10.99) - open/close 5874 (time 1734676798.63 total 295.02 last 21.51) - open/close 6105 (time 1734676809.47 total 305.86 last 21.31) - open/close 6307 (time 1734676820.38 total 316.77 last 18.52) - open/close 6517 (time 1734676832.24 total 328.63 last 17.71) - open/close 6713 (time 1734676842.90 total 339.30 last 18.37) - open/close 7028 (time 1734676853.88 total 350.28 last 28.69) - open/close 7259 (time 1734676863.89 total 360.29 last 23.08) - open/close 7428 (time 1734676874.36 total 370.76 last 16.14) - open/close 7687 (time 1734676885.34 total 381.73 last 23.60) - open/close 7898 (time 1734676897.60 total 394.00 last 17.20) - open/close 8919 (time 1734676907.61 total 404.00 last 102.07) - open/close 10000 (time 1734676915.73 total 412.12 last 133.13) - open/close 11328 (time 1734676925.73 total 422.12 last 132.79) - open/close 12746 (time 1734676935.73 total 432.13 last 141.77) - open/close 14144 (time 1734676945.74 total 442.13 last 139.71) - open/close 15632 (time 1734676955.74 total 452.13 last 148.79) - open/close 17112 (time 1734676965.74 total 462.14 last 147.96) - open/close 18844 (time 1734676975.74 total 472.14 last 173.18) - open/close 20000 (time 1734676981.36 total 477.76 last 205.58) - open/close 21529 (time 1734676991.37 total 487.76 last 152.85) - open/close 22782 (time 1734677001.37 total 497.77 last 125.29) - open/close 24366 (time 1734677011.37 total 507.77 last 158.34) - open/close 25726 (time 1734677021.38 total 517.77 last 135.95) - open/close 26968 (time 1734677031.38 total 527.78 last 124.16) - open/close 28340 (time 1734677041.38 total 537.78 last 137.19) - open/close 29956 (time 1734677051.38 total 547.78 last 161.52) - open/close 31517 (time 1734677061.38 total 557.78 last 156.10) - open/close 32979 (time 1734677071.39 total 567.79 last 146.12) - open/close 34267 (time 1734677081.39 total 577.79 last 128.78) - open/close 35529 (time 1734677091.40 total 587.80 last 126.10) - open/close 36792 (time 1734677101.41 total 597.80 last 126.22) - open/close 38045 (time 1734677111.41 total 607.81 last 125.28) - open/close 39310 (time 1734677121.42 total 617.81 last 126.41) - open/close 40000 (time 1734677126.96 total 623.36 last 124.37) - open/close 41213 (time 1734677136.97 total 633.37 last 121.25) - open/close 42452 (time 1734677146.98 total 643.37 last 123.80) - open/close 43637 (time 1734677156.98 total 653.38 last 118.46) - open/close 44824 (time 1734677166.98 total 663.38 last 118.70) - open/close 46123 (time 1734677176.98 total 673.38 last 129.89) - open/close 47388 (time 1734677186.98 total 683.38 last 126.49) - open/close 48567 (time 1734677196.98 total 693.38 last 117.87) - open/close 49690 (time 1734677206.99 total 703.38 last 112.26) total: 50000 open/close in 706.14 seconds: 70.81 ops/second - unlinked 0 (time 1734677211 ; total 0 ; last 0) - unlinked 10000 (time 1734677264 ; total 53 ; last 53) - unlinked 20000 (time 1734677303 ; total 92 ; last 39) - unlinked 30000 (time 1734677349 ; total 138 ; last 46) - unlinked 40000 (time 1734677397 ; total 186 ; last 48) total: 50000 unlinks in 230 seconds: 217.391312 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg211-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg211-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg211-server stop mds service on oleg211-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg211-server unloading modules via unload_modules_local on: 'oleg211-server' oleg211-server: oleg211-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-client: ssh exited with exit code 2 pdsh@oleg211-client: oleg211-server: ssh exited with exit code 2