== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:50:34 (1708840234) start mds service on oleg415-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg415-server' oleg415-server: oleg415-server.virtnet: executing load_modules_local oleg415-server: Loading modules from /home/green/git/lustre-release/lustre oleg415-server: detected 4 online CPUs by sysfs oleg415-server: Force libcfs to create 2 CPU partitions oleg415-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg415-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg415-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg415-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg415-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-OST0000 oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg415-client.virtnet: -o user_xattr,flock oleg415-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 243 (time 1708840272.12 total 10.00 last 24.29) - open/close 601 (time 1708840282.62 total 20.50 last 34.10) - open/close 907 (time 1708840292.87 total 30.75 last 29.85) - open/close 1064 (time 1708840303.52 total 41.40 last 14.75) - open/close 1441 (time 1708840315.09 total 52.97 last 32.57) - open/close 1920 (time 1708840325.09 total 62.97 last 47.90) - open/close 2273 (time 1708840335.56 total 73.44 last 33.71) - open/close 2707 (time 1708840346.26 total 84.14 last 40.57) - open/close 3117 (time 1708840357.00 total 94.88 last 38.19) - open/close 3462 (time 1708840367.61 total 105.49 last 32.50) - open/close 3625 (time 1708840378.35 total 116.23 last 15.18) - open/close 3908 (time 1708840388.67 total 126.55 last 27.42) - open/close 4294 (time 1708840400.35 total 138.23 last 33.05) - open/close 4742 (time 1708840411.29 total 149.17 last 40.97) - open/close 5135 (time 1708840421.98 total 159.86 last 36.75) - open/close 5364 (time 1708840432.97 total 170.85 last 20.83) - open/close 5748 (time 1708840443.77 total 181.65 last 35.58) - open/close 6031 (time 1708840454.19 total 192.07 last 27.15) - open/close 6383 (time 1708840464.96 total 202.84 last 32.67) - open/close 6821 (time 1708840475.24 total 213.12 last 42.64) - open/close 7289 (time 1708840485.46 total 223.34 last 45.78) - open/close 7735 (time 1708840495.50 total 233.38 last 44.41) - open/close 8113 (time 1708840505.57 total 243.45 last 37.54) - open/close 8334 (time 1708840515.86 total 253.74 last 21.48) - open/close 8659 (time 1708840526.63 total 264.51 last 30.19) - open/close 8936 (time 1708840537.03 total 274.91 last 26.63) - open/close 9378 (time 1708840547.41 total 285.29 last 42.57) - open/close 9623 (time 1708840557.77 total 295.65 last 23.65) - open/close 9826 (time 1708840567.98 total 305.86 last 19.88) - open/close 10000 (time 1708840571.85 total 309.73 last 44.93) - open/close 10173 (time 1708840581.86 total 319.74 last 17.30) - open/close 11599 (time 1708840591.86 total 329.74 last 142.58) - open/close 13161 (time 1708840601.86 total 339.74 last 156.12) - open/close 14798 (time 1708840611.87 total 349.75 last 163.65) - open/close 16325 (time 1708840621.87 total 359.75 last 152.69) - open/close 17720 (time 1708840631.87 total 369.75 last 139.48) - open/close 19170 (time 1708840641.88 total 379.76 last 144.90) - open/close 20000 (time 1708840647.69 total 385.57 last 142.83) - open/close 21318 (time 1708840657.69 total 395.57 last 131.80) - open/close 22708 (time 1708840667.69 total 405.57 last 138.94) - open/close 24053 (time 1708840677.70 total 415.58 last 134.44) - open/close 25345 (time 1708840687.70 total 425.58 last 129.17) - open/close 26491 (time 1708840697.70 total 435.58 last 114.53) - open/close 27581 (time 1708840707.71 total 445.59 last 108.99) - open/close 28646 (time 1708840717.71 total 455.59 last 106.48) - open/close 29838 (time 1708840727.71 total 465.59 last 119.13) - open/close 30946 (time 1708840737.72 total 475.60 last 110.75) - open/close 31915 (time 1708840747.72 total 485.60 last 96.84) - open/close 32822 (time 1708840757.74 total 495.62 last 90.60) - open/close 33876 (time 1708840767.74 total 505.62 last 105.34) - open/close 34845 (time 1708840777.75 total 515.63 last 96.80) - open/close 35833 (time 1708840787.76 total 525.64 last 98.71) - open/close 36706 (time 1708840797.77 total 535.65 last 87.24) - open/close 37598 (time 1708840807.77 total 545.65 last 89.18) - open/close 38516 (time 1708840817.77 total 555.65 last 91.79) - open/close 39327 (time 1708840827.77 total 565.65 last 81.08) - open/close 40000 (time 1708840836.82 total 574.70 last 74.39) - open/close 40798 (time 1708840846.82 total 584.70 last 79.77) - open/close 41684 (time 1708840856.83 total 594.71 last 88.54) - open/close 42507 (time 1708840866.85 total 604.72 last 82.18) - open/close 43234 (time 1708840876.85 total 614.73 last 72.68) - open/close 43963 (time 1708840886.85 total 624.73 last 72.87) - open/close 44691 (time 1708840896.86 total 634.74 last 72.73) - open/close 45439 (time 1708840906.87 total 644.75 last 74.73) - open/close 46200 (time 1708840916.88 total 654.76 last 76.05) - open/close 46954 (time 1708840926.89 total 664.77 last 75.34) - open/close 47628 (time 1708840936.90 total 674.78 last 67.33) - open/close 48319 (time 1708840946.90 total 684.78 last 69.10) - open/close 49003 (time 1708840956.91 total 694.79 last 68.34) - open/close 49661 (time 1708840966.91 total 704.79 last 65.79) total: 50000 open/close in 709.75 seconds: 70.45 ops/second - unlinked 0 (time 1708840973 ; total 0 ; last 0) - unlinked 10000 (time 1708841094 ; total 121 ; last 121) - unlinked 20000 (time 1708841190 ; total 217 ; last 96) - unlinked 30000 (time 1708841265 ; total 292 ; last 75) - unlinked 40000 (time 1708841322 ; total 349 ; last 57) total: 50000 unlinks in 383 seconds: 130.548309 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg415-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg415-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server stop mds service on oleg415-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg415-server unloading modules on: 'oleg415-server' oleg415-server: oleg415-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-client: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-client: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2