== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:03:20 (1734678200) start mds service on oleg326-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg326-server' oleg326-server: oleg326-server.virtnet: executing load_modules_local oleg326-server: Loading modules from /home/green/git/lustre-release/lustre oleg326-server: detected 4 online CPUs by sysfs oleg326-server: MODOPTS_LIBCFS= oleg326-server: Force libcfs to create 2 CPU partitions oleg326-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory oleg326-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg326-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg326-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg326-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg326-server: oleg326-server.virtnet: executing set_default_debug -1 all pdsh@oleg326-client: oleg326-server: ssh exited with exit code 1 Started lustre-OST0000 oleg326-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg326-client.virtnet: -o user_xattr,flock 192.168.203.126@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 258 (time 1734678249.94 total 10.46 last 24.68) - open/close 564 (time 1734678260.69 total 21.20 last 28.48) - open/close 807 (time 1734678270.74 total 31.25 last 24.18) - open/close 1016 (time 1734678281.28 total 41.80 last 19.82) - open/close 1281 (time 1734678291.29 total 51.80 last 26.49) - open/close 1367 (time 1734678302.89 total 63.40 last 7.41) - open/close 1565 (time 1734678313.42 total 73.94 last 18.79) - open/close 1736 (time 1734678323.84 total 84.35 last 16.42) - open/close 2034 (time 1734678334.07 total 94.59 last 29.11) - open/close 2324 (time 1734678344.47 total 104.98 last 27.91) - open/close 2513 (time 1734678354.91 total 115.42 last 18.10) - open/close 2765 (time 1734678365.90 total 126.41 last 22.94) - open/close 2963 (time 1734678375.91 total 136.42 last 19.77) - open/close 3266 (time 1734678386.35 total 146.87 last 29.01) - open/close 3484 (time 1734678396.94 total 157.45 last 20.59) - open/close 3731 (time 1734678407.22 total 167.74 last 24.01) - open/close 3974 (time 1734678417.35 total 177.86 last 24.01) - open/close 4155 (time 1734678427.97 total 188.48 last 17.04) - open/close 4437 (time 1734678438.52 total 199.04 last 26.71) - open/close 4618 (time 1734678448.64 total 209.15 last 17.89) - open/close 4875 (time 1734678458.79 total 219.31 last 25.31) - open/close 5141 (time 1734678468.81 total 229.32 last 26.56) - open/close 5377 (time 1734678478.81 total 239.33 last 23.59) - open/close 5683 (time 1734678489.42 total 249.94 last 28.85) - open/close 5931 (time 1734678500.29 total 260.80 last 22.82) - open/close 6264 (time 1734678511.15 total 271.66 last 30.67) - open/close 6451 (time 1734678521.53 total 282.04 last 18.02) - open/close 6687 (time 1734678531.53 total 292.05 last 23.59) - open/close 6874 (time 1734678542.47 total 302.98 last 17.10) - open/close 7133 (time 1734678552.74 total 313.26 last 25.20) - open/close 7328 (time 1734678562.76 total 323.27 last 19.47) - open/close 7514 (time 1734678572.78 total 333.29 last 18.56) - open/close 7815 (time 1734678583.51 total 344.03 last 28.04) - open/close 8034 (time 1734678594.31 total 354.82 last 20.29) - open/close 8273 (time 1734678604.32 total 364.84 last 23.87) - open/close 8494 (time 1734678614.44 total 374.96 last 21.84) - open/close 8708 (time 1734678624.63 total 385.15 last 21.00) - open/close 8992 (time 1734678634.96 total 395.47 last 27.50) - open/close 9314 (time 1734678645.75 total 406.26 last 29.85) - open/close 9595 (time 1734678655.75 total 416.27 last 28.08) - open/close 9855 (time 1734678666.16 total 426.68 last 24.98) - open/close 11214 (time 1734678676.16 total 436.68 last 135.89) - open/close 12749 (time 1734678686.17 total 446.68 last 153.49) - open/close 14301 (time 1734678696.18 total 456.69 last 155.04) - open/close 15750 (time 1734678706.18 total 466.70 last 144.81) - open/close 17230 (time 1734678716.19 total 476.71 last 147.85) - open/close 18822 (time 1734678726.20 total 486.71 last 159.06) - open/close 20000 (time 1734678733.81 total 494.32 last 154.85) - open/close 21701 (time 1734678743.81 total 504.32 last 170.08) - open/close 23255 (time 1734678753.81 total 514.33 last 155.35) - open/close 24816 (time 1734678763.81 total 524.33 last 156.09) - open/close 26228 (time 1734678773.82 total 534.33 last 141.14) - open/close 27808 (time 1734678783.82 total 544.34 last 157.91) - open/close 29441 (time 1734678793.82 total 554.34 last 163.29) - open/close 30000 (time 1734678797.42 total 557.93 last 155.48) - open/close 31500 (time 1734678807.42 total 567.94 last 149.97) - open/close 32841 (time 1734678817.42 total 577.94 last 134.09) - open/close 34260 (time 1734678827.42 total 587.94 last 141.87) - open/close 35666 (time 1734678837.43 total 597.94 last 140.57) - open/close 36951 (time 1734678847.43 total 607.94 last 128.45) - open/close 38359 (time 1734678857.43 total 617.95 last 140.75) - open/close 39756 (time 1734678867.44 total 627.95 last 139.65) - open/close 41160 (time 1734678877.44 total 637.96 last 140.32) - open/close 42584 (time 1734678887.45 total 647.96 last 142.29) - open/close 43893 (time 1734678897.45 total 657.97 last 130.85) - open/close 45150 (time 1734678907.45 total 667.97 last 125.70) - open/close 46435 (time 1734678917.47 total 677.98 last 128.36) - open/close 47833 (time 1734678927.47 total 687.98 last 139.75) - open/close 49131 (time 1734678937.48 total 697.99 last 129.68) total: 50000 open/close in 705.03 seconds: 70.92 ops/second - unlinked 0 (time 1734678946 ; total 0 ; last 0) - unlinked 10000 (time 1734679003 ; total 57 ; last 57) - unlinked 20000 (time 1734679046 ; total 100 ; last 43) - unlinked 30000 (time 1734679087 ; total 141 ; last 41) - unlinked 40000 (time 1734679122 ; total 176 ; last 35) total: 50000 unlinks in 212 seconds: 235.849060 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg326-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg326-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg326-server stop mds service on oleg326-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg326-server unloading modules via unload_modules_local on: 'oleg326-server' oleg326-server: oleg326-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg326-client: oleg326-client: ssh exited with exit code 2 pdsh@oleg326-client: oleg326-server: ssh exited with exit code 2 pdsh@oleg326-client: oleg326-server: ssh exited with exit code 2 pdsh@oleg326-client: oleg326-client: ssh exited with exit code 2