== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:14:43 (1717017283) start mds service on oleg338-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg338-server' oleg338-server: oleg338-server.virtnet: executing load_modules_local oleg338-server: Loading modules from /home/green/git/lustre-release/lustre oleg338-server: detected 4 online CPUs by sysfs oleg338-server: Force libcfs to create 2 CPU partitions oleg338-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg338-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg338-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg338-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg338-server: oleg338-server.virtnet: executing set_default_debug -1 all pdsh@oleg338-client: oleg338-server: ssh exited with exit code 1 Started lustre-OST0000 oleg338-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg338-client.virtnet: -o user_xattr,flock oleg338-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 160 (time 1717017319.07 total 10.09 last 15.86) - open/close 373 (time 1717017329.52 total 20.54 last 20.37) - open/close 538 (time 1717017339.80 total 30.82 last 16.05) - open/close 732 (time 1717017350.19 total 41.21 last 18.67) - open/close 999 (time 1717017362.08 total 53.10 last 22.45) - open/close 1227 (time 1717017372.78 total 63.80 last 21.31) - open/close 1484 (time 1717017382.79 total 73.80 last 25.69) - open/close 1674 (time 1717017393.32 total 84.34 last 18.03) - open/close 1881 (time 1717017403.93 total 94.95 last 19.51) - open/close 2160 (time 1717017414.17 total 105.19 last 27.26) - open/close 2384 (time 1717017424.79 total 115.81 last 21.09) - open/close 2615 (time 1717017435.59 total 126.61 last 21.39) - open/close 2963 (time 1717017446.43 total 137.45 last 32.09) - open/close 3132 (time 1717017456.92 total 147.94 last 16.11) - open/close 3234 (time 1717017467.77 total 158.78 last 9.41) - open/close 3490 (time 1717017478.40 total 169.42 last 24.06) - open/close 3828 (time 1717017489.37 total 180.39 last 30.82) - open/close 3977 (time 1717017499.38 total 190.40 last 14.89) - open/close 4138 (time 1717017509.67 total 200.69 last 15.64) - open/close 4468 (time 1717017520.48 total 211.50 last 30.52) - open/close 4687 (time 1717017530.48 total 221.50 last 21.90) - open/close 4974 (time 1717017541.08 total 232.10 last 27.09) - open/close 5281 (time 1717017551.98 total 242.99 last 28.17) - open/close 5492 (time 1717017562.58 total 253.59 last 19.91) - open/close 5649 (time 1717017572.75 total 263.77 last 15.44) - open/close 5833 (time 1717017583.37 total 274.39 last 17.31) - open/close 6056 (time 1717017594.16 total 285.18 last 20.68) - open/close 6262 (time 1717017604.86 total 295.88 last 19.25) - open/close 6429 (time 1717017615.18 total 306.19 last 16.19) - open/close 6565 (time 1717017625.19 total 316.20 last 13.59) - open/close 6783 (time 1717017635.20 total 326.21 last 21.78) - open/close 7007 (time 1717017645.57 total 336.59 last 21.59) - open/close 7338 (time 1717017655.92 total 346.94 last 32.00) - open/close 7692 (time 1717017666.88 total 357.90 last 32.29) - open/close 7996 (time 1717017677.08 total 368.10 last 29.80) - open/close 8203 (time 1717017687.41 total 378.43 last 20.04) - open/close 8499 (time 1717017697.41 total 388.43 last 29.60) - open/close 9862 (time 1717017707.42 total 398.44 last 136.24) - open/close 11287 (time 1717017717.42 total 408.44 last 142.49) - open/close 12595 (time 1717017727.42 total 418.44 last 130.72) - open/close 13799 (time 1717017737.43 total 428.45 last 120.37) - open/close 15110 (time 1717017747.43 total 438.45 last 131.02) - open/close 16486 (time 1717017757.44 total 448.46 last 137.52) - open/close 17836 (time 1717017767.44 total 458.46 last 134.97) - open/close 19182 (time 1717017777.44 total 468.46 last 134.55) - open/close 20000 (time 1717017783.88 total 474.90 last 127.01) - open/close 21233 (time 1717017793.89 total 484.91 last 123.22) - open/close 22701 (time 1717017803.89 total 494.91 last 146.80) - open/close 23946 (time 1717017813.89 total 504.91 last 124.49) - open/close 25348 (time 1717017823.89 total 514.91 last 140.16) - open/close 26753 (time 1717017833.90 total 524.92 last 140.40) - open/close 27980 (time 1717017843.91 total 534.93 last 122.61) - open/close 29339 (time 1717017853.91 total 544.93 last 135.83) - open/close 30000 (time 1717017858.96 total 549.98 last 131.06) - open/close 31234 (time 1717017868.96 total 559.98 last 123.37) - open/close 32413 (time 1717017878.96 total 569.98 last 117.87) - open/close 33578 (time 1717017888.97 total 579.99 last 116.45) - open/close 34786 (time 1717017898.97 total 589.99 last 120.77) - open/close 36018 (time 1717017908.97 total 599.99 last 123.18) - open/close 37182 (time 1717017918.98 total 609.99 last 116.35) - open/close 38419 (time 1717017928.98 total 620.00 last 123.70) - open/close 39587 (time 1717017938.98 total 630.00 last 116.78) - open/close 40000 (time 1717017942.07 total 633.09 last 133.53) - open/close 41342 (time 1717017952.07 total 643.09 last 134.17) - open/close 42700 (time 1717017962.08 total 653.10 last 135.72) - open/close 43986 (time 1717017972.08 total 663.10 last 128.54) - open/close 45571 (time 1717017982.09 total 673.11 last 158.45) - open/close 46981 (time 1717017992.09 total 683.11 last 140.96) - open/close 48297 (time 1717018002.09 total 693.11 last 131.56) - open/close 49469 (time 1717018012.10 total 703.12 last 117.10) total: 50000 open/close in 707.58 seconds: 70.66 ops/second - unlinked 0 (time 1717018018 ; total 0 ; last 0) - unlinked 10000 (time 1717018063 ; total 45 ; last 45) - unlinked 20000 (time 1717018097 ; total 79 ; last 34) - unlinked 30000 (time 1717018134 ; total 116 ; last 37) - unlinked 40000 (time 1717018164 ; total 146 ; last 30) total: 50000 unlinks in 181 seconds: 276.243103 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg338-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg338-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg338-server stop mds service on oleg338-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg338-server unloading modules on: 'oleg338-server' oleg338-server: oleg338-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg338-client: oleg338-client: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-client: ssh exited with exit code 2 pdsh@oleg338-client: oleg338-server: ssh exited with exit code 2