== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:16:37 (1688472997) start mds service on oleg272-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg272-server' oleg272-server: oleg272-server.virtnet: executing load_modules_local oleg272-server: Loading modules from /home/green/git/lustre-release/lustre oleg272-server: detected 4 online CPUs by sysfs oleg272-server: Force libcfs to create 2 CPU partitions oleg272-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg272-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg272-server: oleg272-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg272-client: oleg272-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg272-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg272-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg272-server: oleg272-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg272-client: oleg272-server: ssh exited with exit code 1 Started lustre-OST0000 oleg272-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg272-client.virtnet: -o user_xattr,flock oleg272-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 262 (time 1688473030.91 total 10.21 last 25.65) - open/close 551 (time 1688473040.95 total 20.26 last 28.77) - open/close 708 (time 1688473051.63 total 30.94 last 14.70) - open/close 955 (time 1688473061.64 total 40.94 last 24.69) - open/close 1176 (time 1688473071.64 total 50.95 last 22.09) - open/close 1589 (time 1688473082.37 total 61.68 last 38.49) - open/close 1806 (time 1688473093.37 total 72.67 last 19.73) - open/close 2026 (time 1688473103.37 total 82.68 last 21.99) - open/close 2379 (time 1688473113.82 total 93.12 last 33.80) - open/close 2691 (time 1688473124.19 total 103.50 last 30.07) - open/close 3028 (time 1688473136.74 total 116.04 last 26.87) - open/close 3302 (time 1688473146.82 total 126.12 last 27.18) - open/close 3704 (time 1688473157.39 total 136.69 last 38.03) - open/close 4033 (time 1688473167.48 total 146.78 last 32.61) - open/close 4201 (time 1688473178.22 total 157.52 last 15.65) - open/close 4350 (time 1688473188.91 total 168.21 last 13.94) - open/close 4708 (time 1688473199.34 total 178.64 last 34.32) - open/close 5051 (time 1688473209.95 total 189.26 last 32.32) - open/close 5308 (time 1688473220.08 total 199.38 last 25.39) - open/close 5519 (time 1688473231.08 total 210.38 last 19.18) - open/close 5752 (time 1688473241.15 total 220.45 last 23.14) - open/close 5987 (time 1688473251.23 total 230.53 last 23.32) - open/close 6282 (time 1688473261.66 total 240.97 last 28.27) - open/close 6529 (time 1688473271.82 total 251.12 last 24.32) - open/close 6636 (time 1688473282.35 total 261.65 last 10.16) - open/close 6751 (time 1688473292.93 total 272.23 last 10.87) - open/close 6897 (time 1688473303.62 total 282.93 last 13.65) - open/close 7125 (time 1688473313.69 total 293.00 last 22.64) - open/close 7339 (time 1688473323.72 total 303.02 last 21.35) - open/close 7714 (time 1688473334.42 total 313.72 last 35.04) - open/close 8037 (time 1688473344.73 total 324.03 last 31.34) - open/close 9184 (time 1688473354.73 total 334.03 last 114.66) - open/close 10000 (time 1688473358.64 total 337.95 last 208.49) - open/close 12139 (time 1688473368.65 total 347.95 last 213.80) - open/close 14248 (time 1688473378.65 total 357.95 last 210.88) - open/close 16332 (time 1688473388.65 total 367.96 last 208.31) - open/close 18441 (time 1688473398.66 total 377.96 last 210.82) - open/close 20000 (time 1688473406.06 total 385.36 last 210.66) - open/close 22082 (time 1688473416.06 total 395.36 last 208.16) - open/close 24135 (time 1688473426.06 total 405.37 last 205.26) - open/close 26260 (time 1688473436.07 total 415.37 last 212.44) - open/close 28562 (time 1688473446.07 total 425.37 last 230.19) - open/close 30000 (time 1688473452.45 total 431.75 last 225.40) - open/close 32348 (time 1688473462.45 total 441.75 last 234.75) - open/close 34417 (time 1688473472.45 total 451.75 last 206.90) - open/close 36561 (time 1688473482.45 total 461.75 last 214.36) - open/close 38752 (time 1688473492.45 total 471.75 last 219.09) - open/close 40000 (time 1688473498.37 total 477.67 last 210.81) - open/close 42471 (time 1688473508.37 total 487.68 last 247.07) - open/close 45168 (time 1688473518.37 total 497.68 last 269.62) - open/close 47323 (time 1688473528.38 total 507.68 last 215.40) - open/close 49618 (time 1688473538.38 total 517.68 last 229.48) total: 50000 open/close in 519.45 seconds: 96.26 ops/second - unlinked 0 (time 1688473541 ; total 0 ; last 0) - unlinked 10000 (time 1688473565 ; total 24 ; last 24) - unlinked 20000 (time 1688473587 ; total 46 ; last 22) - unlinked 30000 (time 1688473609 ; total 68 ; last 22) - unlinked 40000 (time 1688473632 ; total 91 ; last 23) total: 50000 unlinks in 115 seconds: 434.782623 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg272-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg272-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg272-server stop mds service on oleg272-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg272-server unloading modules on: 'oleg272-server' oleg272-server: oleg272-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg272-client: oleg272-server: ssh exited with exit code 2 pdsh@oleg272-client: oleg272-client: ssh exited with exit code 2 pdsh@oleg272-client: oleg272-client: ssh exited with exit code 2 pdsh@oleg272-client: oleg272-server: ssh exited with exit code 2