== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:37:07 (1726853827) start mds service on oleg146-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing load_modules_local oleg146-server: Loading modules from /home/green/git/lustre-release/lustre oleg146-server: detected 4 online CPUs by sysfs oleg146-server: Force libcfs to create 2 CPU partitions oleg146-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg146-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg146-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg146-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Started lustre-OST0000 oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg146-client.virtnet: -o user_xattr,flock 192.168.201.146@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 215 (time 1726853884.62 total 10.63 last 20.23) - open/close 495 (time 1726853895.33 total 21.34 last 26.14) - open/close 697 (time 1726853908.13 total 34.14 last 15.78) - open/close 893 (time 1726853918.14 total 44.14 last 19.59) - open/close 1113 (time 1726853928.14 total 54.15 last 21.99) - open/close 1332 (time 1726853938.15 total 64.15 last 21.89) - open/close 1476 (time 1726853949.36 total 75.37 last 12.84) - open/close 1747 (time 1726853959.67 total 85.68 last 26.29) - open/close 1937 (time 1726853969.68 total 95.68 last 18.99) - open/close 2132 (time 1726853979.98 total 105.98 last 18.94) - open/close 2405 (time 1726853989.98 total 115.98 last 27.29) - open/close 2643 (time 1726854000.22 total 126.23 last 23.24) - open/close 2868 (time 1726854010.23 total 136.23 last 22.48) - open/close 3088 (time 1726854021.23 total 147.24 last 19.99) - open/close 3309 (time 1726854031.40 total 157.41 last 21.73) - open/close 3507 (time 1726854041.41 total 167.42 last 19.78) - open/close 3686 (time 1726854051.44 total 177.44 last 17.85) - open/close 3933 (time 1726854061.92 total 187.92 last 23.57) - open/close 4138 (time 1726854071.92 total 197.92 last 20.50) - open/close 4311 (time 1726854082.00 total 208.00 last 17.17) - open/close 4461 (time 1726854092.05 total 218.05 last 14.92) - open/close 4641 (time 1726854102.05 total 228.06 last 17.99) - open/close 4884 (time 1726854112.53 total 238.54 last 23.19) - open/close 5197 (time 1726854122.92 total 248.92 last 30.13) - open/close 5374 (time 1726854133.39 total 259.40 last 16.90) - open/close 5584 (time 1726854145.79 total 271.79 last 16.94) - open/close 5867 (time 1726854156.80 total 282.81 last 25.69) - open/close 6060 (time 1726854167.44 total 293.44 last 18.14) - open/close 6348 (time 1726854178.44 total 304.44 last 26.19) - open/close 6701 (time 1726854189.29 total 315.29 last 32.53) - open/close 6997 (time 1726854199.68 total 325.69 last 28.48) - open/close 7150 (time 1726854209.83 total 335.84 last 15.07) - open/close 7257 (time 1726854220.67 total 346.67 last 9.87) - open/close 7475 (time 1726854231.59 total 357.60 last 19.96) - open/close 7609 (time 1726854242.18 total 368.19 last 12.66) - open/close 7856 (time 1726854252.19 total 378.19 last 24.68) - open/close 8107 (time 1726854262.19 total 388.20 last 25.09) - open/close 8311 (time 1726854272.86 total 398.87 last 19.12) - open/close 8509 (time 1726854282.87 total 408.87 last 19.80) - open/close 8807 (time 1726854292.87 total 418.87 last 29.79) - open/close 10000 (time 1726854300.25 total 426.25 last 161.66) - open/close 11675 (time 1726854310.25 total 436.26 last 167.47) - open/close 13200 (time 1726854320.26 total 446.26 last 152.43) - open/close 14824 (time 1726854330.26 total 456.26 last 162.40) - open/close 16269 (time 1726854340.26 total 466.27 last 144.41) - open/close 17630 (time 1726854350.27 total 476.27 last 136.06) - open/close 19347 (time 1726854360.27 total 486.27 last 171.62) - open/close 20000 (time 1726854363.72 total 489.72 last 189.32) - open/close 21450 (time 1726854373.73 total 499.73 last 144.91) - open/close 22885 (time 1726854383.73 total 509.73 last 143.46) - open/close 24594 (time 1726854393.73 total 519.74 last 170.84) - open/close 26297 (time 1726854403.74 total 529.74 last 170.21) - open/close 27679 (time 1726854413.74 total 539.75 last 138.11) - open/close 29063 (time 1726854423.74 total 549.75 last 138.38) - open/close 30000 (time 1726854431.15 total 557.16 last 126.47) - open/close 31527 (time 1726854441.16 total 567.16 last 152.64) - open/close 33105 (time 1726854451.16 total 577.16 last 157.75) - open/close 34462 (time 1726854461.17 total 587.17 last 135.57) - open/close 35891 (time 1726854471.17 total 597.18 last 142.86) - open/close 37383 (time 1726854481.18 total 607.18 last 149.08) - open/close 38694 (time 1726854491.18 total 617.19 last 131.04) - open/close 39982 (time 1726854501.19 total 627.20 last 128.70) - open/close 41364 (time 1726854511.20 total 637.20 last 138.12) - open/close 42838 (time 1726854521.20 total 647.20 last 147.38) - open/close 44092 (time 1726854531.20 total 657.21 last 125.34) - open/close 45455 (time 1726854541.21 total 667.21 last 136.23) - open/close 46892 (time 1726854551.21 total 677.22 last 143.63) - open/close 48105 (time 1726854561.22 total 687.22 last 121.26) - open/close 49513 (time 1726854571.22 total 697.22 last 140.78) total: 50000 open/close in 700.07 seconds: 71.42 ops/second - unlinked 0 (time 1726854575 ; total 0 ; last 0) - unlinked 10000 (time 1726854622 ; total 47 ; last 47) - unlinked 20000 (time 1726854658 ; total 83 ; last 36) - unlinked 30000 (time 1726854695 ; total 120 ; last 37) - unlinked 40000 (time 1726854727 ; total 152 ; last 32) total: 50000 unlinks in 181 seconds: 276.243103 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg146-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg146-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg146-server unloading modules on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg146-client: oleg146-client: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-server: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-server: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-client: ssh exited with exit code 2