== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:08:08 (1772759288) start mds service on oleg116-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg116-server: detected 4 online CPUs by sysfs oleg116-server: MODOPTS_LIBCFS= oleg116-server: Force libcfs to create 2 CPU partitions oleg116-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg116-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg116-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 278 (time 1772759329.41 total 10.49 last 26.49) - open/close 434 (time 1772759339.41 total 20.50 last 15.60) - open/close 741 (time 1772759351.94 total 33.02 last 24.51) - open/close 985 (time 1772759362.11 total 43.20 last 23.98) - open/close 1239 (time 1772759372.35 total 53.44 last 24.80) - open/close 1435 (time 1772759382.40 total 63.48 last 19.51) - open/close 1702 (time 1772759392.77 total 73.86 last 25.74) - open/close 2041 (time 1772759403.33 total 84.41 last 32.11) - open/close 2333 (time 1772759413.63 total 94.72 last 28.34) - open/close 2536 (time 1772759423.74 total 104.83 last 20.08) - open/close 2889 (time 1772759434.11 total 115.20 last 34.05) - open/close 3394 (time 1772759444.99 total 126.08 last 46.41) - open/close 3794 (time 1772759455.55 total 136.64 last 37.88) - open/close 4285 (time 1772759466.50 total 147.58 last 44.86) - open/close 4641 (time 1772759477.14 total 158.22 last 33.46) - open/close 4900 (time 1772759487.49 total 168.57 last 25.01) - open/close 5192 (time 1772759498.18 total 179.26 last 27.33) - open/close 5419 (time 1772759509.25 total 190.34 last 20.49) - open/close 5722 (time 1772759519.75 total 200.84 last 28.86) - open/close 5927 (time 1772759529.75 total 210.84 last 20.50) - open/close 6324 (time 1772759540.42 total 221.50 last 37.23) - open/close 6700 (time 1772759551.11 total 232.19 last 35.17) - open/close 7113 (time 1772759561.35 total 242.43 last 40.33) - open/close 7364 (time 1772759572.03 total 253.12 last 23.49) - open/close 7557 (time 1772759582.03 total 263.12 last 19.30) - open/close 7887 (time 1772759592.45 total 273.53 last 31.68) - open/close 8330 (time 1772759603.27 total 284.35 last 40.96) - open/close 8790 (time 1772759613.27 total 294.35 last 45.99) - open/close 9141 (time 1772759623.56 total 304.64 last 34.10) - open/close 9919 (time 1772759633.93 total 315.01 last 75.05) - open/close 10273 (time 1772759644.29 total 325.38 last 34.15) - open/close 10931 (time 1772759654.34 total 335.42 last 65.51) - open/close 11614 (time 1772759664.34 total 345.42 last 68.29) - open/close 12396 (time 1772759674.63 total 355.71 last 75.99) - open/close 13081 (time 1772759684.63 total 365.71 last 68.50) - open/close 17128 (time 1772759694.63 total 375.72 last 404.61) - open/close 20000 (time 1772759703.07 total 384.15 last 340.43) - open/close 24624 (time 1772759713.07 total 394.15 last 462.38) - open/close 29356 (time 1772759723.07 total 404.15 last 473.11) - open/close 33749 (time 1772759733.07 total 414.16 last 439.23) - open/close 37079 (time 1772759743.07 total 424.16 last 332.99) - open/close 40000 (time 1772759752.65 total 433.73 last 305.08) - open/close 43045 (time 1772759762.65 total 443.73 last 304.44) - open/close 46262 (time 1772759772.65 total 453.74 last 321.60) - open/close 49545 (time 1772759782.65 total 463.74 last 328.25) total: 50000 open/close in 465.00 seconds: 107.53 ops/second - unlinked 0 (time 1772759785 ; total 0 ; last 0) - unlinked 10000 (time 1772759804 ; total 19 ; last 19) - unlinked 20000 (time 1772759821 ; total 36 ; last 17) - unlinked 30000 (time 1772759837 ; total 52 ; last 16) - unlinked 40000 (time 1772759854 ; total 69 ; last 17) total: 50000 unlinks in 83 seconds: 602.409668 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server unloading modules via unload_modules_local on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing unload_modules_local oleg116-server: modules unloaded. pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2