== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:56:34 (1773687394) start mds service on oleg160-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing load_modules_local oleg160-server: Loading modules from /home/green/git/lustre-release/lustre oleg160-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg160-server: detected 4 online CPUs by sysfs oleg160-server: MODOPTS_LIBCFS= oleg160-server: Force libcfs to create 2 CPU partitions oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg160-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 422 (time 1773687440.60 total 10.95 last 38.55) - open/close 834 (time 1773687451.03 total 21.38 last 39.50) - open/close 1304 (time 1773687461.85 total 32.19 last 43.45) - open/close 1675 (time 1773687471.85 total 42.20 last 37.09) - open/close 2006 (time 1773687482.20 total 52.55 last 31.97) - open/close 2437 (time 1773687492.76 total 63.11 last 40.83) - open/close 2786 (time 1773687502.76 total 73.11 last 34.90) - open/close 3116 (time 1773687512.92 total 83.26 last 32.49) - open/close 3539 (time 1773687523.86 total 94.21 last 38.66) - open/close 3938 (time 1773687534.24 total 104.58 last 38.45) - open/close 4356 (time 1773687544.98 total 115.33 last 38.92) - open/close 4854 (time 1773687555.67 total 126.02 last 46.58) - open/close 5290 (time 1773687566.55 total 136.89 last 40.08) - open/close 5702 (time 1773687577.17 total 147.52 last 38.78) - open/close 6057 (time 1773687587.36 total 157.70 last 34.86) - open/close 6387 (time 1773687597.36 total 167.71 last 32.99) - open/close 6723 (time 1773687608.02 total 178.37 last 31.51) - open/close 7033 (time 1773687618.33 total 188.67 last 30.08) - open/close 7412 (time 1773687628.71 total 199.06 last 36.50) - open/close 7721 (time 1773687638.71 total 209.06 last 30.90) - open/close 8081 (time 1773687648.71 total 219.06 last 36.00) - open/close 8424 (time 1773687658.84 total 229.18 last 33.88) - open/close 8748 (time 1773687668.84 total 239.19 last 32.38) - open/close 9110 (time 1773687679.45 total 249.79 last 34.13) - open/close 9526 (time 1773687689.88 total 260.22 last 39.90) - open/close 9888 (time 1773687699.88 total 270.22 last 36.19) - open/close 10000 (time 1773687703.57 total 273.92 last 30.29) - open/close 10386 (time 1773687714.00 total 284.35 last 37.01) - open/close 10739 (time 1773687724.89 total 295.23 last 32.44) - open/close 11126 (time 1773687735.89 total 306.24 last 35.16) - open/close 11540 (time 1773687746.14 total 316.49 last 40.41) - open/close 11933 (time 1773687756.50 total 326.85 last 37.93) - open/close 12232 (time 1773687767.19 total 337.54 last 27.96) - open/close 12605 (time 1773687778.20 total 348.55 last 33.88) - open/close 12939 (time 1773687788.96 total 359.31 last 31.05) - open/close 13234 (time 1773687799.64 total 369.98 last 27.63) - open/close 13548 (time 1773687810.07 total 380.41 last 30.10) - open/close 14176 (time 1773687820.18 total 390.53 last 62.08) - open/close 14449 (time 1773687830.55 total 400.90 last 26.33) - open/close 14659 (time 1773687840.60 total 410.94 last 20.91) - open/close 15480 (time 1773687850.60 total 420.94 last 82.10) - open/close 18995 (time 1773687860.60 total 430.94 last 351.44) - open/close 20000 (time 1773687863.75 total 434.09 last 319.01) - open/close 23047 (time 1773687873.75 total 444.10 last 304.67) - open/close 25955 (time 1773687883.75 total 454.10 last 290.77) - open/close 28588 (time 1773687893.75 total 464.10 last 263.27) - open/close 30000 (time 1773687900.26 total 470.61 last 216.86) - open/close 32612 (time 1773687910.27 total 480.61 last 261.14) - open/close 35373 (time 1773687920.27 total 490.61 last 276.07) - open/close 37476 (time 1773687930.27 total 500.61 last 210.27) - open/close 39562 (time 1773687940.27 total 510.61 last 208.59) - open/close 42313 (time 1773687950.27 total 520.61 last 275.09) - open/close 45202 (time 1773687960.27 total 530.62 last 288.79) - open/close 48555 (time 1773687970.27 total 540.62 last 335.29) total: 50000 open/close in 544.93 seconds: 91.76 ops/second - unlinked 0 (time 1773687976 ; total 0 ; last 0) - unlinked 10000 (time 1773687997 ; total 21 ; last 21) - unlinked 20000 (time 1773688016 ; total 40 ; last 19) - unlinked 30000 (time 1773688034 ; total 58 ; last 18) - unlinked 40000 (time 1773688054 ; total 78 ; last 20) total: 50000 unlinks in 98 seconds: 510.204071 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server unloading modules via unload_modules_local on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing unload_modules_local oleg160-server: modules unloaded. pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2