== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:40:12 (1783143612) start mds service on oleg144-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing load_modules_local oleg144-server: Loading modules from /home/green/git/lustre-release/lustre oleg144-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg144-server: detected 4 online CPUs by sysfs oleg144-server: MODOPTS_LIBCFS= oleg144-server: Force libcfs to create 2 CPU partitions oleg144-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg144-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg144-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg144-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg144-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg144-server: oleg144-server.virtnet: executing set_default_debug -1 all pdsh@oleg144-client: oleg144-server: ssh exited with exit code 1 Started lustre-OST0000 oleg144-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg144-client.virtnet: -o user_xattr,flock 192.168.201.144@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 246 (time 1783143661.87 total 10.12 last 24.30) - open/close 504 (time 1783143672.37 total 20.62 last 24.57) - open/close 861 (time 1783143682.38 total 30.63 last 35.67) - open/close 1141 (time 1783143693.11 total 41.36 last 26.09) - open/close 1414 (time 1783143703.12 total 51.37 last 27.28) - open/close 1740 (time 1783143713.27 total 61.52 last 32.12) - open/close 2067 (time 1783143723.88 total 72.13 last 30.81) - open/close 2347 (time 1783143734.38 total 82.63 last 26.67) - open/close 2569 (time 1783143744.62 total 92.87 last 21.68) - open/close 2882 (time 1783143755.18 total 103.43 last 29.64) - open/close 3082 (time 1783143766.19 total 114.44 last 18.17) - open/close 3331 (time 1783143776.19 total 124.44 last 24.90) - open/close 3550 (time 1783143787.19 total 135.44 last 19.91) - open/close 3819 (time 1783143799.47 total 147.72 last 21.90) - open/close 4082 (time 1783143809.52 total 157.77 last 26.17) - open/close 4334 (time 1783143819.52 total 167.78 last 25.19) - open/close 4573 (time 1783143830.00 total 178.25 last 22.82) - open/close 4757 (time 1783143840.11 total 188.36 last 18.20) - open/close 5026 (time 1783143850.16 total 198.41 last 26.77) - open/close 5272 (time 1783143860.85 total 209.10 last 23.01) - open/close 5460 (time 1783143871.21 total 219.46 last 18.13) - open/close 5709 (time 1783143882.10 total 230.35 last 22.87) - open/close 5933 (time 1783143892.71 total 240.97 last 21.11) - open/close 6133 (time 1783143903.08 total 251.34 last 19.28) - open/close 6724 (time 1783143913.52 total 261.77 last 56.65) - open/close 7095 (time 1783143924.39 total 272.65 last 34.11) - open/close 7481 (time 1783143934.51 total 282.76 last 38.16) - open/close 8098 (time 1783143944.62 total 292.87 last 61.02) - open/close 8629 (time 1783143954.63 total 302.88 last 53.07) - open/close 8986 (time 1783143965.30 total 313.55 last 33.46) - open/close 9520 (time 1783143975.41 total 323.66 last 52.81) - open/close 9917 (time 1783143985.41 total 333.66 last 39.68) - open/close 10318 (time 1783143995.42 total 343.67 last 40.09) - open/close 10741 (time 1783144005.42 total 353.67 last 42.29) - open/close 11062 (time 1783144016.30 total 364.55 last 29.50) - open/close 11483 (time 1783144026.30 total 374.55 last 42.09) - open/close 11935 (time 1783144036.40 total 384.65 last 44.78) - open/close 13563 (time 1783144046.40 total 394.65 last 162.78) - open/close 15650 (time 1783144056.40 total 404.65 last 208.63) - open/close 17759 (time 1783144066.40 total 414.65 last 210.84) - open/close 20000 (time 1783144075.73 total 423.98 last 240.28) - open/close 22276 (time 1783144085.73 total 433.98 last 227.54) - open/close 24713 (time 1783144095.73 total 443.99 last 243.62) - open/close 27196 (time 1783144105.74 total 453.99 last 248.29) - open/close 30000 (time 1783144115.33 total 463.58 last 292.14) - open/close 33876 (time 1783144125.34 total 473.59 last 387.51) - open/close 36411 (time 1783144135.34 total 483.59 last 253.47) - open/close 39866 (time 1783144145.34 total 493.59 last 345.45) - open/close 43023 (time 1783144155.34 total 503.60 last 315.50) - open/close 46130 (time 1783144165.35 total 513.60 last 310.64) - open/close 49490 (time 1783144175.35 total 523.60 last 335.94) total: 50000 open/close in 524.94 seconds: 95.25 ops/second - unlinked 0 (time 1783144178 ; total 0 ; last 0) - unlinked 10000 (time 1783144195 ; total 17 ; last 17) - unlinked 20000 (time 1783144216 ; total 38 ; last 21) - unlinked 30000 (time 1783144238 ; total 60 ; last 22) - unlinked 40000 (time 1783144258 ; total 80 ; last 20) total: 50000 unlinks in 97 seconds: 515.463928 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg144-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg144-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg144-server stop mds service on oleg144-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg144-server unloading modules via unload_modules_local on: 'oleg144-server' oleg144-server: oleg144-server.virtnet: executing unload_modules_local oleg144-server: modules unloaded. pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-client: ssh exited with exit code 2 pdsh@oleg144-client: oleg144-server: ssh exited with exit code 2