== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:09:32 (1782533372) start mds service on oleg117-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing load_modules_local oleg117-server: Loading modules from /home/green/git/lustre-release/lustre oleg117-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg117-server: detected 4 online CPUs by sysfs oleg117-server: MODOPTS_LIBCFS= oleg117-server: Force libcfs to create 2 CPU partitions oleg117-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg117-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg117-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg117-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg117-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg117-server: oleg117-server.virtnet: executing set_default_debug -1 all pdsh@oleg117-client: oleg117-server: ssh exited with exit code 1 Started lustre-OST0000 oleg117-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg117-client.virtnet: -o user_xattr,flock 192.168.201.117@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 310 (time 1782533413.07 total 10.69 last 29.00) - open/close 511 (time 1782533424.00 total 21.62 last 18.39) - open/close 706 (time 1782533434.05 total 31.67 last 19.40) - open/close 902 (time 1782533444.06 total 41.68 last 19.59) - open/close 1111 (time 1782533454.08 total 51.70 last 20.86) - open/close 1311 (time 1782533464.38 total 62.01 last 19.41) - open/close 1839 (time 1782533475.13 total 72.76 last 49.11) - open/close 1967 (time 1782533485.14 total 82.76 last 12.80) - open/close 2164 (time 1782533495.14 total 92.76 last 19.70) - open/close 2343 (time 1782533505.79 total 103.42 last 16.80) - open/close 2620 (time 1782533516.16 total 113.78 last 26.73) - open/close 2851 (time 1782533526.34 total 123.96 last 22.70) - open/close 3196 (time 1782533536.34 total 133.96 last 34.49) - open/close 3540 (time 1782533546.88 total 144.50 last 32.64) - open/close 3644 (time 1782533556.93 total 154.55 last 10.35) - open/close 3869 (time 1782533567.62 total 165.25 last 21.04) - open/close 4032 (time 1782533578.05 total 175.68 last 15.63) - open/close 4230 (time 1782533588.73 total 186.36 last 18.54) - open/close 4341 (time 1782533598.74 total 196.36 last 11.10) - open/close 4610 (time 1782533609.73 total 207.36 last 24.46) - open/close 5010 (time 1782533619.97 total 217.59 last 39.09) - open/close 5368 (time 1782533630.01 total 227.64 last 35.63) - open/close 5599 (time 1782533642.82 total 240.44 last 18.05) - open/close 5711 (time 1782533653.82 total 251.45 last 10.18) - open/close 6105 (time 1782533664.25 total 261.88 last 37.77) - open/close 6415 (time 1782533674.88 total 272.50 last 29.18) - open/close 6822 (time 1782533685.05 total 282.68 last 40.00) - open/close 7044 (time 1782533695.49 total 293.11 last 21.27) - open/close 7260 (time 1782533706.05 total 303.67 last 20.45) - open/close 7405 (time 1782533716.35 total 313.98 last 14.07) - open/close 8066 (time 1782533726.78 total 324.41 last 63.39) - open/close 8432 (time 1782533737.47 total 335.09 last 34.25) - open/close 8881 (time 1782533747.47 total 345.10 last 44.88) - open/close 9365 (time 1782533758.27 total 355.89 last 44.84) - open/close 9975 (time 1782533768.77 total 366.39 last 58.10) - open/close 10523 (time 1782533778.81 total 376.44 last 54.53) - open/close 13964 (time 1782533788.81 total 386.44 last 344.09) - open/close 18099 (time 1782533798.82 total 396.44 last 413.47) - open/close 20000 (time 1782533803.86 total 401.48 last 376.84) - open/close 23775 (time 1782533813.86 total 411.48 last 377.49) - open/close 27647 (time 1782533823.86 total 421.49 last 387.06) - open/close 30000 (time 1782533831.03 total 428.65 last 328.32) - open/close 33380 (time 1782533841.03 total 438.66 last 337.96) - open/close 37009 (time 1782533851.04 total 448.66 last 362.78) - open/close 40000 (time 1782533860.86 total 458.49 last 304.33) - open/close 42372 (time 1782533870.86 total 468.49 last 237.17) - open/close 44783 (time 1782533880.87 total 478.49 last 241.05) - open/close 46943 (time 1782533890.87 total 488.49 last 215.95) - open/close 49525 (time 1782533900.87 total 498.50 last 258.13) total: 50000 open/close in 500.36 seconds: 99.93 ops/second - unlinked 0 (time 1782533904 ; total 0 ; last 0) - unlinked 10000 (time 1782533927 ; total 23 ; last 23) - unlinked 20000 (time 1782533948 ; total 44 ; last 21) - unlinked 30000 (time 1782533975 ; total 71 ; last 27) - unlinked 40000 (time 1782533997 ; total 93 ; last 22) total: 50000 unlinks in 113 seconds: 442.477875 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg117-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg117-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg117-server stop mds service on oleg117-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg117-server unloading modules via unload_modules_local on: 'oleg117-server' oleg117-server: oleg117-server.virtnet: executing unload_modules_local oleg117-server: modules unloaded. pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-client: ssh exited with exit code 2 pdsh@oleg117-client: oleg117-server: ssh exited with exit code 2