== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:05:08 (1777424708) start mds service on oleg118-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg118-server: detected 4 online CPUs by sysfs oleg118-server: MODOPTS_LIBCFS= oleg118-server: Force libcfs to create 2 CPU partitions oleg118-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg118-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg118-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg118-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 648 (time 1777424749.20 total 10.12 last 64.05) - open/close 1257 (time 1777424759.25 total 20.16 last 60.61) - open/close 1836 (time 1777424769.25 total 30.17 last 57.90) - open/close 2417 (time 1777424779.25 total 40.17 last 58.08) - open/close 2975 (time 1777424790.09 total 51.01 last 51.45) - open/close 3623 (time 1777424800.14 total 61.06 last 64.49) - open/close 4204 (time 1777424810.89 total 71.81 last 54.04) - open/close 4742 (time 1777424821.52 total 82.44 last 50.63) - open/close 5375 (time 1777424832.21 total 93.12 last 59.23) - open/close 6039 (time 1777424842.21 total 103.13 last 66.40) - open/close 6554 (time 1777424852.81 total 113.73 last 48.55) - open/close 6855 (time 1777424862.81 total 123.73 last 30.10) - open/close 7329 (time 1777424873.17 total 134.09 last 45.79) - open/close 7983 (time 1777424883.17 total 144.09 last 65.40) - open/close 8458 (time 1777424893.77 total 154.69 last 44.78) - open/close 8962 (time 1777424904.35 total 165.27 last 47.65) - open/close 9452 (time 1777424914.70 total 175.62 last 47.34) - open/close 9950 (time 1777424925.33 total 186.25 last 46.86) - open/close 10485 (time 1777424935.33 total 196.25 last 53.49) - open/close 11133 (time 1777424945.74 total 206.66 last 62.24) - open/close 11765 (time 1777424955.74 total 216.66 last 63.19) - open/close 12383 (time 1777424966.67 total 227.59 last 56.56) - open/close 13036 (time 1777424977.68 total 238.60 last 59.30) - open/close 13545 (time 1777424988.24 total 249.16 last 48.23) - open/close 14201 (time 1777424998.35 total 259.27 last 64.87) - open/close 14742 (time 1777425008.97 total 269.89 last 50.92) - open/close 15331 (time 1777425018.97 total 279.89 last 58.90) - open/close 15835 (time 1777425029.52 total 290.44 last 47.78) - open/close 16427 (time 1777425040.21 total 301.13 last 55.41) - open/close 19685 (time 1777425050.21 total 311.13 last 325.74) - open/close 23372 (time 1777425060.21 total 321.13 last 368.68) - open/close 27152 (time 1777425070.21 total 331.13 last 377.93) - open/close 30000 (time 1777425077.97 total 338.89 last 366.99) - open/close 34171 (time 1777425087.97 total 348.89 last 417.04) - open/close 38102 (time 1777425097.98 total 358.90 last 392.95) - open/close 40000 (time 1777425102.79 total 363.71 last 394.52) - open/close 43491 (time 1777425112.79 total 373.71 last 349.08) - open/close 47084 (time 1777425122.79 total 383.71 last 359.29) total: 50000 open/close in 392.68 seconds: 127.33 ops/second - unlinked 0 (time 1777425133 ; total 0 ; last 0) - unlinked 10000 (time 1777425150 ; total 17 ; last 17) - unlinked 20000 (time 1777425166 ; total 33 ; last 16) - unlinked 30000 (time 1777425181 ; total 48 ; last 15) - unlinked 40000 (time 1777425197 ; total 64 ; last 16) total: 50000 unlinks in 79 seconds: 632.911377 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server unloading modules via unload_modules_local on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing unload_modules_local oleg118-server: modules unloaded. pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2