== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:13:26 (1786115606) start mds service on oleg340-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing load_modules_local oleg340-server: Loading modules from /home/green/git/lustre-release/lustre oleg340-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg340-server: detected 4 online CPUs by sysfs oleg340-server: MODOPTS_LIBCFS= oleg340-server: Force libcfs to create 2 CPU partitions oleg340-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg340-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg340-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg340-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg340-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg340-server: oleg340-server.virtnet: executing set_default_debug -1 all pdsh@oleg340-client: oleg340-server: ssh exited with exit code 1 Started lustre-OST0000 oleg340-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg340-client.virtnet: -o user_xattr,flock 192.168.203.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 245 (time 1786115650.31 total 10.83 last 22.62) - open/close 379 (time 1786115660.48 total 21.01 last 13.17) - open/close 716 (time 1786115670.91 total 31.44 last 32.29) - open/close 987 (time 1786115681.86 total 42.39 last 24.76) - open/close 1247 (time 1786115691.86 total 52.39 last 25.99) - open/close 1439 (time 1786115702.34 total 62.86 last 18.33) - open/close 1675 (time 1786115712.90 total 73.43 last 22.34) - open/close 1877 (time 1786115723.39 total 83.92 last 19.25) - open/close 2172 (time 1786115733.70 total 94.22 last 28.63) - open/close 2447 (time 1786115743.70 total 104.23 last 27.50) - open/close 2694 (time 1786115753.73 total 114.26 last 24.61) - open/close 2914 (time 1786115764.23 total 124.75 last 20.97) - open/close 3212 (time 1786115774.72 total 135.25 last 28.38) - open/close 3509 (time 1786115785.41 total 145.94 last 27.80) - open/close 3715 (time 1786115795.59 total 156.11 last 20.24) - open/close 4347 (time 1786115806.01 total 166.54 last 60.60) - open/close 4762 (time 1786115817.02 total 177.55 last 37.70) - open/close 5328 (time 1786115827.46 total 187.98 last 54.25) - open/close 5702 (time 1786115838.46 total 198.99 last 33.98) - open/close 6009 (time 1786115848.90 total 209.43 last 29.42) - open/close 6143 (time 1786115861.95 total 222.48 last 10.26) - open/close 6646 (time 1786115872.90 total 233.42 last 45.97) - open/close 7300 (time 1786115883.46 total 243.99 last 61.91) - open/close 7932 (time 1786115893.89 total 254.42 last 60.60) - open/close 8545 (time 1786115904.13 total 264.66 last 59.86) - open/close 9223 (time 1786115914.82 total 275.34 last 63.44) - open/close 9501 (time 1786115925.38 total 285.90 last 26.33) - open/close 9916 (time 1786115936.13 total 296.65 last 38.60) - open/close 10553 (time 1786115946.37 total 306.90 last 62.17) - open/close 10826 (time 1786115956.81 total 317.34 last 26.16) - open/close 11505 (time 1786115967.43 total 327.96 last 63.92) - open/close 12010 (time 1786115977.54 total 338.06 last 49.96) - open/close 12435 (time 1786115988.48 total 349.01 last 38.84) - open/close 12898 (time 1786115998.48 total 359.01 last 46.28) - open/close 16434 (time 1786116008.49 total 369.01 last 353.55) - open/close 19714 (time 1786116018.49 total 379.01 last 327.95) - open/close 22940 (time 1786116028.49 total 389.02 last 322.52) - open/close 25447 (time 1786116038.49 total 399.02 last 250.67) - open/close 28117 (time 1786116048.49 total 409.02 last 266.96) - open/close 30000 (time 1786116057.56 total 418.09 last 207.56) - open/close 32320 (time 1786116067.57 total 428.09 last 231.96) - open/close 35642 (time 1786116077.57 total 438.09 last 332.19) - open/close 38670 (time 1786116087.57 total 448.10 last 302.73) - open/close 40000 (time 1786116092.38 total 452.91 last 276.28) - open/close 42533 (time 1786116102.39 total 462.91 last 253.17) - open/close 45537 (time 1786116112.39 total 472.92 last 300.36) - open/close 49284 (time 1786116122.39 total 482.92 last 374.64) total: 50000 open/close in 484.60 seconds: 103.18 ops/second - unlinked 0 (time 1786116125 ; total 0 ; last 0) - unlinked 10000 (time 1786116148 ; total 23 ; last 23) - unlinked 20000 (time 1786116170 ; total 45 ; last 22) - unlinked 30000 (time 1786116191 ; total 66 ; last 21) - unlinked 40000 (time 1786116208 ; total 83 ; last 17) total: 50000 unlinks in 99 seconds: 505.050507 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg340-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg340-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg340-server stop mds service on oleg340-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg340-server unloading modules via unload_modules_local on: 'oleg340-server' oleg340-server: oleg340-server.virtnet: executing unload_modules_local oleg340-server: modules unloaded. pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-client: ssh exited with exit code 2 pdsh@oleg340-client: oleg340-server: ssh exited with exit code 2