== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 01:13:32 (1780809212) start mds service on oleg240-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg240-server' oleg240-server: oleg240-server.virtnet: executing load_modules_local oleg240-server: Loading modules from /home/green/git/lustre-release/lustre oleg240-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg240-server: detected 4 online CPUs by sysfs oleg240-server: MODOPTS_LIBCFS= oleg240-server: Force libcfs to create 2 CPU partitions oleg240-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg240-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg240-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg240-server: oleg240-server.virtnet: executing set_default_debug -1 all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg240-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg240-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg240-server: oleg240-server.virtnet: executing set_default_debug -1 all pdsh@oleg240-client: oleg240-server: ssh exited with exit code 1 Started lustre-OST0000 oleg240-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg240-client.virtnet: -o user_xattr,flock 192.168.202.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 591 (time 1780809252.33 total 10.01 last 59.05) - open/close 1096 (time 1780809263.07 total 20.76 last 46.98) - open/close 1482 (time 1780809273.12 total 30.80 last 38.42) - open/close 2020 (time 1780809283.68 total 41.37 last 50.94) - open/close 2580 (time 1780809294.24 total 51.92 last 53.04) - open/close 3162 (time 1780809305.00 total 62.68 last 54.12) - open/close 3735 (time 1780809315.68 total 73.37 last 53.61) - open/close 4348 (time 1780809326.50 total 84.18 last 56.68) - open/close 4977 (time 1780809337.25 total 94.93 last 58.49) - open/close 5478 (time 1780809347.62 total 105.30 last 48.33) - open/close 6132 (time 1780809358.44 total 116.12 last 60.46) - open/close 6875 (time 1780809368.44 total 126.12 last 74.29) - open/close 7540 (time 1780809379.36 total 137.05 last 60.87) - open/close 8308 (time 1780809389.47 total 147.16 last 75.95) - open/close 9044 (time 1780809399.48 total 157.16 last 73.59) - open/close 9819 (time 1780809409.64 total 167.32 last 76.27) - open/close 10000 (time 1780809412.20 total 169.88 last 70.72) - open/close 10754 (time 1780809422.31 total 179.99 last 74.56) - open/close 11515 (time 1780809432.48 total 190.17 last 74.80) - open/close 12255 (time 1780809443.49 total 201.18 last 67.21) - open/close 12909 (time 1780809454.44 total 212.12 last 59.77) - open/close 13606 (time 1780809464.44 total 222.12 last 69.69) - open/close 14257 (time 1780809474.44 total 232.12 last 65.08) - open/close 14996 (time 1780809484.52 total 242.20 last 73.33) - open/close 15746 (time 1780809494.82 total 252.51 last 72.78) - open/close 16498 (time 1780809504.82 total 262.51 last 75.19) - open/close 17182 (time 1780809515.69 total 273.37 last 62.96) - open/close 17828 (time 1780809527.59 total 285.27 last 54.27) - open/close 18224 (time 1780809537.76 total 295.44 last 38.93) - open/close 18551 (time 1780809547.76 total 305.45 last 32.70) - open/close 20000 (time 1780809550.59 total 308.27 last 513.01) - open/close 25270 (time 1780809560.59 total 318.27 last 526.92) - open/close 30000 (time 1780809570.15 total 327.83 last 494.70) - open/close 34460 (time 1780809580.15 total 337.84 last 445.91) - open/close 39468 (time 1780809590.15 total 347.84 last 500.74) - open/close 44392 (time 1780809600.15 total 357.84 last 492.40) - open/close 49200 (time 1780809610.15 total 367.84 last 480.74) total: 50000 open/close in 369.32 seconds: 135.38 ops/second - unlinked 0 (time 1780809612 ; total 0 ; last 0) - unlinked 10000 (time 1780809625 ; total 13 ; last 13) - unlinked 20000 (time 1780809638 ; total 26 ; last 13) - unlinked 30000 (time 1780809652 ; total 40 ; last 14) - unlinked 40000 (time 1780809664 ; total 52 ; last 12) total: 50000 unlinks in 66 seconds: 757.575745 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg240-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg240-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg240-server stop mds service on oleg240-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg240-server unloading modules via unload_modules_local on: 'oleg240-server' oleg240-server: oleg240-server.virtnet: executing unload_modules_local oleg240-server: modules unloaded. pdsh@oleg240-client: oleg240-client: ssh exited with exit code 2 pdsh@oleg240-client: oleg240-server: ssh exited with exit code 2 pdsh@oleg240-client: oleg240-client: ssh exited with exit code 2 pdsh@oleg240-client: oleg240-server: ssh exited with exit code 2