== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:48:37 (1768009717) start mds service on oleg203-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing load_modules_local oleg203-server: Loading modules from /home/green/git/lustre-release/lustre oleg203-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg203-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg203-server: detected 4 online CPUs by sysfs oleg203-server: MODOPTS_LIBCFS= oleg203-server: Force libcfs to create 2 CPU partitions oleg203-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg203-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg203-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 648 (time 1768009757.80 total 10.71 last 60.48) - open/close 1231 (time 1768009768.36 total 21.27 last 55.21) - open/close 1647 (time 1768009778.60 total 31.52 last 40.62) - open/close 2080 (time 1768009788.90 total 41.82 last 42.02) - open/close 2845 (time 1768009798.95 total 51.87 last 76.13) - open/close 3556 (time 1768009809.90 total 62.81 last 64.97) - open/close 4272 (time 1768009820.91 total 73.82 last 65.03) - open/close 4892 (time 1768009831.66 total 84.57 last 57.67) - open/close 5589 (time 1768009842.54 total 95.45 last 64.06) - open/close 6142 (time 1768009853.10 total 106.01 last 52.37) - open/close 6598 (time 1768009863.27 total 116.19 last 44.82) - open/close 7050 (time 1768009873.51 total 126.43 last 44.13) - open/close 7584 (time 1768009884.14 total 137.05 last 50.27) - open/close 8121 (time 1768009894.83 total 147.74 last 50.23) - open/close 8772 (time 1768009905.51 total 158.43 last 60.92) - open/close 9346 (time 1768009916.14 total 169.05 last 54.03) - open/close 10000 (time 1768009926.11 total 179.02 last 65.57) - open/close 10709 (time 1768009937.07 total 189.98 last 64.70) - open/close 11303 (time 1768009947.07 total 199.98 last 59.39) - open/close 11701 (time 1768009957.42 total 210.33 last 38.46) - open/close 12207 (time 1768009967.98 total 220.89 last 47.93) - open/close 12790 (time 1768009978.73 total 231.65 last 54.20) - open/close 13445 (time 1768009989.74 total 242.65 last 59.51) - open/close 14069 (time 1768010000.36 total 253.28 last 58.74) - open/close 14571 (time 1768010010.86 total 263.77 last 47.82) - open/close 15147 (time 1768010020.86 total 273.78 last 57.59) - open/close 15872 (time 1768010031.85 total 284.76 last 65.98) - open/close 16589 (time 1768010041.85 total 294.76 last 71.70) - open/close 19997 (time 1768010051.85 total 304.77 last 340.75) - open/close 24718 (time 1768010061.85 total 314.77 last 472.08) - open/close 29229 (time 1768010071.85 total 324.77 last 451.04) - open/close 33717 (time 1768010081.85 total 334.77 last 448.74) - open/close 37820 (time 1768010091.86 total 344.78 last 409.99) - open/close 40000 (time 1768010096.66 total 349.58 last 454.26) - open/close 44923 (time 1768010106.66 total 359.58 last 492.22) - open/close 49469 (time 1768010116.66 total 369.58 last 454.54) total: 50000 open/close in 370.87 seconds: 134.82 ops/second - unlinked 0 (time 1768010119 ; total 0 ; last 0) - unlinked 10000 (time 1768010131 ; total 12 ; last 12) - unlinked 20000 (time 1768010143 ; total 24 ; last 12) - unlinked 30000 (time 1768010156 ; total 37 ; last 13) - unlinked 40000 (time 1768010169 ; total 50 ; last 13) total: 50000 unlinks in 63 seconds: 793.650818 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg203-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg203-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg203-server stop mds service on oleg203-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg203-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg203-server' oleg203-server: oleg203-server.virtnet: executing unload_modules_local oleg203-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg203-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-server: ssh exited with exit code 2 pdsh@oleg203-client: oleg203-client: ssh exited with exit code 2