== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:32:35 (1784226755) start mds service on oleg352-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing load_modules_local oleg352-server: Loading modules from /home/green/git/lustre-release/lustre oleg352-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg352-server: detected 4 online CPUs by sysfs oleg352-server: MODOPTS_LIBCFS= oleg352-server: Force libcfs to create 2 CPU partitions oleg352-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg352-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg352-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg352-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 257 (time 1784226798.18 total 10.00 last 25.70) - open/close 400 (time 1784226808.25 total 20.07 last 14.20) - open/close 519 (time 1784226818.26 total 30.08 last 11.90) - open/close 711 (time 1784226828.86 total 40.68 last 18.10) - open/close 1101 (time 1784226838.86 total 50.69 last 39.00) - open/close 1512 (time 1784226849.59 total 61.42 last 38.30) - open/close 1862 (time 1784226860.22 total 72.04 last 32.94) - open/close 1939 (time 1784226870.98 total 82.80 last 7.16) - open/close 2231 (time 1784226882.49 total 94.32 last 25.36) - open/close 2598 (time 1784226892.50 total 104.32 last 36.69) - open/close 3034 (time 1784226902.50 total 114.32 last 43.60) - open/close 3658 (time 1784226912.96 total 124.78 last 59.64) - open/close 4322 (time 1784226923.64 total 135.47 last 62.16) - open/close 4915 (time 1784226933.95 total 145.77 last 57.53) - open/close 5260 (time 1784226944.70 total 156.52 last 32.09) - open/close 5470 (time 1784226955.08 total 166.90 last 20.25) - open/close 5587 (time 1784226965.12 total 176.94 last 11.65) - open/close 5709 (time 1784226977.15 total 188.97 last 10.14) - open/close 5893 (time 1784226987.26 total 199.08 last 18.20) - open/close 6295 (time 1784226997.27 total 209.09 last 40.18) - open/close 6703 (time 1784227007.87 total 219.69 last 38.49) - open/close 6890 (time 1784227018.17 total 230.00 last 18.14) - open/close 7006 (time 1784227029.18 total 241.00 last 10.54) - open/close 7230 (time 1784227039.55 total 251.37 last 21.61) - open/close 7623 (time 1784227050.56 total 262.38 last 35.70) - open/close 7936 (time 1784227061.12 total 272.94 last 29.64) - open/close 8219 (time 1784227072.70 total 284.52 last 24.43) - open/close 8726 (time 1784227083.19 total 295.01 last 48.31) - open/close 9369 (time 1784227094.14 total 305.96 last 58.74) - open/close 10000 (time 1784227102.97 total 314.79 last 71.47) - open/close 10542 (time 1784227113.92 total 325.74 last 49.50) - open/close 10850 (time 1784227124.42 total 336.24 last 29.34) - open/close 11055 (time 1784227135.11 total 346.93 last 19.18) - open/close 11226 (time 1784227145.41 total 357.23 last 16.59) - open/close 12667 (time 1784227155.41 total 367.23 last 144.10) - open/close 14278 (time 1784227165.41 total 377.23 last 161.08) - open/close 16550 (time 1784227175.42 total 387.24 last 227.13) - open/close 18724 (time 1784227185.42 total 397.24 last 217.37) - open/close 20000 (time 1784227190.52 total 402.34 last 250.12) - open/close 22472 (time 1784227200.52 total 412.34 last 247.16) - open/close 25240 (time 1784227210.52 total 422.35 last 276.69) - open/close 28005 (time 1784227220.53 total 432.35 last 276.38) - open/close 30000 (time 1784227228.08 total 439.90 last 264.12) - open/close 32550 (time 1784227238.08 total 449.90 last 254.99) - open/close 35451 (time 1784227248.08 total 459.90 last 290.09) - open/close 37843 (time 1784227258.08 total 469.90 last 239.17) - open/close 40000 (time 1784227267.23 total 479.05 last 235.77) - open/close 42079 (time 1784227277.24 total 489.06 last 207.83) - open/close 43923 (time 1784227287.24 total 499.06 last 184.35) - open/close 46303 (time 1784227297.24 total 509.06 last 237.93) - open/close 47971 (time 1784227307.24 total 519.06 last 166.79) total: 50000 open/close in 528.11 seconds: 94.68 ops/second - unlinked 0 (time 1784227318 ; total 0 ; last 0) - unlinked 10000 (time 1784227350 ; total 32 ; last 32) - unlinked 20000 (time 1784227374 ; total 56 ; last 24) - unlinked 30000 (time 1784227400 ; total 82 ; last 26) - unlinked 40000 (time 1784227422 ; total 104 ; last 22) total: 50000 unlinks in 125 seconds: 400.000000 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg352-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local oleg352-server: modules unloaded. pdsh@oleg352-client: oleg352-client: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-client: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2