== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:24:14 (1786483454) start mds service on oleg206-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing load_modules_local oleg206-server: Loading modules from /home/green/git/lustre-release/lustre oleg206-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg206-server: detected 4 online CPUs by sysfs oleg206-server: MODOPTS_LIBCFS= oleg206-server: Force libcfs to create 2 CPU partitions oleg206-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg206-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg206-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg206-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg206-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-OST0000 oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 196 (time 1786483521.63 total 10.58 last 18.53) - open/close 486 (time 1786483532.58 total 21.52 last 26.50) - open/close 703 (time 1786483543.33 total 32.28 last 20.18) - open/close 1063 (time 1786483554.08 total 43.03 last 33.50) - open/close 1264 (time 1786483564.51 total 53.46 last 19.26) - open/close 1475 (time 1786483575.26 total 64.21 last 19.63) - open/close 1745 (time 1786483587.23 total 76.18 last 22.56) - open/close 1958 (time 1786483597.86 total 86.80 last 20.04) - open/close 2182 (time 1786483608.36 total 97.31 last 21.32) - open/close 2440 (time 1786483619.37 total 108.31 last 23.45) - open/close 2655 (time 1786483630.04 total 118.99 last 20.13) - open/close 2888 (time 1786483640.67 total 129.62 last 21.93) - open/close 3183 (time 1786483651.50 total 140.45 last 27.24) - open/close 3471 (time 1786483662.37 total 151.32 last 26.49) - open/close 3857 (time 1786483673.31 total 162.26 last 35.29) - open/close 4160 (time 1786483683.93 total 172.88 last 28.51) - open/close 4518 (time 1786483694.62 total 183.57 last 33.50) - open/close 4899 (time 1786483705.19 total 194.14 last 36.06) - open/close 5381 (time 1786483715.19 total 204.14 last 48.20) - open/close 5626 (time 1786483725.60 total 214.55 last 23.53) - open/close 5801 (time 1786483735.71 total 224.66 last 17.30) - open/close 6226 (time 1786483745.72 total 234.67 last 42.49) - open/close 6505 (time 1786483756.52 total 245.46 last 25.84) - open/close 6778 (time 1786483766.88 total 255.82 last 26.35) - open/close 7035 (time 1786483777.57 total 266.52 last 24.03) - open/close 7214 (time 1786483787.87 total 276.82 last 17.38) - open/close 7391 (time 1786483797.99 total 286.93 last 17.49) - open/close 7630 (time 1786483808.55 total 297.50 last 22.62) - open/close 8114 (time 1786483818.67 total 307.62 last 47.84) - open/close 8353 (time 1786483828.84 total 317.78 last 23.50) - open/close 8638 (time 1786483839.52 total 328.47 last 26.68) - open/close 9098 (time 1786483849.63 total 338.58 last 45.51) - open/close 9273 (time 1786483859.93 total 348.88 last 16.98) - open/close 9562 (time 1786483869.93 total 358.88 last 28.89) - open/close 9720 (time 1786483880.03 total 368.98 last 15.65) - open/close 10000 (time 1786483885.63 total 374.58 last 50.02) - open/close 12317 (time 1786483895.63 total 384.58 last 231.64) - open/close 15260 (time 1786483905.64 total 394.58 last 294.24) - open/close 18079 (time 1786483915.64 total 404.59 last 281.84) - open/close 20000 (time 1786483923.86 total 412.81 last 233.67) - open/close 22410 (time 1786483933.86 total 422.81 last 240.92) - open/close 24449 (time 1786483943.87 total 432.82 last 203.79) - open/close 25471 (time 1786483953.87 total 442.82 last 102.14) - open/close 26650 (time 1786483963.88 total 452.83 last 117.84) - open/close 27766 (time 1786483973.88 total 462.83 last 111.54) - open/close 28623 (time 1786483983.90 total 472.85 last 85.55) - open/close 29552 (time 1786483993.90 total 482.85 last 92.86) - open/close 30000 (time 1786483998.23 total 487.18 last 103.48) - open/close 31254 (time 1786484008.25 total 497.20 last 125.16) - open/close 32204 (time 1786484018.25 total 507.20 last 94.99) - open/close 32792 (time 1786484028.26 total 517.21 last 58.75) - open/close 33285 (time 1786484038.26 total 527.21 last 49.30) - open/close 34015 (time 1786484048.26 total 537.21 last 72.98) - open/close 34867 (time 1786484058.26 total 547.21 last 85.20) - open/close 35546 (time 1786484068.27 total 557.22 last 67.85) - open/close 36249 (time 1786484078.29 total 567.24 last 70.18) - open/close 36992 (time 1786484088.31 total 577.26 last 74.15) - open/close 37894 (time 1786484098.32 total 587.27 last 90.11) - open/close 38829 (time 1786484108.32 total 597.27 last 93.46) - open/close 39585 (time 1786484118.33 total 607.28 last 75.58) - open/close 40000 (time 1786484127.93 total 616.88 last 43.23) - open/close 40529 (time 1786484137.96 total 626.91 last 52.72) - open/close 41072 (time 1786484147.97 total 636.92 last 54.27) - open/close 41737 (time 1786484157.97 total 646.92 last 66.48) - open/close 42564 (time 1786484167.97 total 656.92 last 82.69) - open/close 43525 (time 1786484177.97 total 666.92 last 96.07) - open/close 44352 (time 1786484187.98 total 676.93 last 82.66) - open/close 45306 (time 1786484197.99 total 686.94 last 95.33) - open/close 46016 (time 1786484207.99 total 696.94 last 70.99) - open/close 46892 (time 1786484217.99 total 706.94 last 87.54) - open/close 47754 (time 1786484228.00 total 716.95 last 86.13) - open/close 48569 (time 1786484238.01 total 726.96 last 81.46) - open/close 49480 (time 1786484248.02 total 736.97 last 91.01) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 738.80 seconds: 67.18 ops/second - unlinked 0 (time 1786484254 ; total 0 ; last 0) - unlinked 10000 (time 1786484360 ; total 106 ; last 106) - unlinked 20000 (time 1786484420 ; total 166 ; last 60) - unlinked 30000 (time 1786484471 ; total 217 ; last 51) - unlinked 40000 (time 1786484522 ; total 268 ; last 51) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 311 seconds: 159.585205 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg206-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg206-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg206-server unloading modules via unload_modules_local on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing unload_modules_local oleg206-server: modules unloaded. pdsh@oleg206-client: oleg206-client: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-server: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-client: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-server: ssh exited with exit code 2