== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 13:41:15 (1783446075) start mds service on oleg242-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg242-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg242-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg242-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 277 (time 1783446166.27 total 10.12 last 27.38) - open/close 557 (time 1783446176.29 total 20.14 last 27.94) - open/close 752 (time 1783446187.32 total 31.16 last 17.69) - open/close 922 (time 1783446197.57 total 41.41 last 16.58) - open/close 1127 (time 1783446208.50 total 52.35 last 18.75) - open/close 1358 (time 1783446218.51 total 62.35 last 23.08) - open/close 1596 (time 1783446228.55 total 72.39 last 23.70) - open/close 1844 (time 1783446239.18 total 83.02 last 23.33) - open/close 2008 (time 1783446249.60 total 93.44 last 15.75) - open/close 2125 (time 1783446259.62 total 103.46 last 11.67) - open/close 2324 (time 1783446269.63 total 113.47 last 19.88) - open/close 2560 (time 1783446280.06 total 123.91 last 22.62) - open/close 2823 (time 1783446290.07 total 133.92 last 26.27) - open/close 3030 (time 1783446300.49 total 144.33 last 19.87) - open/close 3300 (time 1783446311.36 total 155.21 last 24.83) - open/close 3554 (time 1783446322.07 total 165.91 last 23.72) - open/close 3732 (time 1783446332.87 total 176.71 last 16.48) - open/close 4103 (time 1783446342.87 total 186.72 last 37.09) - open/close 4329 (time 1783446353.91 total 197.76 last 20.47) - open/close 4628 (time 1783446363.92 total 207.76 last 29.88) - open/close 4850 (time 1783446374.65 total 218.49 last 20.70) - open/close 4990 (time 1783446385.47 total 229.31 last 12.94) - open/close 5127 (time 1783446395.48 total 239.32 last 13.68) - open/close 5322 (time 1783446405.49 total 249.33 last 19.48) - open/close 5491 (time 1783446415.74 total 259.58 last 16.49) - open/close 5740 (time 1783446426.75 total 270.59 last 22.61) - open/close 5872 (time 1783446437.69 total 281.53 last 12.07) - open/close 6015 (time 1783446448.70 total 292.54 last 12.98) - open/close 6256 (time 1783446459.28 total 303.12 last 22.78) - open/close 6422 (time 1783446469.68 total 313.52 last 15.95) - open/close 6613 (time 1783446479.86 total 323.71 last 18.76) - open/close 6921 (time 1783446489.86 total 333.71 last 30.80) - open/close 7058 (time 1783446499.96 total 343.80 last 13.57) - open/close 7265 (time 1783446510.85 total 354.69 last 19.00) - open/close 7461 (time 1783446521.46 total 365.31 last 18.47) - open/close 7691 (time 1783446531.59 total 375.44 last 22.71) - open/close 7937 (time 1783446542.26 total 386.11 last 23.05) - open/close 8205 (time 1783446552.50 total 396.34 last 26.18) - open/close 8405 (time 1783446562.93 total 406.78 last 19.17) - open/close 8634 (time 1783446573.26 total 417.10 last 22.18) - open/close 8881 (time 1783446584.02 total 427.86 last 22.96) - open/close 9105 (time 1783446594.02 total 437.86 last 22.40) - open/close 9310 (time 1783446604.79 total 448.63 last 19.03) - open/close 9497 (time 1783446615.61 total 459.46 last 17.28) - open/close 9778 (time 1783446625.61 total 469.46 last 28.09) - open/close 9963 (time 1783446636.17 total 480.01 last 17.53) - open/close 10226 (time 1783446646.17 total 490.02 last 26.29) - open/close 10438 (time 1783446657.21 total 501.05 last 19.21) - open/close 10635 (time 1783446667.90 total 511.74 last 18.43) - open/close 10865 (time 1783446677.90 total 521.75 last 22.99) - open/close 11084 (time 1783446687.95 total 531.79 last 21.81) - open/close 11317 (time 1783446697.95 total 541.79 last 23.29) - open/close 11485 (time 1783446708.28 total 552.12 last 16.26) - open/close 11622 (time 1783446719.29 total 563.14 last 12.44) - open/close 11827 (time 1783446730.04 total 573.88 last 19.08) - open/close 11955 (time 1783446740.98 total 584.83 last 11.69) - open/close 12132 (time 1783446751.29 total 595.13 last 17.18) - open/close 12424 (time 1783446761.54 total 605.38 last 28.49) - open/close 12596 (time 1783446771.54 total 615.39 last 17.19) - open/close 12756 (time 1783446782.46 total 626.31 last 14.65) - open/close 12985 (time 1783446793.09 total 636.93 last 21.56) - open/close 13128 (time 1783446803.09 total 646.94 last 14.29) - open/close 13290 (time 1783446813.10 total 656.94 last 16.20) - open/close 13462 (time 1783446823.43 total 667.27 last 16.64) - open/close 14111 (time 1783446833.44 total 677.28 last 64.83) - open/close 15395 (time 1783446843.44 total 687.29 last 128.36) - open/close 16631 (time 1783446853.45 total 697.29 last 123.53) - open/close 17979 (time 1783446863.46 total 707.30 last 134.68) - open/close 19309 (time 1783446873.46 total 717.30 last 132.99) - open/close 20000 (time 1783446878.84 total 722.68 last 128.52) - open/close 21297 (time 1783446888.84 total 732.68 last 129.65) - open/close 22691 (time 1783446898.84 total 742.69 last 139.34) - open/close 24006 (time 1783446908.85 total 752.69 last 131.48) - open/close 25276 (time 1783446918.86 total 762.70 last 126.85) - open/close 26549 (time 1783446928.86 total 772.70 last 127.25) - open/close 27905 (time 1783446938.86 total 782.71 last 135.56) - open/close 29220 (time 1783446948.87 total 792.71 last 131.44) - open/close 30000 (time 1783446954.98 total 798.82 last 127.72) - open/close 31140 (time 1783446964.99 total 808.84 last 113.79) - open/close 32392 (time 1783446975.00 total 818.84 last 125.16) - open/close 33747 (time 1783446985.01 total 828.85 last 135.38) - open/close 35024 (time 1783446995.02 total 838.86 last 127.59) - open/close 36259 (time 1783447005.02 total 848.86 last 123.50) - open/close 37576 (time 1783447015.02 total 858.86 last 131.67) - open/close 38744 (time 1783447025.02 total 868.87 last 116.71) - open/close 40000 (time 1783447034.79 total 878.63 last 128.67) - open/close 41198 (time 1783447044.79 total 888.63 last 119.75) - open/close 42432 (time 1783447054.79 total 898.64 last 123.37) - open/close 43735 (time 1783447064.80 total 908.64 last 130.28) - open/close 45049 (time 1783447074.80 total 918.64 last 131.38) - open/close 46343 (time 1783447084.80 total 928.64 last 129.38) - open/close 47636 (time 1783447094.80 total 938.65 last 129.23) - open/close 48995 (time 1783447104.80 total 948.65 last 135.88) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 953.35 seconds: 52.06 ops/second - unlinked 0 (time 1783447111 ; total 0 ; last 0) - unlinked 10000 (time 1783447184 ; total 73 ; last 73) - unlinked 20000 (time 1783447227 ; total 116 ; last 43) - unlinked 30000 (time 1783447270 ; total 159 ; last 43) - unlinked 40000 (time 1783447313 ; total 202 ; last 43) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 245 seconds: 202.575516 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg242-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg242-server unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local oleg242-server: modules unloaded. pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2