== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 22:17:45 (1780453065) start mds service on oleg318-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions oleg318-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg318-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg318-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg318-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 134 (time 1780453146.38 total 10.24 last 13.08) - open/close 305 (time 1780453156.61 total 20.48 last 16.70) - open/close 448 (time 1780453166.97 total 30.84 last 13.80) - open/close 629 (time 1780453177.72 total 41.59 last 16.84) - open/close 716 (time 1780453188.64 total 52.50 last 7.97) - open/close 814 (time 1780453199.51 total 63.37 last 9.01) - open/close 874 (time 1780453209.52 total 73.38 last 5.99) - open/close 1067 (time 1780453219.77 total 83.63 last 18.83) - open/close 1295 (time 1780453230.52 total 94.39 last 21.20) - open/close 1509 (time 1780453242.24 total 106.11 last 18.26) - open/close 1685 (time 1780453252.28 total 116.15 last 17.54) - open/close 1875 (time 1780453262.60 total 126.47 last 18.41) - open/close 2107 (time 1780453272.60 total 136.47 last 23.19) - open/close 2255 (time 1780453282.61 total 146.47 last 14.80) - open/close 2482 (time 1780453292.61 total 156.48 last 22.69) - open/close 2703 (time 1780453302.61 total 166.48 last 22.10) - open/close 2922 (time 1780453312.62 total 176.49 last 21.88) - open/close 3111 (time 1780453322.84 total 186.71 last 18.50) - open/close 3217 (time 1780453333.06 total 196.93 last 10.37) - open/close 3419 (time 1780453343.09 total 206.95 last 20.15) - open/close 3476 (time 1780453355.52 total 219.39 last 4.58) - open/close 3542 (time 1780453365.56 total 229.42 last 6.58) - open/close 3656 (time 1780453375.56 total 239.43 last 11.39) - open/close 3760 (time 1780453385.57 total 249.44 last 10.39) - open/close 3913 (time 1780453395.97 total 259.84 last 14.70) - open/close 4021 (time 1780453406.91 total 270.78 last 9.88) - open/close 4238 (time 1780453417.39 total 281.26 last 20.69) - open/close 4464 (time 1780453427.44 total 291.31 last 22.50) - open/close 4653 (time 1780453437.56 total 301.42 last 18.68) - open/close 4890 (time 1780453447.63 total 311.50 last 23.52) - open/close 5011 (time 1780453457.64 total 321.51 last 12.09) - open/close 5169 (time 1780453468.22 total 332.09 last 14.94) - open/close 5335 (time 1780453478.24 total 342.11 last 16.56) - open/close 5471 (time 1780453488.90 total 352.77 last 12.76) - open/close 5547 (time 1780453498.91 total 362.77 last 7.60) - open/close 5757 (time 1780453509.46 total 373.33 last 19.89) - open/close 5890 (time 1780453519.81 total 383.68 last 12.86) - open/close 6039 (time 1780453530.25 total 394.12 last 14.27) - open/close 6220 (time 1780453540.26 total 404.13 last 18.08) - open/close 6375 (time 1780453552.05 total 415.92 last 13.15) - open/close 6548 (time 1780453562.13 total 426.00 last 17.16) - open/close 6746 (time 1780453574.96 total 438.82 last 15.44) - open/close 6857 (time 1780453585.78 total 449.65 last 10.26) - open/close 6916 (time 1780453595.82 total 459.69 last 5.88) - open/close 7053 (time 1780453605.95 total 469.82 last 13.52) - open/close 7158 (time 1780453616.06 total 479.92 last 10.39) - open/close 7276 (time 1780453626.12 total 489.99 last 11.73) - open/close 7365 (time 1780453636.37 total 500.23 last 8.69) - open/close 7513 (time 1780453646.59 total 510.46 last 14.47) - open/close 7742 (time 1780453656.63 total 520.50 last 22.81) - open/close 7935 (time 1780453667.02 total 530.89 last 18.57) - open/close 8221 (time 1780453678.02 total 541.89 last 26.00) - open/close 8473 (time 1780453688.06 total 551.92 last 25.12) - open/close 8625 (time 1780453698.58 total 562.44 last 14.45) - open/close 8858 (time 1780453709.27 total 573.14 last 21.78) - open/close 8969 (time 1780453720.00 total 583.86 last 10.35) - open/close 9091 (time 1780453731.64 total 595.51 last 10.48) - open/close 9262 (time 1780453741.65 total 605.52 last 17.08) - open/close 9433 (time 1780453752.61 total 616.48 last 15.60) - open/close 9500 (time 1780453762.62 total 626.48 last 6.70) - open/close 9705 (time 1780453772.64 total 636.50 last 20.46) - open/close 9957 (time 1780453783.55 total 647.42 last 23.08) - open/close 10172 (time 1780453794.59 total 658.46 last 19.48) - open/close 10419 (time 1780453804.60 total 668.46 last 24.69) - open/close 10552 (time 1780453814.61 total 678.48 last 13.28) - open/close 10745 (time 1780453825.21 total 689.07 last 18.22) - open/close 10875 (time 1780453835.21 total 699.08 last 13.00) - open/close 11164 (time 1780453846.22 total 710.09 last 26.25) - open/close 11346 (time 1780453856.85 total 720.71 last 17.13) - open/close 11543 (time 1780453867.39 total 731.25 last 18.69) - open/close 11720 (time 1780453878.10 total 741.97 last 16.52) - open/close 11931 (time 1780453889.08 total 752.95 last 19.21) - open/close 12160 (time 1780453899.10 total 762.97 last 22.86) - open/close 12398 (time 1780453909.95 total 773.82 last 21.94) - open/close 12640 (time 1780453919.95 total 783.82 last 24.19) - open/close 12825 (time 1780453929.98 total 793.84 last 18.46) - open/close 12932 (time 1780453941.24 total 805.11 last 9.50) - open/close 13076 (time 1780453951.62 total 815.48 last 13.88) - open/close 13123 (time 1780453961.99 total 825.85 last 4.53) - open/close 13280 (time 1780453975.11 total 838.98 last 11.97) - open/close 13437 (time 1780453985.89 total 849.76 last 14.56) - open/close 13615 (time 1780453995.90 total 859.77 last 17.79) - open/close 13775 (time 1780454006.41 total 870.27 last 15.23) - open/close 13892 (time 1780454016.41 total 880.28 last 11.70) - open/close 14090 (time 1780454026.95 total 890.81 last 18.79) - open/close 14315 (time 1780454036.97 total 900.84 last 22.43) - open/close 14554 (time 1780454048.07 total 911.93 last 21.55) - open/close 14756 (time 1780454058.43 total 922.30 last 19.49) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-14961) error: No space left on device total: 14961 open/close in 931.68 seconds: 16.06 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 8987: kill: (74314) - No such process - unlinked 0 (time 1780454081 ; total 0 ; last 0) - unlinked 10000 (time 1780454230 ; total 149 ; last 149) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-14961) error: No such file or directory total: 14961 unlinks in 197 seconds: 75.944160 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg318-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server stop mds service on oleg318-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg318-server unloading modules via unload_modules_local on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing unload_modules_local oleg318-server: modules unloaded. pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2