== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:23:37 (1789133017) start mds service on oleg359-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing load_modules_local oleg359-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg359-server: Loading modules from /home/green/git/lustre-release/lustre oleg359-server: detected 4 online CPUs by sysfs oleg359-server: MODOPTS_LIBCFS= oleg359-server: Force libcfs to create 2 CPU partitions oleg359-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg359-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg359-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg359-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Started lustre-OST0000 oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg359-client.virtnet: -o user_xattr,flock 192.168.203.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 269 (time 1789133101.88 total 10.15 last 26.51) - open/close 551 (time 1789133112.50 total 20.77 last 26.54) - open/close 758 (time 1789133123.06 total 31.33 last 19.60) - open/close 1003 (time 1789133133.44 total 41.71 last 23.61) - open/close 1270 (time 1789133144.14 total 52.41 last 24.96) - open/close 1503 (time 1789133154.57 total 62.84 last 22.34) - open/close 1719 (time 1789133164.58 total 72.85 last 21.59) - open/close 2004 (time 1789133175.53 total 83.80 last 26.01) - open/close 2301 (time 1789133186.37 total 94.64 last 27.39) - open/close 2506 (time 1789133196.39 total 104.66 last 20.46) - open/close 2743 (time 1789133206.40 total 114.67 last 23.69) - open/close 2986 (time 1789133216.59 total 124.87 last 23.83) - open/close 3206 (time 1789133226.60 total 134.87 last 21.99) - open/close 3435 (time 1789133236.72 total 144.99 last 22.61) - open/close 3657 (time 1789133247.22 total 155.50 last 21.14) - open/close 3878 (time 1789133257.23 total 165.50 last 22.08) - open/close 4118 (time 1789133267.24 total 175.51 last 23.98) - open/close 4339 (time 1789133277.25 total 185.52 last 22.08) - open/close 4590 (time 1789133287.47 total 195.74 last 24.56) - open/close 4847 (time 1789133298.26 total 206.53 last 23.83) - open/close 5141 (time 1789133309.17 total 217.44 last 26.93) - open/close 5353 (time 1789133320.83 total 229.10 last 18.19) - open/close 5529 (time 1789133330.84 total 239.11 last 17.58) - open/close 5674 (time 1789133340.87 total 249.15 last 14.45) - open/close 5891 (time 1789133351.18 total 259.45 last 21.06) - open/close 6155 (time 1789133362.18 total 270.45 last 24.01) - open/close 6358 (time 1789133372.74 total 281.01 last 19.21) - open/close 6610 (time 1789133382.75 total 291.02 last 25.19) - open/close 6848 (time 1789133392.76 total 301.03 last 23.78) - open/close 7024 (time 1789133402.83 total 311.10 last 17.47) - open/close 7100 (time 1789133413.64 total 321.91 last 7.03) - open/close 7206 (time 1789133424.11 total 332.38 last 10.13) - open/close 7392 (time 1789133434.36 total 342.63 last 18.15) - open/close 7500 (time 1789133444.48 total 352.75 last 10.67) - open/close 7598 (time 1789133454.48 total 362.76 last 9.80) - open/close 7746 (time 1789133464.56 total 372.83 last 14.69) - open/close 7833 (time 1789133475.01 total 383.28 last 8.33) - open/close 8028 (time 1789133485.01 total 393.28 last 19.49) - open/close 8251 (time 1789133495.10 total 403.37 last 22.11) - open/close 8509 (time 1789133505.65 total 413.92 last 24.45) - open/close 8774 (time 1789133515.67 total 423.94 last 26.43) - open/close 9045 (time 1789133525.69 total 433.97 last 27.04) - open/close 9230 (time 1789133536.50 total 444.77 last 17.12) - open/close 9476 (time 1789133547.51 total 455.78 last 22.35) - open/close 9550 (time 1789133557.51 total 465.78 last 7.40) - open/close 9723 (time 1789133572.76 total 481.03 last 11.35) - open/close 9976 (time 1789133582.76 total 491.03 last 25.28) - open/close 10218 (time 1789133592.77 total 501.04 last 24.19) - open/close 10431 (time 1789133603.66 total 511.94 last 19.54) - open/close 10585 (time 1789133614.13 total 522.40 last 14.71) - open/close 10721 (time 1789133624.14 total 532.41 last 13.59) - open/close 10999 (time 1789133635.19 total 543.46 last 25.15) - open/close 11235 (time 1789133645.21 total 553.48 last 23.55) - open/close 11427 (time 1789133660.62 total 568.89 last 12.46) - open/close 11631 (time 1789133670.62 total 578.89 last 20.39) - open/close 11803 (time 1789133680.63 total 588.90 last 17.18) - open/close 11928 (time 1789133691.61 total 599.88 last 11.39) - open/close 12221 (time 1789133701.63 total 609.90 last 29.23) - open/close 12350 (time 1789133711.87 total 620.14 last 12.60) - open/close 12493 (time 1789133721.88 total 630.15 last 14.30) - open/close 12637 (time 1789133732.81 total 641.08 last 13.16) - open/close 12713 (time 1789133745.21 total 653.48 last 6.13) - open/close 12872 (time 1789133756.30 total 664.57 last 14.34) - open/close 12955 (time 1789133767.14 total 675.41 last 7.66) - open/close 13081 (time 1789133777.91 total 686.18 last 11.70) - open/close 13142 (time 1789133787.93 total 696.20 last 6.09) - open/close 13226 (time 1789133797.94 total 706.22 last 8.39) - open/close 13347 (time 1789133808.89 total 717.16 last 11.06) - open/close 13494 (time 1789133819.71 total 727.98 last 13.58) - open/close 13673 (time 1789133832.51 total 740.78 last 13.99) - open/close 13900 (time 1789133843.07 total 751.34 last 21.50) - open/close 14134 (time 1789133853.09 total 761.36 last 23.36) - open/close 14328 (time 1789133863.10 total 771.37 last 19.38) - open/close 14466 (time 1789133873.27 total 781.54 last 13.57) - open/close 14596 (time 1789133884.35 total 792.62 last 11.73) - open/close 14707 (time 1789133896.70 total 804.97 last 8.99) - open/close 14832 (time 1789133907.13 total 815.40 last 11.99) - open/close 15097 (time 1789133917.14 total 825.41 last 26.47) - open/close 15313 (time 1789133927.17 total 835.44 last 21.54) - open/close 15598 (time 1789133937.18 total 845.45 last 28.48) - open/close 15713 (time 1789133947.18 total 855.46 last 11.49) - open/close 15899 (time 1789133957.38 total 865.65 last 18.25) - open/close 16119 (time 1789133968.12 total 876.39 last 20.48) - open/close 16325 (time 1789133978.88 total 887.15 last 19.15) - open/close 16584 (time 1789133989.90 total 898.17 last 23.51) - open/close 16765 (time 1789134000.30 total 908.57 last 17.39) - open/close 16960 (time 1789134010.32 total 918.59 last 19.46) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-17041) error: No space left on device total: 17041 open/close in 921.17 seconds: 18.50 ops/second /home/green/git/lustre-release/lustre/tests/conf-sanity.sh: line 9017: kill: (76006) - No such process - unlinked 0 (time 1789134025 ; total 0 ; last 0) - unlinked 10000 (time 1789134178 ; total 153 ; last 153) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-17041) error: No such file or directory total: 17041 unlinks in 216 seconds: 78.893517 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg359-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg359-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg359-server stop mds service on oleg359-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg359-server unloading modules via unload_modules_local on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing unload_modules_local oleg359-server: modules unloaded. pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2