== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 06:36:51 (1769081811) start mds service on oleg159-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions oleg159-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg159-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg159-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 339 (time 1769081861.91 total 10.42 last 32.54) - open/close 666 (time 1769081871.92 total 20.42 last 32.69) - open/close 1051 (time 1769081882.96 total 31.46 last 34.88) - open/close 1309 (time 1769081893.34 total 41.84 last 24.85) - open/close 1626 (time 1769081903.35 total 51.85 last 31.67) - open/close 1906 (time 1769081913.94 total 62.45 last 26.42) - open/close 2240 (time 1769081923.95 total 72.45 last 33.38) - open/close 2395 (time 1769081934.80 total 83.31 last 14.28) - open/close 2722 (time 1769081947.03 total 95.53 last 26.74) - open/close 2950 (time 1769081957.03 total 105.54 last 22.79) - open/close 3167 (time 1769081968.02 total 116.52 last 19.75) - open/close 3417 (time 1769081978.96 total 127.47 last 22.84) - open/close 3700 (time 1769081989.08 total 137.58 last 27.98) - open/close 3954 (time 1769081999.89 total 148.39 last 23.50) - open/close 4152 (time 1769082010.64 total 159.14 last 18.42) - open/close 4408 (time 1769082020.64 total 169.15 last 25.58) - open/close 4632 (time 1769082031.31 total 179.81 last 21.00) - open/close 4816 (time 1769082041.61 total 190.12 last 17.85) - open/close 5030 (time 1769082052.56 total 201.06 last 19.56) - open/close 5397 (time 1769082063.31 total 211.81 last 34.14) - open/close 5552 (time 1769082073.31 total 221.82 last 15.49) - open/close 5754 (time 1769082083.72 total 232.23 last 19.40) - open/close 6220 (time 1769082093.84 total 242.34 last 46.08) - open/close 6638 (time 1769082104.14 total 252.65 last 40.56) - open/close 6889 (time 1769082115.02 total 263.53 last 23.07) - open/close 7129 (time 1769082125.03 total 273.53 last 23.98) - open/close 7347 (time 1769082135.50 total 284.00 last 20.83) - open/close 7476 (time 1769082145.67 total 294.18 last 12.68) - open/close 7717 (time 1769082155.68 total 304.18 last 24.09) - open/close 7953 (time 1769082165.68 total 314.19 last 23.59) - open/close 8332 (time 1769082175.69 total 324.19 last 37.89) - open/close 8671 (time 1769082186.44 total 334.95 last 31.51) - open/close 8985 (time 1769082196.88 total 345.39 last 30.09) - open/close 9377 (time 1769082207.76 total 356.27 last 36.03) - open/close 9773 (time 1769082217.77 total 366.27 last 39.58) - open/close 10000 (time 1769082225.90 total 374.41 last 27.91) - open/close 10271 (time 1769082236.56 total 385.06 last 25.42) - open/close 10547 (time 1769082246.86 total 395.36 last 26.80) - open/close 10796 (time 1769082256.86 total 405.37 last 24.89) - open/close 11095 (time 1769082267.09 total 415.59 last 29.25) - open/close 12049 (time 1769082277.09 total 425.59 last 95.37) - open/close 14202 (time 1769082287.09 total 435.59 last 215.27) - open/close 16273 (time 1769082297.09 total 445.60 last 207.06) - open/close 17925 (time 1769082307.10 total 455.60 last 165.15) - open/close 19567 (time 1769082317.10 total 465.60 last 164.19) - open/close 20000 (time 1769082319.39 total 467.90 last 188.57) - open/close 22256 (time 1769082329.40 total 477.90 last 225.53) - open/close 24400 (time 1769082339.40 total 487.90 last 214.34) - open/close 26170 (time 1769082349.40 total 497.90 last 176.98) - open/close 27962 (time 1769082359.40 total 507.91 last 179.14) - open/close 29874 (time 1769082369.40 total 517.91 last 191.18) - open/close 31605 (time 1769082379.41 total 527.91 last 173.07) - open/close 33570 (time 1769082389.41 total 537.91 last 196.44) - open/close 36064 (time 1769082399.41 total 547.91 last 249.36) - open/close 38347 (time 1769082409.41 total 557.92 last 228.23) - open/close 40000 (time 1769082416.83 total 565.34 last 222.72) - open/close 41795 (time 1769082426.84 total 575.34 last 179.45) - open/close 43950 (time 1769082436.84 total 585.34 last 215.50) - open/close 46046 (time 1769082446.84 total 595.34 last 209.56) - open/close 48167 (time 1769082456.84 total 605.35 last 212.06) - open/close 49919 (time 1769082466.85 total 615.35 last 175.11) total: 50000 open/close in 615.74 seconds: 81.20 ops/second - unlinked 0 (time 1769082468 ; total 0 ; last 0) - unlinked 10000 (time 1769082502 ; total 34 ; last 34) - unlinked 20000 (time 1769082541 ; total 73 ; last 39) - unlinked 30000 (time 1769082570 ; total 102 ; last 29) - unlinked 40000 (time 1769082600 ; total 132 ; last 30) total: 50000 unlinks in 159 seconds: 314.465424 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg159-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg159-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2