== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:14:40 (1764108880) start mds service on oleg141-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing load_modules_local oleg141-server: Loading modules from /home/green/git/lustre-release/lustre oleg141-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg141-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg141-server: detected 4 online CPUs by sysfs oleg141-server: MODOPTS_LIBCFS= oleg141-server: Force libcfs to create 2 CPU partitions oleg141-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg141-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg141-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg141-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 244 (time 1764108955.57 total 12.92 last 18.88) - open/close 581 (time 1764108965.58 total 22.93 last 33.68) - open/close 783 (time 1764108975.80 total 33.15 last 19.77) - open/close 1110 (time 1764108985.80 total 43.15 last 32.69) - open/close 1348 (time 1764108995.80 total 53.15 last 23.80) - open/close 1608 (time 1764109006.00 total 63.35 last 25.49) - open/close 1899 (time 1764109016.76 total 74.12 last 27.04) - open/close 2085 (time 1764109026.77 total 84.12 last 18.60) - open/close 2250 (time 1764109036.78 total 94.13 last 16.48) - open/close 2450 (time 1764109047.03 total 104.38 last 19.50) - open/close 2716 (time 1764109057.59 total 114.94 last 25.20) - open/close 2995 (time 1764109067.60 total 124.95 last 27.87) - open/close 3112 (time 1764109078.01 total 135.37 last 11.23) - open/close 3168 (time 1764109088.03 total 145.38 last 5.59) - open/close 3296 (time 1764109098.03 total 155.38 last 12.80) - open/close 3494 (time 1764109108.88 total 166.23 last 18.26) - open/close 3724 (time 1764109121.25 total 178.60 last 18.59) - open/close 3935 (time 1764109131.26 total 188.62 last 21.06) - open/close 4147 (time 1764109141.27 total 198.62 last 21.19) - open/close 4338 (time 1764109151.36 total 208.71 last 18.94) - open/close 4593 (time 1764109161.36 total 218.71 last 25.49) - open/close 4704 (time 1764109172.03 total 229.38 last 10.40) - open/close 4987 (time 1764109182.04 total 239.39 last 28.28) - open/close 5501 (time 1764109192.70 total 250.05 last 48.21) - open/close 5682 (time 1764109203.19 total 260.54 last 17.25) - open/close 5909 (time 1764109213.96 total 271.31 last 21.08) - open/close 6069 (time 1764109224.63 total 281.98 last 14.99) - open/close 6313 (time 1764109234.86 total 292.22 last 23.85) - open/close 6515 (time 1764109244.87 total 302.22 last 20.19) - open/close 6682 (time 1764109254.88 total 312.23 last 16.68) - open/close 6795 (time 1764109265.39 total 322.74 last 10.75) - open/close 7106 (time 1764109275.40 total 332.75 last 31.09) - open/close 7269 (time 1764109287.42 total 344.77 last 13.56) - open/close 7384 (time 1764109298.10 total 355.45 last 10.77) - open/close 7646 (time 1764109309.11 total 366.46 last 23.79) - open/close 7768 (time 1764109320.71 total 378.06 last 10.52) - open/close 8066 (time 1764109330.71 total 388.06 last 29.79) - open/close 8159 (time 1764109341.11 total 398.46 last 8.94) - open/close 8278 (time 1764109351.68 total 409.03 last 11.26) - open/close 8599 (time 1764109361.91 total 419.26 last 31.38) - open/close 8780 (time 1764109371.91 total 429.26 last 18.09) - open/close 8977 (time 1764109382.35 total 439.70 last 18.88) - open/close 9196 (time 1764109393.60 total 450.95 last 19.46) - open/close 9419 (time 1764109405.17 total 462.52 last 19.27) - open/close 9571 (time 1764109416.05 total 473.40 last 13.97) - open/close 9766 (time 1764109426.11 total 483.46 last 19.40) - open/close 9882 (time 1764109436.67 total 494.02 last 10.98) - open/close 10000 (time 1764109440.87 total 498.22 last 28.09) - open/close 10200 (time 1764109451.19 total 508.54 last 19.38) - open/close 10403 (time 1764109461.19 total 518.54 last 20.30) - open/close 10745 (time 1764109471.19 total 528.54 last 34.20) - open/close 10999 (time 1764109481.52 total 538.87 last 24.59) - open/close 11300 (time 1764109492.27 total 549.62 last 27.99) - open/close 11561 (time 1764109505.27 total 562.62 last 20.08) - open/close 11920 (time 1764109515.28 total 572.63 last 35.88) - open/close 13198 (time 1764109525.28 total 582.63 last 127.71) - open/close 14310 (time 1764109535.28 total 592.64 last 111.19) - open/close 15443 (time 1764109545.29 total 602.64 last 113.29) - open/close 16560 (time 1764109555.29 total 612.64 last 111.64) - open/close 17582 (time 1764109565.30 total 622.65 last 102.12) - open/close 18678 (time 1764109575.30 total 632.65 last 109.53) - open/close 19771 (time 1764109585.30 total 642.66 last 109.29) - open/close 20000 (time 1764109587.61 total 644.96 last 99.18) - open/close 21223 (time 1764109597.62 total 654.97 last 122.24) - open/close 22494 (time 1764109607.62 total 664.97 last 127.05) - open/close 23592 (time 1764109617.63 total 674.98 last 109.72) - open/close 24861 (time 1764109627.64 total 684.99 last 126.84) - open/close 26392 (time 1764109637.64 total 694.99 last 153.05) - open/close 27697 (time 1764109647.64 total 704.99 last 130.42) - open/close 29194 (time 1764109657.65 total 715.00 last 149.68) - open/close 30000 (time 1764109662.09 total 719.44 last 181.46) - open/close 31248 (time 1764109672.09 total 729.44 last 124.78) - open/close 32376 (time 1764109682.09 total 739.44 last 112.79) - open/close 33540 (time 1764109692.10 total 749.45 last 116.33) - open/close 34989 (time 1764109702.10 total 759.45 last 144.84) - open/close 36182 (time 1764109712.10 total 769.45 last 119.28) - open/close 37503 (time 1764109722.10 total 779.46 last 132.05) - open/close 38795 (time 1764109732.11 total 789.46 last 129.08) - open/close 39961 (time 1764109742.12 total 799.47 last 116.56) - open/close 41109 (time 1764109752.12 total 809.47 last 114.76) - open/close 42244 (time 1764109762.12 total 819.47 last 113.50) - open/close 43409 (time 1764109772.12 total 829.47 last 116.49) - open/close 44817 (time 1764109782.12 total 839.47 last 140.78) - open/close 45940 (time 1764109792.13 total 849.48 last 112.22) - open/close 47101 (time 1764109802.14 total 859.49 last 116.03) - open/close 48478 (time 1764109812.14 total 869.49 last 137.68) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 877.61 seconds: 56.55 ops/second - unlinked 0 (time 1764109822 ; total 0 ; last 0) - unlinked 10000 (time 1764109892 ; total 70 ; last 70) - unlinked 20000 (time 1764109934 ; total 112 ; last 42) - unlinked 30000 (time 1764109961 ; total 139 ; last 27) - unlinked 40000 (time 1764109994 ; total 172 ; last 33) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 204 seconds: 243.294113 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg141-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing unload_modules_local oleg141-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg141-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2