== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 21:18:57 (1763345937) start mds service on oleg155-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg155-server' oleg155-server: oleg155-server.virtnet: executing load_modules_local oleg155-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg155-server: Loading modules from /home/green/git/lustre-release/lustre oleg155-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg155-server: detected 4 online CPUs by sysfs oleg155-server: MODOPTS_LIBCFS= oleg155-server: Force libcfs to create 2 CPU partitions oleg155-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg155-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg155-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg155-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0000 oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 257 (time 1763346030.27 total 10.93 last 23.50) - open/close 404 (time 1763346040.28 total 20.94 last 14.69) - open/close 564 (time 1763346051.19 total 31.85 last 14.67) - open/close 699 (time 1763346061.19 total 41.86 last 13.49) - open/close 914 (time 1763346071.34 total 52.00 last 21.20) - open/close 1123 (time 1763346081.46 total 62.12 last 20.64) - open/close 1347 (time 1763346091.47 total 72.13 last 22.39) - open/close 1562 (time 1763346101.63 total 82.29 last 21.16) - open/close 1801 (time 1763346111.65 total 92.31 last 23.85) - open/close 1984 (time 1763346121.66 total 102.32 last 18.28) - open/close 2152 (time 1763346131.68 total 112.34 last 16.77) - open/close 2290 (time 1763346142.21 total 122.87 last 13.10) - open/close 2465 (time 1763346152.22 total 132.88 last 17.49) - open/close 2669 (time 1763346163.45 total 144.11 last 18.16) - open/close 2865 (time 1763346173.46 total 154.12 last 19.58) - open/close 3110 (time 1763346186.30 total 166.96 last 19.09) - open/close 3410 (time 1763346196.67 total 177.33 last 28.92) - open/close 3584 (time 1763346206.68 total 187.35 last 17.38) - open/close 3718 (time 1763346216.71 total 197.37 last 13.37) - open/close 3956 (time 1763346228.65 total 209.31 last 19.92) - open/close 4206 (time 1763346238.66 total 219.32 last 24.99) - open/close 4448 (time 1763346249.26 total 229.92 last 22.83) - open/close 4682 (time 1763346259.77 total 240.43 last 22.26) - open/close 4906 (time 1763346269.77 total 250.43 last 22.40) - open/close 5056 (time 1763346280.13 total 260.79 last 14.48) - open/close 5288 (time 1763346291.09 total 271.75 last 21.16) - open/close 5493 (time 1763346301.09 total 281.75 last 20.50) - open/close 5690 (time 1763346311.65 total 292.31 last 18.66) - open/close 5848 (time 1763346321.66 total 302.32 last 15.78) - open/close 5989 (time 1763346331.85 total 312.51 last 13.84) - open/close 6068 (time 1763346341.85 total 322.51 last 7.90) - open/close 6239 (time 1763346354.36 total 335.02 last 13.67) - open/close 6448 (time 1763346364.37 total 345.03 last 20.88) - open/close 6643 (time 1763346374.37 total 355.03 last 19.49) - open/close 6864 (time 1763346384.71 total 365.37 last 21.38) - open/close 7008 (time 1763346395.00 total 375.66 last 13.99) - open/close 7201 (time 1763346405.13 total 385.79 last 19.06) - open/close 7483 (time 1763346415.13 total 395.79 last 28.19) - open/close 7691 (time 1763346427.00 total 407.66 last 17.52) - open/close 7901 (time 1763346437.02 total 417.68 last 20.96) - open/close 8123 (time 1763346447.02 total 427.68 last 22.20) - open/close 8330 (time 1763346457.03 total 437.69 last 20.70) - open/close 8535 (time 1763346467.70 total 448.36 last 19.20) - open/close 8756 (time 1763346478.46 total 459.12 last 20.54) - open/close 8994 (time 1763346488.71 total 469.37 last 23.24) - open/close 9249 (time 1763346499.44 total 480.10 last 23.77) - open/close 9476 (time 1763346510.51 total 491.17 last 20.50) - open/close 9705 (time 1763346520.52 total 501.18 last 22.87) - open/close 9886 (time 1763346530.99 total 511.65 last 17.30) - open/close 10000 (time 1763346536.03 total 516.69 last 22.59) - open/close 10270 (time 1763346546.04 total 526.70 last 26.99) - open/close 10417 (time 1763346556.10 total 536.76 last 14.61) - open/close 10636 (time 1763346566.44 total 547.11 last 21.17) - open/close 10857 (time 1763346577.52 total 558.18 last 19.96) - open/close 11104 (time 1763346588.20 total 568.86 last 23.13) - open/close 11323 (time 1763346599.30 total 579.97 last 19.72) - open/close 11480 (time 1763346609.32 total 589.98 last 15.68) - open/close 11740 (time 1763346619.32 total 599.98 last 25.99) - open/close 11930 (time 1763346629.68 total 610.34 last 18.35) - open/close 12160 (time 1763346640.36 total 621.02 last 21.53) - open/close 12270 (time 1763346651.19 total 631.85 last 10.16) - open/close 12475 (time 1763346661.19 total 641.85 last 20.50) - open/close 12654 (time 1763346671.34 total 652.00 last 17.64) - open/close 12840 (time 1763346681.60 total 662.26 last 18.13) - open/close 13072 (time 1763346691.61 total 672.27 last 23.18) - open/close 13234 (time 1763346701.62 total 682.28 last 16.19) - open/close 13433 (time 1763346711.85 total 692.51 last 19.45) - open/close 13609 (time 1763346721.85 total 702.51 last 17.59) - open/close 13730 (time 1763346731.86 total 712.52 last 12.09) - open/close 13921 (time 1763346742.34 total 723.00 last 18.23) - open/close 14171 (time 1763346752.94 total 733.60 last 23.57) - open/close 14350 (time 1763346763.37 total 744.03 last 17.17) - open/close 14575 (time 1763346773.55 total 754.21 last 22.10) - open/close 14777 (time 1763346783.98 total 764.64 last 19.37) - open/close 15004 (time 1763346793.99 total 774.65 last 22.68) - open/close 15188 (time 1763346804.35 total 785.01 last 17.76) - open/close 15410 (time 1763346814.44 total 795.10 last 22.00) - open/close 15682 (time 1763346824.44 total 805.10 last 27.20) - open/close 15903 (time 1763346834.45 total 815.11 last 22.08) - open/close 16090 (time 1763346844.71 total 825.37 last 18.23) - open/close 16317 (time 1763346854.71 total 835.37 last 22.69) - open/close 16546 (time 1763346864.73 total 845.39 last 22.86) - open/close 16728 (time 1763346875.06 total 855.72 last 17.62) - open/close 17224 (time 1763346885.06 total 865.72 last 49.59) - open/close 17962 (time 1763346895.06 total 875.72 last 73.77) - open/close 18760 (time 1763346905.08 total 885.74 last 79.66) - open/close 19548 (time 1763346915.09 total 895.75 last 78.75) - open/close 20000 (time 1763346920.79 total 901.45 last 79.28) - open/close 20707 (time 1763346930.80 total 911.46 last 70.64) - open/close 21425 (time 1763346940.80 total 921.46 last 71.75) - open/close 22137 (time 1763346950.81 total 931.47 last 71.14) - open/close 22910 (time 1763346960.82 total 941.48 last 77.21) - open/close 23664 (time 1763346970.84 total 951.50 last 75.29) - open/close 24448 (time 1763346980.84 total 961.50 last 78.37) - open/close 25219 (time 1763346990.85 total 971.51 last 77.06) - open/close 25999 (time 1763347000.85 total 981.51 last 77.95) - open/close 26779 (time 1763347010.86 total 991.52 last 77.94) - open/close 27537 (time 1763347020.87 total 1001.53 last 75.75) - open/close 28306 (time 1763347030.87 total 1011.53 last 76.85) - open/close 29083 (time 1763347040.88 total 1021.54 last 77.64) - open/close 29815 (time 1763347050.88 total 1031.54 last 73.18) - open/close 30000 (time 1763347053.31 total 1033.97 last 76.16) - open/close 30704 (time 1763347063.33 total 1043.99 last 70.26) - open/close 31435 (time 1763347073.33 total 1053.99 last 73.09) - open/close 32137 (time 1763347083.34 total 1064.01 last 70.13) - open/close 32798 (time 1763347093.35 total 1074.01 last 66.07) - open/close 33436 (time 1763347103.36 total 1084.02 last 63.74) - open/close 34132 (time 1763347113.36 total 1094.02 last 69.57) - open/close 34857 (time 1763347123.37 total 1104.03 last 72.45) - open/close 35621 (time 1763347133.38 total 1114.04 last 76.35) - open/close 36439 (time 1763347143.39 total 1124.05 last 81.70) - open/close 37217 (time 1763347153.41 total 1134.07 last 77.64) - open/close 37930 (time 1763347163.41 total 1144.08 last 71.26) - open/close 38693 (time 1763347173.42 total 1154.08 last 76.25) - open/close 39428 (time 1763347183.43 total 1164.09 last 73.45) - open/close 40000 (time 1763347191.70 total 1172.36 last 69.14) - open/close 40688 (time 1763347201.70 total 1182.36 last 68.79) - open/close 41341 (time 1763347211.70 total 1192.37 last 65.29) - open/close 41988 (time 1763347221.71 total 1202.37 last 64.65) - open/close 42572 (time 1763347231.73 total 1212.39 last 58.31) - open/close 43261 (time 1763347241.74 total 1222.40 last 68.80) - open/close 43937 (time 1763347251.75 total 1232.41 last 67.55) - open/close 44594 (time 1763347261.76 total 1242.42 last 65.66) - open/close 45297 (time 1763347271.76 total 1252.42 last 70.29) - open/close 45980 (time 1763347281.76 total 1262.42 last 68.26) - open/close 46662 (time 1763347291.77 total 1272.43 last 68.17) - open/close 47348 (time 1763347301.78 total 1282.44 last 68.54) - open/close 48053 (time 1763347311.78 total 1292.45 last 70.44) - open/close 48740 (time 1763347321.81 total 1302.47 last 68.53) - open/close 49400 (time 1763347331.81 total 1312.48 last 65.97) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 1315.91 seconds: 37.72 ops/second - unlinked 0 (time 1763347340 ; total 0 ; last 0) - unlinked 10000 (time 1763347497 ; total 157 ; last 157) - unlinked 20000 (time 1763347664 ; total 324 ; last 167) - unlinked 30000 (time 1763347819 ; total 479 ; last 155) - unlinked 40000 (time 1763347965 ; total 625 ; last 146) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 762 seconds: 65.133858 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg155-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg155-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg155-server stop mds service on oleg155-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg155-server stop mds service on oleg155-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg155-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg155-server' oleg155-server: oleg155-server.virtnet: executing unload_modules_local oleg155-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg155-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg155-client: oleg155-client: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-server: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-client: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-server: ssh exited with exit code 2