== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:09:33 (1781046573) Stopping clients: oleg212-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg212-client.virtnet /mnt/lustre2 (opts:-f) oleg212-server: oleg212-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg212-server' oleg212-server: oleg212-server.virtnet: executing load_modules_local oleg212-server: Loading modules from /home/green/git/lustre-release/lustre oleg212-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg212-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg212-server: detected 4 online CPUs by sysfs oleg212-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg212-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg212-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg212-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg212-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg212-server: oleg212-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg212-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg212-client.virtnet: -o user_xattr,flock oleg212-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1974 (time 1781046631.67 total 10.00 last 197.35) - open/close 3800 (time 1781046641.67 total 20.00 last 182.57) - open/close 6156 (time 1781046651.67 total 30.01 last 235.57) - open/close 8624 (time 1781046661.67 total 40.01 last 246.79) - open/close 10000 (time 1781046666.50 total 44.83 last 285.25) - open/close 13374 (time 1781046676.50 total 54.83 last 337.24) - open/close 15446 (time 1781046686.50 total 64.83 last 207.18) - open/close 17972 (time 1781046696.50 total 74.84 last 252.54) - open/close 20000 (time 1781046705.34 total 83.68 last 229.38) - open/close 22163 (time 1781046715.35 total 93.69 last 216.13) - open/close 24633 (time 1781046725.36 total 103.69 last 246.89) - open/close 26777 (time 1781046735.36 total 113.70 last 214.28) - open/close 28733 (time 1781046745.36 total 123.70 last 195.57) - open/close 30000 (time 1781046751.65 total 129.98 last 201.52) - open/close 31963 (time 1781046761.65 total 139.99 last 196.29) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 143.62 seconds: 226.94 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 3398 (time 1781046777.02 total 10.00 last 339.72) - /unlink 7851 (time 1781046787.02 total 20.00 last 445.23) - /unlink 10000 (time 1781046791.92 total 24.90 last 438.89) - /unlink 15244 (time 1781046801.92 total 34.90 last 524.20) - /unlink 20000 (time 1781046810.98 total 43.96 last 525.37) - /unlink 25444 (time 1781046820.98 total 53.96 last 544.28) - /unlink 29849 (time 1781046830.98 total 63.96 last 440.41) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 71.16 seconds: 458.05 ops/second Waiting for MDT destroys to complete - open/close 4171 (time 1781046859.94 total 10.00 last 417.08) - open/close 7192 (time 1781046869.94 total 20.00 last 302.04) - open/close 9725 (time 1781046879.94 total 30.00 last 253.28) - open/close 12687 (time 1781046889.95 total 40.00 last 296.16) - open/close 15180 (time 1781046899.95 total 50.01 last 249.12) - open/close 17998 (time 1781046909.95 total 60.01 last 281.80) - open/close 20000 (time 1781046917.59 total 67.65 last 262.25) - open/close 22878 (time 1781046927.59 total 77.65 last 287.77) - open/close 26173 (time 1781046937.59 total 87.65 last 329.44) - open/close 30000 (time 1781046947.28 total 97.34 last 394.79) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 102.82 seconds: 316.99 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 7991 (time 1781046964.23 total 10.00 last 799.07) - /unlink 10000 (time 1781046967.97 total 13.74 last 536.67) - /unlink 15977 (time 1781046977.97 total 23.74 last 597.66) - /unlink 20000 (time 1781046983.99 total 29.76 last 668.85) - /unlink 26674 (time 1781046993.99 total 39.76 last 667.34) - /unlink 30000 (time 1781046998.80 total 44.57 last 691.51) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 49.03 seconds: 664.80 ops/second Waiting for MDT destroys to complete - open/close 3145 (time 1781047036.65 total 10.00 last 314.49) - open/close 6440 (time 1781047046.65 total 20.00 last 329.47) - open/close 10000 (time 1781047056.55 total 29.90 last 359.81) - open/close 13157 (time 1781047066.55 total 39.90 last 315.66) - open/close 15745 (time 1781047076.55 total 49.90 last 258.68) - open/close 18470 (time 1781047086.55 total 59.90 last 272.49) - open/close 20000 (time 1781047093.28 total 66.63 last 227.46) - open/close 22045 (time 1781047103.29 total 76.64 last 204.35) - open/close 25192 (time 1781047113.29 total 86.64 last 314.65) - open/close 29338 (time 1781047123.29 total 96.64 last 414.58) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 103.99 seconds: 313.44 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 6829 (time 1781047142.18 total 10.00 last 682.86) - /unlink 10000 (time 1781047146.29 total 14.12 last 769.95) - /unlink 16985 (time 1781047156.29 total 24.12 last 698.48) - /unlink 20000 (time 1781047161.07 total 28.90 last 630.87) - /unlink 25939 (time 1781047171.07 total 38.90 last 593.88) - /unlink 30000 (time 1781047177.14 total 44.97 last 668.99) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 48.42 seconds: 673.12 ops/second Waiting for MDT destroys to complete - open/close 3415 (time 1781047215.14 total 10.00 last 341.49) - open/close 7390 (time 1781047225.14 total 20.00 last 397.47) - open/close 10000 (time 1781047232.06 total 26.92 last 377.26) - open/close 13097 (time 1781047242.06 total 36.92 last 309.69) - open/close 16778 (time 1781047252.06 total 46.92 last 368.03) - open/close 20000 (time 1781047260.99 total 55.85 last 360.98) - open/close 23734 (time 1781047270.99 total 65.85 last 373.40) - open/close 27650 (time 1781047280.99 total 75.85 last 391.54) - open/close 30000 (time 1781047285.40 total 80.26 last 532.71) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 85.67 seconds: 380.48 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 8033 (time 1781047302.10 total 10.00 last 803.28) - /unlink 10000 (time 1781047304.99 total 12.89 last 681.52) - /unlink 17963 (time 1781047314.99 total 22.89 last 796.28) - /unlink 20000 (time 1781047317.67 total 25.57 last 758.59) - /unlink 27573 (time 1781047327.67 total 35.57 last 757.23) - /unlink 30000 (time 1781047330.49 total 38.39 last 860.56) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 41.79 seconds: 779.92 ops/second Waiting for MDT destroys to complete - open/close 4133 (time 1781047364.64 total 10.00 last 413.27) - open/close 7735 (time 1781047374.64 total 20.00 last 360.18) - open/close 10000 (time 1781047380.38 total 25.74 last 394.71) - open/close 14713 (time 1781047390.38 total 35.74 last 471.24) - open/close 19209 (time 1781047400.39 total 45.74 last 449.56) - open/close 23527 (time 1781047410.39 total 55.74 last 431.77) - open/close 27969 (time 1781047420.39 total 65.74 last 444.19) - open/close 30000 (time 1781047424.42 total 69.77 last 503.79) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 75.28 seconds: 432.97 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 7439 (time 1781047441.27 total 10.00 last 743.88) - /unlink 10000 (time 1781047444.66 total 13.40 last 754.01) - /unlink 18216 (time 1781047454.66 total 23.40 last 821.59) - /unlink 20000 (time 1781047457.04 total 25.77 last 751.36) - /unlink 27784 (time 1781047467.04 total 35.78 last 778.11) - /unlink 30000 (time 1781047469.76 total 38.49 last 814.88) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 41.58 seconds: 783.90 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg212-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg212-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg212-server stop mds service on oleg212-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg212-server stop mds service on oleg212-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg212-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg212-server' oleg212-server: oleg212-server.virtnet: executing unload_modules_local oleg212-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg212-server: LNET ready to unload oleg212-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg212-client: oleg212-client: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-client: ssh exited with exit code 2 pdsh@oleg212-client: oleg212-server: ssh exited with exit code 2