== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:14:15 (1759842855) Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f) oleg160-server: oleg160-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing load_modules_local oleg160-server: Loading modules from /home/green/git/lustre-release/lustre oleg160-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg160-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg160-server: detected 4 online CPUs by sysfs oleg160-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg160-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg160-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg160-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock oleg160-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 4432 (time 1759842910.08 total 10.00 last 443.16) - open/close 9267 (time 1759842920.08 total 20.00 last 483.45) - open/close 14078 (time 1759842930.08 total 30.00 last 481.01) - open/close 18815 (time 1759842940.08 total 40.01 last 473.61) - open/close 20000 (time 1759842942.50 total 42.43 last 489.34) - open/close 24804 (time 1759842952.50 total 52.43 last 480.35) - open/close 29460 (time 1759842962.51 total 62.43 last 465.47) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 69.08 seconds: 471.80 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 8368 (time 1759842980.64 total 10.00 last 836.80) - /unlink 17239 (time 1759842990.64 total 20.00 last 887.02) - /unlink 20000 (time 1759842994.00 total 23.36 last 822.62) - /unlink 28787 (time 1759843004.00 total 33.36 last 878.63) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 37.74 seconds: 863.73 ops/second Waiting for MDT destroys to complete - open/close 4493 (time 1759843048.28 total 10.00 last 449.27) - open/close 9466 (time 1759843058.28 total 20.00 last 497.29) - open/close 14215 (time 1759843068.28 total 30.00 last 474.84) - open/close 19907 (time 1759843078.28 total 40.00 last 569.19) - open/close 24717 (time 1759843088.28 total 50.00 last 480.93) - open/close 29380 (time 1759843098.29 total 60.01 last 466.21) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 66.94 seconds: 486.91 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 8181 (time 1759843116.71 total 10.00 last 818.07) - /unlink 10000 (time 1759843118.75 total 12.04 last 892.28) - /unlink 18459 (time 1759843128.75 total 22.04 last 845.84) - /unlink 26800 (time 1759843138.75 total 32.04 last 834.05) - /unlink 30000 (time 1759843142.28 total 35.57 last 907.77) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 38.49 seconds: 846.83 ops/second Waiting for MDT destroys to complete - open/close 4813 (time 1759843185.27 total 10.00 last 481.28) - open/close 9557 (time 1759843195.27 total 20.00 last 474.35) - open/close 14423 (time 1759843205.27 total 30.00 last 486.55) - open/close 19180 (time 1759843215.27 total 40.00 last 475.69) - open/close 23729 (time 1759843225.27 total 50.00 last 454.86) - open/close 28190 (time 1759843235.27 total 60.00 last 446.10) - open/close 30000 (time 1759843239.50 total 64.23 last 428.24) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 69.91 seconds: 466.25 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 8917 (time 1759843256.59 total 10.00 last 891.64) - /unlink 17805 (time 1759843266.59 total 20.00 last 888.76) - /unlink 20000 (time 1759843269.84 total 23.25 last 675.42) - /unlink 29177 (time 1759843279.84 total 33.25 last 917.64) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 37.06 seconds: 879.46 ops/second Waiting for MDT destroys to complete - open/close 4500 (time 1759843317.99 total 10.00 last 449.99) - open/close 9013 (time 1759843327.99 total 20.00 last 451.23) - open/close 10000 (time 1759843330.95 total 22.97 last 332.79) - open/close 14692 (time 1759843340.95 total 32.97 last 469.15) - open/close 18781 (time 1759843350.95 total 42.97 last 408.87) - open/close 20000 (time 1759843353.39 total 45.41 last 499.97) - open/close 24892 (time 1759843363.39 total 55.41 last 489.11) - open/close 29446 (time 1759843373.40 total 65.41 last 455.33) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 72.52 seconds: 449.47 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 8768 (time 1759843391.93 total 10.00 last 876.75) - /unlink 17579 (time 1759843401.93 total 20.00 last 881.10) - /unlink 20000 (time 1759843404.50 total 22.58 last 940.21) - /unlink 29292 (time 1759843414.50 total 32.58 last 929.19) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 36.51 seconds: 892.83 ops/second Waiting for MDT destroys to complete - open/close 4786 (time 1759843481.99 total 10.00 last 478.52) - open/close 9637 (time 1759843491.99 total 20.00 last 485.03) - open/close 14481 (time 1759843501.99 total 30.00 last 484.34) - open/close 19390 (time 1759843511.99 total 40.01 last 490.86) - open/close 24277 (time 1759843521.99 total 50.01 last 488.67) - open/close 29072 (time 1759843531.99 total 60.01 last 479.46) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 67.31 seconds: 484.20 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 8931 (time 1759843550.66 total 10.00 last 893.08) - /unlink 17174 (time 1759843560.66 total 20.00 last 824.28) - /unlink 20000 (time 1759843564.33 total 23.66 last 771.17) - /unlink 27943 (time 1759843574.33 total 33.67 last 794.29) - /unlink 30000 (time 1759843576.74 total 36.08 last 850.78) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 38.90 seconds: 837.95 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg160-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing unload_modules_local oleg160-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg160-server: LNET ready to unload oleg160-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2