== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 21:09:25 (1767319765) Stopping clients: oleg346-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg346-client.virtnet /mnt/lustre2 (opts:-f) Stopping client oleg346-client.virtnet /mnt/lustre opts:-f Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server waited 0 for 5 ST mdt lustre-MDT0000 lustre-MDT0000_UUID 5 waited 2 for 5 ST mdt lustre-MDT0000 lustre-MDT0000_UUID 3 pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Stopping /mnt/lustre-ost2 (opts:-f) on oleg346-server pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 oleg346-server: oleg346-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing load_modules_local oleg346-server: Loading modules from /home/green/git/lustre-release/lustre oleg346-server: detected 4 online CPUs by sysfs oleg346-server: MODOPTS_LIBCFS= oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1053: echo: write error: Device or resource busy oleg346-server: Force libcfs to create 2 CPU partitions oleg346-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1054: echo: write error: Device or resource busy Formatting mgs, mds, osts Format ost1: /dev/vde Format mds1: /dev/vdc Format mds2: /dev/vdd Format ost2: /dev/vdf start mds service on oleg346-server pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Commit the device label on /dev/vdc Started lustre-MDT0000 start mds service on oleg346-server pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Commit the device label on /dev/vdd Started lustre-MDT0001 oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg346-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg346-server pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg346-server: oleg346-server.virtnet: executing set_default_debug -1 all pdsh@oleg346-client: oleg346-server: ssh exited with exit code 1 Commit the device label on /dev/vde Started lustre-OST0000 oleg346-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg346-client.virtnet: -o user_xattr,flock 192.168.203.146@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 896 (time 1767320048.31 total 10.01 last 89.52) - open/close 1770 (time 1767320058.33 total 20.02 last 87.26) - open/close 2576 (time 1767320068.34 total 30.03 last 80.55) - open/close 3390 (time 1767320078.36 total 40.06 last 81.17) - open/close 4237 (time 1767320088.36 total 50.06 last 84.70) total: 4500 open/close in 53.55 seconds: 84.04 ops/second - /unlink 1460 (time 1767320107.11 total 10.00 last 145.97) - /unlink 2861 (time 1767320117.11 total 20.01 last 140.00) - /unlink 4490 (time 1767320127.12 total 30.02 last 162.81) total: 4500 /unlink in 30.09 seconds: 149.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 806 (time 1767320141.72 total 10.03 last 80.39) - open/close 1609 (time 1767320151.73 total 20.03 last 80.26) - open/close 2329 (time 1767320161.73 total 30.04 last 71.97) - open/close 3170 (time 1767320171.73 total 40.04 last 84.06) - open/close 3926 (time 1767320181.74 total 50.04 last 75.57) total: 4500 open/close in 57.05 seconds: 78.87 ops/second - /unlink 1453 (time 1767320204.06 total 10.00 last 145.25) - /unlink 2946 (time 1767320214.06 total 20.00 last 149.29) - /unlink 4401 (time 1767320224.07 total 30.01 last 145.43) total: 4500 /unlink in 30.69 seconds: 146.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 627 (time 1767320239.72 total 10.00 last 62.69) - open/close 1303 (time 1767320249.73 total 20.01 last 67.53) - open/close 2053 (time 1767320259.74 total 30.02 last 74.95) - open/close 2888 (time 1767320269.76 total 40.04 last 83.30) - open/close 3688 (time 1767320279.77 total 50.05 last 79.94) - open/close 4416 (time 1767320289.77 total 60.05 last 72.78) total: 4500 open/close in 61.29 seconds: 73.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1478 (time 1767320305.35 total 10.00 last 147.78) - /unlink 3083 (time 1767320315.36 total 20.01 last 160.31) - /unlink 4452 (time 1767320325.37 total 30.02 last 136.85) total: 4500 /unlink in 30.48 seconds: 147.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 857 (time 1767320340.51 total 10.01 last 85.66) - open/close 1640 (time 1767320350.52 total 20.01 last 78.25) - open/close 2526 (time 1767320360.53 total 30.02 last 88.49) - open/close 3431 (time 1767320370.54 total 40.03 last 90.43) - open/close 4231 (time 1767320380.54 total 50.04 last 79.95) total: 4500 open/close in 52.60 seconds: 85.56 ops/second - /unlink 1651 (time 1767320398.13 total 10.00 last 165.05) - /unlink 3243 (time 1767320408.13 total 20.01 last 159.16) total: 4500 /unlink in 29.03 seconds: 154.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 549 (time 1767320431.49 total 10.00 last 54.88) - open/close 1477 (time 1767320441.51 total 20.03 last 92.58) - open/close 2313 (time 1767320451.51 total 30.03 last 83.59) - open/close 3096 (time 1767320461.52 total 40.04 last 78.23) - open/close 4011 (time 1767320471.53 total 50.05 last 91.36) total: 4500 open/close in 55.36 seconds: 81.29 ops/second - /unlink 1456 (time 1767320490.94 total 10.02 last 145.34) - /unlink 2608 (time 1767320500.95 total 20.03 last 115.09) - /unlink 4271 (time 1767320510.95 total 30.03 last 166.23) total: 4500 /unlink in 31.51 seconds: 142.83 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 968 (time 1767320527.59 total 10.01 last 96.68) - open/close 1721 (time 1767320537.60 total 20.02 last 75.24) - open/close 2521 (time 1767320547.61 total 30.03 last 79.89) - open/close 3299 (time 1767320557.61 total 40.04 last 77.78) - open/close 4061 (time 1767320567.62 total 50.04 last 76.14) total: 4500 open/close in 55.05 seconds: 81.74 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1595 (time 1767320586.93 total 10.00 last 159.47) - /unlink 3045 (time 1767320596.93 total 20.01 last 144.90) total: 4500 /unlink in 28.75 seconds: 156.53 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 847 (time 1767320620.75 total 10.00 last 84.66) - open/close 1860 (time 1767320630.77 total 20.02 last 101.13) - open/close 2826 (time 1767320640.78 total 30.03 last 96.53) - open/close 3672 (time 1767320650.80 total 40.05 last 84.44) total: 4500 open/close in 49.84 seconds: 90.29 ops/second - /unlink 1558 (time 1767320675.30 total 10.00 last 155.77) - /unlink 3110 (time 1767320685.30 total 20.00 last 155.18) total: 4500 /unlink in 29.09 seconds: 154.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 902 (time 1767320709.12 total 10.01 last 90.13) - open/close 1761 (time 1767320719.13 total 20.02 last 85.81) - open/close 2743 (time 1767320729.15 total 30.03 last 98.05) - open/close 3670 (time 1767320739.15 total 40.04 last 92.66) total: 4500 open/close in 49.31 seconds: 91.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1250 (time 1767320763.08 total 10.02 last 124.81) - /unlink 2965 (time 1767320773.09 total 20.02 last 171.39) total: 4500 /unlink in 28.98 seconds: 155.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 838 (time 1767320797.34 total 10.01 last 83.68) - open/close 1734 (time 1767320807.35 total 20.02 last 89.54) - open/close 2585 (time 1767320817.35 total 30.02 last 85.07) - open/close 3382 (time 1767320827.35 total 40.03 last 79.69) - open/close 4254 (time 1767320837.36 total 50.03 last 87.17) total: 4500 open/close in 52.60 seconds: 85.55 ops/second - /unlink 1555 (time 1767320855.45 total 10.00 last 155.44) - /unlink 3176 (time 1767320865.45 total 20.01 last 162.07) total: 4500 /unlink in 28.45 seconds: 158.18 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 701 (time 1767320888.79 total 10.01 last 70.05) - open/close 1394 (time 1767320898.79 total 20.01 last 69.29) - open/close 2318 (time 1767320908.80 total 30.02 last 92.34) - open/close 3249 (time 1767320918.81 total 40.03 last 92.99) - open/close 4255 (time 1767320928.82 total 50.04 last 100.50) total: 4500 open/close in 52.68 seconds: 85.42 ops/second - /unlink 1707 (time 1767320944.61 total 10.00 last 170.65) - /unlink 3260 (time 1767320954.61 total 20.01 last 155.26) total: 4500 /unlink in 28.66 seconds: 157.00 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 903 (time 1767320978.20 total 10.02 last 90.11) - open/close 1754 (time 1767320988.20 total 20.02 last 85.07) - open/close 2746 (time 1767320998.20 total 30.03 last 99.18) - open/close 3639 (time 1767321008.22 total 40.04 last 89.21) - open/close 4410 (time 1767321018.22 total 50.04 last 77.09) total: 4500 open/close in 51.28 seconds: 87.75 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1595 (time 1767321034.27 total 10.00 last 159.50) - /unlink 2977 (time 1767321044.27 total 20.00 last 138.18) total: 4500 /unlink in 29.29 seconds: 153.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 1049 (time 1767321067.44 total 10.02 last 104.67) - open/close 1806 (time 1767321077.44 total 20.02 last 75.69) - open/close 2706 (time 1767321087.45 total 30.03 last 89.90) - open/close 3574 (time 1767321097.45 total 40.04 last 86.74) - open/close 4375 (time 1767321107.47 total 50.06 last 79.97) total: 4500 open/close in 51.54 seconds: 87.31 ops/second - /unlink 1525 (time 1767321122.69 total 10.01 last 152.29) - /unlink 3166 (time 1767321132.69 total 20.02 last 164.04) total: 4500 /unlink in 28.54 seconds: 157.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 965 (time 1767321155.63 total 10.01 last 96.45) - open/close 1894 (time 1767321165.63 total 20.01 last 92.88) - open/close 2840 (time 1767321175.64 total 30.02 last 94.51) - open/close 3667 (time 1767321185.64 total 40.02 last 82.68) total: 4500 open/close in 49.08 seconds: 91.68 ops/second - /unlink 1626 (time 1767321208.74 total 10.00 last 162.54) - /unlink 3243 (time 1767321218.74 total 20.01 last 161.68) total: 4500 /unlink in 27.30 seconds: 164.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 906 (time 1767321240.56 total 10.01 last 90.51) - open/close 1917 (time 1767321250.56 total 20.02 last 101.04) - open/close 2850 (time 1767321260.56 total 30.02 last 93.29) - open/close 3836 (time 1767321270.56 total 40.02 last 98.60) total: 4500 open/close in 46.77 seconds: 96.22 ops/second - /unlink 1872 (time 1767321291.69 total 10.00 last 187.20) - /unlink 3490 (time 1767321301.69 total 20.00 last 161.75) total: 4500 /unlink in 27.14 seconds: 165.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 960 (time 1767321322.82 total 10.00 last 95.97) - open/close 1889 (time 1767321332.83 total 20.01 last 92.80) - open/close 2892 (time 1767321342.84 total 30.03 last 100.16) - open/close 3865 (time 1767321352.84 total 40.03 last 97.27) total: 4500 open/close in 47.61 seconds: 94.51 ops/second - /unlink 1855 (time 1767321374.45 total 10.00 last 185.48) - /unlink 3554 (time 1767321384.45 total 20.00 last 169.88) total: 4500 /unlink in 24.61 seconds: 182.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 909 (time 1767321403.79 total 10.01 last 90.85) - open/close 1794 (time 1767321413.81 total 20.02 last 88.34) - open/close 2698 (time 1767321423.81 total 30.02 last 90.40) - open/close 3460 (time 1767321433.81 total 40.02 last 76.19) - open/close 4391 (time 1767321443.81 total 50.03 last 93.07) total: 4500 open/close in 50.98 seconds: 88.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1504 (time 1767321459.18 total 10.01 last 150.32) - /unlink 3215 (time 1767321469.18 total 20.01 last 171.07) total: 4500 /unlink in 27.61 seconds: 162.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 923 (time 1767321491.35 total 10.00 last 92.26) - open/close 1820 (time 1767321501.36 total 20.01 last 89.64) - open/close 2589 (time 1767321511.36 total 30.01 last 76.89) - open/close 3451 (time 1767321521.36 total 40.02 last 86.17) - open/close 4400 (time 1767321531.37 total 50.02 last 94.84) total: 4500 open/close in 50.99 seconds: 88.25 ops/second - /unlink 1664 (time 1767321546.15 total 10.00 last 166.33) - /unlink 2918 (time 1767321556.15 total 20.01 last 125.33) total: 4500 /unlink in 29.66 seconds: 151.73 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 927 (time 1767321580.28 total 10.01 last 92.64) - open/close 1896 (time 1767321590.29 total 20.01 last 96.87) - open/close 2762 (time 1767321600.29 total 30.02 last 86.54) - open/close 3655 (time 1767321610.30 total 40.03 last 89.19) total: 4500 open/close in 49.94 seconds: 90.10 ops/second - /unlink 1638 (time 1767321634.64 total 10.01 last 163.71) - /unlink 3059 (time 1767321644.64 total 20.01 last 142.06) total: 4500 /unlink in 29.06 seconds: 154.86 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11271: 113104 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg346-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg346-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg346-server stop mds service on oleg346-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg346-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg346-server' oleg346-server: oleg346-server.virtnet: executing unload_modules_local oleg346-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg346-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.