== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 14:51:07 (1787856667) Stopping clients: oleg310-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg310-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg310-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg310-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg310-server oleg310-server: oleg310-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing load_modules_local oleg310-server: Loading modules from /home/green/git/lustre-release/lustre oleg310-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg310-server: detected 4 online CPUs by sysfs oleg310-server: MODOPTS_LIBCFS= oleg310-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg310-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg310-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg310-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 621 (time 1787856883.53 total 10.02 last 62.00) - open/close 1254 (time 1787856893.54 total 20.02 last 63.25) - open/close 1943 (time 1787856903.55 total 30.03 last 68.85) - open/close 2571 (time 1787856913.55 total 40.03 last 62.79) - open/close 3168 (time 1787856923.55 total 50.03 last 59.69) - open/close 3787 (time 1787856933.55 total 60.04 last 61.89) - open/close 4346 (time 1787856943.56 total 70.04 last 55.87) total: 4500 open/close in 72.23 seconds: 62.30 ops/second - /unlink 1404 (time 1787856961.99 total 10.00 last 140.34) - /unlink 2644 (time 1787856971.99 total 20.01 last 123.90) - /unlink 3763 (time 1787856982.00 total 30.01 last 111.87) total: 4500 /unlink in 36.72 seconds: 122.57 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 738 (time 1787857004.07 total 10.01 last 73.75) - open/close 1363 (time 1787857014.08 total 20.01 last 62.47) - open/close 2073 (time 1787857024.09 total 30.02 last 70.94) - open/close 2757 (time 1787857034.09 total 40.03 last 68.36) - open/close 3509 (time 1787857044.10 total 50.03 last 75.18) - open/close 4217 (time 1787857054.11 total 60.04 last 70.72) total: 4500 open/close in 64.04 seconds: 70.27 ops/second - /unlink 1344 (time 1787857072.32 total 10.01 last 134.28) - /unlink 2674 (time 1787857082.34 total 20.02 last 132.85) - /unlink 3933 (time 1787857092.35 total 30.03 last 125.78) total: 4500 /unlink in 34.45 seconds: 130.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 643 (time 1787857111.65 total 10.00 last 64.28) - open/close 1313 (time 1787857121.66 total 20.01 last 66.97) - open/close 2004 (time 1787857131.67 total 30.02 last 69.03) - open/close 2650 (time 1787857141.67 total 40.02 last 64.56) - open/close 3280 (time 1787857151.70 total 50.05 last 62.85) - open/close 3837 (time 1787857161.72 total 60.07 last 55.57) - open/close 4446 (time 1787857171.73 total 70.08 last 60.83) total: 4500 open/close in 70.89 seconds: 63.48 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1185 (time 1787857187.74 total 10.02 last 118.30) - /unlink 2262 (time 1787857197.75 total 20.02 last 107.62) - /unlink 3365 (time 1787857207.75 total 30.02 last 110.30) total: 4500 /unlink in 39.40 seconds: 114.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 686 (time 1787857231.66 total 10.00 last 68.57) - open/close 1391 (time 1787857241.67 total 20.01 last 70.48) - open/close 2096 (time 1787857251.68 total 30.02 last 70.40) - open/close 2773 (time 1787857261.68 total 40.02 last 67.68) - open/close 3433 (time 1787857271.69 total 50.03 last 65.93) - open/close 4048 (time 1787857281.69 total 60.04 last 61.49) total: 4500 open/close in 66.83 seconds: 67.33 ops/second - /unlink 1311 (time 1787857302.69 total 10.01 last 130.98) - /unlink 2552 (time 1787857312.69 total 20.01 last 124.08) - /unlink 3806 (time 1787857322.70 total 30.02 last 125.30) total: 4500 /unlink in 35.26 seconds: 127.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 662 (time 1787857342.78 total 10.01 last 66.14) - open/close 1316 (time 1787857352.78 total 20.02 last 65.36) - open/close 2041 (time 1787857362.80 total 30.03 last 72.39) - open/close 2779 (time 1787857372.81 total 40.04 last 73.71) - open/close 3453 (time 1787857382.82 total 50.05 last 67.34) - open/close 4103 (time 1787857392.83 total 60.06 last 64.96) total: 4500 open/close in 66.34 seconds: 67.83 ops/second - /unlink 1333 (time 1787857413.27 total 10.01 last 133.22) - /unlink 2462 (time 1787857423.27 total 20.01 last 112.86) - /unlink 3615 (time 1787857433.28 total 30.02 last 115.22) total: 4500 /unlink in 36.91 seconds: 121.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 719 (time 1787857454.55 total 10.00 last 71.89) - open/close 1486 (time 1787857464.55 total 20.00 last 76.70) - open/close 2204 (time 1787857474.56 total 30.01 last 71.78) - open/close 2900 (time 1787857484.57 total 40.01 last 69.54) - open/close 3604 (time 1787857494.57 total 50.01 last 70.39) - open/close 4309 (time 1787857504.57 total 60.02 last 70.49) total: 4500 open/close in 62.88 seconds: 71.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1071 (time 1787857521.92 total 10.00 last 107.05) - /unlink 2170 (time 1787857531.92 total 20.01 last 109.86) - /unlink 3431 (time 1787857541.93 total 30.02 last 126.01) total: 4500 /unlink in 38.85 seconds: 115.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 708 (time 1787857566.32 total 10.01 last 70.72) - open/close 1471 (time 1787857576.33 total 20.03 last 76.19) - open/close 2167 (time 1787857586.33 total 30.03 last 69.58) - open/close 2878 (time 1787857596.35 total 40.04 last 71.02) - open/close 3593 (time 1787857606.35 total 50.04 last 71.47) - open/close 4348 (time 1787857616.37 total 60.06 last 75.34) total: 4500 open/close in 61.97 seconds: 72.62 ops/second - /unlink 1323 (time 1787857633.13 total 10.02 last 132.08) - /unlink 2616 (time 1787857643.13 total 20.02 last 129.25) - /unlink 3987 (time 1787857653.14 total 30.03 last 137.00) total: 4500 /unlink in 33.89 seconds: 132.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 739 (time 1787857671.81 total 10.01 last 73.85) - open/close 1475 (time 1787857681.82 total 20.02 last 73.50) - open/close 2222 (time 1787857691.83 total 30.03 last 74.65) - open/close 2966 (time 1787857701.84 total 40.04 last 74.33) - open/close 3657 (time 1787857711.84 total 50.04 last 69.05) - open/close 4407 (time 1787857721.85 total 60.05 last 74.97) total: 4500 open/close in 61.26 seconds: 73.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1419 (time 1787857737.06 total 10.01 last 141.73) - /unlink 2783 (time 1787857747.06 total 20.01 last 136.39) - /unlink 4124 (time 1787857757.06 total 30.02 last 134.06) total: 4500 /unlink in 33.18 seconds: 135.62 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 677 (time 1787857776.00 total 10.00 last 67.67) - open/close 1328 (time 1787857786.00 total 20.00 last 65.09) - open/close 2050 (time 1787857796.02 total 30.03 last 72.05) - open/close 2707 (time 1787857806.03 total 40.03 last 65.65) - open/close 3370 (time 1787857816.04 total 50.04 last 66.26) - open/close 4088 (time 1787857826.04 total 60.05 last 71.76) total: 4500 open/close in 66.84 seconds: 67.33 ops/second - /unlink 1311 (time 1787857847.91 total 10.01 last 131.01) - /unlink 2582 (time 1787857857.91 total 20.01 last 127.09) - /unlink 3841 (time 1787857867.91 total 30.01 last 125.87) total: 4500 /unlink in 35.07 seconds: 128.31 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 594 (time 1787857888.28 total 10.01 last 59.35) - open/close 1190 (time 1787857898.32 total 20.05 last 59.33) - open/close 1838 (time 1787857908.33 total 30.06 last 64.77) - open/close 2450 (time 1787857918.35 total 40.09 last 61.04) - open/close 3132 (time 1787857928.36 total 50.09 last 68.15) - open/close 3845 (time 1787857938.37 total 60.10 last 71.26) total: 4500 open/close in 69.42 seconds: 64.83 ops/second - /unlink 1198 (time 1787857962.81 total 10.00 last 119.75) - /unlink 2471 (time 1787857972.81 total 20.00 last 127.29) - /unlink 3688 (time 1787857982.82 total 30.01 last 121.64) total: 4500 /unlink in 36.63 seconds: 122.84 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 709 (time 1787858004.51 total 10.01 last 70.84) - open/close 1326 (time 1787858014.52 total 20.02 last 61.65) - open/close 2044 (time 1787858024.52 total 30.02 last 71.80) - open/close 2693 (time 1787858034.53 total 40.03 last 64.83) - open/close 3292 (time 1787858044.54 total 50.04 last 59.85) - open/close 3968 (time 1787858054.54 total 60.04 last 67.57) total: 4500 open/close in 67.80 seconds: 66.37 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1127 (time 1787858077.43 total 10.02 last 112.52) - /unlink 2367 (time 1787858087.43 total 20.02 last 123.99) - /unlink 3775 (time 1787858097.45 total 30.03 last 140.65) total: 4500 /unlink in 36.72 seconds: 122.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 652 (time 1787858119.01 total 10.00 last 65.20) - open/close 1338 (time 1787858129.02 total 20.02 last 68.50) - open/close 2045 (time 1787858139.03 total 30.02 last 70.67) - open/close 2736 (time 1787858149.03 total 40.02 last 69.07) - open/close 3404 (time 1787858159.05 total 50.04 last 66.71) - open/close 4077 (time 1787858169.08 total 60.07 last 67.10) total: 4500 open/close in 66.71 seconds: 67.45 ops/second - /unlink 1371 (time 1787858191.50 total 10.01 last 137.00) - /unlink 2707 (time 1787858201.50 total 20.01 last 133.59) - /unlink 4081 (time 1787858211.50 total 30.01 last 137.33) total: 4500 /unlink in 33.07 seconds: 136.10 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 717 (time 1787858229.35 total 10.00 last 71.68) - open/close 1502 (time 1787858239.37 total 20.02 last 78.35) - open/close 2253 (time 1787858249.39 total 30.04 last 74.95) - open/close 2960 (time 1787858259.40 total 40.05 last 70.68) - open/close 3679 (time 1787858269.40 total 50.05 last 71.86) - open/close 4378 (time 1787858279.41 total 60.06 last 69.86) total: 4500 open/close in 61.65 seconds: 72.99 ops/second - /unlink 1386 (time 1787858295.05 total 10.00 last 138.60) - /unlink 2725 (time 1787858305.07 total 20.01 last 133.73) - /unlink 4052 (time 1787858315.07 total 30.02 last 132.64) total: 4500 /unlink in 33.42 seconds: 134.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 717 (time 1787858333.30 total 10.00 last 71.68) - open/close 1413 (time 1787858343.30 total 20.01 last 69.59) - open/close 2124 (time 1787858353.32 total 30.02 last 71.01) - open/close 2841 (time 1787858363.33 total 40.03 last 71.58) - open/close 3577 (time 1787858373.34 total 50.04 last 73.54) - open/close 4195 (time 1787858383.36 total 60.06 last 61.68) total: 4500 open/close in 63.68 seconds: 70.67 ops/second - /unlink 1315 (time 1787858401.21 total 10.01 last 131.41) - /unlink 2733 (time 1787858411.22 total 20.01 last 141.74) - /unlink 4076 (time 1787858421.22 total 30.01 last 134.25) total: 4500 /unlink in 33.19 seconds: 135.59 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 716 (time 1787858439.40 total 10.04 last 71.35) - open/close 1392 (time 1787858449.41 total 20.04 last 67.58) - open/close 2023 (time 1787858459.44 total 30.07 last 62.91) - open/close 2660 (time 1787858469.44 total 40.07 last 63.67) - open/close 3344 (time 1787858479.45 total 50.08 last 68.34) - open/close 3974 (time 1787858489.46 total 60.09 last 62.95) total: 4500 open/close in 68.38 seconds: 65.81 ops/second - /unlink 1423 (time 1787858512.54 total 10.01 last 142.22) - /unlink 2796 (time 1787858522.54 total 20.01 last 137.25) - /unlink 4141 (time 1787858532.54 total 30.01 last 134.47) total: 4500 /unlink in 32.49 seconds: 138.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 727 (time 1787858548.90 total 10.01 last 72.60) - open/close 1393 (time 1787858558.91 total 20.03 last 66.52) - open/close 2044 (time 1787858568.92 total 30.04 last 65.01) - open/close 2759 (time 1787858578.94 total 40.06 last 71.36) - open/close 3409 (time 1787858588.96 total 50.07 last 64.93) - open/close 4088 (time 1787858598.96 total 60.07 last 67.90) total: 4500 open/close in 66.01 seconds: 68.17 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1297 (time 1787858618.82 total 10.01 last 129.57) - /unlink 2635 (time 1787858628.83 total 20.02 last 133.63) - /unlink 3828 (time 1787858638.83 total 30.03 last 119.26) total: 4500 /unlink in 36.58 seconds: 123.01 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 739 (time 1787858661.01 total 10.01 last 73.81) - open/close 1426 (time 1787858671.02 total 20.02 last 68.66) - open/close 2146 (time 1787858681.03 total 30.02 last 71.96) - open/close 2850 (time 1787858691.03 total 40.03 last 70.35) - open/close 3532 (time 1787858701.04 total 50.04 last 68.16) - open/close 4118 (time 1787858711.05 total 60.04 last 58.56) total: 4500 open/close in 66.13 seconds: 68.05 ops/second - /unlink 1308 (time 1787858732.36 total 10.00 last 130.77) - /unlink 2477 (time 1787858742.36 total 20.00 last 116.89) - /unlink 3947 (time 1787858752.36 total 30.00 last 146.98) total: 4500 /unlink in 33.98 seconds: 132.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 674 (time 1787858771.02 total 10.00 last 67.37) - open/close 1417 (time 1787858781.03 total 20.01 last 74.28) - open/close 2141 (time 1787858791.05 total 30.03 last 72.27) - open/close 2862 (time 1787858801.05 total 40.03 last 72.04) - open/close 3578 (time 1787858811.07 total 50.05 last 71.50) - open/close 4290 (time 1787858821.08 total 60.06 last 71.09) total: 4500 open/close in 62.94 seconds: 71.49 ops/second - /unlink 1569 (time 1787858837.71 total 10.00 last 156.88) - /unlink 2891 (time 1787858847.71 total 20.00 last 132.17) - /unlink 4162 (time 1787858857.72 total 30.01 last 126.97) total: 4500 /unlink in 32.33 seconds: 139.18 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11369: 95357 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg310-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg310-server unloading modules via unload_modules_local on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing unload_modules_local oleg310-server: modules unloaded.