== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:09:02 (1786979342) Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg419-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg419-server oleg419-server: oleg419-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg419-server: detected 4 online CPUs by sysfs oleg419-server: MODOPTS_LIBCFS= oleg419-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg419-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg419-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg419-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 846 (time 1786979543.26 total 10.01 last 84.56) - open/close 1667 (time 1786979553.26 total 20.01 last 82.09) - open/close 2436 (time 1786979563.28 total 30.03 last 76.74) - open/close 3156 (time 1786979573.29 total 40.04 last 71.94) - open/close 4028 (time 1786979583.30 total 50.05 last 87.12) total: 4500 open/close in 56.11 seconds: 80.20 ops/second - /unlink 1560 (time 1786979603.57 total 10.00 last 155.97) - /unlink 3108 (time 1786979613.57 total 20.00 last 154.76) total: 4500 /unlink in 29.83 seconds: 150.88 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 716 (time 1786979638.00 total 10.00 last 71.59) - open/close 1561 (time 1786979648.00 total 20.01 last 84.45) - open/close 2462 (time 1786979658.01 total 30.01 last 90.06) - open/close 3214 (time 1786979668.01 total 40.01 last 75.20) - open/close 4010 (time 1786979678.01 total 50.01 last 79.59) total: 4500 open/close in 56.24 seconds: 80.01 ops/second - /unlink 1380 (time 1786979698.59 total 10.01 last 137.88) - /unlink 2940 (time 1786979708.59 total 20.01 last 155.90) - /unlink 4355 (time 1786979718.59 total 30.02 last 141.48) total: 4500 /unlink in 31.20 seconds: 144.23 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 800 (time 1786979733.88 total 10.00 last 79.98) - open/close 1512 (time 1786979743.88 total 20.01 last 71.17) - open/close 2285 (time 1786979753.89 total 30.01 last 77.27) - open/close 3031 (time 1786979763.89 total 40.02 last 74.56) - open/close 3774 (time 1786979773.90 total 50.03 last 74.24) total: 4500 open/close in 59.30 seconds: 75.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1537 (time 1786979797.58 total 10.00 last 153.70) - /unlink 3024 (time 1786979807.58 total 20.01 last 148.62) - /unlink 4472 (time 1786979817.59 total 30.01 last 144.76) total: 4500 /unlink in 30.36 seconds: 148.22 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 783 (time 1786979832.81 total 10.01 last 78.24) - open/close 1436 (time 1786979842.82 total 20.01 last 65.25) - open/close 2175 (time 1786979852.82 total 30.02 last 73.86) - open/close 3034 (time 1786979862.84 total 40.03 last 85.80) - open/close 3758 (time 1786979872.84 total 50.04 last 72.33) total: 4500 open/close in 59.68 seconds: 75.40 ops/second - /unlink 1541 (time 1786979897.39 total 10.02 last 153.84) - /unlink 3003 (time 1786979907.39 total 20.02 last 146.20) - /unlink 4384 (time 1786979917.39 total 30.02 last 138.10) total: 4500 /unlink in 31.13 seconds: 144.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 740 (time 1786979932.81 total 10.03 last 73.74) - open/close 1492 (time 1786979942.82 total 20.04 last 75.17) - open/close 2247 (time 1786979952.84 total 30.06 last 75.34) - open/close 3041 (time 1786979962.84 total 40.06 last 79.37) - open/close 3775 (time 1786979972.86 total 50.08 last 73.31) total: 4500 open/close in 59.31 seconds: 75.87 ops/second - /unlink 1445 (time 1786979998.22 total 10.00 last 144.49) - /unlink 2635 (time 1786980008.23 total 20.01 last 118.93) - /unlink 4090 (time 1786980018.23 total 30.01 last 145.49) total: 4500 /unlink in 33.43 seconds: 134.63 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 708 (time 1786980037.40 total 10.02 last 70.65) - open/close 1458 (time 1786980047.41 total 20.03 last 74.94) - open/close 2209 (time 1786980057.41 total 30.03 last 75.07) - open/close 2961 (time 1786980067.43 total 40.05 last 75.07) - open/close 3719 (time 1786980077.43 total 50.05 last 75.78) - open/close 4378 (time 1786980087.44 total 60.06 last 65.83) total: 4500 open/close in 61.63 seconds: 73.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1366 (time 1786980104.21 total 10.00 last 136.60) - /unlink 2719 (time 1786980114.21 total 20.00 last 135.30) - /unlink 4097 (time 1786980124.22 total 30.00 last 137.74) total: 4500 /unlink in 32.80 seconds: 137.19 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 801 (time 1786980141.74 total 10.01 last 80.00) - open/close 1633 (time 1786980151.74 total 20.01 last 83.19) - open/close 2336 (time 1786980161.74 total 30.02 last 70.26) - open/close 3060 (time 1786980171.75 total 40.03 last 72.37) - open/close 3582 (time 1786980181.75 total 50.03 last 52.17) - open/close 4055 (time 1786980191.78 total 60.05 last 47.19) total: 4500 open/close in 68.63 seconds: 65.57 ops/second - /unlink 1469 (time 1786980216.33 total 10.00 last 146.90) - /unlink 2813 (time 1786980226.33 total 20.00 last 134.39) - /unlink 4260 (time 1786980236.34 total 30.01 last 144.63) total: 4500 /unlink in 32.18 seconds: 139.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 776 (time 1786980253.85 total 10.00 last 77.60) - open/close 1567 (time 1786980263.85 total 20.01 last 79.05) - open/close 2368 (time 1786980273.85 total 30.01 last 80.09) - open/close 3099 (time 1786980283.86 total 40.01 last 73.07) - open/close 3863 (time 1786980293.87 total 50.03 last 76.28) total: 4500 open/close in 57.41 seconds: 78.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1358 (time 1786980316.41 total 10.00 last 135.74) - /unlink 2764 (time 1786980326.42 total 20.01 last 140.52) - /unlink 4270 (time 1786980336.42 total 30.01 last 150.56) total: 4500 /unlink in 31.82 seconds: 141.40 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 720 (time 1786980352.87 total 10.01 last 71.96) - open/close 1448 (time 1786980362.88 total 20.01 last 72.76) - open/close 2100 (time 1786980372.91 total 30.04 last 64.99) - open/close 2665 (time 1786980382.92 total 40.06 last 56.43) - open/close 3235 (time 1786980392.93 total 50.07 last 56.95) - open/close 3964 (time 1786980402.93 total 60.07 last 72.87) total: 4500 open/close in 66.92 seconds: 67.25 ops/second - /unlink 1344 (time 1786980424.69 total 10.01 last 134.23) - /unlink 2690 (time 1786980434.69 total 20.01 last 134.59) - /unlink 4035 (time 1786980444.70 total 30.02 last 134.39) total: 4500 /unlink in 33.38 seconds: 134.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 676 (time 1786980463.43 total 10.01 last 67.56) - open/close 1243 (time 1786980473.43 total 20.01 last 56.68) - open/close 1789 (time 1786980483.43 total 30.01 last 54.57) - open/close 2504 (time 1786980493.45 total 40.03 last 71.38) - open/close 3155 (time 1786980503.50 total 50.08 last 64.79) - open/close 3882 (time 1786980513.50 total 60.08 last 72.68) total: 4500 open/close in 68.86 seconds: 65.35 ops/second - /unlink 1279 (time 1786980536.38 total 10.00 last 127.87) - /unlink 2664 (time 1786980546.39 total 20.01 last 138.37) - /unlink 3845 (time 1786980556.40 total 30.02 last 117.99) total: 4500 /unlink in 34.93 seconds: 128.82 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 693 (time 1786980575.61 total 10.01 last 69.24) - open/close 1378 (time 1786980585.62 total 20.01 last 68.49) - open/close 2068 (time 1786980595.62 total 30.02 last 68.96) - open/close 2765 (time 1786980605.64 total 40.03 last 69.59) - open/close 3390 (time 1786980615.64 total 50.04 last 62.47) - open/close 4123 (time 1786980625.66 total 60.05 last 73.19) total: 4500 open/close in 65.92 seconds: 68.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1386 (time 1786980646.69 total 10.00 last 138.54) - /unlink 2765 (time 1786980656.69 total 20.01 last 137.84) - /unlink 4107 (time 1786980666.69 total 30.01 last 134.20) total: 4500 /unlink in 32.72 seconds: 137.54 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 767 (time 1786980684.01 total 10.02 last 76.55) - open/close 1511 (time 1786980694.01 total 20.02 last 74.39) - open/close 2326 (time 1786980704.02 total 30.03 last 81.43) - open/close 3092 (time 1786980714.03 total 40.04 last 76.54) - open/close 3798 (time 1786980724.04 total 50.05 last 70.52) total: 4500 open/close in 58.65 seconds: 76.73 ops/second - /unlink 1474 (time 1786980747.30 total 10.00 last 147.40) - /unlink 2862 (time 1786980757.30 total 20.00 last 138.79) - /unlink 4330 (time 1786980767.30 total 30.00 last 146.78) total: 4500 /unlink in 31.29 seconds: 143.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 787 (time 1786980783.33 total 10.01 last 78.61) - open/close 1531 (time 1786980793.34 total 20.02 last 74.35) - open/close 2283 (time 1786980803.34 total 30.02 last 75.17) - open/close 2979 (time 1786980813.35 total 40.03 last 69.53) - open/close 3614 (time 1786980823.36 total 50.05 last 63.41) - open/close 4375 (time 1786980833.41 total 60.09 last 75.78) total: 4500 open/close in 61.65 seconds: 73.00 ops/second - /unlink 1412 (time 1786980849.71 total 10.01 last 141.10) - /unlink 2733 (time 1786980859.71 total 20.01 last 132.08) - /unlink 4167 (time 1786980869.71 total 30.01 last 143.38) total: 4500 /unlink in 32.58 seconds: 138.13 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 781 (time 1786980887.51 total 10.01 last 78.04) - open/close 1520 (time 1786980897.52 total 20.01 last 73.85) - open/close 2310 (time 1786980907.52 total 30.02 last 79.00) - open/close 3127 (time 1786980917.53 total 40.03 last 81.60) - open/close 3849 (time 1786980927.53 total 50.03 last 72.17) total: 4500 open/close in 59.05 seconds: 76.21 ops/second - /unlink 1377 (time 1786980951.14 total 10.01 last 137.62) - /unlink 2824 (time 1786980961.15 total 20.01 last 144.66) - /unlink 4205 (time 1786980971.15 total 30.01 last 138.05) total: 4500 /unlink in 31.92 seconds: 140.96 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 751 (time 1786980987.58 total 10.00 last 75.09) - open/close 1464 (time 1786980997.59 total 20.02 last 71.19) - open/close 2248 (time 1786981007.60 total 30.02 last 78.37) - open/close 2975 (time 1786981017.60 total 40.02 last 72.69) - open/close 3766 (time 1786981027.61 total 50.03 last 79.02) total: 4500 open/close in 59.80 seconds: 75.25 ops/second - /unlink 1493 (time 1786981052.32 total 10.00 last 149.27) - /unlink 2899 (time 1786981062.33 total 20.01 last 140.43) - /unlink 4249 (time 1786981072.33 total 30.02 last 134.92) total: 4500 /unlink in 31.93 seconds: 140.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 700 (time 1786981090.01 total 10.01 last 69.92) - open/close 1385 (time 1786981100.01 total 20.02 last 68.47) - open/close 2186 (time 1786981110.04 total 30.04 last 79.91) - open/close 2961 (time 1786981120.05 total 40.05 last 77.43) - open/close 3720 (time 1786981130.07 total 50.07 last 75.71) - open/close 4492 (time 1786981140.08 total 60.08 last 77.15) total: 4500 open/close in 60.16 seconds: 74.80 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1422 (time 1786981154.58 total 10.01 last 142.08) - /unlink 2781 (time 1786981164.58 total 20.01 last 135.90) - /unlink 4114 (time 1786981174.58 total 30.01 last 133.26) total: 4500 /unlink in 32.83 seconds: 137.06 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 811 (time 1786981192.19 total 10.01 last 80.99) - open/close 1538 (time 1786981202.20 total 20.03 last 72.61) - open/close 2168 (time 1786981212.23 total 30.05 last 62.86) - open/close 2828 (time 1786981222.23 total 40.05 last 65.97) - open/close 3553 (time 1786981232.24 total 50.06 last 72.45) - open/close 4282 (time 1786981242.24 total 60.06 last 72.89) total: 4500 open/close in 63.08 seconds: 71.34 ops/second - /unlink 1290 (time 1786981259.86 total 10.00 last 128.95) - /unlink 2448 (time 1786981269.86 total 20.01 last 115.78) - /unlink 3757 (time 1786981279.86 total 30.01 last 130.88) total: 4500 /unlink in 35.00 seconds: 128.56 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 726 (time 1786981298.94 total 10.00 last 72.56) - open/close 1474 (time 1786981308.94 total 20.01 last 74.75) - open/close 2132 (time 1786981318.95 total 30.02 last 65.73) - open/close 2828 (time 1786981328.96 total 40.03 last 69.57) - open/close 3505 (time 1786981338.98 total 50.05 last 67.57) - open/close 4270 (time 1786981348.98 total 60.05 last 76.47) total: 4500 open/close in 62.97 seconds: 71.47 ops/second - /unlink 1377 (time 1786981365.90 total 10.00 last 137.66) - /unlink 2730 (time 1786981375.91 total 20.01 last 135.19) - /unlink 4159 (time 1786981385.92 total 30.02 last 142.80) total: 4500 /unlink in 32.58 seconds: 138.12 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11316: 95265 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg419-server unloading modules via unload_modules_local on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local oleg419-server: modules unloaded.