== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:41:16 (1781224876) Stopping clients: oleg248-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg248-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg248-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg248-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg248-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg248-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg248-server oleg248-server: oleg248-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing load_modules_local oleg248-server: Loading modules from /home/green/git/lustre-release/lustre oleg248-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg248-server: detected 4 online CPUs by sysfs oleg248-server: MODOPTS_LIBCFS= oleg248-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg248-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg248-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg248-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H all_char /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 610 (time 1781225126.33 total 10.01 last 60.97) - open/close 1247 (time 1781225136.33 total 20.01 last 63.64) - open/close 1893 (time 1781225146.34 total 30.02 last 64.54) - open/close 2516 (time 1781225156.35 total 40.03 last 62.25) - open/close 3204 (time 1781225166.36 total 50.04 last 68.76) - open/close 3893 (time 1781225176.36 total 60.04 last 68.85) total: 4500 open/close in 68.86 seconds: 65.35 ops/second - /unlink 1536 (time 1781225201.13 total 10.02 last 153.29) - /unlink 2914 (time 1781225211.14 total 20.03 last 137.73) - /unlink 4081 (time 1781225221.14 total 30.03 last 116.65) total: 4500 /unlink in 33.64 seconds: 133.78 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 600 (time 1781225241.04 total 10.02 last 59.86) - open/close 1386 (time 1781225251.04 total 20.02 last 78.60) - open/close 2159 (time 1781225261.04 total 30.03 last 77.27) - open/close 2928 (time 1781225271.05 total 40.03 last 76.89) - open/close 3727 (time 1781225281.05 total 50.03 last 79.86) - open/close 4400 (time 1781225291.07 total 60.05 last 67.18) total: 4500 open/close in 61.27 seconds: 73.45 ops/second - /unlink 1512 (time 1781225307.82 total 10.00 last 151.17) - /unlink 3053 (time 1781225317.82 total 20.00 last 154.10) - /unlink 4431 (time 1781225327.82 total 30.00 last 137.77) total: 4500 /unlink in 30.77 seconds: 146.26 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 815 (time 1781225344.46 total 10.02 last 81.36) - open/close 1553 (time 1781225354.47 total 20.03 last 73.74) - open/close 2394 (time 1781225364.47 total 30.03 last 84.07) - open/close 3239 (time 1781225374.48 total 40.03 last 84.46) - open/close 4139 (time 1781225384.49 total 50.04 last 89.90) total: 4500 open/close in 55.42 seconds: 81.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1394 (time 1781225405.29 total 10.00 last 139.38) - /unlink 2570 (time 1781225415.29 total 20.01 last 117.54) - /unlink 4079 (time 1781225425.30 total 30.01 last 150.87) total: 4500 /unlink in 33.00 seconds: 136.38 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 828 (time 1781225444.21 total 10.01 last 82.70) - open/close 1694 (time 1781225454.22 total 20.01 last 86.59) - open/close 2500 (time 1781225464.22 total 30.02 last 80.58) - open/close 3301 (time 1781225474.22 total 40.02 last 80.04) - open/close 4039 (time 1781225484.25 total 50.05 last 73.59) total: 4500 open/close in 57.04 seconds: 78.89 ops/second - /unlink 1419 (time 1781225506.87 total 10.00 last 141.85) - /unlink 2849 (time 1781225516.88 total 20.01 last 142.94) - /unlink 3908 (time 1781225526.88 total 30.01 last 105.90) total: 4500 /unlink in 34.00 seconds: 132.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 656 (time 1781225546.75 total 10.02 last 65.50) - open/close 1242 (time 1781225556.76 total 20.02 last 58.57) - open/close 2001 (time 1781225566.77 total 30.04 last 75.77) - open/close 2552 (time 1781225576.79 total 40.05 last 55.04) - open/close 3182 (time 1781225586.79 total 50.05 last 62.98) - open/close 3778 (time 1781225596.79 total 60.05 last 59.58) - open/close 4467 (time 1781225606.80 total 70.07 last 68.81) total: 4500 open/close in 70.51 seconds: 63.82 ops/second - /unlink 1266 (time 1781225623.06 total 10.01 last 126.53) - /unlink 2726 (time 1781225633.07 total 20.01 last 145.95) - /unlink 3973 (time 1781225643.07 total 30.01 last 124.67) total: 4500 /unlink in 34.86 seconds: 129.08 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 725 (time 1781225663.99 total 10.01 last 72.43) - open/close 1318 (time 1781225673.99 total 20.01 last 59.29) - open/close 2019 (time 1781225684.01 total 30.03 last 69.96) - open/close 2763 (time 1781225694.02 total 40.04 last 74.36) - open/close 3346 (time 1781225704.03 total 50.05 last 58.24) - open/close 4094 (time 1781225714.09 total 60.10 last 74.37) total: 4500 open/close in 66.24 seconds: 67.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1381 (time 1781225735.29 total 10.00 last 138.09) - /unlink 2814 (time 1781225745.29 total 20.00 last 143.30) - /unlink 4112 (time 1781225755.30 total 30.01 last 129.73) total: 4500 /unlink in 33.45 seconds: 134.55 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 838 (time 1781225775.45 total 10.00 last 83.76) - open/close 1458 (time 1781225785.45 total 20.01 last 61.97) - open/close 2132 (time 1781225795.46 total 30.01 last 67.39) - open/close 2678 (time 1781225805.47 total 40.03 last 54.50) - open/close 3145 (time 1781225815.48 total 50.03 last 46.70) - open/close 3657 (time 1781225825.48 total 60.03 last 51.19) - open/close 4226 (time 1781225835.49 total 70.04 last 56.84) total: 4500 open/close in 73.24 seconds: 61.44 ops/second - /unlink 1246 (time 1781225853.68 total 10.00 last 124.60) - /unlink 2457 (time 1781225863.68 total 20.00 last 121.10) - /unlink 3539 (time 1781225873.68 total 30.00 last 108.15) total: 4500 /unlink in 39.60 seconds: 113.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 710 (time 1781225898.03 total 10.00 last 70.99) - open/close 1490 (time 1781225908.04 total 20.01 last 77.92) - open/close 2195 (time 1781225918.05 total 30.02 last 70.42) - open/close 2963 (time 1781225928.07 total 40.03 last 76.71) - open/close 3702 (time 1781225938.07 total 50.04 last 73.85) - open/close 4445 (time 1781225948.07 total 60.04 last 74.30) total: 4500 open/close in 60.61 seconds: 74.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1446 (time 1781225963.88 total 10.00 last 144.54) - /unlink 2897 (time 1781225973.89 total 20.01 last 145.01) - /unlink 4026 (time 1781225983.89 total 30.01 last 112.88) total: 4500 /unlink in 35.04 seconds: 128.44 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 810 (time 1781226004.59 total 10.00 last 80.98) - open/close 1435 (time 1781226014.59 total 20.00 last 62.50) - open/close 2155 (time 1781226024.62 total 30.04 last 71.72) - open/close 2787 (time 1781226034.63 total 40.04 last 63.18) - open/close 3508 (time 1781226044.63 total 50.05 last 72.07) - open/close 4238 (time 1781226054.64 total 60.05 last 72.97) total: 4500 open/close in 63.75 seconds: 70.59 ops/second - /unlink 1278 (time 1781226073.65 total 10.00 last 127.75) - /unlink 2616 (time 1781226083.65 total 20.01 last 133.78) - /unlink 4115 (time 1781226093.66 total 30.02 last 149.69) total: 4500 /unlink in 33.16 seconds: 135.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 776 (time 1781226112.63 total 10.01 last 77.54) - open/close 1581 (time 1781226122.63 total 20.01 last 80.47) - open/close 2503 (time 1781226132.63 total 30.02 last 92.15) - open/close 3401 (time 1781226142.63 total 40.02 last 89.79) - open/close 4072 (time 1781226152.65 total 50.04 last 66.98) total: 4500 open/close in 58.89 seconds: 76.41 ops/second - /unlink 1309 (time 1781226178.24 total 10.00 last 130.86) - /unlink 2214 (time 1781226188.25 total 20.01 last 90.47) - /unlink 3231 (time 1781226198.26 total 30.01 last 101.61) - /unlink 4359 (time 1781226208.27 total 40.03 last 112.61) total: 4500 /unlink in 41.71 seconds: 107.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 711 (time 1781226225.84 total 10.00 last 71.07) - open/close 1508 (time 1781226235.84 total 20.01 last 79.69) - open/close 2164 (time 1781226245.85 total 30.01 last 65.57) - open/close 2911 (time 1781226255.86 total 40.02 last 74.65) - open/close 3724 (time 1781226265.87 total 50.03 last 81.17) - open/close 4434 (time 1781226275.87 total 60.03 last 70.99) total: 4500 open/close in 60.93 seconds: 73.86 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 992 (time 1781226292.62 total 10.00 last 99.18) - /unlink 2637 (time 1781226302.63 total 20.01 last 164.43) - /unlink 4010 (time 1781226312.63 total 30.01 last 137.30) total: 4500 /unlink in 33.52 seconds: 134.24 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 804 (time 1781226331.95 total 10.00 last 80.38) - open/close 1598 (time 1781226341.95 total 20.00 last 79.39) - open/close 2385 (time 1781226351.96 total 30.01 last 78.66) - open/close 3187 (time 1781226361.98 total 40.04 last 79.99) - open/close 3902 (time 1781226371.99 total 50.04 last 71.44) total: 4500 open/close in 58.49 seconds: 76.94 ops/second - /unlink 1002 (time 1781226395.88 total 10.02 last 99.99) - /unlink 1864 (time 1781226405.89 total 20.02 last 86.17) - /unlink 3241 (time 1781226415.89 total 30.03 last 137.63) total: 4500 /unlink in 38.58 seconds: 116.65 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 765 (time 1781226439.47 total 10.00 last 76.49) - open/close 1600 (time 1781226449.48 total 20.01 last 83.43) - open/close 2361 (time 1781226459.49 total 30.02 last 76.02) - open/close 3095 (time 1781226469.50 total 40.03 last 73.34) - open/close 3640 (time 1781226479.53 total 50.06 last 54.34) - open/close 4175 (time 1781226489.53 total 60.06 last 53.50) total: 4500 open/close in 66.16 seconds: 68.02 ops/second - /unlink 1184 (time 1781226511.24 total 10.00 last 118.36) - /unlink 2522 (time 1781226521.24 total 20.00 last 133.79) - /unlink 3829 (time 1781226531.24 total 30.00 last 130.70) total: 4500 /unlink in 34.97 seconds: 128.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 721 (time 1781226551.33 total 10.01 last 72.03) - open/close 1316 (time 1781226561.36 total 20.03 last 59.37) - open/close 1976 (time 1781226571.36 total 30.03 last 65.99) - open/close 2602 (time 1781226581.37 total 40.04 last 62.54) - open/close 3388 (time 1781226591.37 total 50.05 last 78.58) - open/close 4356 (time 1781226601.39 total 60.06 last 96.65) total: 4500 open/close in 61.88 seconds: 72.72 ops/second - /unlink 1472 (time 1781226617.86 total 10.00 last 147.20) - /unlink 2907 (time 1781226627.86 total 20.00 last 143.44) - /unlink 4421 (time 1781226637.87 total 30.01 last 151.27) total: 4500 /unlink in 31.21 seconds: 144.20 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 602 (time 1781226654.39 total 10.01 last 60.12) - open/close 1300 (time 1781226664.40 total 20.02 last 69.77) - open/close 2058 (time 1781226674.40 total 30.02 last 75.75) - open/close 2808 (time 1781226684.41 total 40.03 last 74.93) - open/close 3614 (time 1781226694.42 total 50.04 last 80.50) - open/close 4129 (time 1781226704.44 total 60.06 last 51.41) total: 4500 open/close in 66.75 seconds: 67.41 ops/second - /unlink 1362 (time 1781226726.91 total 10.02 last 135.98) - /unlink 2734 (time 1781226736.92 total 20.02 last 137.15) - /unlink 4038 (time 1781226746.92 total 30.02 last 130.37) total: 4500 /unlink in 33.48 seconds: 134.41 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 827 (time 1781226765.84 total 10.01 last 82.64) - open/close 1635 (time 1781226775.84 total 20.01 last 80.77) - open/close 2434 (time 1781226785.89 total 30.06 last 79.54) - open/close 3333 (time 1781226795.89 total 40.06 last 89.90) - open/close 4148 (time 1781226805.91 total 50.07 last 81.35) total: 4500 open/close in 54.12 seconds: 83.14 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1302 (time 1781226825.07 total 10.01 last 130.04) - /unlink 2577 (time 1781226835.07 total 20.02 last 127.44) - /unlink 3908 (time 1781226845.07 total 30.02 last 133.07) total: 4500 /unlink in 33.66 seconds: 133.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 675 (time 1781226865.27 total 10.00 last 67.48) - open/close 1363 (time 1781226875.28 total 20.01 last 68.72) - open/close 1890 (time 1781226885.28 total 30.02 last 52.68) - open/close 2507 (time 1781226895.29 total 40.02 last 61.68) - open/close 3121 (time 1781226905.30 total 50.03 last 61.35) - open/close 3909 (time 1781226915.30 total 60.04 last 78.75) total: 4500 open/close in 67.44 seconds: 66.72 ops/second - /unlink 1408 (time 1781226937.25 total 10.02 last 140.51) - /unlink 2466 (time 1781226947.25 total 20.02 last 105.79) - /unlink 3695 (time 1781226957.27 total 30.04 last 122.73) total: 4500 /unlink in 36.69 seconds: 122.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 750 (time 1781226979.36 total 10.01 last 74.91) - open/close 1436 (time 1781226989.37 total 20.02 last 68.54) - open/close 2174 (time 1781226999.37 total 30.02 last 73.79) - open/close 2827 (time 1781227009.39 total 40.04 last 65.21) - open/close 3451 (time 1781227019.40 total 50.05 last 62.33) - open/close 4154 (time 1781227029.40 total 60.05 last 70.29) total: 4500 open/close in 64.05 seconds: 70.26 ops/second - /unlink 1421 (time 1781227048.34 total 10.01 last 142.01) - /unlink 2850 (time 1781227058.35 total 20.01 last 142.85) - /unlink 4127 (time 1781227068.35 total 30.01 last 127.66) total: 4500 /unlink in 33.40 seconds: 134.72 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11202: 94162 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg248-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg248-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg248-server stop mds service on oleg248-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg248-server stop mds service on oleg248-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg248-server unloading modules via unload_modules_local on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing unload_modules_local oleg248-server: modules unloaded.