== conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 20:40:12 (1763343612) Stopping clients: oleg316-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg316-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg316-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg316-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg316-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg316-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg316-server oleg316-server: oleg316-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing load_modules_local oleg316-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg316-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg316-server: Loading modules from /home/green/git/lustre-release/lustre oleg316-server: detected 4 online CPUs by sysfs oleg316-server: MODOPTS_LIBCFS= oleg316-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg316-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg316-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg316-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg316-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg316-server: oleg316-server.virtnet: executing set_default_debug -1 all pdsh@oleg316-client: oleg316-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg316-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg316-client.virtnet: -o user_xattr,flock 192.168.203.116@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 fail_loc=0x1312 fail_val=5 striped dir -i0 -c1 -H crush /mnt/lustre/d135.conf-sanity seq.cli-lustre-OST0000-super.width=0x1ffffff mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' Wrap arround changelog catalog lustre-MDT0000: clear the changelog for cl1 to record #12998 - open/close 787 (time 1763343785.41 total 10.00 last 78.68) - open/close 1606 (time 1763343795.41 total 20.01 last 81.83) - open/close 2432 (time 1763343805.42 total 30.02 last 82.54) - open/close 3237 (time 1763343815.42 total 40.02 last 80.50) - open/close 4078 (time 1763343825.42 total 50.02 last 84.10) total: 4500 open/close in 55.07 seconds: 81.71 ops/second - /unlink 1441 (time 1763343843.38 total 10.00 last 144.07) - /unlink 2838 (time 1763343853.38 total 20.01 last 139.61) - /unlink 4228 (time 1763343863.38 total 30.01 last 138.97) total: 4500 /unlink in 32.04 seconds: 140.47 ops/second lustre-MDT0000: clear the changelog for cl1 to record #25998 - open/close 716 (time 1763343879.56 total 10.01 last 71.56) - open/close 1489 (time 1763343889.56 total 20.01 last 77.29) - open/close 2294 (time 1763343899.57 total 30.01 last 80.46) - open/close 3122 (time 1763343909.57 total 40.01 last 82.78) - open/close 3841 (time 1763343919.58 total 50.02 last 71.84) total: 4500 open/close in 58.75 seconds: 76.60 ops/second - /unlink 1511 (time 1763343942.37 total 10.00 last 151.06) - /unlink 2988 (time 1763343952.38 total 20.01 last 147.66) - /unlink 4452 (time 1763343962.38 total 30.01 last 146.33) total: 4500 /unlink in 30.43 seconds: 147.90 ops/second lustre-MDT0000: clear the changelog for cl1 to record #38998 - open/close 826 (time 1763343976.02 total 10.00 last 82.56) - open/close 1562 (time 1763343986.03 total 20.01 last 73.55) - open/close 2356 (time 1763343996.04 total 30.02 last 79.32) - open/close 3070 (time 1763344006.05 total 40.03 last 71.34) - open/close 3835 (time 1763344016.08 total 50.06 last 76.25) total: 4500 open/close in 59.39 seconds: 75.77 ops/second lustre-MDT0000: clear the changelog for cl1 to record #51998 - /unlink 1389 (time 1763344039.04 total 10.01 last 138.79) - /unlink 2707 (time 1763344049.04 total 20.01 last 131.73) - /unlink 4096 (time 1763344059.04 total 30.01 last 138.88) total: 4500 /unlink in 32.71 seconds: 137.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #64998 - open/close 792 (time 1763344075.72 total 10.00 last 79.16) - open/close 1541 (time 1763344085.72 total 20.01 last 74.89) - open/close 2288 (time 1763344095.72 total 30.01 last 74.69) - open/close 3039 (time 1763344105.74 total 40.02 last 74.98) - open/close 3698 (time 1763344115.74 total 50.02 last 65.89) - open/close 4400 (time 1763344125.75 total 60.03 last 70.13) total: 4500 open/close in 61.61 seconds: 73.04 ops/second - /unlink 1328 (time 1763344141.55 total 10.00 last 132.80) - /unlink 2745 (time 1763344151.55 total 20.01 last 141.62) - /unlink 4137 (time 1763344161.56 total 30.01 last 139.14) total: 4500 /unlink in 32.97 seconds: 136.50 ops/second lustre-MDT0000: clear the changelog for cl1 to record #77998 - open/close 756 (time 1763344178.56 total 10.02 last 75.48) - open/close 1464 (time 1763344188.57 total 20.02 last 70.74) - open/close 2190 (time 1763344198.58 total 30.03 last 72.54) - open/close 2882 (time 1763344208.58 total 40.03 last 69.18) - open/close 3657 (time 1763344218.59 total 50.04 last 77.45) total: 4500 open/close in 59.90 seconds: 75.13 ops/second - /unlink 1601 (time 1763344240.98 total 10.00 last 160.04) - /unlink 3058 (time 1763344250.99 total 20.01 last 145.64) total: 4500 /unlink in 29.45 seconds: 152.81 ops/second lustre-MDT0000: clear the changelog for cl1 to record #90998 - open/close 773 (time 1763344274.38 total 10.01 last 77.23) - open/close 1573 (time 1763344284.39 total 20.02 last 79.88) - open/close 2344 (time 1763344294.40 total 30.03 last 77.05) - open/close 3108 (time 1763344304.41 total 40.04 last 76.34) - open/close 3891 (time 1763344314.41 total 50.05 last 78.24) total: 4500 open/close in 58.53 seconds: 76.89 ops/second lustre-MDT0000: clear the changelog for cl1 to record #103998 - /unlink 1488 (time 1763344336.53 total 10.00 last 148.73) - /unlink 2987 (time 1763344346.54 total 20.01 last 149.84) - /unlink 4382 (time 1763344356.54 total 30.01 last 139.46) total: 4500 /unlink in 31.14 seconds: 144.49 ops/second lustre-MDT0000: clear the changelog for cl1 to record #116998 - open/close 674 (time 1763344371.95 total 10.01 last 67.36) - open/close 1397 (time 1763344381.95 total 20.01 last 72.24) - open/close 2112 (time 1763344391.97 total 30.03 last 71.40) - open/close 2903 (time 1763344401.98 total 40.04 last 78.98) - open/close 3610 (time 1763344411.99 total 50.05 last 70.65) - open/close 4209 (time 1763344422.00 total 60.06 last 59.84) total: 4500 open/close in 65.04 seconds: 69.19 ops/second - /unlink 1349 (time 1763344441.30 total 10.01 last 134.82) - /unlink 2694 (time 1763344451.30 total 20.01 last 134.46) - /unlink 4068 (time 1763344461.30 total 30.01 last 137.39) total: 4500 /unlink in 33.25 seconds: 135.34 ops/second lustre-MDT0000: clear the changelog for cl1 to record #129998 - open/close 745 (time 1763344479.03 total 10.03 last 74.28) - open/close 1435 (time 1763344489.04 total 20.04 last 68.92) - open/close 2192 (time 1763344499.05 total 30.05 last 75.64) - open/close 2983 (time 1763344509.06 total 40.06 last 78.99) - open/close 3730 (time 1763344519.07 total 50.07 last 74.64) total: 4500 open/close in 60.00 seconds: 74.99 ops/second lustre-MDT0000: clear the changelog for cl1 to record #142998 - /unlink 1369 (time 1763344542.62 total 10.01 last 136.82) - /unlink 2752 (time 1763344552.63 total 20.01 last 138.20) - /unlink 4231 (time 1763344562.63 total 30.01 last 147.89) total: 4500 /unlink in 32.01 seconds: 140.58 ops/second lustre-MDT0000: clear the changelog for cl1 to record #155998 - open/close 811 (time 1763344578.16 total 10.01 last 81.05) - open/close 1575 (time 1763344588.16 total 20.01 last 76.38) - open/close 2334 (time 1763344598.16 total 30.01 last 75.86) - open/close 3101 (time 1763344608.17 total 40.02 last 76.68) - open/close 3862 (time 1763344618.17 total 50.02 last 76.09) total: 4500 open/close in 58.32 seconds: 77.16 ops/second - /unlink 1329 (time 1763344641.10 total 10.00 last 132.87) - /unlink 2639 (time 1763344651.11 total 20.01 last 130.96) - /unlink 4026 (time 1763344661.12 total 30.02 last 138.50) total: 4500 /unlink in 33.76 seconds: 133.28 ops/second lustre-MDT0000: clear the changelog for cl1 to record #168998 - open/close 755 (time 1763344679.54 total 10.01 last 75.43) - open/close 1510 (time 1763344689.55 total 20.02 last 75.41) - open/close 2241 (time 1763344699.57 total 30.03 last 73.00) - open/close 3007 (time 1763344709.57 total 40.04 last 76.59) - open/close 3777 (time 1763344719.57 total 50.04 last 76.96) total: 4500 open/close in 59.34 seconds: 75.84 ops/second - /unlink 1377 (time 1763344742.58 total 10.00 last 137.64) - /unlink 2779 (time 1763344752.58 total 20.00 last 140.20) - /unlink 4051 (time 1763344762.58 total 30.01 last 127.18) total: 4500 /unlink in 33.47 seconds: 134.46 ops/second lustre-MDT0000: clear the changelog for cl1 to record #181998 - open/close 839 (time 1763344779.83 total 10.00 last 83.86) - open/close 1635 (time 1763344789.86 total 20.03 last 79.38) - open/close 2355 (time 1763344799.87 total 30.04 last 71.93) - open/close 3161 (time 1763344809.88 total 40.05 last 80.52) - open/close 3981 (time 1763344819.89 total 50.07 last 81.88) total: 4500 open/close in 56.24 seconds: 80.02 ops/second lustre-MDT0000: clear the changelog for cl1 to record #194998 - /unlink 1393 (time 1763344839.90 total 10.00 last 139.29) - /unlink 2762 (time 1763344849.90 total 20.00 last 136.89) - /unlink 4156 (time 1763344859.90 total 30.01 last 139.34) total: 4500 /unlink in 32.90 seconds: 136.79 ops/second lustre-MDT0000: clear the changelog for cl1 to record #207998 - open/close 788 (time 1763344877.17 total 10.00 last 78.77) - open/close 1591 (time 1763344887.17 total 20.00 last 80.29) - open/close 2405 (time 1763344897.17 total 30.01 last 81.36) - open/close 3171 (time 1763344907.18 total 40.01 last 76.57) - open/close 3974 (time 1763344917.19 total 50.03 last 80.20) total: 4500 open/close in 56.99 seconds: 78.96 ops/second - /unlink 1401 (time 1763344937.93 total 10.00 last 140.04) - /unlink 2754 (time 1763344947.93 total 20.01 last 135.24) - /unlink 4139 (time 1763344957.94 total 30.01 last 138.44) total: 4500 /unlink in 32.92 seconds: 136.70 ops/second lustre-MDT0000: clear the changelog for cl1 to record #220998 - open/close 805 (time 1763344974.82 total 10.00 last 80.49) - open/close 1429 (time 1763344984.82 total 20.00 last 62.40) - open/close 2162 (time 1763344994.83 total 30.00 last 73.29) - open/close 2898 (time 1763345004.83 total 40.01 last 73.56) - open/close 3650 (time 1763345014.84 total 50.01 last 75.16) - open/close 4359 (time 1763345024.84 total 60.01 last 70.90) total: 4500 open/close in 62.28 seconds: 72.25 ops/second - /unlink 1399 (time 1763345042.14 total 10.00 last 139.90) - /unlink 2737 (time 1763345052.14 total 20.00 last 133.80) - /unlink 4048 (time 1763345062.14 total 30.00 last 131.07) total: 4500 /unlink in 33.23 seconds: 135.42 ops/second lustre-MDT0000: clear the changelog for cl1 to record #233998 lustre-MDT0000: clear the changelog for cl1 to record #246998 - open/close 717 (time 1763345079.65 total 10.00 last 71.70) - open/close 1431 (time 1763345089.65 total 20.00 last 71.39) - open/close 2126 (time 1763345099.66 total 30.01 last 69.47) - open/close 2794 (time 1763345109.68 total 40.03 last 66.66) - open/close 3550 (time 1763345119.69 total 50.04 last 75.49) - open/close 4282 (time 1763345129.70 total 60.05 last 73.15) total: 4500 open/close in 63.02 seconds: 71.41 ops/second - /unlink 1415 (time 1763345146.49 total 10.01 last 141.37) - /unlink 2804 (time 1763345156.50 total 20.01 last 138.85) - /unlink 4164 (time 1763345166.50 total 30.02 last 135.92) total: 4500 /unlink in 32.60 seconds: 138.03 ops/second lustre-MDT0000: clear the changelog for cl1 to record #259998 - open/close 790 (time 1763345182.80 total 10.00 last 78.97) - open/close 1568 (time 1763345192.81 total 20.02 last 77.70) - open/close 2264 (time 1763345202.81 total 30.02 last 69.56) - open/close 2969 (time 1763345212.82 total 40.03 last 70.47) - open/close 3727 (time 1763345222.82 total 50.03 last 75.76) - open/close 4461 (time 1763345232.83 total 60.03 last 73.39) total: 4500 open/close in 60.74 seconds: 74.09 ops/second - /unlink 1362 (time 1763345247.51 total 10.00 last 136.17) - /unlink 2624 (time 1763345257.52 total 20.00 last 126.18) - /unlink 3961 (time 1763345267.52 total 30.01 last 133.67) total: 4500 /unlink in 34.11 seconds: 131.93 ops/second lustre-MDT0000: clear the changelog for cl1 to record #272998 - open/close 764 (time 1763345285.69 total 10.00 last 76.39) - open/close 1509 (time 1763345295.69 total 20.00 last 74.49) - open/close 2133 (time 1763345305.69 total 30.00 last 62.40) - open/close 2824 (time 1763345315.71 total 40.02 last 68.97) - open/close 3584 (time 1763345325.71 total 50.03 last 75.96) - open/close 4349 (time 1763345335.71 total 60.03 last 76.50) total: 4500 open/close in 61.95 seconds: 72.64 ops/second lustre-MDT0000: clear the changelog for cl1 to record #285998 - /unlink 1564 (time 1763345350.77 total 10.00 last 156.34) - /unlink 2902 (time 1763345360.77 total 20.01 last 133.75) - /unlink 4323 (time 1763345370.77 total 30.01 last 142.10) total: 4500 /unlink in 31.10 seconds: 144.71 ops/second lustre-MDT0000: clear the changelog for cl1 to record #298998 - open/close 807 (time 1763345385.07 total 10.01 last 80.63) - open/close 1637 (time 1763345395.08 total 20.02 last 82.92) - open/close 2458 (time 1763345405.08 total 30.02 last 82.06) - open/close 3240 (time 1763345415.09 total 40.03 last 78.15) - open/close 4085 (time 1763345425.10 total 50.04 last 84.43) total: 4500 open/close in 55.18 seconds: 81.55 ops/second - /unlink 1447 (time 1763345442.79 total 10.01 last 144.54) - /unlink 2958 (time 1763345452.80 total 20.01 last 151.07) total: 4500 /unlink in 29.95 seconds: 150.25 ops/second lustre-MDT0000: clear the changelog for cl1 to record #311998 - open/close 798 (time 1763345476.30 total 10.00 last 79.79) - open/close 1572 (time 1763345486.30 total 20.00 last 77.37) - open/close 2433 (time 1763345496.32 total 30.02 last 85.98) - open/close 3274 (time 1763345506.33 total 40.03 last 84.03) - open/close 4098 (time 1763345516.34 total 50.04 last 82.33) total: 4500 open/close in 55.18 seconds: 81.55 ops/second - /unlink 1343 (time 1763345534.88 total 10.00 last 134.30) - /unlink 2596 (time 1763345544.88 total 20.00 last 125.29) - /unlink 3795 (time 1763345554.89 total 30.01 last 119.81) total: 4500 /unlink in 35.96 seconds: 125.14 ops/second kill changelog reader /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 11098: 93976 Terminated coproc COPROC $LFS changelog --follow $service (wd: ~) lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 Cleanup test_135 umount lustre on /mnt/lustre..... Stopping client oleg316-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg316-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg316-server stop mds service on oleg316-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg316-server stop mds service on oleg316-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg316-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg316-server' oleg316-server: oleg316-server.virtnet: executing unload_modules_local oleg316-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg316-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded.