Apache Lucene.NET
This release contains several important bug fixes and performance enhancements.
There are 2 breaking changes that may affect some users when reading indexes that were created from version 4.8.0-beta00015 and all prior 4.8.0 beta versions (not including 3.0.3).
Lucene.Net.Documents.DateTools has been modified to return milliseconds since Unix epoch (that is, since Jan 1, 1970 at 00:00:00 UTC) by default to match Java Lucene. This only affects users who explicitly use Lucene.Net.Documents.DateTools in their application and store the result (in .NET ticks) in their index.If you are affected by either of the above issues, it is recommended to regenerate your indexes during upgrading. However, if that is not feasible, we have provided the following workarounds.
Lucene.Net.Documents.DateTools.StringToTime(string) or Lucene.Net.Documents.DateTools.Round(long) (a long) in your index, you are storing .NET ticks. There are now optional parameters inputRepresentation and outputRepresentation on these methods to specify whether the long value represents .NET ticks, .NET ticks as milliseconds, or millisenonds since the Unix epoch. To exactly match version 4.8.0-beta00015 and prior (including prior major versions):
Lucene.Net.Documents.DateTools.StringToTime(string, NumericRepresentation) should specify NumericRepresentation.TICKS for outputRepresentation.Lucene.Net.Documents.DateTools.Round(long, NumericRepresentation, NumericRepresentation) should specify NumericRepresentation.TICKS_AS_MILLISECONDS for inputRepresentation and NumericRepresentation.TICKS for outputRepresentation.It is recommended that all .NET Framework users migrate as soon as possible.
Lucene.Net.Support.WeakDictionary<TKey, TValue> was used in .NET Framework and .NET Standard 2.0 due to missing APIs, but there is now a better solution using Prism.Core's weak events in combination with ConditionalWeakTable<TKey, TValue>, which means memory management is handled entirely by the GC in Lucene.Net.Index.IndexReader, Lucene.Net.Search.FieldCacheImpl, Lucene.Net.Search.CachingWrappingFilter and Lucene.Net.Facet.Taxonomy.CachedOrdinalsReader. See #613.Lucene.Net.Util.StringHelper.GOOD_FAST_HASH_SEED: converted from a static field to a property and marked obsolete. Added a new property GoodFastHashSeed. Removed SystemProperties call to populate the value of the field, since NUnit only allows us to generate a seed per test, and we need a way to inject the seed value for repeatability.Lucene.Net.TestFramework: Added LuceneSetUpFixtureBuilder class to load either a subclass or our default instance of LuceneTestFrameworkInitializer. Also added LuceneTestCase.SetUpFixture to control initialization of LuceneTestFrameworkInitializer so it is only called on setup and teardown for the assembly. Added Initialize() method to LuceneTestFrameworkInitializer that must be used when setting factories during testing.Lucene.Net.TestFramework.Util.LuceneTestCase: Deprecated GetClassType() method and added TestType propertyLucene.Net.TestFramework.Util.AbstractBeforeAfterRule``: Removed LuceneTestCaseparameter fromBefore()andAfter()` methods.Lucene.Net.Util.NumberFormat and Lucene.Net.QueryParsers.Flexible.Standard.Config.NumberDateFormat to accept IFormatProvider rather than CultureInfo and changed Lucene.Net.Util.NumberFormat.Culture property to Lucene.Net.Util.NumberFormat.FormatProvider.Lucene.Net.Misc: Made DocFreqComparer and TotalTermFreqComparer into static singletons, only accessible by the Default property.Lucene.Net.Search.FieldComparer: Redesigned implementation to use reference types for numerics (from J2N) to avoid boxing.Lucene.Net.Search.FieldCache.IParser: Renamed method from TermsEnum() to GetTermsEnum() to match other APIsLucene.Net.Queries: ObjectVal() returns a J2N.Numerics.Number-derived type rather than a value type cast to object. Direct casts to int, long, double, single, etc. will no longer work without first casting to the J2N.Numerics.Number-derived type. Alternatively, use the corresponding Convert.ToXXX() method for the type you wish to retrieve from the object.Lucene.Net.Suggest.Fst.FSTCompletionLookup/WFSTCompletionLookup: Changed Get() to return long? instead of object to eliminate boxing/unboxingLucene.Net.Index.MergePolicy::FindForcedMerges(): Removed unnecessary nullable from FindForcedMerges() and all MergePolicy subclassesLucene.Net.Replicator: Changed callback signature from Func<bool?> to Action, since the return value had no semantic meaningLucene.Net.Index.DocValuesFieldUpdates: Refactored so the subclasses will handle getting the values from DocValuesFieldUpdatesIterator or DocValuesUpdate via a cast rather than boxing the value. Also marked internal (as well as all members of BufferedUpdates), since this was not supposed to be part of the public API.Lucene.Net.Index.SegmentInfos: Changed Info() method to an indexer (.NET Convention)Lucene.Net.Documents.DateTools - Added NumericRepresentation enum to allow converting to/from long in the following formats:
Unix Epoch (default): Milliseconds since Jan 1, 1970 12:00:00 AM UTC.
Ticks: The raw ticks from DateTime or DateTimeOffset.
Ticks as Milliseconds: This is for compatibility with prior versions of Lucene.NET (3.0.3 and 4.8.0-beta00001 - 4.8.0-beta00015). The conversion done on input values is time * TimeSpan.TicksPerMillisecond and the conversion to output values is time / TimeSpan.TicksPerMillisecond.
The long return value from Lucene.Net.Documents.DateTools.StringToTime(string, NumericRepresentation) has been changed from NumericRepresentation.TICKS to NumericRepresentation.UNIX_TIME_MILLISECONDS by default.
The long input parameter provided to Lucene.Net.Documents.DateTools.Round(long, NumericRepresentation, NumericRepresentation) has been changed from NumericRepresentation.TICKS_AS_MILLISECONDS to NumericRepresentation.UNIX_TIME_MILLISECONDS by default.
The long return value from Lucene.Net.Documents.DateTools.Round(long, NumericRepresentation, NumericRepresentation) has changed from NumericRepresentation.TICKS to NumericRepresentation.UNIX_TIME_MILLISECONDS by default.
Lucene.Net.Documents.DateTools - De-nested Resolution enum and renamed DateResolution.Lucene.Net.QueryParsers.Flexible.Standard: Changed numeric nodes to accept and return J2N.Numerics.Number-derived types instead of object.Lucene.Net.Util.Fst: Changed API to use J2N.Numerics.Int64 instead of long? for generic closing type as it was designed to use reference equality comparison.Lucene.Net.Util.Fst: Added class constraints to each generic FST type and reverted to reference equality comparisons.Lucene.Net.Util.Fst.Int32sRefFSTEnum: Added MoveNext() method and marked Next() method obsolete. This change had already been done to BytesRefFSTEnum, which made them inconsistent.Lucene.Net.QueryParsers.Flexible: Removed unnecessary nullable value types from ConfigurationKeys and configuration setters/getters in StandardQueryParser. Added AbstractQueryConfig.TryGetValue() method to allow retrieving value types so they can be defaulted properly.Lucene.Net.Queries.Function.ValueSources.EnumFieldSource::ctor() - changed enumIntToStringMap to accept IDictionary<int, string> instead of IDictionary<int?, string> (removed unnecessary nullable)Lucene.Net.TestFramework.Store.MockDirectoryWrapper: Renamed AssertNoUnreferencedFilesOnClose to AssertNoUnreferencedFilesOnDispose
Lucene.Net.Spatial: Upgraded to new Spatial4n NuGet package that unifies the types from Spatial4n.Core and Spatial4n.Core.NTSLucene.Net.Spatial.Prefix.Tree.Cell: Renamed m_outerInstance > m_spatialPrefixTree and constructor parameter outerInstance > spatialPrefixTree
Lucene.Net.Spatial.Prefix.AbstractPrefixTreeFilter.BaseTermsEnumTransverser: renamed m_outerInstance > m_filter, constructor parameter outerInstance > filter
Lucene.Net.Spatial.Prefix.AbstractPrefixTreeFilter: De-nested BaseTermsEnumTraverserclassLucene.Net.Spatial.Prefix.Tree.GeohashPrefixTree.Factory: de-nested and renamed GeohashPrefixTreeFactory
Lucene.Net.Spatial.Prefix.Tree.QuadPrefixTree.Factory: de-nested and renamed QuadPrefixTreeFactory
Lucene.Net.Spatial.Prefix.AbstractVisitingPrefixTreeFilter: De-nested VisitorTemplate class and changed protected field m_prefixGridScanLevel to a public property named PrefixGridScanLevel.Lucene.Net.Spatial.Query: Renamed UnsupportedSpatialOperation > UnsupportedSpatialOperationException to match .NET conventionsLucene.Net.Replicator.Http.HttpReplicatorTest::TestBasic(). This was failing intermittently due to the Timeout value being set to 1 second instead of the 60 second value that was used in Java. It has been increased to the .NET default of 100 seconds.Lucene.Net.Replicator.IndexAndTaxonomyReplicationClientTest::TestConsistencyOnExceptions() and Lucene.Net.Replicator.IndexReplicationClientTest::TestConsistencyOnExceptions() were failing due to to exceptions being raised on the worker thread and missing locks, which have both been addressed.[SuppressCodecs] attribute where required for custom Lucene.NET testsTermsEnum.MoveNext() methods to return a check for null rather than returning true
Lucene.Net.TestFramework.Index.BasePostingsFormatTestCase: Removed IndexOptions.NONE from the list of available options, since it is not a valid test optionLucene.Net.Grouping.Term.TermAllGroupHeadsCollector: Use NumericUtils.SingleToSortableInt32() to compare floating point numbers (Fixes AllGroupHeadCollectorTest.TestRandom() on .NET Framework x86).Lucene.Net.Tests.Util.TestPriorityQueue: Fixed issues with comparers after introducing J2N.Randomizer, which produces negative random numbers.Lucene.Net.Codecs.SimpleText.SimpleTextFieldsReader::NextDoc(): Fixed assert that was throwing on BytesRef.Utf8ToString()
Lucene.Net.Util.Automation.MinimizationOperations::MinimizeHopcroft(): Fixed range in OpenBitSet.Clear()
Lucene.Net.Tests.QueryParser.Flexible.Standard.TestQPHelper: Use ParseExact() method to specify the date format, so it works across cultures.Lucene.Net.Search.Suggest.Analyzing.BlendedInfixSuggester: Apply patch from https://issues.apache.org/jira/browse/LUCENE-6093 to fix ArgumentNullException if there were discarded trailing characters in the query (Thanks @Maxwellwr)Use StringComparer.OrdinalforSort()` methods, where appropriateLucene.Net.QueryParser.Flexible.Standard: Fixed calendar handling on .NET CoreLucene.Net.Suggest.Jaspell.JaspellTernarySearchTree: Fixed random number generator so it produces random numbersLucene.Net.TestFramework.Util.TestUtil::RandomAnalysisString(): Fixed ArgumentOutOfRangeException when passed a maxLength of 0.Lucene.Net.Search.DisjunctionMaxScorer: Fixed x86 floating point precision issue on .NET FrameworkLucene.Net.Expressions.ScoreFunctionValues: Fixed x86 floating point precision issue on .NET FrameworkLucene.Net.Spatial.Prefix.SpatialOpRecursivePrefixTreeTest: Ported over patch from https://github.com/apache/lucene/commit/e9906a334b8e123e93b917c3feb6e55fed0a8c57 (from 4.9.0).Lucene.Net.Index.TestDuelingCodecs::TestEquals(): There was a missing ! in Lucene.Net.Codecs.BlockTreeTermsReader.IntersectEnum.Frame::Load() that was inverting the logic, causing this test to fail intermittently.Lucene.Net.Search.TestJoinUtil::TestMultiValueRandomJoin(): Fixed x86 floating point precision issue on .NET FrameworkLucene.Net.Search.Spell.TestSpellChecker::TestConcurrentAccess(): Fixed issues that were causing the test to hang due to concurrency problems.ControlledRealTimeReopenThread to correctly handle timing (thanks @rclabo)Lucene.Net.Support.Collections.ReverseComparer<T>: Replaced CaseInsensitiveComparer with J2N.Collections.Generic.Comparer<T>. This only affects tests..github/workflows: Updated website/documentation configs to use subdirectory glob patterns for paths.Lucene.Net.Search.FieldCacheImpl.Cache<TKey, TValue>::Put(): Logic was inverted on innerCache field so the value was being updated if exists, when it should not be updated in this caseLucene.Net.Search.FieldCacheImpl::Cache<TKey, TValue> (Put + Get): Fixed issue with InitReader() being called prior to adding the item to the cache when it should be called afterLucene.Net.Spatial.Query.SpatialArgs::ctor(): Set operation and shape fields rather than calling the virtual properties to set them (which can cause initialization issues for subclasses)Lucene.Net.TestFramework.Search.CheckHits::CheckHitCollector(): Removed unnecessary call to Convert.ToInt32() and simplified collection initialization.System.Threading.Thread.Interrupt() when writing indexes due to the high possibility in .NET that it could break a Commit() or cause a deadlock.[Serializable] exceptions on all target platforms (previously, exceptions were not serializable in .NET Core)[Serializable] support for all classes except for the following (See #567 for a complete list)
IComparer<T> implementations that are publicly exposed directly or through collectionsLucene.Net.TestFramework.Util.LuceneTestCase::NewSearcher(): Added missing event handler to shut down LimitedConcurrencyLevelTaskScheduler to prevent it from accepting new work when we are attempting to end the background process.Lucene.Net.Support: Factored out ICallable<V> and ICompletionService<V> interfaces, as they are not neededLucene.Net.Search.NumericRangeQuery: Eliminated boxing when converting from T to the numeric type and when comparing equalityLucene.Net.Suggest.Jaspell: Use J2N numeric types to eliminate boxingLucene.Net.Search.FieldCache: Use J2N parsers and formattersLucene.Net.Classification.Utils.DatasetSplitter: Removed duplicate calls to field methods and stored values in local variables. Use default round-trip format from J2N.Lucene.Net.Search.FieldCacheRangeFilter: Use HasValue and Value for nullable value types rather casting and comparing to nullLucene.Net.Facet: Added culture-sensitve ToString() overload on FacetResult and LabelAndValue
Lucene.Net.Facet.FacetResult: Added nullable reference type supportLucene.Net.Facet.DrillDownQuery: Added collection initializer supportLucene.Net.Documents.DateTools - Added support for TimeZoneInfo when converting to/from stringLucene.Net.QueryParsers.Flexible.Standard.Config.NumberDateFormat: Added constructor overload to format a date without a time.Lucene.Net.QueryParsers.Flexible.Standard.Config.NumberDateFormat: Added NumericRepresentation property to set the representation to use for both Format() and Parse().Lucene.Net.QueryParsers - Added support for TimeZoneInfo when converting to/from string (Classic and Flexible query parsers)Lucene.Net.QueryParsers.Classic.QueryParserBase: Use TryParse() instead of Parse() to parse numeric values. Use the current culture, but fall back to invariant culture.Lucene.Net.Search.FieldCacheRangeFilter: Eliminated boxing in Equals() checkLucene.Net.Expressions.SimpleBindings: Added collection initializer support. Updated DistanceFacetsExample and ExpressionAggregationFacetsExample to demonstrate usage.LuceneTestCase
lucene-cli: Added embedded readme to NuGet package and updated build to update docs with release version numberIndexWriter tests to Lucene.Net.Tests._I-J to make both Lucene.Net.Tests._E-I and Lucene.Net.Tests._I-J run less than 2 minutes. This cuts the total time on Azure DevOps by around 5 minutes.LuceneDocsPlugins projectNIOFSDirectory bottleneck on multiple instances by switching from a static shared lock to a lock per FileStream instance.Lucene.Net.Store: Updated the FSDirectory documentation to remove irrelevant Java info and replace it with performance characteristics of the .NET implementation.WeakDictionary<TKey, TValue> in favor of weak events using Prism.Core
Lucene.Net.Spatial: Enabled nullable reference type supportLucene.Net.Spatial: Added guard clauses, where appropriateLucene.Net.TestFramework: Fixed random seed functionality so it is repeatable, so random tests can be more easily debugged. The random seed and how to configure a test assembly to repeat the same result is appended to the output message of the test (which becomes visible upon failure). The J2N.Randomizer class was used to provide random numbers, which uses the same implementation on every OS, so the random seeds are portable across operating systems.lucene-cli: Added multitarget support for .NET Core 3.1, .NET 5.0, and .NET 6.0This release contains important bug fixes, performance enhancements, concurrency improvements, and improved debugging support (full stack traces, consistent exception types, attributes for debug view, and structurally formattable lists).
Much of the exception handling has been changed so it is recommended to test thoroughly, especially if your application relies on catching exceptions from Lucene.NET for control flow. The full extent of the exception handling changes are not documented here, but can be viewed at https://github.com/apache/lucenenet/pull/476/files.
Lucene.Net.Index.IndexWriter::Dispose(): Using Thread.Interrupt() to shutdown background threads in .NET is problematic because System.Threading.ThreadInterruptedException could be thrown on any lock statement with contention on it. This includes lock statements on code that we depend on or custom components that are engaged during a Commit() (such as a custom Directory implementation). These exceptions may cause Commit() to fail in unexpected ways during IndexWriter.Dispose(). While this affected all prior releases to a much larger degree, this release provides a partial solution using UninterruptableMonitor.Enter() to ensure these exceptions are ignored and the Thread.Interrupt() state restored, which greatly reduces the chance a Commit() could be broken or a deadlock can occur. This problem will not affect applications that do not call Thread.Interrupt() to shut down a thread. It is recommended never to use Thread.Interrupt() in conjunction with IndexWriter, ConcurrentMergeScheduler, or ControlledRealTimeReopenThread.lucene-cli: Changed exit codes to well-defined constants to make testing simplerLucene.Net.Documents.Extensions namespace and added tests for DocumentExtensions in Lucene.Net.Tests._J-S, Lucene.Net.Tests.ICU and Lucene.Net.Tests.Facet. Added guard clauses and updated documentation of Document extension methods and some related fields.Lucene.Net.TestFramework.Util.TestUtil: Renamed method parameters from abbreviations to whole words to follow .NET API conventions and improved documentation.Lucene.Net.Grouping: Refactored and improved GroupingSearch Search API and added GroupByField() and GroupByFunction() methods.Lucene.Net.Join types to Lucene.Net.Search.Join namespaceCardinality() methods to Cardinality property. Added obsolete Cardinality() extension methods to the namespace of each of the pertinent types for backward compatibility.
Lucene.Net.Index.RandomAccessOrds
Lucene.Net.Util.FixedBitSet
Lucene.Net.Util.Int64BitSet
Lucene.Net.Util.OpenBitSet
Lucene.Net.Util.PForDeltaDocIdSet
Lucene.Net.Util.WAH8DocIdSet
Lucene.Net.Index.Term: Changed Text() method into Text property. Added an obsolete Text() extension method to Lucene.Net.Index namespace for backward compatibility.Lucene.Net.BinaryDocValuesField: Changed fType static field to TYPE (as it was in Lucene) and added obsolete fType field for backward compatibility.GetFilePointer() methods into properties named Position to match FileStream. Types affected: Lucene.Net.Store.IndexInput (and subclasses), Lucene.Net.Store.IndexOutput (and subclasses). Added obsolete extension methods for each type in Lucene.Net.Store namespace for backward compatibility.Lucene.Net.QueryParser: Factored out NLS/IMessage/Message support and changed exceptions to use string messages so end users can elect whether or not to use .NET localization, as is possible with any other .NET exception type.Lucene.Net.QueryParsers.Flexible.Messages: Removed entire namespace, as we have refactored to use .NET localization rather than NLSLucene.Net.Util: Removed BundleResourceManagerFactory and IResourceManagerFactory, as these were only to support NLS. The new approach to localizing messages can be achieved by registering QueryParserMessages.SetResourceProvider(SomeResource.ResourceManager, SomeOtherResource.ResourceManager) at application startup using any ResourceManager instance or designer-generated resource's ResourceManager property.Lucene.Net.Support.Time in favor of J2N.Time. Replaced all calls (except Lucene.Net.Tests.Search.TestDateFilter) that were Environment.TickCount and Time.CurrentTimeMilliseconds() to use Time.NanoTime() / Time.MillisecondsPerNanosecond for more accurate results. This may break some concurrent applications that are synchronizing with Lucene.NET components using Environment.TickCount.Lucene.Net.Highlighter.VectorHiglight.ScoreOrderFragmentsBuilder.ScoreComparer: Implemented singleton pattern so the class can only be used via the Default property.Lucene.Net.QueryParser.Flexible.Core.Nodes.IQueryNode: Added RemoveChildren() method from Lucene 8.8.1 to fix broken RemoveFromParent() method behavior (applies patch LUCENE-5805). This requires existing IQueryNode implementations to implement RemoveChildren() and TryGetTag().Lucene.Net.QueryParser.Flexible.Core.Nodes.IQueryNode: Added TryGetTag() method to simplify looking up a tag by name.Lucene.Net.Analysis.Stempel.Egothor.Stemmer.MultiTrie: Changed protected m_tries field from List<Trie> to IList<Trie>
Lucene.Net.Search.BooleanQuery: Changed protected m_weights field from List<Weight> to IList<Weight>
Lucene.Net.Search.DisjunctionMaxQuery: Changed protected m_weights field from List<Weight> to IList<Weight>
Lucene.Net.Grouping.GroupingSearch::GroupByFieldOrFunction<TGroupValue>(): Fixed casting bug of allGroupsCollector.Groups by changing the cast to ICollection instead of IList.appsettings.json file optional. This was causing a fatal FileNotFoundException after installing lucene-cli without adding an appsettings.json file.Lucene.Net.Codecs.SimpleText.SimpleTextStoredFieldsWriter + Lucene.Net.Codecs.SimpleText.SimpleTextTermVectorsWriter: Fixed Abort() methods to correctly swallow any exceptions thrown by Dispose() to match the behavior of Lucene 4.8.0.Lucene.Net NuGet does not compile under Visual Studio 2017. Downgraded Lucene.Net.CodeAnalysis.CSharp and Lucene.Net.CodeAnalysis.VisualBasic from .NET Standard 2.0 to .NET Standard 1.3 to fix.IsTokenaized as true and NumericType as NumericType.NONE, and to set to NumericType.NONE (rather than null) if the field has no numeric type.Lucene.Net.Analysis.Common.Util.CharArraySet: Throw NotSupportedException when the set is readonly, not InvalidOperationException to match .NET collection behaviorLucene.Net.Codecs.Bloom.BloomFilteringPostingsFormat::FieldsConsumer(): Throw NotSupportedException rather than InvalidOperationException
Lucene.Net.Codecs.Lucene42.Lucene42DocValuesProducer::LoadNumeric(): Throw AssertionError rather than InvalidOperationException
Lucene.Net.Store.CompoundFileDirectory::ReadEntries(): throw AssertionError rather than InvalidOperationException
Lucene.Net.Util.Packed.DirectPackedReader::Get(): Throw AssertionError rather than InvalidOperationException
Lucene.Net.Facet: Throw InvalidOperationException rather than ThreadStateException
Lucene.Net.Grouping.BlockGroupingCollector: Throw NotSupportedException rather than InvalidOperationException
Lucene.Net.Tests.Index.TestUniqueTermCount: Throw NotSupportedException rather than InvalidOperationException
float.MinValue and double.MinValue to float.Epsilon and double.Epsilon because those are the .NET equivalent constants to Float.MIN_VALUE and Double.MIN_VALUE in JavaLucene.Net.Search.ControlledRealTimeReopenThread - Fixed time calculation issue that was causing wait to happen for unusually long time periods.Lucene.Net.Tests.Search.TestMultiThreadTermVectors: Removed stray [Test] attribute that was causing extra overhead with no benefitLucene.Net.Support.WeakDictionary: Changed WeakKey to use WeakReference<T> instead of WeakReference to avoid problems with garbage collectionLucene.Net.Highlighter.VectorHiglight.ScoreOrderFragmentsBuilder.ScoreComparer: Implemented singleton pattern so the class can only be used via the Default property.Lucene.Net.Index.IndexReader: Use ConditionalWeakTable<TKey, TValue>/WeakDictionary<TKey, TValue> to ensure dead elements are pruned and garbage collectedLucene.Net.Index.TestIndexWriter::TestThreadInterruptDeadlock() and Lucene.Net.Index.TestIndexWriter::TestTwoThreadsInterruptDeadlock() that were failing due to a difference in .NET Thread.Interrupt() behavior. In Java, an InterruptedException is never thown from synchronized (this) (the equivalent of lock (this)), but .NET may throw ThreadInterruptedException in cases where there is contention on the lock. The patch fixes our immediate problem of these 2 tests failing and deadlocks occurring, but is only a partial fix. See #526 for an explanation.Lucene.Net.Tests.Suggest.Suggest.Analyzing.TestFreeTextSuggester::TestRandom(): LookupResult calculation in the test was using different order of parentheses than the production code. This bug existed in Java, but apparently the order makes no difference on that platform. This test was getting a false positive because it was using List<T>.ToString() to make the result comparison, which J2N's List<T> corrects.Lucene.Net.Util.NumericUtils::SingleToSortableInt32() method to compare the raw bits for equality. This change doesn't impact performance or behavior of the application as using an approximate float comparison would.
Lucene.Net.Expressions.TestExpressionSorts::TestQueries()
Lucene.Net.Sandbox.TestSlowFuzzyQuery::TestTieBreaker()
Lucene.Net.Sandbox.TestSlowFuzzyQuery::TestTokenLengthOpt()
Lucene.Net.Search.TestBooleanQuery::TestBS2DisjunctionNextVsAdvance()
Lucene.Net.Search.TestFuzzyQuery::TestTieBreaker()
Lucene.Net.Search.TestSearchAfter::TestQueries()
Lucene.Net.Search.TestTopDocsMerge::TestSort_1()
Lucene.Net.Search.TestTopDocsMerge::TestSort_2()
Lucene.Net.Tartarus.Snowball: Refactored to use Func<bool> instead of a Reflection call to execute stemmer code as in the original C# port: https://github.com/snowballstem/snowball
GroupingSearch tests to demonstrate usageappsettings.json file with the default settingsLucene.Net.Tests.Cli: Added InstallationTest to install lucene-cli and run it to ensure it can be installed and has basic functionality.Lucene.Net.Analysis.OpenNLP: Updated to OpenNLP 1.9.1.1 and added strong naming support.Lucene.Net.IndexWriter.ReaderPool: Swapped in ConcurrentDictionary<TKey, TValue> instead of Dictionary<TKey, TValue> to take advantage of the fact ConcurrentDictionary<TKey, TValue> supports deleting while iterating.Lucene.Net.Queries.Mlt.MoreLikeThis: Fixed boxing issues with RetrieveTerms() and RetrieveInterestingTerms() methods by changing object[] to a class named ScoreTerm (same refactoring as Lucene 8.2.0).Lucene.Net.CodeAnalysis: Added Version.props file to make it possible to manually bump the assembly number by one revision on any code change (VS requires this, see: dotnet/roslyn#4381 (comment)).Lucene.Net.TestFramework: Removed FEATURE_RANDOMIZEDCONTEXT and deleted all files related to Java randomizedtesting that were partially ported bits of its test runner.Lucene.Net.TestFramework, Lucene.Net.Support: Added [DebuggerStepThrough] attribute to all assertion methods so the debugger stops in the code that fails the assert not inside of the assert method (affects only internal Lucene.NET development).Lucene.Net.Support.ExceptionHandling: Added ExceptionExtensions class with methods named after the Java exception types so future porting efforts can use similar catch blocks with the same behavior as in Java (i.e. catch (Exception e) when (e.IsIllegalStateException()).Lucene.Net.Support.ExceptionHandling: Added exception classes with the same names as Java exception types so future porting efforts can use similar catch blocks with the same behavior as in Java (i.e throw IllegalStateException.Create("This is the message")).Lucene.Net.Tests.AllProjects project containing tests to confirm that all exceptions thrown by .NET and NUnit are correctly identified by ExceptionExtensions methods.Lucene.Net.Documents.FieldType::Freeze(): Changed from void return to return this FieldType to allow direct chaining of the method in field initializers. Chained the Freeze() method in all static field initializers of Field subclasses to eliminate extra helper load methods. Marked BinaryDocValuesField.fType static field obsolete and added TYPE static field (as it was in Lucene).Lucene.Net.QueryParsers.Flexible.Core.Messages: Redesigned QueryParserMessages.cs so that it is just a facade around a IResourceProvider implementation that provides the actual fallback logic. Added a QueryParserResourceProvider implementation that can be passed zero to many ResourceProvider instances to override and optionally localize the default resource messages.BytesRef. In addition to the decimal bytes values it now shows the BytesRef as a UTF8 string. If the BytesRef is not a UTF8 string that representation will be the string's fingerprint signature.Lucene.Net.Grouping: Fix SonarQube's "Any() should be used to test for emptiness" / Code SmellLucene.Net.Support: Factored out Number class in favor of using J2N's parsers and formattersLucene.Net.Highlighter: Implemented IFormattable and added culture-aware ToString() overload to WeightedPhraseInfo and WeightedFragInfo
Lucene.Net.Highlighter: Use RemoveAll() extension method rather than allocating separate collections to track which enumerated items to remove.Parse/TryParse that accept offsets rather than allocating substringsLucene.Net.Support.ListExtensions::SubList(): Factored out in favor of J2N's List<T>.GetView() method. Many calls to List<T>.GetRange() were updated to J2N.Collections.Generic.List<T>.GetView(), which reduces unnecessary allocations.Lucene.Net.Util.UnicodeUtil::ToString(): Updated to cascade the call to J2N.Character.ToString() which has been optimized to use the stack for small strings.FEATURE_THREAD_YIELD and FEATURE_THREAD_PRIORITY, changed all applicable calls from Thread.Sleep(0) back to Thread.Yield() as they were in Lucene.System.Collections.Generic.List<T> to J2N.Collections.Generic.List<T>, which is structurally equatable and structurally formattable.Lucene.Net.Util.ListExtensions: Added optimized path for J2N.Collections.Generic.List<T> in AddRange() and Sort() extension methodsnet6.0
This release contains bug fixes and minor performance improvements
The lucene-cli tool requires an appsettings.json file, but none was shipped. Upon running lucene on the command line, the following error will be presented:
F:\Projects\lucenenet>lucene
Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'.
at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info)
at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload)
at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load()
at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
Adding a text file named appsettings.json to the location specified in the error message with opening and closing brackets will prevent the exception.
{
}
IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.867 (2004/?/20H1)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.104
[Host] : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00005 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00006 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00007 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00008 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00009 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00010 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00011 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00012 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00013 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00014 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
InvocationCount=1 IterationCount=15 LaunchCount=2
UnrollFactor=1 WarmupCount=10
| Method | Job | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|
| IndexFiles | 4.8.0-beta00005 | 905.6 ms | 131.82 ms | 197.30 ms | 43000.0000 | 8000.0000 | 7000.0000 | 220.99 MB |
| IndexFiles | 4.8.0-beta00006 | 707.1 ms | 18.57 ms | 26.04 ms | 44000.0000 | 8000.0000 | 7000.0000 | 220.99 MB |
| IndexFiles | 4.8.0-beta00007 | 712.2 ms | 16.45 ms | 23.06 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.04 MB |
| IndexFiles | 4.8.0-beta00008 | 785.7 ms | 17.37 ms | 25.46 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.54 MB |
| IndexFiles | 4.8.0-beta00009 | 824.9 ms | 32.86 ms | 48.17 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.34 MB |
| IndexFiles | 4.8.0-beta00010 | 789.6 ms | 16.40 ms | 24.04 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.35 MB |
| IndexFiles | 4.8.0-beta00011 | 805.4 ms | 21.26 ms | 31.82 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.37 MB |
| IndexFiles | 4.8.0-beta00012 | 827.8 ms | 13.95 ms | 20.89 ms | 56000.0000 | 7000.0000 | 6000.0000 | 287.03 MB |
| IndexFiles | 4.8.0-beta00013 | 793.6 ms | 13.63 ms | 19.55 ms | 44000.0000 | 8000.0000 | 7000.0000 | 220.22 MB |
| IndexFiles | 4.8.0-beta00014 | 812.0 ms | 21.97 ms | 30.79 ms | 44000.0000 | 8000.0000 | 7000.0000 | 220.29 MB |
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.867 (2004/?/20H1)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.104
[Host] : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00005 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00006 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00007 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00008 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00009 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00010 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00011 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00012 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00013 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
4.8.0-beta00014 : .NET Core 3.1.13 (CoreCLR 4.700.21.11102, CoreFX 4.700.21.11602), X64 RyuJIT
IterationCount=15 LaunchCount=2 WarmupCount=10
| Method | Job | Mean | Error | StdDev | Median | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|---|
| SearchFiles | 4.8.0-beta00005 | 421.1 ms | 111.47 ms | 163.38 ms | 326.3 ms | 18000.0000 | 1000.0000 | - | 82.11 MB |
| SearchFiles | 4.8.0-beta00006 | 349.8 ms | 24.03 ms | 35.97 ms | 338.9 ms | 18000.0000 | 1000.0000 | - | 82.11 MB |
| SearchFiles | 4.8.0-beta00007 | 333.6 ms | 17.36 ms | 25.98 ms | 336.8 ms | 18000.0000 | 1000.0000 | - | 81.9 MB |
| SearchFiles | 4.8.0-beta00008 | 191.7 ms | 7.17 ms | 10.51 ms | 187.9 ms | 17000.0000 | 1000.0000 | - | 80.13 MB |
| SearchFiles | 4.8.0-beta00009 | 186.6 ms | 8.56 ms | 12.55 ms | 184.0 ms | 17000.0000 | 1000.0000 | - | 80.13 MB |
| SearchFiles | 4.8.0-beta00010 | 182.2 ms | 6.69 ms | 9.16 ms | 181.6 ms | 17000.0000 | 1000.0000 | - | 79.85 MB |
| SearchFiles | 4.8.0-beta00011 | 208.9 ms | 17.73 ms | 26.54 ms | 207.9 ms | 17000.0000 | 1000.0000 | - | 79.85 MB |
| SearchFiles | 4.8.0-beta00012 | 192.3 ms | 10.99 ms | 16.46 ms | 187.8 ms | 18000.0000 | 1000.0000 | - | 81.11 MB |
| SearchFiles | 4.8.0-beta00013 | 177.4 ms | 7.74 ms | 11.59 ms | 175.1 ms | 14000.0000 | 1000.0000 | - | 65.78 MB |
| SearchFiles | 4.8.0-beta00014 | 172.7 ms | 5.93 ms | 8.88 ms | 168.9 ms | 14000.0000 | 1000.0000 | - | 65.78 MB |
TaskMergeScheduler, a merge scheduler that was added to support .NET Standard 1.xLucene.Net.TestFramework: Removed the public LuceneTestCase.ConcurrentMergeSchedulerFactories classLucene.Net.Index.DocTermOrds: Fixed issue with enumerator (OrdWrappedTermsEnum) incorrectly returning true when the value is null.Lucene.Net.Analysis.Common: Fixed TestRollingCharBuffer::Test() to prevent out of memory exceptions when running with Verbose enabledHunspellStemFilter does not work with zero affixLucene.Net.Facet.Taxonomy.CachedOrdinalsReader: Fixed synchronization issue between adding new items to the cache and reading RamBytesUsed methodLucene.Net.Spatial.Util.ShapeFieldCacheProvider: Fixed atomicity issue with loading the cache by using Lazy<T>. Lucene.Net.TestFramework.Support.Confguration.TestConfigurationFactory: Use Lazy<T> to ensure the configurationCache.GetOrAdd() factory is atomic.Lucene.Net.TestFramework.Search.ShardSearchingTestBase: Fixed possible KeyNotFoundExceptionwhen getting the value fromcollectionStatisticsCache`Lucene.Net.Spatial.Prefix.PrefixTreeFactory: Use Lazy<T> in ConcurrentDictionary to make the valueFactory atomic.Lucene.Net.Benchmark.ByTask.Feeds.SpatialDocMaker: Since Dictionary<TKey, TValue>.this[key]is not marked virtual in .NET, subclassingDictionary<string, string>is not a valid approach. So we implementIDictionary<string, string>` instead.Lucene.Net.Facet - Reverted locking in to the state it was in Lucene 4.8.1, however we are still making use of ReaderWriterLockSlim to improve read performance of caches. Also, removed the 1 second lock timeout from Cl2oTaxonomyWriterCache.[AwaitsFix] attribute to known failing testsdotnet CLI commands dotnet build and dotnet test rather than dotnet msbuild and dotnet vstest
Lucene.Net.Tests.Facet
Console.WriteLine() work in unit tests.Lucene.Net.Tests.Join
Lucene.Net.Tests.Classification
Lucene.Net.Classification: Removed leading underscore from private/internal member variablesazure-pipelines.yml: Added RunX86Tests option to explicitly enable x86 tests without having to run a full nightly buildLucene.Net.Tests.Codecs
AnonymousClass
private
Lucene.Net.Tests.Queries
FEATURE_CLONEABLE and the MSBuild property IncludeICloneable
Lucene.Net.Tests.Expressions
Lucene.Net.Util.VirtualMethod: Removed unnecessary call to Convert.ToInt32()
Lucene.Net.Util.AttributeSource: Restored comment from Lucene indicating it doesn't matter if multiple threads compete to populate the ConditionalWeakTable.Lucene.Net.Analysis.OpenNLP.Tools.OpenNLPOpsFactory: Simplified logic by using GetOrAdd() instead of TryGetValue. Lucene.Net.TestFramework.Util (LuceneTestCase + TestUtil): Refactored the CleanupTemporaryFiles() method to be more in line with the original Java implementation, including not allowing new files/directories to be added to the queue concurrently with the deletion process. Lucene.Net.Join.ToParentBlockJoinCollector: Changed from ConcurrentQueue<T> to Queue<T> because we are dealing with a collection declared within the same method so there is no reason for the extra overhead. Lucene.Net.Tests.Suggest.Spell.TestSpellChecker: Replaced ConcurrentBag<T> with ConcurrentQueue<T> because we need to be sure the underlying implementation guarantees order and the extra call to Reverse() was just slowing things down. Lucene.Net.TestFramework.Search.ShardSearchingTestBase: Display the contents of the collection to the console using Collections.ToString().Lucene.Net.Search.SearcherLifetimeManager: Added comment to indicate the reason we useLazy<T>` is to make the create operation atomic. Directory.Build.Targets: Added FEATURE_DICTIONARY_REMOVE_CONTINUEENUMERATION so we can support this feature in .NET 5.x + when we add a target.Lucene.Net.Search.Suggest.Fst.FSTCompletion: Use Stack<T> rather than List<T>.Reverse(). Also, removed unnecessary lock in CheckExistingAndReorder(), as it is only used in a single thread at a time.Lucene.Net.Search.Suggest.SortedInputEnumerator: Removed unnecessary call to Reverse() and allocation of HashSet<T>
Lucene.Net.Search.FieldCacheImpl: Reverted locking back to the state of Lucene 4.8.0.FEATURE_THREAD_INTERRUPT since all supported targets now support thread interrupts. Note also that Lucene depends on thread interrupts to function properly, so disabling this feature would be invalid.JavaDocToMarkdownConverter tool from the normal build workflow of the API docs. This frees us up to update the "namespace" documentation with .NET-specific information and code examples.Lucene.Net.Documents.Document: Added culture-sensitive overloads of GetValues(), Get() and GetStringValue() that accept format and IFormatProvider and implemented IFormattable on Document and LazyDocument.Commit() method to AnalyzingInfixSuggester (from LUCENE-5889)This release contains important bug fixes and performance enhancements.
The lucene-cli tool requires an appsettings.json file, but none was shipped. Upon running lucene on the command line, the following error will be presented:
F:\Projects\lucenenet>lucene
Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'.
at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info)
at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload)
at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load()
at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
Adding a text file named appsettings.json to the location specified in the error message with opening and closing brackets will prevent the exception.
{
}
IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.
J2N versions prior to version 2.0.0-beta-0012 had an infinite recursion bug on Xamarin.Android which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.630 (2004/?/20H1)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.100
[Host] : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00005 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00006 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00007 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00008 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00009 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00010 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00011 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00012 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00013 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
InvocationCount=1 IterationCount=15 LaunchCount=2
UnrollFactor=1 WarmupCount=10
| Method | Job | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|
| IndexFiles | 4.8.0-beta00005 | 628.1 ms | 8.41 ms | 12.05 ms | 43000.0000 | 8000.0000 | 7000.0000 | 220.82 MB |
| IndexFiles | 4.8.0-beta00006 | 628.3 ms | 13.19 ms | 19.33 ms | 44000.0000 | 8000.0000 | 7000.0000 | 220.67 MB |
| IndexFiles | 4.8.0-beta00007 | 617.2 ms | 8.44 ms | 11.83 ms | 44000.0000 | 8000.0000 | 7000.0000 | 220.73 MB |
| IndexFiles | 4.8.0-beta00008 | 620.6 ms | 5.62 ms | 8.41 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.06 MB |
| IndexFiles | 4.8.0-beta00009 | 632.8 ms | 12.57 ms | 18.43 ms | 44000.0000 | 8000.0000 | 7000.0000 | 220.95 MB |
| IndexFiles | 4.8.0-beta00010 | 862.3 ms | 51.13 ms | 74.95 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.22 MB |
| IndexFiles | 4.8.0-beta00011 | 636.5 ms | 11.06 ms | 15.87 ms | 44000.0000 | 8000.0000 | 7000.0000 | 221.09 MB |
| IndexFiles | 4.8.0-beta00012 | 668.8 ms | 14.78 ms | 21.66 ms | 56000.0000 | 7000.0000 | 6000.0000 | 286.63 MB |
| IndexFiles | 4.8.0-beta00013 | 626.7 ms | 7.78 ms | 10.91 ms | 43000.0000 | 8000.0000 | 7000.0000 | 219.8 MB |
BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.630 (2004/?/20H1)
Intel Core i7-8850H CPU 2.60GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=5.0.100
[Host] : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00005 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00006 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00007 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00008 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00009 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00010 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00011 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00012 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
4.8.0-beta00013 : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
IterationCount=15 LaunchCount=2 WarmupCount=10
| Method | Job | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
|---|---|---|---|---|---|---|---|---|
| SearchFiles | 4.8.0-beta00005 | 274.8 ms | 7.01 ms | 10.28 ms | 18000.0000 | 1000.0000 | - | 82.12 MB |
| SearchFiles | 4.8.0-beta00006 | 283.4 ms | 7.78 ms | 11.64 ms | 18000.0000 | 1000.0000 | - | 82.13 MB |
| SearchFiles | 4.8.0-beta00007 | 291.5 ms | 8.91 ms | 13.33 ms | 18000.0000 | 1000.0000 | - | 81.9 MB |
| SearchFiles | 4.8.0-beta00008 | 162.3 ms | 5.50 ms | 8.23 ms | 17000.0000 | 1000.0000 | - | 80.13 MB |
| SearchFiles | 4.8.0-beta00009 | 165.6 ms | 2.61 ms | 3.90 ms | 17000.0000 | - | - | 80.13 MB |
| SearchFiles | 4.8.0-beta00010 | 159.4 ms | 2.84 ms | 4.17 ms | 17000.0000 | 1000.0000 | - | 79.85 MB |
| SearchFiles | 4.8.0-beta00011 | 160.8 ms | 1.93 ms | 2.77 ms | 17000.0000 | 1000.0000 | - | 79.85 MB |
| SearchFiles | 4.8.0-beta00012 | 169.2 ms | 6.48 ms | 9.49 ms | 18000.0000 | 1000.0000 | - | 81.11 MB |
| SearchFiles | 4.8.0-beta00013 | 161.6 ms | 3.28 ms | 4.80 ms | 14000.0000 | 1000.0000 | - | 65.78 MB |
Lucene.Net.Search.FieldCache: Added interface ICreationPlaceholder and changed CreationPlaceholder class to CreationPlaceHolder<TValue>.Lucene.Net.Store.NativeFSLockFactory: Modified options to allow read access on non-Windows operating systems. This caused the copy constructor of RAMDirectory to throw "The process cannot access the file 'file path' because it is being used by another process" excpetions.Lucene.Net.Util.Automaton.State: Removed Equals() implementation; it was intended to use reference equality as a unique key. This caused random IndexOperationExceptions to occur when using FuzzyTermsEnum/FuzzyQuery.ArgumentException message for all analyzer factories so it will display the dictionary contentsJ2N.Collections.Generic.List<T> so the call to ToString() will automatically list the exception messagesLucene.Net.TestFramework.Analysis.MockTokenizer: Pass the AttributeFactory argument that is provided as per the documentation comment. Note this bug exists in Lucene 4.8.0, also.Lucene.Net.Analysis.Common.Tartarus.Snowball.Among: Fixed MethodObject property to return private field instead of itselfLucene.Net.Document.CompressionTools: Pass the offset and length to the underlying MemoryStream
Microsoft.Extensions.Configuration version to 2.0.0 on .NET Standard 2.0 and 2.1Directory pathsusing statementsDump() method and replaced with Console.WriteLine() for clarityMultiPhraseQuery
Lucene.Net.Search.FieldCacheImpl: Removed unnecessary dictionary lookupFieldCacheImpl casting/boxingLucene.Net.Util.Automaton.SortedInt32Set: Removed unnecessary IEquatable<T> implementations and converted FrozenInt32Set into a struct.Lucene.Net.Util.Bits: Removed unnecessary GetHashCode() method from MatchAllBits and MatchNoBits (didn't exist in Lucene)Lucene.Net.Util.Counter: Changed Get() to Value property and added implicit operator.CreateDirectory() method virtual so that derived classes can provide their own Directory implementation, allowing for benchmarking of custom Directory providers (e.q LiteDB)Debugging.Assert() to use generic parameters and string.Format() to reduce allocations. Use J2N.Text.StringFormatter to automatically format arrays and collections so the processing of converting it to a string is deferred until an assert fails.Lucene.Net..Index: Calling IndexOptions.CompareTo() causes boxing. Added new IndexOptionsComparer class to be used in codecs instead.IndexWriter and subclasses which need more work)TermsEnum subclassesLucene.Net.Index.DocValuesProducer: Optimized checks in AddXXXField() methodsLucene.Net.Index: Changed FieldInfos, FreqProxTermsWriterPerField, IndexWriter, LogMergePolicy, SegmentCoreReaders, and SegmentReader to take advantage of the fact that TryGetValue() returns a booleanFieldCacheImpl delegate capture introduced in #348Lucene.Net.TestFramework: Added Assert.DoesNotThrow() overloadsThis release contains important bug fixes and performance enhancements.
After installation, when upgrading from versions of Lucene.Net 4.8.0-beta00009 or higher may require a restart of all instances of Visual Studio in order to reload the code analysis analyzer.
The lucene-cli tool requires an appsettings.json file, but none was shipped. Upon running lucene on the command line, the following error will be presented:
F:\Projects\lucenenet>lucene
Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'.
at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info)
at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload)
at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load()
at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
Adding a text file named appsettings.json to the location specified in the error message with opening and closing brackets will prevent the exception.
{
}
IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.
J2N versions prior to version 2.0.0-beta-0012 had an infinite recursion bug on Xamarin.Android which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.
Lucene.Net.Facet: Renamed LRUHashMap > LruDictionary. Changed all members to be virtual to allow users to provide their own LRU cache.Lucene.Net.Facet.FacetsConfig: Removed ProcessSSDVFacetFields from public API (as was done in Lucene), avoid lock (this)Lucene.Net.Facet.TaxonomyReader: Changed DoClose() to Dispose(bool) and implemented proper dispose pattern. Avoid lock (this).Lucene.Net.Facet.WriterCache: Renamed NameInt32CacheLRU > NameIntCacheLru, NameHashInt32CacheLRU > NameHashInt32CacheLru. Refactored to utilize a generic type internally using composition to avoid boxing/unboxing without exposing the generic closing type publicly. Added public INameInt32CacheLru as a common interface between NameIntCacheLru and NameHashInt32CacheLru.Lucene.Net.Facet.Taxonomy.TaxonomyReader: Restructured ChildrenIterator into ChildrenEnumerator
Lucene.Net.Facet.Taxonomy.CategoryPath: Changed FullPathLength from a method to a propertyLucene.Net.Facet.DrillSideways: Changed ScoreSubDocsAtOnce from a method to a propertyLucene.Net.Facet: Refactored OrdAndValue into a generic struct that can be used in both TopOrdAndSingleQueue and TopOrdAndInt32Queue. Added Insert method to Util.PriorityQueue<T> to allow adding value types without reading the previous value for reuse.Lucene.Net.Analysis.Common.Miscellaneous.CapitalizationFilter: Changed default behavior to use invariant culture instead of the current thread's culture to match Lucene, which seems more natural when using filters inside of analyzers. This also fits more in line with how other filters are selected.Lucene.Net.Analysis.Compound.Hyphenation.TernaryTree: Renamed Iterator > Enumerator, Keys() > GetEnumerator()
Lucene.Net.Benchmarks.ByTask.Feeds.DirContentSource: Renamed Iterator > Enumerator
NGramTokenAnymousInnerClassHelper::IsTokenChar(int) that was causing surrogate pairs to fail in the TestUTF8FullRange() tests of NGramTokenizerTest and EdgeNGramTokenizerTest
ArgumentExceptions being thrown from char.ConvertToUtf32(string, int) by reverting back to CodePointAt() method in TestCharTokenizers.TestCrossPlaneNomalization().Lucene.Net.QueryParser.Surround.Query.ComposedQuery::MakeLuceneSubQueriesField(): Added missing using block on enumeratorLucene.Net.Analysis.Common: Fixed classes that were originally using invariant culture to do so again. J2N's Character class default is to use the current culture, which had changed from the prior Character class from Lucene.Net.Support that used invariant culture. Fixes TestICUFoldingFilter::TestRandomStrings().Lucene.Net.Spatial.Util.ShapeFieldCache: Removed unnecessary array allocationLucene.Net.TestFramework: Fixed LineFileDocs to read byte by byte the same way that Lucene does, except using a BufferedStream to improve performance.Lucene.Net.TestFramework: Fixed NightlyAttribute, WeeklyAttribute, AwaitsFixAttribute, and SlowAttribute so they work at the class levelLucene.Net.Analysis.Icu.Segmentation.ICUTokenizer: Corrected call to ICU4N.UChar.IsWhiteSpace() rather than System.Char.IsWhiteSpace(), which may return different results.Lucene.Net.TestFramework.Search.SearchEquivalenceTestBase: Fixed exception when using OpenBitSet.FastGet() instead of OpenBitSet.Get(), since the size of the bit set is unknown.Lucene.Net.Index.DocumentsWriterFlushControl: Fixed issue due to misbehaving locking on Monitor.TryEnter(), the code was restructured to disallow any thread that doesn't have a lock into InternalTryCheckoutForFlush() so the threads do note compete for a lock.Lucene.Net.Facet: Fixed null reference exception in DrillSidewaysScorer from patch in Lucene 4.10.4 https://issues.apache.org/jira/browse/LUCENE-6001
Lucene.Net.Facet.Taxonomy.WriterCache.Cl2oTaxonomyWriterCache: Fixed locking on Dispose() method and made it safe to call dispose multiple timesLucene.Net.ICU: Added locking to ICUTokenizer to only allow a single thread to manipulate the BreakIterator at a time. This is a temporary fix to get the tests to pass until a solution is found for making BreakIterator threadsafe.Lucene.Net.Replicator: Fixed an issue in IndexInputStream that meant the read method could return a number larger than what was passed in for read count or what the buffer could hold, it should instead return the total number of bytes that was read into the buffer, which logically can't be bigger than the buffer it self.Lucene.Net.Tests.Index.TestIndexWithThreads::TestRollbackAndCommitWithThreads(): Must catch and ignore AssertionException, as was done in LuceneLucene.Net.Search.TopScoreDocCollector: Disabled optimizations on .NET Framework because of float comparison failures on x86 in Release mode. Fixes TestSearchAfter::TestQueries(), TestTopDocsMerge::TestSort_1(), TestTopDocsMerge::TestSort_2().Lucene.Net.Sandbox.Queries.SlowFuzzyTermsEnum: Disabled optimizations on .NET Framework because of float comparison failures on x86 in Release mode. Fixes TestTokenLengthOpt().Lucene.Net.Search.FuzzyTermsEnum: Disabled optimizations for Accept() method on .NET Framework because of float comparison failures on x86 in Release mode. Fixes TestTokenLengthOpt().J2N.BitConversion that were calling the overload that normalizes NaN when they should have been calling the raw bit conversion instead (as was done in Lucene).Lucene.Net.Configuration: Removed the IConfigurationRoot interface from the ConfigurationRoot class when targeting a version of Microsoft.Extensions.Configuration less than 2.0. This will allow the end user to upgrade Microsoft.Extensions.Configuration seamlessly to versions 2.0 or higher.Lucene.Net.CodeAnalysis: Separated CSharp and VisualBasic into different assemblies to prevent cross-language dependency issues when using analyzersFSTTester to delete while iterating forward instead of using .ElementAt() to iterate in reverse, which takes about 3x longerLucene.Net.Facet.Taxonomy.WriterCache.NameInt32CacheLRU: Changed from Dictionary to ConcurrentDictionary so we can delete items from the cache while forward iterating through it.Lucene.Net.Index.FieldInfos: Changed Builder.FieldInfo() method to TryGetFieldInfo() to optimize check for valueICollection<T>.ToArray() extension method that uses ICollection<T>.CopyTo(), which takes precedence over the LINQ IEnumerable<T>.ToArray() extension method. Benchmarks show about a 1/3 increase in performance.Lucene.Net.Support.IO.FileSupport::CreateTempFile(): Optimized the check for invalid charactersDirectory.Build.props: Disabled warnings for features that require .NET Standard 2.1FirstOrDefault(), LastOrDefault(), Skip(), First(), and Last()
Lucene.Net.Support.ListExtensions: Factored out BinarySearch in favor of implementation from J2NLucene.Net.Suggest.FreeTextSuggester: Converted from SubList().Clear() to RemoveRange()
LineFileDocs to unzip the file to a temp directory once per test run instead of using a MemoryStream to pick a random line from the file on each test. This significantly improves performance of many of the tests.Lucene.Net.Analysis.Icu.Segmentation.ScriptIterator: Removed static constructor and initialized static state inlineAnalyzer classes to using the Analyzer.NewAnonymous() method to declare Analyzers inline.Lucene.Net.Tests.Util.TestCollectionUtil: Optimized by using array instead of list for sorting testsLucene.Net.Util: Switched implementation of DisposableThreadLocal with that from RavenDB, with permission from its maintainers (https://issues.apache.org/jira/browse/LUCENENET-640?focusedCommentId=17033146&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17033146). The new implementation improves GC during several operations.Lucene.Net.TestFramework.Util.LuceneTestCase: Removed TaskMergeScheduler completely from random testingBytesRef instances outside of the loops that they were nested in so they can be reused in each iteration (as was done in Lucene)Lucene.Net.Facet.Taxonomy.CachedOrdinalsReader: Refactored locking to make reads more efficientLucene.Net.Facet.Taxonomy.Directory.DirectoryTaxonomyReader: Refactored to use ReaderWriterLockSlim to make reads more efficientLucene.Net.Facet.Taxonomy.Directory.DirectoryTaxonomyWriter: Refactored locking for better efficiencyLucene.Net.Facet.Taxonomy.WriterCache.Cl2oTaxonomyWriterCache: Added proper dispose patternLucene.Net.Facet.Taxonomy.WriterCache.LruTaxonomyWriterCache: Added proper dispose patternLucene.Net.Facet.Taxonomy.Directory.TaxonomyIndexArrays: Changed to use LazyInitializer and avoid lock (this)Lucene.Net.Tests.Facet: Convert int to string in the invariant cultureLucene.Net.Analysis.ICU: Updated Segmentation files to Lucene 8.6.1 to account for the latest features of ICULucene.Net.Util.AttributeSource: Eliminated unnecessary try catch and made more efficient by using TryGetValue instead of ContainsKey followed by a lookupLucene.Net.Util: Streamlined DefaultAttributeFactory to make the get/update process of creating an attribute WeakReference atomicThreadState allocator during indexing. Technically, this is something from releases/lucene-solr/4.8.1, but profiling indicates it makes a huge difference in multithreaded scenariosNETSTANDARD.FEATURE_STACKTRACE is enabled rather than disabled when the System.Diagnositcs.StackTrace class is available.Lucene.Net.Constants: Refactored to use System.Runtime.InteropServices.RuntimeInformation on .NET FrameworkLucene.Net.Expressions: Eliminated .NET settings file and reused JavascriptCompiler.properties file in .NET Framework so we don't have to branch for different target platforms. Simplified reading the settings by using J2N PropertyExtensions.Lucene.Net.Support.AssemblyUtils: restructured to use IEnumerable<T> for deferred executionLucene.Net.Index.Terms/TermsEnum, Lucene.Net.Suggest: Refactored iterators into enumerators. Deprecated the iterators.Lucene.Net.Util.FilterIterator<T>: Converted to FilterEnumerator<T> using a predicate passed into the constructor rather than having to subclass. Deprecated FilterIterator<T>. Swapped only usage in FieldFilterAtomicReader with a LINQ query/yield return, since performance is better.Lucene.Net.Util.MergedIterator<T>: Converted to MergedEnumerator<T> and deprecated MergedIterator<T>
Lucene.Net.Codecs.Memory.DirectDocValuesConsumer: Renamed IteratorAnonymousInnerClassHelper > Enumerator, IterableAnonymousInnerClassHelper > EnumerableAnonymousInnerClassHelper
DeadlockAttribute to identify tests that are known to have threading contention issues and may deadlock during test runsRelease build by using the system property "assert": "true". This is necessary to ensure all of the test conditions are being hit in all builds and to enable more thorough CheckIndex from the command line.Lucene.Net.TestFramework: Added ability to turn off asserts when running tests by ignoring a few tests that require the asserts to be enabled in order to pass. This makes it possible to ensure that Lucene.NET works properly with asserts disabled. This feature didn't exist in Lucene.Lucene.Net.Search.FieldCacheDocIdSet: Added public constructor with predicate parameter for filtering without having to create a subclassThis release contains a critical patch for .NET Framework users that use
Microsoft.Extensions.Configurationhigher than version 1.1.2. See #311 for details.
This release contains impactful performance enhancements.
The lucene-cli tool requires an appsettings.json file, but none was shipped. Upon running lucene on the command line, the following error will be presented:
F:\Projects\lucenenet>lucene
Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'.
at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info)
at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload)
at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load()
at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
Adding a text file named appsettings.json to the location specified in the error message with opening and closing brackets will prevent the exception.
{
}
IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.
J2N versions prior to version 2.0.0-beta-0012 had an infinite recursion bug on Xamarin.Android which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.
Lucene.Net.Analysis.Compound: Changed protected m_tokens field from LinkedList<T> to Queue<T> for better throughputLucene.Net.Configuration: Removed IConfigurationBuilder implementation that prevents .NET Framework users from being able to upgrade to a higher version than 1.1.2. Lucene.Net.TestFramework: Reverted BaseTermVectorsFormatTestCase to use the original 5000 iterations instead of 500. Reverted TestUtil.RandomSimpleString(Random) to default to a maximum string length of 10 instead of 20, which was slowing down several tests. Lucene.Net.TestFramework: Refactored Assert class to use custom comparisons for all members, since NUnit's Assert implementation uses very slow fluent expressions to do comparisons, which are not practical to use inside of tight loops.LinkedList<T> with Queue<T>` where there is a performance advantageCaseInsensitiveComparer by using its singleton static propertySystem.Collections.Generic.Dictionary because indexer of J2N's dictionary is slowerLucene.Net.TestFramework.Util.Fst.FSTTester: Use System.Collections.Generic.Dictionary for better performanceNUnit.Framework.Assert to Lucene.Net.TestFramework.Assert, which are several orders of magnitude fasterLucene.Net.Facet: Optimized DirectoryTaxonomyReader by reducing locking, removing unnecessary casts, and using LazyInitializer for the taxonomy array initializationLucene.Net.Tests.Analysis.Common: Changed Hunspell StemmerTestBase to use more optimized assert to compare arraysLucene.Net.Tests.Support.TestApiConsistency: Added regex filter to exclude public fields from the scanThis release contains impactful performance enhancements.
The lucene-cli tool requires an appsettings.json file, but none was shipped. Upon running lucene on the command line, the following error will be presented:
F:\Projects\lucenenet>lucene
Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'.
at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info)
at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload)
at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load()
at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
Adding a text file named appsettings.json to the location specified in the error message with opening and closing brackets will prevent the exception.
{
}
IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.
J2N versions prior to version 2.0.0-beta-0012 had an infinite recursion bug on Xamarin.Android which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.
Lucene.Net.Suggest: Narrowed return type of Contexts property from IEnumerable<BytesRef> to ICollection<BytesRef> to improve performance for certain operationsLucene.Net.TestFramework.Util.LuceneTestCase: Refactored to correctly load static fields and converted them into static properties. The names were changed to match the conventions of .NET properties.BitSet instead of BitArray so we don't have to resort to slow extension methods for certain operationsLucene.Net.Support.Collections::ToString(): Fixed overloads to write "null" when the collection passed is null rather than throw an exception Lucene.Net.Codecs: Fixed testing condition for BaseTermVectorsFormatTestCase on TermVectorsReaders by throwing InvalidOperationException
Type.GetTypeInfo() and call properties/methods on the Type instance directly.Lucene.Net.Util.AttributeSource: Create built-in attributes directly rather than using Activator.CreateInstance()
Lucene.Net.Util.AttributeSource: Optimized creation of string to identify attribute type based on attribute interface nameLucene.Net.TestFramework: Added overloads of Assert.AreEqual for collections where J2N's aggressive mode can be switched offLucene.Net.TestFramework: Replaced overloads from NUnit.Framework.CollectionAssert with optimized implementations using J2N comparersLucene.Net.TestFramework: Compile expensive string concatenation out of the release build by using System.Diagnostics.Debug.Assert
Lucene.Net.Util.Automaton: Fixed State class to initialize and trim more efficiently.
Fixes the performance of Lucene.Net.Util.Automaton.TestBasicOperations::TestEmptyLanguageConcatenate().Lucene.Net.TestFramework.Search.RandomSimilarityProvider::ToString(): Use StringBuilder for better efficiencyLucene.Net.TestFramework.Util.LuceneTestCase: Cache codecType and similarityName as strings so they don't have to be regenerated for each testLucene.Net.TestFramework: Use Assert.IsFalse() rather than Assert.That()
Lucene.Net.Index.Term: Optimized equality checkingLucene.Net.Util (BytesRef + CharsRef): Implemented IEquatable<T>
Lucene.Net.Tests.Index.TestDocumentsWriterDeleteQueue: Updated comparisons to reduce memory allocationsLucene.Net.TestFramework: Changed ConcurrentMergeSchedulerFactories.Values to only return the TaskMergeScheduler rarely, since it is no longer a default setting and is slowing down testsLucene.Net.TestFramework: Added overloads of Assert.Throws to supply messagesLucene.Net.TestFramework: Use the Microsoft.Extensions.Configuration.EnvironmentVariables provider instead of our custom one.Lucene.Net.Analysis.Tokenizer: Allow enabling "asserts" for testingazure-pipelines.yml: Changed to use pipeline caching instead of build caching for better performanceLucene.Net.Tests (core) - Rearranged how test projects are split so parallel jobs can process them fasterLucene.Net.Tests.Analysis.Phonetic.Language.Bm.PhoneticEnginePeformanceTest::Test(): Changed to use Stopwatch for more accurate timingLucene.Net.Tests: Changed private/internal member variables to consistently use camelCase instead of PascalCaseLucene.Net.Analysis: Removed unnecessary allocations caused by calling ToString() rather than passing the ICharTermAttribute directlyLucene.Net.Analysis.Common: Slight optimization of ToUpper and ToLower methods, which are used in several analyzersLucene.Net.Codecs.SimpleText: Using decimal is 30% faster than using BigInteger for addition and subtractionSlow and Nightly attributes where applicable for better testing performanceLucene.Net.TestFramework: Added logging of system properties to the output of testsLucene.Net.Util.ArrayUtil::GetNaturalComparer<T>(): Use statically cached singleton instance of comparerLucene.Net.TestFramework.RandomExtensions: Added missing overload of NextInt64(long) to choose only max upper boundLucene.Net.TestFramework: Completed implementation of Nightly, Weekly, AwaitsFix and Slow attributesThe lucene-cli tool requires an appsettings.json file, but none was shipped. Upon running lucene on the command line, the following error will be presented:
F:\Projects\lucenenet>lucene
Unhandled exception. System.IO.FileNotFoundException: The configuration file 'appsettings.json' was not found and is not optional. The physical path is 'C:\Users\shad\.dotnet\tools\.store\lucene-cli\4.8.0-beta00010\lucene-cli\4.8.0-beta00010\tools\netcoreapp3.1\any\appsettings.json'.
at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info)
at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload)
at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load()
at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
at Lucene.Net.Cli.Program.Main(String[] args) in D:\a\1\s\src\dotnet\tools\lucene-cli\Program.cs:line 27
Adding a text file named appsettings.json to the location specified in the error message with opening and closing brackets will prevent the exception.
{
}
IMPORTANT: There must be at least opening and closing curly brackets in the file, or it won't be parsed as valid JSON.
J2N versions prior to version 2.0.0-beta-0012 had an infinite recursion bug on Xamarin.Android which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.
Microsoft.Extensions.DependencyInjection to provide custom codecs to the test environment so they can be used during testing of other components.Lucene.Net.ICU to utilize the new UCultureInfo class.This release contains performance improvements and bug fixes.
The
Lucene.Net.Supportnamespace has been phased out, and most of the components that previously existed there have been refactored and moved to J2N, a library to fill in gaps in functionality between the JDK and .NET.
Xamarin.Android which caused fatal crashes in Lucene.NET. Upgrading J2N to 2.0.0-beta-0012 or higher will prevent these crashes from occurring.#239 - Changed Append() overloads of OpenStringBuilder to use startIndex/count rather than start/end to match the .NET convention
#241 - Made public fields into properties in ArabicStemmer
Lucene.Net.Collections: Factored out unmodifiable methods and related classes in favor of J2N's AsReadOnly() extension methods
SWEEP: Lucene.Net.Support: Factored out Equatable, EquatableList, and EquatableSet, and replaced with collections from J2N
SWEEP: Lucene.Net.Support: Factored out Collections.Equals() and Collections.GetHashCode()
Lucene.Net.Support.Collections: Factored out Swap() and Shuffle() in favor of J2N's implementation
Lucene.Net.Support: Factored out IdentityComparer, IdentityHashMap, and IdentityHashSet and used J2N.Runtime.CompilerServices.IdentityEqualityComparer in conjunction with standard Dictionary and HashSet types
Factored out TreeSet and TreeDictionary from Lucene.Net.Support in favor of J2N.Collections.Generic.SortedSet and J2N.Collections.Generic.SortedDictionary
Lucene.Net.Support.PriorityQueue: Factored out in favor of J2N's implementation
Lucene.Net.Support: Refactored ConcurrentHashSet into ConcurrentSet - a wrapper class that can be used to synchronize any set object (ordered or not), similar to how it was done in Java. Changed ordered concurrent set types back to the original type from Lucene.
Lucene.Net.Search.FieldComparer: Replaced CompareTo() calls with JCG.Comparer<T>.Default.Compare(), factoring out Lucene.Net.Support.SignedZeroComparer in the process.
Factored out Character, ICharSequence, StringBuilderCharSequenceWrapper, StringBuilderExtensions, StringCharSequenceWrapper, and most StringExtensions methods in favor of J2N's implementation
Changed semantics of CharTermAttribute.Append() overloads to act more like the .NET StringBuilder class.
null parameter on a single parameter overload will be a no-op instead of appending the text "null".null parameter on a 3 parameter overload will be a no-op if both startIndex and count are 0, otherwise it will throw an ArgumentNullException.Lucene.Net.Support: Factored out GeneralKeyedCollection and AttributeItem in favor of J2N.Collections.Generic.LinkedDictionary
SWEEP: Upgraded to account for breaking changes (AsCharSequence() and BitOperation) in J2N.
Lucene.Net.Support.Collections: Factored out ImplementsGenericInterface() in favor of J2N's implementation
Lucene.Net.Support.Collections: Factored out Singleton() method and used collection initializers instead
Lucene.Net.Support.Collections: Removed NewSetFromMap method and related SetFromMap class
Lucene.Net.Support: Added ConcurrentHashSet class for internal use, and factored out the use ConcurrentSet throughout the project
Removed Lucene.Net.Support.ConcurrentHashMapWrapper from the public API and renamed ConcurrentDictionary
Marked ExceptionToClassNameConventionAttribute, ExceptionToNetNumericConventionAttribute, ExceptionToNullableEnumConventionAttribute, and WritableArrayAttribute internal
Factored out StringExtensions
Factored out DictionaryExtensions
Factored out LurchTable in favor of J2N's implementation
Lucene.Net.Support.Threading: Deleted ThreadLock, DisposableThreadLocalProfiler
Factored out Lucene.Net.Support.AssemblyExtensions in favor of J2N's implementation
Lucene.Net.Support: Marked all types internal
Moved Lucene.Net.Support.SystemConsole to Lucene.Net.Util namespace
Lucene.Net.Support: Moved ExceptionExtensions to Lucene.Net.Util namespace
Lucene.Net.Support.ListExtensions: Moved AddRange, Sort, TimSort, and IntroSort extension methods to Lucene.Net.Util.ListExtensions.
Lucene.Net.Support.NumberFormat: Moved to Lucene.Net.Util namespace
Lucene.Net.TestFramework.Support.JavaCompatibility.AbstractBeforeAfterRule: Moved from Lucene.Net.Support namespace to Lucene.Net.Util
Lucene.Net.TestFramework.Support: Changed namespace of ApiScanTestBase, CultureInfoSupport, and ExceptionSerializationTestBase to Lucene.Net.Util
Lucene.Net.Util.NumberFormat, Lucene.Net.QueryParsers.Flexible.Standard.Config.NumberDateFormat: Changed protected locale field to private, made property named Culture, and changed constructors and methods to use "culture" rather than "locale" for parameter names
Lucene.Net.Support: Moved SystemProperties class to Lucene.Net.Util namespace
Lucene.Net.Benchmark.Support: Moved EnglishNumberFormatExtensions to Lucene.Net.Util namespace
FragNum value on Highlight.TextFragment
Lucene.Net.Search.BooleanClause::Equals(BooleanClause): Fixed potential null reference exception when Query is set to nullFileStream::Flush() in FSIndexOutput::Flush() that was preventing persistence to disk from occuring at the necessary phaseShuffle() method calls to ensure the same psuedo-random sequence is used based on the seedLucene.Net.Index.DocumentsWriterDeleteQueue::DeleteSlice(): Removed extra Debug.Assert() statement that wasn't in Lucene which caused Lucene.Net.Index.TestIndexWriterUnicode::TestEmbeddedFFFF() test to fail when running in debug modesrc/docs folderIComparer<T> implementations with the built in .NET Comparer.Create() factory methodIAppendable on OpenStringBuilder
IndexReader dependenciesWeakIdentityMap in favor of .NET's ConditionalWeakTable
Lucene.Net.Highlighter, Lucene.Net.Tests.Spatial: Swapped in LinkedHashSet from J2N, like in the original Lucene implementationLucene.Net.Support.HashMap in favor of J2N.Collections.Generic.Dictionary
System.Collections.Generic.SortedDictionary in favor of J2N.Collections.Generic.SortedDictionary
Lucene.Net.Analysis.Kuromoji.Dict.UserDictionary: Swapped out C5 TreeDictionary for J2N.Collections.Generic.SortedDictionary
System.Collections.Generic.SortedSet to J2N.Collections.Generic.SortedSet
TreeSet
System.Collections.Generic.HashSet for J2N.Collections.Generic.HashSet
Arrays.AsList(), which was causing both additional operational complexity and unnecessary memory allocationsLucene.Net.Util.Automaton.State: Implemented IEquatable<State>, changed enumerator to a structFEATURE_HASHSET_CAPACITY, since J2N now has the full .NET Core 3.x implementation with a capacity constructorLucene.Net.Util.Fst.ListOfOutputs::Merge(): Streamlined so we don't have so many castsLucene.Net.Util.Fst: Use J2N.Collections.List<T> for the closing type of Outputs<T> to ensure the outputs can be compared for structural equalityLucene.Net.TestFramework.Search.AssertingScorer: Changed to use ConditionalWeakTable/WeakDictionary
FieldCache and its dependent classesInternalsVisibleTo attributes based on ItemGroup/InternalsVisibleTo elementsLucene.Net.Support.DictionaryExtensions: Optimized Put() method, added guard clauses to Put and PutAll
Lucene.Net.Support.DictionaryExtensions: Factored out Load() and Store() methods in favor of J2N's implementationLucene.Net.Analysis.TokenStream: Removed Reflection code that is used to force the end user to make TokenStream subclasses or their IncrementToken() method sealedLucene.Net.Support.IO.FileStreamExtensions::Read(): Moved to StreamExtensions class and optimized to read bytes in bulk instead of one byte at a timeLucene.Net.Tests.Index (TestBagOfPositions + TestBagOfPostings): Fixed performance issue due to converting int to string and string to int in the current cultureLucene.Net.Store (FSDirectory + BufferedIndexOutput): Refactored FSDirectory.FSIndexOutput to utilize the FileStream buffer only, rather than using both a FileStream buffer and the buffer in BufferedIndexOutput.TokenStream subclasses or their IncrementToken() method sealed at design time, rather than running Reflection code at runtime to make this checkThis release contains impactful performance improvements and bug fixes.
NOTE: The
Lucene.Net.Supportnamespace in the coreLucene.Netassembly is being phased out for external use. Much of what used to be in this namespace has been made into first-class components and moved to J2N, a library to fill in gaps in functionality between the JDK and .NET. Please do not add any new dependencies onLucene.Net.Supportin the coreLucene.Netlibrary. Also, please open a new JIRA ticket if you have a dependency on a component fromLucene.Net.Supportthat is not available in J2N.
This release drops support for .NET Standard 1.6 and adds support for .NET Standard 2.1.
Lucene.Net.Codecs (in Lucene.Net library) - Changed the following methods to properties with the same name for API consistency
Lucene.Net.Codecs.DefaultCodecFactory::AvailableServices()
Lucene.Net.Codecs.DefaultDocValuesFormatFactory::AvailableServices()
Lucene.Net.Codecs.DefaultPostingsFormatFactory::AvailableServices()
Lucene.Net.Codecs.Codec::AvailableCodecs()
Lucene.Net.Codecs.DocValuesFormat::AvailableDocValuesFormats()
Lucene.Net.Codecs.PostingsFormat::AvailablePostingsFormats()
Lucene.Net.Analysis.Kuromoji - Changed the following methods to properties named Instance for API consistency
Lucene.Net.Analysis.Kuromoji.Dict.CharacterDefinition::GetInstance()
Lucene.Net.Analysis.Kuromoji.Dict.ConnectionCosts::GetInstance()
Lucene.Net.Analysis.Kuromoji.Dict.TokenInfoDictionary::GetInstance()
Lucene.Net.Analysis.Kuromoji.Dict.UnknownDictionary::GetInstance()
Lucene.Net.Support.Collections::AddAll() - Factored out in favor of ISet<T>.UnionWith()
Lucene.Net.Support.IdentityComparer::ctor() - Factored out in favor of new Default static propertyLucene.Net.Support.DictionaryExtensions::EntrySet() - Factored out because IDictionary<TKey, TValue> is already enumerable and copying its contents to another data structure for the purpose of enumeration is wasteful and unnecessaryLucene.Net.Support.SetExtensions::AddAll() - Factored out in favor of UnionWith(), Lucene.Net.Support.DictionaryExtensions.PutAll(), or AddRange() depending on collection typeLucene.Net.Support.HashMap<TKey, TValue> - Changed behavior of indexer property to throw a KeyNotFoundException if the key doesn't exist. The exception can already be avoided by using the TryGetValue() method.Lucene.Net.Support.Buffer - Factored out in favor of J2N.IO.Buffer
Lucene.Net.Support.ByteBuffer - Factored out in favor of J2N.IO.ByteBuffer
Lucene.Net.Support.LongBuffer - Factored out in favor of J2N.IO.LongBuffer
Lucene.Net.Support.Number - Factored out the following methods in favor of their counterparts in J2N
Signum() > J2N.MathExtensions::Signum()
SingleToInt32Bits() > J2N.BitConversion::SingleToInt32Bits()
SingleToRawInt32Bits() > J2N.BitConversion::SingleToRawInt32Bits()
Int64BitsToDouble() > J2N.BitConversion::Int64BitsToDouble()
DoubleToInt64Bits() > J2N.BitConversion::DoubleToInt64Bits()
DoubleToRawInt64Bits() > J2N.BitConversion::DoubleToRawInt64Bits()
ToBinaryString() > J2N.IntegralNumberExtensions::ToBinaryString()
URShift() > J2N.Numerics.BitOperationExtensions::TripleShift()
ToString(long, radix) > J2N.IntegralNumberExtensions::ToString(long, radix)
BitCount() > J2N.Numerics.BitOperationExtensions::PopCount()
NumberOfLeadingZeros() > J2N.IntegralNumberExtensions::LeadingZeroCount()
NumberOfTrailingZeros() > J2N.IntegralNumberExtensions::TrailingZeroCount()
RotateLeft() > J2N.IntegralNumberExtensions::RotateLeft()
RotateRight() > J2N.IntegralNumberExtensions::RotateRight()
Lucene.Net.Support.Number - Removed the following methods
FlipEndian()
IsNumber()
ToInt64()
Lucene.Net.Support.MathExtensions - Factored out the following methods in favor of their counterparts in J2N
ToDegrees() > J2N.MathExtensions::ToDegrees()
ToRadians() > J2N.MathExtensions::ToRadians()
Lucene.Net.Support.AtomicBoolean - Factored out in favor of J2N.Threading.Atomic.AtomicBoolean
Lucene.Net.Support.AtomicInt32 - Factored out in favor of J2N.Threading.Atomic.AtomicInt32
Lucene.Net.Support.AtomicInt64 - Factored out in favor of J2N.Threading.Atomic.AtomicInt64
Lucene.Net.Support.AtomicObject - Factored out in favor of J2N.Threading.Atomic.AtomicReference
Lucene.Net.Support.AtomicReferenceArray - Factored out in favor of J2N.Threading.Atomic.AtomicReferenceArray
Lucene.Net.Support.Threading.ThreadClass - Factored out in favor of J2N.Threading.ThreadJob
Lucene.Net.Support.Character - Factored out the following methods in favor of their counterparts in J2N
Digit()
ForDigit()
Lucene.Net.Support.StringTokenizer - Factored out in favor of J2N.Text.StringTokenizer
Lucene.Net.Support.CultureContext - Factored out in favor of J2N.Globalization.CultureContext
Lucene.Net.Support.IndexWriterConfigExtensions - Moved to Lucene.Net.Index.Extensions namespaceLucene.Net.Documents.IndexableFieldExtensions - Moved to Lucene.Net.Documents.Extensions namespaceLucene.Net.Documents.DocumentExtensions - Moved to Lucene.Net.Documents.Extensions namespaceLucene.Net.Support.IResourceManagerFactory - Moved to Lucene.Net.Util namespaceLucene.Net.Support.BundleResourceMangerFactory Moved to Lucene.Net.Util namespaceLucene.Net.Tests.Replicator.IndexAndTaxonomyReplicationClientTest::TestConsistencyOnExceptions().Lucene.Net.Tests.Analysis.Common/Analysis/Util/TestCharArrayMap_TestCharArrayMap() - Was failing in Turkish, lowercasing must be done in the invariant culture to match LuceneLucene.Net.Tests.Util.Fst.TestFSTs::TestPrimaryKeys() - Fixed sorting issue that was causing the test to fail with negative valuesLucene.Net.Tests.Util.TestVersionComparer::TestVersions() - Fixed version number conversion to use the invariant culture so the test consistently passesLucene.Net.Tests.Search.TestSearchAfter::TestQueries() - Fixed number-to-string conversion to use the invariant culture so the test consistently passesLucene.Net.Analysis.Common - Fixed to use invariant culture converting numbers and uppercasing/lowercasingLucene.Net.Benchmark - Fixed to use invariant culture converting numbers and uppercasing/lowercasingLucene.Net.Facet - Fixed to use invariant culture for number conversionLucene.Net.Misc - Fixed to use invariant culture for number conversionLucene.Net.Suggest - Fixed to use invariant culture for number conversionLucene.Net.Grouping - Fixed to use invariant culture for number conversionLucene.Net.Spatial - Fixed to use invariant culture for number conversionLucene.Net.Tests.Replicator - Fixed issue with incorrect error message due to missing Dispose() callLucene.Net.Tests.Replicator.LocalReplicatorTest::TestObtainMissingFile() - Added missing catch block for DirectoryNotFoundException
Lucene.Net.Support.Codecs - Fixed initialization locking between parallel tasks. Also added guard clauses and inlined variable declarations.Lucene.Net.Store.NativeFSLockFactory to be thread-safe on non-Windows operating systemsLucene.Net.TestFramework.Index.ThreadedIndexingAndSearchingTestCase - WeakDictionary<TKey, TValue> must be wrapped to make it thread safeLucene.Net.Codecs.Lucene45.Lucene45DocValuesConsumer::MISSING_ORD - Correct value should be -1L to match LuceneLucene.Net.Util.VirtualMethod to its original location in the Lucene.Net assemblyLucene.Net.Analysis.Phonetic - Create culture during static initialization to improve performanceLucene.Net.Index.CheckIndex - Added guard clause to ensure dir is not nullLucene.Net.Support.IO.FileSupport::CreateTempFile() - Simplified error handling to use a when clause to only catch the relevant exceptions (improving thread safety)IDictionary<TKey, TValue> type that is provided will determine how the component behaves.Lucene.Net.Support.BitArrayExtensions::Cardinality() - Replaced implementation with one that benchmarked more than 3x faster, at the cost of a small amount of RAMDirectory.Build.targets fileWeakConditionalTable<TKey, TValue> over WeakDictionary<TKey, TValue> when the features Lucene requires are available.Lucene.Net.Support.Arrays - Changed Equals() and GetHashCode() to be backed by J2N.Collections.ArrayEqualityComparer<T>.OneDimensional for improved performanceLucene.Net.Support.Arrays::ToString() - Changed implementation to the same as Apache Harmony for improved performanceLucene.Net.Support.Character
ToChars() - Cascade to J2N.Character.ToChars(), which benchmarked 5x fasterToUpper() - Cascade to J2N.Character.ToUpper()
ToLower() - Cascade to J2N.Character.ToLower()
IsLetter() - Cascade to J2N.Character.IsLetter()
Lucene.Net.Support.DictionaryExtensions::Load() - Cascade the call to J2N.PropertyExtensions::LoadProperties()
Lucene.Net.Support.DictionaryExtensions::Store() - Cascade the call to J2N.PropertyExtensions::SaveProperties()
lucene-cli to target netcoreapp3.1
PackageIconUrl elementbuild.ps1 - Added testname/framework to test script to show which projects are runningnet451 to net48
netcoreapp2.1 to netcoreapp2.2
netcoreapp3.1
debug artifact for .pdb files so they aren't pushed into our Apache release distribution binary.rat-excludes file for Apache RATLucene.Net.TestFramework on NuGetLucene.Net.Tests.TestFramework (ported compatible tests from Lucene 8.2.0, since Lucene 4.8.0 did not have any tests)Lucene.Net.Codecs.DefaultCodecFactory - Added CustomCodecTypes property that can be used to initialize custom codec types without requiring a subclassLucene.Net.Codecs.DefaultDocValuesFormatFactory - Added CustomDocValuesFormatTypes property that can be used to initialize custom codec types without requiring a subclassLucene.Net.Codecs.DefaultPostingsFormatFactory - Added CustomPostingsFormatTypes property that can be used to initialize custom codec types without requiring a subclassLucene.Net.Analysis.OpenNLP on NuGetLucene.Net.Analysis.Morfologik on NuGet