Tomasnorre Crawler Versions Save

Libraries and scripts for crawling the TYPO3 page tree. Used for re-caching, re-indexing, publishing applications etc.

11.0.8

3 weeks ago

Fixed

  • Fix loading middleware order to make forced indexing work again @cweiske
  • Avoid deprecations in PHP 8.2
  • Ensure that directory for lock file for enabling/disabling crawler exits

12.0.0

3 weeks ago

Added

  • AbstractBackendModuleController
  • BackendModuleController
  • BackendModuleCrawlerLogController
  • BackendModuleCrawlerProcessController
  • BackendModuleStartCrawlingController
  • Support for PHP 8.2
  • Support for PHP 8.3

Changed

  • !!! Switch constants in ProcessQueueCommand from public to private
  • !!! Add interface to CrawlStrategy filename
  • !!! Switch getExtensionIcon() in TcaUtility from protected to private
  • !!! Add Type Hints for id parameters on RequestForm Render method
  • !!! Switch runPollSuccessHooks() in CrawlerInitialization from protected to private
  • !!! Switch isRequestHashMatchingQueueRecord() in FrontendUserAuthenticator from protected to private
  • !!! CrawlerHookInterface doesn't take a CrawlerController as input anymore
  • !!! ResultHandler moved to Controller/Backend/Helper
  • !!! UrlBuilder moved to Controller/Backend/Helper
  • !!! Mark all classes @internal that are not already marked as internal

Fixed

  • If indexed_search is installed activate procInstructions for indexed_search
  • Avoid deprecations in PHP 8.2
  • Prevent undefined array key uid and username in FrontendUserAuthentication
  • Fix loading middleware order to make forced indexing work again @cweiske
  • Change tense in flush command message @cweiske
  • Ensure that directory for lock file for enabling/disabling crawler exits

Deprecated

Functions & Properties

  • CrawlerController->drawURLs_PIfilter()

Removed

  • Obsolete columns (first and last time) from Process Overview
  • !!! Remove constructor form RequestFormInterface
  • !!! Support for TYPO3 10
  • !!! Support for TYPO3 11 (Only in main branch, v12.0.0)
  • !!! PHP support for <8.0
  • !!! PHP support for <8.1 (Only in main branch, v12.0.0)
  • Remove CLI_hooks from ProcessQueueCommand
  • !!! ClassAliasMap & LegacyClassesForIde Migrations
  • !!! Remove Symfony Components <6.4 from composer.json

Classes

  • !!! AbstractCommandTests
  • !!! IndexedSearchCrawlerHook
  • !!! CrawlerApi
  • !!! IndexedSearchCrawlerFilesHook
  • !!! BackendModule
  • !!! AbstractRequestForm
  • !!! LogRequestForm
  • !!! RequestFormFactory
  • !!! RequestFormInterface
  • !!! StartRequestForm
  • !!! MultiProcessRequestForm
  • !!! BackendUtility

Functions & Properties

  • !!! CrawlerController->modMenu()
  • !!! ConfigurationRepository->getCrawlerConfigurationRecords()
  • !!! ConfigurationService->getUrlService()
  • !!! FrontendUserAuthenticator->queueRepository
  • !!! ProcessRepository->countActive()
  • !!! ProcessRepository->findByProcessId()
  • !!! ProcessRepository->getLimitFromItemCountAndOffset()
  • !!! ProcessQueueCommandTest->queueRepository
  • !!! QueueRepository->isPageInQueueTimed()
  • !!! QueueRepository->getAvailableSets()
  • !!! QueueRepository->countAllUnassignedPendingItems()
  • !!! QueueRepository->countPendingItemsGroupedByConfigurationKey()
  • !!! QueueRepository->getSetIdWithUnprocessedEntries()
  • !!! QueueRepository->getTotalQueueEntriesByConfiguration()
  • !!! QueueRepository->getLastProcessedEntriesTimestamps()
  • !!! QueueRepository->getLastProcessedEntries()
  • !!! QueueRepository->getPerformanceData()
  • !!! QueueRepository->findByQueueId()

11.0.7

4 months ago

Fixed

Full Changelog: https://github.com/tomasnorre/crawler/compare/11.0.6...11.0.7

11.0.6

4 months ago

Fixed

  • Zip extension with PHP 7.4 builds instead of PHP 8.1

Full Changelog: https://github.com/tomasnorre/crawler/compare/11.0.5...11.0.6

11.0.5

4 months ago

Fixed

  • If indexed_search is installed, activate procInstructions for indexed_search
  • Ensure undefined indexes is not throwing a warning in PHP 8

Full Changelog: https://github.com/tomasnorre/crawler/compare/11.0.4...11.0.5

11.0.4

2 years ago

Fixed

  • Crawler could not index frontend because of trustedHostPattern mismatch
  • Fatal PHP error is thrown in the backend crawler log
  • Prevent buildQueue from creating entries for non-localized pages
  • CrawlerController checking if URL is correct ObjectType fixed

Deprecated

Functions & Properties

  • ConfigurationRepository->getCrawlerConfigurationRecords()
  • ProcessRepository->findByProcessId()
  • QueueRepository->countAllUnassignedPendingItems()
  • QueueRepository->countPendingItemsGroupedByConfigurationKey()
  • QueueRepository->getSetIdWithUnprocessedEntries()
  • QueueRepository->getTotalQueueEntriesByConfiguration()
  • QueueRepository->getLastProcessedEntriesTimestamps()
  • QueueRepository->getLastProcessedEntries()
  • QueueRepository->getPerformanceData()
  • QueueRepository->isPageInQueueTimed()
  • QueueRepository->getAvailableSets()
  • QueueRepository->findByQueueId()

New Contributors

11.0.3

2 years ago

Added

  • PHP 8.1 compatibility

Fixed

  • PHP8 undefined array key when creating/updating/saving configuration record
  • RestrictedPages is getting crawled when Crawler Configuration contains the belonging UserGroup

Contributors

@brotkrueml @tomasnorre

11.0.2

2 years ago

Added

  • Docs: Troubleshooting - Site based baseVariants not used
  • Docs: Example on how to exclude pages recursively from being crawled
  • Docs: Respecting Categories in News to prevent double indexing
  • Docs: Set correct main branch name

Fixed

  • DateTime parse error in backend module when displaying log
  • PHP8 undefined array keys in CrawlerController, ConfigurationService and middlewares
  • Text not displayed in confirm dialog in backend module

New Contributors

11.0.1

2 years ago

Changed

  • Use more expressive message for excluded doktypes when building queue

Fixed

  • Set result_data back to json_encode($result) to ensure correct result handling in backend
  • Removes QueueRepository from FrontendUserAuthenticator Middleware, to ensure FE plugins can be rendered
  • PHP 8.0 compatibility problems. Undefined indexes resolved
  • Change requests to write to Header to ensure StaticFileCache is working
  • Switched back to fetch() instead of fetchAssociative() to keep support for legacy TYPO3 installations

11.0.0

2 years ago

Only added functionality is added here, please see the https://github.com/tomasnorre/crawler/blob/main/CHANGELOG.md for more changes.

Added

  • Support for TYPO3 11 LTS
  • PHP 8.0 Support
  • Display error messages in buildQueue command in CLI
  • PSR-14 ModifySkipPageEvent
  • PSR-14 AfterUrlCrawledEvent
  • PSR-14 InvokeQueueChangeEvent
  • PSR-14 AfterUrlCrawledEvent
  • PSR-14 BeforeQueueItemAddedEvent
  • PSR-14 AfterQueueItemAddedEvent