The FSCrawler team is pleased to announce the FSCrawler 2.7 release!
FSCrawler
FS Crawler offers a simple way to index binary files into elasticsearch.
Usage
Download FSCrawler 2.7:
wget https://repo1.maven.org/maven2/fr/pilato/elasticsearch/crawler/fscrawler-es7/2.7/fscrawler-es7-2.7.zip
Start FS crawler with:
bin/fscrawler job_name
FS crawler will read a local file (default to ~/.fscrawler/{job_name}/_settings.json).
If the file does not exist, FS crawler will propose to create your first job.
$ bin/fscrawler job_name
18:28:58,174 WARN [f.p.e.c.f.FsCrawler] job [job_name] does not exist
18:28:58,177 INFO [f.p.e.c.f.FsCrawler] Do you want to create it (Y/N)?
y
18:29:05,711 INFO [f.p.e.c.f.FsCrawler] Settings have been created in [~/.fscrawler/job_name/_settings.json]. Please review and edit before relaunch
Create a directory named /tmp/es or c:\tmp\es, add some files you want to index in it and start again:
$ bin/fscrawler job_name
18:30:34,330 INFO [f.p.e.c.f.FsCrawlerImpl] Starting FS crawler
18:30:34,332 INFO [f.p.e.c.f.FsCrawlerImpl] FS crawler started in watch mode. It will run unless you stop it with CTRL+C.
18:30:34,682 INFO [f.p.e.c.f.FsCrawlerImpl] FS crawler started for [job_name] for [/tmp/es] every [15m]
More details in the documentation.
New features
-
#991: Add Workplace Search connector.
-
#1203: Add FTP crawler. By helsonxiao.
-
#1211: Add
file.content_type field on folders.
-
#1210: Add
file.filename field on folders.
-
#1179: Automatically create Custom Sources.
-
#1037: Split console logs and actual logs and add a banner :).
-
#1036: Support ssl verification configurable. By TommyLike.
-
#1035: Log index errors in documents.log.
-
#1031: Add an external Log4J2 configuration file.
-
#907: Add
path_prefix option.
-
#820: Generate FSCrawler docker images. By toto1310.
-
#776: Report HEAP size at startup.
-
#752: Add option to ignore symlinks. By budachst.
-
#715: Allow custom index name in the REST API. By kikkauz.
-
#698: Add Cross-Origin Resource Sharing (CORS) headers to RestServer. By isaac-ipl.
-
#692: Allow running OCR but not on PDF files.
-
#673: Add support for YAML configuration.
-
#663: Add Patterns table to includes and excludes. By wrathagom.
Fixed Bugs
-
#1224: Fix NPE in Console when running with Docker.
-
#1217: Check if date is null when formatting it to RFC3339.
-
#1204: Split build and deploy phases for Docker images.
-
#1201: 2.7 - Docker image broken. By agrantdeakin.
-
#1194: Elasticsearch node settings should not be null by default.
-
#1193: Corrupt PDF can lead to a StackOverflow.
-
#1137: Ignore errors when parsing a 0 byte file.
-
#1085: fscrawler.bat added a CD to move to the appropriate directory. By CircuitGuy.
-
#1084: InputStream must have > 0 bytes. By yuanzhian.
-
#1066: Start fscrawler instead of internal services.
-
#1041: Fixed an issue that caused an error when running in a windows environment. By muraken720.
-
#1006: Running fscrawler with no argument now lists existing jobs. By janhoy.
-
#1005: Fix ENTRYPOINT in Dockerfile to allow variable substitution. By Maijin.
-
#994: Using cloud id gives "invalid IPv6 Address". By tdaroly.
-
#973: Fix SSH crawling from Windows machine.
-
#899: FSCrawler can't index .doc or .docx elements. By LaaKii.
-
#895: java.lang.NoSuchMethodError: parsing some Word files. By mwaltersbmc.
-
#860: Bug Syntax error in fscrawler file, to init fscrawler. By CarlosRCDev.
-
#847: sun.jnu.encoding=UTF-8 added in .bat and .sh both. By shahariaazam.
-
#834: FS Crawler freezes when crawling a 0 byte TXT file. By dansfelix.
-
#819: Fix Percentage computation.
-
#760: Allow passing test parameters to Maven CLI.
-
#714: fix release-drafter. By jetersen.
-
#701: Change log level and display logs only if filters on content.
-
#691: OCR without pdf_ocr. By Newmski.
-
#686: Wait for healthy index when creating the index.
-
#681: SSH dirs should be seen as dirs and not files.
-
#680: trying to index remote files with ssh - files seen as folder. By sblanc0054.
-
#660: Fix authentication when sending announcement email.
Main changes
-
#1218: Isolate WorkplaceSearchClient and ElasticsearchClient.
-
#1213: Switch back to Java 11.
-
#1049: Update Dockerfile to use JDK14. By mario-89.
-
#1212: Let's use JsonPath.
-
#1207: Generate only 2 docker images.
-
#1206: Detect when fscrawler runs in foreground and adapt logs.
-
#1205: Add logs to the console when running a Docker instance.
-
#1172: Move CI from Travis to GitHub actions.
-
#872: Add more information to the _simulate API.
-
#700: Add dependency convergence checks.
-
#695: Exclude the PDFParser from the DefaultParser.
-
#694: Display full names when catching parsing errors.
-
#693: Move
fs.pdf_ocr setting to fs.ocr.pdf_strategy.
-
#675: Warn in case of Tika error.
-
#1219: Update to Elasticsearch 7.14.0 and 6.8.18.
-
#1180: Bump tika.version from 1.26 to 1.27.
Removed
-
#978: files lost. By bluebell1990.
Have fun!
-FSCrawler team