* Elasticsearch store updates

* Adds ACL check on indicator search
* On token edit, adds ability to update fields other than group

The above should resolve some of the new test errors from pull #500.

* Fix pip backtracking issue

Pin setuptools due to maxmind conflict and pin pip version due to backtracking issue introduced in new versions

* pin setuptools

due to maxmind conflict

Co-authored-by: Michael Davis <mdavis@ren-isac.net>

* Elasticsearch store updates

* Adds ACL check on indicator search
* On token edit, adds ability to update fields other than group

The above should resolve some of the new test errors from pull #500.

* Fix pip backtracking issue

Pin setuptools due to maxmind conflict and pin pip version due to backtracking issue introduced in new versions

* pin setuptools

due to maxmind conflict

* move most authN/Z functions up from store

* moves authN/Z to another mzq socket and introduces new Auth service via plugin
* changes most funcs to xfer token dict rather than token str (to minimize store access per request)
* introduces backend socket (by reusing hunter_sink) that bypasses auth for internal-to-cif requests
* above changes give Hunters ability to search the CIF store (opens up new hunter possibilities)
* fixes some tests related to the above
* general cleanup

* allow querying by rdata field

* adds fqdn_wl hunter

* convert to ujson

* remove unused libs

* take advantage of tag negation

using !whitelist tag negation in internal feed searches ensure unneeded indicators never returned, so no need to filter them out

* tweak error msging

* reorder auth error so it's actually caught before general exception
* add runtime error with useful output

* cleanup / tweaks

Co-authored-by: Michael Davis <mdavis@ren-isac.net>

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-UJSON-2940619

Co-authored-by: snyk-bot <snyk-bot@snyk.io>

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-NLTK-1053952
- https://snyk.io/vuln/SNYK-PYTHON-NLTK-1660190
- https://snyk.io/vuln/SNYK-PYTHON-NLTK-1932014
- https://snyk.io/vuln/SNYK-PYTHON-NLTK-2313655
- https://snyk.io/vuln/SNYK-PYTHON-NLTK-460224
- https://snyk.io/vuln/SNYK-PYTHON-UJSON-2359034
- https://snyk.io/vuln/SNYK-PYTHON-UJSON-2940619
- https://snyk.io/vuln/SNYK-PYTHON-UJSON-2942122
- https://snyk.io/vuln/SNYK-PYTHON-URLLIB3-1014645
- https://snyk.io/vuln/SNYK-PYTHON-URLLIB3-1533435

Co-authored-by: snyk-bot <snyk-bot@snyk.io>

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-UJSON-2942122

The following vulnerabilities are fixed by pinning transitive dependencies:
- https://snyk.io/vuln/SNYK-PYTHON-SETUPTOOLS-3113904

Co-authored-by: snyk-bot <snyk-bot@snyk.io>

* Check `expires` and `revoked` token fields

* Implement strtobool

* Standardize print statements

* Implement token tweaks

* add `last_edited_at`  and `last_edited_by`, `created_at`, and `created_by` fields for token updates in ES store
* `token_cache` dict implemented as mp.Manager proxy to share cache b/w processes
* token `search()` caches token result for fast lookup
* token `auth_search()` now checks cache first, possible 100x speed increase
* token `edit()` changes in-mem cache and uses ES versioning to prevent stale cache stomping b/w diff router instances
* token cache flushed at `TOKEN_CACHE_DELAY` interval and written using new bulk=True edit param
* add pytests for the above

* Add some logic and improved error handling

* Eliminate hardcoded `fqdn` itype

Fix the assumption that a hostname for a url will always be an `fqdn` itype, e.g. `http://10.10.10.1/Malware.exe`

* Skip ja3 gatherer if source is already `ja3er.com`

* Add `find_relatives` and `sort` param frontend func

* Enables passing a `?sort=` param to API (defaults to `?sort=-reporttime,-lasttime` which specifies reporttime DESC and lasttime DESC)
 * feed pulls default to (and can't be changed from) `-confidence,-reporttime`
* Adds `?find_relatives=` API param which default to `False` for all searches
 * Enables finding related/familial indicators (super/subnets for IPs and related SSDEEP)

Backend CIF searches such as allowlist searches for feed pulls or searches done during upsert checks will enforce `find_relatives=False`

* change `find_relatives` frontend default to `False`

* Add ssdeep as newly supported `itype`

* add appropriate new fields to analyze/tokenize on store
* includes new search filters for finding related ssdeep matches
* add ability to pull ssdeep feeds
* add new unit tests

* Refactor hunters

* Add spamhaus DQS support
* lower fqdn hunter conf such that no further hunters run on their creations
* each hunter now has a set of supported itypes and mtypes
* factored out each hunter's prereq check into own func
* additional info such as mtype and nolog passed to each hunter for flexibility
* make http filters a set and better validate input

* Indicator index tweaks

* .init() will only try to update mappings if schema is diff
* if there is a diff and index is open, IllegalOperation returned
* if another proc already closing/reopening, catch/retry

* Fix upserts matching on overlapping tags

* fix issue where identical indicator w/ additional tags would previously upsert into existing indicator with fewer tags that had overlap, e.g.:
 * provider=example.com, confidence=5, indicator=bad.tld, tags=[old]
 * provider=example.com, confidence=5, indicator=badl.tld, tags[old,new]
 previously, the new indicator would be upserted into the old indicator and the new tag info would be lost. it now creates an entirely new indicator

* URL param keys now lowercased on every frontend request for better UX in the event a human specifies a non-lowercase param (like Asn=0000 instead of asn=0000)

* check for existence of `indicator` field at frontend during submission to ensure no malformed indicators are sent on

* Update to TLP2

* adjust `white` to `clear` to align with tlp2

* Fix upsert group match

Upserts weren't correctly matching diff groups in certain cases, causing an indicator w/ all other identical fields to upsert match and increase count of existing indicator rather than create a new indicator. This was due to the API only allowing the url param `groups` (plural), but several cif clients making use of `group` (singular).

Also fixed upsert matches on optional fields. e.g., if user-specified, but optional, field like `rdata` is specified in one indicator but not another, all other fields being equal, those indicators are diff and therefore shouldn't upsert match.
* fix the above issue so that if a field isn't specified in the upsert search, it's specificially excluded in the ES search

* misc fixes

* Enable API multiparams

* support multiple URL params w/ the same name

There's no web std for how to pass multiple values to a URL param like `/feed?tags=malware,exploit` and some clients (like python requests lib) takes multiple values like `{ 'tags': [ 'exploit', 'malware' ] }` and passes those in as `/feed?tags=exploit&tags=malware`. Unfortunately, by default Flask ignores all URL params of the same name after the first one, so it would receive such a request at the API and only consider the `exploit` tag. This change normalizes multiple params of the same name into a single key whose value is a comma-delimieted string.

Therefore, `/feed?tags=malware&tags=exploit` is re-interpreted as `tags=malware,exploit` and therefore correctly considers all passed values to the API.

Additionally, this commit supports a SHARDS_PER_INDEX variable for ES indices.

* Refactor a few gatherers

* misc fixes

* Move `reporttime` parsing from specific stores to main

* Remove deprecated use of _request_ctx_stack

* Align with tlp2

* URL param filtering now happens closer to frontend

* Fix some things related to sort tweaks

* Add test for `reporttime`