Repository navigation
Settings
git edited this page Sep 26, 2016
·
31 revisions
Moloch uses a tiered system for configuration variables. A tiered system allows Moloch to share one config file for many machines. The ordering of sections within the config file doesn't matter.
Order of config variables:
- [optional] The section titled with the node name is used first. Moloch will always tag sessions with node: <node name>
- [optional] If a node has a nodeClass variable, the section titled with the nodeClass name is used next. Sessions will be tagged with node:<node class name> which is useful if watching different network classes.
- The section titled "default" is used last.
| Setting | Default | Description |
| bpf | EMPTY | The bpf filter used to reduce traffic. Used both on live and file traffic. |
| certFile | EMPTY | Public certificate to use for https, if not set then http will be used. keyFile must also be set. |
| dontSaveBPFs | EMPTY | Semicolon ';' separated list of bpf filters which when matched for a session causes the remaining pcap from being saved for the session. It is possible to specify the number of packets to save per filter by ending with a :num. For example dontSaveBPFs = port 22:5 will only save 5 packets for port 22 sessions. Currently only the initial packet is matched against the bpfs. |
| dontSaveTags | EMPTY | Semicolon ';' separated list of tags which once capture sets for a session causes the remaining pcap from being saved for the session. It is likely that the initial packets WILL be saved for the session since tags usually aren't set until after several packets. It is possible to specify the number of packets to save per filter by ending with a :num. |
| dropGroup | EMPTY | Group to drop privileges to. The pcapDir must be writable by this group or to the user specified by dropUser |
| dropUser | EMPTY | User to drop privileges to. The pcapDir must be writable by this user or to the group specified by dropGroup |
| elasticsearch | localhost:9200 | A comma separated list of host:port pairs to use to connect to the Elasticsearch cluster. If not using a VIP, a different Elasticsearch node can be specified for each Moloch node. |
| freeSpaceG |
5% (>= v0.12) 100 (< v0.12) |
Delete pcap files when free space is lower then this. This does NOT delete the session records in the database. It is recommended this value is between 5% and 10% of the disk. Database deletes are done by the db.pl expire. Can also be specified using a percentage. |
| geoipASNFile | EMPTY | Path to the maxmind geoip ASN file. Download free version |
| geoipFile | EMPTY | Path to the maxmind geoip country file. Download free version |
| httpRealm | Moloch | HTTP Digest Realm - Must be in the default section of configuration file. Changing the value will cause all previous stored passwords to no longer work. |
| icmpTimeout | 10 | For ICMP sessions, Moloch writes a session record after this many seconds of inactivity since the last session save. |
| interface | EMPTY | Semicolon ';' separated list of interfaces to listen on for live traffic. (Prior to 0.14 only 1 interface is allowed) |
| keyFile | EMPTY | Private certificate to use for https, if not set then http will be used. certFile must also be set. |
| maxFileSizeG | 4 | The max raw pcap file size in gigabytes. The disk should have room for at least 10*maxFileSizeG files. |
| maxFileTimeM | 0 | The max time in minutes between rotating pcap files. Useful if there is an external utility that needs to look for closed files every so many minutes. Setting to 0 means only use maxFileSizeG |
| maxPackets | 10000 | Moloch writes a session record after this many packets since the last save. |
| maxStreams | 1500000 | An aproximiate maximum number of active sessions Moloch/libnids will try and monitor |
| minPacketsSaveBPFs | (since 0.14.2) Semicolon ';' separated list of bpf filters which when matched for a session have their SPI data NOT saved to Elasticsearch. PCAP data is still saved however. It is possible to specify the number of min packets required for SPI to be saved by ending with a :num. A use case is a scanning host inside the network that you only want to capture if their is a conversation "tcp and host 10.10.10.10:1". | |
| packetThreads | 1 | Number of threads to use to process packets. Basically how much CPU to dedicate, increase this if you get errors about dropping packets. |
| passwordSecret | EMPTY | Server-to-server and password hash secret - Must be in default section. Since elasticsearch is wide open by default, we encrypt the stored password hashes with this so a malicous person can't insert a working new account. It is also used for secure server-to-server communication. Don't set for no user authentication. Changing the value will make all previously stored passwords no longer work. |
| parseCookieValue | false | Parse HTTP request cookie values, cookie keys are always parsed. |
| parseQSValue | false | Parse HTTP query string values, query string keys are always parsed. |
| parseSMB | true | Parse extra SMB traffic info |
| parseSMTP | true | Parse extra SMTP traffic info |
| pcapDir | EMPTY | Semicolon separated list of directories to save pcap files to. The directory to save to is currently picked using round robin, need to add a smarter algorithm eventually. |
| pcapDirTemplate | EMPTY | (Since 0.14.2) When set, this strftime template is appended to pcapDir and allows multiple directories to be created based on time. |
| pcapDirAlgorithm | round-robin |
When pcapDir is a list of directories, this determines how Moloch chooses which directory to use for each new pcap file.
Possible values: round-robin (rotate sequentially), max-free-percent (choose the directory on the filesystem with the highest percentage of available space), max-free-bytes (choose the directory on the filesystem with the highest number of available bytes). |
| parsersDir | /data/moloch/parsers ; ./parsers | Semicolon separated list of directories to load parsers from |
| pluginsDir | EMPTY | Semicolon separated list of directories to load plugins from |
| plugins | EMPTY | Semicolon separated list of plugins to load and the order to load them in. Must include the trailing .so |
| rootPlugins | EMPTY | Semicolon separated list of plugins to load as root and the order to load them in. Must include the trailing .so |
| rirFile | EMPTY | Path of the RIR assignments file. Download |
| rotateIndex | daily |
Specifies how often to create a new index in Elasticsearch. Use daily or hourly for busy live instances, use weekly or monthly for research instances. Changing the value will cause previous sessions to be unreachable thru the interface. If using the multi viewer then all Moloch clusters must have the same value.
Possible values are: hourly, daily, weekly, monthly. |
| smtpIpHeaders | EMPTY | Semicolon separated list of SMTP Headers that have ips, need to have the terminating colon ':' |
| spiDataMaxIndices | 1 | Specify the max number of indices we calculate spidata for. Elasticsearch will blow up if we allow the spiData to search too many indices. |
| tcpSa 8000 veTimeout | 480 | For TCP sessions, Moloch writes a session record after this many seconds since the last session save, no matter if active or inactive. |
| tcpTimeout | 480 | For TCP sessions, Moloch writes a session record after this many seconds of inactivity since the last session save. |
| titleTemplate | _cluster_ - _page_ _-view_ _-expression_ |
|
| udpTimeout | 60 | For UDP sessions, Moloch writes a session record after this many seconds of inactivity since the last session save. |
| userNameHeader | EMPTY | Header to use for determining the username to check in the database for instead of using http digest. Use this if you have apache or something else doing the auth. The Web Auth Header checkbox must be set for user. |
| viewerPlugins | EMPTY | Semicolon separated list of viewer plugins to load and the order to load them in. Must include the trailing .js |
| viewPort | 8005 | Port for viewer to listen on |
| viewUrl | http[s]://hostname:[viewport] | The URL used to accecss this viewer instance |
| webBasePath | / | The base url for Moloch web requests. Must end with a / or bad things will happen. |
| yara | EMPTY | Where to load Yara rules from. |
| Setting | Default | Description |
| compressES | false | Compress requests to ES, reduces ES bandwidth by ~80% at the cost of increased CPU. MUST have "http.compression: true" in elasticsearch.yml file |
| dbBulkSize | 20000 | Size of indexing request to send to Elasticsearch. Increase if monitoring a high bandwidth network. |
| dbFlushTimeout | 5 | Number of seconds before we force a flush to Elasticsearch |
| maxESConns | 20 | Max number of connections to Elasticsearch from capture process |
| maxESRequests | 500 | Max number of Elasticsearch requests outstanding in queue |
| offlineFilenameRegex | (?i)\.(pcap|cap)$ | Regexp to control which filenames are processed when using the -R option to moloch-capture. Since 0.11.3 |
| packetsPerPoll | 50000 | Number of packets to ask libnids/libpcap to read per poll/spin. Increasing may hurt stats and Elasticsearch performance, decreasing may cause more dropped packets. |
| pcapBufferSize | 100000 | pcap library buffer size, see man pcap_set_buffer_size |
| pcapWriteMethod | normal |
|
| pcapWriteSize | 262144 | Buffer size when writing pcap files. Should be a multiple of the raid 5/xfs stripe size and multiple of 4096 if using direct/thread-direct pcapWriteMethod |
| prefix | EMPTY | It is possible for multiple Moloch clusters to live inside the same Elasticsearch cluster by giving each Moloch cluster a different prefix value. (Added 0.11.3) |
| usersElasticsearch | EMPTY | It is possible for multiple Moloch clusters to share the same users table, so that the same accounts and settings work across all clusters. (Added 0.11.3) |
| usersPrefix | [PREFIX] if set otherwise EMPTY | Like prefix but only for the users table. |
| Setting | Default | Description |
| logESRequests | false | Write to stdout Elasticsearch requests |
| logEventXPackets | 50000 | Write to stdout info every X packets. Set to -1 to never log status. |
| logFileCreation | false | Write to stdout file creation information |
| logUnknownProtocols | false | Write to stdout unknown IP protocols |
The moloch-clusters section is used to describe the various moloch-clusters that are available to forward traffic to either manually or through the CRON functionality. Each line represents a single cluster, with the name just being any unique string.
[moloch-clusters]
cluster1=url:https://moloch.example.com:8005;passwordSecret:password;name:Cluster
cluster2=url:https://cluster2.example.com:8005;passwordSecret:foo;name:Test Cluster| url | The base url to use to contact cluster |
| passwordSecret | The passwordSecret for the remote cluster, if it is different from current cluster |
| name | Friendly name to display in UI |
The multi viewer is useful when you have multiple Moloch clusters that you want to search across. To use an extra viewer process and a multies process must be started. These two processes can share the same config file and node name section. The viewer process uses the SAME configuration values as above.
A sample configuration for multi viewer. The elasticsearch variable points to the multies.js process.
[multi-viewer]
elasticsearch=127.0.0.1:8200
viewPort = 8009
viewHost = localhost
multiES = true
multiESPort = 8200
multiESHost = localhost
multiESNodes = escluster1.example.com:9200,prefix:PREFIX;escluster2.example.com:9200| Setting | Default | Description |
| multiES | false | This is the multiES node |
| multiESPort | 8200 | Port that multies.js should listen on |
| multiESHost | EMPTY | Host interface that multies.js should listen on |
| multiESNodes | EMPTY | Semicolon separated list of Elasticsearch nodes that MultiES should connect to. The first node listed will be considered the primary node and is used for users/views/queries. An optional "prefix:" element can follow each host if that cluster was setup with an Elasticsearch prefix. |
override-ips is a special section that overrides the MaxMind databases for the fields set, but fields not set will still use MaxMind (example if you set tags but not country it will use MaxMind for the country) Spaces and capitalization are very important.
[override-ips]
10.1.0.0/16=tag:ny-office;country:USA;asn:AS0000 This is neat| tag | A single tag to set for matches |
| country | A 3 character country code to set for matches |
| asn | An ASN value to set for matches |
| rir | A RIR value to set for matches |
This section makes it easy to specify HTTP Request headers to index. They will be searchable in the UI using http.[HEADERNAME]
[headers-http-request]
referer=type:string;count:true;unique:true| type |
|
| unique | true by default, only index unique values |
| count | false by default, create a second field http.[HEADERNAME].cnt with the number items found |
This section makes it easy to specify HTTP Request headers to index. They will be searchable in the UI using http.[HEADERNAME]
[headers-http-response]
location=type:string
server=type:stringThis section makes it easy to specify email headers to index. They will be searchable in the UI using email.[HEADERNAME]
[headers-email]
x-priority=type:integerShould no longer be needed, but it allowed the definition of fields to be done in the config file instead of programmatically with the new fields API.
Starting with Moloch 0.11.4 it is possible to configure right click actions on any SPI data fields. Currently the only actions supported are url opening actions, but other action types will be added in the future. Right click actions can be added based on either the field name or the category of the data. They can be added either in the configuration file or by enabled WISE data sources.
Configuration File Sample:
[right-click]
VTIP=url:https://www.virustotal.com/en/ip-address/%TEXT%/information/;name:Virus Total IP;category:ip
VTHOST=url:https://www.virustotal.com/en/domain/%HOST%/information/;name:Virus Total Host;category:host
VTURL=url:https://www.virustotal.com/latest-scan/%URL%;name:Virus Total URL;category:urlEach line in the right-click section contains possible actions
| Piece | Notes |
| [key]= | The key must be unique and is also used as the right click menu name if the name field is missing |
| url | The url to open if selected. There are some basic URL substitutions defined below. |
| name | The menu text to display |
| category | If the field that is right clicked on has this category then display this menu item. All right click entries must have either a category or fields variable defined. |
| fields | A comma separated list of field names. If the field that is right clicked on has one the field expressions in the list then display this menu item. All right click entries must have either a category or fields variable defined. |
| regex | A regex to use on the right clicked text to extract a piece for the URL. The first matching group is substituted for %REGEX% in the url. If the regex doesn't match at all then the menu isn't displayed. |
| users | A comma separated list of user names that can see the right click item. If not set then all users can see the right click item. |
The possible URL Substitutions are
| URL Substitutions | Notes |
| %TEXT% | The text clicked on in raw form |
| %URL% | An URL extracted from the text clicked on |
| %HOST% | A hostname extracted from the text clicked on. Sometimes the same as %TEXT% sometimes a subset. |
| %REGEX% | The first regex group match |
The categories that Moloch uses are
| Category | Notes |
| asn | An ASN field |
| country | A three letter country code |
| host | A domain or host name |
| ip | An ip address |
| md5 | A md5 of a payload, such as for http body or smtp attachment |
| port | The TCP/UDP port |
| rir | The Regional Internet Registry |
| url | A URL |
| user | A user name or email address |