8000 FAQ · newdominic/moloch Wiki · GitHub
[go: up one dir, main page]
Skip to content

FAQ

Jump to bottom Edit New page
amaddalena edited this page Sep 23, 2016 · 94 revisions

Table of Contents

General

How do I reset Moloch?

Leave elasticsearch running: shutdown all running viewer or capture processes so no new data is recorded.

To delete all the SPI data stored in elasticsearch, use the db.pl script with either the init or wipe commands. The only difference between the two commands is that wipe leaves the added users so they don't need to be re-added. 

    /data/moloch/db/db.pl ESHOST:ESPORT wipe

Next, you need to delete the pcap files. The PCAP files are stored on the file system in raw format. You need to do this on all of the capture machines 

    /bin/rm -f /data/moloch/raw/*

What OSes are supported?

Moloch is no longer supported on 32 bit machines. Our deployment is on Centos 6 with the elrepo 3.x kernel upgrade for packet performance increases. A large amount of development is done on Mac OS X 10.11 using MacPorts, however, it has never been tested in a production setting. :)

The following OSes should work (easybutton-singlehost tries to support):

  • Centos 5, 6, 7
  • Ubuntu 12.04, 14.04, 16.04
  • FreebSD 9, 10 (wiseService doesn't work, but the wise plugin does)
Feel free to update with other known configurations!

Elasticsearch

How many elasticsearch nodes or machines do I need?

The answer, of course, is "it depends". Factors include:

  • How much memory each box has
  • How many days you want to store meta data (SPI data) for
  • How fast the disks are
  • What percentage of the traffic is HTTP
  • The average transfer rate of all the interfaces
  • If the sessions are long lived or short lived
  • How fast response times should be for operators
  • How many operators are querying at the same time

Some important things to remember when designing your cluster:

  • 1Gbps of network traffic requires ~200GB of disk a day. For example, to sto 8000 re 14 days of 2.5Gbps average traffic you need 14*2.5*200 or ~7TB of disk space.
  • SPI data is usually kept longer then PCAP data. For example, you may store pcap for a week but SPI data for a month.
  • Have at least 3% of disk space available in memory. For example, if the cluster has 7TB of data then 7*0.03 or 210GB of memory is the minimum recommended. Note, the more days store the memory ratio can actually decrease.
  • Assign half the memory to elasticsearch (but no more then 30G per node) and half the memory to disk cache.
  • Use version 2.x of elasticsearch.

A rough guideline for NODES is 1/4 * Average_Total_Gigabit_Sec * Number_of_Days_of_History. If you have large machines, then you can run multiple nodes per MACHINE.

The good news is that it is easy to add new nodes in the future, so feel free to start with less nodes. As a temporary fix to capacity problems, you can reduce the number of days of meta data that are stored. Just use the elasticsearch head interface to delete the oldest sessions-* index.

Data never gets deleted

Make sure you set up something like the db/daily.sh script to run once a day, during off peak. This script optimizes the previous days data and deletes the oldest data.

When do I add additional nodes? Why are queries slow?

If queries are too slow the easiest fix is to add additional elasticsearch nodes. You should also use the bigdesk http://moloches-machine:9200/_plugin/bigdesk/ and watch the heap memory graph. If it is approaching the max, it is time to add more nodes or reduce the number of days being stored.

Elasticsearch doesn't do well if Java hits an OutOfMemory condition. If you ever have one, you should immediately delete the oldest sessions-* index, update the daily.sh script to delete more often, and restart the elasticsearch cluster. Then you should order more machines. :)

Adding additional nodes

See detailed instructions in the Multiple-Host-HOWTO wiki page

Removing nodes

  • First, configure elasticsearch to exclude the nodes being removed. In this example, nodes moloches-05a, moloches-05b, moloches-06a, moloches-06b are all being removed. Either _name or _ip can be used to match. 
    curl -XPUT http://localhost:9200/_cluster/settings -d '{
        "transient" : {
            "cluster.routing.allocation.exclude._name" : "moloches-05*,moloches-06*"
        }
    }'
  • Now, look at localhost:9200/_plugin/head and make sure some indexes start to move.
  • If reducing the number of nodes and not just replacing/changing nodes, then elasticsearch must be configured to allow multiple shards per node. At minimum, elasticsearch must be configured allowing 2 shards per node, although a large number may be required if removing many nodes. 
    curl -XPUT 'localhost:9200/sessions*/_settings' -d '{
        "index.routing.allocation.total_shards_per_node": 2
    }'
  • Once the nodes no longer have any session* shards, shut them down: 
    curl -XPOST 'http://localhost:9200/_cluster/nodes/moloches-05*,moloches-m06*/_shutdown'
  • If there are many shards that need to be redistributed, the defaults might take days, which is good for the cluster, but it might make you crazy. Increase the speed from default 3 streams at 20mb (60mb/sec) to something higher like 6 streams at 50mb (300mb/sec). Obviously, adjust for the speed of the new nodes' disks and network. 
    curl -XPUT localhost:9200/_cluster/settings -d '{"transient":{
       "indices.recovery.concurrent_streams":6,
       "indices.recovery.max_bytes_per_sec":"50mb"}
    }'

How to enable elasticsearch replication

Turning on replication will consume twice the disk space on the nodes and increase the network bandwidth between nodes, so make sure you actually need replication.

To change future days (since 0.14.1): 

    db/db.pl <ESHOST:ESPORT> upgrade --replicas 1

To change past days, but not the current day (since 0.14.1): 

    db/db.pl <ESHOST:ESPORT> expire <type> <num> --replicas 1

How do I upgrade elasticsearch?

Rolling upgrades are supported for elasticesearch 2.x when doing a minor upgrade. In all other cases you must shutdown the entire cluster and restart it with the new version. 

    cd elasticsearch-NEWVERSION
    bin/plugin -install mobz/elasticsearch-head
    bin/plugin -install lukas-vlcek/bigdesk
  • Edit the ES start script so it has the correct version, /data/moloch/bin/run_es.sh (if using single hostconfig)
  • Shutdown current elasticsearch node: 
    curl -XPOST http://localhost:9200/_shutdown # prior to ES 2
    kill <pid> # since ES 2
  • Start the new version back up: 
    /data/moloch/bin/run_es.sh

How do I upgrade to ES 2.x from ES 1.x?

ES 2.x is only supported by Moloch 0.12.3 and later.

  • Make sure the cluster is GREEN and you are running at least ES 1.7.3, the upgrade has only been tested with 1.7.3. If not using 1.7.3 upgrade to that first.
  • Run the following command twice (just to make sure everything is upgraded :). This makes sure all ES files are using a recent version of lucene data format. This is only really needed if ES 0.x was ever used, but doesn't hurt. 
    curl -XPOST 'http://localhost:9200/_upgrade?pretty&human&only_ancient_segments'
  • Make sure the cluster is GREEN.
  • Upgrade the Moloch db scheme while still on the OLD version of ES. 
    db.pl 127.0.0.1:9200 upgrade
  • Make sure the cluster is GREEN.
  • Completely shutdown everything (ES, viewer, cluster). Do NOT do a rolling ES restart.
  • YOU MUST SHUTDOWN EVERYTHING, even if you normally don't, this time you MUST!
  • Upgrade ES to 2.x (see the instructions above).
  • Make sure that in elasticsearch.yml you've set "gateway.recover_after_nodes:" to number of ALL nodes in the cluster (data + master only).
  • Start just 1 or 2 nodes and make sure new ES actually starts. If it doesn't, you can usually go back to previous version of ES since you haven't fully started ES yet.
  • DO NOT start the full ES cluster until all capture/viewer process are shutdown AND you tried 1 ES node by itself.
  • Start full ES cluster.
  • Wait for cluster to go GREEN.
  • Start viewers and captures.

ES 2 won't start - java.lang.IllegalStateException ... was created before v0.90.0 and wasn't upgraded

  • Go back to a working version of ES like 1.7.3.
  • Make sure you followed the ES2 instructions above.
  • Run _upgrade one more time.
  • If the index it is complaining that it's on a non master node and there is a _state directory at the top level of the index (not in a shard directory), delete it. See elasticsearch issue #16044.

ES 2 - org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large

This seems to happen with old indices that are upgraded from ES versions before 2 to ES 2, because old indices didn't use the doc_values settings. Increasing the size of the fielddata cache will fix the problem.

Either edit your elasticsearch.yml and add: 

    indices.breaker.fielddata.limit: 80%

or run the command: 

    curl -XPUT localhost:9200/_cluster/settings -d '{
      "persistent" : {
        "indices.breaker.fielddata.limit" : "80%"
      }
    }'

If that doesn't work, you can try increasing the heap size per node, but don't go over 31G. For more information, read the heap sizing guide

Capture

What kind of capture machines should we buy?

Some things to remember when selecting a machine:

  • 1Gbps of network traffic requires 11TB of disk a day. For example, to store 7 days of 2.5Gbps average traffic you need 7*2.5*11 or 192.5TB of disk space.
  • The total bandwidth number must include both RX and TX bandwidth. For example, a 10G link can really produce up to 20G of traffic to capture, 10G in each direction. Include both directions in your calculations.
  • Related to the above, capturing a busy link will require multiple connections to the moloch-capture machine(s), since both the TX/RX links are being monitored. For example, a 10G link that is doing an average of 4 Gbps each direction should use two 10G moloch-capture links. Otherwise, packets may drop during bursts since the total of 8Gbps is near the max 10Gbps already.
  • Moloch requires all packets from the same 5-tuple to be processed by the same moloch-capture process.

When selecting Moloch capture boxes, standard "Big Data" boxes might be the best bet. ($10k-$15k each) Look for:

  • CASE: There are many 4RU boxes out there. If space is an issue, there are more expensive 2RU that still hold over 20 drives (example HPE Apollo 4200)
  • MEMORY: 64GB to 96GB
  • OS DISKS: We like RAID 1 small drives. SSDs are nice but not required.
  • CAPTURE DISKS: 20+ x 4TB or 6TB SATA drives. We like RAID 5 with 1 hot spare.
  • RAID: A hardware RAID card with at least 1G cache.
  • NIC: We like newer Intel base NICs, but most should work fine (might want to get one compatible with PFRING).
  • CPU: At least 2 x 6 cores. The higher the average Gbps, the more speed/cores required.

We are big fans of using Network Packet Brokers ($10k - $50k). They allow multiple taps/mirrors to be aggregated and load balanced across multiple moloch-capture machines. This allows for easy horizontal scaling as there are increases in bandwidth, retention policy, or required tools. As an added bonus, any traffic filtering/shaping can be moved to the NPB.

The goal of Moloch is to use commodity hardware. If you start thinking about using SSDs or expensive NICs, think about if it would just be cheaper to buy one more box. This usually gains more retention, and can bring the cost of each box down.

What kind of packet capture speeds can moloch-capture handle?

Moloch allows multiple threads to be used to process the packets. On simple commodity hardware, it is easy to get 3Gbps or more, depending on the number of CPUs available to Moloch and what else the machine is doing. Many times the limiting factor is the speed of the disks and RAID system. See Architecture and Multiple-Host-HOWTO for more information.

To test the RAID device use: 

    dd bs=256k count=50000 if=/dev/zero of=/THE_MOLOCH_PCAP_DIR/test oflag=direct

This is the MAX disk performance. If you don't want to drop any packets, you shouldn't average more then ~80% of the value returned. If using a RAID and want to not drop packets during a future rebuild, you probably don't want to count on more then 60% of the value returned.

Prior to version 0.14 the recommended assumption was 1.5Gbps.

Why am I dropping packets?

Network Card Config

Make sure the network card is configured correctly by increasing the ring buf to max size and turning off most of the card's features. The features are not useful anyway, since we want to capture what is on the network instead of what the local OS sees. Example configuration: 

# Set ring buf size, see max with ethool -g eth0
ethtool -G eth0 rx 4096 tx 4096
# Turn off feature, see available features with ethtool -k eth0
ethtool -K eth0 rx off tx off gs off tso off gso off

Moloch Config

Version prior to 0.14 use pcapWriteMethod=thread-direct for a separate thread for all disk reads/writes. For versions since 0.14 use pcapWriteMethod=simple.

Kernel and TPACKET_V3 support

Make sure you are running Kernel 3.2 or later and libpcap 1.5.2 or later for a 15-25% boost in capture rate supported. You should verify that moloch-capture has been configured and compiled correctly by running nm moloch-capture | grep pcap_read_linux_mmap_v3

For those stuck on Centos 6 use elrepo and install kernel-ml-headers AND kernel-ml. Both are needed as well as a force install of the headers. Download from http://elrepo.org/linux/kernel/el6/x86_64/RPMS/

Disk

Make sure swap has been disabled, or at the very least, isn't writing to the disk being used for pcap.

Make sure the RAID isn't in the middle of a rebuild or something worse. Most RAID cards will have a status of OPTIMAL when things are all good and DEGRADED or SUBOPTIMAL when things are bad.

Another possible reason is the disk IO subsystem can't keep up which is causing moloch-capture to pause and fall behind reading the packets off the network. For versions since 0.14.0 use pcapWriteMethod=simple, for versions prior to 0.14 try pcapWriteMethod=thread-direct.

If you are using xfs make sure you use mount options defaults,inode64,noatime

  • Don't run capture and elasticsearch on the same machine.
  • Make sure you actually have enough disk write thru capacity and disks. For example, for a 1G link with RAID 5 you may need:
    • At least 4 spindles if using a RAID 5 card with write cache enabled.
    • At least 8 spindles (or more) if using a RAID 5 card with write cache disabled.
  • Make sure your RAID card can actually handle the write rate. Many onboard RAID 5 controllers can not handle sustained 1G write rates.
  • Switch to RAID 0 from RAID 5 if you can live with the TOTAL data loss on a single disk failure.

If using EMC for disks:

  • Make sure write cache is enabled for the LUNs. 8000
  • If it is a CX with SATA drives, RAID-3 is optimized for large sequential I/O.
  • Monitor EMC lun queue depth, may be too many hosts sharing it.

To check your disk IO run iostat -xm 5 and look at the following:

  • wMB/s will give you the current write rate, does it match up with what you expect?
  • avgqu-sz should be near or less then 1, otherwise linux is queueing instead of doing
  • await should be near or less then 10, otherwise the IO system is slow, which will slow moloch-capture down.

Other things to do/check:

  • If using RAID 5 make sure you have write cache enabled on the RAID card.
    • Adaptec Example: arcconf SETCACHE 1 LOGICALDRIVE 1 WBB
  • Maybe taskset to give moloch-capture its own CPU, although with the new pcapWriteMethod thread or thread-direct setting, this isn't needed, and may hurt.

Other

  • There are conflicting reports that disabling irqbalancer may help.
  • Check that the CPU you are giving moloch-capture isn't handling lots of interrupts (cat /proc/interrupts).
  • Make sure other processes aren't using the same CPU as moloch-capture.

WISE

  • Cyclical packet drops may be caused by bad connectivity to the wise server. Verify that the wiseService responds quickly curl http://moloch-wise.hostname:8081/views on the moloch-capture host that is dropping packets.

How do I import existing PCAPs?

Think of the 'moloch-capture' binary much like you would 'tcpdump'. 'moloch-capture' can listen to live network interface(s), or read from historic packet capture files.

${moloch_dir}/bin/moloch-capture -c [config_file] -r [pcap_file]

For an entire directory, use -R [pcap]

See ${moloch_dir}/bin/moloch-capture --help for more info

How do I monitor multiple interfaces?

Versions since 0.14.2 support a semicolon ';' separated list of interfaces to listen on for live traffic.

Versions prior 0.14.2 require multiple moloch-capture processes, since a capture process will only monitor a single interface. To monitor multiple interfaces on a single machine, you will need multiple capture processes.

  • Since, by default, moloch uses the unqualified hostname as the name of the moloch node, you'll need to come up with a naming scheme. Appending a, b, c, ... or the interface number to the hostname are possible methods.
  • Edit /data/moloch/etc/config.ini, and create a section for each of the moloch nodes. Assuming the defaults are correct in the [default] section, the only thing thing that MUST be set is the interface item. It is also common to have each moloch node talk to a different elasticsearch node if running a cluster of elasticsearch nodes. 
 [moloch-m01a]
 interface=eth2
 [moloch-m01b]
 interface=eth5
  • If hostname on the machine doesn't return a FQDN, you'll also need to set a viewUrl.
  • Create two start up scripts. You will now need to specify the moloch node name with the -n option and change the log files so they are separate. 
 TDIR=/data/moloch
 cd ${TDIR}/bin
 /bin/rm -f ${TDIR}/capturea.log.old
 /bin/mv ${TDIR}/logs/capturea.log ${TDIR}/logs/capturea.log.old
 ${TDIR}/bin/moloch-capture -n moloch-m01a -c ${TDIR}/etc/config.ini > ${TDIR}/logs/capturea.log 2>&1

and 

 TDIR=/data/moloch
 cd ${TDIR}/bin
 /bin/rm -f ${TDIR}/captureb.log.old
 /bin/mv ${TDIR}/logs/captureb.log ${TDIR}/logs/captureb.log.old
 ${TDIR}/bin/moloch-capture -n moloch-m01b -c ${TDIR}/etc/config.ini > ${TDIR}/logs/captureb.log 2>&1
  • Add both scripts to inittab or upstart

You only need to run one viewer on the machine. Unless it is started with the -n option, it will still use the hostname as the node name, so any special settings need to be set there (although default is usually good enough).

Moloch capture crashes

Please file a ticket on github with the stack trace.

  • You'll need to allow suid or user changing programs to save core dumps. Use sysctl to change until the next reboot. Setting it to 0 will change it back to the default. 
sysctl -w fs.suid_dumpable=2
  • The user that moloch switches to must be able to write to the directory that moloch-capture is running in.
  • Run moloch-capture and get it to crash.
  • Look for the most recent core file.
  • Run gdb (you may need to install the gdb package first) 
gdb /data/moloch/bin/moloch-capture corefilename
  • Get the back trace using the bt command

If it is easy to reproduce, sometimes it's easier to just run gdb as root:

  • Run gdb moloch-capture as root.
  • Start moloch in gdb with run ALL_THE_ARGS_USED_FOR_MOLOCH-CAPTURE_GO_HERE.
  • Wait for crash.
  • Get the backtrace using bt command.
  • Sometimes, you need to put a break point in g_log b g_log

Capture log is full of libnids errors at start

libnids isn't used since 0.14, upgrade to avoid the errors.

When starting Capture, capture log is full of errors like 

Feb  * 1:24:45 nids.c:917 moloch_nids_syslog(): NIDSIP: Invalid IP header - Length issue, was packet truncated?

Many ethernet cards have advanced features for offloading operations. Moloch doesn't like this, plus it can hide what packets are actually on the network. The most common problems are the rx-checksumming and the generic-segmentation-offload. You can disable both with ethtool. 

ethtool -K eth0 rx off tx off gs off tso off gso off

After that, the errors should be gone. If not, you can use ethtool -k eth0 to find other features of the network card and turn those off also.

PCAP Deletion

PCAP deletion is actually handled by the viewer process, so make sure the viewer process is running on all capture boxes. The viewer process checks every minute to see how much space is available, and if it is below freeSpaceG, then it will start deleting the oldest file. Note, freeSpaceG can also be a percentage, newer versions of moloch use freeSpaceG=5% for the default. The viewer process will always leave at least 10 PCAP files on the disk, so make sure there is room for at least maxFileSizeG * 10 capture files on disk, or by default 120G.

WARNING - Moloch versions before and including 0.11.2 would always keep 20 files, version 0.11.3 and 0.11.4 would always keep 100 files, and version 0.11.5 and later always keeps 10 files. So for example with 0.11.3 a minimum of maxFileSizeG * 100 storage would be required, or by default 1200G. Bad developer!

dontSaveBPFs doesn't work

Turns out BPF filters are tricky. :) When the network is using vlans, then at compile time, BPFs need to know that fact. So instead of a nice simple dontSaveBPFs=tcp port 443:10 use something like dontSaveBPFs=tcp port 443 or (vlan and tcp port 443):10. Basically FILTER or (vlan and FILTER). Information from here.

Viewer

Where do I learn more about the expressions available

Click on the owl

Exported pcap files are corrupt, sometimes session detail fails

  • The most common cause of this problem is that the timestamps between the moloch machines are different. Make sure ntp is running everywhere, or that the time stamps are in sync.

Map counts are wrong

  • The source and destination ips are each counted, so the map should total twice the number of sessions.
  • Currently elasticsearch only has accurate counts up to 2 billion uniques.
  • Some countries aren't shown, but can still be searched using their ISO-3.

Self signed or internal root cert issues

When using self signed or internal root signed certs, both viewer and browsers will have issues.

Browser security exception/replacing cert

By default the easybutton-singlehost.sh script creates a self signed cert. This will cause good browsers to throw a security exception.

Quick Fix

Most browsers have a method for adding a self signed cert so its accepted and stops showing warnings. Start with this google query for suggestions on your browser.

Correct Fix

  • Contact your PKI folks and ask for a new cert, or purchase one.
    • They may require a csr, you can use the /data/moloch/etc/moloch.csr file or create a new one.
    • If you will (or think you will) have multiple viewer machines, it is easier to give them ALL the host names at once so the single cert can have multiple alt names in it.
  • Replace the /data/moloch/etc/moloch.crt and moloch.key files on all the viewer nodes.
  • Restart the viewer (node) process.

Viewer issues with multiple nodes

This is only an issue when using multiple viewer nodes and they try and proxy requests to each other. Node.js will try and verify the certs chain correctly, and with certs that are self signed, or signed by a cert that Node.js doesn't know about viewer will output "proxy errors".

Bad Fix

Add export NODE_TLS_REJECT_UNAUTHORIZED=0 to the viewer startup script. This will cause Node.js to not verify the cert when talking to other https servers. This means it is possible to perform MITM attacks whenever the Node.js server talks to another https server. This is insecure, only use for proof of concept.

Correct Fix

Contact your PKI folks and ask for a new cert, or purchase one (they are $25 or less now).

Possible Fix

  • Create a file with the internal root cert or self signed certs.
  • Place on all viewer machines.
  • Set caTrustFile=FILEPATH in the default section and restart all the viewers.

What browsers are supported?

Recent versions of Chrome, Firefox, and Safari should all work fairly equally. Development is done mostly with Chrome on a Mac, so it gets the most attention. Firefox on Windows is used heavily by the analysts. Currently, no testing or development is done with IE or Opera.

Viewer doesn't run after upgrading Node.js

The packages that viewer depends on must be reinstalled after upgrading Node.js on a machine. 

cd /data/moloch/viewer
mv node_modules node_modules.save
npm cache clean
npm install

Error: getaddrinfo EADDRINFO

This seems to be caused when proxying requests from one viewer node to another and the machines don't use FQDNs for their hostnames and the short hostnames are not resolvable by DNS. You can check if your machine uses FQDNs by running the hostname command. There are several options to resolve the error:

  1. Configure the OS to use FQDNs.
  2. Make it so DNS can resolve the shortnames or add the shortnames to the hosts file.
  3. Edit config.ini and add a viewUrl for each node. This part of the config file must be the same on all machines. (We recommend you just use the same config file everywhere). Example: 
[node1_eth0]
interface=eth0
viewUrl=http://node1.fqdn
[node1_eth1]
interface=eth1
viewUrl=http://node1.fqdn
[node2]
interface=eth1
viewUrl=http://node2.fqdn

How do I proxy Moloch using Apache

Apache, and other web servers, can be used to provide authentication or other services for Moloch. While operators will use the proxy, the viewer processes still need direct access to each other.

  • Install apache, turn on the auth method of your choice, and make sure mod_ssl is loaded
  • Set up a SSL proxy: 
SSLProxyEngine On
#ProxyRequests On # You probably don't want this line
ProxyPass        /moloch/ https://localhost:8005/ retry=0
ProxyPassReverse /moloch/ https://localhost:8005/
  • Restart apache.
  • Using the Moloch UI (by going directly to a non proxy moloch) make sure the "Web Auth Header" is checked for the users.
  • Edit Moloch's config.ini
    • Create a new section for the moloch proxy.
    • Set userNameHeader to the lower case version of the header apache is setting.
    • Set the webBasePath to the path used above. All other sections should NOT have a webBasePath.
    • Add a viewHost, so externals can't just set the userNameHeader and access moloch with no auth: 
[moloch-proxy]
userNameHeader=remote_user
webBasePath = /moloch/
viewPort = 8005
viewHost = localhost
  • Start the moloch-proxy viewer.
  • To prevent the users from going directly to moloch in the future, scramble their passwords. You might want to leave an admin user that doesn't use the apache auth. Or you can temporarily add one with the addUser.js script.

How do I search multiple Moloch clusters

It is possible to search multiple moloch clusters by setting up a special multiple moloch viewer and a special multies process. Multies is similiar to Elasticsearch tribe nodes, except it was created before tribe nodes and can deal with multiple indexes having the same name. One big limitation currently is that all the moloch clusters have to use the same rotateIndex setting.

To use, create another config.ini file or section in a shared config file. Both multies.js and the special "all" viewer can use the same node name. 

# viewer/multies node name (-n allnode)
[allnode]
# The host and port multies is running on, set with multiESHost:multiESPort usually just run on the same host
elasticsearch=127.0.0.1:8200
# This is a special multiple moloch cluster viewer
multiES=true
# Port the multies.js program is listening on, elasticsearch= must match
multiESPort = 8200
# Host the multies.js program is listening on, elasticsearch= must match
multiESHost = localhost
# Semicolon list of elasticsearch instances, one per moloch cluster.  The first one listed will be used for settings
multiESNodes = es-cluster1.example.com:9200;es-cluster2.example.com:9200

Now you need to start up both the multies.js program and viewer.js with the config file. All other viewer settings, including webBasePath can still be used.

By default, the users table comes from the first cluster listed in multiESNodes. This can be overridden by setting usersElasticsearch and optionally usersPrefix in the multi viewer config file.

How do I reset my password?

An admin can change anyone's password on the Users tab by clicking the Settings link in the Actions column next to the user.

A password can also be changed by using the addUser script, which will replace the entire account if the same userid is used. All preferences and views will be cleared, so creating a secondary admin account may be a better option if you need to change an admin users password. After creating a secondary admin account, change the users password and then delete the secondary admin account. 

node addUser -c <configfilepath> <user id> <user friendly name> <password> [--admin]

Error: Couldn't connect to remote viewer, only displaying SPI data

First, look at viewer.log on both the viewer machine and the remote machine and see if there are any obvious errors. The most common problems are:

  1. The remote machine doesn't return a FQDN from the hostname command AND the viewer machine can't resolve just the hostname. Before version 0.14.1 the domainname command was ignore. To fix this, do ONE of the following:
    1. Make it so the remote machines returns a FQDN (hostname "fullname" as root and edit /etc/sysconfig/network)
    2. Set a viewHost in each node section of the config.ini. If you don't have a node section for each host, you'll need to create one.
    3. Edit /etc/resolv.conf and add search foo.example.com, where foo.example.com is the subdomain of the hosts. Basically, you want it so "telnet shortname 8005" works on the viewer machine to the remote machine.
  2. The remote machine is using a self signed cert. To fix this, either turn off https or see the certificate answer above.
  3. The remote machine can't open the pcap. Make sure the dropUser user can read the pcap files. Check the directories in the path too.

Moloch Wiki

Add a custom sidebar

Clone this wiki locally
0