Repository navigation
Expand file tree
/
Copy pathREADME
More file actions
317 lines (240 loc) · 11.3 KB
/
Copy pathREADME
File metadata and controls
317 lines (240 loc) · 11.3 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
This document describes how to install the pWeb DNS Gateway and
Crawler on Ubuntu 13.04 (Raring Ringtail) and other versions of
Ubuntu. The DNS Gateway and Crawler have been tested on
Ubuntu 12.04 LTS, 12.10, and 13.04. The DNS Gateway and
Crawler/Poller have a number of dependencies that can be easily
installed using a few simple commands.
Earlier versions of the DNS Gateway and Poller/Crawler have also
been tested on Windows Server 2008 (equivalent to Windows XP/Vista
for desktops) and Windows Server 2008 R2 (equivalent to Windows 7
for desktops). However, installing the dependencies and compiling
on Windows is much more complicated, and isn't described here.
Additionally, the software has not been tested on Windows in
quite some time and will likely need some source code modifications
to compile successfully.
You should have received a source distribution of the DNS Gateway
and Crawler well as a binary distribution. (You probably opened
this file from the source distribution). The binary distribution
is the simplest way to install the DNS Gateway and Crawler on
Ubuntu 13.04.
----------------------------------------------------------------------
Installation Instructions for Ubuntu 13.04 from BINARY
----------------------------------------------------------------------
Note: These instructions are for Ubuntu 13.04 (raring) only. For
installation on other versions of Ubuntu, see the following section.
First install the dependencies:
$ sudo apt-get install libboost-program-options1.53.0 \
libboost-thread1.53.0 \
libboost-date-time1.53.0 \
libboost-chrono1.53.0 \
libboost-serialization1.53.0 \
libcurl3 \
liblog4cpp5
Now install the DNS Gateway and poller binaries. The following
commands assume that you have extracted the DNS Gateway/Crawler
binary archive to <pweb>:
$ cd <pweb>
$ sudo install dnsgw.bin/dnsgw /usr/local/bin
$ sudo install crawler_poller.bin/poller /usr/local/bin
Now you are ready to proceed to the section "Configuring
the DNS Gateway and Crawler."
----------------------------------------------------------------------
Installation Instructions for Ubuntu 12.04, 12.10 or 13.04 from SOURCE
----------------------------------------------------------------------
If you are installing the software on Ubuntu 12.04 or 12.10
you need to install the Boost libraries from Alexander Pokluda's
PPA, because the official repositories for Ubuntu 12.04 and 12.10
provide only older versions of Boost. (The software has been
developed and tested with Boost 1.53. Newer versions of Boost
should work, but haven't been tested, because they don't exist yet!).
Follow the instructions on the PPA archive page to install
the Boost 1.53 PPA: https://launchpad.net/~apokluda/+archive/boost1.53.
Now install the dependencies:
$ sudo apt-get install cmake libboost1.53-all-dev \
libcurl4-openssl-dev liblog4cpp5-dev build-essential
We will use CMake to generate GNU Makefiles that are used to
compile the DNS Gateway and Crawler/Poller. The recommended way
to use CMake is to do "out of source builds". This means that
we will leave the source directory untouched and use a
separate directory to compile the software. The following
commands assume that you have extracted the DNS Gateway/Crawler
source archive to <src>:
$ mkdir pweb
$ cd pweb
$ cmake <src>
The compilation process uses a precompiled header to reduce the compile
time. Due to a bug in the build system, we need to link the
precompiled header source file to the build directory:
$ ln <src>/shared/stdhdr.hpp .
Now we can continue with the compilation:
$ make
$ make install
Now continue with the next section.
----------------------------------------------------------------------
Configuring the DNS Gateway and Crawler
----------------------------------------------------------------------
This section assumes that you have completed either the section about
installing from BINARY or SOURCE above.
In order to run the DNS Gateway and Crawler as a daemon,
first install the daemon package:
$ sudo apt-get install daemon
Now the DNS Gateway and Crawler can be launched on system
startup and controlled as a system service using Upstart.
Create the following two files in /etc/init:
--- /etc/init/dnsgw.conf ---
description "pWeb DNS Gateway Server"
author "Alexander Pokluda <apokluda@uwaterloo.ca>"
start on runlevel [2345]
stop on runlevel [!2345]
exec daemon --user=daemon:daemon --name=dnsgw --respawn -- /usr/local/bin/dnsgw -c /etc/dnsgw.conf
--- end of file---
--- /etc/init/crawler.conf ---
description "pWeb Device Crawler - Poller Process"
author "Alexander Pokluda <apokluda@uwaterloo.ca>"
start on runlevel [2345]
stop on runlevel [!2345]
exec daemon -c --user=daemon:daemon --name=poller --respawn -- /usr/local/bin/poller -c /etc/poller.conf
--- end of file ---
Note that the crawler init script starts only one poller without a manager process.
The above scripts use DNS Gateway and Crawler configuration files /etc.
The following configuration files can be used as a starting point for your own.
Note that dnsgw.conf tells the DNS Gateway to listen for DNS queries on
127.0.0.1:5533. The next section explains how to run all the pWeb components on a
single machine with the DNS Gateway listening on the loopback interface.
When running the DNS Gateway in a production environment, you must tell the
DNS Gateway to listen on a public interface and on port 53 and point both
the DNS Gateway and poller to your Home Agents.
--- /etc/dnsgw.conf ---
# pWeb DNS Gateway Configuration File
#
# The option names and values are the same on the command line and in
# this configuration file. More information about each option can
# be see by running
#
# $ dnsgw -h
#
# on the command line.
# The verbosity of messages printed to the log file
# Valid options are: DEBUG, INFO, NOTICE, WARN, ERROR, CRIT, ALERT, FATAL = EMERG
log_level=INFO
# Path to the log file
log_file=/var/log/dnsgw.log
# The hostname of the dnsgw
nshostname=dnsgw1.pwebproject.net.
# The interface to use for listening for DNS requests
iface=127.0.0.1
# The port to use for listing for DNS requests
port=5533
# The port to use to receive responses from the Home Agent
nsport=56142
# The suffix that must be removed from DNS queries to get
# a device name. (This device name will be sent to the Home Agent).
suffix=dht.pwebproject.net.
# The TTL value to insert into DNS responses
ttl=0
# Home Agents that can respond to queries for device name lookups.
# Any number of home agents can be specified by listing the 'home_agent'
# parameter multiple times. The value has the format '<host name>:<port>'
# where <port> is the port number of the TCP interface.
# If the Home Agents are not able to answer a query locally, they
# will lookup the device's IP address in the Plexus network on behalf
# of the DNS Gateway.
#
# Example:
#home_agent=cn101.cs.uwaterloo.ca:20000
#home_agent=cn102.cs.uwaterloo.ca:20000
home_agent=localhost:20000
--- end of file ---
--- /etc/poller.conf ---
# pWeb Crawler - Poller Process Configuration File
#
# The option names and values are the same on the command line and in
# this configuration file. More information about each option can
# be seen by running
#
# $ poller -h
#
# on the command line.
# The verbosity of messages printed to the log file
# Valid options are: DEBUG, INFO, NOTICE, WARN, ERROR, CRIT, ALERT, FATAL = EMERG
log_level=INFO
# Path to the log file
log_file=/var/log/poller.log
# The polling interval for each Home Agent. A value of x means
# that each Home Agent will be polled once every x seconds.
interval=5
# The URLs of the Apache Solr server to send updates to for the
# devices and content cores (databases)
soldevrurl=http://localhost:8081/solr/pweb_devices/update
solconrurl=http://localhost:8081/solr/pweb_content/update
# Home Agents to seed the crawling process. Any number of Home Agents
# can be specified. The poller process will use the default HTTP port
# for the home agents. If you are running your own Home Agent(s),
# you need to put there addresses here!
#
# Example:
#home_agent=cn101.cs.uwaterloo.ca
#home_agent=cn102.cs.uwaterloo.ca
home_agent=localhost
--- end of file ---
If you want the DNS Gateway to accept DNS
813A
queries on the default
port of 53 (eg, for a production server), you need to give it permission
to bind to privileged ports. This can be done with the command
$ sudo setcap 'cap_net_bind_service=+ep' /usr/local/bin/dnsgw
When run as a daemon, the DNS Gateway and poller will not
have permission to create their log files in the /var/log
directory. In order to allow the to write to this directory,
we have to create their log files manually:
$ sudo touch /var/log/dnsgw.log && sudo chown daemon:daemon /var/log/dnsgw.log
$ sudo touch /var/log/poller.log && sudo chown daemon:daemon /var/log/poller.log
Once the above configuration files are in place and you have run
the above commands, you should be able to start the DNS Gateway
and poller manually with the commands
$ sudo service dnsgw start
$ sudo service crawler start
You can monitor the DNS Gateway and poller log files in real-time
with the command
$ tail -f /var/log/dnsgw.log
or
$ tail -f /var/log/poller.log
If the DNS Gateway and Poller started successfully, you should
see the following output in each log file:
--- dnsgw.log ---
2013-08-08 17:43:38,069 [INFO] Listening on all interfaces port 56142 for home agent connections
2013-08-08 17:43:38,069 [INFO] Listening on 127.0.0.1 port 5533 for UDP connections
2013-08-08 17:43:38,069 [INFO] Listening on 127.0.0.1 port 5533 for TCP connections
--- poller.log ---
2013-08-08 18:05:07,750 [INFO] --- pWeb Crawler Poller Process v0.0.0 started ---
2013-08-08 18:05:07,750 [INFO] Home Agent 'localhost' has been assigned for monitoring
2013-08-08 18:05:07,753 [NOTICE] Now monitoring 1 Home Agents
------------------------------------------------------------
Configuring Delegation to the DNS Gateway
------------------------------------------------------------
In order for the DNS Gateway to be useful, you need to
configure some sort of delegation to the DNS Gateway. For
example, the pWeb Project Team has registered pwebproject.net
and have multiple BIND DNS Servers configured to handle
queries for that domain. The bind servers are configured to
delegate queries for *.dht.pwebproject.net to a pool of
DNS Gateway instances. This setup is beyond the scope of this
README, but the following shows how you can set up DNSmasq
on your local machine to forward queries to *.dht.pwebproject.net
to a DNS Gateway also running on your local machine. This
section assumes that you have already followed the instructions
above and that you are installing and running/demonstrating
everything on a single machine that is running Ubuntu 13.04
with the Unity desktop environment installed and using
the Network Manager GUI.
Use you favourite text editor to create the file
/etc/NetworkManager/dnsmasq.d/dnsgw.conf
Add the following two lines to that file:
server=/dht.pwebproject.net/127.0.0.1#5533
host-record=dnsgw1.pwebproject.net,127.0.0.1
You should now have a fully functioning DNS Gateway and Crawler
setup. Next you will need to install the Solr and Django. Instructions
and configuration files for setting up these services are provided
in separate archives. You will also need to install and set up
your Home Agent(s) and registration web server if you haven't done so
already.
If you need help, please feel free to contact me at apokluda@uwaterloo.ca.
Good luck!