1
INTERNET CLIENT PROGRAMMING USING PYTHON
C U A U H T E M O C C A R B A J A L I T E S M C E M A P R I L 0 6 , 2 0 1 3
INTERNET CLIENT PROGRAMMING USING PYTHON C U A U H T E M O C C A R - - PowerPoint PPT Presentation
INTERNET CLIENT PROGRAMMING USING PYTHON C U A U H T E M O C C A R B A J A L I T E S M C E M A P R I L 0 6 , 2 0 1 3 1 INTRODUCTION In the previous lecture we took a look at low-level networking communication protocols using sockets.
1
C U A U H T E M O C C A R B A J A L I T E S M C E M A P R I L 0 6 , 2 0 1 3
2
3
4
parameters
necessary headers)
5
import smtplib, socket frad = "sholden@holdenweb.com" toads = ["bookuser@holdenweb.com", "nosuchuser@holdenweb.com", "sholden@holdenweb.com"] msg = """To: Various recipients From: Steve Holden <sholden@holdenweb.com>
"""
6
try: server = smtplib.SMTP('10.0.0.1') result = server.sendmail(frad, toads, msg) server.quit() if result: for r in result.keys(): print "Error sending to", r rt = result[r] print "Code", rt[0], ":", rt[1] else: print "Sent without errors" except smtplib.SMTPException, arg: print "Server could not send mail", arg
7
can be very useful to be able to send emails from the Raspberry Pi to arbitrary recipients.
MTA running on the Pi (like Sendmail, Postfix, Exim, QMail, etc.), which can also receive and store emails.
going to cover the possibility of sending emails, not receiving.
people tend to use GMail, Yahoo! Mail and other major email service providers and they store their emails
Raspberry Pi can come in handy in many situations.
8
the Pi and you could program the Pi to send you an email
threshold values,
with summarized system data.
and set up some motion detection software, which would send you an email as soon as it detects motion in a given area
Raspberry Pi and you would like to provide your readers with the possibility to subscribe to the posts.
9
10
the host’s IP address or a domain name like tutorialspoint.com. This is optional argument.
specify a port where SMTP server is listening. Usually this port would be 25.
machine then you can specify just localhost as of this option.
import smtplib smtpObj = smtplib.SMTP( [host [, port [, local_hostname]]] )
11
as a list with 1 address),
with all RCPT commands can be passed as rcpt_options. (If you need to use different ESMTP options to different recipients you have to use the low-level methods such as mail(), rcpt() and data() to send the message.)
12
name of the local host. Examine the response for ESMTP option and store them for use by has_extn(). Also sets several informational attributes: the message returned by the server is stored as the ehlo_resp attribute, does_esmtp is set to true or false depending on whether the server supports ESMTP, and esmtp_features will be a dictionary containing the names of the SMTP service extensions this server supports, and their parameters (if any).
13
14
import smtplib to = 'carbajal@itesm.mx' gmail_user = 'carbajal.cuauhtemoc@gmail.com' gmail_pwd = '28/Julio/1955' smtpserver = smtplib.SMTP("smtp.gmail.com",587) smtpserver.ehlo() smtpserver.starttls() smtpserver.ehlo smtpserver.login(gmail_user, gmail_pwd) header = 'To:' + to + '\n' + 'From: ' + gmail_user + '\n' + 'Subject:testing \n' print header msg = header + '\n this is test msg from my RPi \n\n' smtpserver.sendmail(gmail_user, to, msg) print 'done!' smtpserver.close() email-example2.py
15
http://learn.adafruit.com/arduino-lesson-17-email-sending-movement-detector 16
17
int pirPin = 7; int minSecsBetweenEmails = 60; // 1 min long lastSend = -minSecsBetweenEmails * 1000l; void setup() { pinMode(pirPin, INPUT); Serial.begin(9600); }
18
void loop() { long now = millis(); if (digitalRead(pirPin) == HIGH) { if (now > (lastSend + minSecsBetweenEmails * 1000l)) { Serial.println("MOVEMENT"); lastSend = now; } else { Serial.println("Too soon"); } } delay(500); }
19
import time import serial import smtplib TO = 'putyour@email.here' GMAIL_USER = 'putyour@email.here' GMAIL_PASS = 'putyourpasswordhere' SUBJECT = 'Intrusion!!' TEXT = 'Your PIR sensor detected movement' ser = serial.Serial('/dev/ttyACM0', 9600)
20
def send_email(): print("Sending Email") smtpserver = smtplib.SMTP("smtp.gmail.com",587) smtpserver.ehlo() smtpserver.starttls() smtpserver.ehlo smtpserver.login(GMAIL_USER, GMAIL_PASS) header = 'To:' + TO + '\n' + 'From: ' + GMAIL_USER header = header + '\n' + 'Subject:' + SUBJECT + '\n‘ print header msg = header + '\n' + TEXT + ' \n\n‘ smtpserver.sendmail(GMAIL_USER, TO, msg) smtpserver.close()
21
while True: message = ser.readline() print(message) if message[0] == 'M' : send_email() time.sleep(0.5)
22
23
24
parameters
25
mailbox
26
import poplib, rfc822, sys, StringIO SRVR = 'pop.googlemail.com' PORT = '995' USER = "carbajal.cuauhtemoc@gmail.com" PASS = "28/Julio/1955" try: p = poplib.POP3_SSL(SRVR, PORT) except: print "Can't contact %s" % (SRVR, ) sys.exit(-1) try: print p.user(USER) print p.pass_(PASS) except: print "Authentication failure" sys.exit(-2)
27
msglst = p.list()[1] for m in msglst: mno, size = m.split() lines = p.retr(mno)[1] print "----- Message %s" % (mno, ) file = StringIO.StringIO("\r\n".join(lines)) msg = rfc822.Message(file) body = file.readlines() addrs = msg.getaddrlist("to") for rcpt, addr in addrs: print "%-15s %s" % (rcpt, addr) print len(body), "lines in message body" print "-----" p.quit()
pop-example.py (C3PO)
28
widely used protocols on the Internet, invoked whenever a user wanted to transfer files between Internet-connected computers.
1. File download: documents, source code, media like images or movies
typed your e-mail address as the password.
between computer accounts, since trying to transfer large files with Telnet clients was often a dicey proposition.
2. Anonymous upload.
files, and their solution was to set up FTP servers that allowed files to be written into a directory whose contents could not, then, be listed back again. That way, users could not see (and hopefully could not guess!) the names of the files that other users had just submitted and get to them before the site administrators did.
29
3. To support the synchronization of entire trees of files between computer accounts.
push entire directory trees from one of their accounts to another, and server administrators could clone or install new services without having to re-build them from scratch on a new machine.
protocol worked, or of the many separate commands needed to transfer so many different files: instead, they hit a button and a large batch operation would run and then complete.
4. Interactive, full-fledged file management
like a Unix shell account itself, and—as we shall see—the protocol borrows from shell accounts both the idea of a “current working directory” and of a cd command to move from one directory to another. Later clients mimicked the idea of a Mac-like interface, with folders and files drawn on the computer screen.
FTP finally came into play: it supported not only the operations of listing directories and uploading and downloading files, but of creating and deleting directories, adjusting file permissions, and re-naming files.
30
clear and can be viewed by anyone observing network traffic.
several operations all over the same network connection. Modern Internet services, with millions of users, prefer protocols like HTTP that consist of short, completely self-contained requests, instead of long-running FTP connections that require the server to remember things like a current working directory.
filesystem that the owner wanted exposed, tended to simply expose the entire filesystem, letting users cd to / and snoop around to see how the system was configured. True, you could run the server under a separate ftp user and try to deny that user access to as many files as possible; but many areas of the Unix filesystem need to be publicly readable simply so that normal users can use the programs there.
filesystem, this more or less violated the original intention: that an FTP session would look like a Telnet command-line prompt, even down to the fact that full pathnames were used that started at the filesystem root.
31
32
1. For file download, HTTP is the standard protocol on today’s Internet, protected with SSL when necessary for security. Instead of exposing system-specific file name conventions like FTP, HTTP supports system-independent URLs. 2. Anonymous upload is a bit less standard, but the general tendency is to use a form on a web page that instructs the browser to use an HTTP POST operation to transmit the file that the user selects. 3. File synchronization has improved immeasurably since the days when a recursive FTP file copy was the only common way to get files to another computer. Instead of wastefully copying every file, modern commands like rsync or rdist efficiently compare files at both ends of the connection and copy only the ones that are new or have changed. 4. Full filesystem access is actually the one area where FTP can still commonly be found on today’s Internet: thousands of cut-rate ISPs continue to support FTP, despite its insecurity, as the means by which users copy their media and (typically) PHP source code into their web account. A much better alternative today is for service providers to support SFTP instead.
33
and e-mail address)
remote host and FTP server
34
The client and server communicate using the FTP protocol on the command or control port data; is transferred using the data port.
35
maxblocksize[, rest]])
36
37
class ftplib.FTP([host[, user[, passwd[, acct[, timeout]]]]])
method call connect(host) is made. When user is given, additionally the method call login(user, passwd, acct) is made (where passwd and acct default to the empty string when not given). The optional timeout parameter specifies a timeout in seconds for blocking operations like the connection attempt (if is not specified, the global default timeout setting will be used). class ftplib.FTP_TLS([host[, user[, passwd[, acct[, keyfile[, certfile[, timeout]]]]]]])
control connection before authenticating. Securing the data connection requires the user to explicitly ask for it by calling the prot_p() method. keyfile and certfile are optional – they can contain a PEM formatted private key and certificate chain file name for the SSL connection.
38
>>> from ftplib import FTP >>> f = FTP('ftp.ibiblio.org') >>> print "Welcome:", f.getwelcome() Welcome: 220 ProFTPD Server >>> f.login() "230-\n Welcome to ftp.ibiblio.org... ...Anonymous access granted, restrictions apply" >>> print "Current working directory:", f.pwd() Current working directory: / >>> f.quit
39
#!/usr/bin/env python # ASCII download – ftp-text_example.py # Downloads README from remote and writes it to disk. import os from ftplib import FTP if os.path.exists('README'): raise IOError('refusing to overwrite your README file') def writeline(data): fd.write(data) fd.write(os.linesep) f = FTP('ftp.kernel.org') f.login() f.cwd('/pub/linux/kernel') fd = open('README', 'w') f.retrlines('RETR README', writeline) fd.close() f.quit() ftp-text_example.py C3PO
40
>>> from ftplib import FTP >>> ftp = FTP('ftp.kernel.org') # connect to host, default port >>> ftp.login() # user anonymous, passwd anonymous@ >>> f.cwd('/pub') >>> ftp.retrlines('LIST') # list directory contents
1 ftp ftp 578 Mar 18 2003 README_ABOUT_BZ2_FILES drwxr-xr-x 6 ftp ftp 4096 Dec 01 2011 dist drwxr-xr-x 13 ftp ftp 4096 Nov 16 2011 linux drwxr-xr-x 3 ftp ftp 4096 Sep 23 2008 media drwxr-xr-x 17 ftp ftp 4096 Jun 06 2012 scm drwxr-xr-x 2 ftp ftp 4096 Dec 01 2011 site drwxr-xr-x 13 ftp ftp 4096 Nov 27 2011 software drwxr-xr-x 3 ftp ftp 4096 Apr 30 2008 tools '226 Directory send OK.‘ >>> ftp.retrbinary('RETR README_ABOUT_BZ2_FILES',open('README','wb').write) '226 Transfer complete.' >>> ftp.quit()
41
>>> from ftplib import FTP_TLS >>> ftps = FTP_TLS('ftp.python.org') >>> ftps.login() # login anonymously before securing control channel >>> ftps.prot_p() # switch to secure data connection >>> ftps.retrlines('LIST') # list directory content securely total 9 drwxr-xr-x 8 root wheel 1024 Jan 3 1994 . drwxr-xr-x 8 root wheel 1024 Jan 3 1994 .. drwxr-xr-x 2 root wheel 1024 Jan 3 1994 bin drwxr-xr-x 2 root wheel 1024 Jan 3 1994 etc d-wxrwxr-x 2 ftp wheel 1024 Sep 5 13:43 incoming drwxr-xr-x 2 root wheel 1024 Nov 17 1993 lib drwxr-xr-x 6 1094 wheel 1024 Sep 13 19:07 pub drwxr-xr-x 3 root wheel 1024 Jan 3 1994 usr
312 Aug 1 1994 welcome.msg '226 Transfer complete.' >>> ftps.quit()
42
class Writer: def __init__(self, file): self.f = open(file, "w") def __call__(self, data): self.f.write(data) self.f.write('\n') print data FILENAME = "AutoIndent.py" writer = Writer(FILENAME) import ftplib ftp = ftplib.FTP('127.0.0.1', 'book', 'bookpw') ftp.retrlines("RETR %s" % FILENAME, writer)
43
44
identified resource
local file
45
>>> import urllib >>> f = urllib.urlopen("http://www.python.org/") >>> page = f.read() >>> len(page) 21801 >>> h=f.info() >>> h.getheader("Server") 'Apache/2.2.16 (Debian)' >>> h.getheaders("Date") ['Mon, 01 Apr 2013 20:32:46 GMT'] >>> h.type 'text/html'
46
an instance to urllib._urlopener
47
given scheme
48
49
import CGIHTTPServer, BaseHTTPServer httpd = BaseHTTPServer.HTTPServer(('', 8888), CGIHTTPServer.CGIHTTPRequestHandler) httpd.serve_forever()
50
51
52
the URL ie. filename.
in if it finds that file then sends back to the browser
requested a wrong file.
either the received file or error message.
53
CGI Architecture Diagram
54
Server supports CGI and it is configured to handle CGI Programs. All the CGI Programs to be executed by the HTTP server are kept in a pre- configured directory. This directory is called CGI Directory and by convention it is named as /var/www/cgi-bin. By convention CGI files will have extention as .cgi but you can keep your files with python extension .py as well.
bin directory in /var/www. If you want to specify any other directory to run your CGI scripts, comment the following lines in the httpd.conf file:
<Directory "/var/www/cgi-bin"> AllowOverride None Options ExecCGI Order allow,deny Allow from all </Directory> <Directory "/var/www/cgi-bin"> Options All </Directory>
and you are able to run any other CGI program like Perl or Sheel etc.
55
hello.py. This file is being kept in /var/www/cgi-bin directory and it has following content. Before running your CGI program make sure you have chage mode of file using chmod 755 hello.py UNIX command to make file executable.
#!/usr/bin/python print "Content-type:text/html\r\n\r\n" print '<html>' print '<head>' print '<title>Hello Word - First CGI Program</title>' print '</head>' print '<body>' print '<h2>Hello Word! This is my first CGI program</h2>' print '</body>' print '</html>'
Hello Word! This is my first CGI program
56
57
multi-valued form inputs
vs list)
58
59
60
#!/usr/bin/python import cgi, cgitb; cgitb.enable() fields = ["subnum", "reviewer", "comments"] form = cgi.FieldStorage() vlist = [] for f in fields: vlist.append("%s=%s" % (f, form.getfirst(f))) print pgtmpl = """Content-Type: text/html <html><body><head><title>Hello!</title></head> %s </body></html> """ % "<br>".join(vlist)
Support module for Common Gateway Interface (CGI) scripts.
Store a sequence of fields, reading multipart/form-data. This method always returns only one value associated with form field name. Return a string which is the concatenation of the strings in the iterable iterable.
61
62
# Time server program from socket import * import time s = socket(AF_INET, SOCK_STREAM) # Create TCP socket s.bind(("",8888)) # Bind to port 8888 s.listen(5) # Start listening while 1: client,addr = s.accept() # Wait for a connection print "Got a connection from ", addr client.send(time.ctime(time.time())) # Send time back client.close()
63
using send() and recv().
# Time client program from socket import * s = socket(AF_INET,SOCK_STREAM) # Create TCP socket s.connect((“127.0.0.1",8888)) # Connect to server tm = s.recv(1024) # Receive up to 1024 bytes s.close() # Close connection print "The time is", tm
64
conversion, etc...)
machine).
socket.gethostbyname(hostname) # Get IP address for a host socket.gethostname() # Name of local machine socket.ntohl(x) # Convert 32-bit integer to host order socket.ntohs(x) # Convert 16-bit integer to host order socket.htonl(x) # Convert 32-bit integer to network order socket.htons(x) # Convert 16-bit integer to network order
65
Stream socket (TCP)
Datagram socket (UDP)
66
# Accept a new connection
# Bind to an address and port
# Close the socket
# Return integer file descriptor
# Get name of remote machine
# Get socket address as (ipaddr,port)
# Get socket options
# Start listening for connections
# Turn socket into a file object
# Receive data
# Send data
# Set socket options
# Shutdown one or both halves of connection
67
# Simple time server import SocketServer import time # This class actually implements the server functionality class TimeHandler(SocketServer.BaseRequestHandler): def handle(self): self.request.send(time.ctime(time.time())) # Create the server server = SocketServer.TCPServer(("",8888),TimeHandler) server.serve_forever()
68
FTP protocol
SMTP (mail) protocol
News
Gopher
POP3 mail server
IMAP4 mail server
Telnet protocol
HTTP protocol
low-level.
doing.
69
GET /index.html HTTP/1.0 Connection: Keep-Alive Host: www.python.org User-Agent: Mozilla/4.61 [en] (X11; U; SunOS 5.6 sun4u) [blank line]
HTTP/1.0 200 OK Content-type: text/html Content-length: 72883 Headers: blah [blank line] Data ...
70
import httplib h = httplib.HTTP("www.python.org") h.putrequest('GET','/index.html') h.putheader('User-Agent','Lame Tutorial Code') h.putheader('Accept','text/html') h.endheaders() errcode,errmsg, headers = h.getreply() f = h.getfile() # Get file object for reading data data = f.read() f.close()
71
remote servers
import urllib f = urllib.urlopen("http://www.python.org/index.html") data = f.read() f.close()
urllib.quote(str) # Quotes a string for use in a URL urllib.quote_plus(str) # Also replaces spaces with ’+’ urllib.unquote(str) # Opposite of quote() urllib.unquote_plus(str) # Opposite of quote_plus() urllib.urlencode(dict) # Turns a dictionary of key=value # pairs into a HTTP query-string
urllib.quote("beazley@cs") # Produces "beazley%40cs" urllib.unquote("%23%21/bin/sh") # Produces "/bin/sh"
72
Functions for manipulating URLs
import urlparse t = urlparse.urlparse("http://www.python.org/index.html") # Produces (’http’,’www.python.org’,’/index.html’,’’,’’,’’)
string
url = urlparse.urlunparse(('http','www.python.org','foo.html', 'bar=spam','')) # Produces "http://www.python.org/foo.html?bar=spam"
urlparse.urljoin("http://www.python.org/index.html","help.h tml") # Produces "http://www.python.org/help.html"
73
response to HTTP requests.
<form method="GET" action="cgi-bin/spam.cgi"> Your name: <input type="text" name="name" size=30><p> Your email: <input type="text" name="email" size=40><p> <input type="submit" value="Submit"></form>
GET /cgi- bin/spam.cgi?name=Dave+Beazley&email=beazley%40cs HTTP/1.0
parameters
74
import pywapi import string yahoo_result = pywapi.get_weather_from_yahoo(‘10001') print "Yahoo says: It is " + string.lower(yahoo_result['condition']['text']) + " and " + yahoo_result['condition']['temp'] + "C now in New York.\n\n" The module provides a python wrapper around the Yahoo! Weather API. http://code.google.com/p/python-weather-api/ python-weather-api.py C3PO
75
76