{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Advanced Topics\n", "\n", "In this lecture, we will go over some advanced topics like\n", "- Templates\n", "- Argparse\n", "- MultiThreading\n", "- Networking\n", "- C Extension\n", "- PyCrypto\n", "\n", "**Created and updated by** John C. S. Lui on August 14, 2020.\n", "\n", "**Important note:** *If you want to use and modify this notebook file, please acknowledge the author.*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Topic 1: Template\n", "\n", "- Part of the String module\n", "- Allows for data transformation without having to edit the application\n", "- Can be modified with subclasses\n", "\n", "# Operations\n", "- Takes a string as a template with placeholder variables\n", "- **Substitute** the values with a dictionary, with the key being the placeholder name\n", "- Placeholder names follow the same rules as variable data type in Python\n", "- Template(“\\$name is friends with $friend”)\n", "\n", "# Example\n", "- Let's create a program which will output all the items in a users *e-cart*\n", "- Create a template for that output then fill it with the cart info.\n", "\n", "Let's illustrate." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Example 1" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# import Template\n", "\n", "from string import Template \n", "\n", "def Main():\n", " cart = [] # start with an empty list\n", " \n", " # append dictionary with 3 keys (item, price, qty)\n", " cart.append(dict(item=\"iPhone\", price=4000.0, qty=1)) \n", " cart.append(dict(item=\"Samsung\", price=3999.9, qty=1))\n", " cart.append(dict(item=\"Huawei\", price=15999.2, qty=4))\n", "\n", " # create a template with '$' being placeholder\n", " t = Template(\"$qty items of $item = $price\")\n", " \n", " total = 0\n", " print(\"For this cart:\")\n", " for data in cart:\n", " print(t.substitute(data), end='; ') # the use of template with substitute\n", " print('Unit price is: ', data[\"price\"]/data[\"qty\"])\n", " total += data[\"price\"]\n", " print(\"\\nTotal: \" + str(total))\n", " \n", "if __name__ == '__main__':\n", " Main()\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Some common template errors\n", "\n", "- No Placeholder match, this will result in a *KeyError* \n", "- Bad placeholder, this will result in a *Value Error*\n", "\n", "## Solution: *safe_substitute()* \n", "\n", "- Templates can handle these errors and give us back a string if we use the method *safe_substitute()*\n", "- We can also get the placeholders in the returned string if there is an error with it. E.g., : Template(“\\$name had \\$money”), output: **“Jim had \\$money”**\n", "\n", "## Make your own delimiters\n", "- One can set a customized delimiter for the placeholder variables by overriding the Template Class\n", "- E.g., change our delimeter to the Hash symbol (#)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Example 2" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "\n", "from string import Template\n", "\n", "class MyTemplate(Template):\n", " delimiter = '#'\n", "\n", "def Main():\n", " cart = []\n", " cart.append(dict(item=\"iPhone\", price=4000.0, qty=1))\n", " cart.append(dict(item=\"Samsung\", price=3999.9, qty=1))\n", " cart.append(dict(item=\"Huawei\", price=15999.2, qty=4))\n", "\n", " # Now we have our own delimeter '#'\n", " t = MyTemplate(\"#qty items of #item = #price\")\n", " total = 0\n", " print(\"For this cart:\")\n", " for data in cart:\n", " print(t.substitute(data), end='; ') # the use of template with substitute\n", " print('Unit price is: ', data[\"price\"]/data[\"qty\"])\n", " total += data[\"price\"]\n", " print(\"Total: \" + str(total))\n", " \n", "\n", "if __name__ == '__main__':\n", " Main()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Motiviation of using template\n", "\n", "- Saves time typing \n", "- Reduces code size.\n", "- Extremely useful for webpage’s since a webpage generally follows the same template but with different data.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Topic #2: Argparse\n", "\n", "## Motiviation of using template\n", "\n", "- Argparse module allows argument parsing for our python programs\n", "- Automatically generates the usage\n", "- Has built-in help functions\n", "- Auto formats the output for the console\n", "\n", "\n", "## Argparse usage\n", "\n", "- Interfaces with the python system module so to grab the arguments from the command line.\n", "- Supports checking and making sure required arguments are provided.\n", "- E.g., *python example.py 10*\n", "- **USAGE FORMAT**:\n", " - parser = argparse.ArgumentParser()
\n", " parser.add_argument(\"num\", help=“help text”,type=int)
\n", " args = parser.parse_args()\n", "- **Position Arguments**:\n", " - Positional arguments are *required arguments* that we need for our program \n", " - Positional arguments do not require the dash(-) because it is not an option\n", " \n", "## Example: fibonacci number" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# MAKE SURE TO RUN IN THE TERMINAL !!!!!!\n", "\n", "# fibonacci_1.py: example of a fibonacci number\n", "# try to run it with %python3 fibonacci_1.py\n", "# %python3 fibonacci_1.py -h\n", "# %python3 fibonacci_1.py 12\n", "\n", "import argparse\n", "\n", "def fib(n):\n", " a, b = 0, 1\n", " for i in range(n):\n", " a, b = b, a+b\n", " return a\n", "\n", "\n", "def Main():\n", " parser = argparse.ArgumentParser()\n", " parser.add_argument(\"num\", help=\"The Fibonacci number you wish to calculate.\", type=int)\n", "\n", " # get the argument\n", " args = parser.parse_args()\n", "\n", " result = fib(args.num)\n", " print(\"The \" + str(args.num)+ \"th fibonacci number is \" + str(result))\n", "\n", "if __name__ == '__main__':\n", " Main()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Optional Arguments\n", "\n", "- The optional arguments are, well, *optional*.\n", "- The -h option is already inbuilt by default\n", "- We can create as many optional arguments as we like and argparse will handle it.\n", "- Like the positional arguments the help will be automatically added to the help output.\n", "- Example:\n", " - parser.add_argument(“--quiet\", help=“help text”, action=“store_true”)\n", " \n", "# Fibonacci program\n", "\n", "- We modify fibonacci_1.py\n", "- Will take the Fibonacci number to output using a position argument\n", "- Optional: output number to file. “--output”\n", "- Add a short option aswell. “-o”\n", "- Add help for the optional output\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# MAKE SURE TO RUN IN THE TERMINAL !!!!!!\n", "\n", "# fibonacci_2.py: example of a fibonacci number\n", "# try to run it with %python3 fibonacci_2.py\n", "# %python3 fibonacci_2.py -h\n", "# %python3 fibonacci_2.py 12\n", "# %python3 fibonacci_2.py 12 -o\n", "# %python3 fibonacci_2.py 12 -o\n", "\n", "\n", "import argparse\n", "\n", "def fib(n):\n", " a, b = 0, 1\n", " for i in range(n):\n", " a, b = b, a+b\n", " return a\n", "\n", "\n", "def Main():\n", " parser = argparse.ArgumentParser()\n", " parser.add_argument(\"num\", help=\"The Fibonacci number you wish to calculate.\", type=int)\n", " parser.add_argument(\"-o\", \"--output\", help=\"Output the result to a file\", action=\"store_true\")\n", "\n", " args = parser.parse_args()\n", " \n", " result = fib(args.num)\n", " print(\"The \" + str(args.num)+ \"th fib number is \" + str(result))\n", "\n", " if args.output: # If this is true, write it to a file fibonacci.txt\n", " f = open(\"fibonacci.txt\",\"a\")\n", " f.write(str(result) + '\\n')\n", "\n", "if __name__ == '__main__':\n", " Main()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Mutually Exclusive Arguments\n", "\n", "- You can select one option only, but not both\n", "- This can be done with a group\n", "- Automatically generates an output informing the user to select one, should they try to use both (or more)\n", "- Let's modify the program so that user can choose either have a verbose output or a quiet output but not both via *mutually exclusive group*\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# MAKE SURE TO RUN IN THE TERMINAL !!!!!!\n", "\n", "# fibonacci_3.py: example of a fibonacci number\n", "# try to run it with %python3 fibonacci_3.py\n", "# %python3 fibonacci_3.py -h\n", "# %python3 fibonacci_3.py 12\n", "# %python3 fibonacci_3.py 12 -o\n", "# %python3 fibonacci_3.py 12 -q\n", "\n", "\n", "\n", "import argparse\n", "\n", "def fib(n):\n", " a, b = 0, 1\n", " for i in range(n):\n", " a, b = b, a+b\n", " return a\n", "\n", "\n", "def Main():\n", " parser = argparse.ArgumentParser()\n", " group = parser.add_mutually_exclusive_group()\n", " group.add_argument(\"-v\", \"--verbose\", action=\"store_true\")\n", " group.add_argument(\"-q\", \"--quiet\", action=\"store_true\")\n", " parser.add_argument(\"num\", help=\"The Fibonacci number you wish to calculate.\", type=int)\n", " parser.add_argument(\"-o\", \"--output\", help=\"Output the result to a file\", action=\"store_true\")\n", "\n", " args = parser.parse_args()\n", " \n", " result = fib(args.num)\n", " if args.verbose:\n", " print(\"The \" + str(args.num)+ \"th fib number is \" + str(result))\n", " elif args.quiet:\n", " print(result)\n", " else:\n", " print(\"Fib(\"+str(args.num)+\") = \" + str(result))\n", "\n", " if args.output:\n", " f = open(\"fibonacci.txt\",\"a\")\n", " f.write(str(result) + '\\n')\n", "\n", "if __name__ == '__main__':\n", " Main()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Summary of Argparse\n", "\n", "- Argparse makes it easy to handle command line options and arguments\n", "- It generates and formats usage and help\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Topic 3: Multithreading in Python\n", "\n", "- Threads are separate concurrent running programs\n", "- However they run within one *process*, meaning they can share data between one another easier than separate programs or processes\n", "- One challenge of threads is that they are not easy to manage. Especially when a piece of data is being used by more than one thread.\n", "- Threads can be used just for quick tasks like calculating a result from an algorithm or for running slow processes in the background while the program continues.\n", "Given example.\n", "- We can create many threads to try and find an answer faster. E.g., you need to hash 100 passwords with md5. One can have 5-10 threads each hashing a password making the total time 5-10 times faster!\n", "\n", "\n", "\n", "## Example: timer program\n", "\n", "- A basic timer program is essentially a \"hello world\" program for threading\n", "- Each timer thread will output the current time\n", "- THen wait for a certain time before outputting again" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# import threading and timer modules\n", "\n", "from threading import Thread\n", "import time\n", "\n", "def timer(name, delay, repeat): # takes 3 arguments: name, delay, and # of times to repeat\n", " print(\"Timer: \" + name + \" Started\")\n", " while repeat > 0:\n", " time.sleep(delay) # delay for some time for this thread\n", " print(name + \": \" + str(time.ctime(time.time()))) # print current time for this thread\n", " repeat -= 1\n", " print(\"Timer: \" + name + \" Completed\")\n", "\n", "def Main(): # we start 2 threads\n", " t1 = Thread(target=timer, args=(\"Timer1\", 1, 5)) # initialize a thread\n", " t2 = Thread(target=timer, args=(\"Timer2\", 2, 5))\n", " t1.start() # start the thread\n", " t2.start()\n", " \n", " print(\"Main complete\")\n", "\n", "if __name__ == '__main__':\n", " Main()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Asynchronous tasks\n", "\n", "- Some tasks can take a long time. Input and output for example can take a long time.\n", "- Some programs are required to be real time. So we can setup threads to run in the background to write a file or search for items while the user can still interact with the interface or commandline.\n", "\n", "\n", "## Custom threads\n", "\n", "- We can make our own thread subclasses\n", "- These are useful for making task specific threads that we can simply reuse as well as add features to the thread.\n", "\n", "# Example\n", "- Let's write a simple threading program that writes a file in the background\n", "- Another \"customized\" thread will be created to take a string and safe it to a file in the background\n", "- We will make it sleep for a second so we can see it is working" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import time\n", "import threading\n", "\n", "class AsyncWrite(threading.Thread): # define my AsyncWrite class\n", " def __init__(self, text, out):\n", " threading.Thread.__init__(self)\n", " self.text = text\n", " self.out = out\n", "\n", " def run(self):\n", " f = open(self.out, \"a\") # open and append text to a file\n", " f.write(self.text + '\\n') \n", " f.close()\n", " time.sleep(2)\n", " print(\"Finished Background file write to \" + self.out)\n", " \n", "\n", "def Main():\n", " message = input(\"Enter a string to store:\" )\n", " background = AsyncWrite(message, 'out.txt')\n", " background.start() # start my thread class and run the class \"run\" function\n", " print(\"The program can continue while it writes in another thread\")\n", " print(\"100 + 400 = \", 100+400)\n", "\n", " background.join() # the main() will wait for the background to finish, then continue\n", " \n", " \n", " print(\"Waited until thread was complete\")\n", "\n", "if __name__ == '__main__':\n", " Main()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Locks and semaphore\n", "\n", "- We use a *lock* to \"lock access\" to one thread\n", "- Because threads run simultaneously, there is no guarantee that threads won't try to use a variable at the same time\n", "\n", "## Modified Timer program\n", "- We now add a loct that will lock the thread currently printing the time\n", "- This program shows an example of what a lock does" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "\n", "import threading \n", "import time\n", "\n", "tLock = threading.Lock() # create a lock\n", "\n", "def timer(name, delay, repeat): # takes 3 arguments: name, delay, and # of times to repeat\n", " print(\"Timer: \" + name + \" Started\")\n", " tLock.acquire() # the thread tries to get a lock\n", " print(name + \" has acquired the lock\")\n", " while repeat > 0:\n", " time.sleep(delay) # delay for some time for this thread\n", " print(name + \": \" + str(time.ctime(time.time()))) # print current time for this thread\n", " repeat -= 1\n", " print(name + \" is releasing the lock\")\n", " tLock.release()\n", " print(\"Timer: \" + name + \" Completed\")\n", "\n", "def Main(): # we start 2 threads\n", " t1 = threading.Thread(target=timer, args=(\"Timer1\", 1, 5)) # initialize a thread\n", " t2 = threading.Thread(target=timer, args=(\"Timer2\", 2, 5))\n", " t1.start() # start the thread\n", " t2.start()\n", " \n", " print(\"Main complete\")\n", "\n", "if __name__ == '__main__':\n", " Main()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Semaphores\n", "\n", "- Semaphores like locks restrict access to a thread.\n", "- However, semaphores can allow more than one lock to be acquired.\n", "- You may have 10 threads running, but you only want 2 or 3 to have access to a piece of data at a time.\n", "\n", "## When to use threading\n", "\n", "- Say you are writing a GUI, then you want at least two threads. One for the GUI and the other one to do all the work in the background. This can avoid the GUI to\n", " be unresponsive.\n", "- If the program is running on a multi-core machines, then there will be performance improvement \n", "- It’s great for servers that deal with TCP connections to be multi-Threaded as you want to be able to handle more than 1 request at a time" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Topic 4: Networking\n", "\n", "- Many network programs are written using the client/server model\n", "- Client: A program which initiates requests, e.g, web browser\n", "- Server: A program which waits for requests, processes them, and replies to requests, e.g., web server\n", "- There are other models, like peer-to-peer (P2P), for network applications. E.g., Skype, game servers, BitTorrent, etc.\n", "- In the P2P model, clients connect to other clients without the use of a centralized server\n", "\n", "## Basic terminologies\n", "- IP addresses\n", "- Port, e.g, port 80 for web server (port 1-1024 are reserved). So try to use something above 1024 but less than 65535.\n", "\n", "\n", "## Sockets\n", "\n", "- Sockets are the programming abstractions for network connections\n", "- Sockets allow two end-points (e.g., client/server or client/client) to communicate in a bidirectional manner\n", "- Once they are connected, both sides can transmit\n", "- use sockets to send data and receive data\n", "- Sockets support the common transport protocols, e.g., TCP and UDP.\n", "- Methods:\n", " - *socket(socket_family, socket_type)*
\n", " The constructer creates a new socket.\n", " - *bind((hostname,port))*
\n", " bind takes a turple of a host address and port\n", " - *listen()*
\n", " starts listening for TCP connections\n", " - *accept()*
\n", " Accepts a connection when found.(returns new socket)\n", " - *connect((hostname,port))*
\n", " Takes a turple of the address and port.\n", " - *recv(buffer)*
\n", " Tries to grab data from a TCP connection. The buffer size determines
\n", " how many bytes of data to receive at a time.\n", " - *send(bytes)*
\n", " Attempts to send the bytes given to it.\n", " - *close()*
\n", " Closes a socket/connection and frees the port.\n", " \n", "## TCP\n", "\n", "- Transmission Control Protocol.\n", "- Reliable Connection based Protocol\n", "- Ordered & Error checked (simple checksum)\n", "- Used by Web Browsers, Email, SSH, FTP, etc\n", "\n", "## Example\n", "\n", "- Lets create a basic client server program that uses TCP to connect and send text to a server. The server then replies with that text capitalized.\n", "- We will build the Server first (tcpserver.py) then the Client (tcpclient.py)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# RUN ON TERMINAL: filename is tcpserver.py\n", "\n", "# SERVER\n", "\n", "import socket\n", "\n", "def Main():\n", " host = '127.0.0.1'\n", " port = 5000\n", "\n", " s = socket.socket()\n", " s.bind((host,port))\n", "\n", " s.listen(1)\n", " c, addr = s.accept()\n", " print(\"Connection from: \" + str(addr))\n", " while True:\n", " data = c.recv(1024).decode('utf-8')\n", " if not data:\n", " break\n", " print(\"from connected user: \" + data)\n", " data = data.upper()\n", " print(\"sending: \" + data)\n", " c.send(data.encode('utf-8'))\n", " c.close()\n", "\n", "if __name__ == '__main__':\n", " Main()\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# RUN ON TERMINAL: filename is tcpclient.py\n", "# Client code\n", "import socket\n", "\n", "def Main():\n", " host = '127.0.0.1'\n", " port = 5000\n", "\n", " s = socket.socket()\n", " s.connect((host, port))\n", "\n", " message = input(\"-> \")\n", " while message != 'q':\n", " s.send(message.encode('utf-8'))\n", " data = s.recv(1024).decode('utf-8')\n", " print('Received from server: ' + data)\n", " message = input(\"-> \")\n", " s.close()\n", "\n", "if __name__ == '__main__':\n", " Main()\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# C Extension\n", "\n", "- Python can work with C programming language (and other languages).\n", "- Creation of wrappers which bind python objects to C functions.\n", "- There is many sub-topic’s of C Extensions\n", "- Why is this useful? Perhaps you already have a library of C functions that you want to turn into a python module to use.\n", "\n", "\n", "## Python Header\n", "\n", "- Everything in the Python header starts with the prefix Py or PY, we use the header file for C extension.\n", "- The **PyObject** type is always used as a pointer and it handles all the data parsing between Python and C\n", "- E.g., static PyObject\\* myFunc(PyObject\\* self)\n", "\n", "## Python.h Functions\n", "\n", "- PyArg_ParseTuple(args, format, …)
\n", " Handles getting the arguments from Python.\n", "- Py_BuildValue(format, …)
\n", " Handles turning values into PyObject pointers.\n", "- PyModule_Create(moduleDef)
\n", " Initializes the module and wraps the method pointers using the module definitions\n", "- If you want your function to return nothing, return the Py_None value.\n", "\n", "## PyMethodDef\n", "\n", "- The PyMethodDef structure is one of the most critical parts because the compiler won't pick up any errors inside.\n", "- The structure **must always end with terminating NULL and 0 values**. \\{NULL, NULL, 0, NULL\\}\n", "- Here we tell Python if the function has argument, no arguments or arguments and keywords\n", "\n", "## MethodDef Example\n", "\n", "- static PyMethodDef myMethods[] = {
\n", "       \n", " {\"func1\", func1, METH_NOARGS, \"func1 doc\"},
\n", "      \n", " {\"func2\", func2, METH_VARARGS, \"func2 doc\"},
\n", "      \n", " {NULL, NULL, 0, NULL}
\n", "    }\n", "- Pattern: pyMethodName, function, functionType, Doc\n", "\n", "## PyModuleDef (for Python 3)\n", "\n", "- The PyModuleDef structure is what we use to tell the PyModule_Create() function what information to use to create the module\n", "- We need to give it a name, documentation, tell Python of we will control the module state and thte structure of methods to include in the module\n", "- static struct PyModuleDef myModule = {
\n", "       \n", " PyModuleDef_HEAD_INIT,
\n", "       \n", " \"myModule\", \\#name of module.
\n", "       \n", " \"Fibonacci Module\", # Module Docs
\n", "       \n", " -1, # -1, the module state is global
\n", "       \n", " myMethods # method def structure
\n", "    \n", " };\n", " \n", "## Our C program\n", "\n", "- sudo apt-get install python-dev (only in **Ubuntu**)\n", "- Lets create a simple Fibonacci function in C and create the bindings for a module\n", "- We will also add a version function so we can see a function that doesn't take arguments\n", "- Lets call it myModule.c\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Examine the file \"myModule.c\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "/* Our myModule.c */\n", "\n", "#include \n", " \n", "int Cfib(int n)\n", "{\n", " if (n < 2)\n", " return n;\n", " else\n", " return Cfib(n-1) + Cfib(n-2);\n", "}\n", " \n", "/* This is our wrapper function */\n", "\n", "static PyObject* fib(PyObject* self, PyObject* args)\n", "{\n", " int n;\n", " \n", "/* for error check */\n", "/* pass args from Python to C */\n", " if (!PyArg_ParseTuple(args, \"i\", &n)) /* check the interger argument */\n", " return NULL;\n", " \n", " return Py_BuildValue(\"i\", Cfib(n)); /* turn a C value into Python value */\n", "}\n", "\n", "/* this one does not have any argument */\n", "static PyObject* version(PyObject* self)\n", "{\n", " return Py_BuildValue(\"s\", \"Version 1.0\"); /* return a string */\n", "}\n", "\n", "/* our method definition */\n", "\n", "static PyMethodDef myMethods[] = {\n", " {\"fib\", fib, METH_VARARGS, \"Calculate the Fibonacci numbers.\"},\n", " {\"version\", (PyCFunction) version, METH_NOARGS, \"Returns the version.\"},\n", " {NULL, NULL, 0, NULL}\n", "};\n", " \n", "static struct PyModuleDef myModule = {\n", " PyModuleDef_HEAD_INIT,\n", " \"myModule\", /* name of module. */\n", " \"Fibonacci Module\",\n", " -1,\n", " myMethods\n", "};\n", "\n", "\n", "PyMODINIT_FUNC PyInit_myModule(void)\n", "{\n", " return PyModule_Create(&myModule);\n", "}\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The setup script\n", "- There is a utility module that comes with Python to make the building/linking easy\n", "- Lets call out setup script setup.py\n", "- We our extension, it will be outputed into a build directory\n", "- We need to copy the *module.so* into the directory of our python code (or to the Python libs folder). THem we can do the *import*\n", "- Let's create a test.py.program\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# setup.py\n", "\n", "from distutils.core import setup, Extension\n", "\n", "# variable to store our module\n", "module = Extension('myModule', sources = ['myModule.c'])\n", "\n", "setup (name = 'PackageName',\n", " version = '1.0',\n", " description = 'This is a package for myModule',\n", " ext_modules = [module])\n", "\n", "\n", "# At the terminal, do \"python3 setup.py build\" to create build folder\n", "# Or at the terminal, do \"python3 setup.py install\" to create the bild folder" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using it !\n", "\n", "- Now we have built our extension, it will be output into a *build* directory\n", "\n", "- We need to copy the module.so file into the directory of our python code (or to the Python libs folder). Then we can import it.\n", "\n", "- Let's create a test.py program" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "\n", "import myModule\n", "\n", "print(myModule.fib(10))\n", "print(myModule.version())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# PyCryto\n", "\n", "- Python has a great package for Cryptographic modules.\n", "- It contains Symmetric and Asymmetric Ciphers, Hashing algorithms, Cryptographic Protocols, Public-key encryption and signature algorithms and it’s own crypto-strong random functions.\n", "\n", "\n", "## What is Cryptography\n", "\n", "- Typically Cryptography refers to encryption of plaintext (readable) into ciphertext (unreadable ). And the reverse, decryption of ciphertext into plaintext.\n", "- A Cipher is used with a **Key** which will produce what looks like a random output. The strength of a cryptographic algorithm is measured in how easily an adversary can break the encryption.\n", "\n", "## AES (Advanced Encryption Standard)\n", "\n", "- Symmetric Cipher\n", "- 16 byte block size\n", "- The keys can be 128, 192 or 256 bits long\n", "- Has many block cipher modes. Please refer
\n", " http://en.wikipedia.org/wiki/Block_cipher_mode_of_operation\n", "\n", "\n", "## IV (Initialization Vector)\n", "\n", "- Used to randomize and produce distinct ciphertext’s for certain cipher modes.\n", "- **IMPORTANT:** One should **NEVER** reuse the same IV for two separate encryptions with the same key/password\n", "- The IV can be known for most modes\n", "\n", "## Hashing a Password\n", "\n", "- Because a cipher requires a key of certain length, it’s useful to hash \n", " the users password to produce the same length key every time.\n", "- SHA256 produces a 16 byte output and works great with AES-256.\n", "\n", "## File Encryption Program\n", "\n", "- Lets create a file encrypting and decrypting program.\n", "- Lets call it encrypt.py" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# encrypt.py\n", "\n", "import os\n", "from Crypto.Cipher import AES\n", "from Crypto.Hash import SHA256\n", "from Crypto import Random\n", "\n", "def encrypt(key, filename):\n", " chunksize = 64*1024\n", " outputFile = \"(encrypted)\"+filename\n", " filesize = str(os.path.getsize(filename)).zfill(16)\n", " IV = Random.new().read(16)\n", "\n", " encryptor = AES.new(key, AES.MODE_CBC, IV)\n", "\n", " with open(filename, 'rb') as infile:\n", " with open(outputFile, 'wb') as outfile:\n", " outfile.write(filesize.encode('utf-8'))\n", " outfile.write(IV)\n", "\n", " while True:\n", " chunk = infile.read(chunksize)\n", "\n", " if len(chunk) == 0:\n", " break\n", " elif len(chunk) % 16 != 0:\n", " chunk += b' ' * (16 - (len(chunk) % 16))\n", "\n", " outfile.write(encryptor.encrypt(chunk))\n", "\n", "\n", "\n", "def decrypt(key, filename):\n", " chunksize = 64*1024\n", " outputFile = filename[11:]\n", "\n", " with open(filename, 'rb') as infile:\n", " filesize = int(infile.read(16))\n", " IV = infile.read(16)\n", "\n", " decryptor = AES.new(key, AES.MODE_CBC, IV)\n", "\n", " with open(outputFile, 'wb') as outfile:\n", " while True:\n", " chunk = infile.read(chunksize)\n", "\n", " if len(chunk) == 0:\n", " break\n", "\n", " outfile.write(decryptor.decrypt(chunk))\n", " outfile.truncate(filesize)\n", "\n", "\n", "def getKey(password):\n", " hasher = SHA256.new(password.encode('utf-8'))\n", " return hasher.digest()\n", "\n", "def Main():\n", " choice = input(\"Would you like to (E)ncrypt or (D)ecrypt?: \")\n", "\n", " if choice == 'E':\n", " filename = input(\"File to encrypt: \")\n", " password = input(\"Password: \")\n", " encrypt(getKey(password), filename)\n", " print(\"Done.\")\n", " elif choice == 'D':\n", " filename = input(\"File to decrypt: \")\n", " password = input(\"Password: \")\n", " decrypt(getKey(password), filename)\n", " print(\"Done.\")\n", " else:\n", " print(\"No Option selected, closing...\")\n", "\n", "if __name__ == '__main__':\n", " Main()\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.3" } }, "nbformat": 4, "nbformat_minor": 2 }